Text: How do I share the vocab between the source and target language for machine translation

[thak123]

Hi I do I create combined vocabulary from the source and target fields from the multi30k dataset
. I am interested in having a shared encoder which can represent source as well the target words.

SRC = Field(tokenize=tokenize_de,
            init_token='<sos>',
            eos_token='<eos>',
            lower=True,
            batch_first=True)

TRG = Field(tokenize=tokenize_en,
            init_token='<sos>',
            eos_token='<eos>',
            lower=True,
            batch_first=True)

train_data, valid_data, test_data = Multi30k.splits(exts=('.de', '.en'),
                                                    fields=(SRC, TRG))

SRC.build_vocab(train_data, min_freq=2)
TRG.build_vocab(train_data, min_freq=2)

[zhangguanheng66]

We will re-write the translation dataset in next release and you will be able to generate vocab and pass it to a specific dataset.

For the current version, any thoughts? @mttk

[thak123]

Any quick fix for time being ?

[zhangguanheng66]

Just curious, the src and tgt sequences are different languages. Why do you want them to share vocab?

[thak123]

Trying to model src and tgt in the same space like Universal Sentence Encoder setting. Hoping they will align. Running experiments out of curiosity

[thak123]

I think I will project them into their individual embedding and then try to align them in higher layers

[bentrevett]

@thak123

SRC.build_vocab(train_data.src, train_data.trg, min_freq=2)
TRG.vocab = SRC.vocab

This will cause your SRC and TRG to share a single vocab. You can then use one embedding layer for both languages instead of one per language.

[thak123]

omg... @bentrevett thanks for the code.