While I'm implementing a dialogue model with seq2seq model, I got some problems. For example, I can separate two fields into source and target, but I don't wanna make two different vocab set, instead merge them. To explain as a code.
## Create two different fields as 'src' and 'tgt'
src = Field()
tgt = Field()
dataset = torchtext.data.TabularDataset(
path="data_path", format='tsv',
fields=[('src', src), ('tgt', tgt)],
)
## This implementation creates different vocab set
## but they have to be merged as one vocab set
src.build_vocab(dataset)
tgt.build_vocab(dataset)
## After then I can implement merging two vocab set,
## but I guess this makes my codes more dirty.
## I need a function to make more codes clean.
Naively, I can make it in this current version, however, it would be better to add a function to be more functional and clean. Could someone give me a wiser answer? I'm just newbie about this framework, maybe I couldn't find some other options I should have known.
Thanks.
The original conception of Fields was that Vocab objects would correspond one-to-one with Field objects. The easiest thing for you to do would be to use a single Field for both source and target, but you can also do something like
src.build_vocab(dataset.src, dataset.tgt)
tgt.vocab = src.vocab
if you want.
I guess this is what I want to find!
I'll close this issue.
Thanks!
The original conception of
Fields was thatVocabobjects would correspond one-to-one withFieldobjects. The easiest thing for you to do would be to use a singleFieldfor both source and target, but you can also do something likesrc.build_vocab(dataset.src, dataset.tgt) tgt.vocab = src.vocabif you want.
Cool , this method solve my problem!
Most helpful comment
The original conception of
Fields was thatVocabobjects would correspond one-to-one withFieldobjects. The easiest thing for you to do would be to use a singleFieldfor both source and target, but you can also do something likeif you want.