Text: Maybe we need a function to merge vocab set for two different ""Field"".

Created on 13 Feb 2018  路  3Comments  路  Source: pytorch/text

While I'm implementing a dialogue model with seq2seq model, I got some problems. For example, I can separate two fields into source and target, but I don't wanna make two different vocab set, instead merge them. To explain as a code.

## Create two different fields as 'src' and 'tgt'
src = Field()
tgt = Field()

dataset = torchtext.data.TabularDataset(
    path="data_path", format='tsv',
    fields=[('src', src), ('tgt', tgt)],
)

## This implementation creates different vocab set 
## but they have to be merged as one vocab set
src.build_vocab(dataset)
tgt.build_vocab(dataset)

## After then I can implement merging two vocab set, 
## but I guess this makes my codes more dirty. 
## I need a function to make more codes clean.

Naively, I can make it in this current version, however, it would be better to add a function to be more functional and clean. Could someone give me a wiser answer? I'm just newbie about this framework, maybe I couldn't find some other options I should have known.

Thanks.

Most helpful comment

The original conception of Fields was that Vocab objects would correspond one-to-one with Field objects. The easiest thing for you to do would be to use a single Field for both source and target, but you can also do something like

src.build_vocab(dataset.src, dataset.tgt)
tgt.vocab = src.vocab

if you want.

All 3 comments

The original conception of Fields was that Vocab objects would correspond one-to-one with Field objects. The easiest thing for you to do would be to use a single Field for both source and target, but you can also do something like

src.build_vocab(dataset.src, dataset.tgt)
tgt.vocab = src.vocab

if you want.

I guess this is what I want to find!
I'll close this issue.

Thanks!

The original conception of Fields was that Vocab objects would correspond one-to-one with Field objects. The easiest thing for you to do would be to use a single Field for both source and target, but you can also do something like

src.build_vocab(dataset.src, dataset.tgt)
tgt.vocab = src.vocab

if you want.

Cool , this method solve my problem!

Was this page helpful?
0 / 5 - 0 ratings