Nmt: Does the Vocabulary File have to be provided?

[ArmageddonKnight]

Dear Developers,

I noticed that in nmt.py line 130, there is a line that says the following:

parser.add_argument("--vocab_prefix", type=str, default=None, help="""\
    Vocab prefix, expect files with src/tgt suffixes.If None, extract from
    train files.\
    """)

But when I ignored it, I got the error from this line (line 354 from nmt.py):

## Vocab
# Get vocab file names first
if hparams.vocab_prefix:
    src_vocab_file = hparams.vocab_prefix + "." + hparams.src
    tgt_vocab_file = hparams.vocab_prefix + "." + hparams.tgt
else:
    raise ValueError("hparams.vocab_prefix must be provided.")

So can I safely ignore the vocabulary file? If yes, how should I do that?

Thank you so much for your precious time.

[slamandar]

I use the vocabulary file created by the previous version of seq2seq model, which is created according to the frequency of each word desc, and each line is one word.

[oahziur]

@ArmageddonKnight You can't ignore the vocabulary file. In addition to slamandar@'s notes, we expect the first three tokens in your vocab file are special tokens (UNK, SOS, EOS) by default, you can remove this restriction by setting --check_special_token=False.

You can check here for details.

[ArmageddonKnight]

Thanks for the reply :+1:

[GxvgiuU]

for those who use generate_vocab.py in seq2seq, note not to print out the count after each word.

[mohammedayub44]

Does it matter if we resort the tokens in the vocabulary file (asc or desc) or change in order, for nmt process to work correctly ?