Parlai: Model zoo agents should have an args to evaluate on training set

[koustuvsinha]

Is your feature request related to a problem? Please describe.

Somewhat related to my previous issue #1907, now my use case is extracting model predictions on _both_ train and test set. For Seq2Seq trained model (from model zoo), I can extract the predictions on the test set just fine. However, on setting --dt train, it defaults to training mode and fails on trying to find the optimizer:

[ Main ParlAI Arguments: ]
[  batchsize: 1 ]
[  datapath: /private/home/koustuvs/mlp/parlai_koustuvs/data ]
[  datatype: train:ordered ]
[  download_path: /private/home/koustuvs/mlp/parlai_koustuvs/downloads ]
[  hide_labels: False ]
[  image_mode: raw ]
[  init_opt: None ]
[  multitask_weights: [1] ]
[  numthreads: 1 ]
[  show_advanced_args: False ]
[  task: convai2 ]
[ ParlAI Model Arguments: ]
[  dict_class: parlai.agents.legacy_agents.seq2seq.dict_v0:DictionaryAgent ]
[  init_model: None ]
[  model: legacy:seq2seq:0 ]
[  model_file: /private/home/koustuvs/mlp/parlai_koustuvs/data/models/convai2/seq2seq/convai2_self_seq2seq_model ]
[ PytorchData Arguments: ]
[  batch_length_range: 5 ]
[  batch_sort_cache_type: pop ]
[  batch_sort_field: text ]
[  numworkers: 4 ]
[  pytorch_context_length: -1 ]
[  pytorch_datapath: None ]
[  pytorch_include_labels: True ]
[  pytorch_preprocess: False ]
[  pytorch_teacher_batch_sort: False ]
[  pytorch_teacher_dataset: None ]
[  pytorch_teacher_task: None ]
[  shuffle: False ]
[ ParlAI Image Preprocessing Arguments: ]
[  image_cropsize: 224 ]
[  image_size: 256 ]
[ Seq2Seq Arguments: ]
[  attention: none ]
[  attention_length: 48 ]
[  attention_time: post ]
[  beam_log_freq: 0.0 ]
[  beam_size: 1 ]
[  bidirectional: False ]
[  decoder: same ]
[  dropout: 0.1 ]
[  embedding_type: random ]
[  embeddingsize: 128 ]
[  gpu: -1 ]
[  gradient_clip: 0.1 ]
[  hiddensize: 128 ]
[  history_replies: label_else_model ]
[  init_model: None ]
[  learningrate: 1 ]
[  lookuptable: unique ]
[  momentum: -1 ]
[  no_cuda: False ]
[  numlayers: 2 ]
[  numsoftmax: 1 ]
[  optimizer: sgd ]
[  person_tokens: False ]
[  rank_candidates: False ]
[  report_freq: 0.001 ]
[  rnn_class: lstm ]
[  softmax_layer_bias: False ]
[  topk: 1 ]
[  truncate: -1 ]
[ Dictionary Arguments: ]
[  bpe_debug: False ]
[  dict_endtoken: __end__ ]
[  dict_file: None ]
[  dict_initpath: None ]
[  dict_language: english ]
[  dict_lower: False ]
[  dict_max_ngram_size: -1 ]
[  dict_maxtokens: -1 ]
[  dict_minfreq: 0 ]
[  dict_nulltoken: __null__ ]
[  dict_starttoken: __start__ ]
[  dict_textfields: text,labels ]
[  dict_tokenizer: re ]
[  dict_unktoken: __unk__ ]
[ warning: overriding opt['model'] to legacy:seq2seq:0 (previously: seq2seq )]
[ warning: overriding opt['model_file'] to /private/home/koustuvs/mlp/parlai_koustuvs/data/models/convai2/seq2seq/convai2_self_seq2seq_model (previously: /Users/edinan/ParlAI/data/models/convai2/seq2seq/convai2_self_seq2seq_model )]
[ Using CUDA ]
[ Loading existing model params from /private/home/koustuvs/mlp/parlai_koustuvs/data/models/convai2/seq2seq/convai2_self_seq2seq_model ]
Dictionary: loading dictionary from /private/home/koustuvs/mlp/parlai_koustuvs/data/models/convai2/seq2seq/convai2_self_seq2seq_model.dict
[ num words =  18803 ]
/private/home/koustuvs/miniconda3/envs/dialog/lib/python3.7/site-packages/torch/nn/_reduction.py:43: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
  warnings.warn(warning.format(ret))
[creating task(s): convai2]
[loading fbdialog data:/private/home/koustuvs/mlp/parlai_koustuvs/data/ConvAI2/train_self_original.txt]
  0%|                                                                         | 0/17878 [00:00<?, ?it/s]preinitializing pytorch cuda buffer
/private/home/koustuvs/miniconda3/envs/dialog/lib/python3.7/site-packages/torch/nn/functional.py:1339: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.
  warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.")
Traceback (most recent call last):
  File "data.py", line 662, in <module>
    data.extract_all_models()
  File "data.py", line 624, in extract_all_models
    self.extract_interactions()
  File "data.py", line 543, in extract_interactions
    world.parley()
  File "/private/home/koustuvs/mlp/parlai_koustuvs/parlai/core/worlds.py", line 274, in parley
    acts[1] = agents[1].act()
  File "/private/home/koustuvs/mlp/parlai_koustuvs/parlai/agents/legacy_agents/seq2seq/seq2seq_v0.py", line 890, in act
    return self.batch_act([self.observation])[0]
  File "/private/home/koustuvs/mlp/parlai_koustuvs/parlai/agents/legacy_agents/seq2seq/seq2seq_v0.py", line 855, in batch_act
    predictions, cand_preds = self.predict(xs, ys, cands, cand_inds, is_training)
  File "/private/home/koustuvs/mlp/parlai_koustuvs/parlai/agents/legacy_agents/seq2seq/seq2seq_v0.py", line 709, in predict
    self.zero_grad()
  File "/private/home/koustuvs/mlp/parlai_koustuvs/parlai/agents/legacy_agents/seq2seq/seq2seq_v0.py", line 610, in zero_grad
    self.optimizer.zero_grad()
AttributeError: 'Seq2seqAgent' object has no attribute 'optimizer'

Which I guess is the expected behavior, since you would want the optimizer on training. However, my use case is I just want the predictions of the pretrained model (here seq2seq) on the training set.

On inspecting closely, I think this line is the issue, where you implicitly set is_training to True if you encounter labels in observation dict (which is the case for --dt train anyway). For my case, I just edited this particular line to set is_training to False. Thus a feature request for this model would be to provide a flag which enables setting is_training to false if I want only the predictions on the training set.

[koustuvsinha]

Same for kvmemnn agent, relevant line number. Updating the issue title to reflect it to be a more general use case.

[koustuvsinha]

For kvmenn it seems its not that straightforward hack:

1. Apparently candidates are only built in eval mode. Thus, cands is an empty list
2. This check fails - I think its anyway wrong as now cands is an empty list which is False for first condition ([] is None --> False) and cands[0] fails being an empty list

It seems the observation dict itself doesn't contain label_candidates on training mode.. probably this gets set in the agent?

[koustuvsinha]

I got it working after several hacks, which might be useful in building this flag:

1. In kvmemnn agent file, change is_training to True.
2. cands is now an empty list as we take the candidates from fixedCands, so in this line explicitly test for len(cands)==0 before cands[0] and reorder such that we first test for self.take_next_utt
3. Again, put or len(cands) == 0 before checking for cands[0] is None in this line.
4. In kvmenn modules.py, change ys is not None to be False

[jaseweston]

if im understanding right, you can use: -dt train:evalmode

On Fri, Aug 16, 2019 at 12:15 AM Koustuv Sinha [e-mail]
wrote:

I got it working after several hacks, which might be useful in building
this flag:

1. In kvmemnn agent
   https://github.com/facebookresearch/ParlAI/blob/master/projects/personachat/kvmemnn/kvmemnn.py#L606
   file, change is_training to True.
2. cands is now an empty list as we take the candidates from fixedCands,
   so in this line
   https://github.com/facebookresearch/ParlAI/blob/master/projects/personachat/kvmemnn/kvmemnn.py#L645
   explicitly test for len(cands)==0 before cands[0] and reorder such
   that we first test for self.take_next_utt
3. Again, put or len(cands) == 0 before checking for cands[0] is None in
   this line
   https://github.com/facebookresearch/ParlAI/blob/master/projects/personachat/kvmemnn/kvmemnn.py#L710
   .
4. In kvmenn modules.py
   https://github.com/facebookresearch/ParlAI/blob/master/projects/personachat/kvmemnn/modules.py#L84,
   change ys is not None to be False

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/facebookresearch/ParlAI/issues/1928?email_source=notifications&email_token=ACUOJ6CCH7NPN5S6APBVGOTQEYSVHA5CNFSM4IMCULGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4NTRWI#issuecomment-521877721,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACUOJ6BMYQK6KV5G5WGGR5DQEYSVHANCNFSM4IMCULGA
.

[koustuvsinha]

Bummer, tried changing batchsize > 1 for kvmemnn as it was taking too long (~24hours) for inference on the full training set, and got this error:

RuntimeError: Kvmemnn model does not support batchsize > 1, try training with numthreads > 1 instead.

I will open a separate feature request for that.

[koustuvsinha]

@jaseweston ahh you are right!! yes evalmode serves the purpose perfectly! missed that in the docs.. closing this issue.

[jaseweston]

you csn use hogwild: -nt 40. if you have the cpu..

On Fri, Aug 16, 2019 at 12:20 AM Koustuv Sinha [e-mail]
wrote:

Bummer, tried changing batchsize > 1 for kvmemnn as it was taking too
long (~24hours) for inference on the full training set, and got this error:

RuntimeError: Kvmemnn model does not support batchsize > 1, try training with numthreads > 1 instead.

I will open a separate feature request for that.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/facebookresearch/ParlAI/issues/1928?email_source=notifications&email_token=ACUOJ6BMG34WTKTBRER4DILQEYTHTA5CNFSM4IMCULGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4NTXMI#issuecomment-521878449,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACUOJ6FC7CRFMZOP5ZMFO7DQEYTHTANCNFSM4IMCULGA
.

[koustuvsinha]

@jaseweston nice, i'll try that. many thanks :)

[stephenroller]

Yes evalmode is what you’re looking for.

Kvmemnn won’t be updated to support bs>1, so it’s better to use multi threading.