Parlai: Issue using TF-IDF

[jazzminewang]

Logs when trying to run construction / evaluation commands on this README:

python examples/train_model.py -m tfidf_retriever -t personachat -mf /tmp/personachat_tfidf -dt train:ordered -eps 1. 

[ running eval: valid ]
Traceback (most recent call last):
  File "examples/train_model.py", line 16, in <module>
    TrainLoop(opt).train()
  File "/home/ml/jwang301/Development/ParlAI/parlai/scripts/train_model.py", line 557, in train
    v_report = run_eval(valid_world, opt, 'valid', write_log=True)
  File "/home/ml/jwang301/Development/ParlAI/parlai/scripts/train_model.py", line 166, in run_eval
    valid_world.parley()
  File "/home/ml/jwang301/Development/ParlAI/parlai/core/worlds.py", line 249, in parley
    acts[1] = agents[1].act()
  File "/home/ml/jwang301/Development/ParlAI/parlai/agents/tfidf_retriever/tfidf_retriever.py", line 225, in act
    doc_ids, doc_scores = self.ranker.closest_docs(
AttributeError: 'TfidfRetrieverAgent' object has no attribute 'ranker'.

Going through the code, it looks like it's looking for a file at "/tmp/personachat_tfidf.tfidf.npz" in order to initialize the ranker attribute - but I have no such file.

[klshuster]

Hi Jasmine,

When you run that script above, before [ running eval: valid ] is printed, do you see anything that says [ Saving to /tmp/personachat_tfidf.tfidf ]? The script should build and save the tfidf matrix to /tmp/personachat_tfidf.tfidf.npz

[jazzminewang]

This is my entire log. The NO RANKER FOUND is just an else statement I added to see if it found anything at that file path.

[ Main ParlAI Arguments: ] [ datapath: /home/ml/jwang301/Development/ParlAI/data ] [ datatype: train:ordered ] [ download_path: /home/ml/jwang301/Development/ParlAI/downloads ] [ hide_labels: False ] [ image_mode: raw ] [ numthreads: 1 ] [ show_advanced_args: False ] [ task: personachat ] [ Batching Arguments: ] [ batchsize: 1 ] [ include_labels: True ] [ PytorchData Arguments: ] [ batch_length_range: 5 ] [ batch_sort_cache_type: pop ] [ batch_sort_field: text ] [ numworkers: 4 ] [ pytorch_context_length: -1 ] [ pytorch_datapath: None ] [ pytorch_include_labels: True ] [ pytorch_preprocess: False ] [ pytorch_teacher_batch_sort: False ] [ pytorch_teacher_dataset: None ] [ pytorch_teacher_task: None ] [ shuffle: False ] [ ParlAI Model Arguments: ] [ dict_class: None ] [ model: tfidf_retriever ] [ model_file: /tmp/personachat_tfidf ] [ Training Loop Arguments: ] [ dict_build_first: True ] [ display_examples: False ] [ eval_batchsize: None ] [ evaltask: None ] [ load_from_checkpoint: False ] [ max_train_time: -1 ] [ num_epochs: 1.0 ] [ save_after_valid: False ] [ save_every_n_secs: -1 ] [ validation_cutoff: 1.0 ] [ validation_every_n_epochs: -1 ] [ validation_every_n_secs: -1 ] [ validation_max_exs: -1 ] [ validation_metric: accuracy ] [ validation_metric_mode: None ] [ validation_patience: 10 ] [ validation_share_agent: False ] [ Tensorboard Arguments: ] [ tensorboard_comment: ] [ tensorboard_log: False ] [ tensorboard_metrics: None ] [ tensorboard_tag: None ] [ Dictionary Loop Arguments: ] [ dict_include_test: False ] [ dict_include_valid: False ] [ dict_maxexs: -1 ] [ log_every_n_secs: 2 ] [ Dictionary Arguments: ] [ bpe_debug: False ] [ dict_endtoken: __end__ ] [ dict_file: None ] [ dict_initpath: None ] [ dict_language: english ] [ dict_lower: False ] [ dict_max_ngram_size: -1 ] [ dict_maxtokens: -1 ] [ dict_minfreq: 0 ] [ dict_nulltoken: __null__ ] [ dict_starttoken: __start__ ] [ dict_textfields: text,labels ] [ dict_tokenizer: re ] [ dict_unktoken: __unk__ ] [ ParlAI Image Preprocessing Arguments: ] [ image_cropsize: 224 ] [ image_size: 256 ] [ Retriever Arguments: ] [ index_by_int_id: True ] [ remove_title: False ] [ retriever_hashsize: 16777216 ] [ retriever_mode: values ] [ retriever_ngram: 2 ] [ retriever_num_retrieved: 5 ] [ retriever_numworkers: None ] [ retriever_tokenizer: simple ] [ tfidf_context_length: -1 ] [ tfidf_include_labels: True ] [ building dictionary first... ] [ dictionary already built .] NO RANKER FOUND :( at /tmp/personachat_tfidf.tfidf.npz [creating task(s): personachat] Creating single task [loading fbdialog data:/home/ml/jwang301/Development/ParlAI/data/Persona-Chat/personachat/train_self_original.txt] [ training... ] [ time:74.0s total_exs:65722 epochs:1.0 time_left:0s ] {'exs': 1, 'accuracy': 1.0, 'f1': 1.0, 'bleu': 1.0} [ num_epochs completed:1.0 time elapsed:74.48239541053772s ] [creating task(s): personachat] Creating single task [loading fbdialog data:/home/ml/jwang301/Development/ParlAI/data/Persona-Chat/personachat/valid_self_original.txt] [ running eval: valid ] Traceback (most recent call last): File "examples/train_model.py", line 16, in <module> TrainLoop(opt).train() File "/home/ml/jwang301/Development/ParlAI/parlai/scripts/train_model.py", line 557, in train v_report = run_eval(valid_world, opt, 'valid', write_log=True) File "/home/ml/jwang301/Development/ParlAI/parlai/scripts/train_model.py", line 166, in run_eval valid_world.parley() File "/home/ml/jwang301/Development/ParlAI/parlai/core/worlds.py", line 249, in parley acts[1] = agents[1].act() File "/home/ml/jwang301/Development/ParlAI/parlai/agents/tfidf_retriever/tfidf_retriever.py", line 228, in act doc_ids, doc_scores = self.ranker.closest_docs( AttributeError: 'TfidfRetrieverAgent' object has no attribute 'ranker' .

[jazzminewang]

Also - is there a dict of contractions that this model is using to map out data? All the convai2 baseline validation outputs have contractions.

For example, "you'd" is expanded into "you would" in the TF_IDF output.

[klshuster]

Can you please change the model file path (e.g. /tmp/personachat_tfidf_1), run the command again, and then paste the whole output?

[stephenroller]

Also - is there a dict of contractions that this model is using to map out data? All the convai2 baseline validation outputs have contractions.

For example, "you'd" is expanded into "you would" in the TF_IDF output.

This was a regrettable preprocessing step done in PersonaChat, and is forever baked into the data.

[jazzminewang]

@klshuster that works! just a documentation update then. Thanks so much! :)

[jazzminewang]

Also - is there a dict of contractions that this model is using to map out data? All the convai2 baseline validation outputs have contractions.
For example, "you'd" is expanded into "you would" in the TF_IDF output.

This was a regrettable preprocessing step done in PersonaChat, and is forever baked into the data.

@stephenroller Do you have a mapping of these contractions available to share? As I'm labelling this dataset with the convai2 output I'd like to keep it consistent.

[stephenroller]

I haven't seen the list anywhere. I only know about this from also noticing the very unnatural contractions when working with the data myself, and asking Jason.

[jazzminewang]

@jaseweston would you be able to provide a mapping of these contractions?

[stephenroller]

I just spent some effort looking in internal and external repos, and couldn't find the list anywhere. @saizheng might have the list of contractions used in personachat, but I really believe it has been lost to the sands of time :(