Parlai: Invalid value error because of labels data on Movie Dialog Dataset

[jojonki]

Hi, I tried to train Memory Networks with #moviedd-qa. But I encountered below problem. I think labels data on Movie Dialog is invalid because it contains multiple entities. In cands, it is from entities.txt and only contains _single_ vocab like 'Three Kings' and 'hungury'.

• python: 3.6.1
• Ubuntu 16.04, 64bit
• ParlAI: f43343f4bf, Wed Sep 20 17:15:55

python examples/train_model.py -m memnn -t "#moviedd-qa" --hops 1 -e 5 -bs 1 --gpu 0
[ Main ParlAI Arguments: ] 
[  task: #moviedd-qa ]
[  download_path: /home/jonki/work/clean/ParlAI/downloads ]
[  datatype: train ]
[  image_mode: raw ]
[  numthreads: 1 ]
[  batchsize: 1 ]
[  datapath: /home/jonki/work/clean/ParlAI/data ]
[ ParlAI Model Arguments: ] 
[  model: memnn ]
[  model_file: None ]
[  dict_class: None ]
[ Dictionary Arguments: ] 
[  dict_file: None ]
[  dict_initpath: None ]
[  dict_language: english ]
[  dict_max_ngram_size: -1 ]
[  dict_minfreq: 0 ]
[  dict_nulltoken: __NULL__ ]
[  dict_endtoken: __END__ ]
[  dict_unktoken: __UNK__ ]
[  dict_starttoken: __START__ ]
[  dict_maxexs: 100000 ]
[ MemNN Arguments: ] 
[  learning_rate: 0.01 ]
[  embedding_size: 128 ]
[  hops: 1 ]
[  mem_size: 100 ]
[  time_features: True ]
[  position_encoding: False ]
[  output: rank ]
[  rnn_layers: 2 ]
[  dropout: 0.1 ]
[  optimizer: adam ]
[  no_cuda: False ]
[  gpu: 0 ]
[ Training Loop Arguments: ] 
[  evaltask: None ]
[  display_examples: False ]
[  num_epochs: 5.0 ]
[  max_train_time: -1 ]
[  log_every_n_secs: 2 ]
[  validation_every_n_secs: -1 ]
[  validation_max_exs: -1 ]
[  validation_patience: 5 ]
[  validation_metric: accuracy ]
[  dict_build_first: True ]
[ building dictionary first... ]
Tried to build dictionary but `--dict-file` is not set. Set this param so the dictionary can be saved.
[ Using CUDA ]
[creating task(s): moviedialog:Task:1]
[building data: /home/jonki/work/clean/ParlAI/data/MovieDialog]

unpacking moviedialog.tar.gz
unpacking p6tyohj.tgz
[loading fbdialog data:/home/jonki/work/clean/ParlAI/data/MovieDialog/movie_dialog_dataset/task1_qa/task1_qa_train.txt]
[ training... ]
[ time:1006s parleys:1 total_exs:1 time_left:483939705s ] {'total': 1, 'accuracy': 0, 'f1': 0, 'hits@k': {1: 0, 5: 0, 10: 0, 50: 0, 100: 0}}
[ time:2008s parleys:2 total_exs:2 time_left:483044539s ] {'total': 1, 'accuracy': 0, 'f1': 0, 'hits@k': {1: 0, 5: 0, 10: 0, 50: 0, 100: 0}}
[ time:3059s parleys:3 total_exs:3 time_left:490416900s ] {'total': 1, 'accuracy': 0, 'f1': 0, 'hits@k': {1: 0, 5: 0, 10: 0, 50: 0, 100: 0}}
Traceback (most recent call last):
  File "/home/jonki/work/clean/ParlAI/examples/train_model.py", line 217, in <module>
    main()
  File "/home/jonki/work/clean/ParlAI/examples/train_model.py", line 131, in main
    world.parley()
  File "/home/jonki/work/clean/ParlAI/parlai/core/worlds.py", line 243, in parley
    acts[1] = agents[1].act()
  File "/home/jonki/work/clean/ParlAI/parlai/agents/memnn/memnn.py", line 315, in act
    return self.batch_act([self.observation])[0]
  File "/home/jonki/work/clean/ParlAI/parlai/agents/memnn/memnn.py", line 306, in batch_act
    predictions = self.predict(xs, cands, ys)
  File "/home/jonki/work/clean/ParlAI/parlai/agents/memnn/memnn.py", line 141, in predict
    label_inds = [cand_list.index(self.labels[i]) for i, cand_list in enumerate(cands)]
  File "/home/jonki/work/clean/ParlAI/parlai/agents/memnn/memnn.py", line 141, in <listcomp>
    label_inds = [cand_list.index(self.labels[i]) for i, cand_list in enumerate(cands)]
ValueError: 'Morgan Freeman, Mickey Rourke, Ellen Barkin' is not in list

[jojonki]

In this situation, label_candidates looks strange because it includes multi entities in one element, e.g, label_candidates[0] = ('Top Hat, Kitty Foyle, The Barkleys of Broadway'). Should it be like label_candidates[0] = ('Top Hat'), label_candidates[1] = ('Kitty Foyle'), label_candidates[2] = 'The Barkleys of Broadway') ?
The above cands is a array of entities, so this error exactly occurs when the answer contains multi entities because of key error. Like below episode.

1 what terms are applicable to My Left Foot? bd-r, cerebral, fiona shaw, jim sheridan, cerebral palsy

And some entities contain ,, so it is not so simple to split label_candidates by ,. For example, Bashu, the Little Stranger is exactly one entity. So it is not easy to create correct label_candidates. Below is episode it is not easy to handle, because Larceny, Inc. contains ,, but one entity.

1 what does Jane Wyman star in? The Lost Weekend, Stage Fright, All That Heaven Allows, Pollyanna, The Yearling, Magnificent Obsession, Johnny Belinda, Larceny, Inc., Magic Town, Here Comes the Groom

[jojonki]

Does anyone try this issue?

[jaseweston]

hi! sorry for the delay. indeed, the labels look wrong on that dataset, just as you described. we need to fix them..

[jojonki]

@jaseweston Hi, thank you for your reply. Yes, it is easier to use it if the candidates were split correctly.

[jojonki]

labels in this dataset probably should be split by | regarding to below code.
https://github.com/facebookresearch/ParlAI/blob/master/parlai/core/fbdialog_teacher.py#L190

[alexholdenmiller]

still working on this, ran into a few issues that make the parsing more complicated. thanks again for flagging.

[jojonki]

@alexholdenmiller Thank you!
I found another problem of entities which contains two ,. There are dance, girl and Dance, Girl, Dance in entities.txt. But in training set, below candidate_labels contain Dance, Girl which doesn't exist in entities.txt. It probably comes from this problem.

1 Dorothy Arzner directed which movies? Dance, Girl, Christopher Strong, The Last of Mrs. Cheyney, The Bride Wore Red, Craig's Wife, Honor Among Lovers

According to movie_kb.txt, Dorothy Arzner directed Dance, Girl, Dance, not dance nor girl. I hope it would be helpful.

[alexholdenmiller]

Yes exactly, that's what has given me pause. I'll have an updated version soon which resolves that!

[alexholdenmiller]

data has been fixed, #325 will force parlai to download a new copy of the data, but you can immediately access the fixed data by just doing rm -rf data/MovieDialog

[jojonki]

Thank you! I also confirmed #326 .

[jojonki]

movie_kb.txt is still using , for entities. Do you have a plan to update this file? I want to use this file to reproduce your study. And movie_kb.txt looks same as wiki_entities_kb.txt in http://www.thespermwhale.com/jaseweston/babi/movieqa.tar.gz So if you update the file, both should may be updated.

• Key-Value Memory Networks for Directly Reading Documents