Rasa: TypeError when I evaluate my NER model

[Nomiluks]

Evaluation Command

!python -m rasa_nlu.test \
    --data export_data_format_train.json \
    --model model/ner_model \
    --mode evaluation

Config_file

pipeline:

• name: "SpacyNLP"
• name: "tokenizer_spacy"
• name: "ner_crf"
• name: "ner_synonyms"
• name: "CRFEntityExtractor"

Error

/home/user/anaconda2/envs/jd_matcher/lib/python3.6/runpy.py:125: RuntimeWarning: 'rasa_nlu.test' found in sys.modules after import of package 'rasa_nlu', but prior to execution of 'rasa_nlu.test'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))

2019-04-30 11:55:12 INFO     rasa_nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-en'.
2019-04-30 11:55:12 INFO     rasa_nlu.training_data.loading  - Training data format of export_data_format_train.json is rasa_nlu
2019-04-30 11:55:12 INFO     rasa_nlu.training_data.training_data  - Training data stats: 
    - intent examples: 0 (0 distinct intents)
    - Found intents: 
    - entity examples: 3 (9 distinct entities)
    - found entities: 'soft_skills', 'technical_skills', 'degree', 'salary', 'job_position', 'degree_major', 'job_location', 'overall_work_experience', 'technology_work_experience'

Traceback (most recent call last):
  File "/home/user/anaconda2/envs/jd_matcher/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/user/anaconda2/envs/jd_matcher/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/user/anaconda2/envs/jd_matcher/lib/python3.6/site-packages/rasa_nlu/test.py", line 1001, in <module>
    main()
  File "/home/user/anaconda2/envs/jd_matcher/lib/python3.6/site-packages/rasa_nlu/test.py", line 995, in main
    cmdline_args.histogram)
  File "/home/user/anaconda2/envs/jd_matcher/lib/python3.6/site-packages/rasa_nlu/test.py", line 744, in run_evaluation
    intent_targets = [None] * test_data.training_examples
TypeError: can't multiply sequence by non-int of type 'list'

[erohmensing]

Thanks for creating a separate issue, I'll take care of it -- I've mentioned the fix in #3070 if you want to patch it locally.

[Nomiluks]

Thanks for creating a separate issue, I'll take care of it -- I've mentioned the fix in #3070 if you want to patch it locally.

thanks. Looking forward

[Nomiluks]

@erohmensing

intent_targets = [None] * len(test_data.training_examples)

@erohmensing I have added following changes in rasa_nlu/test.py code:

# intent_targets = [None] * test_data.training_examples
 intent_targets = [None] * len(test_data.training_examples)

But getting following error can you help me to fix this one?

/home/user/anaconda2/envs/jd_matcher/lib/python3.6/runpy.py:125: RuntimeWarning: 'rasa_nlu.test' found in sys.modules after import of package 'rasa_nlu', but prior to execution of 'rasa_nlu.test'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
2019-05-02 14:29:13 INFO     rasa_nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-en'.
2019-05-02 14:29:13 INFO     rasa_nlu.training_data.loading  - Training data format of export_data_format_test.json is rasa_nlu
2019-05-02 14:29:13 INFO     rasa_nlu.training_data.training_data  - Training data stats: 
    - intent examples: 0 (0 distinct intents)
    - Found intents: 
    - entity examples: 0 (0 distinct entities)
    - found entities: 

2019-05-02 14:29:13 INFO     __main__  - Running model for predictions:
0it [00:00, ?it/s]
2019-05-02 14:29:13 INFO     __main__  - Entity evaluation results:
2019-05-02 14:29:13 INFO     __main__  - Evaluation for entity extractor: CRFEntityExtractor 
Traceback (most recent call last):
  File "/home/user/anaconda2/envs/jd_matcher/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/user/anaconda2/envs/jd_matcher/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/user/anaconda2/envs/jd_matcher/lib/python3.6/site-packages/rasa_nlu/test.py", line 1002, in <module>
    main()
  File "/home/user/anaconda2/envs/jd_matcher/lib/python3.6/site-packages/rasa_nlu/test.py", line 996, in main
    cmdline_args.histogram)
  File "/home/user/anaconda2/envs/jd_matcher/lib/python3.6/site-packages/rasa_nlu/test.py", line 780, in run_evaluation
    report_folder)
  File "/home/user/anaconda2/envs/jd_matcher/lib/python3.6/site-packages/rasa_nlu/test.py", line 421, in evaluate_entities
    merged_targets, merged_predictions)
  File "/home/user/anaconda2/envs/jd_matcher/lib/python3.6/site-packages/rasa_nlu/test.py", line 181, in get_evaluation_metrics
    output_dict=output_dict)
  File "/home/user/.local/lib/python3.6/site-packages/sklearn/metrics/classification.py", line 1568, in classification_report
    name_width = max(len(cn) for cn in target_names)
ValueError: max() arg is an empty sequence

[erohmensing]

Hi @Nomiluks , what happened to your entity examples? Earlier you had this:

2019-04-30 11:55:12 INFO     rasa_nlu.training_data.training_data  - Training data stats: 
    - intent examples: 0 (0 distinct intents)
    - Found intents: 
    - entity examples: 3 (9 distinct entities)
    - found entities: 'soft_skills', 'technical_skills', 'degree', 'salary', 'job_position', 'degree_major', 'job_location', 'overall_work_experience', 'technology_work_experience'

Now it says there are no entity examples. I think that has something to do with the problem.

[Nomiluks]

Hi @Nomiluks , what happened to your entity examples? Earlier you had this:

2019-04-30 11:55:12 INFO     rasa_nlu.training_data.training_data  - Training data stats: 
  - intent examples: 0 (0 distinct intents)
  - Found intents: 
  - entity examples: 3 (9 distinct entities)
  - found entities: 'soft_skills', 'technical_skills', 'degree', 'salary', 'job_position', 'degree_major', 'job_location', 'overall_work_experience', 'technology_work_experience'

Now it says there are no entity examples. I think that has something to do with the problem.

Thanks for pointing this out for me. Actually file was not formatted properly at my end. Issue is fixed !!

[erohmensing]

Awesome 🚀

[erohmensing]

Sorry @Nomiluks, I should have mentioned -- awesome that you have it fixed locally. I've implemented the change and it'll be released in rasa_nlu 0.15.1, so whenever that comes out, don't be afraid of updating and losing your local fix 🙂

[Nomiluks]

Sorry @Nomiluks, I should have mentioned -- awesome that you have it fixed locally. I've implemented the change and it'll be released in rasa_nlu 0.15.1, so whenever that comes out, don't be afraid of updating and losing your local fix

thanks