rasa lint might be a good idea

[koaning]

Description of Problem:
I am playing with rasa, let's say for the first time. The basics worked and I got a new intent working. This worked fine and now I want to add a form. I change a yml/md file here and there and I get this error:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/vincent/Development/koaning-rasa/venv/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/Users/vincent/Development/koaning-rasa/venv/lib/python3.6/site-packages/rasa/__main__.py", line 76, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/vincent/Development/koaning-rasa/venv/lib/python3.6/site-packages/rasa/cli/interactive.py", line 60, in interactive
    zipped_model = train.train(args)
  File "/Users/vincent/Development/koaning-rasa/venv/lib/python3.6/site-packages/rasa/cli/train.py", line 75, in train
    kwargs=extract_additional_arguments(args),
  File "/Users/vincent/Development/koaning-rasa/venv/lib/python3.6/site-packages/rasa/train.py", line 40, in train
    kwargs=kwargs,
  File "uvloop/loop.pyx", line 1417, in uvloop.loop.Loop.run_until_complete
  File "/Users/vincent/Development/koaning-rasa/venv/lib/python3.6/site-packages/rasa/train.py", line 91, in train_async
    kwargs,
  File "/Users/vincent/Development/koaning-rasa/venv/lib/python3.6/site-packages/rasa/train.py", line 130, in _train_async_internal
    new_fingerprint = await model.model_fingerprint(file_importer)
  File "/Users/vincent/Development/koaning-rasa/venv/lib/python3.6/site-packages/rasa/model.py", line 208, in model_fingerprint
    stories = await file_importer.get_stories()
  File "/Users/vincent/Development/koaning-rasa/venv/lib/python3.6/site-packages/rasa/importers/importer.py", line 261, in get_stories
    stories = await asyncio.gather(*stories)
  File "/Users/vincent/Development/koaning-rasa/venv/lib/python3.6/site-packages/rasa/importers/rasa.py", line 55, in get_stories
    exclusion_percentage,
  File "/Users/vincent/Development/koaning-rasa/venv/lib/python3.6/site-packages/rasa/core/training/dsl.py", line 210, in read_from_files
    f, domain, interpreter, template_variables, use_e2e
  File "/Users/vincent/Development/koaning-rasa/venv/lib/python3.6/site-packages/rasa/core/training/dsl.py", line 238, in read_from_file
    return await reader.process_lines(lines)
  File "/Users/vincent/Development/koaning-rasa/venv/lib/python3.6/site-packages/rasa/core/training/dsl.py", line 346, in process_lines
    raise ValueError(msg)
ValueError: ('Error in line 12: Invalid to parse arguments in line \' form{"name": "schedule_form"\'. Failed to decode parametersas a json object. Make sure the eventname is followed by a proper json object. Error: Expecting \',\' delimiter: line 1 column 25 (char 24)', "Invalid story file format. Failed to parse '/Users/vincent/Development/koaning-rasa/data/stories.md'")

Five minutes later and I still don't know what file nlu.md/stories.md/domain.yml I need to edit/fix. Something that could really help here is a command line tool that would lint the files. Probably a good idea to add rasa lint. This seems to be a package where setting files are central to the functionality, without a linter it can be rather daunting for a newcomer to debug their code.

I've tried using this:

> rasa data validate 

but it yields the same issue.

Overview of the Solution:

Preferably something like;

> rasa lint 
Oddities found: 
- domain.yml cannot be parsed -> check line 12 

Examples

There's some other things that could be done in the linter.

> rasa lint 
Oddities found: 
- domain.yml cannot be parsed -> check line 12 
Un-used intents: 
- i_scream is never used

Definition of Done:

As a new user I mainly care about having tools that keep me on the "happy path" even when I make a config typo. At the moment the docs are a bit daunting. A linter tool could help.

[sara-tagger]

Thanks for submitting this feature request 🚀@Ghostvv will get back to you about it soon!✨

[Ghostvv]

Good point, I agree that the error is not clear at all, even for experience user. @wochinge @tabergma @tmbo what do you think?

[erohmensing]

rasa lint could also act to sort/dump the config and domain to avoid unnecessary git diffs. We could also create a pre-commit hook to implement this functionality automatically if users opt-in by activating the hook

[koaning]

pre-commit is a very useful feature. albeit advanced. i can imagine the
typical scientist not using it. so i would def do this in an opt-in
fashion.

sort/dump is behavior like ‘black’ which can certainly be useful. but also
here it might be good to opt in. tools that automatically change files can
be scary.

[erohmensing]

yes, black was the inspiration for that idea 🙂

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tmbo]

I think as a first step towards this, the error message should be improved to help the user better. even if there is a lint command, users will run into this.

[koaning]

This is something that I hope to pick up once I start at rasa.

Better error message are always a good thing, but a linter can help on different angles.

• It gives the user something to fall back to.
• Linters can also check things in order. Sometimes the spot where you misconfigured something and the spot where the error happens are not the same.

[koaning]

This is a reminder to myself, this feature would have saved me 30 minutes today.

if domain.yml is invalid the training loop proceeds and assumes the file does not exist.

➜  pokemon-demo-bot rasa train; 
/Users/vincent/Development/pokemon-demo-bot/venv/lib/python3.7/site-packages/rasa/utils/common.py:351: UserWarning: Loading domain from 'domain.yml' failed. Using empty domain. Error: 'The provided yaml file is invalid. You can use http://www.yamllint.com/ to validate the yaml syntax of your file.'
Training NLU model...
2020-04-29 17:14:31 INFO     rasa.nlu.training_data.training_data  - Training data stats:
2020-04-29 17:14:31 INFO     rasa.nlu.training_data.training_data  - Number of intent examples: 48 (9 distinct intents)
2020-04-29 17:14:31 INFO     rasa.nlu.training_data.training_data  -   Found intents: 'deny', 'greet', 'mood_great', 'bot_challenge', 'demand_joke', 'confirm_exists', 'affirm', 'mood_unhappy', 'goodbye'
2020-04-29 17:14:31 INFO     rasa.nlu.training_data.training_data  - Number of response examples: 0 (0 distinct responses)
2020-04-29 17:14:31 INFO     rasa.nlu.training_data.training_data  - Number of entity examples: 3 (1 distinct entities)
...

This may be considered bad behavior, but running the linter beforehand would have worked.

[erohmensing]

Thinking about this as well.

Both of these seem like issues that came from an invalid (by rasa but not yaml standards) domain
https://forum.rasa.com/t/i-have-enterd-the-intents-in-domain-but-it-say-entines-are-not-defined-in-domain/28338/3?u=erohmensing
https://forum.rasa.com/t/telegram-not-displaying-bot-messages/28372

We also have a domain schema, apparently, which i've never seen before. Would be great if we could make sure this was up to date (with, for example, the different types of responses, not just that it's a map, but tbh pretty sure responses are seq), and validate domain files against it. https://github.com/RasaHQ/rasa/blob/master/rasa/core/schemas/domain.yml

[tmbo]

I think mainly this should be checked before the training, a separate linting command is cherry on top