Rasa: Model regression tests fail because of a dependency issue

[dakshvar22]

Rasa version: PR

Operating system (windows, osx, ...): Ubuntu

Issue: Model regression tests for some configurations start failing because of a dependency issue. This is weird because it's not consistent across configurations of the regression tests which means it has nothing to do with the code and the configurations and is a bit random probably.

[m-vdb]

I've introduced the new pydoc-markdown dependency, and I didn't take a look at the setup for model regression tests. My bad. Let me take a quick look to see if I can fix it

[m-vdb]

it actually seems to me that it's not linked to pydoc-markdown (sure, there is an internal error from pip, but that's not stopping the script). The real error is:

[TypeError]
expected string or bytes-like object

Which happens at various stages of the installation process on various builds. There are this issue and this one that talk about a possible problem with virtualenv.

[dakshvar22]

Hmm, so would the recommendation be to try re-running the jobs in that case?

[m-vdb]

maybe, but it doesn't seem like a suitable solution. Will retry the one you mentioned in the desc of this PR

[m-vdb]

but apparently the "retry" didn't work @dakshvar22 . How can we retry those runs?

[dakshvar22]

I just re-ran them by adding the appropriate labels on the PR. :)

[m-vdb]

it looks like dependencies got installed this time 🤷 😕

[dakshvar22]

They do get installed for the first configuration always and then the next one fails. 😕

[m-vdb]

how different are the configurations in terms of OS?

[dakshvar22]

They all run on the same OS configuration.

[dakshvar22]

And it failed for the second configuration :)

[m-vdb]

yeah, just saw that. Comparing the log and they are identical. It really looks like an internal issue of pip or virtualenv. I cannot reproduce locally with Python 3.7 (but the tests run Python 3.8, will try that)

[m-vdb]

by following the code inside pip module, the error really comes from the fact that it doesn't find a METADATA file:

FileNotFoundError: [Errno 2] No such file or directory: '/home/github/.cache/pypoetry/virtualenvs/rasa-ip_mm8IB-py3.8/lib/python3.8/site-packages/pydoc_markdown-3.3.0.post1.dist-info/METADATA'

which is really here when you install the package on python 3.7 or 3.8 🤔

[m-vdb]

so since the METADATA is really present in the files hosted on pypi.rasa.com, could it be an issue that is happening when loading the cache (using @actions/cache)?

[m-vdb]

I've pushed a small commit to your branch to see if it has an impact, re-running the tests ⌛

[dakshvar22]

Looks like that didn't help, as the test still failed.

[m-vdb]

yes, saw that :/ that's really weird. Let's see if @tczekajlo can take a closer look at this as he is here this week. He is more familiar with the model regression tests setup than me, for sure.

[tczekajlo]

I'll try to take a look at it as soon as possible :)