Scout: Fix cli download tests

[northwestwitch]

Because the test fails all the time when Ensembl is down

[northwestwitch]

Might actually not be due to Ensembl being down since the service is patched

[northwestwitch]

It's not at all Ensembl-related, all tests under tests/commands/download/ fail! Must be some sneaky library update during the night! 🤔

[northwestwitch]

I don't know if I'll have time to check and fix it today so any help is welcome. To reproduce, create a new conda environment with python=3.8 and git clone scout into it. Then install test requirements and then scout itself with the command: pip install -e .

[mikaell]

Yeah, everything I push fails. I'll have a look. This is a real show stopper for me.

[northwestwitch]

Some tests failing:

[mikaell]

I installed 3.8 and I can reproduce the error locally. But Github test runs 3.7 -and locally 3.7 worked. I'm confused.

[northwestwitch]

I installed 3.8 and I can reproduce the error locally. But Github test runs 3.7 -and locally 3.7 worked. I'm confused.

Doesn't have to do with the python version, but probably some lib that gets installed from the most recent version if you install scout on a fresh env

[mikaell]

Check this out.

3.7

(py3.7) Mikael@mikaellmbp:scout$ pytest tests/commands/download/test_download_ensembl_cmd.py
=============================================================================== test session starts ===============================================================================
platform darwin -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /miniconda3/envs/py3.7/bin/python
cachedir: .pytest_cache
rootdir: /Users/Mikael/Work/scout_master/scout, configfile: pytest.ini
plugins: flask-1.0.0, mock-3.2.0, cov-2.10.1
collected 5 items

tests/commands/download/test_download_ensembl_cmd.py::test_download_ensembl_cmd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/Mikael/Work/scout_master/scout/tests/commands/download/test_download_ensembl_cmd.py(28)test_download_ensembl_cmd()
-> assert result.exit_code == 0
(Pdb) p result.exit_code
0
(Pdb) p result.output
'2020-12-10 13:33:14 mikaellmbp scout.commands.download.ensembl[4780] ERROR Download ensembl results to /var/folders/vc/5c_gq8jj3r5fgmwq7r5k8xk00000gp/T/tmp0ei92y6k\n'

3.8

(py3.8) Mikael@mikaellmbp:scout$ pytest tests/commands/download/test_download_ensembl_cmd.py
=============================================================================== test session starts ===============================================================================
platform darwin -- Python 3.8.5, pytest-6.1.2, py-1.9.0, pluggy-0.13.1 -- /miniconda3/envs/py3.8/bin/python
cachedir: .pytest_cache
rootdir: /Users/Mikael/Work/scout_master/scout, configfile: pytest.ini
plugins: flask-1.1.0, mock-3.3.1
collected 5 items

tests/commands/download/test_download_ensembl_cmd.py::test_download_ensembl_cmd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/Mikael/Work/scout_master/scout/tests/commands/download/test_download_ensembl_cmd.py(28)test_download_ensembl_cmd()
-> assert result.exit_code == 0
(Pdb) p result.exit_code
0
(Pdb) p result.output
''

[northwestwitch]

Looks like they both fail but python 3.7 tells you why

[mikaell]

result.output does not get set. So the assert test will fail.

Don't think matching a string in the output is as important as checking for resultcode == 0. Shall we remove it?

[northwestwitch]

But python 3.7 says this: ERROR Download ensembl results to /var/folders/vc/5c_gq8jj3r5fgmwq7r5k8xk00000gp/T/tmp0ei92y6k\n'

So it could actually be more than just matching a string

[northwestwitch]

I think it would be useful to understand what changed since yesterday, since until yesterday out PRs passed these tests and now they don't

[mikaell]

Looks like they both fail but python 3.7 tells you why

3.7 success

The error was from me testing different prints. Sorry to confuse you.

(Pdb) p result.output
'2020-12-10 13:47:37 mikaellmbp scout.commands.download.ensembl[5049] INFO Download ensembl results to /var/folders/vc/5c_gq8jj3r5fgmwq7r5k8xk00000gp/T/tmp5xh7mpfe\n'

My likely suspect now is Click.

[mikaell]

Click was not the culprit. Running Python 3.7 with Click updated to 7.1.2 it still works.

(py3.7) Mikael@mikaellmbp:scout$ pip list |grep click
click                7.1.2
(py3.7) Mikael@mikaellmbp:scout$ pytest tests/commands/download/test_download_ensembl_cmd.py
============================================================================== test session starts ==============================================================================
platform darwin -- Python 3.7.7, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /miniconda3/envs/py3.7/bin/python
cachedir: .pytest_cache
rootdir: /Users/Mikael/Work/scout_master/scout, configfile: pytest.ini
plugins: flask-1.0.0, mock-3.2.0, cov-2.10.1
collected 5 items

tests/commands/download/test_download_ensembl_cmd.py::test_download_ensembl_cmd
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/Mikael/Work/scout_master/scout/tests/commands/download/test_download_ensembl_cmd.py(28)test_download_ensembl_cmd()
-> assert result.exit_code == 0
(Pdb) p result.output
'2020-12-10 13:52:13 mikaellmbp scout.commands.download.ensembl[5102] INFO Download ensembl results to /var/folders/vc/5c_gq8jj3r5fgmwq7r5k8xk00000gp/T/tmpe_ew_ji4\n'
(Pdb) c

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB continue (IO-capturing resumed) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PASSED                                                                                    [ 20%]

[mikaell]

I suggest to remove the string matching test. I literally looks for a log message printed before any action takes place.

Ensemble.py:

def ensembl(out_dir, skip_tx, exons, build):
    """Download the Ensembl information"""
    out_dir = pathlib.Path(out_dir)
    out_dir.mkdir(parents=True, exist_ok=True)
    LOG.info("Download ensembl results to %s", out_dir)   # String  we assert in test_download_ensembl_cmd.py


    print_ensembl(out_dir, "genes", build)

    if not skip_tx:
        print_ensembl(out_dir, "transcripts", build)

    if exons:
        print_ensembl(out_dir, "exons", build)

_Yeah, the "to" was added by me, otherwise I wouldn't have believed the meaninglessness of the tests._

_Well, these sort of test is everywhere, trying to match a string written to stdout. I suppose the root cause must be found_

[mikaell]

Reran tests for an old, merged pr. They fail now too: https://github.com/Clinical-Genomics/scout/pull/2238/checks?check_run_id=1530986070

[dnil]

My local tests fail on PyMongo collections() at the moment - that was the deprecated one that was supposed to go obsolete in Python 3.9, but possibly someone moved that one up. I'll start there at least, but it may just be my local..

[northwestwitch]

Today looks like entropy is finally closing in on this repo

[mikaell]

In requirements changing coloredlogs from 14.0 to 14.2 introduces the error locally for me.

I don't know what colorlogs does. I'm all for removing it.

[dnil]

Well they are very pretty.. :wink: Lets see if there is something else affected - I wonder if that would not mess with logger generally?

[dnil]

PyMongo went 3.11.2 but that was 2 Dec according to pypi. (And it was just my mongomock that was out of date; fixed and works).

[northwestwitch]

In requirements changing coloredlogs from 14.0 to 14.2 introduces the error locally for me.

I don't know what colorlogs does. I'm all for removing it.

Yes it's coloredlogs, let's remove it!

[northwestwitch]

coloredlogs was updated today, see here: https://coloredlogs.readthedocs.io/en/latest/changelog.html#release-14-1-2020-12-10

[dnil]

"updated 1 h ago" on PyPi to a new major. We could also just freeze it to <15 until they have time to fix it.

[northwestwitch]

Or we just fix the tests removing those lines?

[mikaell]

I vote for finding a working version and freeze. _Or I don't know at the moment. Finding this bug took the whole day_

[northwestwitch]

"updated 1 h ago" on PyPi to a new major. We could also just freeze it to <15 until they have time to fix it.

They released 4 versions during the last hours, reminds me of us during one of these glorious release days 🤣

[dnil]

Based on the link @northwestwitch posted I would guess just <15 will do it: they had a new 14.3 as well earlier today, which may or may not work. I'll try it as soon as my baseline local is done, but at least 14.0 seems to have been working - and probably 14.2..

[dnil]

Hehe yes I see it now.. 14.0 it is for now..

[northwestwitch]

14.2 doesn't work for me

[dnil]

And 14.0 does work here. Given that all later were released today, culminating in that 15.0:

"This is backwards incompatible (which is why I’m bumping the major version number) however the old behavior has been reported to be rather problematic (see is_syslog_supported() for details) so this seems like the best choice."

[henrikstranneheim]

Time for some lock files using poetry or pipenv - Måns favourite topic

[dnil]

Yes, well, it is not breaking in production yet but stopping development for hours is not super either I guess. I think coloredlogs will probably solve this before long, but lets freeze it for now.. 🥶

[dnil]

Yep, I confirm also 14.1 is bugged.

[dnil]

Whereas 14.0 works.