Scout: Update docs and code to require loqusdb version >= 2.5 if used

[bjhall]

This is may be less of a bug report and more of a support issue. But there are some bugs in this as well which I may add to an other issue.

Since the move to using the CLI for LoqusDB we have not been able to configure it properly. We've added this to the config.py:

LOQUSDB_SETTINGS = dict(
    binary_path="/correct/path/to/loqusdb"
)

But at this point:

https://github.com/Clinical-Genomics/scout/blob/750c805cefba3e06ca5737d021d71b69bec694e0/scout/server/blueprints/variant/views.py#L53-L57

current_app.config["LOQUSDB_SETTINGS"] is not set. So data["observations"] is not populated. I tried printing the whole dictionary and it is really not there!

However if I add {{config}} anywhere in variant.html, it shows that LOQUSDB_SETTINGS is set according to the values in config.py. Why are the config data different in these two places? Do I need to configure loqusdb somewhere else as well?

[bjhall]

By the way! It would be nice (for us in Lund) if obviously breaking changes like this (ie requiring reconfiguration) could be marked somehow in the CHANGELOG (or otherwise communicated).

[moonso]

Oh sorry @bjhall if this passed silently! We should definitely be better on communicating these things. I have updated the admin docs but understand that we should have made a better release note.

[bjhall]

No worries @moonso! I try to screen for any regressions everytime we update, but of course more information helps.

[moonso]

And to your problem, I can not really understand why it doesn't work. We use it in production and it is easy to set up locally to test. The idea is to install loqusdb in its own environment and provide the path to the executable in that environment. This is mainly to avoid dependency conflicts. For example now loqusdb(version>2.0) has a dependency on mongodb>3.7 and scout has mongodb<3.7.
When I set it up in my local computer I create a conda environment called loqusdb where I install the program. Then I point to that executable in the flask config like:

LOQUSDB_SETTINGS = dict(
    binary_path="/Users/mansmagnusson/miniconda3/envs/loqusdb/bin/loqusdb"
)

The config is only necessary if port, host and database name is not default (27017, localhost, loqusdb)

And when running I see the information from loqus. I'll be happy to help more if there are any problems.

[bjhall]

Yes that is exactly how we've set it up.

And then in scout/scout/server/blueprints/variant/views.py when it checks if current_app.config.get("LOQUSDB_SETTINGS"): the LOQUSDB_SETTINGS does not exist in the config dict.

But in template scout/server/blueprints/variants/templates/variants/variant.html it also checks {% if config['LOQUSDB_SETTINGS'] %} and then it is set! However the observations variable is not populated because of the above, so I get:

I suppose this could be some weird issue with out deployment/configuration, but I don't really understand it...

[moonso]

Hmm no that sounds strange... Have you tried to do it local?

[bjhall]

Well it's in a test environment (we have Windows laptops at Region Skåne........) but I've tried using both "scout serve" (using the built in web server) and properly deployed it. So on second thought it is probably not related to the deployment.

Any tips on where to start debugging why LOQUSDB_SETTING is not in the current_app.config dict!? Where is it initially set?

[moonso]

It is populated in server.app.create_app. Then how flask does the current_app I'm no expert in. Guess that @northwestwitch could give a better input.

[bjhall]

I think I'm starting to understand the problems (there were many). I think most of the problems are on our side. I'll close this issue and start a new one if anything needs to be fixed in Scout. I'm still figuring this out...

[bjhall]

@moonso I'm reopening this with more information.

I've had several problems,, some of which I've solved. One problem was that I used an old version of loqusdb (2.4). The scout code makes it appear that versions <2.5 are supported, but they are not. For instance --to-json is not supported in 2.4 but is required for _fetch_variant() to work.

Once I updated loquddb to 2.5 I started getting this error message

Something went wrong with loqus
Traceback (most recent call last):
  File "/opt/loqusdb2/venv/bin/loqusdb", line 11, in <module>
    load_entry_point('loqusdb', 'console_scripts', 'loqusdb')()
  File "/opt/loqusdb2/venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/loqusdb2/venv/lib/python3.6/site-packages/click/core.py", line 696, in main
    _verify_python3_env()
  File "/opt/loqusdb2/venv/lib/python3.6/site-packages/click/_unicodefun.py", line 124, in _verify_python3_env
    ' mitigation steps.' + extra
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Consult https://click.palletsprojects.com/en/7.x/python3/ for mitigation steps.

This system supports the C.UTF-8 locale which is recommended.
You might be able to resolve your issue by exporting the
following environment variables:

    export LC_ALL=C.UTF-8
    export LANG=C.UTF-8

Which I have not been able to solve despite testing everything... We're running python 3.6.8 and apparently this is fixed in 3.7. Maybe we just need to update python...

[moonso]

Woooow that does not look so fun. I heard something about this in our slack channel. Perhaps @ewels might have some input??

[moonso]

Hi @bjhall , did you test to update python/export the variables? From what I understand it is always good to set the variables like above when running python. In any case if we add a check in scout that loqusdb>=2.5 is used (and update docs), can we close this issue then?

[bjhall]

I got distracted by other things, so I haven't figured this out yet. Settings the env variables did not work for me, and I have not yet tested upgrading Python to 3.7.

But you can close the issue and I'll start a new one if needed, when I've figured this out.

[bjhall]

Just FYI, I solved the locale/encoding problem by updating to Python 3.7. Maybe it should be noted somewhere that python>=3.7 is a requirement for loqusdb if you want the scout integration to work.

[ewels]

Ugh, click and its locale settings 🤦‍♂ I would be careful about writing that py3.7 is a requirement - might be worth checking that it wasn't to do with creating or changing your environment somehow (as a side effect of updating to 3.7). I've seen similar bugs crop up and be fixed with some extremely random behaviour in the past. No tips really apart from trial and error. See this old PR: https://github.com/bioconda/bioconda-recipes/pull/5541 for some previous experience if it helps.

[dnil]

Great to hear that you got it working! As a case in favour of retaining py3.6, our stage environment still uses it for both scout and loqusdb. Sorry for all the fuss @bjhall - the idea was simply to make the two packages more independent. If you run them in separate environments, eg virtual conda envs they play well together even with py3.6.

[bjhall]

Thanks for your input @ewels and @dnil . If I ever find the time I'll do some more experimenting on the old environment, but att the moment I'm happy to have it working on 3.7.

@dnil I actually had loqusdb and scout running on in separate py3.6 environments before, but using virtualenv rather than conda. But it is entirely possible that my environment was broken in some other way.

It would be interesting to compare our deployment strategies of Scout at some point. I just got it working through trial and error 3 years ago, and now I'm just sticking to the same strategy, but it's probably not optimal.