Galaxy: Wrap display_application cheetah fill_template in try/except

[mvdbeek]

as we do for tools.
Otherwise starting without a datatype_conf.xml file (or a very empty one) could lead to

galaxy.web.framework.decorators ERROR 2019-06-04 16:38:43,484 [p:74,w:1,m:0] [uWSGIWorker1Core0] Uncaught exception in exposed API method:
Traceback (most recent call last):
File "/galaxy/server/lib/galaxy/web/framework/decorators.py", line 282, in decorator
rval = func(self, trans, *args, **kwargs)
File "/galaxy/server/lib/galaxy/webapps/galaxy/api/history_contents.py", line 82, in index
return self.__index_v2(trans, history_id, **kwd)
File "/galaxy/server/lib/galaxy/webapps/galaxy/api/history_contents.py", line 891, in __index_v2
user=trans.user, trans=trans, view='detailed', **serialization_params))
File "/galaxy/server/lib/galaxy/managers/base.py", line 692, in serialize_to_view
return self.serialize(item, all_keys, **context)
File "/galaxy/server/lib/galaxy/managers/hdas.py", line 369, in serialize
return super(HDASerializer, self).serialize(hda, keys, user=user, **context)
File "/galaxy/server/lib/galaxy/managers/datasets.py", line 604, in serialize
serialized = super(DatasetAssociationSerializer, self).serialize(dataset_assoc, keys, **context)
File "/galaxy/server/lib/galaxy/managers/base.py", line 605, in serialize
returned[key] = self.serializers[key](item, key, **context)
File "/galaxy/server/lib/galaxy/managers/hdas.py", line 376, in serialize_display_apps
for display_app in hda.get_display_applications(trans).values():
File "/galaxy/server/lib/galaxy/model/__init__.py", line 2756, in get_display_applications
return self.datatype.get_display_applications_by_dataset(self, trans)
File "/galaxy/server/lib/galaxy/datatypes/data.py", line 551, in get_display_applications_by_dataset
value = value.filter_by_dataset(dataset, trans)
File "/galaxy/server/lib/galaxy/datatypes/display_applications/application.py", line 309, in filter_by_dataset
if link_value.filter_by_dataset(data, trans):
File "/galaxy/server/lib/galaxy/datatypes/display_applications/application.py", line 95, in filter_by_dataset
if fill_template(filter_elem.text, context=context) != filter_elem.get('value', 'True'):
File "/galaxy/server/lib/galaxy/util/template.py", line 91, in fill_template
raise first_exception or e
NotFound: cannot find ‘site_id’

[afgane]

Probably related. Happens when trying to expand a history dataset and seemingly having an incorrect combination of datatypes and display configuration:

[pid: 49|app: 0|req: 75/126] 10.0.17.31 () {56 vars in 1197 bytes} [Tue Jun 4 20:11:20 2019] GET /galaxy/api/users/current => generated 89 bytes in 10 msecs (HTTP/1.1 200) 3 headers in 139 bytes (1 switches on core 3)
/galaxy/server/.venv/local/lib/python2.7/site-packages/Cheetah/Compiler.py:1630: UserWarning:
You don't have the C version of NameMapper installed! I'm disabling Cheetah's useStackFrames option as it is painfully slow with the Python version of NameMapper. You should get a copy of Cheetah with compiled C version of NameMapper.
"\nYou don't have the C version of NameMapper installed! "
galaxy.web.framework.decorators ERROR 2019-06-04 20:11:35,305 [p:49,w:1,m:0] [uWSGIWorker1Core0] Uncaught exception in exposed API method:
Traceback (most recent call last):
File "/galaxy/server/lib/galaxy/web/framework/decorators.py", line 282, in decorator
rval = func(self, trans, *args, **kwargs)
File "/galaxy/server/lib/galaxy/webapps/galaxy/api/history_contents.py", line 176, in show
return self.__show_dataset(trans, id, **kwd)
File "/galaxy/server/lib/galaxy/webapps/galaxy/api/history_contents.py", line 263, in __show_dataset
**self._parse_serialization_params(kwd, 'detailed'))
File "/galaxy/server/lib/galaxy/managers/base.py", line 692, in serialize_to_view
return self.serialize(item, all_keys, **context)
File "/galaxy/server/lib/galaxy/managers/hdas.py", line 367, in serialize
return super(HDASerializer, self).serialize(hda, keys, user=user, **context)
File "/galaxy/server/lib/galaxy/managers/datasets.py", line 604, in serialize
serialized = super(DatasetAssociationSerializer, self).serialize(dataset_assoc, keys, **context)
File "/galaxy/server/lib/galaxy/managers/base.py", line 605, in serialize
returned[key] = self.serializers[key](item, key, **context)
File "/galaxy/server/lib/galaxy/managers/hdas.py", line 374, in serialize_display_apps
for display_app in hda.get_display_applications(trans).values():
File "/galaxy/server/lib/galaxy/model/__init__.py", line 2746, in get_display_applications
return self.datatype.get_display_applications_by_dataset(self, trans)
File "/galaxy/server/lib/galaxy/datatypes/data.py", line 551, in get_display_applications_by_dataset
value = value.filter_by_dataset(dataset, trans)
File "/galaxy/server/lib/galaxy/datatypes/display_applications/application.py", line 309, in filter_by_dataset
if link_value.filter_by_dataset(data, trans):
File "/galaxy/server/lib/galaxy/datatypes/display_applications/application.py", line 95, in filter_by_dataset
if fill_template(filter_elem.text, context=context) != filter_elem.get('value', 'True'):
File "/galaxy/server/lib/galaxy/util/template.py", line 91, in fill_template
raise first_exception or e
NotFound: cannot find 'dataset'

[natefoo]

A bit more context, I believe the original is likely to be encountered if using the default build_sites.yml without running the setup script to copy the build site files referenced within from their samples. On Main I read the UCSC build sites from a copy in CVMFS that is automatically updated weekly, and igv and rviewer are modified to read directly from the samples.

[afgane]

Little by little, instead of just trying to add configurations, we removed them all and realized this is happening with all default settings on a standalone docker image. So here's an image that exhibits this behavior and I'm not sure where to poke at it so am hoping someone can perhaps try it locally and chime in?

docker run -it -p 8080:8080 galaxy/galaxy:19.05sqlite bash
uwsgi --yaml config/galaxy.yml

(alternatively, can use sh run.sh and the behavior is the same)
If a bam, bed, or fasta file is uploaded (not sure about any others), and you try to expand the history dataset, the above site_id error is triggered. The error does not show up with a fastqsanger dataset for example.

[dannon]

@afgane I'll give it a shot now.

[dannon]

I really feel like I owe someone (@WilliamHolden) an apology for tagging this one beginner friendly. I thought it'd be a fun way to see more of the Python side of the stack. It goes into the weeds a bit.

This isn't a configuration issue with datatypes or display applications at all, that was a red herring.

I walked through exactly what was happening at template time. We're passing in an object to fill_template with the right values -- it looks something like this if you inspect it:

{'qp': <function quote_plus at 0x7f72ea83c5f0>, 'site_name': 'web current', 'APP': <galaxy.app.UniverseApplication object at 0x7f72ccc8fd90>, 'site
_id': 'web_link_main', 'site_dbkeys': ['hg_g1k_v37'], 'BASE_URL': 'http://localhost:8080', 'site_organisms': ['b37'], 'url_for': <function url_for at 0x7f72e55875f0>, 'datase
t': <galaxy.model.HistoryDatasetAssociation(1) at 0x7f72c4596490>, 'site_link': 'http://www.broadinstitute.org/igv/projects/current/igv.php'}

Note that site_id and all the rest of the variables that the template needs are indeed defined.

Cheetah will insist that those aren't in the namespace, though, due to the code here: https://github.com/CheetahTemplate3/cheetah3/blob/master/Cheetah/NameMapper.py#L214

Looks right, but we're actually passing our (non-2.7 actual OrderedDict) odict, per https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/display_applications/application.py#L62. The problem here is that our odict is not a Mapping, and hasattr of course doesn't find keys on it, which causes all of this to fail.

The reason we have only seen this on this particular container is that it's using the purepy NameMapper linked above because the C version of NameMapper is missing, and the C version does object inspection and handles hasKey differently.

I don't know why C-based NameMapper is missing in this container or how the dependencies were installed, but fixing that is probably a 'quick fix' (which should probably be done anyway) for this particular deployment.

Another fix (which we should probably also do anyway) is is to use a real OrderedDict, or even just a dict here. I'm testing https://github.com/galaxyproject/galaxy/pull/8104, but while that happens -- is there any reason at all we still need galaxy.util.odict? Is there any reason not to swap all of those to core collections.OrderedDicts?

[nuwang]

@dannon That's a veritable tour de force of debugging, awesome stuff!

[afgane]

Above referenced commit sorted out the issue on the container image for the time being.

[dannon]

For just a tiny bit more context, when we looked into why the container wasn't using the c-based NameMapper it was due to libpython2.7.so.1.0 being missing, which is what that https://github.com/CloudVE/galaxy-kube-playbook/commit/c685b1c4b0f5bffb62c98248c92ed5ca39bff4c8 addresses. We'll still want to swap out our odict implementation for OrderedDicts as in https://github.com/galaxyproject/galaxy/pull/8104, but there are a couple kinks to work out there yet.

[natefoo]

@afgane Is your container not using the Cheetah wheels from wheels.galaxyproject.org?

[nuwang]

@natefoo The docker file doesn't do anything much other than installing the basic requirements for running ansible-galaxy (git and python-virtualenv) and then running the ansible-galaxy role via a playbook file.
https://github.com/CloudVE/galaxy-kube-playbook/blob/c7c27705608530d87d270fdd5a1b7e2f31f15a49/Dockerfile#L46

The default is to use galaxy wheels right?

[natefoo]

Yeah it is. You're not using dev, right?

[nuwang]

@natefoo The defaults here: https://github.com/galaxyproject/ansible-galaxy/blob/3b18f4759911cc786d3bd77a4d94434cf41d0311/tasks/dependencies.yml#L14
Is that correct?

[ghrasko]

I would like to add that the same issue @afgane had mentioned is occuring in my fresh test installation:

• Galaxy release_19.05 install on Ubuntu server VM (Oracle VirtualBox 64 bit on Windows 10)

FASTQ file history items open in the history panel correctly when I click on the title. Most other formats produce the error.

I am a beginner in this and probably won't be able to run detailed debugging. I was happy to find this thread anyhow. I don't understand, what could be done at this point, if anything.

[natefoo]

This would potentially occur if for some reason you are building Cheetah3 from source instead of using wheels.

@ghrasko If you can try these steps we could verify:

1. Activate your virtualenv (by default it is in .venv/ in the Galaxy directory) with:
   . ./.venv/bin/activate
2. Uninstall Cheetah3:
   pip uninstall -y cheetah3
3. Install Cheetah3:
   pip install --no-cache-dir cheetah3==3.2.2

And then paste the output (especially of step 3).

[ghrasko]

Hi @natefoo,

I installed Galaxy the standard way on a fresh Ubuntu server. I only installed python 2.7.15 in advance. Python 3.6.8 is already coming with Ubuntu, I hope it is not causing problem.

I reinstalled cheetah3 the way you had described. This is the output of the install step:

Collecting cheetah3==3.2.2
  Downloading https://files.pythonhosted.org/packages/3b/16/60baf6aea3de483da186
650f4a1d483a24b96a3915345fdf57fd5b44ef1b/Cheetah3-3.2.2-cp27-cp27mu-manylinux1_x
86_64.whl (662kB)
Installing collected packages: cheetah3
Successfully installed cheetah3-3.2.2

After this I ran Galaxy, but nothing had changed. I uploaded several file types. I had problems with these. View (in the main window) is working fine, only the opening in the History window causes the error:

• gtf (Cuffmerge output)
• bam (TopHat output)

These open correctly in the History window:

• tabular (CuffDiff output)
• sra
• fastsanger.qz

The error is shown in the console as follows:

FILE "lib/galaxy/util/template.py", line 91, in fill_template
  raise first_exception or e
NotFound: cannot find 'site_id'

In the Galaxy window in the browser there is a red error box. The detail page shows this:

{
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36",
  "onLine": true,
  "version": "19.05",
  "xhr": {
    "readyState": 4,
    "responseText": "{\"err_msg\": \"Uncaught exception in exposed API method:\", \"err_code\": 0}",
    "responseJSON": {
      "err_msg": "Uncaught exception in exposed API method:",
      "err_code": 0
    },
    "status": 500,
    "statusText": "Internal Server Error"
  },
  "options": {
    "parse": true,
    "silent": true,
    "emulateHTTP": false,
    "emulateJSON": false,
    "textStatus": "error",
    "errorThrown": "Internal Server Error"
  },
  "url": "http://192.168.0.123:8080/api/histories/f597429621d6eb2b/contents/datasets/e85a3be143d5905b",
  "model": {
    "state": "ok",
    "deleted": false,
    "purged": false,
    "name": "Galaxy25-[TopHat_on_data_5__accepted_hits].bam",
    "accessible": true,
    "data_type": "",
    "file_ext": "",
    "file_size": 0,
    "meta_files": [],
    "misc_blurb": "",
    "misc_info": "",
    "tags": [],
    "history_id": "f597429621d6eb2b",
    "history_content_type": "dataset",
    "hid": 12,
    "visible": true,
    "model_class": "HistoryDatasetAssociation",
    "update_time": "2019-08-08T11:30:17.774Z",
    "extension": "bam",
    "type_id": "dataset-e85a3be143d5905b",
    "id": "e85a3be143d5905b",
    "create_time": "2019-08-08T11:30:00.269Z",
    "url": "/api/histories/f597429621d6eb2b/contents/e85a3be143d5905b",
    "dataset_id": "e85a3be143d5905b",
    "type": "file",
    "urls": {
      "purge": "/datasets/e85a3be143d5905b/purge_async",
      "display": "/datasets/e85a3be143d5905b/display/?preview=True",
      "edit": "/datasets/edit?dataset_id=e85a3be143d5905b",
      "download": "/datasets/e85a3be143d5905b/display?to_ext=",
      "report_error": "/dataset/errors?id=e85a3be143d5905b",
      "rerun": "/tool_runner/rerun?id=e85a3be143d5905b",
      "show_params": "/datasets/e85a3be143d5905b/show_params",
      "visualization": "/visualization",
      "meta_download": "/dataset/get_metadata_file?hda_id=e85a3be143d5905b&metadata_name="
    }
  },
  "user": {
    "id": "f2db41e1fa331b3e",
    "username": "ghrasko",
    "total_disk_usage": 1288509994,
    "nice_total_disk_usage": "1.2 GB",
    "quota_percent": null,
    "is_admin": true,
    "preferences": {},
    "tags_used": [],
    "deleted": false,
    "purged": false,
    "quota": null
  }
}

If I close these dataset items (hide the details) in the History window, everything is fine. However if I leave one open and refresh the screen (for example by selecting an item from the top menu (Analyse, Workflow...) or if I refresh the History window all items dissapear from the history. it shows the number of items, but the window is empty. I can purge the History anyhow. It seems, it is purely a display problem.

[natefoo]

@dannon any idea on how @ghrasko could check if this is the same issue you found?

[ghrasko]

The console logs are very similar. Now I compared mine with the one presented in the opening ticket.

[ghrasko]

Any ideas?