Galaxy: Galaxy 16.04: tools with dynamic select lists that are passed as RuntimeValue objects fail due to no legal values when attempting to render html

[gregvonkuster]

Including this version of the Extract Genomic DNA tool in a workflow provides a good example: https://toolshed.g2.bx.psu.edu/view/iuc/extract_genomic_dna/702970e4a134.

The tool includes this parameter, which results in the bug:

<param name="reference_genome" type="select" label="Using reference genome">
    <options from_data_table="twobit">
        <filter type="data_meta" key="dbkey" ref="input" column="0"/>
    </options>
    <validator type="no_options" message="A built-in reference genome is not available for the build associated with the selected input file"/>
</param>

The "input" value of the ref attribute in the above tag ultimately calls the filter_options() method in the ~/lib/galaxy/tools/parameters/dynamic_options code file, passing ref as a RuntimeValue object. The following code in that method is executed:

        ref = other_values.get( self.ref_name, None )
        is_data = isinstance( ref, galaxy.tools.wrappers.DatasetFilenameWrapper )
        is_data_list = isinstance( ref, galaxy.tools.wrappers.DatasetListWrapper ) or isinstance( ref, list )
        is_data_or_data_list = is_data or is_data_list
        if not isinstance( ref, self.dynamic_option.tool_param.tool.app.model.HistoryDatasetAssociation ) and not is_data_or_data_list:
            return []  # not a valid dataset

Since ref is a RuntimeValue object, the empty list is always returned, and the workflow fails with this tool error:

"traceback": "Traceback (most recent call last):\n  File \"lib/galaxy/web/framework/decorators.py\", line 261, in decorator\n    rval = func( self, trans, *args, **kwargs)\n  File \"lib/galaxy/webapps/galaxy/api/workflows.py\", line 458, in invoke\n    request_params=work_request_params\n  File \"lib/galaxy/workflow/run.py\", line 115, in queue_invoke\n    modules.populate_module_and_state( trans, workflow, workflow_run_config.param_map, allow_tool_state_corrections=workflow_run_config.allow_tool_state_corrections )\n  File \"lib/galaxy/workflow/modules.py\", line 1342, in populate_module_and_state\n    raise exceptions.MessageException( message )\nMessageException: Workflow cannot be run because of validation errors in some steps: {'reference_genome_cond|reference_genome': 'Parameter reference_genome requires a value, but has no legal values defined.'}\n", "err_msg": "Workflow cannot be run because of validation errors in some steps: {'reference_genome_cond|reference_genome': 'Parameter reference_genome requires a value, but has no legal values defined.'

[guerler]

Thanks for reporting this. Can you try to load the workflow form with the above tool and then change the initially selected dataset? Does the reference genome field properly populate like in the screenshot here:

[gregvonkuster]

@guerler I have made sure the dynamic select list on this tool is set as "Set at runtime" within the workflow editor. I then loaded the workflow by choosing "Run" from the workflow menu, then selected various input datasets with different dbkeys, and the select list for the above tool does not refresh. It just remains with the first option in the dynamic select list:

[guerler]

I can confirm that the dynamic list if set to "Set at runtime" does not properly refresh/fill on initial load since at least 14.10. However, after forcing the first refresh e.g. by selecting a new dataset, the list continues to properly refresh. Can you share your loc file? Maybe that will allow me to reproduce the issue you are encountering now. Thanks again.

[gregvonkuster]

@guerler
In case it helps, I'm using an automated process that consists of a set of scripts that set parameters at runtime for certain tools in the workflow when I make the api call gi.workflows.invoke(workflow(), sending inputs and params. These parameter settings affect tools that use reference genomes for specified dbkeys, and the Extract Genomic DNA tool is an example. So in my scenario, the fact that the select list does not refresh is really not the problem since I am making the api call.

This process worked in Galaxy 15.10, so changes since then have resulted in this new behavior. Perhaps whatever is needed to fix the refresh for the select list will also fix this broken api call?

The underlying issue seems to be that a RuntimeValue object is not properly handled throughout the workflow process. So the likely fix centers around either enhancing existing methods to properly handle these new objects throughout the workflow or somehow change the RuntimeValue object into a HistoryDatasetAssociation or some other supported object before it proceeds through the workflow.

Regarding my .loc file, my Galaxy instance uses Data Managers, so I have installed this tool for use with the Extract Genomic DNA tool: https://toolshed.g2.bx.psu.edu/view/devteam/data_manager_twobit_builder/74b09c8e5f6e

Here is the .loc file I'm using which was produced by the Data Manager.

twobit.loc.txt

[WilliamKMLai]

+1 on fixing this. This is critical for fixing our ChIP-exo pipeline. Thanks! :-)

[dshao]

+2 on fixing this. This is critical for fixing our ChIP-exo pipeline. Thanks! :-)

[guerler]

@WilliamKMLai and @dshao, sorry for the inconvenience. I am on it. Just to clarify this, triggering a refresh as mentioned above does not fix the issue right or are you guys executing the workflow through the api too?

[gregvonkuster]

@guerler I've tried manually triggering the refresh on the Run Workflow form using the short workflow I sent to you, and the dynamic options are not rendered for the Extract Genomic DNA tool. I have made sure to set the parameters as "Set at runtime" in the workflow editor:

[gregvonkuster]

@guerler @WilliamKMLai @dshao To clarify, we are all executing these workflows both manually and via the API, but the critical blocker for all of us right now is via the API.

[guerler]

Alright. Thanks a lot for all that info.

[jmchilton]

@guerler You disappeared from IRC - I don't know of any existing tests that cover dynamic lists like this.

[guerler]

Alright, can you help to add one? It would be great to distinguish this particular issue from other issues.

[guerler]

I created a test workflow using the dbkey_filter_input.xml tool from https://github.com/galaxyproject/galaxy/blob/dev/test/functional/tools/dbkey_filter_input.xml: And executed the workflow through the api using: https://bitbucket.org/galaxy/galaxy-central/src/3f95c21b1a85af3ae1260eb957ea041d1de8bd5b/scripts/api/workflow_execute_parameters.py?at=default&fileviewer=file-view-default:
I noticed that the default for input step ids seems to have changed from step_id to order_index by default which would fix #2437. ping @jmchilton

In 16.04:
python workflow_execute_parameters.py 9565a0da1d1bafbbad5db294a3df9dcf http://127.0.0.1:8080/api/workflows 8c49be448cfe29bc 'hist_id=f6509e58756240f2' '0=hda=43b77686951b80f1' 'param=dbkey_filter_input=index=hg18canon'

In 15.10:
python workflow_execute_parameters.py 9565a0da1d1bafbbad5db294a3df9dcf http://127.0.0.1:8080/api/workflows 8c49be448cfe29bc 'hist_id=f6509e58756240f2' '2360=hda=43b77686951b80f1' 'param=dbkey_filter_input=index=hg18canon'

Both of these fail without https://github.com/galaxyproject/galaxy/pull/2456.

[gregvonkuster]

@guerler Thanks for working on this! This PR https://github.com/galaxyproject/galaxy/pull/2456 does not correct the behavior in my case. I am still seeing this exception:

galaxy.tools ERROR 2016-06-05 12:27:48,342 Exception caught while attempting tool execution:
Traceback (most recent call last):
  File "/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/__init__.py", line 1142, in handle_single_execution
    job, out_data = self.execute( trans, incoming=params, history=history, rerun_remap_job_id=rerun_remap_job_id, mapping_over_collection=mapping_over_collection, execution_cache=execution_cache )
  File "/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/__init__.py", line 1220, in execute
    return self.tool_action.execute( self, trans, incoming=incoming, set_output_hid=set_output_hid, history=history, **kwargs )
  File "/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/actions/__init__.py", line 216, in execute
    history, inp_data, inp_dataset_collections = self._collect_inputs(tool, trans, incoming, history, current_user_roles)
  File "/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/actions/__init__.py", line 201, in _collect_inputs
    inp_data = self.collect_input_datasets( tool, incoming, trans, current_user_roles=current_user_roles )
  File "/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/actions/__init__.py", line 149, in collect_input_datasets
    tool.visit_inputs( param_values, visitor )
  File "/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/__init__.py", line 1066, in visit_inputs
    visit_input_values( self.inputs, values, callback )
  File "/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/parameters/__init__.py", line 110, in visit_input_values
    callback_helper( input, input_values, name_prefix, label_prefix, parent_prefix=parent_prefix, context=context )
  File "/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/parameters/__init__.py", line 75, in callback_helper
    new_value = callback( **args )
  File "/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/actions/__init__.py", line 112, in visitor
    input_datasets[ prefix + input.name ] = process_dataset( value )
  File "/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/actions/__init__.py", line 61, in process_dataset
    if not data.datatype.matches_any( formats ):
AttributeError: 'RuntimeValue' object has no attribute ‘datatype'

I notice that you are not using bioblend for your API calls in this script: https://bitbucket.org/galaxy/galaxy-central/src/3f95c21b1a85af3ae1260eb957ea041d1de8bd5b/scripts/api/workflow_execute_parameters.py?at=default&fileviewer=file-view-default, but my understanding is that bioblend should be used when working with the Galaxy API. Can you clarify this?

My current API calls that fail with the above exception (but with the patch from your PR) look like this:

workflow_invocation_dict = gi.workflows.invoke_workflow(workflow_id, inputs=inputs, params=params, history_id=history_id)

Do I need to eliminate the use of bioblend in my API calls in order for the above PR to fix this behavior?

Thanks!

[guerler]

I used the direct api call in order to keep the test case as simple as possible. The second issue you refer to is related to the other ticket right? Currently, I can reproduce it only if I do not parse input datasets for tool modules which contain dataset selectors. I suggest to fix the issue in this ticket first using the PR above and then work on the second ticket.

[gregvonkuster]

Ok, thanks. I'll wait for the 2nd fix and see if my API calls using bioblend will work. Please let me know if I can help.

[martenson]

2456 has been merged

[guerler]

This has been fixed right?

[gregvonkuster]

@guerler Yes, many thanks for fixing this!