Galaxy: Update workflow editor for newer workflow features

[jmchilton]

• [ ] Allow implicit connections.
• [x] Allow step parameter connections to tool input parameters.
• [ ] Expose step UUIDs. (It has been requested, but I think most users will not want to see this.)
• [x] Build editor widget for defining input parameters (right now it only has a select drop down for type - but it should be a full input definition like used by Forms - optional, default, select options, etc...).
• [x] Allow modifying step labels on tool nodes. See also #1264

[mvdbeek]

• Allow step parameter connections to tool input parameters.

@jmchilton Uhh, never knew we could do this, is there a test I can look at ?

[jmchilton]

@mvdbeek

This is an important concept I want to push, but I've failed to grok or put the effort into the GUI to make it happen. The backend is quite old at this point and supports typed non-data parameters:

https://github.com/galaxyproject/galaxy/commit/45962f32344a00162f5d04b94096c8a570f2adf1 (see tests added in that commit)

Non-data inputs to workflows is just a start, my CWL branch has support for tools producing non-data values (https://github.com/common-workflow-language/galaxy/pull/47/commits/2ba9ed72e14e036db13596cf6c3d42a68ef7d34a) and connecting them between steps in a workflow.

Any help gaining traction on any of this would be most welcome, let me know what I can do to help you help me 😃.

[mvdbeek]

This is an important concept I want to push, but I've failed to grok or put the effort into the GUI to make it happen. The backend is quite old at this point and supports typed non-data parameters:

That is already great to have (I don't think we can do boolean or color in the current interface, so that's a pretty concrete gain), now we need to figure out how to represent this in the UI. Do you think we should add another noodle type ? I am a little worried about the clutter, but I guess we could have different views for parameters if that turns out to be a problem ?
Or should this work pretty much like the regular workflow parameters work ?

Non-data inputs to workflows is just a start, my CWL branch has support for tools producing non-data values (common-workflow-language@2ba9ed7) and connecting them between steps in a workflow.

This is just amazing, I think this opens a few usecases that have traditionally been difficult with Galaxy, like calculating a normalisation factor in one step and then using that factor in the next step, or inferring the sequencing adapter for thousands of datasets and then using the correct adapter for each dataset, or dynamically optimising parameters based on the input, like the number of rows / columns in a plot ... I knew you're working on this, but this is just amazing.

[jmchilton]

As a start, I think they should just be step boxes and connected via noodles. There should be a an option on the side tool form to turn any input into an input with a connector. We can then optimize the step display for all input types (data, collection, input) to be a lot smaller - the "boxes" (or maybe circles) really just need the label and an out arrow to reduce visual clutter. After that and after we've implemented implicit connections in the GUI, maybe different noodle colors for data, mapping, implicit, non-data connections may make sense to make things clearer?

I'd rather see steps taken to formalize existing workflow parameters as "string" inputs and sweep them under this umbrella. We are loosing a lot of type information and interpreting strings in problematic ways - I think connections would help that. Visually also I'm not sure the current way workflow parameters are represented makes it easy to connect them to things. I'll admit this preference is subjective and colored by other hats I wear. My inspiration for how inputs should work in the GUI is to make everything really consistent - the way both Galaxy represents everything as steps on the backend or that CWL has formal parameters for all inputs. Rabix Composer - http://docs.rabix.io/the-workflow-editor - does a really good job representing data and non-data inputs together in a really uniform way. I think we should be making progress toward uniformity for CWL and because it is the right way to think about workflows IMO.

[mvdbeek]

Totally agree, just looked at Rabix composer today and it is nice, without a doubt. I haven't seen how non-data parameters are represented, but I may just need to look closer. The thought crossed my mind if there isn't anything re-usable for us in there ...

[mvdbeek]

I'd rather see steps taken to formalize existing workflow parameters as "string" inputs and sweep them under this umbrella

They can be integers or floats as well at this point, but I agree that typing them is a good thing.

[mvdbeek]

I started working on the second point, hopefully I'll have something done by the end of the week.

[innovate-invent]

* Expose step UUIDs. (It has been requested, but I think most users will not want to see this.)

I would very much like this, even if it is hidden under a menu or something. Anything that allows me not to have to download and parse the ga file.