Galaxy: Workflow version management

[mvdbeek]

I think it would be great if we could have a little more control over versions. This includes a couple of things (please amend if you have more ideas):

• [x] Extend API to select specific workflow versions (WIP https://github.com/galaxyproject/galaxy/pull/6409)
• [x] Allow running workflows without importing them (https://github.com/galaxyproject/galaxy/pull/6428)
• [ ] Show notification if newer workflow version is available
• [ ] Have "Reference Workflow" at a URL, allow easy updating from hosted workflows
• [x] Allow selecting workflow version in the workflow menu and workflow editor (WIP https://github.com/galaxyproject/galaxy/pull/6409)
• [ ] Add changelog and explicit version management

[gregvonkuster]

Way back when workflows were first introduced, I was thinking that a workflow version would be at least partially defined by the versions of the tools contained within the workflow. Workflows have come a long way since then and I've not thought about this for a while, so I'm not sure how relevant it still is. But at the time, I felt that if a workflow layout remains the same (i.e., all tools and connections remain the same), then the action of upgrading a tool within the workflow would somehow increment the version of the workflow. If this no longer makes sense, feel free to ignore. ;)

[hexylena]

I mentioned some of these points originally in https://github.com/galaxyproject/galaxy/issues/298, would definitely love to see this!

[mvdbeek]

then the action of upgrading a tool within the workflow would somehow increment the version of the workflow

Right, that is still the case. But we don't have a way to go back and forth between versions currently, which is item 1 on the list. We'd probably also want to be able to exchange versions across galaxy instances without transferring the entire set of Workflow objects associated with the most recent workflow and reference a specific version ("Hey, please run version 1.3 of this workflow"). That'd be the last step.

[mvdbeek]

(deleted comments about fixed bugs, they were confusing me)

[hexylena]

@mvdbeek you can also "hide" comments with reason: "outdated" which may be better in the future than deleting them :)

[innovate-invent]

I would personally love to see the underlying version management be done by git. Format the json (I would prefer the workflows to be stored in yaml) so that it lends itself to git parsing such as having each tool on a separate line.
Then workflow sharing can be done using your favorite git repository host.
Just provide galaxy with the uri to clone.

[mvdbeek]

I would personally love to see the underlying version management be done by git.

Sure, but that's outside of the scope of Galaxy for now. I think it'll be a mess to deal with branches and untagged commits. We probably want to keep it simple, not going beyond anything more complex than major/minor/patch. I do agree that importing and tracking versions via some URI scheme (item 3 and 4 in the list) would be cool, but someone needs to work out the details.

Now that we have simple date-based internal versions I personally think that extending the utility of parameters, branching and other things that lead to less copy-paste for slightly different workflows is more important. But if someone wants to have a go at it it will be a nice project.

[pcm32]

One thing I've noticed is that when I export a workflow and then re-import it, versions start back from zero. Is this intended or a bug? I would expect versions to be persisted to the exported file. To be clear, when I mean export is by doing Workflow -> My workflow -> Share (drop down menu) -> Download. I have noticed that other avenues produce different output (which is a bit annoying to be honest). Happy to open a different issue for this if needed.

[mvdbeek]

Yes, that is expected. The versions you select now are just internal versions, one per saved change.
Explicit versions are the the last bullet point, but there's a lot of details to consider for implementing this.