Galaxy: Archival and replication of user analyses

[mvdbeek]

This was brought up on gitter by @lparsons, and we also had questions about this at the admin workshop in Oslo.

I am currently seeing 2 possible routes to archive an analysis in Galaxy, and they should be documented better, probably in the admin section of the GTN.

To me the ideal archival strategy would be to work exclusively with workflows and to keep input datasets, a workflow, and something resembling the workflow invocation, so that the analysis can be faithfully reproduced, including workflow parameters.

Another possibility is exporting the entire history, which currently has the drawback of not supporting collection (there's some work in https://github.com/galaxyproject/galaxy/pull/4532),
and not all analyses are confined to a single history.

[jennaj]

This is a good idea. Similiar to what I keep on a wish-list. Maybe some of the ideas would be useful: https://github.com/jennaj/support-known-issues/wiki/History-usability-ideas

[mblue9]

This is a great idea. I really need a way for users to export a record of what was actually run (versions, parameters, input dataset names...). Is this the issue for that?

Being able to extract a record of the workflow invocation would be good but if some steps are run outside the workflow then being able to extract from the history (aswell) would be good.

These ideas in Jen's list sound like what I'm looking for:

Extract history structure similar to how extract workflow works now. Elements clickable same as in workflow canvas flavored view. Note: History menu > Show structure results in a server error (and has for a while)

This would be great!:

Output the history content - datasets, tool version/paramaters, status, etc - into a tab-delimited file.

[Takadonet]

I have hired a four month co-op student to work on ability to export a record(provenance) of a history in both a machine and human readable formats.

[mblue9]

OMG @Takadonet that is some great news!!!! 🙌 🎉

[hexylena]

Any results from them @Takadonet? :)

[Takadonet]

We had a prototype working by end of the term. It did not work well with collections which is most of our workflows that we run at our lab.

Believe the route we are leading toward will be using John's Workflow invocation

[jennaj]

This came up at Galaxy Help the other day.

Even a PDF type of output would be some improvement. But I really really like the option of outputting into some type of formatted tab file that could be parsed.

Versioning would be an important consideration here, too ...

https://help.galaxyproject.org/t/can-i-download-a-printable-version-of-a-workflow/2076