Galaxy: [RFC] Decompose the rest of Galaxy into packages/... ASAP

[jmchilton]

I had been thinking the decomposed packages/ (#7524) would be an alternative to the @natefoo / @mvdbeek attempts to publish all of Galaxy to PyPI (https://github.com/galaxyproject/galaxy/issues/1472 | https://github.com/galaxyproject/galaxy/pull/921 | https://github.com/galaxyproject/galaxy/pull/5243). I was thinking the same modules would exist in a big galaxy-app thing for installing and in small libraries for embedding functionality into other applications. This would be a bit messy because it would be important the Galaxy libraries were never installed in a galaxy-app environment for instance - because there would be competing module definitions. In this way, it would be better to just have a single source of truth. Furthermore, we're making rapid progress on placing code into packages/.

Though it might slow down the Installable Galaxy initiative, I think we should try to put all the code into packages/ and just have them be the way to install Galaxy from PyPI.
Never publishing a big wheel with all the code in its big monolithic form would also mean that someday we could actually make packages/ the source of truth for particular modules - if we continue to grow to the point where we want to decompose development processes, governance, etc.. of parts of the Galaxy code base. I know a lot of people like the big monolithic approach and I'm not saying we would ever need to change... but realistically I think it doesn't scale indefinitely and it would be good to have the option open.

I think the path forward would be something like:

1) Put all the remaining modules in a galaxy-webapp package.

• Concurrently we could then try to decompose this into galaxy-core, galaxy-web-framework, and then break galaxy-core down even more... but this would all be optional, as-needed decompositions.
  2) Create a package galaxy-app (maybe?) that depends on our pinned requirements list and depends on fixed, pinned versions of all the Galaxy libraries. So it is really just a recipe for how to build a fixed, tested thing from our libraries. Everything else - scripts, web content, etc... would be placed in galaxy-webapp or things that it gets decomposed into.
  3) All the work we need to do for https://github.com/galaxyproject/galaxy/pull/921 would still need to be done, but probably for either galaxy-webapp or galaxy-app.

[mvdbeek]

Though it might slow down the Installable Galaxy initiative, I think we should try to put all the code into packages/ and just have them be the way to install Galaxy from PyPI.

💯 I don't think you can slow down something that isn't moving :).
This sounds like a good plan to me, it'll have a lot of advantages down the road.

[natefoo]

+1

I don't think you can slow down something that isn't moving :).

I did get halfway through merging dev into #921 a few months ago. 😆

[mvdbeek]

sorry, didn't know that :). Anyway, some of this stuff we can probably merge in smaller consecutive PRs ? Lots of nice commits in there.

[jmchilton]

Such a nice hackathon project to be honest:

• These packages breakdown in really independent ways
• There are a million little independent refactorings needed
• So many small details to get tests working, dependencies correct, etc...
• Sphinx docs could be added for all the projects - mostly focusing on doctests and better READMEs at first - but we could think about how to split out some of the docs for things like the mulled utilities (there are docs in galaxy-lib but not in Galaxy, that needs to be fixed).
• People could work on what they care about - galaxy-auth, galaxy-viz (or maybe galaxy-viz-framework). Everyone could have ownership of a nice unit of work.
• I think everyone who worked on it would have a sense of the problems with Galaxy's current circular dependencies and would be more careful about addressing them as regular development continued in the future.

[mvdbeek]

Could be our next go to standard project, after py3 support alsmost being done :)

[bgruening]

Not yet official but please feel free to add it to https://docs.google.com/document/d/17fwd0VylUHPgcRxnm2zfQBRP8xeUROGI1hlvwSebVFo/edit?usp=sharing

[martenson]

fwiw since we started talking sub packages I have thought this is the endgame, so 👍 from me