Napari: Use namespace packages internally for "builtin" plugins

[sofroniewn]

There have been a couple of discussions of namespace packages and our plugins in the past, including #139 from @kne42, but I'd like to revisit, particularly as I'm working on #1800 and we have things like #1593 raised and we've already made napari-svg

The big question I have is around on repo structure/ package organization for our "builtin plugins" and if we can have multiple packages in same repo? Or do we need one repo per package?

There's good discussion of this on the namespace packages page https://packaging.python.org/guides/packaging-namespace-packages

If we go the many package per repo approach we could make a plugins directory and add

• napari-console (would be product of this PR)
• napari-svg (right now is own repo)
• napari-movie (would be where #780 ends up living)
• napari-io (would be just our "builtins")

We also would need to have "builtins" anymore as everything could be distributed like this.

I've already run into a bunch of inconveniences with napari-svg where you need to make synchronized changes across both repos, and the question of where to make new issues for beginners is very real. I also find myself less good at monitoring issues in other repos.

We'd need to adjust our tagging/ release process etc so we can independently release each of these.

Very curious what others think? If we want to try this out first step would just be to move over napari-svg!!

[jni]

@sofroniewn Yeah I'm super not sure! I don't mean that in a negative way, just in a "it's a pickle!" way! 😂 I do agree that the many-repo approach can get out of hand. On the other hand, I've heard @stefanv say before that managing namespace packages is painful. Not sure whether that has evolved much since the scikits.stuff happened. Additionally, it makes the napari repo itself rather unwieldy. I also don't know what we would consider "worthy" of including in the main repo, vs what is actually a plugin, separately maintained...?

I think this would be a good discussion to have in the team/euro meetings! And maybe people from the Jupyter side like @choldgraf have comments about single repo vs monorepo...?

[choldgraf]

my personal preference is for more, smaller, more tightly-scoped packages rather than a single mono-repo, but I think it depends on how you want the package ecosystem to evolve. E.g. with JupyterBook, we've gone with lots of little sphinx-<name> repositories (like sphinx-thebe) because those repos are designed to be modular and reusable across the Sphinx ecosystem. In my opinion it helps keep the issues / maintenance burden / etc more tightly-scoped around each and helps people engage with only the tools they want to contribute to instead of requiring that they learn a whole stack. But this works for the kind of technology that Jupyter Book offers, I'm not sure what the nuances are for Napari, so all I can do is just offer the perspective from Jupyter Book.

One other thing probably worth noting - a challenge with monorepos that I've noticed across projects is that it makes it really hard to know which subsets of the repo are being well maintained / monitored / etc. With tightly-scoped repos, I can look at the issues, contributor list, etc and know how things are going. With monorepos this is trickier because I have to filter open issues to include only the ones relevant to a folder, and in practice the repository upkeep isn't good enough to keep really good tags etc to be easy to follow. As a result I just assume monorepos of the "list of extensions" variety are generally a grab-bag of quality that I can't trust without doing a deep dive.

[sofroniewn]

This is very helpful input @choldgraf, thanks!!

Given the comments from both @jni and @choldgraf I'm now leaning against the monorepo approach. I think things will scale better with multirepo and ultimately make maintenance burden and discoverability lighter.

I'll leave this open for more discussion though.