Pysal: Meta-package not building for 3.6

[sjsrey]

The import system for the meta package is broken on 3.6 but not 3.7.
See: Travis build 3.7 vs.3.6

[jGaboardi]

I spent some time going through the v2.2.0rc PR without finding anything obvious that was causing this. If it's passing on 3.7 & not on 3.6 on Travis, and passing for both locally it seems like that points to an issue with version of 3.6 we are pulling from Miniconda. Especially since the v2.1 passing tests before its release.

[knaaptime]

can you get tests to pass locally on 3.6?

[jGaboardi]

@knaaptime me? No, I have only had to time to review the PR (no actual testing). I thought that's way @sjsrey 's email from last week said, though. Did I misread it?

[knaaptime]

no, we're on the same page. Just trying to make sure we can parse issues arising from travis/conda/package changes

[jGaboardi]

FWIW, the spaghetti tarball build was failing for several days, seemingly due to reasons way upstream, and is now fixed. A similar issue also occurred last month. It's very likely not related to this, but might be worth a try to rerun the build just to make sure?

[jGaboardi]

Issues with importlib seem to have been the culprit in both occurrences of the spaghetti failures.

[knaaptime]

@sjsrey and I spent the morning working on this and have lots to report:

I think the tests were failing because we didnt actually specify pytest as a dependency anywhere, so we were getting a single executable that wasnt tied to a python version (i.e. pytest 3.7 being used to test against python 3.6 and 3.7 environments. But the larger issue this raised is: is our current metapackage approach too complex? and we've decided that yes, it could be much simpler, and the approach is as follows:

Currently, the metapackage downloads all the subpackages, copies their source code into the model/explore/viz structure, and re-writes their import schemes. Moving forward, we think an easier solution would be as follows. On PySAL release day:

1) freeze the subpackage released versions and build a requirements.txt file that includes each of the subpackages, pinned to the proper version.
2) the pysal metapackage will no longer download or manipulate source code. Instead, it will import the subpackages in the proper __init__ files (e.g. now pysal/model/__init__.py will have import tobler import spreg, etc) but no source code underneath--i.e. users of the metapackage will consume the subpackages directly
3) the init files will also have a little utility function that checks to see whether the version of the subpackage in the environment matches the version pinned to that pysal release and raises a warning in case the user is on a newer subpackage (its possible there could be API changes in this case)
4) Unit testing is now done only on the subpackages; the metapackage will only test whether everything imports successfully (and maybe one simple test per subpackage, just so we can see something running)
5) when the metapackage is built, it will use requirements.txt (the canonical location for specifying deps) to create two environment.yml files:
- one that essentially mirrors requirements.txt and has subpackages pinned to the "pysal release version". We setup nightly testing on this environment, which tells us whether upstream changes in numpy, scipy, etc have broken our release
- one that includes subpackages without pinning them to specific versions. We setup nightly testing on this environment too, to tell us whether changes to the subpackages have induced breakage in the meta

[knaaptime]

this should also help us with some of the confusion about the best way to interact with the library without redoing the examples... when you use a package from the meta, you should use it directly, e.g. from pysal.explore import giddy (since this is exactly what's happening under the hood anyway)

then, we need only redirect people to the subpackage documentation

[jGaboardi]

@knaaptime & @sjsrey awesome work!

[jGaboardi]

How will the stuff you learned/decided on today affect the current PySAL meta release? Will this new schema be implemented immediately?

[sjsrey]

@jGaboardi we need input/discussion from/with all the devs and interested users

[knaaptime]

there's a branch on serge's fork that implements this idea and it seems to work, so we figured we'd next ask for buy-in from the other devs

[ljwolf]

I wholeheartedly approve of this and am glad to see it now considered :smile: It solves a ton of problems, not just about distribution but also about documentation consistency (e.g. #1147).

However, in the past, we discussed this option and I recall there being worry that the upstream would look "empty."

I think this is ok, but want to raise it so we can agree it's ok.

[knaaptime]

I want to do a bit of a push on what i think of as the "soft documentation" materials... stuff like the readme and website prose that serve as people's first entry to the library. If done right, i think reworking some of that material can help smooth the "emptyness" issue

[weikang9009]

Sounds like a great solution! - making the integration of the metapackage easier and more explicit to end-users. One question (and maybe this is not a question after all ) I have is about the implication of this new structure on the API documentation for the metapackage. If there is no source code for the metapackage, can we still build docs like this on top of it?

[knaaptime]

actually the idea is to explicitly not build pysal-only docs. Instead, that link you used would essentially point here (1) to help solidify the concept of pysal as a proper metapackage, and (2) to help ensure that things #1147 happen less often because there's a single source of documentation

[knaaptime]

the idea being to more or less imitate the tidyverse model, where each of the subpackages are first-class citizens, but unified under a common banner

[knaaptime]

(more as a reminder to myself than anything else, but we'll also need to release spvcm and inequality on conda-forge)

[darribas]

Hello!

I'm too removed from release stuff to make any meaningful comments, but this sounds like a good streamlining exercise, I'm +1. One question is whether the api will change when you access a module from pysal (would the from pysal.explore.esda import Moran still be the way to go?), and to what extent that'll overlap or not with a package by package approach (can/should users be able to import esda directly?).

As for docs, I like the idea of relying on the original ones rather than replicating. Maybe one goal for the meta package is to pin and point easily from pysal.org to each package version of the docs? I think this is what's proposed but just to clarify for me. Also, would it be possible in the docs of a package to go back to the meta website for the version it's pinned to? For example, I go to pysal.org and check the docs for 2.2, then takes me to the docs for esda 2.2, could I got back to the meta website pysal.org from there? That's make the experience more integrated rather than leaving the user feeling like (s)he's "left" PySAL-land. Just food for thought.

[sjsrey]

Closing as we are going with the metalite approach.