Wrye-bash: Moving dependency management to pipenv

[GandaG]

An essential part of any python project is dependency management, a topic that is in constant evolution and hopefully will soon end (PEP 582). Since we're developing an app whose only deployment is via freezing we won't need to enter the horrifying world of packaging (warning: drama).

First, our goals:

1. Isolation - isolating our dependencies gives a few advantages over system packages:

   • Control over versioning, since we don't have to depend on the system to (not) update them, although this is arguably a bigger problem on linux than windows.
   • Independence from other scripts/packages, no need to coordinate who needs what version of which package.
   • This is arguably less of an issue while we're in python 2 since most things have been updated already (hint hint) but we'll definitely need this when we move to python 3.

2. Deterministic Builds - this requires the above and gives us:

   • Environment uniformity, since this ensures that my dev env will have exactly the same dependencies as yours which helps with...
   • Bug prevention, no more "it works on my computer...".
   • Build reproducibility, we can checkout any commit and build the exact same app as we would have then.

Are there issues with this? Of course, but they are very much minor in comparison to the advantages above:

1. We lose the ability to freely run python, everything has to go through the virtualenv. This implies some sort of replacement shell, shims, activate scripts or running everything with an additional prefix.
2. Git hooks become much harder to automate.
3. Additional steps to setup the environment.

As for the software to manage, we only have two realistic options:

1. pipenv - meant as a replacement for pip and virtualenv, this automates setting up the virtualenv and installing the packages all in one.
   
   
   
   • Usage is pretty simple, with a single, very specific use-case which happens to be ours.
   
   
   • Most IDEs have plugins for this.
   
   
   • Is able to setup shortcuts for commonly run commands, so we could do pipenv run build instead of the much longer pipenv run python scripts/build.py.
   
   
   • Plagued with bugs and regressions for a while, not sure if it has already stabilized since I don't use it much.
   
   
   • Is quite infamous for having a very opinionated lead dev which may contribute to some of its instability.
   
   
   • Unable to upgrade specific packages, will always attempt to upgrade as much as possible within the constraints you set in the Pipfile.
   
   
   • As for using its virtualenv, we need to either setup a new shell via py -2 -m pipenv shell or just prefix everything with run, such as py -2 -m pipenv run python scripts/build.py.
   
   
2. poetry - same as above, plus also packages and publishes to PyPI (which we don't care about)
   
   
   
   • Also don't have much experience with this but the cli seems largely similar to pipenv.
   
   
   • Here is a comparison to pipenv. Most of the advantages over the above seem geared towards libraries tbh.
   
   
   • Uses the standardized pyproject.toml instead of Pipfile.
   
   
   • Slightly younger project, although it seems mature enough to be seriously considered.
   
   
   • No shortcuts like above.
   
   
   • Never heard of IDE plugins for poetry, but I'm not sure.
   
   
   • Also needs to run with a prefix, py -2 -m poetry run python ... but doesn't seem to have the option to start a replacement shell. Requires filling in some basic metadata, but meh.
   
   

Overall, pipenv seems a better fit to me.

In order to implement this, there a few barriers:

• [x] Find the correct (original) Tools folder - we need some of those scripts during building.
• [ ] Properly include all distutils modules in the executable (see: 7583a4e) - virtualenvs only contain a "fake" distutils that links to the original and py2exe is a bit dumb.

Related branches: ganda-469-virtualenv.

Did I forget anything? Discuss!

[Infernio]

While we're on the topic of packaging and whatnot, does anyone know why we still bundle an ancient version of chardet? Is there something preventing us from moving to the PyPI version? It supports py2 and py3, so it's not #460.

[Utumno]

Cause nobody bothered - do you think we should add it as a dependency? Usual pros and cons but I think the pros outweigh the cons

[Infernio]

I'd like to get this done before #460 - as @GandaG mentioned, right now having to install WB dependencies system-wide isn't really a problem, because WB is the only thing I use py2 for, but on py3 it will be a big problem for me. Even bigger on Linux, where quite a few applications rely on the package manager providing the latest versions of everything.

I'll try rebasing that commit (along with the 64bit branch, because I'd like to start building 64bit nightlies) and sticking it onto nightly tomorrow.

[Infernio]

The first 64bit nightly also happens to be the first pipenv nightly :)
Some random notes:

• pipenv crashes python in powershell for me(?)
• It works in git bash and pycharm however, and...
• pipenv in pycharm is really nice. When you edit the pipfile, it will automatically offer to lock dependencies for you, the migration is literally as simple as 'go to settings > project interpreter and change it from system interpreter to pipenv environment', at which point it will automatically install all dependencies for you
• build script needed a few edits - I copy-pasted sys_real_prefix from setup.py since that was missing, but @GandaG should have a look over all that at some point. It did build correctly, so it seems to be working now

[Infernio]

For some reason, pipenv is failing to install dependencies or crashing for pretty much everyone. Arthmoor, Verayth and mpy21953 reported on Discord and AFK Mods that pipenv creates the virtualenv, then says it's installing dependencies, but when they try to py -2 -m pipenv run bash, it will error out, saying it couldn't find lz4/wxPython/etc. Running py -2 -m pipenv uninstall --all produces this:

PS C:\dev\wrye-bash-git> python.exe -m pipenv uninstall --all
Un-installing all [dev-packages] and [packages]...
Found 0 installed package, skip purging.
Environment now purged and fresh!

Note the Found 0 installed package, skip purging.. For Verayth, it also crashes if launched via git bash - the opposite of my situation, where it crashes in powershell but works fine in git bash.

In summary, pipenv seems to be highly unstable, at least on Windows. I'll try switching to poetry in a few days to see if that improves the situation at all.

[Verayth]

I haven't tried it yet, but supposedly, "Downgrading to Python 2.7.16 + pip 18.1 fix the crashes."

https://github.com/pypa/pipenv/issues/4016

[Verayth]

verified able to start bash using pipenv with Python 2.7.16 on Win10/Powershell:

choco uninstall python2
rm -r C:\Python27
choco install python2 --version 2.7.16
python.exe -m pip install pipenv
python.exe -m pipenv sync --dev
python.exe -m pipenv run bash

[Utumno]

Hey guys how do we build now? :P

EDIT:

• installed python 64 to C:\Python27-64\
• in pycharm I added the 64 bit interpreter
• installed pipenv add on on this interpreter (took ages turns out has many dependencies, most of them due to py2/3)
• 'go to settings > project interpreter and change it from system interpreter to pipenv environment' - I had to add the path to pipenv executable at C:\Python27-64\Scripts\pipenv.exe
• Installing dependencies from pipfile.lock 🤞 - did it! we have quite a few dependencies turns out - wxPython4 must have added a couple (numpy ?!)
• And gotcha

C:\Users\MrD\.virtualenvs\wrye-bash-INI3OnZK\Scripts\python.exe C:/Dropbox/eclipse_workspaces/python/wrye-bash/scripts/build.py -c
Building on Python 2.7.17 (v2.7.17:c2f86d86e6, Oct 19 2019, 21:01:17) [MSC v.1500 64 bit (AMD64)]
Traceback (most recent call last):
  File "C:/Dropbox/eclipse_workspaces/python/wrye-bash/scripts/build.py", line 546, in <module>
    with clean_repo():
  File "C:\Python27-64\lib\contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "C:/Dropbox/eclipse_workspaces/python/wrye-bash/scripts/build.py", line 519, in clean_repo
    include_ignored=True,
  File "C:\Users\MrD\.virtualenvs\wrye-bash-INI3OnZK\lib\site-packages\pygit2\repository.py", line 869, in stash
    check_error(err)
  File "C:\Users\MrD\.virtualenvs\wrye-bash-INI3OnZK\lib\site-packages\pygit2\errors.py", line 64, in check_error
    raise GitError(message)
_pygit2.GitError: cannot create blob from 'C:/Dropbox/eclipse_workspaces/python/wrye-bash/.__/_appdata/LOOT/Skyrim Special Edition/': it is a directory

Process finished with exit code 1

[Metallicow]

umm yea wxPython added numpy and pillow. I don't recall they are actually required but some widgets will use them if available. For example the agw thumbnailer will use PIL if available and it runs so much faster starting up. iirc numpy was for some unit tests and building

[Infernio]

Hey guys how do we build now? :P

py -2-64 -m pipenv run build --commit

On 15. Mar 2020, at 12:52, Utumno [e-mail] wrote:

Hey guys how do we build now? :P

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[Utumno]

Guys pipenv seems unmaintained https://github.com/pypa/pipenv/releases -> 2018.11.26 (2018-11-26)

[Infernio]

There are recent commits: https://github.com/pypa/pipenv/commits/master, and a release milestone for march: https://github.com/pypa/pipenv/milestone/7
Would have to see if anything comes of it :P

[Infernio]

Something is seriously broken: https://github.com/wrye-bash/wrye-bash/runs/810961242
For some reason pipenv is now creating a new virtualenv on every CI run and taking ~10 minutes to do so :/

[Infernio]

Moved this back to 308 and dropped the commit from nightly:

1. pipenv breaks in PowerShell for me
2. pipenv is completely broken for some people, unless they downgrade both pip and python
3. the source bundle we build includes commands (e.g. pipenv run bash) that don't work because they're assuming the repo paths - this breaks actually running WB as the python version
4. and finally, it's caused our CI times to suddenly reach 14 minutes

Screw this, I think we should stick to a requirements.txt with fixed dependency versions for now :/

[Infernio]

On the backburner - pipenv is unusable on py2, and we'll want to reconsider our options once we are on py3.