Flynn: Custom git pip packages not being installed

[mcescalante]

I've got a single dependency in my Pipfile that's being installed fine, from git (open source repo). It's a patched version of a library while I wait for a merge and release to PyPy:

requests-respectful = { git = '[email protected]:mcescalante/requests-respectful.git', editable = 'true' }

When I deploy to flynn, everything deploys and releases successfully (no errors on Pipfile.lock install, Pipfile.lock is updated), but when I navigate to my application I'm met with the error:

No module named 'requests_respectful'

I further checked by running flynn run bash and then pip list. The package is missing from the returned list, indicating that it wasn't installed.

I looked quickly over the issues in the upstream Heroku buildpack that's used for Python to see if there were outstanding issues, but this deploy actually works on Heroku itself fine, and there are no major outstanding issues apparent to me upstream that would cause this. You can find the Pipfile that I'm using in the repo: https://github.com/mcescalante/oh-fitbit-integration/blob/master/Pipfile

[lmars]

@mcescalante I suspect the install is failing as you are using an SSH URI but Flynn does not trust the GitHub host key.

I would suggest either using a HTTP URI:

https://github.com/mcescalante/requests-respectful.git

or updating the gitreceive app so that it trusts the GitHub SSH host key:

flynn -a gitreceive env set SSH_CLIENT_HOSTS="$(ssh-keyscan -H github.com)"

[mcescalante]

@lmars thanks for the reply! I saw your gitreceive command in another issue thread and have already added my own key before opening this issue - adding my own key resolved a previous error during the Pipfile.lock installation about not being able to clone the repo.

I'll give both of these a shot and see what happens and report back!

[mcescalante]

Just made both changes, and am seeing the same exact errors in my application but no problems on deploy. Here are my deploy logs, showing no errors for install:

➜  oh-fitbit-integration git:(fix-flynn) git push flynn fix-flynn:master                     
Counting objects: 4, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 557 bytes | 557.00 KiB/s, done.
Total 4 (delta 3), reused 0 (delta 0)
-----> Building fitbit...
-----> Python app detected
-----> Installing pip
-----> Installing requirements with latest pipenv...
       Installing dependencies from Pipfile.lock (82d7ab)…
-----> $ python manage.py collectstatic --noinput
       /app/.heroku/python/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
       """)
       DEBUG 2018-04-09 19:58:42,724 base 822 xxxx Configuring Raven for host: None
       INFO 2018-04-09 19:58:42,725 base 822 xxxx Raven is not configured (logging is disabled). Please see the documentation for more information.
       120 static files copied to '/tmp/build/app/staticfiles', 134 post-processed.

-----> Discovering process types
       Procfile declares types -> release, web, worker
-----> Compiled slug size is 67.38 MiB
-----> Creating release...
=====> Application deployed
To https://myclusterurlhere-redacted/fitbit.git
   da75e82..f32f030  fix-flynn -> master

And flynn run bash and pip list output:

➜  oh-fitbit-integration git:(fix-flynn) flynn run bash
flynn@hash-redaced:~$ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
amqp (2.2.2)
arrow (0.12.1)
billiard (3.5.0.3)
celery (4.1.0)
certifi (2018.1.18)
chardet (3.0.4)
defusedxml (0.5.0)
dj-database-url (0.5.0)
Django (2.0.4)
gunicorn (19.7.1)
idna (2.6)
kombu (4.1.0)
multidict (4.1.0)
oauthlib (2.0.7)
pip (9.0.3)
pipenv (11.9.0)
psycopg2 (2.7.4)
PyJWT (1.6.1)
python-dateutil (2.7.2)
python-social-auth (0.3.6)
python3-openid (3.1.0)
pytz (2018.3)
PyYAML (3.12)
raven (6.6.0)
redis (2.10.6)
requests (2.18.4)
requests-oauthlib (0.8.0)
setuptools (39.0.1)
six (1.11.0)
social-auth-core (1.7.0)
urllib3 (1.22)
vcrpy (1.11.1)
vine (1.1.4)
virtualenv (15.2.0)
virtualenv-clone (0.3.0)
wheel (0.31.0)
whitenoise (3.3.1)
wrapt (1.10.11)
yarl (1.1.1)

[titanous]

This is a duplicate of #3932. I believe that this is due to some assumptions that are made in the python buildpack about paths resulting in an invalid/incorrect path in an egg-link file.

[lmars]

There is also this recently closed issue: https://github.com/heroku/heroku-buildpack-python/issues/654.

Try the latest buildpack with:

flynn env set BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-python#v132

[mcescalante]

@titanous ah, yeah, that sounds correct. That issue slipped right by me, I suppose because I was looking for the wrong words in my issue searches before opening this 😄.

@lmars great idea, I just gave that a shot but sadly I'm still getting the No module named 'requests_respectful' with my buildpack set in env to BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-python#v132

[Darkless012]

Well, since I've got stuck on this issue (thinking it is new pipenv issue), I'd like to add Flynn vs Heroku comparison, for the sake of completeness. (in my case used luminol package as editable dependency in Pipenv file)

Flynn:

flynn@x:~$ python
Python 3.6.6 (default, Aug  1 2018, 21:10:48) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', '/app', '/app/.heroku/python/lib/python36.zip', '/app/.heroku/python/lib/python3.6', '/app/.heroku/python/lib/python3.6/lib-dynload', '/app/.heroku/python/lib/python3.6/site-packages']

flynn@x:~$ cat /app/.heroku/python/lib/python3.6/site-packages/easy-install.pth
/tmp/build//app/src/luminol/src

Heroku:

~ $ python
Python 3.6.6 (default, Aug  1 2018, 21:10:15) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.path)
['', '/app', '/app/.heroku/python/lib/python36.zip', '/app/.heroku/python/lib/python3.6', '/app/.heroku/python/lib/python3.6/lib-dynload', '/app/.heroku/python/lib/python3.6/site-packages', '/app/src/luminol/src']


~ $ cat /app/.heroku/python/lib/python3.6/site-packages/easy-install.pth
/app/src/luminol/src

[Darkless012]

TLDR; quickfix:

store script below into /bin/post_compile folder:

#!/usr/bin/env bash

set +e
find .heroku/python/lib/python*/site-packages/ -name "*.pth" -print0 2> /dev/null | xargs -r -0 -n 1 sed -i -e "s#[^>]*/app/#$(pwd)/#" &> /dev/null
set -e

EXPLANATION:

Python buildpack (https://github.com/heroku/heroku-buildpack-python/blob/v144/bin/compile)

does this particular steps in following order:

#L178: source "$BIN_DIR/steps/hooks/pre_compile"
#L201: source "$BIN_DIR/steps/pipenv-python-version"
#L256: source "$BIN_DIR/steps/eggpath-fix"
#L295: source "$BIN_DIR/steps/pip-install"
#L359: source "$BIN_DIR/steps/hooks/post_compile"
#L363: source "$BIN_DIR/steps/eggpath-fix2

And the interesting parts here are in /eggpath-fix and /eggpath-fix2

In the /eggpath-fix
It replaces (via sed) s#/app/#/$(pwd)/# the /app/ ~> /$(pwd)/

In the /eggpath-fix2
It replaces (via sed) s#$(pwd)#/app# the $(pwd) ~> /app

For some reason Heroku ends up after eggpath-fix2 in this state:
$(pwd) is evaluated to /tmp/build_HASH
/tmp/build_HASH/src/luminol/src ~> /app/src/luminol/src

But Flynn ends up after eggpath-fix2 in state:
$(pwd) is evaluated to /tmp/build/app
/tmp/build//tmp/build/app/src/luminol/src ~> /tmp/build//app/src/luminol/src

Which you can see in previous comment.
Not sure where and what when wrong (that substitutions went crazy) but the proposed fix (ran as post_compile script which happens nearly before eggpath-fix2) and should correct any wrong set paths during flynn's python buildpack execution using sed:

s#[^>]*/app/#$(pwd)/#
[^>]* (non-greedy matching) - courtesy of http://0x2a.at/blog/2008/07/sed--non-greedy-matching/

so this should fix /tmp/build//tmp/build/app/src/luminol/src ~> /tmp/build/app/src/luminol/src
which will eventually gets crunched by eggpath-fix2 into /app/src/luminol/src resulting in correct python paths.

Extended version of the TLDR; script with debug

#!/usr/bin/env bash

DEBUG=true

if [ "$DEBUG" = true ] ; then
    echo "PWD : $(pwd)"
    echo "Looking for .pth packages"
    find .heroku/python/lib/python*/site-packages/ -name "*.pth"
    echo "------- Original content(s) -------"
    find .heroku/python/lib/python*/site-packages/ -name "*.pth" -print0 2> /dev/null | xargs -r -0 -n 1 cat
fi

set +e
# http://0x2a.at/blog/2008/07/sed--non-greedy-matching/
find .heroku/python/lib/python*/site-packages/ -name "*.pth" -print0 2> /dev/null | xargs -r -0 -n 1 sed -i -e "s#[^>]*/app/#$(pwd)/#" &> /dev/null
set -e

if [ "$DEBUG" = true ] ; then
    echo "------- Replaced content(s) -------"
    find .heroku/python/lib/python*/site-packages/ -name "*.pth" -print0 2> /dev/null | xargs -r -0 -n 1 cat
fi

[titanous]

Flynn is unmaintained and our infrastructure will shut down on June 1, 2021. See the README for details.