pipenv, version 11.10.0
I have a submodule ./dep1 in my git repo and am adding it to pipenv via pipenv install -e dep1, which works just fine and adds the line
"abc123" = {path = "./dep1", editable = true}
to my Pipfile. Then I use a Dockerfile as follows:
FROM kennethreitz/pipenv
COPY ./app /app
COPY ./dep1 /app/dep1
CMD python3 start.py
where I added the third line, so the submodule is available in the WORKDIR. However, upon docker build I am getting the following error
+ pipenv install --deploy --system
Installing -e ./dep1…
Error: An error occurred while installing -e ./dep1!
./dep1 should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+
This is unexpected, since ./dep1 is a path to a local project, relative to the WORKDIR in the docker image with the Pipfile, which is /app. Am I missing something or is this a bug?
peter-ki
All 17 comments
Hi @peter-ki! Thanks for using pipenv! I need to update the docker instance to use the most current version of pipenv. Can you try rewriting your dockerfile to pull from python:3.6 or whichever version you are using and then pip install --upgrade pipenv? I'm wondering if our latest release fixes this issue. Additionally could you post your pipfile and Dockerfile? Also would it be possible to launch an interactive docker session and run the commands requested by the issue template so I can see the output? I know that's a lot, docker issues can be problematic to troubleshoot.
erinxocon
on 25 Jun 2018
The dockerfile would be closer to ours in the repo:
FROM ubuntu:18.04
# -- Install Pipenv:
RUN apt update && apt install python3-pip git -y && pip3 install pipenv
ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8
# -- Install Application into container:
RUN set -ex && mkdir /app
WORKDIR /app
# -- Adding Pipfiles
ONBUILD COPY Pipfile Pipfile
ONBUILD COPY Pipfile.lock Pipfile.lock
# -- Install dependencies:
ONBUILD RUN set -ex && pipenv install --deploy --system
@erinxocon thank you very much for the fast response!
Interesting. I tried to work with the Dockerfile you provided, which seems to take the process a step further. I get a different error, though:
+ pipenv install --deploy --system
Installing -e file:///app/data-cache...
Obtaining file:///app/data-cache
Installing collected packages: DataCache
Running setup.py develop for DataCache
Successfully installed DataCache
Adding -e file:///app/data-cache to Pipfile's [packages]...
Your Pipfile.lock (34a2ab) is out of date. Expected: (6bd758).
Aborting deploy.
The Pipfile is
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[packages]
kafka = "*"
pandas = "*"
numpy = "*"
"1bc1e69" = {path = "./data-cache", editable = true}
[dev-packages]
[requires]
python_version = "3.6"
and the Dockerfile now is
FROM ubuntu:18.04
# -- Install Pipenv:
RUN apt update && apt install python3-pip git -y && pip3 install pipenv
ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8
# -- Install Application into container:
RUN set -ex && mkdir /app
COPY ./services /app
COPY ./data-cache /app/data-cache
WORKDIR /app
# -- Adding Pipfiles
COPY Pipfile Pipfile
COPY Pipfile.lock Pipfile.lock
# -- Install dependencies:
RUN set -ex && pipenv install --deploy --system
CMD python3 start.py
Hm...
I also could run in the container interactively. But what does commands requested by the issue template refer to?
peter-ki
on 25 Jun 2018
@peter-ki Thanks for the info. So when you create a bug report we ask you to run the command python -m pipenv.help and paste the output, it contains diagnostic info for use in troubleshooting the problem. And since yours is with a local package it would be helpful to get a capture of that. So if you could run python -m pipenv.help in the interactive session and paste the output that would be appreciated!
erinxocon
on 25 Jun 2018
@peter-ki Also what vesion of pipenv are you using to generate the lockfile? Can you try updating to our latest release using pip/pip3 install pipenv --upgrade and regenerate the lockfile and then rebuild the docker container with the new lockfile? The error is telling you that your lockfile is out of date and can't be used with --deploy
erinxocon
on 25 Jun 2018
Ah sorry, got it. Here you go:
root@36f1e613ddbd:/app# python3 -m pipenv.help
$ python -m pipenv.help output
Pipenv version: '2018.6.25'
Pipenv location: '/usr/local/lib/python3.6/dist-packages/pipenv'
Python location: '/usr/bin/python3'
Other Python installations in PATH:
3.6:/usr/bin/python3.6m3.6:/usr/bin/python3.63.6.5:/usr/bin/python3
PEP 508 Information:
{'implementation_name': 'cpython',
'implementation_version': '3.6.5',
'os_name': 'posix',
'platform_machine': 'x86_64',
'platform_python_implementation': 'CPython',
'platform_release': '4.9.87-linuxkit-aufs',
'platform_system': 'Linux',
'platform_version': '#1 SMP Wed Mar 14 15:12:16 UTC 2018',
'python_full_version': '3.6.5',
'python_version': '3.6',
'sys_platform': 'linux'}
System environment variables:
LC_ALLLS_COLORSLESSCLOSELANGKAFKA_SERVER_ENDPOINTSHOSTNAMEPWDHOMETERMSHLVLPATHLESSOPEN_PYTHONDONTWRITEBYTECODEPIP_PYTHON_PATH
Pipenv–specific environment variables:
Debug–specific environment variables:
PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binLANG:C.UTF-8PWD:/app
Contents of Pipfile ('/app/Pipfile'):
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[packages]
kafka = "*"
pandas = "*"
numpy = "*"
"1bc1e69" = {path = "./data-cache", editable = true}
datacache = {editable = true, file = "file:///app/data-cache"}
[dev-packages]
[requires]
python_version = "3.6"
Contents of Pipfile.lock ('/app/Pipfile.lock'):
{
"_meta": {
"hash": {
"sha256": "599ae093fa031e777034f7217b53754baeca64542ef99125d2603a6b2834a2ab"
},
"pipfile-spec": 6,
"requires": {
"python_version": "3.6"
},
"sources": [
{
"name": "pypi",
"url": "https://pypi.python.org/simple",
"verify_ssl": true
}
]
},
"default": {
"1bc1e69": {
"editable": true,
"path": "./data-cache"
},
"kafka": {
"hashes": [
"sha256:35a49e9a432f6792aad48b2ab09d03867357aaa06eacba2678b7fd0808156e5c",
"sha256:ffb3348cd918c31344b1a2eb61f692cbc423c83b1b06466a86de3db8a3f6a018"
],
"index": "pypi",
"version": "==1.3.5"
},
"numpy": {
"hashes": [
"sha256:07379fe0b450f6fd6e5934a9bc015025bb4ce1c8fbed3ca8bef29328b1bc9570",
"sha256:085afac75bbc97a096744fcfc97a4b321c5a87220286811e85089ae04885acdd",
"sha256:2d6481c6bdab1c75affc0fc71eb1bd4b3ecef620d06f2f60c3f00521d54be04f",
"sha256:2df854df882d322d5c23087a4959e145b953dfff2abe1774fec4f639ac2f3160",
"sha256:381ad13c30cd1d0b2f3da8a0c1a4aa697487e8bb0e9e0cbeb7439776bcb645f8",
"sha256:385f1ce46e08676505b692bfde918c1e0b350963a15ef52d77691c2cf0f5dbf6",
"sha256:4d278c2261be6423c5e63d8f0ceb1b0c6db3ff83f2906f4b860db6ae99ca1bb5",
"sha256:51c5dcb51cf88b34b7d04c15f600b07c6ccbb73a089a38af2ab83c02862318da",
"sha256:589336ba5199c8061239cf446ee2f2f1fcc0c68e8531ee1382b6fc0c66b2d388",
"sha256:5edf1acc827ed139086af95ce4449b7b664f57a8c29eb755411a634be280d9f2",
"sha256:6b82b81c6b3b70ed40bc6d0b71222ebfcd6b6c04a6e7945a936e514b9113d5a3",
"sha256:6c57f973218b776195d0356e556ec932698f3a563e2f640cfca7020086383f50",
"sha256:758d1091a501fd2d75034e55e7e98bfd1370dc089160845c242db1c760d944d9",
"sha256:8622db292b766719810e0cb0f62ef6141e15fe32b04e4eb2959888319e59336b",
"sha256:8b8dcfcd630f1981f0f1e3846fae883376762a0c1b472baa35b145b911683b7b",
"sha256:97fa8f1dceffab782069b291e38c4c2227f255cdac5f1e3346666931df87373e",
"sha256:9d69967673ab7b028c2df09cae05ba56bf4e39e3cb04ebe452b6035c3b49848e",
"sha256:9e1f53afae865cc32459ad211493cf9e2a3651a7295b7a38654ef3d123808996",
"sha256:a4a433b3a264dbc9aa9c7c241e87c0358a503ea6394f8737df1683c7c9a102ac",
"sha256:baadc5f770917ada556afb7651a68176559f4dca5f4b2d0947cd15b9fb84fb51",
"sha256:c725d11990a9243e6ceffe0ab25a07c46c1cc2c5dc55e305717b5afe856c9608",
"sha256:d696a8c87315a83983fc59dd27efe034292b9e8ad667aeae51a68b4be14690d9",
"sha256:e1864a4e9f93ddb2dc6b62ccc2ec1f8250ff4ac0d3d7a15c8985dd4e1fbd6418"
],
"index": "pypi",
"version": "==1.14.5"
},
"pandas": {
"hashes": [
"sha256:211cfdb9f72f26d2ede21c751d27e08fed4434d47fb9bb82ebc8ff753888b8b6",
"sha256:28fd087514616549a0e3259cd68ac88d7eaed6bd3062017a7f312e27941266bd",
"sha256:2fb7c63138bd5ead296b18b2cb6abd3a394f7581e5ae052b02b27df8244b03ca",
"sha256:372435456c349a8d39ff001967b161f6bd29d4c3de145a4cf9b366648defbb1f",
"sha256:3790a3348ab0f416e58061d21693cb662fbb2f638001b94bf2b2199fedc1b1c2",
"sha256:437a6e906a6717a9ed2627cf6e7895b63dfaa0172567cbd75a553f55cf78cc17",
"sha256:50b52af2af2e15f4aeb2fe196da073a8c131fa02e433e105d95ce40016df5690",
"sha256:720daad75b5d35dd1b446842210c4f3fd447464c9c0884972f3f12b213a9edd1",
"sha256:b4fb71acbc2709b8f5993cb4b5445d8182864f11c39787e317aae39f21206270",
"sha256:b704fd73022342cce612996de495a16954311e0c0cf077c1b83d5cf0b9656a60",
"sha256:cbbecca0c7af6a2160b2d6ba30becc286824a98c61dcc6a41fada664f226424c",
"sha256:d2a071de755cc8ee7784e1b4c7b9b643d951d35c8adea7d64fe7c57cff9c47a7",
"sha256:d8154c5c68713a82461aba735832f0b4692be8a45a0a340a303bf90d6f80f36f",
"sha256:e1b86f7c55467ce1f6c12715f2fd1817f4a909b5c8c39bd4b5d2415ef2b04bd8",
"sha256:fcc63e8134516e93e16eb4ceac9afaa51f4adc5bf58efddae7cbc562f5b77dd0"
],
"index": "pypi",
"version": "==0.23.1"
},
"python-dateutil": {
"hashes": [
"sha256:1adb80e7a782c12e52ef9a8182bebeb73f1d7e24e374397af06fb4956c8dc5c0",
"sha256:e27001de32f627c22380a688bcc43ce83504a7bc5da472209b4c70f02829f0b8"
],
"version": "==2.7.3"
},
"pytz": {
"hashes": [
"sha256:65ae0c8101309c45772196b21b74c46b2e5d11b6275c45d251b150d5da334555",
"sha256:c06425302f2cf668f1bba7a0a03f3c1d34d4ebeef2c72003da308b3947c7f749"
],
"version": "==2018.4"
},
"six": {
"hashes": [
"sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
"sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
],
"version": "==1.11.0"
}
},
"develop": {}
}
The pipenv is the latest. To be sure I did run
pip3 install pipenv --upgrade
pipenv clean
pipenv update
But the same issue remains.
Reviewing the diagnostic info I realize the Pipfiles differ. There is the extra line
datacache = {editable = true, file = "file:///app/data-cache"}
in the Pipfile in the docker, which is missing in the Pipfile on the host.
peter-ki
on 25 Jun 2018
when the pipfile and lockfile are in sync let me know if you can rebuild the docker container!
erinxocon
on 25 Jun 2018
also try removing the following line:
"1bc1e69" = {path = "./data-cache", editable = true}
From:
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[packages]
kafka = "*"
pandas = "*"
numpy = "*"
"1bc1e69" = {path = "./data-cache", editable = true}
datacache = {editable = true, file = "file:///app/data-cache"}
[dev-packages]
[requires]
python_version = "3.6"
I don't think you need them both, the one I'm asking to be removed is an older syntax and we've since changed how we resolve local packages.
erinxocon
on 25 Jun 2018
Well, the line
"1bc1e69" = {path = "./data-cache", editable = true}
got created on the host and the line
datacache = {editable = true, file = "file:///app/data-cache"}
got added upon
RUN set -ex && pipenv install --deploy --system
in the docker. So the Pipfiles are in sync before pipenv install is called in docker. To test, I omitted pipenv install line from the Dockerfile and the image got built. I attached to the container and ran
pipenv install --deploy --system --verbose
interactively, while looking at the Pipfile before and after.
Deleting the first line and keeping the second does not work, because on the host "file:///app/data-cache" does not resolve, so pipenv update fails with the following error:
Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
ne_round
for dep in self._iter_dependencies(best_match):
File "/usr/local/Cellar/pipenv/11.10.0/libexec/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 275, in _iter_dependencies
for dependency in self.repository.get_dependencies(ireq):
File "/usr/local/Cellar/pipenv/11.10.0/libexec/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 215, in get_dependencies
legacy_results = self.get_legacy_dependencies(ireq)
File "/usr/local/Cellar/pipenv/11.10.0/libexec/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 234, in get_legacy_dependencies
dist = ireq.get_dist()
File "/usr/local/Cellar/pipenv/11.10.0/libexec/lib/python3.6/site-packages/pipenv/vendor/pip9/req/req_install.py", line 1069, in get_dist
egg_info = self.egg_info_path('').rstrip('/')
File "/usr/local/Cellar/pipenv/11.10.0/libexec/lib/python3.6/site-packages/pipenv/vendor/pip9/req/req_install.py", line 483, in egg_info_path
filenames = os.listdir(base)
FileNotFoundError: [Errno 2] No such file or directory: '/app/data-cache'
Sorry don't know what line of thought I was on. I have thought of something else.
FROM ubuntu:18.04
# -- Install Pipenv:
RUN apt update && apt install python3-pip git -y && pip3 install pipenv
ENV LC_ALL C.UTF-8
ENV LANG C.UTF-8
# -- Install Application into container:
RUN set -ex && mkdir /app
COPY ./services /app
COPY ./data-cache /app/data-cache
WORKDIR /app
# -- Adding Pipfiles
COPY Pipfile Pipfile
COPY Pipfile.lock Pipfile.lock
# -- Install dependencies:
RUN set -ex && pipenv install && pipenv --deploy --system
CMD python3 start.py
Can you try that dockerfile with your original pipfile before i asked you to remove that line? It should relock the pipfile after the install which will allow deploy to work. I won't be able to create a small test for another hour or so, apologies for asking you to test so many things.
erinxocon
on 25 Jun 2018
Cool, that works (after fixing the line which should read
RUN set -ex && pipenv install && pipenv install --deploy --system
, I guess) while
RUN set -ex && pipenv install --deploy --system
fails. Could you give a short explanation?
peter-ki
on 26 Jun 2018
Anyway, thank you very much for resolving the issue!
peter-ki
on 26 Jun 2018
I don't think my solution is the proper way, I'm working on another, but essentially, when the pipfile references the local package, the lockfile becomes different for each platform, your host machine and the docker instance since they are in difference places. pipenv install creates a virtualenv, and relocks the pipfile and then pipenv install --deploy --system uses that new updated lockfile for the install. So we are essentially creating a virtualenv when it's not needed to sync lockfiles after you install the new local dep...I'm looking into a better Dockerfile solution now.
erinxocon
on 26 Jun 2018
I am 100% confused about this, @erinxocon and I have been hashing it out all afternoon. This is definitely not a good thing and we will sort out why it's happening. Sorry for the issue, --deploy should never modify your pipfile!
techalchemy
on 26 Jun 2018
Oh... I think I understand what happened... the new requirements library resolves the names of python packages. It must have a bug in it that does two things
1, It resolves to absolute paths and converts to URIs
- It ignores existing entries with other names
These are not good.
techalchemy
on 26 Jun 2018
(For the record it was much simpler than that, thankfully)
techalchemy
on 26 Jun 2018
Related issues
hynek
·
3Comments
fbender
·
3Comments
konstin
·
3Comments
jacebrowning
·
3Comments
marc-fez
·
3Comments
Most helpful comment
I don't think my solution is the proper way, I'm working on another, but essentially, when the pipfile references the local package, the lockfile becomes different for each platform, your host machine and the docker instance since they are in difference places.
pipenv installcreates a virtualenv, and relocks the pipfile and thenpipenv install --deploy --systemuses that new updated lockfile for the install. So we are essentially creating a virtualenv when it's not needed to sync lockfiles after you install the new local dep...I'm looking into a better Dockerfile solution now.