Pipenv: create requirements.txt without virtualenv?

Created on 31 Aug 2018 · 15Comments · Source: pypa/pipenv

hi! Is there a way to create a requrirements.txt file without creating a virtualenv? I'm trying to build a series of lambdas that each have separate Pipfiles.

After cding into each lambdas fodler, I'm running pipenv lock -r | pip install -r /dev/stdin --target src. Unfortunately, it creates a virtualenv per invocation which wastes time.

Is there an easy way to just translate the Pipfile.lock -> requirements.txt format without creating the virtualenv?

Thanks!

Discussion

Source

AlJohri

👍1

Most helpful comment

here's a slightly more full-featured version of @uranusjr 's workaround for this issue, which I would still love to see the pipenv maintainers reconsider.

def pipenv_lock(lockfile: str):
    """If you rely on `pipenv lock -r to do this for you, it will force-create a venv for you.
     This is a workaround that gives the same result.
    """
    from pipenv.vendor.requirementslib import Requirement
    # this is imported late because it very rudely does a basicConfig on logging
    # which will prevent basicConfig in your own scripts from taking effect
    section = 'default' if lockfile == 'requirements.txt' else 'develop'
    with open('Pipfile.lock') as f:
        with open(lockfile, 'w') as outf:
            data = json.load(f)
            for source in data.get('_meta', {}).get('sources', []):
                url = source['url']
                if source['name'] == 'pypi':
                    outf.write(f'-i {url}\n')
                else:
                    outf.write(f'--extra-index-url {url}\n')
            for name, entry in data[section].items():
                r = Requirement.from_pipfile(name, entry)
                outf.write(r.as_line() + '\n')

petergaultney on 29 Aug 2019

👍5

All 15 comments

By accessing the underlying requirementslib:

import json
from requirementslib import Requirement

with open('Pipfile.lock') as f:
    data = json.load(f)
for name, entry in data['default'].items():
    r = Requirement.from_pipfile(name, entry)
    print(r.as_line())

If you don’t want to install requirementslib separately, Pipenv vendors a (slightly outdated) version in pipenv.vendor.requirementslib.

uranusjr on 31 Aug 2018

thanks @uranusjr!

would you consider integrating this into the Pipenv cli or a small independent CLI within requirementslib

AlJohri on 31 Aug 2018

Requirementslib can load full pipfiles even. I see no reason to force people to make a virtualenv...

techalchemy on 31 Aug 2018

@techalchemy But that version is not in Pipenv yet. We really should make a release :p

Regarding the virtualenv, IMO it depends on what lock -r actually does. If it’s locking into a requirements.txt (instead of Pipfile.lock), a virtualenv should be required; if it’s simply exporting Pipfile.lock, however, I’d agree the virtualenv shouldn’t be required.

(Side note, this is why I implemented this as a separate command in Passa as passa freeze, to make the distinction clear.)

uranusjr on 31 Aug 2018

Reopening until we figure out how this feature should be implemented.

uranusjr on 31 Aug 2018

I mean it’s intended to export the lockfile. If it needs to regenerate it that requires a virtualenv. If one exists and is current, it can load the existing lockfile and export. It is not intended to simply dump the environment as there is already a command for that in pip ;)

techalchemy on 31 Aug 2018

no need to integrate this

kennethreitz42 on 1 Sep 2018

FWIW, I'm getting this weird case where if no virtualenv exists, the piped command both try to create a virtualenv at the same time. This is the command:

pipenv lock -r | pipenv run pip install -r /dev/stdin --target src

I'm getting by, by triggering the creation of the virtualenv with a noop for now. Might be a good case for a pipenv new command.

pipenv run true

Multiple virtualenvs created at once traceback:

building public-pool-scraper...
Creating a virtualenv for this project...
Pipfile: /codebuild/output/src782213354/src/apps/public-pool-scraper/Pipfile
Creating a virtualenv for this project...
Pipfile: /codebuild/output/src782213354/src/apps/public-pool-scraper/Pipfile
Using /usr/local/bin/python3.6m (3.6.5) to create virtualenv...
Using /usr/local/bin/python3.6m (3.6.5) to create virtualenv...
Running virtualenv with interpreter /usr/local/bin/python3.6m
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu/bin/python3.6m
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 2343, in <module>
    main()
  File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 712, in main
    symlink=options.symlink)
  File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 927, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 1233, in install_python
    shutil.copyfile(executable, py_executable)
  File "/usr/local/lib/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
OSError: [Errno 26] Text file busy: '/root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu/bin/python3.6m'

Virtualenv location: 
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 593, in lock
    ensure_project(three=three, python=python, pypi_mirror=pypi_mirror)
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 614, in ensure_project
    path_to_python = which('python') or which('py')
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 115, in which
    p = os.path.join(location, 'bin', command)
  File "/usr/local/lib/python3.6/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Running virtualenv with interpreter /usr/local/bin/python3.6m
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu/bin/python3.6m
Also creating executable in /root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu/bin/python
Installing setuptools, pip, wheel...
  Complete output from command /root/.local/share/v...2xlKu/bin/python3.6m - setuptools pip wheel:
  Failed to import the site module
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu/lib/python3.6/site.py", line 67, in <module>
  File "/root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu/lib/python3.6/os.py", line 27, in <module>
ModuleNotFoundError: No module named 'stat'
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 2343, in <module>
    main()
  File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 712, in main
    symlink=options.symlink)
  File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 947, in create_environment
    download=download,
  File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 904, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 796, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /root/.local/share/v...2xlKu/bin/python3.6m - setuptools pip wheel failed with error code 1
Error while trying to remove the /root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu env: 
No such file or directory

Virtualenv location: 
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 701, in run
    do_run(command=command, args=args, three=three, python=python, pypi_mirror=pypi_mirror)
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 2244, in do_run
    ensure_project(three=three, python=python, validate=False, pypi_mirror=pypi_mirror)
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 614, in ensure_project
    path_to_python = which('python') or which('py')
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 115, in which
    p = os.path.join(location, 'bin', command)
  File "/usr/local/lib/python3.6/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

AlJohri on 2 Sep 2018

Why would you do that?

Sent from my iPhone

On Sep 2, 2018, at 11:32 AM, Al Johri notifications@github.com wrote:

FWIW, I'm getting this weird case where if no virtualenv exists, the piped command both try to create a virtualenv at the same time. This is the command:

pipenv lock -r | pipenv run pip install -r /dev/stdin --target src
I'm getting by triggering the creation of the virtualenv with a noop for now. Might be a good case for a pipenv new command.

pipenv run true
Multiple virtualenvs created at once traceback:

building public-pool-scraper...
Creating a virtualenv for this project...
Pipfile: /codebuild/output/src782213354/src/apps/public-pool-scraper/Pipfile
Creating a virtualenv for this project...
Pipfile: /codebuild/output/src782213354/src/apps/public-pool-scraper/Pipfile
Using /usr/local/bin/python3.6m (3.6.5) to create virtualenv...
Using /usr/local/bin/python3.6m (3.6.5) to create virtualenv...
Running virtualenv with interpreter /usr/local/bin/python3.6m
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu/bin/python3.6m
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 2343, in
main()
File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 712, in main
symlink=options.symlink)
File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 927, in create_environment
site_packages=site_packages, clear=clear, symlink=symlink))
File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 1233, in install_python
shutil.copyfile(executable, py_executable)
File "/usr/local/lib/python3.6/shutil.py", line 121, in copyfile
with open(dst, 'wb') as fdst:
OSError: [Errno 26] Text file busy: '/root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu/bin/python3.6m'

Virtualenv location:
Traceback (most recent call last):
File "/usr/local/bin/pipenv", line 11, in
sys.exit(cli())
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
return self.main(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
return callback(args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 593, in lock
ensure_project(three=three, python=python, pypi_mirror=pypi_mirror)
File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 614, in ensure_project
path_to_python = which('python') or which('py')
File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 115, in which
p = os.path.join(location, 'bin', command)
File "/usr/local/lib/python3.6/posixpath.py", line 80, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Running virtualenv with interpreter /usr/local/bin/python3.6m
Using base prefix '/usr/local'
New python executable in /root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu/bin/python3.6m
Also creating executable in /root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu/bin/python
Installing setuptools, pip, wheel...
Complete output from command /root/.local/share/v...2xlKu/bin/python3.6m - setuptools pip wheel:
Failed to import the site module
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu/lib/python3.6/site.py", line 67, in
File "/root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu/lib/python3.6/os.py", line 27, in

ModuleNotFoundError: No module named 'stat'

...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 2343, in
main()
File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 712, in main
symlink=options.symlink)
File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 947, in create_environment
download=download,
File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 904, in install_wheel
call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
File "/usr/local/lib/python3.6/site-packages/virtualenv.py", line 796, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /root/.local/share/v...2xlKu/bin/python3.6m - setuptools pip wheel failed with error code 1
Error while trying to remove the /root/.local/share/virtualenvs/public-pool-scraper-LqI2xlKu env:
No such file or directory

Virtualenv location:
Traceback (most recent call last):
File "/usr/local/bin/pipenv", line 11, in
sys.exit(cli())
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
return self.main(args, *kwargs)
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
return callback(args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 701, in run
do_run(command=command, args=args, three=three, python=python, pypi_mirror=pypi_mirror)
File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 2244, in do_run
ensure_project(three=three, python=python, validate=False, pypi_mirror=pypi_mirror)
File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 614, in ensure_project
path_to_python = which('python') or which('py')
File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 115, in which
p = os.path.join(location, 'bin', command)
File "/usr/local/lib/python3.6/posixpath.py", line 80, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.

kennethreitz42 on 2 Sep 2018

This confuses me. We’ve discussed an init command in the past. I’m on the fence, Kenneth was -1 I think and TP was +1. Lots of people run pipenv --python x.y with no other commands as essentially an init command anyway.

techalchemy on 2 Sep 2018

Pipenv initially had an init command, and it was removed, based on usability studies.

Sent from my iPhone

On Sep 2, 2018, at 12:07 PM, Dan Ryan notifications@github.com wrote:

This confuses me. We’ve discussed an init command in the past. I’m on the fence, Kenneth was -1 I think and TP was +1. Lots of people run pipenv --python x.y with no other commands as essentially an init command anyway.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub, or mute the thread.

kennethreitz42 on 2 Sep 2018

I thought I was remembering that. People just ran the commands above to initialize either way.

techalchemy on 2 Sep 2018

Why would you do that?

AWS Lambda requires a flat folder with all packages installed. I'm running the same command (pipenv lock -r | pipenv run pip install -r /dev/stdin --target src) in production on codepipeline and for local development via the aws-sam-cli.

Locally, I need to prefix pipenv run to pip install so that pip doesn't see my globally installed packages. Otherwise, locally, I get weird issues like:

awscli 1.15.83 has requirement botocore==1.10.82, but you'll have botocore 1.11.6 which is incompatible.

In production, the virtualenv doesn't exist so the weird parallel creation of virtualenv occurs.

Might be too much of an edgecase.

AlJohri on 2 Sep 2018

👍4

here's a slightly more full-featured version of @uranusjr 's workaround for this issue, which I would still love to see the pipenv maintainers reconsider.

def pipenv_lock(lockfile: str):
    """If you rely on `pipenv lock -r to do this for you, it will force-create a venv for you.
     This is a workaround that gives the same result.
    """
    from pipenv.vendor.requirementslib import Requirement
    # this is imported late because it very rudely does a basicConfig on logging
    # which will prevent basicConfig in your own scripts from taking effect
    section = 'default' if lockfile == 'requirements.txt' else 'develop'
    with open('Pipfile.lock') as f:
        with open(lockfile, 'w') as outf:
            data = json.load(f)
            for source in data.get('_meta', {}).get('sources', []):
                url = source['url']
                if source['name'] == 'pypi':
                    outf.write(f'-i {url}\n')
                else:
                    outf.write(f'--extra-index-url {url}\n')
            for name, entry in data[section].items():
                r = Requirement.from_pipfile(name, entry)
                outf.write(r.as_line() + '\n')

petergaultney on 29 Aug 2019

👍5

+1 to this and the solution above. I have a use case where the production build environment doesn't expose credentials needed for some package sources, so requirements file generation is a must. Just running lock/creating the venv fails due to this limitation.

@petergaultney's workaround above is great, we also just need to add BAD_PACKAGE support so things like setuptools/six don't get caught in --require-hashes mode.