Pipenv: Generate requirements.txt from Pipfile.lock

[Zebradil]

How to generate requirements.txt file from existing Pipfile.lock without locking?
When I run pipenv lock -r it ignores existing Pipfile.lock and does locking process again.

There is a workaround for this:

$ pipenv sync
$ pipenv run pip freeze

In my particular situation I'm building docker image and using requirements.txt in Dockerfile. I'd like to avoid creating virtual environment on the host machine just to be able to create requirements.txt.

[frostming]

Pipenv doesn't provide a way for this, you can seek for other pipfile utility libraries such as pipfile-requirements

[anuj9196]

you can run

pipenv run pip freeze > requirements.txt

[Zebradil]

you can run

pipenv run pip freeze > requirements.txt

That's what I mentioned as a workaround in the first post.

But it works only if you have your pipenv environment synchronized (all packages are installed).
Extracting dependencies directly from Pipfile.lock is more convenient for me:

jq -r '.default
        | to_entries[]
        | .key + .value.version' \
    Pipfile.lock > requirements.txt

Blog post
jq tool

[frostming]

LOL, I have already mentioned that library.

[Zebradil]

I, personally, dislike to have a dedicated library for that. Also, there is a higher chance that a team member already has jq or some other general purpose tool installed.

[anuj9196]

you can even run

 pipenv lock --requirements > requirements.txt

[Zebradil]

It will not work as you expect, because, as I wrote:

When I run pipenv lock -r it ignores existing Pipfile.lock and does locking process again.

In other words, it performs an update, which potentially can destroy a distribution. Imagine, you generate requirements.txt to use it in Dockerfile to build a docker image. Locally your application works, but when you generate requirements.txt using pipenv lock, requirements might get updated to incompatible or just broken versions (hopefully, it's a rare case, though). And you will not know this before running the image. So, you'll need to test the app again after running pipenv lock.

If you don't want to use jq, then better use the approach I proposed in the first post with pipenv sync (which doesn't do update).

[ye]

@Zebradil your jq oneliner approach is much simpler than @frostming 's own pipfile-requirements package (100+ lines python code) since I've already installed jq, no other dependencies required, which is great.

However, after a few git commits I noticed the difference between what pipenv lock --requirements outputs and what jq gleans through Pipfile.lock file and prints out:

• jq output doesn't have -i https://pypi.org/simple as the very first line, as opposed to what pipenv lock --r always insert as the very first line.
• jq output doesn't include the annotation for packages. For example: pipenv lock --r output has this lineappnope==0.1.0 ; sys_platform == 'darwin', but in jq output, it's appnope==0.1.0. Another example is pipenv lock -r generates pexpect==4.7.0 ; sys_platform != 'win32' whereas jq generates pexpect==4.7.0, not sure if this matters or not.
• the requirements file package ordering is different between the two outputs. jq which presumably takes the package order in Pipfile.lock file, which always sorts by ascending order in alphabets and character length, for example, flask is in front of flask-sqlalchemy or any flask-XXXXX packages, whereas pipenv lock --r outputs flask behind flask-sqlalchemy, which is different from the order in the Pipfile.lock. This is a major annoyance because it doesn't generate a minimum git diff. I would consider this is a bug in pipenv.

[Zebradil]

Hi @ye, nice comparison of the methods. It might help people choosing proper solution for their particular situation and avoid caveats.

Yes, as you said, the proposed approach with jq has limited functionality. It is possible to extend it to add annotations and package index URL, but I have no need for this right now.

To avoid having differences in generated requirements.txt one should consider using the same approach every time. In a similar way, using different code formatting tools might lead to inconsistent results. So, I don't see a problem here.

[rcastill]

you can run

pipenv run pip freeze > requirements.txt

That's what I mentioned as a workaround in the first post.

But it works only if you have your pipenv environment synchronized (all packages are installed).
Extracting dependencies directly from Pipfile.lock is more convenient for me:

jq -r '.default
        | to_entries[]
        | .key + .value.version' \
    Pipfile.lock > requirements.txt

Blog post
jq tool

Hi,

Thanks for your solution. I ran into the same problem but I also need the definition of sources created by pipenv lock -r, i.e.: -i, --extra-index-url. This is because I work with private sources.

@Zebradil I think you mentioned that.

So I created yet another minimal no-dependencies script in python that includes that functionality. It also expands env vars in case that you have your sources defined that way in your Pipfile.

If anybody wants to take a look I'll leave it here: https://gist.github.com/rcastill/dab85c234dd10fa7af56755116c75aee

[dms-cat]

In case it helps anyone else, here's how to include the hashes in the results:

 jq --raw-output '.default | to_entries[] | .key + .value.version + (.value.hashes | map(" --hash=\(.)") | join(""))' Pipfile.lock

This creates entries like

paramiko==2.6.0 --hash=sha256:99f0179bdc176281d21961a003ffdb2ec369daac1a1007241f53374e376576cf --hash=sha256:f4b2edfa0d226b70bd4ca31ea7e389325990283da23465d572ed1f70a7583041

Which makes pip enforce the hashes.

If you want to include only the requirements which were in the original requirements file (provided they were already locked to a specific version with ==):

 jq --raw-output '.default | to_entries[] | .key + .value.version + (.value.hashes | map(" --hash=\(.)") | join(""))' Pipfile.lock | grep --file=<(grep --only-matching --perl-regexp '^.*(?===)' requirements.txt | tr '[:upper:]' '[:lower:]') > new.txt && mv new.txt requirements.txt

tr is necessary because requirements.txt files may contain mixed case package names but pipenv install -r requirements.txt lowercases them in Pipfile.

[BeyondEvil]

Here's a small python script in case you want to turn the Pipfile (not the lock-file) into a requirements.txt file.

import configparser


def main():
    parser = configparser.ConfigParser()
    parser.read("Pipfile")

    packages = "packages"
    with open("requirements.txt", "w") as f:
        for key in parser[packages]:
            value = parser[packages][key]
            f.write(key + value.replace("\"", "") + "\n")


if __name__ == "__main__":
    main()

[linusguan]

@frostming Hi, I found https://github.com/frostming/pipfile-requirements useful but why it didn't get integrated into pipenv?

[frostming]

@linusguan The tool exists for those who don't want to install the big pipenv library, when you have installed pipenv, you can use pipenv lock -r

[linusguan]

@frostming I find it quite useful for use with other tools that do not support pipfile.lock.
The problem with pipenv lock -r is it updates pipfile.lock so I cannot use it to produce deterministic build along with other tools. Something like pipenv lock -r --ignore-pipfile would be ideal.

[sergeyt]

Here is yet another python script to generate requirements.txt from Pipfile.lock file with hashes:

import os
import json

__dir__ = os.path.dirname(os.path.realpath(__file__))


def read_json_file(path):
    with open(path) as f:
        return json.load(f)


def main():
    root = read_json_file(os.path.join(__dir__, 'Pipfile.lock'))

    for name, pkg in root["default"].items():
        version = pkg["version"]
        sep = lambda i: "" if i == len(pkg["hashes"]) - 1 else " \\"
        hashes = [f'--hash={t}{sep(i)}' for i, t in enumerate(pkg["hashes"])]
        tail = '' if len(hashes) == 0 else f' {hashes[0]}'
        print(f'{name} {version}{tail}')
        for h in hashes[1:]:
            print(f'    {h}')


if __name__ == "__main__":
    main()

[harshit98]

@Zebradil Thanks! Your solution really worked for me.

• I first install jq tool using brew install jq
• Then used your script to generate requirements.txt from Pipfile.lock

[jacobisaliveandwell]

It looks like this can be solved with the --keep-outdated flag, or am I mistaken?

pipenv lock --keep-outdated -d -r > requirements.txt

P.S. that's an annoyingly verbose flag to solve this

[paytonrules]

Unfortunately @jacobisaliveandwell the --keep-outdated flag appears to update subdependencies: https://github.com/pypa/pipenv/issues/3975

[jacobisaliveandwell]

@paytonrules That's a bug, but the spirit of the flag is still the answer to this issue.

P.S. No need to thumbs down for that :-(

[Darkless012]

Just want to mention that from stated workarounds, there are differences:
pipenv run pip freeze returns case-sensitive package names (eg. PyYAML)
pipenv lock --requirements returns all lower-case package names (eg. pyyaml)

[jacobisaliveandwell]

@Darkless012 you should open another ticket describing that, and reference this issue as related.

[earthgecko]

Pure bash, just packages, nothing else, in case someone cannot or does not want to install jq, just in case it helps someone,

cat Pipfile.lock \
  | grep -B1 '"hashes"\|"version": ' \
  | grep -v '"markers": \|"hashes": ' \
  | grep ": {\|version" \
  | sed -e 's/: {$//g' \
  | tr '\n' ',' | tr -s ' ' ' ' \
  | sed -e 's/, "version": "//g;s/", "/ /g;s/"//g;s/,//g' \
  | tr ' ' '\n' \
| grep -v "^$" > requirements.txt

[westurner]

Is there anything that also copies the hashes from the Pipfile to requirements.txt (e.g. given something like a platform str) so that pip install --require-hashes works?

• https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode
• https://pip.pypa.io/en/stable/reference/pip_hash/

$ pip install --help
# ...
  --require-hashes            Require a hash to check each requirement against, for repeatable installs. This option is implied when any package in a
                              requirements file has a --hash option.

[fridex]

You can use micropipenv that can convert Pipenv.lock (also poetry.lock) files to requirements.txt (raw requirements.txt or pip-tools compatible). See https://github.com/thoth-station/micropipenv/#micropipenv-requirements--micropipenv-req