Pipenv: Can't lock package with `git` and `extras`

Created on 3 Jul 2018  路  11Comments  路  Source: pypa/pipenv
Issue description

A package spec with git and extras fails to lock. This happens regardless of the editable flag.

Expected result

The package should be installed with all extras.

Actual result
[I] ~/test $ pipenv --version
pipenv, version 2018.7.1
[I] ~/test $ ll
total 8
-rw-r--r--  1 jacobhayes    96B Jul  3 13:12 Pipfile
[I] ~/test $ cat Pipfile
[packages]
requests = {git = "https://github.com/requests/requests.git", extras = ["security"]}
[I] ~/test $ pipenv --rm ; pipenv install --verbose
Removing virtualenv (/Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt)...
Creating a virtualenv for this project...
Pipfile: /Users/jacobhayes/test/Pipfile
Using /usr/local/opt/python/bin/python3.6 (3.6.5) to create virtualenv...
鉅婣lready using interpreter /usr/local/opt/python/bin/python3.6
Using base prefix '/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt/bin/python3.6
Also creating executable in /Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt/bin/python
Installing setuptools, pip, wheel...done.
Setting project for test-qFl43Lkt to /Users/jacobhayes/test

Virtualenv location: /Users/jacobhayes/.local/share/virtualenvs/test-qFl43Lkt
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 435, in install
    selective_upgrade=selective_upgrade,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1943, in do_install
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1312, in do_init
    pypi_mirror=pypi_mirror,
  File "/usr/local/lib/python3.6/site-packages/pipenv/core.py", line 1140, in do_lock
    project.write_lockfile(lockfile)
  File "/usr/local/lib/python3.6/site-packages/pipenv/project.py", line 648, in write_lockfile
    content, indent=4, separators=(u',', u': '), sort_keys=True,
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 201, in encode
    chunks = list(chunks)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 430, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 404, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 437, in _iterencode
    o = _default(o)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'ArrayElement' is not JSON serializable
Steps to replicate
[packages]
requests = {git = "https://github.com/requests/requests.git", extras = ["security"]}

pipenv install

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: '/usr/local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/local/opt/python/bin/python3.6'

Other Python installations in PATH:

  • 2.7: /usr/local/bin/python2.7
  • 2.7: /usr/local/bin/python2.7
  • 2.7: /usr/bin/python2.7
  • 3.6: /usr/local/bin/python3.6m

  • 3.6: /usr/local/bin/python3.6

  • 2.7.15: /usr/local/bin/python

  • 2.7.10: /usr/bin/python
  • 2.7.15: /usr/local/bin/python2
  • 3.6.5: /usr/local/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.5',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.6.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.6.0: Tue May  8 15:22:16 PDT '
                     '2018; root:xnu-4570.61.1~1/RELEASE_X86_64',
 'python_full_version': '3.6.5',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

  • Apple_PubSub_Socket_Render
  • COLORFGBG
  • COLORTERM
  • COPYFILE_DISABLE
  • EDITOR
  • GOPATH
  • HOME
  • ITERM_PROFILE
  • ITERM_SESSION_ID
  • LANG
  • LOGNAME
  • PAGER
  • PATH
  • PWD
  • SECURITYSESSIONID
  • SHELL
  • SHLVL
  • SSH_AUTH_SOCK
  • TERM
  • TERM_PROGRAM
  • TERM_PROGRAM_VERSION
  • TERM_SESSION_ID
  • TMPDIR
  • TMUX
  • TMUX_PANE
  • TMUX_PLUGIN_MANAGER_PATH
  • USER
  • XPC_FLAGS
  • XPC_SERVICE_NAME
  • __CF_USER_TEXT_ENCODING
  • PYTHONDONTWRITEBYTECODE
  • PIP_PYTHON_PATH

Pipenv鈥搒pecific environment variables:

Debug鈥搒pecific environment variables:

  • PATH: /Users/jacobhayes/bin:/Users/jacobhayes/bin:/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/bin:/Users/jacobhayes/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
  • SHELL: /usr/local/bin/fish
  • EDITOR: vim
  • LANG: en_US.UTF-8
  • PWD: /Users/jacobhayes/test

Contents of Pipfile ('/Users/jacobhayes/test/Pipfile'):

[packages]
requests = {git = "https://github.com/requests/requests.git", extras = ["security"]}

Type
Source
picture JacobHayes
👍8

Most helpful comment

This bug is a 3.7+ bug and should only be happening since the combination of 3.7 + 2018.7.1.

This is definitely happening on 3.6.* for me.

picture theunraveler on 10 Jul 2018
👍5

All 11 comments

This works with v2018.05.18, but fails in v2018.6.25 and v2018.7.1

JacobHayes picture JacobHayes on 3 Jul 2018

Interesting! I can definitely replicate this.

techalchemy picture techalchemy on 3 Jul 2018

git bisect tells me that https://github.com/pypa/pipenv/commit/18530dfb46bbe6aef9dc23ce8774ddca6990db65 is the culprit, if that helps.

theunraveler picture theunraveler on 4 Jul 2018

I did some digging today and an easy fix for this is to make the following change in Project.write_lockfile():

    def write_lockfile(self, content):
        """Write out the lockfile.
        """
        newlines = self._lockfile_newlines

        def encode_objects(obj):
            if isinstance(obj, ArrayElement): # prettytoml.elements.array.ArrayElement
                return obj.primitive_value
            return obj

        s = simplejson.dumps(   # Send Unicode in to guarentee Unicode out.
            content, indent=4, separators=(u',', u': '), sort_keys=True,
            default=encode_objects
        )
        with atomic_open_for_write(self.lockfile_location, newline=newlines) as f:
            f.write(s)
            if not s.endswith(u'\n'):
                f.write(u'\n')  # Write newline at end of document. GH #319.
bryanforbes picture bryanforbes on 5 Jul 2018

thanks for digging into this, I know very little about the toml parser -- do you have a sense of what's going on here?

techalchemy picture techalchemy on 6 Jul 2018

When the lockfile is read from the file, it just copies the structure over from the parsed TOML. I'm not sure why, but the TOML structure returns a specialized ArrayElement instead of a Python list (which is why simplejson.dumps() throws an error). Rather than iterate through the TOML coming out of the parser, it looked easier to have simplejson.dumps() convert the ArrayElements to lists.

bryanforbes picture bryanforbes on 6 Jul 2018

Development since v2018.5.18 completely killed our deployment. This bug and #2317 are easily worked around (patch and external variable substitution) but #2552 is probably not. It would be great to have pipenv tested so that these issues don't find their way to the open :)
Otherwise great work, its much appreciated

notEvil picture notEvil on 10 Jul 2018
👍1

This bug is a 3.7+ bug and should only be happening since the combination of 3.7 + 2018.7.1. I took the weekend off of working on this project, but we鈥檇 definitely merge a failing test / fix if we haven鈥檛 already (not totally caught up here)

techalchemy picture techalchemy on 10 Jul 2018

This bug is a 3.7+ bug and should only be happening since the combination of 3.7 + 2018.7.1.

This is definitely happening on 3.6.* for me.

theunraveler picture theunraveler on 10 Jul 2018
👍5

@bryanforbes would you be willing to submit a PR with your changes or mind if I do?

JacobHayes picture JacobHayes on 10 Jul 2018

@JacobHayes you're more than welcome to do the PR. I'm swamped this week.

bryanforbes picture bryanforbes on 11 Jul 2018
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jeyraof picture jeyraof  路  3Comments
jacebrowning picture jacebrowning  路  3Comments
xi picture xi  路  3Comments
ansrivas picture ansrivas  路  3Comments
leileigong picture leileigong  路  3Comments