Pipenv: RuntimeError: maximum recursion depth exceeded

[philfreo]

Trying to convert my production app, I get a recursion error:

$ cat requirements.txt | wc -l
     201

$ pipenv lock   
Requirements.txt found, instead of Pipfile! Converting…
Traceback (most recent call last):
  File "/Users/philfreo/Library/Python/2.7/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/cli.py", line 1787, in lock
    ensure_project(three=three, python=python)
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/cli.py", line 577, in ensure_project
    ensure_pipfile(validate=validate)
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/cli.py", line 291, in ensure_pipfile
    import_requirements()
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/cli.py", line 251, in import_requirements
    project.add_package_to_pipfile(str(package.req), dev=dev)
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/project.py", line 413, in add_package_to_pipfile
    p = self._pipfile
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/project.py", line 212, in _pipfile
    pfile = self.parsed_pipfile
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/project.py", line 207, in parsed_pipfile
    return contoml.loads(contents)
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/patched/contoml/__init__.py", line 15, in loads
    elements = parse_tokens(tokens)
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/patched/prettytoml/parser/__init__.py", line 17, in parse_tokens
    return _parse_token_stream(TokenStream(tokens))
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/patched/prettytoml/parser/__init__.py", line 29, in _parse_token_stream
    elements, pending = toml_file_elements(token_stream)
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 375, in toml_file_elements
    captured = capture_from(token_stream).find(one).or_find(file_entry_element).or_empty()

.......... middle of very large stack trace omitted (lmk if you want the full version) ......

    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 98, in more
    c = capture_from(ts).find(token(token_type)).and_find(zero_or_more_tokens(token_type))
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 58, in factory
    raise ParsingError('Expected a token of type {}'.format(token_type))
  File "/Users/philfreo/Library/Python/2.7/lib/python/site-packages/pipenv/patched/prettytoml/tokens/__init__.py", line 30, in __repr__
    return "{}-{}".format(self.priority, self._name)
RuntimeError: maximum recursion depth exceeded

It makes some good progress before crashing:

$ cat Pipfile | wc -l 
     193

[kennethreitz42]

hmm, i'm not sure why. You might want to convert the file by hand — it wouldn't take too long. or, break it into smaller chunks and use pipenv install -r smaller-chunk.txt.

[kennethreitz42]

this may just be a limitation of the way our toml library is designed.

[kennethreitz42]

I'd try the smaller chunk approach, break it into 4 files or so.

[philfreo]

thx, will try that.

[philfreo]

I split it up among 4 files. First 3 succeeded. 4th one failed with the same error. I tried splitting that into even smaller files but the same thing happened.

Basically, after having installed a certain number of packages, installing even 1 more package fails with recursion error. Doesn't matter which package.

After having installed a bunch of packages:

wc -l requirements-original.txt Pipfile Pipfile.lock
     201 requirements-original.txt
     193 Pipfile
    1436 Pipfile.lock

Then installing any one more package gives this:

pipenv install python-baseconv==1.2.0
Traceback (most recent call last):
  File "/Users/philfreo/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/cli.py", line 1593, in install
    ensure_project(three=three, python=python, system=system, warn=True)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/cli.py", line 552, in ensure_project
    if project.required_python_version:
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/project.py", line 59, in required_python_version
    required = self.parsed_pipfile.get('requires', {}).get('python_full_version')
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/project.py", line 213, in parsed_pipfile
    return contoml.loads(contents)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/contoml/__init__.py", line 15, in loads
    elements = parse_tokens(tokens)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/__init__.py", line 17, in parse_tokens
    return _parse_token_stream(TokenStream(tokens))
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/__init__.py", line 29, in _parse_token_stream
    elements, pending = toml_file_elements(token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 375, in toml_file_elements
    captured = capture_from(token_stream).find(one).or_find(file_entry_element).or_empty()
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 372, in one
    c1 = capture_from(ts1).find(file_entry_element).and_find(toml_file_elements)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 375, in toml_file_elements
    captured = capture_from(token_stream).find(one).or_find(file_entry_element).or_empty()
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 372, in one
    c1 = capture_from(ts1).find(file_entry_element).and_find(toml_file_elements)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 375, in toml_file_elements
    captured = capture_from(token_stream).find(one).or_find(file_entry_element).or_empty()
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 372, in one
    c1 = capture_from(ts1).find(file_entry_element).and_find(toml_file_elements)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 375, in toml_file_elements
    captured = capture_from(token_stream).find(one).or_find(file_entry_element).or_empty()
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 372, in one
    c1 = capture_from(ts1).find(file_entry_element).and_find(toml_file_elements)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 375, in toml_file_elements
    captured = capture_from(token_stream).find(one).or_find(file_entry_element).or_empty()
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 372, in one
    c1 = capture_from(ts1).find(file_entry_element).and_find(toml_file_elements)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 375, in toml_file_elements
    captured = capture_from(token_stream).find(one).or_find(file_entry_element).or_empty()
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 372, in one
    c1 = capture_from(ts1).find(file_entry_element).and_find(toml_file_elements)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 365, in file_entry_element
    or_find(table_body_element)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 79, in or_find
    return Capturer(self._token_stream).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 349, in table_body_element
    captured = capture_from(token_stream).find(table_body_elements)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 341, in table_body_elements
    or_find(two).\
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 79, in or_find
    return Capturer(self._token_stream).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 336, in two
    and_find(table_body_elements)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 340, in table_body_elements
    find(one).\

...... repeating pattern .....

  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 79, in or_find
    return Capturer(self._token_stream).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 335, in two
    find(empty_line_elements).\
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 359, in empty_line_elements
    captured = capture_from(token_stream).find(empty_line_tokens)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 354, in empty_line_tokens
    c1 = capture_from(ts1).find(space_element).and_find(line_terminator_element)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 115, in space_element
    captured = capture_from(token_stream).find(zero_or_more_tokens(tokens.TYPE_WHITESPACE))
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 108, in factory
    captured = capture_from(token_stream).find(more).or_find(two).or_find(zero)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 98, in more
    c = capture_from(ts).find(token(token_type)).and_find(zero_or_more_tokens(token_type))
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 113, in capture_from
    return Capturer(token_stream)
RuntimeError: maximum recursion depth exceeded

[fruch]

just out of curiosity, what are you building there with 201 dependencies? (only seen such things in node.js till now ;))

[philfreo]

It’s https://close.io. We don’t have 201 direct dependences, but we were using pip’s --no-deps option to get more of a guarantee that subdependency versions wouldn’t randomly change on us and break something unexpectedly in production.

After porting to pipenv, I expect to be able to remove the subdependencies from Pipfile due to the safety provided by Pipfile.lock. However I want to do that as a subsequent step in order to avoid a bunch of subdependency version bumps that would be too large of a change all at once. And my assumption is the recursion error here is based on Pipfile.lock’s size which wouldn’t shrink anyway.

[kennethreitz42]

I'll see about falling back to the second parser today. Having your pipfile handy would be great.

[kennethreitz42]

okay v7.8.2 released, which falls back to another parser if this one fails.

[kennethreitz42]

This should (hopefully) fix your issue

[kennethreitz42]

to be clear, we're using contoml in attempt to preserve comments in the pipfile.

[kennethreitz42]

it appears to choke on large files though, so i made the code fallback to 'toml' parser when that one fails.

[kennethreitz42]

Closing, as i believe this is now fixed.

[kennethreitz42]

Confirmed working with a 196 line toml file. that wasn't easy to make.

[kennethreitz42]

FYI, pipenv automatically uses --no-deps, so you might be able to prune this file a bit.

[philfreo]

fyi I tried again on 7.8.3 and I get the same error when trying to install 1 more package.

wc -l Pipfile
     276 Pipfile

I'm going to try to prune Pipfile of subdependencies.

[kennethreitz42]

can you show the exception?

[kennethreitz42]

that's an outrageously large Pipfile

[kennethreitz42]

maybe our other parser can't handle it either

[philfreo]

Yeah, I'm going to work on making it smaller.

$ pipenv install zbase32==1.1.5
Traceback (most recent call last):
  File "/Users/philfreo/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/cli.py", line 1614, in install
    ensure_project(three=three, python=python, system=system, warn=True)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/cli.py", line 570, in ensure_project
    if project.required_python_version:
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/project.py", line 59, in required_python_version
    required = self.parsed_pipfile.get('requires', {}).get('python_full_version')
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/project.py", line 211, in parsed_pipfile
    return contoml.loads(toml.dumps(data, preserve=True))
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/contoml/__init__.py", line 15, in loads
    elements = parse_tokens(tokens)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/__init__.py", line 17, in parse_tokens
    return _parse_token_stream(TokenStream(tokens))
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/__init__.py", line 29, in _parse_token_stream
    elements, pending = toml_file_elements(token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 375, in toml_file_elements
    captured = capture_from(token_stream).find(one).or_find(file_entry_element).or_empty()
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 372, in one
    c1 = capture_from(ts1).find(file_entry_element).and_find(toml_file_elements)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 375, in toml_file_elements
    captured = capture_from(token_stream).find(one).or_find(file_entry_element).or_empty()
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 372, in one
    c1 = capture_from(ts1).find(file_entry_element).and_find(toml_file_elements)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 375, in toml_file_elements
    captured = capture_from(token_stream).find(one).or_find(file_entry_element).or_empty()
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 372, in one
    c1 = capture_from(ts1).find(file_entry_element).and_find(toml_file_elements)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)

........... more .........


  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 340, in table_body_elements
    find(one).\
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 330, in one
    and_find(table_body_elements)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 340, in table_body_elements
    find(one).\
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 329, in one
    find(key_value_pair).\
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 317, in key_value_pair
    and_find(value).\
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 202, in value
    or_find(inline_table_element)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 79, in or_find
    return Capturer(self._token_stream).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 303, in inline_table_element
    and_find(internal).\
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 297, in internal
    c = capture_from(ts).find(one).or_find(key_value).or_empty()
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 294, in one
    and_find(internal)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 297, in internal
    c = capture_from(ts).find(one).or_find(key_value).or_empty()
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 294, in one
    and_find(internal)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 297, in internal
    c = capture_from(ts).find(one).or_find(key_value).or_empty()
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 290, in one
    find(key_value).\
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 281, in key_value
    and_find(space_element).\
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 115, in space_element
    captured = capture_from(token_stream).find(zero_or_more_tokens(tokens.TYPE_WHITESPACE))
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 108, in factory
    captured = capture_from(token_stream).find(more).or_find(two).or_find(zero)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 98, in more
    c = capture_from(ts).find(token(token_type)).and_find(zero_or_more_tokens(token_type))
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 109, in and_find
    return Capturer(self.pending_tokens, self.value()).find(finder)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 108, in factory
    captured = capture_from(token_stream).find(more).or_find(two).or_find(zero)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 98, in more
    c = capture_from(ts).find(token(token_type)).and_find(zero_or_more_tokens(token_type))
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/recdesc.py", line 33, in find
    element, pending_ts = finder(self._token_stream)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/patched/prettytoml/parser/parser.py", line 58, in factory
    raise ParsingError('Expected a token of type {}'.format(token_type))
RuntimeError: maximum recursion depth exceeded

[kennethreitz42]

weird, it should be using the other parser. let me take a look.

[kennethreitz42]

try v7.8.5

[philfreo]

Trying.

Sidenote: When I run pipenv install -r requirements-original.txt and then it says "Discovering imports from local codebase…" I get nervous that it's doing something _other_ than just installing what's in the .txt file.

[kennethreitz42]

it shouldn't be doing that unless -r isn't given..

[kennethreitz42]

fixed in v7.8.6!

[philfreo]

On 7.8.5:

$ pipenv install -r requirements-original.txt
...
Installing dependencies from Pipfile.lock…
Traceback (most recent call last):▉▉▉▉ 31/201 — 00:00:17
  File "/Users/philfreo/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/cli.py", line 1663, in install
    do_init(dev=dev, allow_global=system, ignore_pipfile=ignore_pipfile, system=system, skip_lock=skip_lock, verbose=verbose, concurrent=concurrent)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/cli.py", line 1230, in do_init
    skip_lock=skip_lock, verbose=verbose, concurrent=concurrent)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/cli.py", line 788, in do_install_dependencies
    index=index
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/cli.py", line 1303, in pip_install
    c = delegator.run(pip_command, block=block)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/delegator.py", line 257, in run
    c.run(block=block, binary=binary)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/delegator.py", line 149, in run
    s = PopenSpawn(self._popen_args, **pexpect_kwargs)
  File "/Users/philfreo/.local/venvs/pipenv/lib/python2.7/site-packages/pipenv/vendor/pexpect/popen_spawn.py", line 45, in __init__
    self.proc = subprocess.Popen(cmd, **kwargs)
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1227, in _execute_child
    errpipe_read, errpipe_write = self.pipe_cloexec()
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1179, in pipe_cloexec
    r, w = os.pipe()
OSError: [Errno 24] Too many open files

I also tried it with --sequential and that didn't change things.

(I'm still going to try pruning to a smaller file, haven't yet)

[kennethreitz42]

that is... weird.

[kennethreitz42]

can you please privately share your pipfile with me?

[kennethreitz42]

this is a goldmine of edgecases

[kennethreitz42]

[email protected]

[kennethreitz42]

we confirmed we fixed the toml bug though, yay!

[fruch]

looks like you guys having fun :)

[kennethreitz42]

$ pipenv install --skip-lock works for me, so far...

[kennethreitz42]

[kennethreitz42]

I'm running MacOS. What OS are you on?

[philfreo]

Same

[kennethreitz42]

it might be related to git... i'm surprised --sequential didn't fix your problem.

[kennethreitz42]

Yeah, succeeded over here (aside from lxml failing to build)

[philfreo]

At this point I'm more excited to prune the file first anyway. Of course there's bugs with that too :)

So I'm working through reviewing it manually.

[kennethreitz42]

Did you try pipenv check --unused?

[philfreo]

Not yet. It's just a wrapper around pipreqs, right? (And would have the same bug). I figured I'd start there with just my requirements file, for simpler changes first.

[kennethreitz42]

yeah it's a wrapper around pip reqs. will probably give you a lot of false positives, but it'd be a place to start.

[kennethreitz42]

Cannot reproduce, even with 4+ git deps.

[philfreo]

I ran git gc, pipenv --rm, pruned a bunch of dependencies from my requirements.txt file, and ran rm Pipfile Pipfile.lock. Then I ran pipenv install. It gave me OSError: [Errno 24] Too many open files again.

However I then ran pipenv install --sequential and this time it worked! Not really sure what the specific trick was compared to before, but cool to see it working!!

[techalchemy]

FYI if you run into this error again you can check the per-process file handle limit with ulimit -n and then determine how many file handles pipenv is actually using as follows (for example, with fish as your shell):

for pid in ( ps aux | grep -i pipenv | awk '{ print $2 }')
    lsof -p $pid
end | wc -l

You can run without wc -l for the actual handles if you want to get a better idea of what is happening. I am guessing it is a circular dependency resolution relating to concurrent downloads which will basically create infinite file handles until it hits the ulimit cap. Essentially that would mean ulimit is doing its job.

It would be hard to troubleshoot something like this, but I'd be curious to see the output of that if it happens again or if you can reproduce.

[wy193777]

I run into this problem on one of my Windows machine. It works on Ubuntu and another Windows. On the windows has the problem I run pipenv with py -m pipenv install. On Ubuntu I simply use pipenv install.