Poetry: [RecursionError] maximum recursion depth exceeded while calling a Python object

[qiankunxienb]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS Catalina 10.15.6
• Poetry version: 1.1.3
• Link of a Gist with the contents of your pyproject.toml file:

Issue

When I run poetry add scrapy, it raise an error:

[RecursionError]
maximum recursion depth exceeded while calling a Python object

However, when I run poetry shell first to enter the virtual env，and then run poetry add scrapy, then everying works fine.

[abn]

@qiankunxienb I am unable to reproduce the issue see below. If this is is still occurring, can you please add an example pyproject.toml file along with the output of the command when run with -vvv please.

Using a container (podman | docker)

podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -e
pip install -q poetry==1.1.3
poetry new foobar
pushd foobar
sed -i /pytest/d pyproject.toml
poetry add scrapy
EOF

Created package foobar in foobar
/foobar /
Creating virtualenv foobar-lWDpn5M1-py3.8 in /root/.cache/pypoetry/virtualenvs
Using version ^2.4.0 for Scrapy

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 28 installs, 0 updates, 0 removals

  • Installing pycparser (2.20)
  • Installing six (1.15.0)
  • Installing attrs (20.2.0)
  • Installing cffi (1.14.3)
  • Installing cssselect (1.1.0)
  • Installing idna (2.10)
  • Installing lxml (4.5.2)
  • Installing pyasn1 (0.4.8)
  • Installing w3lib (1.22.0)
  • Installing automat (20.2.0)
  • Installing constantly (15.1.0)
  • Installing cryptography (3.1.1)
  • Installing hyperlink (20.0.1)
  • Installing incremental (17.5.0)
  • Installing itemadapter (0.1.1)
  • Installing parsel (1.6.0)
  • Installing pyasn1-modules (0.2.8)
  • Installing zope.interface (5.1.2)
  • Installing pyhamcrest (2.0.2)
  • Installing jmespath (0.10.0)
  • Installing itemloaders (1.0.3)
  • Installing protego (0.1.16)
  • Installing pydispatcher (2.0.5)
  • Installing pyopenssl (19.1.0)
  • Installing queuelib (1.5.0)
  • Installing service-identity (18.1.0)
  • Installing twisted (20.3.0)
  • Installing scrapy (2.4.0)

[qiankunxienb]

Strange, when I add -vvv argument, the issue can not be reproduced.

[seandstewart]

Popping in here to say that I'm getting this consistently when I attempt to export my lock file. Full traceback:

❯ poetry export -f requirements.txt --without-hashes -vvv

  Stack trace:

  992  ~/.poetry/lib/poetry/_vendor/py3.7/clikit/console_application.py:131 in run
        129│             parsed_args = resolved_command.args
        130│ 
      → 131│             status_code = command.handle(parsed_args, io)
        132│         except KeyboardInterrupt:
        133│             status_code = 1

  991  ~/.poetry/lib/poetry/_vendor/py3.7/clikit/api/command/command.py:120 in handle
        118│     def handle(self, args, io):  # type: (Args, IO) -> int
        119│         try:
      → 120│             status_code = self._do_handle(args, io)
        121│         except KeyboardInterrupt:
        122│             if io.is_debug():

  990  ~/.poetry/lib/poetry/_vendor/py3.7/clikit/api/command/command.py:171 in _do_handle
        169│         handler_method = self._config.handler_method
        170│ 
      → 171│         return getattr(handler, handler_method)(args, io, self)
        172│ 
        173│     def __repr__(self):  # type: () -> str

  989  ~/.poetry/lib/poetry/_vendor/py3.7/cleo/commands/command.py:92 in wrap_handle
         90│         self._command = command
         91│ 
      →  92│         return self.handle()
         93│ 
         94│     def handle(self):  # type: () -> Optional[int]

  988  ~/.poetry/lib/poetry/console/commands/export.py:73 in handle
        71│             dev=self.option("dev"),
        72│             extras=self.option("extras"),
      → 73│             with_credentials=self.option("with-credentials"),
        74│         )
        75│ 

  987  ~/.poetry/lib/poetry/utils/exporter.py:44 in export
         42│             dev=dev,
         43│             extras=extras,
      →  44│             with_credentials=with_credentials,
         45│         )
         46│ 

  986  ~/.poetry/lib/poetry/utils/exporter.py:61 in _export_requirements_txt
         59│ 
         60│         for dependency_package in self._poetry.locker.get_project_dependency_packages(
      →  61│             project_requires=self._poetry.package.all_requires, dev=dev, extras=extras
         62│         ):
         63│             line = ""

  985  ~/.poetry/lib/poetry/packages/locker.py:367 in get_project_dependency_packages
        365│             project_requires=selected,
        366│             locked_packages=repository.packages,
      → 367│             with_nested=True,
        368│         ):
        369│             try:

  984  ~/.poetry/lib/poetry/packages/locker.py:316 in get_project_dependencies
        314│             packages_by_name=packages_by_name,
        315│             project_level_dependencies=project_level_dependencies,
      → 316│             nested_dependencies=dict(),
        317│         )
        318│ 

  ...  Previous frame repeated 970 times

   12  ~/.poetry/lib/poetry/packages/locker.py:272 in __walk_dependency_level
        270│             packages_by_name=packages_by_name,
        271│             project_level_dependencies=project_level_dependencies,
      → 272│             nested_dependencies=nested_dependencies,
        273│         )
        274│ 

   11  ~/.poetry/lib/poetry/packages/locker.py:272 in __walk_dependency_level
        270│             packages_by_name=packages_by_name,
        271│             project_level_dependencies=project_level_dependencies,
      → 272│             nested_dependencies=nested_dependencies,
        273│         )
        274│ 

   10  ~/.poetry/lib/poetry/packages/locker.py:240 in __walk_dependency_level
        238│                 # if this is not done, we can end-up with incorrect nested dependencies
        239│                 marker = requirement.marker
      → 240│                 requirement = locked_package.to_dependency()
        241│                 requirement.marker = requirement.marker.intersect(marker)
        242│             else:

    9  ~/.poetry/lib/poetry/_vendor/py3.7/poetry/core/packages/package.py:361 in to_dependency
        359│ 
        360│         if not self.python_constraint.is_any():
      → 361│             dep.python_versions = self.python_versions
        362│ 
        363│         if self._source_type not in ["directory", "file", "url", "git"]:

    8  ~/.poetry/lib/poetry/_vendor/py3.7/poetry/core/packages/dependency.py:117 in python_versions
        115│                 parse_marker(
        116│                     self._create_nested_marker(
      → 117│                         "python_version", self._python_constraint
        118│                     )
        119│                 )

    7  ~/.poetry/lib/poetry/_vendor/py3.7/poetry/core/version/markers.py:683 in parse_marker
        681│         return AnyMarker()
        682│ 
      → 683│     parsed = _parser.parse(marker)
        684│ 
        685│     markers = _compact_markers(parsed.children)

    6  ~/.poetry/lib/poetry/_vendor/py3.7/poetry/core/_vendor/lark/lark.py:391 in parse
        389│         """
        390│         try:
      → 391│             return self.parser.parse(text, start=start)
        392│         except UnexpectedToken as e:
        393│             if on_error is None:

    5  ~/.poetry/lib/poetry/_vendor/py3.7/poetry/core/_vendor/lark/parser_frontends.py:128 in parse
        126│ 
        127│         token_stream = self.lex(text, lambda: parser_state[0])
      → 128│         return self._parse(token_stream, start, set_parser_state)
        129│ ###}
        130│ 

    4  ~/.poetry/lib/poetry/_vendor/py3.7/poetry/core/_vendor/lark/parser_frontends.py:53 in _parse
         51│                 raise ValueError("Lark initialized with more than 1 possible start rule. Must specify which start rule to parse", start)
         52│             start ,= start
      →  53│         return self.parser.parse(input, start, *args)
         54│ 
         55│ 

    3  ~/.poetry/lib/poetry/_vendor/py3.7/poetry/core/_vendor/lark/parsers/lalr_parser.py:36 in parse
         34│ 
         35│     def parse(self, *args):
      →  36│         return self.parser.parse(*args)
         37│ 
         38│ 

    2  ~/.poetry/lib/poetry/_vendor/py3.7/poetry/core/_vendor/lark/parsers/lalr_parser.py:114 in parse
        112│             _action, arg = get_action(token)
        113│             assert(_action is Reduce)
      → 114│             reduce(arg)
        115│             if state_stack[-1] == end_state:
        116│                 return value_stack[-1]

    1  ~/.poetry/lib/poetry/_vendor/py3.7/poetry/core/_vendor/lark/parsers/lalr_parser.py:78 in reduce
         76│                 s = []
         77│ 
      →  78│             value = self.callbacks[rule](s)
         79│ 
         80│             _action, new_state = states[state_stack[-1]][rule.origin.name]

  RecursionError

  maximum recursion depth exceeded

  at ~/.poetry/lib/poetry/_vendor/py3.7/poetry/core/_vendor/lark/parse_tree_builder.py:127 in __call__
      123│                 else:   # Optimize for left-recursion
      124│                     filtered = children[i].children
      125│             else:
      126│                 filtered.append(children[i])
    → 127│         return self.node_builder(filtered)
      128│ 
      129│ def _should_expand(sym):
      130│     return not sym.is_term and sym.name.startswith('_')
      131│ 

I've confirmed that this issue is _only_ occurring on poetry v1.1.3. Downgrading back to v1.1.2 resolves the RecursionError. I'm running these installations using the officially supported installer, not via pip or anything like that.

Here's a link to the current pyproject.toml: https://github.com/seandstewart/typical/blob/main/pyproject.toml

Also, I should note that I upgraded because I noticed poetry was exporting all of my optional dependencies by default...

[estyrke]

I have the exact same issue as @seandstewart - install works fine, but export does not. Not sure if it is related to the OP or if it is better handled as a new issue?

[abn]

Thank you for the reproducer @seandstewart.

Can you please try the fix at #3237. @estyrke @seandstewart @qiankunxienb

Using pipx

pipx install --suffix=@3237 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/3237/head'

Using a container (podman | docker)

podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -xe
pip install -q git+https://github.com/python-poetry/poetry.git@refs/pull/3237/head
install -d foobar
pushd foobar
curl -sLO https://raw.githubusercontent.com/seandstewart/typical/main/pyproject.toml
poetry lock
poetry export -f requirements.txt --without-hashes
EOF

Note that the lock step will take a a minute or so.

[sergekir]

Same problem here.
poetry@3237 worked fine and it took only a few seconds for 150+ deps 👍
But it added python version restriction on every package.
Is it expected?
In pyproject.toml there is python = "^3.8" and i got:
flake8==3.8.4; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")

[abn]

@sergekir at the moment this comes via propagation from dependency markers so, it does not take into account the pypt python constraint. The per dependency python constraint is expected at the moment since we are simply propagating whats in the lock file. Will see if we can remove that later on.

[estyrke]

@abn, poetry@3237 works great for me!

[RollerKnobster]

Was able to reproduce the maximum recursion depth exceeded on any dependency-resolving action on v1.0.9 but the issue was quickly fixed by updating to v1.1.3. Not sure if this is helpful or not.

Also providing the stacktrace from -vvv.

[seandstewart]

Hey @abn sorry for the late response - confirmed poetry@3237 works for me!