Dvc: On `dvc repro`: NetworkXError: nbunch is not a node or a sequence of nodes

[courentin]

Hello dvc team!

I encountered an error during dvc repro just after freezing some stages.

Here is the trace:

2020-11-12 19:17:45,465 DEBUG: No lock entry found for 'bash_projects/TDNNF_child_EN/dvc.yaml:create_hires_feats_train'
2020-11-12 19:17:45,974 DEBUG: fetched: [(412119,)]
2020-11-12 19:17:45,978 ERROR: unexpected error - nbunch is not a node or a sequence of nodes.
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/.local/share/virtualenvs/speech-api-EI_ft4iY/lib/python3.7/site-packages/networkx/classes/graph.py", line 1874, in bunch_iter
    for n in nlist:
TypeError: 'PipelineStage' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.local/share/virtualenvs/speech-api-EI_ft4iY/lib/python3.7/site-packages/dvc/main.py", line 76, in main
    ret = cmd.run()
  File "/home/ubuntu/.local/share/virtualenvs/speech-api-EI_ft4iY/lib/python3.7/site-packages/dvc/command/repro.py", line 29, in run
    stages = self.repo.reproduce(target, **self._repro_kwargs)
  File "/home/ubuntu/.local/share/virtualenvs/speech-api-EI_ft4iY/lib/python3.7/site-packages/dvc/repo/__init__.py", line 54, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/ubuntu/.local/share/virtualenvs/speech-api-EI_ft4iY/lib/python3.7/site-packages/dvc/repo/scm_context.py", line 4, in run
    result = method(repo, *args, **kw)
  File "/home/ubuntu/.local/share/virtualenvs/speech-api-EI_ft4iY/lib/python3.7/site-packages/dvc/repo/reproduce.py", line 89, in reproduce
    active_graph = _get_active_graph(self.graph)
  File "/home/ubuntu/.local/share/virtualenvs/speech-api-EI_ft4iY/lib/python3.7/site-packages/dvc/repo/reproduce.py", line 58, in _get_active_graph
    if not active.in_degree(node):
  File "/home/ubuntu/.local/share/virtualenvs/speech-api-EI_ft4iY/lib/python3.7/site-packages/networkx/classes/reportviews.py", line 354, in __call__
    return self.__class__(self._graph, nbunch, weight)
  File "/home/ubuntu/.local/share/virtualenvs/speech-api-EI_ft4iY/lib/python3.7/site-packages/networkx/classes/reportviews.py", line 339, in __init__
    else list(G.nbunch_iter(nbunch))
  File "/home/ubuntu/.local/share/virtualenvs/speech-api-EI_ft4iY/lib/python3.7/site-packages/networkx/classes/graph.py", line 1882, in bunch_iter
    raise NetworkXError(msg)
networkx.exception.NetworkXError: nbunch is not a node or a sequence of nodes.
------------------------------------------------------------
2020-11-12 19:17:46,119 DEBUG: Version info for developers:
DVC version: 1.9.1 (pip)
---------------------------------
Platform: Python 3.7.3 on Linux-5.4.0-1029-aws-x86_64-with-debian-buster-sid
Supports: http, https, s3
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p1
Workspace directory: ext4 on /dev/nvme0n1p1
Repo: dvc, git

Unfreezing the stages makes the repro work again.

Here is what's in my Pipfile.lock:

        "networkx": {
            "hashes": [
                "sha256:cdfbf698749a5014bf2ed9db4a07a5295df1d3a53bf80bf3cbd61edf9df05fa1",
                "sha256:f8f4ff0b6f96e4f9b16af6b84622597b5334bf9cae8cf9b2e42e7985d5c95c64"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==2.4"
        },

Do you have any ideas?
Tell me if you need more info

Thank you for your work!

[pared]

@courentin would it be possible to share the dag structure and information about which stages have been frozen with us?

[skshetry]

@courentin, ping. Does that still happen? Could you please provide more information as @pared requested? Thanks.

[courentin]

Yes sorry, I'll do it through the dvc chat!

[pared]

@courentin @efiop prepared a fix, that might fix your issue. It can be installed from master or from next release (most likely 1.11.8)

[efiop]

@courentin 1.11.8 is out with a fix, could you upgrade and give it a try? Closing for now.

[morganednl]

Hello,
I just had the same issue, I confirm 1.11.8 fixed the issue, thank you!