Conan: AssertionError (non-deterministic?) in graph evaluation

[himikof]

I have just seen the following error in our CI:

$ conan install -u -r artifactory --build=outdated ./packaging -s python=cpython -s python.version=3.6
Version range '^2.1' required by 'fbthrift/2018.05.21.00@igs/testing' resolved to 'gflags/2.2.0@igs/testing'
Version range '^2.1' required by 'folly/2018.05.21.00@igs/testing' resolved to 'gflags/2.2.0@igs/testing'
Version range '^0.3.5' required by 'folly/2018.05.21.00@igs/testing' resolved to 'glog/0.3.5@igs/testing'
Version range '^1.1' required by 'folly/2018.05.21.00@igs/testing' resolved to 'double-conversion/1.1.5@igs/testing'
glog/0.3.5@igs/testing requirement gflags/[^2.2]@igs/testing overriden by folly/2018.05.21.00@igs/testing to gflags/2.2.0@igs/testing 
Version range '^2.2' required by 'glog/0.3.5@igs/testing' valid for downstream requirement 'gflags/2.2.0@igs/testing'
Version range '^2.1' required by 'wangle/2018.05.21.00@igs/testing' resolved to 'gflags/2.2.0@igs/testing'
wangle/2018.05.21.00@igs/testing: WARN: Can't update, no package in remote
wangle/2018.05.21.00@igs/testing: Package is up to date
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/conans/model/ref.py", line 70, in loads
    name, version, user, channel = tokens
ValueError: not enough values to unpack (expected 4, got 2)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/conans/client/command.py", line 314, in install
    reference = ConanFileReference.loads(args.path_or_reference)
  File "/usr/local/lib/python3.6/dist-packages/conans/model/ref.py", line 73, in loads
    "OpenCV/1.0.6@user/stable" % text)
conans.errors.ConanException: Wrong package recipe reference ./packaging
Write something like OpenCV/1.0.6@user/stable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/conans/client/command.py", line 1219, in run
    method(args[0][1:])
  File "/usr/local/lib/python3.6/dist-packages/conans/client/command.py", line 325, in install
    install_folder=args.install_folder)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/conan_api.py", line 79, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/conan_api.py", line 482, in install
    no_imports=no_imports)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/manager.py", line 258, in install
    profile.build_requires)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/graph/graph_manager.py", line 108, in load_graph
    binaries_analyzer.evaluate_graph(graph, build_mode, update, remote_name)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/graph/graph_binaries.py", line 138, in evaluate_graph
    self._evaluate_node(node, build_mode, update, evaluated_references, remote_name)
  File "/usr/local/lib/python3.6/dist-packages/conans/client/graph/graph_binaries.py", line 47, in _evaluate_node
    assert node.binary is None
AssertionError

Unfortunately, this error did not reproduce after retrying exactly the same build (with the same cache state), everything was OK.
Likely relevant info about our CI setup: the entire .conan directory is cached between jobs, but it is only stored at the end of a successful job. So, while the .conan contents were quite possibly written by an older conan version, they were exactly the same for the unsuccessful and successful tries.

Environment info: Conan 1.5.1, Ubuntu 18.04 x86_64

[zscgeek]

I am getting the same AssertionError.

I can see it on conan info . and other commands.

Here is the backtrace on my system. I can provide more details if needed. I am also running Conan 1.5.1, Ubuntu 18.04 x86_64.

```Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/conans/client/command.py", line 1219, in run
method(args[0][1:])
File "/usr/local/lib/python3.6/dist-packages/conans/client/command.py", line 472, in info
build=args.dry_build)
File "/usr/local/lib/python3.6/dist-packages/conans/client/conan_api.py", line 79, in wrapper
return f(args, *kwargs)
File "/usr/local/lib/python3.6/dist-packages/conans/client/conan_api.py", line 562, in info
check_updates=update, build_mode=build)
File "/usr/local/lib/python3.6/dist-packages/conans/client/manager.py", line 193, in info_get_graph
build_mode=build_mode, remote_name=remote_name)
File "/usr/local/lib/python3.6/dist-packages/conans/client/manager.py", line 156, in _get_deps_graph
profile_build_requires=profile.build_requires)
File "/usr/local/lib/python3.6/dist-packages/conans/client/graph/graph_manager.py", line 111, in load_graph
profile_build_requires)
File "/usr/local/lib/python3.6/dist-packages/conans/client/graph/graph_manager.py", line 88, in _recurse_build_requires
remote_name, profile_build_requires)
File "/usr/local/lib/python3.6/dist-packages/conans/client/graph/graph_manager.py", line 108, in load_graph
binaries_analyzer.evaluate_graph(graph, build_mode, update, remote_name)
File "/usr/local/lib/python3.6/dist-packages/conans/client/graph/graph_binaries.py", line 138, in evaluate_graph
self._evaluate_node(node, build_mode, update, evaluated_references, remote_name)
File "/usr/local/lib/python3.6/dist-packages/conans/client/graph/graph_binaries.py", line 47, in _evaluate_node
assert node.binary is None
AssertionError

ERROR:
insectoidsombrero@e82a07b10921:/host/SIOInstaller$
```

[zscgeek]

Note that I also get the error on my Ubuntu 16.04 test box.

[memsharded]

Thanks very much for your reports. Indeed this seems a bug, will try to fix as soon as possible and release it.

[memsharded]

I have reviewed the code, and haven't been able to find the flaw so far. Everything seems correct, the assignment of value to the node.binary is done only after the assert, so in theory the assert seems correct.

Maybe if @zscgeek could provide more details, that would help (as @himikof error seems more random). If you don't want to share details publicly, don't hesitate to email recipes, or any necessary data to [email protected]. Many thanks!

[zscgeek]

@memsharded - Let me know what info would be helpful and I can try to package it up.

It happens with a pretty complex set of dependancies so it's not easy to just package everything up, but I would be happy to run some traces or send some selected stuff over.

[memsharded]

@zscgeek If I add some prints to trace execution in one of my branches, would be able to run your project from my source branch to capture them? That would be very helpful.

[zscgeek]

Sure! No problem
On Fri, Jul 6, 2018 at 06:44 James [e-mail] wrote:

@zscgeek https://github.com/zscgeek If I add some prints to trace
execution in one of my branches, would be able to run your project from my
source branch to capture them? That would be very helpful.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/conan-io/conan/issues/3166#issuecomment-402998908,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADMX23ID2uF4RJkTBNJnd1enYyIojwLks5uDz-YgaJpZM4VBAo2
.

[memsharded]

Great!

This is my branch: https://github.com/memsharded/conan/tree/feature/investigate_assertion_binary
Fork: https://github.com/memsharded/conan, branch feature/investigate_assertion_binary

It is just a few prints, but I hope they can bring some light to this issue. Please copy or send me the whole output of running it. Many thanks @zscgeek !!

[memsharded]

Hi @zscgeek

Any chance of capturing that info? Conan 1.6 is getting closer, and I would like to check this before if possible. Many thanks!

[dkgs]

Hello @memsharded I managed to create a use case causing the problem. I can't run you branch cause I don't have any python dev environment nor a properly internet connected pc (too much restrictions).
Just point the $CONAN_USER_HOME to where you have decompressed the archive and just run a conan info path_to_conanfile.txt

To be lightweight I tried to delete the source dir, the build dir : problem was still there. But when deleting the package dir the AssertionError disappeared.

Problem reproduced on linux x64 & windows x64.

Note : some packages may have been created by a prior version of conan (the 0.28.1 if my memory's good).

Link to the archive of the local repo : https://send.firefox.com/download/5aab8f4400/#6s2rVSghz22CkMVufdip3g
conanfile.txt

[memsharded]

Many thanks @dkgs I have been able to reproduce. Will investigate it as soon as possible.

[memsharded]

It seems that it was a subtle bug when there are several transitive private requirements. Your tgz file resulted to be extremely useful, thanks a million!!!

I have submitted a fix in https://github.com/conan-io/conan/pull/3214. It includes a test that covers the case, but it also works fine with your data. If you want to try running from sources from my branch to test it, that would also be useful.

I think conan 1.6 will be released soon, but if it is urgent, we could try to release a patch version for 1.5.X.
Thanks again for your help.

[dkgs]

Glad to hear that :) No rush on my side, we reverted to 1.4. We are able to wait till the 1.6 is released.
Cheers

[memsharded]

@dkgs Conan 1.6 released! Please try and report if necessary. Thanks!

[dkgs]

@memsharded Updated to 1.6.0 : no problem so far !
Thanks :)

[truezq]

@memsharded Hi,I have got the same error, But I found it's caused by the registry.txt
myconan http://10.78.223.175:8081/artifactory/api/conan/conan-local True ---This conan server is unavailable
Hello/0.1@demo/testing myconan
After remove the two line up, no problem so far.

[memsharded]

Hi @truezq

Can't say if it is the same error, I would say similar but not exactly this one. It could be that you had something dirty there from (quite) older conan versions? In any case, if it is solved cleaning the registry.txt, great, thanks for reporting!