Drake: Python-based Unit Tests Continue to Run Despite C++ Build Error

[liangfok]

When a C++ build fails, the Python-based unit tests are still being run. This results in the confusing GUI shown below.

When we click on a unit test, we see:

ImportError: No module named pydrake

To get the actual error, we need to check the full console log, which in this case is:

00:11:50 /home/ubuntu/workspace/linux-clang-continuous-debug/drake/examples/schunk_wsg/schunk_wsg_simulation.cc:73:45: error: no member named 'kContinuousSampling' in namespace 'drake::systems'
00:11:50     this->DeclareAbstractInputPort(systems::kContinuousSampling);
00:11:50                                    ~~~~~~~~~^

Ideally, to prevent confusion, the python-based unit tests should not run when the C++ build process fails.

[david-german-tri]

@liangfok could you provide a link to the instance in your screenshot?

I agree this is annoying, but I don't have a sense of how hard it would be to resolve. Could some Kitware folks weigh in?

[liangfok]

@david-german-tri: Yes, here are some recent examples:

• https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-clang-continuous-debug/1336/
• https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-clang-continuous-debug/1335/
• https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-clang-continuous-open-source-debug/286/
• https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-clang-continuous-open-source-debug/285/

[mwoehlke-kitware]

...why doesn't the build error show up as an error?

[RussTedrake]

agreed. the non-reporting of the build errors are the bigger problem. i'm going to bump the priority.

[david-german-tri]

Wait, so Drake build errors don't show up in CDash at all anymore? I agree with priority: high in that case!

[mwoehlke-kitware]

That would be horrible... but it seems only a few builds are affected. I can't reproduce so far, and we've obviously had failing builds recently... for example, linux-gcc-ninja-continuous-release/1053 seems to have the exact same error, and was built at almost the same time and probably the same SHA, but it caught the error as expected...

[liangfok]

Yeah, the problem appears to be transient, though I'm noticing it happen more frequently recently. I'll update this issue as I encounter it.

[mwoehlke-kitware]

Please note also if you see it happening on builds that are not linux+make+clang+debug...

This is really screwy; make clearly failed, but somehow the failure disappeared between when make bailed out of a few levels of build tree and the ctest_build(...) that invoked make in the first place.

[liangfok]

Again: https://drake-jenkins.csail.mit.edu/job/linux-clang-experimental-debug/1394/

[mwoehlke-kitware]

So... it turns out a bug managed to sneak into CMake master that broke launchers with the make generator. This caused ctest/cdash to be unable to recognize errors when building C++ sources. This affected the bleeding-edge CMake we were using on the CI machines (the bug is not present in any released version). It's been fixed now and the AMI's updated.