Drake: testURDFmex is flaky

[david-german-tri]

In the continuous build, it sometimes fails with "Index exceeds matrix dimensions". example

This looks like a software flake, not an infrastructure flake, so developer investigation is needed. I'll assign it to myself but may not have time to work on it today. Feel free to grab it if you're interested/blocked.

[billhoffman]

Easier to see in CDash output then digging in the full output log on jenkins:
https://drake-cdash.csail.mit.edu/testDetails.php?test=280510&build=4890,

[jwnimmer-tri]

I'm a fan of fixing this ASAP, but merging PRs would be much easier (fewer CI false positives) if we just completely disabled the test for now.

[billhoffman]

We can add it to the exclude on the CI side if you want.

[david-german-tri]

It's not getting fixed today (at least by me), so yes, please do.

[billhoffman]

https://github.com/billhoffman/drake/commit/b48ff467de5e06a5637d8aacb9482e30b288ff3d

[RussTedrake]

We should not have the testExclude file anymore. If a test is flakey, we should comment it out in the cmakelists.txt and add a note. Otherwise the build servers are still doing something different than the developers.

[billhoffman]

OK, I think the best way to move forward would be to create a PR that comments out all of the tests we are currently excluding. Then when that is merged into master we wait a few days and remove the exclude stuff. Tests can then be enabled one at a time in a new PR and tested before they get re-enabled.

[RussTedrake]

Your original list of excludes was made because you did not have the known issue logic. I went through and made all of the tests pass on my machine — at least in the two or three times I ran them all. Perhaps we can try a build without any exclusions and see where we’re at?

On Apr 6, 2016, at 6:17 PM, Bill Hoffman [email protected] wrote:

OK, I think the best way to move forward would be to create a PR that comments out all of the tests we are currently excluding. Then when that is merged into master we wait a few days and remove the exclude stuff. Tests can then be enabled one at a time in a new PR and tested before they get re-enabled.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub https://github.com/RobotLocomotion/drake/issues/2013#issuecomment-206593502

[RussTedrake]

Just took a look. Flakey is right. It appears to be failing in a different place almost every time:
https://drake-cdash.csail.mit.edu/testSummary.php?project=1&name=systems/plants/test/testURDFmex&date=2016-04-06
That's scary. While I can still see the value in muting this, I would prioritize fixing it. Might be a memory issue.

[david-german-tri]

Clearly, I dropped the ball on this. Sorry about that! I will mark it priority: high and try to take a look this week.

[david-german-tri]

priority: high is clearly not accurate; it seems unlikely this will ever get TRI attention, being a MATLAB test, so I'm going to move it to the RLG backlog.

[jwnimmer-tri]

All issues must have owners; feel free to reassign elsewhere as needed (or close), etc.

[jwnimmer-tri]

Closing as won't-fix, because #6369 will be removing the relevant software.

If I'm confused and this should in fact stay open, please re-open an clarify the remaining unsolved problem(s) in a new comment here.