Drake: Mysterious CI Error - could not lock config file
The Error
15:10:33 Cloning into 'drake/doc/textbook'...
15:10:33 Submodule path 'drake/doc/textbook': checked out 'e051917c15a513e1924465060134164c85fe5141'
15:10:33 error: could not lock config file /home/ubuntu/workspace/linux-gcc-ninja-continuous-matlab-debug/.git/modules/drake/doc/textbook/config: File exists
15:10:33 Failed to register url for submodule path 'HTMLBook'
15:10:33 Failed to recurse into submodule path 'drake/doc/textbook'
Example Console Logs
liangfok
All 36 comments
A prior example:
https://drake-cdash.csail.mit.edu/viewBuildError.php?buildid=169296
Cloning into 'externals/xfoil'...
error: could not lock config file /home/ubuntu/workspace/linux-gcc-ninja-experimental/.git/modules/externals/xfoil/config: File exists
jwnimmer-tri
on 5 Nov 2016
This happens up to a few times per week or more. Let get it onto Kitware's radar.
jwnimmer-tri
on 8 Nov 2016
I have yet to be inconvenienced myself, but @jamiesnape @BetsyMcPhail, PTAL?
david-german-tri
on 8 Nov 2016
@hongkai-dai experienced this here:
https://drake-jenkins.csail.mit.edu/job/linux-gcc-ninja-experimental-open-source-debug/804/console
liangfok
on 21 Nov 2016
jwnimmer-tri
on 22 Nov 2016
Has anyone observed this on a non-Ninja build?
jamiesnape
on 22 Nov 2016
I just experienced this issue here:
https://drake-jenkins.csail.mit.edu/job/linux-gcc-ninja-experimental/1170/
liangfok
on 29 Nov 2016
Failing with snopt: linux-clang-ninja-continuous-debug-pre-drake
Cloning into 'externals/snopt'...
error: could not lock config file /home/ubuntu/workspace/linux-clang-ninja-continuous-debug/.git/modules/externals/snopt/config: File exists
Lucy-tri
on 30 Nov 2016
jwnimmer-tri
on 1 Dec 2016
liangfok
on 5 Dec 2016
liangfok
on 5 Dec 2016
liangfok
on 5 Dec 2016
liangfok
on 6 Dec 2016
We have enough information now and are trying to reproduce. Post if there is a failure that is not Linux and Ninja.
jamiesnape
on 6 Dec 2016
RobotLocomotion/drake-ci#62 will hopefully give us some information if it happens again, and should also work around the problem by retrying the failed git command. Builds that have git errors should show as 'unstable'. (For build-cop purposes, these can be treated as "clean", but the strace output should be inspected for clues.)
mwoehlke-kitware
on 13 Dec 2016
This cropped up again. This time affecting libccd. Per @jamiesnape's request, this is in a mac build.
SeanCurtis-TRI
on 10 Jan 2017
@SeanCurtis-TRI can you provide a link to the console log or CI build page?
liangfok
on 10 Jan 2017
Sorry, here you go: https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/mac-clang-ninja-nightly-minimal-debug/160/
SeanCurtis-TRI
on 10 Jan 2017
(Aside: CDash links persist longer than Jenkins links)
jamiesnape
on 10 Jan 2017
I suspected that might be the case, which is why the original posting used a cdash link.
SeanCurtis-TRI
on 10 Jan 2017
The build should not have failed outright, however. Looking closer, there are issues other than the "could not lock config file" problem:
[9/46] cd /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug && "/usr/local/Cellar/cmake/HEAD-ce71bd9/bin/ctest" --launch --target-name TARGET_NAME --build-dir /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build --output /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/CMakeFiles/submodule-sync -- /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/ci/tools/git-wrapper.bash submodule --quiet sync --
fatal: Not a git repository: /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/.git/modules/externals/ccd
fatal: Not a git repository: /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/.git/modules/externals/ccd
[10/46] cd /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug && "/usr/local/Cellar/cmake/HEAD-ce71bd9/bin/ctest" --launch --target-name TARGET_NAME --build-dir /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build --output /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/ccd-prefix/src/ccd-stamp/ccd-download -- /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/ci/tools/git-wrapper.bash submodule update --init --recursive -- externals/ccd && "/usr/local/Cellar/cmake/HEAD-ce71bd9/bin/ctest" --launch --target-name TARGET_NAME --build-dir /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build --output /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/ccd-prefix/src/ccd-stamp/ccd-download -- /usr/local/Cellar/cmake/HEAD-ce71bd9/bin/cmake -E touch /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/ccd-prefix/src/ccd-stamp/ccd-download
FAILED: ccd-prefix/src/ccd-stamp/ccd-download
cd /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug && "/usr/local/Cellar/cmake/HEAD-ce71bd9/bin/ctest" --launch --target-name TARGET_NAME --build-dir /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build --output /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/ccd-prefix/src/ccd-stamp/ccd-download -- /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/ci/tools/git-wrapper.bash submodule update --init --recursive -- externals/ccd && "/usr/local/Cellar/cmake/HEAD-ce71bd9/bin/ctest" --launch --target-name TARGET_NAME --build-dir /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build --output /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/ccd-prefix/src/ccd-stamp/ccd-download -- /usr/local/Cellar/cmake/HEAD-ce71bd9/bin/cmake -E touch /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/build/ccd-prefix/src/ccd-stamp/ccd-download
Cloning into 'externals/ccd'...
error: could not lock config file /Users/elcapitan/workspace/mac-clang-ninja-nightly-minimal-debug/.git/modules/externals/ccd/config: File exists
fatal: Could not set 'remote.origin.url' to 'https://github.com/danfis/libccd.git'
fatal: clone of 'https://github.com/danfis/libccd.git' into submodule path 'externals/ccd' failed
Cloning into 'externals/ccd'...
Submodule path 'externals/ccd': checked out '16b9379fb6e8610566fe5e1396166daf7106f165'
The failure despite succeeding on retry is due to a stupid logic error in the wrapper, fixed by RobotLocomotion/drake-ci#70. What is interesting however are the errors from the submodule sync attempt, which makes me wonder if all the "could not lock config file" errors are merely symptomatic of something having gone wrong earlier...
mwoehlke-kitware
on 10 Jan 2017
Great job sleuthing!
liangfok
on 10 Jan 2017
liangfok
on 17 Jan 2017
liangfok
on 26 Jan 2017
liangfok
on 6 Feb 2017
Demoting high -> medium, because the git retry logic is helping, and because in BazelDrake land we won't be shelling out to git commands for submodules anymore.
jwnimmer-tri
on 6 Mar 2017
liangfok
on 6 Mar 2017
liangfok
on 9 Mar 2017
liangfok
on 13 Mar 2017
liangfok
on 22 Mar 2017
liangfok
on 26 Mar 2017
liangfok
on 30 Mar 2017
liangfok
on 2 Apr 2017
liangfok
on 15 Apr 2017
liangfok
on 31 May 2017
Related issues
mntan3
路
4Comments
amcastro-tri
路
4Comments
RussTedrake
路
4Comments
palmieri
路
4Comments
Islam0mar
路
4Comments