Drake: Need to Add Instructions On How To Recover From A Git Submodule "fatal: reference not in tree" Error

[liangfok]

_I recently encountered the following problem and figured it may be worthwhile to document the fix since it was not the first time I encountered it._

Documentation to Add

Problem Scenario

Suppose you merge or rebase the latest code from master and encounter a build error that looks like the following (even if you build at the super-build level):

fatal: reference is not a tree: 27929f1fd6e928ba536784e82ec67f7d474725c6
Unable to checkout '27929f1fd6e928ba536784e82ec67f7d474725c6' in submodule path 'externals/google_styleguide'
make[2]: *** [google_styleguide-prefix/src/google_styleguide-stamp/google_styleguide-update] Error 1
make[1]: *** [CMakeFiles/google_styleguide.dir/all] Error 2
make: *** [all] Error 2

Solution

Here's how to fix it (these instructions are for this specfic submodule, but they can be easily modified for other submodules):

$ cd drake-distro/externals/google_styleguide
$ git remote set-url origin https://github.com/mwoehlke-kitware/styleguide.git
$ git fetch origin
$ git checkout 27929f1fd6e928ba536784e82ec67f7d474725c6

The correct remote URL for origin is determined by looking in drake-distro/.gitmodules. The SHA was determined by the aforementioned error. You can also get the updated SHA by looking at the submodule in Github, for example here.

Once the above change is made, you should be able to build Drake like normal.

[jwnimmer-tri]

Related to (and thus beetlejuicing) #2757.

[sammy-tri]

Thank you! This has been annoying me for some time.

[bradking]

Just running git submodule sync should be sufficient.

The super-build is supposed to cause a git submodule sync to be run first to take care of such changes. For example:

$ ninja -v
[1/109] cd /.../drake-distro && /usr/bin/git submodule --quiet sync --
...

That should happen on every build (unless AUTO_UPDATE_EXTERNALS is off). Is anyone seeing this not happen?

[sammy-tri]

I actually do have AUTO_UPDATE_EXTERNALS set to off. I'm working with a forked external, and I haven't managed to find a workflow I like (needing a bunch of version bumping commits on my fork of drake vs. having my changes to the external rolled back when I run the superbuild). I'd been unable to find a manual workaround for google_styleguide on my own (hence the thank you above)

[liangfok]

I encountered this problem even though my AUTO_UPDATE_EXTERNALS is on:

$ cat drake-distro/build/CMakeCache.txt | grep AUTO_UPDATE
AUTO_UPDATE_EXTERNALS:BOOL=ON

It's possible I failed to run the build at the super-build level the first time I got the submodule update. If I try to build at the super-build level later, will the git submodule sync still work?

[bradking]

@sammy-tri please try git submodule sync next time you need this. I think that is what we should document. We should generalize this issue to "document workflow with AUTO_UPDATE_EXTERNALS off".

@liangfok the super-build should _always_ start with git submodule sync if AUTO_UPDATE_EXTERNALS is on. It is a no-op if it is not needed.

[liangfok]

@bradking OK thanks. I'm 100% certain I tried running the build at the super-build level after encountering the above-mentioned "fatal: reference not in tree" error. I then went into the submodule's directory and found that the origin remote URL was not updated.

$ cd drake-distro/externals/google_styleguide
$ git remote -v
origin  https://github.com/google/styleguide.git (fetch)
origin  https://github.com/google/styleguide.git (push)

I just wanted to confirm with you that running git submodule sync is supposed to first update origin's URL and then do the git checkout of the new SHA. Is this what's expected to happen? If so, next time this happens I'll try running git submodule sync to see if that solves the problem.

[liangfok]

Update

I found another clone of Drake on my machine that was not updated with the new google_styleguide submodule. I updated it with the latest on master:

$ cd drake-distro
$ git fetch upstream master

I then tried to build at the super-build level, but it failed with the above-mentioned error:

$ cd drake-distro/build
$ make
...
fatal: reference is not a tree: 27929f1fd6e928ba536784e82ec67f7d474725c6
Unable to checkout '27929f1fd6e928ba536784e82ec67f7d474725c6' in submodule path 'externals/google_styleguide'
make[2]: *** [google_styleguide-prefix/src/google_styleguide-stamp/google_styleguide-update] Error 1
make[1]: *** [CMakeFiles/google_styleguide.dir/all] Error 2
make: *** [all] Error 2

I went into the google_styleguide submodule and verified that the problem is due to the origin URL not being updated:

$ cd drake-distro/externals/google_styleguide
$ git remote -v
origin  https://github.com/google/styleguide.git (fetch)
origin  https://github.com/google/styleguide.git (push)

I then went into the super-build level and ran git submodule sync and it successfully updated the remote:

$ cd drake-distro
$ git submodule sync
Synchronizing submodule url for 'drake/cmake'
Synchronizing submodule url for 'externals/bot_core_lcmtypes'
Synchronizing submodule url for 'externals/bullet'
Synchronizing submodule url for 'externals/director'
Synchronizing submodule url for 'externals/dreal'
Synchronizing submodule url for 'externals/eigen'
Synchronizing submodule url for 'externals/gflags'
Synchronizing submodule url for 'externals/google_styleguide'
Synchronizing submodule url for 'externals/googletest'
Synchronizing submodule url for 'externals/ipopt'
Synchronizing submodule url for 'externals/lcm'
Synchronizing submodule url for 'externals/libbot'
Synchronizing submodule url for 'externals/nlopt'
Synchronizing submodule url for 'externals/snopt'
Synchronizing submodule url for 'externals/spdlog'
Synchronizing submodule url for 'externals/spotless'
Synchronizing submodule url for 'externals/swig_matlab'
Synchronizing submodule url for 'externals/swigmake'
Synchronizing submodule url for 'externals/yaml_cpp'

$ cd externals/google_styleguide/

$ git remote -v
origin  https://github.com/mwoehlke-kitware/styleguide.git (fetch)
origin  https://github.com/mwoehlke-kitware/styleguide.git (push)

With this change, I was able to compile Drake at the super-build level:

$ cd drake-distro/build
$ make
...
[100%] Built target drake

Conclusions

• Running a build at the super-build level did not update the URL of the origin remote of google_styleguide.
• Manually running git submodule sync within drake-distro updates the URL of origin of google_styleguide.

[bradking]

$ cd drake-distro/build
$ make

It looks like commit #2896 (9147917ae050d3992fdb87769844d0c53551b065) broke this for non-Ninja superbuilds. The logic to add the dependency on the submodule-sync target is now incorrectly conditioned to appear only for Ninja.

[bradking]

I've re-opened #2757 with this information since that issue is about the actual fix and this issue is about documentation.

[liangfok]

Thanks @bradking. I will now close this issue since I have great faith that the build system will soon be fixed to no longer require manual intervention due to this failure mode.