Wav2letter: Did fbgemm git hashes change?

[EdwinWenink]

I am following this tutorial for building wav2letter-inference and running the Simple Streaming ASR Example. I cannot however successfully build the example (I did successfully complete all previous steps). The culprit seems to be that the git hash specified in cmake files for fbgemm is incorrect. This issue suggests that the git hashes of the fbgemm repo have changed:

> make interactive_streaming_asr_example -j $(nproc)
make[1]: Entering directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[2]: Entering directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[3]: Entering directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[3]: Entering directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[3]: Entering directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[3]: Leaving directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[3]: Entering directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[3]: Leaving directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[3]: Leaving directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[3]: Entering directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
[  4%] Performing update step for 'cereal'
[  4%] Performing update step for 'fbgemm'
[ 28%] Built target wav2letter-libraries
fatal: bad object b35b183e45a8b9af50efe29c7fd1b2460ad16182
[ 28%] Performing configure step for 'cereal'
loading initial cache file /home/edwin/Documents/AI/Master/NML/wav2letter/build/inference/cereal/tmp/cereal-cache-.cmake
-- Configuring done
-- Generating done
-- Build files have been written to: /home/edwin/Documents/AI/Master/NML/wav2letter/build/inference/cereal/src/cereal
[ 28%] Performing build step for 'cereal'
make[4]: Entering directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build/inference/cereal/src/cereal'
make[4]: Leaving directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build/inference/cereal/src/cereal'
[ 28%] No install step for 'cereal'
[ 28%] Completed 'cereal'
make[3]: Leaving directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
[ 28%] Built target cereal
make[3]: Entering directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[3]: Leaving directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
[ 32%] Built target streaming_inference_common
make[3]: Entering directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[3]: Entering directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[3]: Leaving directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[3]: Leaving directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
[ 32%] Built target util_example
[ 32%] Built target streaming_inference_modules_feature
fatal: reference is not a tree: b35b183e45a8b9af50efe29c7fd1b2460ad16182
CMake Error at /home/edwin/Documents/AI/Master/NML/wav2letter/build/inference/inference/module/fbgemm/tmp/fbgemm-gitupdate.cmake:146 (message):
  Failed to checkout tag: 'b35b183e45a8b9af50efe29c7fd1b2460ad16182'


make[3]: *** [inference/inference/module/CMakeFiles/fbgemm.dir/build.make:96: inference/inference/module/fbgemm/src/fbgemm-stamp/fbgemm-update] Error 1
make[3]: Leaving directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[2]: *** [CMakeFiles/Makefile2:865: inference/inference/module/CMakeFiles/fbgemm.dir/all] Error 2
make[2]: Leaving directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make[1]: *** [CMakeFiles/Makefile2:968: inference/inference/examples/CMakeFiles/interactive_streaming_asr_example.dir/rule] Error 2
make[1]: Leaving directory '/home/edwin/Documents/AI/Master/NML/wav2letter/build'
make: *** [Makefile:424: interactive_streaming_asr_example] Error 2

An easier way to reproduce this is to manually go to the fbgemm folder ( cd wav2letter/build/inference/inference/module/fbgemm ) and try to checkout the hash:

git checkout b35b183e45a8b9af50efe29c7fd1b2460ad16182
fatal: reference is not a tree: b35b183e45a8b9af50efe29c7fd1b2460ad16182

If I manually change the hash in one of the cmake files in fbgemm/tmp/ , the build process progresses a bit further but stops at the next location the b35b.. hash occurs.

[Chreschtf]

have the same issue

[bigcmos]

I had the same issue with a different repo that depends on fbgemm: https://github.com/pytorch/pytorch/issues/35149

I was able to work around it by finding the new hash for the same commit.

I searched the logs for the hash you provided and it appears that the new hash should be this: 4491fb4de96f390084af50073239d747b40ac11b

Which the commit message matches the one here:
https://github.com/pytorch/FBGEMM/commits/b35b183e45a8b9af50efe29c7fd1b2460ad16182

If you manually checkout the new commit hash in fbgemm it may unblock your build.

git checkout 4491fb4de96f390084af50073239d747b40ac11b
Note: checking out '4491fb4de96f390084af50073239d747b40ac11b'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 4491fb4... Add an API to unintrusively unpack PackedGemmMatrixFP16 (#201)

[EdwinWenink]

Much obliged, thanks for the effort. Based on the commit messages I indeed find the same hash ( here, for completeness). For me, the workaround requires replacing the hash a few times in different makefiles so, in addition, I do think an update to those makefiles would be appreciated.

[jacobkahn]

@EdwinWenink @bigcmos — thanks for flagging. Somehow, it looks like the hashes changed. We aren't sure what happened, https://github.com/facebookresearch/wav2letter/commit/8158c03bf266109ae2951877b2945d3c3307e996 updates the hash to the current FBGEMM master, which fixes the problem. We'll continue keeping an eye on it.