Watchman: Binaries seem broken on linux

[stephenh]

I tried this build:

https://github.com/facebook/watchman/runs/289290807

And after extracting the zip and cd'ing into linux/bin, chmod u+x watchman, running ./watchman gives:

$ ./watchman
./watchman: error while loading shared libraries: _artifacts/linux/lib/libglog.so.0: cannot open shared object file: No such file or directory

After some guessing/googling, what technically (not sure if it's "correct") works is, while in the same bin directory, doing mkdir -p _artifacts/linux mv ../lib _artifacts_linux. I.e. just put the libs at the _artifacts/... path it was looking for.

Now ./watchman almost works but I get:

$ ./watchman
2019-11-05T13:23:28,518: [watchman] while computing sockname: failed to create /tmp/fbcode_builder_getdeps-ZhomeZrunnerZworkZwatchmanZwatchmanZbuildZfbcode_builder/installed/watchman/var/run/watchman/stephen-state: No such file or directory

I'm going to fallback to the non-binary / install-from-source route.

[wez]

Thanks for trying these!

The build process rewrites the shared library paths to not be the "randomized" local build paths, but the result still isn't super useful outside of the CI environment.

I think a reasonable game plan will be to build these as if they were to be installed with a prefix of something like /opt/com.github.facebook.watchman/. Making that happen is a little indirect because the things that builds these is generic across a number of the FB opensource builds and it may take a couple of iterations to get it right.

[stephenh]

Thanks for trying these!

Np! I really appreciate the work on them, as it'll make watchman much easier to get.

I'm not an expert on C/C++ native things, but FWIW my vote would be (if possible) to have 100% static binaries.

I.e. given my level of native-tool-chain expertise is "okay, I can _technically_ run make and kind of debug shared lib issues by leaning heavily on stackoverflow each time" (which I imagine is common among watchman users), the 100% static approach of "just put watchman in /usr/local/bin and you're done" would be pretty great.

[wez]

Yeah, I'm with you on the concept of static linkage.
It turns out to be more complicated than anyone would like because some combinations of third party dependencies on some platforms strongly resist being built statically, or don't like it when some of their dependencies are built statically. The process used in our CI builds (almost) everything from source using their unmodified build processes and even with those set to prefer static linkage we end up with things like glog being dynamic :-/

[stephenh]

Ah, gotcha. I didn't realize it was that nuanced/pita.

Fwiw I'm happy to "try again" at some point, so will lurk on this issue, but otherwise don't really have any time or primarily expertise in native development to be much help here. :-)

I just wanted to file this issue was an FYI in case it was an unknown issue (understood the CI builds are beta, but I didn't see any open issues for this in the tracker so figured it wouldn't hurt).

Thanks!

[omidraha]

I have the same issue, while trying to use binary downloads for Ubuntu 18.04

$ watchman 
watchman: error while loading shared libraries: _artifacts/linux/lib/libgflags.so.2.2: cannot open shared object file: No such file or directory

Info:

$ ls /usr/local/bin/ | grep watchman
watchman

Where is correct path for putting these two static lib files:

libgflags.so.2.2 
libglog.so.0

I put them here:

ls /usr/local/lib/ | grep libg
libgflags.so.2.2
libglog.so.0

I also try something like this:

$ ls /usr/local/bin/_artifacts/linux/lib/
libgflags.so.2.2  libglog.so.0

And this:

$ export LD_LIBRARY_PATH=/usr/local/lib/

But no ones worked for me.

[bormansquirrel]

@omidraha The process you followed is correct, but you moved the lib files into /usr/local/bin/_artifacts/linux/lib/ instead of /usr/local/lib/_artifacts/linux/lib/

[kumarharsh]

Just faced the same error in Mac. Sadly the binary gotten from brew is too old (4.9.0_3 instead of 4.9.4), and getting the artifacts from CI to work require a lot of jumping through the hoops. Hope this gets resolved soon-ish :)

[DhruvDh]

@omidraha The process you followed is correct, but you moved the lib files into /usr/local/bin/_artifacts/linux/lib/ instead of /usr/local/lib/_artifacts/linux/lib/

after doing that, I get

2020-01-16T15:56:59,873: [watchman] while computing sockname: failed to create /tmp/fbcode_builder_getdeps-ZhomeZrunnerZworkZwatchmanZwatchmanZbuildZfbcode_builder/installed/watchman/var/run/watchman/root-state: No such file or directory

[bormansquirrel]

@DhruvDh Sorry my mistake, my comment above is wrong ... the only way I got that working is following exactly the steps @omidraha mentioned but running watchman from the LD_LIBRARY_PATH (/usr/local/lib)

[wez]

We're still far from what I'd consider "done" here, but things are a bit better right now if you're on ubuntu.

https://github.com/facebook/watchman/actions/runs/91748642
has a zip that extracts to:

$ tree linux
linux
├── bin
│   └── watchman
└── lib
    ├── libgflags.so.2.2
    └── libglog.so.0

2 directories, 3 files

Those object files are built to be installed to /usr/local/bin (you'll need to chmod +x /usr/local/bin/watchman) and /usr/local/lib with no requirement to mess with the LD_LIBRARY_PATH.

You'll also need to setup the state dir:

sudo mkdir -p /usr/local/var/run/watchman
sudo chmod 2777 /usr/local/var/run/watchman

This particular build requires the ubuntu boost package(s) to be installed.

We're working towards at least getting this documented in the main docs, but also to make this a bit easier to install and use as well as publishing these things to a github release automatically.

[stephenh]

@wez Thanks! Fwiw / experience report, I tried this locally (I'm on Ubuntu 19.10) and am getting:

$ watchman --version
watchman: error while loading shared libraries: libboost_context.so.1.65.1: cannot open shared object file: No such file or directory

I poked around a bit about "do I have libboost-context" already? And it looks like I do:

[I] ~/Downloads $ apt-cache search libboost | grep context
libboost-context1.67.0 - provides a sort of cooperative multitasking on a single thread
libboost-context-dev - provides a sort of cooperative multitasking on a single thread (default version)
libboost-context1.67-dev - provides a sort of cooperative multitasking on a single thread
[I] ~/Downloads $ dpkg --listfiles libboost-context1.67.0
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libboost_context.so.1.67.0
/usr/share
/usr/share/doc
/usr/share/doc/libboost-context1.67.0
/usr/share/doc/libboost-context1.67.0/changelog.Debian.gz
/usr/share/doc/libboost-context1.67.0/copyright
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/libboost-context1.67.0
[I] ~/Downloads $ ls /usr/lib/x86_64-linux-gnu/libboost_context.so.1.67.0
/usr/lib/x86_64-linux-gnu/libboost_context.so.1.67.0

But it's version 1.67.0.... Should I try to force install version 1.65.1? No rush, I can poke around on debugging / teaching myself to fish as well, but if you know off your head / need to work into the WIP docs. Thanks!

[wez]

@stephenh ah, this is likely an ubuntu 16 vs later version issue. I just added a mode for CI that can build using system boost instead of building it ourselves and that is defaulting to on to make it quicker for the CI. We'll either need to turn it off, or run different jobs for different linux flavors.

[timhughes]

Fedora 32
Linux 5.6.12-300.fc32.x86_64

Followed the instruction at https://facebook.github.io/watchman/docs/install the
best I could. Went to the Linux section.

Downloaded from https://github.com/facebook/watchman/actions/runs/106826672

thughes@krypton [0] $ cat /etc/redhat-release 
Fedora release 32 (Thirty Two)
thughes@krypton [0] $ uname -a
Linux krypton 5.6.12-300.fc32.x86_64 #1 SMP Mon May 11 16:47:13 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
thughes@krypton [0] $ cd ~/Downloads/watchman/linux/
thughes@krypton [0] $ sudo cp bin/watchman /usr/local/bin/
[sudo] password for thughes:
thughes@krypton [0] $ sudo chmod 755 /usr/local/bin/watchman
thughes@krypton [0] $ watchman
watchman: error while loading shared libraries: /usr/local/lib/libgflags.so.2.2: cannot open shared object file: No such file or directory
thughes@krypton [0] $ sudo cp lib/* /usr/local/lib/
thughes@krypton [0] $ watchman
watchman: error while loading shared libraries: libboost_context.so.1.65.1: cannot open shared object file: No such file or directory
thughes@krypton [0] $ dnf whatprovides '*/libboost_context.so.1.65.1'
Last metadata expiration check: 0:02:05 ago on Sun 17 May 2020 18:28:29 BST.
Error: No Matches found
thughes@krypton [0] $ dnf info boost-devel
Last metadata expiration check: 0:02:49 ago on Sun 17 May 2020 18:28:29 BST.
Available Packages
Name         : boost-devel
Version      : 1.69.0
Release      : 17.fc32
Architecture : i686
Size         : 9.9 M
Source       : boost-1.69.0-17.fc32.src.rpm
Repository   : updates
Summary      : The Boost C++ headers and shared development libraries
URL          : http://www.boost.org
License      : Boost and MIT and Python
Description  : Headers and shared object symbolic links for the Boost C++ libraries.

Name         : boost-devel
Version      : 1.69.0
Release      : 17.fc32
Architecture : x86_64
Size         : 9.9 M
Source       : boost-1.69.0-17.fc32.src.rpm
Repository   : updates
Summary      : The Boost C++ headers and shared development libraries
URL          : http://www.boost.org
License      : Boost and MIT and Python
Description  : Headers and shared object symbolic links for the Boost C++ libraries.

Failed back to install from source and ran into this issue.

thughes@krypton [0] $ make
make  all-am
make[1]: Entering directory '/home/thughes/git/watchman'
  CXX      watchman-ChildProcess.o
  CXX      watchman-ContentHash.o
  CXX      watchman-CookieSync.o
  CXX      watchman-FileDescriptor.o
  CXX      watchman-FileInformation.o
  CXX      watchman-InMemoryView.o
  CXX      watchman-Pipe.o
  CXX      watchman-PubSub.o
  CXX      watchman-ThreadPool.o
  CXX      watchman-QueryableView.o
  CXX      watchman-spawn.o
  CXX      watchman-opt.o
  CXX      watchman-cfg.o
  CXX      watchman-clockspec.o
  CXX      watchman-checksock.o
  CXX      watchman-error_category.o
  CXX      watchman-fstype.o
  CXX      watchman-log.o
  CXX      watchman-ignore.o
In file included from ./watchman_string.h:15,
                 from thirdparty/jansson/jansson.h:10,
                 from watchman.h:9,
                 from ignore.cpp:4:
thirdparty/libart/src/art-inl.h: In instantiation of ‘static unsigned char art_tree<ValueType, KeyType>::keyAt(const unsigned char*, uint32_t, uint32_t) [with ValueType = unsigned char; KeyType = w_string; uint32_t = unsigned int]’:
thirdparty/libart/src/art-inl.h:661:36:   required from ‘art_tree<ValueType, KeyType>::Leaf* art_tree<ValueType, KeyType>::longestMatch(const unsigned char*, uint32_t) const [with ValueType = unsigned char; KeyType = w_string; uint32_t = unsigned int]’
ignore.cpp:25:73:   required from here
thirdparty/libart/src/art-inl.h:61:18: error: comparison of unsigned expression in ‘>= 0’ is always true [-Werror=type-limits]
   61 |     w_assert(idx >= 0 && idx <= key_len,
      |              ~~~~^~~~
./watchman_log.h:27:9: note: in definition of macro ‘w_check’
   27 |   if (!(e)) {                                                                  \
      |         ^
thirdparty/libart/src/art-inl.h:61:5: note: in expansion of macro ‘w_assert’
   61 |     w_assert(idx >= 0 && idx <= key_len,
      |     ^~~~~~~~
cc1plus: all warnings being treated as errors
make[1]: *** [Makefile:3549: watchman-ignore.o] Error 1
make[1]: Leaving directory '/home/thughes/git/watchman'
make: *** [Makefile:1263: all] Error 2

At this point I would have given up but I am evaluating Buck/Pants/Bazel and eventually ran into this link https://github.com/facebook/watchman/issues/789#issuecomment-602058433 which suggested using ./configure --enable-lenient which although mentioned in with the other flags may be better broken down into a troubleshooting section.

This has worked and I have now got a working watchman on Fedora 32. It wasn't easy and is a large barrier to entry for someone trying it fresh.

[TechhDan]

We're still far from what I'd consider "done" here, but things are a bit better right now if you're on ubuntu.

https://github.com/facebook/watchman/actions/runs/91748642
has a zip that extracts to:

$ tree linux
linux
├── bin
│   └── watchman
└── lib
    ├── libgflags.so.2.2
    └── libglog.so.0

2 directories, 3 files

Those object files are built to be installed to /usr/local/bin (you'll need to chmod +x /usr/local/bin/watchman) and /usr/local/lib with no requirement to mess with the LD_LIBRARY_PATH.

You'll also need to setup the state dir:

sudo mkdir -p /usr/local/var/run/watchman
sudo chmod 2777 /usr/local/var/run/watchman

This particular build requires the ubuntu boost package(s) to be installed.

This worked for me, thanks! Ubuntu 18.04 LTS

[wez]

Starting today, we have weekly tags with binaries uploaded to them: https://github.com/facebook/watchman/releases/latest
I'd appreciate your feedback on how well the linux binaries work in practice!

[stephenh]

@wez worked on the 1st try! Amazing, thanks!!

[timhughes]

Thanks @wez Worked fine first go on Fedora release 32 (Thirty Two) 5.7.7-200.fc32.x86_64

[luboslives]

Just did a fresh install on a CentOS 7 VPS, following the official instructions exactly. I tried to initialize my first watch but was greeted with:

watchman: error while loading shared libraries: libcrypto.so.1.1: cannot open shared object file: No such file or directory

Any suggestions/directions?