Scalapb: The sbt plugin cannot run protoc in NixOS

[jvican]

I'm using NixOS (linux machine) and I'm experiencing an error when I compile the protos from our build in sbt/zinc:

protoc-jar: embedded: bin/3.5.1/protoc-3.5.1-linux-x86_64.exe
protoc-jar: executing: [/tmp/protocjar6482673021349331976/bin/protoc.exe, --plugin=protoc-gen-scala=/tmp/protocbridge4181242464017027422, --scala_out=grpc:/home/jvican/Code/bloop/zinc/internal/zinc-persist/target/scala-2.12/src_managed/main, -I/home/jvican/Code/bloop/zinc/internal/zinc-persist/src/main/protobuf, -I/home/jvican/Code/bloop/zinc/internal/zinc-persist/target/protobuf_external, -I/home/jvican/Code/bloop/zinc/internal/zinc-testing/src/main/protobuf, -I/home/jvican/Code/bloop/zinc/internal/zinc-testing/target/protobuf_external, -I/home/jvican/Code/bloop/zinc/internal/compiler-interface/src/main/protobuf, -I/home/jvican/Code/bloop/zinc/internal/compiler-interface/target/protobuf_external, -I/home/jvican/Code/bloop/zinc/internal/zinc-persist/src/main/protobuf, -I/home/jvican/Code/bloop/zinc/internal/zinc-persist/target/protobuf_external, -I/home/jvican/Code/bloop/zinc/internal/zinc-core/src/main/protobuf, -I/home/jvican/Code/bloop/zinc/internal/zinc-core/target/protobuf_external, -I/home/jvican/Code/bloop/zinc/internal/zinc-classfile/src/main/protobuf, -I/home/jvican/Code/bloop/zinc/internal/zinc-classfile/target-2.12/protobuf_external, -I/home/jvican/Code/bloop/zinc/internal/zinc-apiinfo/src/main/protobuf, -I/home/jvican/Code/bloop/zinc/internal/zinc-apiinfo/target-2.12/protobuf_external, -I/home/jvican/Code/bloop/zinc/internal/zinc-classpath/src/main/protobuf, -I/home/jvican/Code/bloop/zinc/internal/zinc-classpath/target-2.12/protobuf_external, -I/home/jvican/Code/bloop/zinc/internal/compiler-bridge/src/main/protobuf, -I/home/jvican/Code/bloop/zinc/internal/compiler-bridge/target-2.12/protobuf_external, /home/jvican/Code/bloop/zinc/internal/zinc-persist/src/main/protobuf/schema.proto]
protoc-jar: caught exception, retrying: java.io.IOException: Cannot run program "/tmp/protocjar6482673021349331976/bin/protoc.exe": error=2, No such file or directory

It looks like the logic to run protoc fallbacks on running a windows executable (note the use of the .exe script). For context, Nix doesn't place installed binaries in the common locations (/usr/bin/bash or /bin/bash for example leave somewhere else, so to get the right location of their binaries you need to use the standard #!/usr/bin/env bash). It may happen that some of the scripts that run the protoc bridge are not finding the correct binary. I've tried to follow the logic in the sbt plugin but it didn't bring me anywhere.

How can I get scalapb working on NixOS?

[thesamet]

See https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/scalapb/ijOn6tQngrc/Thn833ZOAwAJ
and https://github.com/os72/protoc-jar/issues/45

protoc-jar does not work on NixOS, I think it is the lack of glicbc. The workaround is to install protoc locally and have the plugin execute it directly and not throught sbt-protoc. Details on how to do that are in the Google Groups link.

[jvican]

Thank you for the pointers. I couldn't fix the issue by installing glibc, so I installed protoc via Nix and then modified the path in the plugin via:

PB.runProtoc in Compile := (args => scala.sys.process.Process("/path/to/protoc", args)!)

[bobymicroby]

For everyone who stumbles upon this issue, you can just nix-env -iA nixos.glibc nixos.patchelf , then
patchelf --set-interpreter /nix/store/7gx4kiv5m0i7d7qkixq2cwzbr10lvxwc-glibc-2.27/lib64/ld-linux-x86-64.so.2 ...protoc-gen-grpc-java-1.17.0-linux-x86_64.exe

[raboof]

The problem is that the downloaded protoc binaries have the path to /lib64/ld-linux-x86-64.so.2 hard-coded in the .interp ELF header, but NixOS doens't put it there (because NixOS allows different interpreters to exist side-by-side and wants to avoid accidentally depending on a global/arbitrary/indeterministic one).

Post-processing the downloaded executables with patchelf --set-interpreter or using the system protoc by setting PB.runProtoc both work but seemed cumbersome. Luckily NixOS has a buildFHSUserEnv utility so you can run sbt in a more FHS-ish sandbox.

I'm using the following shell.nix:

{ pkgs ? import <nixpkgs> {} }:

(pkgs.buildFHSUserEnv {
  name = "akka-grpc";

  targetPkgs = pkgs: [ pkgs.sbt pkgs.glibc ];

  runScript = "sbt";
}).env

To start sbt in this sandbox, run nix-shell --pure in the directory containing this shell.nix and you should be able to use sbt-protoc as usual.

[Zhen-hao]

The problem is that the downloaded protoc binaries have the path to /lib64/ld-linux-x86-64.so.2 hard-coded in the .interp ELF header, but NixOS doens't put it there (because NixOS allows different interpreters to exist side-by-side and wants to avoid accidentally depending on a global/arbitrary/indeterministic one).

Post-processing the downloaded executables with patchelf --set-interpreter or using the system protoc by setting PB.runProtoc both work but seemed cumbersome. Luckily NixOS has a buildFHSUserEnv utility so you can run sbt in a more FHS-ish sandbox.

I'm using the following shell.nix:

{ pkgs ? import <nixpkgs> {} }:

(pkgs.buildFHSUserEnv {
  name = "akka-grpc";

  targetPkgs = pkgs: [ pkgs.sbt pkgs.glibc ];

  runScript = "sbt";
}).env

To start sbt in this sandbox, run nix-shell --pure in the directory containing this shell.nix and you should be able to use sbt-protoc as usual.

so far this is the best solution.
all other solutions require either

1. patching your OS or user profile in an impure manner, which is against the Nix philosophy and can cause problems elsewhere (yes, I've been there)
2. changing your sbt project. it won't be a problem if you are the owner of the project. if not, good luck on getting your PR merged.

With this buildFHSUserEnv solution, I can use vscode for editing and code navigation as usual and the pure nix-shell to run tests.

Note that the merged NixOS/nixpkgs#69057 has been reverted and hence no longer an option.

[raboof]

so far this is the best solution.

I'm glad it is useful to you! By the way, https://github.com/NixOS/nixpkgs/pull/80457 makes buildFHSuserEnv much nicer.

it won't be a problem if you are the owner of the project. if not, good luck on getting your PR merged.

Even though I maintain akka-grpc, I'm still using the buildFHSuserEnv solution instead of changing PB.runProtoc because the latter would require everyone who wants to build the library to have the right version of protoc installed.

[thesamet]

I have some good news for those on NixOS! Latest version of sbt-protoc (1.0.0-RC1) tries to work around that protoc.exe and other protoc plugins are expecting ld-linux in a standard location.

The workaround works like this: sbt-protoc looks for a NIX_CC environment variable, which is then used to locate a local dynamic linker to your environment. The dynamic linker is then used to run protoc itself and any downloaded plugins. There is no elf-patching involved, we just executing the linker directly and passing it protoc as the first argument.

@raboof 's solution works great, however I wanted to have a solution that doesn't require a chrooted environment so it can be used with alternatives to nix-shell such as lorri.

To summarize, the following shell.nix should be sufficient:

{ pkgs ? import <nixpkgs> {} }:

pkgs.mkShell {
  buildInputs = [
    pkgs.sbt
  ];
}

[Zhen-hao]

@thesamet it seems to work as you described on the unstable channel. Thanks!

[raboof]

Works well here, too - this is great!