Scalapb: scala-native binaries for scalapbc CLI tools?

[mroth]

It appears #387 added support for Scala Native compilation. Would it be possible to have a release task that uses this or GraalVM native-image to generate native Linux binary command line tools for scalapbc and protoc-gen-scala? For running those in CI environments, it would be _greatly_ advantageous to not require the current JVM/bash dependencies.

[thesamet]

The code generator relies on Java protocol buffers, so that won't run as-is on Scala native, not sure about GraalVM. To remove the dependencies on Java protocol buffers, ScalaPB code generator would have to depend on ScalaPB's runtime which is possible, but requires some work.

[thesamet]

I was able to create native binaries with GraalVM native image. See commit 5f371382 (it's on a separate branch). You can download the binary at https://github.com/scalapb/ScalaPB/releases/tag/v0.9.1-test - can you try it out and let me know?

[mroth]

Cool! I tested it in a few different container images (alpine:latest and ubuntu:18.04), some notes that I hope can be helpful:

• Currently this only works on glibc-based Linux distributions such as Ubuntu, and fails on Alpine Linux. I'd highly recommend statically linking instead of dynamic linking for distribution (simply add a --static argument to GraalVM native-image tool*) -- the binary will be slightly larger overall, but it will should then on any Linux system including MUSL based distributions like the very popular Alpine Linux used often in Docker containers and CI as well).
• Re: file permissions, the binary currently downloads without the execute bit set (so an end user must chmod +x it before running). I suspect this is potentially due to the raw download from GitHub as I believe GraalVM native-image produces binaries that are execute bit set already? (looking at my process for making native binaries for scalafmt, that appears to be the case). Thus if If your plan is to distribute the binary release files, I'd recommend zip/tgz the binary file that goes to release -- in particular tgz can help preserve permissions on the file (though I believe zip preserve file permissions as well now on more recent versions).
• Minor note: is the name change from protoc-gen-scala to protoc-gen-scalapb part of other 0.9.1 changes, or related to this directly?

Looking good in general! Exciting to see this. LMK anything else I can do to help.

*: EDIT: ahh, looks like this is built on Travis-CI, looks like that is on Ubuntu but I'm not sure what packages they have installed by default, so it may be necessary to sudo apt install zlib1g zlib1g-dev for the --static to work. (LMK if this works and if not I can try to figure out what is needed in their environment.)

[thesamet]

Check out https://github.com/scalapb/ScalaPB/releases/tag/v0.9.1-test2 - now it's statically compiled and wrapped in a zip.
Not sure I follow the question on a rename from protoc-gen-scala to protoc-gen-scalapb . The latter is the name used for this in published artifacts in prior version.

[mroth]

Testing

Seems to be working! 🚀

Here's a quick Docker file I used to test in Alpine, in case you want to play with it yourself:

FROM alpine:latest
ARG RELEASE=https://github.com/scalapb/ScalaPB/releases/download/v0.9.1-test2/protoc-gen-scalapb-0.9.1-test2-linux-x86_64.zip
ARG SAMPLE=https://raw.githubusercontent.com/protocolbuffers/protobuf/master/examples/addressbook.proto

RUN apk add --no-cache protobuf protobuf-dev wget file tree
WORKDIR /root
RUN wget --quiet $RELEASE \
    && unzip protoc*.zip -d /usr/local/bin \
    && rm protoc*.zip

RUN wget --quiet $SAMPLE

RUN ls -lh /usr/local/bin/protoc-gen-scalapb
RUN file /usr/local/bin/protoc-gen-scalapb
RUN protoc --scalapb_out=. addressbook.proto
RUN tree .

And some relevant sections from the build output:

Step 8/11 : RUN ls -lh /usr/local/bin/protoc-gen-scalapb
-rwxr-xr-x    1 root     root       13.0M Aug 27 13:47 /usr/local/bin/protoc-gen-scalapb

Step 9/11 : RUN file /usr/local/bin/protoc-gen-scalapb
/usr/local/bin/protoc-gen-scalapb: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.32, BuildID[sha1]=10fc065166cb77925a1422d1175069c4d8836ced, with debug_info, not stripped

Step 10/11 : RUN protoc --scalapb_out=. addressbook.proto

Step 11/11 : RUN tree .
.
├── addressbook.proto
└── com
    └── example
        └── tutorial
            └── addressbook
                ├── AddressBook.scala
                ├── AddressbookProto.scala
                └── Person.scala

Binary naming

Re:

Not sure I follow the question on a rename from protoc-gen-scala to protoc-gen-scalapb . The latter is the name used for this in published artifacts in prior version.

Huh, in the release I was using prior, this is what I have downloaded on my workstation:

$ tree ~/Downloads/scalapbc-0.9.0/bin                                        
/Users/mroth/Downloads/scalapbc-0.9.0/bin
├── protoc-gen-scala
├── protoc-gen-scala.bat
├── scalapbc
└── scalapbc.bat

In particular, the protoc-gen-* plugin naming determines what is the name to be passed to protoc, if you look in the sample output from the Testing section above, this necessitates changing the invocation params to protoc from --scala_out to --scalapb_out. (Doesn't matter to me either way, just was making sure this is intentional!)

[thesamet]

Released ScalaPB 0.9.1 with native codgen binaries : https://github.com/scalapb/ScalaPB/releases

[mroth]

@thesamet awesome! I've noticed that in the current release 0.9.1 downloads, the linux/osx binaries (e.g. protoc-gen-scalapb-0.9.1-osx-x86_64.zip etc) use the filename protoc-gen-scalapb, whereas in the java download version (scalapbc-0.9.1.zip), you have bin/protoc-gen-scala and bin/protoc-gen-scala.bat. This could cause a documentation issue in that one would use --scala_out and the other would use --scalapb_out. (I'll be handling this by renaming the file locally for now, but thought you might want to standardize?)

Really cool to see tho!

[mroth]

Hmm, it appears that there may have been a regression in 0.9.1 release since -test2 as well. The file is now again showing up as dynamically linked instead of statically linked, and thus won't execute properly on Alpine Linux.

$ file *
protoc-gen-scalapb: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=0bfc55fd921ab327696a5e46aa9979ff0e7c50b4, with debug_info, not stripped

[thesamet]

I'll standardize. Looks like I messed up the condition https://github.com/scalapb/ScalaPB/blob/master/build.sbt#L382 (forgot to lowercase or something...)

[thesamet]

File names are standardized in 0.9.4, and the linux binary is static.