Prysm: Add ARM support

[penta314]

It has been found that the binaries are not built for ARM, hence people trying to run a node in for example Rasberry Pi 3 are getting crashes.

[prestonvanloon]

This requires build docker images/binaries with --platforms=@io_bazel_rules_go//go/toolchain:linux_arm64.

The validator client builds for ARM, but not the beacon-chain due to some cgo dependencies in go-ethereum.

[prestonvanloon]

I made a brief attempt at trying to cross compile for ARM, but running into weird issues with cgo.
Commit ea790c5a1340822c617c9ad1a62194d60adf1af9

[diglos]

Hi,
It fails compiling directly on ARM64 as well, see attached log.
beacon-chain-build-arm64.txt

[prestonvanloon]

The binaries should build on ARM as of e71e91c9aaa6ddf2a15a6fb329a95b7d88b8a254, but we don't officially support ARM binaries at this time.

[diglos]

Hi,
yes, it compiles fine on ARM64, thanks for noticing. I will be doing some testing with this.
Regards!

[XinyaYang0506]

@diglos Hi,
I am trying to use this on raspberry pi, and on the way figuring out how to compile this repo from the source. Can you give me some guidance? thanks!

[diglos]

Hi Xinya,
the tricky part is compile bazel for ARM. This is possible on a ARM64 board with 4GB RAM but I don't know if the Raspberry pi can do this. You may try this guide (section 3):
https://github.com/samjabrahams/tensorflow-on-raspberry-pi/blob/master/GUIDE.md
and, once you have the bazel binary, follow this page instructions to build the prysm validator and beacon chain.

[XinyaYang0506]

I will try this, thanks!

[prestonvanloon]

I think this was blocked by crosstool rewrite (https://github.com/bazelbuild/bazel/issues/5380).

Most likely, we need to add some cc toolchains for cross compilation. See: https://docs.bazel.build/versions/master/tutorial/cc-toolchain-config.html

There is a tutorial here for clang: https://github.com/bazelbuild/rules_go/blob/master/go/crosstool.rst

Other artifacts:

• https://github.com/bazelbuild/rules_go/issues/1642
• https://github.com/bazelbuild/rules_go/issues/1020

Now blocked by https://github.com/bazelbuild/rules_go/issues/2089

[jbriquet]

Here are the notes I have taken when compiling bazel for an ARM64 (4 cores A53, 2GB RAM).

Add repository to find openjdk-8 source
sudo add-apt-repository ppa:webupd8team/java

Install JDK8 and dependencies
sudo apt-get install build-essential openjdk-8-jdk python zip unzip

Download Bazel

wget https://github.com/bazelbuild/bazel/releases/download/0.25.3/bazel-0.25.3-dist.zip
unzip bazel-0.25.3-dist.zip -d bazel-0.25.3-dist

Install Bazel on targeted device
cd bazel-0.25.3-dist
export BAZEL_JAVAC_OPTS="-J-Xmx1g -J-Xms200m" \
export EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk"  \
./compile.sh


# Successful compilation on ARM64 - 2cores, 4GB

INFO: Elapsed time: 1449.174s, Critical Path: 139.99s
INFO: 1777 processes: 1538 local, 239 worker.
INFO: Build completed successfully, 1824 total actions

[prestonvanloon]

Thanks @jbriquet! My last comment is referring to cross complation. We’d love to build ARM64 binaries for you without unreasonable build environment, but I am glad to see that you were able to build one after 1450s(!!!).

ARM builds will soon be a priority for us and we’ll be providing regular ARM binaries for tagged releases soon :)

[diglos]

Great news!
I'm releasing beacon-chain and validator binaries in the next Ethereum on ARM release for NanoPC-T4 and RockPro64. So far, they work quite well :-)

[rauljordan]

This is resolved by #2970

[prestonvanloon]

Verifying that building on linux amd64 for arm64 works!

bazel build //beacon-chain //validator --platforms=@io_bazel_rules_go//go/toolchain:linux_arm64

[prestonvanloon]

Recent BLS PR has broken ARM builds #3670. Reopening this issue for tracking.

[diglos]

Hi,
I just ran into this issue trying to compile Prysm for the Raspberry Pi 4. Thanks for reopening this.

[diglos]

Hi,
Any ETA to fix this?
Thanks!

[nisdas]

hey @diglos, with #4181 merged in, we have ARM support added in

[diglos]

Hi,
Thanks @nisdas.
It compiles on ARM64 (NanoPC-T4) but fails on ARM32 (Raspberry Pi 4) for both, beacon chain and validator. Please check attached log.

prysm-arm7-error.txt

[nisdas]

hey @diglos , thanks for testing this out. It seems like we might not be giving the correct archive for bls, I will try and see where it fails

[wikidemic]

There’s an army of RPi4 standing by to join test network.
Any progress towards successful compile yet?

[nisdas]

Hey sorry currently we haven't been able to get it to work on arm32 yet, only arm64 is currently supported. I havent been able to dedicate the time to resolving this, I hope this can be done soon

[mangoleap]

I'm trying to get it to run on a NanoPC T4 (ARM 64). Installed docker and pulled the images.

When I run

docker run -v $HOME/prysm-data:/data -p 4000:4000 --name beacon-node gcr.io/prysmaticlabs/prysm/beacon-chain:latest --no-genesis-delay --datadir=/data

I get the error

standard_init_linux.go:211: exec user process caused "exec format error"

That's all, nothing else. Any idea how to proceed?

[wikidemic]

I

I'm trying to get it to run on a NanoPC T4 (ARM 64). Installed docker and pulled the images.

When I run

docker run -v $HOME/prysm-data:/data -p 4000:4000 --name beacon-node gcr.io/prysmaticlabs/prysm/beacon-chain:latest --no-genesis-delay --datadir=/data

I get the error

standard_init_linux.go:211: exec user process caused "exec format error"

That's all, nothing else. Any idea how to proceed?

@diglos seems to be on the ball here. If you don’t answer for your docker image question, suggest you look up diglox image here on github - diglos/userpatches.

I’m considering NanoPC T4 purchase from Friendly Arm this week. So I’m tracking developer responsiveness here before I pull trigger. Let us know if you make progress!

[prestonvanloon]

The current docker image is compiled for AMD64 so it will not work on ARM64. It's not compiled for ARM and I am not sure about any workaround.

To give you guys a sense of timeline, we most likely won't get to this until after the holiday.

We'd welcome anyone to attempt a resolution here. I have a few ideas if someone wants to attempt resolving builds for ARM64.

[diglos]

Hi @mangoleap.

Are you running a Ethereum on ARM image?. If so, you can install it by typing (last version):

sudo apt-get update && sudo apt-get install prysm-beacon prysm-validator

@prestonvanloon, @wikidemic, it compiles fine on ARM64 (NanoPC-T4 and RockPro64) after #4181 merged. ARM7 (Raspberry Pi 4), doesn't work at the moment.

[mangoleap]

@diglos That worked, thank you!

If anyone wants to try it:

Config files are at

/etc/ethereum/prysm-beacon.conf
/etc/ethereum/prysm-validator.conf

Chain and validator can be started with

sudo systemctl start prysm-beacon
sudo systemctl start prysm-validator

Auto start up on boot can be enabled with

sudo systemctl enable prysm-beacon.service
sudo systemctl enable prysm-validator.service

Real time output from both services can be watched with

sudo journalctl -u prysm-beacon -f
sudo journalctl -u prysm-validator -f

[rauljordan]

Done by @SuburbanDad and included in our https://docs.prylabs.network

[jgresham]

Link to install docs: https://docs.prylabs.network/docs/install/arm/

[wikidemic]

I just attempted to load the ‘Unofficial’ version(2.1.0) of Bazel for ARM architecture using docs below. Unfortunately, it did not work as the current version is now at 2.2 and does not incorporate this Arm edit (by SuburbanDad)

I haven’t coded in quite a while so I cann’t offer too much detail other than saying Bazel doesn’t load on my RPi4. Installation script error notes that Unofficial Bazel isn’t current version and references a config file that requires editing before it is happy.

Hope that helps.
Mc

Sent from my iPad

On Mar 6, 2020, at 6:35 PM, Johns Gresham [e-mail] wrote:


Link to install docs: https://docs.prylabs.network/docs/install/arm/

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[SuburbanDad]

@wikidemic prysm has pinned to bazel 2.1.1 for now. I added an arm64 build of 2.1.1:
https://github.com/SuburbanDad/bazel/releases/tag/2.1.1-arm64

give it a whirl

[jgresham]

Edit: It appears that although the raspi4 comes with a 64 bit cpu, then default OS (Raspbian Buster) on raspi4 is 32 bit. Is there any particular 64 bit OS that you all would recommend for the raspi4?

@SuburbanDad Should the bazel binaries on your releases work on a raspi4? I having issues after running step 3 from here https://docs.prylabs.network/docs/install/arm/ so I am trying to build prysm from source.

Here is the result from trying to use your bazel binaries on raspi4:

pi@raspberrypi:~/prysm $ ./bazel --version
-bash: ./bazel: cannot execute binary file: Exec format error
pi@raspberrypi:~/prysm $ file bazel
bazel: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=09b54219645538b60aaaf5829af803cf3fad9fd4, not stripped

Here is result from trying to run prysm on a raspi4:

pi@raspberrypi:~/prysm $ ./prysm.sh --clear-db --datadir=$HOME/prysm
pi@raspberrypi:~/prysm $ ls dist/
beacon-chain-v0.3.4-linux-armv7l  validator-v0.3.4-linux-armv7l
pi@raspberrypi:~/prysm $ ./prysm.sh beacon-chain
Latest Prysm version is v0.3.4.
Beacon chain is up to date.
Validator is up to date.
Starting Prysm beacon-chain
/home/pi/prysm/dist/beacon-chain-v0.3.4-linux-armv7l: line 1: Not: command not found

In the meantime, I am trying to build bazel 2.1.1 on a raspi4.

[SuburbanDad]

Hey @jgresham,

Edited after reading your edit:
the unofficial / dev bazel binaries I have here: https://github.com/SuburbanDad/bazel/releases
should work on any debian based arm64 os.

Ubuntu 19.10 is probably the easiest route. Personally I am running an unofficial distro of 18.04 from:
https://jamesachambers.com/raspberry-pi-4-ubuntu-server-desktop-18-04-3-image-unofficial/

Since it has pi-specific kernel optimizations from raspbian. I am hopeful that 20.04 will have better support when it comes out next month.

On Sun, Mar 15, 2020, 3:40 PM Johns Gresham [e-mail]
wrote:

@SuburbanDad https://github.com/SuburbanDad Should the bazel binaries
on your releases work on a raspi4? I having issues after running step 3
from here https://docs.prylabs.network/docs/install/arm/ so I am trying
to build prysm from source.

Here is the result from trying to use your bazel binaries on raspi4:

pi@raspberrypi:~/prysm $ ./bazel --version
-bash: ./bazel: cannot execute binary file: Exec format error
pi@raspberrypi:~/prysm $ file bazel
bazel: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=09b54219645538b60aaaf5829af803cf3fad9fd4, not stripped

Here is result from trying to run prysm on a raspi4:

pi@raspberrypi:~/prysm $ ./prysm.sh --clear-db --datadir=$HOME/prysm
pi@raspberrypi:~/prysm $ ls dist/
beacon-chain-v0.3.4-linux-armv7l validator-v0.3.4-linux-armv7l
pi@raspberrypi:~/prysm $ ./prysm.sh beacon-chain
Latest Prysm version is v0.3.4.
Beacon chain is up to date.
Validator is up to date.
Starting Prysm beacon-chain
/home/pi/prysm/dist/beacon-chain-v0.3.4-linux-armv7l: line 1: Not: command not found

In the meantime, I am trying to build bazel 2.1.1 on a raspi4.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/prysmaticlabs/prysm/issues/2546#issuecomment-599274699,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOSEBJRG27FVPNPOED4ACD3RHVKMHANCNFSM4HL3T3WA
.

[amaurer]

I found that the prysm.sh attempted download of the binaries failed from https://api.github.com/repos/prysmaticlabs/prysm/releases/latest

Line one of the binary file was "Not Found" which resulted in an error - line 1: Not: command not found

Turns out I was hitting the rate-limit imposed by github for unauthenticated requests.

{"message":"API rate limit exceeded for x.x.x.x . (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://developer.github.com/v3/#rate-limiting"}

I authenticated using curl -u "username" https://api.github.com/repos/prysmaticlabs/prysm/releases/latest -o ./releases.json and edited the curl line in prysm.sh to use the json file instead. Beacon node is up and running now