Code-server: Add arm32v7 builder

[Matt-Gleich]

Create a new file called arm.Dockerfile that allows you to run it on arm32v7 CPUs such as the Raspberry Pi's CPU.

[nhooyr]

Could you elaborate on what the current problem is and how this helps?

[Matt-Gleich]

At the moment the docker image for code-server only works on x86 processors (desktop processor) and returns the following error when you try to run on an arm32v7 processor (such as the Raspberry Pi):

standard_init_linux.go:211: exec user process caused "exec format error"

This is due to the fact that the docker image is built on an x86 processor. In this case, we don't even need to change the base image thanks to the fact that node has an arm image already available. So, in theory, this adds support for running code-server on a Raspberry Pi!

One problem with this is that in order for it to work on a arm32v7 processor it needs to be built on an arm32v7 processor. Sadly github-actions and travis-ci both don't support you to run your CI on arm32 CPUs. There are a few solutions to this tough:

1. Run a Jenkins server on a Raspberry Pi that you or someone on the team buys. I made a docker image that comes with Jenkins and docker installed https://hub.docker.com/repository/docker/mattgleich/arm32v7jenkins-docker
2. Try building it on an x86 with a CPU arm emulation tool such as QEMU

I am going to work to get this working and will submit a PR once done

[sr229]

@nhooyr we tried building it over at Drone but it keeps OOMing, we're still investigating what we can do with the build toolchain since it should work under normal circumstances since we run with 128GB RAM in Drone Cloud

[nhooyr]

That is unfortunate news. Not sure what we can do.

[nhooyr]

https://discourse.drone.io/t/drone-cloud-arm32-ooms-when-building/6622

[nhooyr]

Ah so we need to compile for ARM32 on ARM64.

[sr229]

I'll see on how to compile the project using 32-bit libs but AFAIK documentation to do it for Node.js is quite scarce but I'll keep looking.

[sr229]

@nhooyr AFAIK we should be compiling this on a 32-bit capable container, preferrably Raspbian, in a ARM64 environment (Raspbian can run 32-bit userland against a 64 bit kernel so it shouldn't be an issue).

[aptalca]

@sr229 I'm with linuxserver.io and we publish a code-server image. We use native builders with our jenkins and have a bunch of arm32v7 devices. If you point me to your specific build instructions, I'll try to build it and let you know.

[aptalca]

I tried a build on an odroid xu4, running armbian buster (arm32v7), kernel 4.14.165.
Ran in an ubuntu bionic docker container with node 12.15.0 and yarn 1.21.1
Followed the steps from https://github.com/cdr/code-server#development
yarn watch resulted in the following error:

root@3d29582a8c4c:/tmp/vscode/src/vs/server# yarn watch
yarn run v1.21.1
$ cd ../../../ && yarn watch
$ gulp watch --max_old_space_size=4095
[15:29:46] Node flags detected: --max_old_space_size=4095
[15:29:46] Respawned to PID: 8612

<--- Last few GCs --->

[8612:0x1e47950]     2561 ms: Mark-sweep 20.7 (29.5) -> 20.7 (29.5) MB, 52.8 / 0.0 ms  (average mu = 0.606, current mu = 0.001) allocation failure GC in old space requested
[8612:0x1e47950]     2614 ms: Mark-sweep 20.7 (29.5) -> 20.7 (22.0) MB, 53.4 / 0.0 ms  (average mu = 0.437, current mu = 0.001) last resort GC in old space requested
[8612:0x1e47950]     2669 ms: Mark-sweep 20.7 (22.0) -> 20.7 (22.0) MB, 54.6 / 0.0 ms  (average mu = 0.278, current mu = 0.001) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x35b0e235 <JSObject>
    0: builtin exit frame: compileFunction(this=0x2b042315 <JSGlobal Object>,0x2fd0a595 <JSArray[5]>,0x2fd0a585 <JSArray[0]>,0x25340275 <undefined>,0x253403bd <false>,0x25340275 <undefined>,0,0,0x2fd1f5e9 <String[#53]: /tmp/vscode/node_modules/typescript/lib/typescript.js>,0xb0580089 <Very long string[7797825]>,0x2b042315 <JSGlobal Object>)

    1: _compile [0x4407861d] [internal/modules/cjs/loader....

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[nhooyr]

@aptalca yea thats the same error @sr229 was getting. We need to cross compile on 64bit arm for 32bit arm.

[aptalca]

Oh I see now, my mistake. I hadn't read the linked thread on drone and incorrectly assumed drone was using arm64 builders for arm32 work and that cross compile was the issue. That's why I tried on native arm32.

[nhooyr]

Some discussion at https://github.com/cdr/code-server/issues/35

[deftdawg]

My process for cobbling together builds is as follows:

1. Build Code Server on aarch64 docker container on a X86_64 host using qemu-aarch64-static via binfmt (linking needs ~12GB of RAM; impossible on a 32-bit arch which can address a max of 4GB per process)
2. Shipping the compiled directory to a native arm6 system (steps are the same for arm32v7); in my case an RPi Zero W / RPi 2
3. Going into code-server-*-built and running npm rebuild on everything (node arm6 unofficial build needed)

That produces something that can be run via the node arm6 unofficial build via node ./build/code-server*/out/vs/server/main.js --auth none

The packing can't be done on a RPi2(1GB), because build.ts is trying to pass the entire packed binary to fs.writeFile as a buffer which easily exceeds the 1GB; from what I've read it should be possible to use streams to reduce that footprint to something more sane (streams -> https://medium.com/dev-bits/writing-memory-efficient-software-applications-in-node-js-5575f646b67f)

I'm not sure if streams could also help with the 12GB ram footprint of the build phase; I don't even know where to start investigating that one as build has way more going on.

[sr229]

I'm going to assume the build script is passing the entire thing as a buffer hence the OOM errors. I think this is something @code-asher would likely be happy to investigate to optimize the builds.

[code-asher]

The next version will be out soon and it no longer packages into a
single binary so that should simplify building a bit.

[jsco2t]

Question (and if this needs to be a new issue just let me know):

I noticed that _some_ arm binaries are being produced as part of the official releases. Ex: https://github.com/cdr/code-server/releases/tag/2.1698

I grabbed the ARM64 versions and deployed them onto a RaspberryPi just to see if they would work. I'm getting a bad image format style error:

code-server2.1698-vsc1.41.1-linux-arm64 $ ./code-server
-bash: ./code-server: cannot execute binary file: Exec format error

This leads me to believe that the current ARM binaries aren't compiled for the version of ARM the raspberryPi I am using runs (I have access to a Pi Zero W, as well as a Pi 4). Is there any chance we can get full build instructions for ARM if we want to try to compile and run the server on a raspberry pi.

• I know that running the server on a RaspberryPi device, especially the 2.x code versions, will result in an OOM error - even if I managed to get the binary to run. I also understand the experience will be substandard. That being said - I still love the idea of being able to run this on ARM devices.

[code-asher]

Hopefully with v3 we'll be able to distribute arm64 releases that work a bit better but in the meantime check out https://github.com/cdr/code-server/blob/master/doc/CONTRIBUTING.md#build for how to build.

[skabber]

I tried building on my RaspberryPi 4 according to the CONTRIBUTING.md file but ran into this error.

pi@raspberrypi:~/Projects/code-server $ yarn build
yarn run v1.22.4
$ yarn runner build
$ cd ./ci && NODE_OPTIONS=--max_old_space_size=32384 ts-node ./build.ts build
[default] rootPath is "/home/pi/Projects/code-server"
[default] codeServerVersion is "3.0.1"
[build] cleaning up old build...took 8ms
[build] building vs code...failed
Command failed: yarn gulp compile-build
/usr/bin/node[16083]: ../src/node_platform.cc:414:std::shared_ptr<node::PerIsolatePlatformData> node::NodePlatform::ForIsolate(v8::Isolate*): Assertion `data' failed.
Aborted
error Command failed with exit code 134.

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. 

[code-asher]

That's a new one. Are you building with Node 12?

[encombhat]

That's a new one. Are you building with Node 12?

Same problem, node 10.17.0, yarn 1.22.4, debian buster armhf

[skabber]

Once I switched from node 10 to 12, I get a JavaScript heap out of memory error.

[b00giZm]

Once I switched from node 10 to 12, I get a JavaScript heap out of memory error.

Same for me.

Node: v12.16.1
Yarn: 1.22.4

[deftdawg]

Gents, kindly read the entire thread before "me too" posting... Per previous posts (https://github.com/cdr/code-server/issues/1337#issuecomment-585522283), it requires 12GB of RAM to build.

[nhooyr]

Ya'll can test the npm package: yarn global add code-server. 🎉

[jguittard]

@nhooyr can you elaborate on this great news please? ^^

[nhooyr]

You can install node on a arm32v7 CPU and then run npm install -g @coder/code-server to install and use code-server.

[jguittard]

Awesome! Which _exact_ sem-ver of node is required?

[nhooyr]

There is no restriction, the native modules are compiled against whatever version of node you have installed :)

[jguittard]

I’ll finally be able to use my iPad as a dev device with my pi4 🎉

[skabber]

Works for me!
Thanks

[dreamorosi]

For future reference, the installation process requires also yarn and if you don’t have libx11-dev, libxkbfile-dev and libsecret-1-dev it won’t compile.

Other than that it works great, finally we can run it on Raspberry Pi!

[jguittard]

@dreamorosi came up to first two dependencies, thanks for the third ;)

[Sebobo]

Works great thanks !

[dreamorosi]

Has anyone been able to run this version on raspberry pi via HTTPS and use it on iPadOS?

The HTTP without certificate works fine but as we know with limitations (no clipboard). If I add certificates or I use reverse proxy it works until the login screen but then it shows a blank screen.

I think the issue is localized to iPadOS because if I had some up to access thru laptop it works fine also with HTTPS.

@jguittard @Sebobo

[nhooyr]

@dreamorosi See https://github.com/cdr/code-server/issues/715

[dreamorosi]

@dreamorosi See https://github.com/cdr/code-server/issues/715

What should I look at exactly? In that thread there are mentions to an inexistent app, a paid 3rd party service and ssh access.

[nhooyr]

Sorry linked the wrong issue at first. #715 is what I meant to link. You need a certificate signed by a real CA for iPadOS, can't use a self signed certificate.

[nhooyr]

The npm package is now just code-server. https://www.npmjs.com/package/code-server

We were able to acquire it from the previous owner.

[nhooyr]

The npm PR is now official, see https://github.com/cdr/code-server#npm for docs.

Keeping this issue open until we have a armv7 builder for static releases.

[radokristof]

@nhooyr Thanks it works great on Raspberry. Wonder if you could publish images on Docker Hub built for arm32v7 as well?

Thanks!

[aptalca]

If anyone's still interested, the linuxserver.io image is now multi-arch, supporting amd64, arm32v7 and arm64v8.
It also has a bunch of mods you can install for things like docker&docker-compose, python2&3, nodejs, powershell, golang, etc.
https://github.com/linuxserver/docker-code-server

EDIT: well, we tested 3.3.0 with multi-arch and it worked great on both arm32 and arm64. 3.3.1 got released right before we merged and published and unfortunately it doesn't seem to work on arm. Also noticed 3.3.1 added a whole lot of node modules as dependencies as the image size grew considerably between 3.3.0 and 3.3.1: https://github.com/cdr/code-server/issues/1704
Sorry to announce here prematurely. Our dev image with 3.3.0 seems to work still, which you can find at this docker hub endpoint: lsiodev/code-server:v3.3.0-pkg-4c123a3c-dev-9137da0f154cbad6946846cce08ee42ddd603a1a

[nhooyr]

@aptalca That's awesome!

I'll be sure to add to the README under the docker section.

Any ideas why 3.3.1 doesn't work on ARM? The release is tested on arm64 by CI so it should be good.

And I've responded re #1704

[aptalca]

@nhooyr Thanks. I'll open a new issue with logs from the arm image of 3.3.1

[aptalca]

With a little bit of help from @nhooyr we fixed up the images and pushed a new build of 3.3.1 to https://github.com/linuxserver/docker-code-server
This one works on arm (passed basic gui tests)

[nhooyr]

@aptalca Congrats! Did you manually test or do you have some automated tests? I'd love to check them out if possible.

[nhooyr]

And do you guys host your own arm32 builders?

[aptalca]

Our jenkins does automated checks, just a basic docker run, take a screenshot and get the logs and packages, but for some reason in this case I guess it took the screenshot a little too early, before the gui came up: https://lsio-ci.ams3.digitaloceanspaces.com/linuxserver/code-server/v3.3.1-ls43/index.html

Yes, we have arm32 and arm64 builders: https://ci.linuxserver.io/
It also spins up some aws slaves (x86_64 and arm64) when there is a backlog

[nhooyr]

That's awesome, we'd love to adopt that test in our CI with credit of course. Can you share the source?

[aptalca]

Sure it's all open on github
This is the overview of our CI: https://github.com/linuxserver/pipeline-triggers
The jenkinsfiles are templated and this is the builder: https://github.com/linuxserver/docker-jenkins-builder
It takes in jenkins-vars.yml (and others) and generates the Jenkinsfile
In the Jenkinsfile, this is the part that does the test: https://github.com/linuxserver/docker-code-server/blob/master/Jenkinsfile#L483-L531
And it basically runs this tester script: https://github.com/linuxserver/docker-ci/blob/master/ci/ci.py
and that uses this image to take the screenshot: https://github.com/linuxserver/docker-tester

You probably just need to rip out the test part and the screenshot parts. We had to centralize it and template it all since we use them over a lot of repos.

[nhooyr]

@aptalca Thank you so much! That will be incredibly useful. Currently our CI only installs an extension and confirms that worked but the screenshot approach is far more robust.

[aptalca]

Sure, let us know if we can help in any way. You can pop into our discord to discuss further or if you have questions: https://discord.gg/YWrKVTn

[Rathna-K]

Hi @aptalca @nhooyr trying to pull a docker image for coder to run RPi4. Where should I go for an “official image” ? Meaning one will be supported in the future? is it Linux Server io? Or something else? I saw a lot of threads around this in here but still unclear on which one to use.
Oh also, if I use Linuxserver.io image how can I use port 80 as I have a reverse proxy already. I see you expose 8443 so I am assuming you are adding self signed cert?
TIA

[aptalca]

@Rathna-K
linuxserver.io is third party, not official.
If your pi is running a 64 bit OS, you can use either image. But if it is running a 32 bit OS like the stock raspbian image, then linuxserver one is your only choice for docker image at the moment.

The container port for the linuxserver one is 8443 because we published that image before code-server changed their default port from 8443 to 80. We didn't want to break anything for our existing users so we kept it as 8443. But you can map that port to whatever port you like on your host anyway since it's docker.
It serves http by default and we recommend a reverse proxy like this one: https://github.com/linuxserver/docker-letsencrypt it comes with a preset proxy conf that you can easily enable: https://github.com/linuxserver/reverse-proxy-confs/blob/master/code-server.subdomain.conf.sample

One cool thing about the linuxserver image is the list of mods we provide, that install popular dev packages in the container: https://mods.linuxserver.io/?mod=code-server
They are enabled by a simple env var when creating the image and thus are persistent through container recreation unlike things that are manually installed inside a running container (they disappear along with the container during update/recreation)

[nhooyr]

Yes, in terms of an arm32 docker image, your only choice at the moment is linuxserver.io.

For an officially supported solution you can use the NPM package and follow our setup guide. https://github.com/cdr/code-server/blob/master/doc/guide.md

[tidux]

The Pi 4 has full aarch64 support coming with Raspbian, and it already exists via Ubuntu and Gentoo, so it might be easier to just replace the hardware.

[nhooyr]

Yea but most users will still be using arm32v7 so we'll be sure to add a builder in the next few weeks.

[sr229]

If it helps you @nhooyr, Docker has native cross building support in Beta, so you just need to install the Edge version to use the BuildKit project.

[nhooyr]

I gave that a shot a few weeks ago @sr229

Unfortunately, qemu's user mode emulation kept erroring out when building code-server :(

Don't remember the specifics right now. I'll give it a shot again at some point.

[imgmatt]

Ya'll can test the npm package: yarn global add code-server. 🎉

Hey, apologies for the bump... not sure if I'm being a complete bozo but this has me stumped. Tried this a few times on my Pi 3B + running node v12 and all dependencies. I'm getting a successful install (success Installed "[email protected]" with binaries) but code-server doesn't run. Not sure the install even did anything, even with --unsafe-perm added.

[code-asher]

Does it give any errors? If it's something like "command not found" then you might just need to add the yarn binary directory to your PATH environment variable. I think by default it's ~/.yarn/bin.

[imgmatt]

Does it give any errors? If it's something like "command not found" then you might just need to add the yarn binary directory to your PATH environment variable. I think by default it's ~/.yarn/bin.

That worked, thanks! code-server is now running without throwing any errors in terminal but getting 'refused to connect' in the browser when loading the default address. Seems like default Raspbian install might need some firewall/port forward updating.

[code-asher]

Are you connecting directly to code-server? If that's the case you might need to add --host 0.0.0.0. By default it listens on localhost which is not accessible externally.

[imgmatt]

Are you connecting directly to code-server? If that's the case you might need to add --host 0.0.0.0. By default it listens on localhost which is not accessible externally.

I'm connecting from another machine on my local network, but it was running @ 127.0.0.1:8080. Also tried 0.0.0.0 but still refused.

If I VNC into the Pi, I can connect to code-server in that browser though - not ideal.

[code-asher]

Hmm yeah if you're just accessing from another local machine via the ip then 0.0.0.0 will be necessary. But if that's still not working then it could definitely be a firewall issue.

[code-asher]

I'm not super familiar with Raspbian but a cursory search seems to suggest it doesn't have a firewall enabled by default. Have you enabled iptables or ufw or something?

[sr229]

I think we should still supply prebuilt binaries and have postinstall download them (fallback to compilation if cannot find a prebuilt) to save time for them.