Rancher: Make Rancher ARM compatible

[deniseschannon]

Main components

• [ ] cattle [rancher/cattle#1739]
• [ ] strongswan-package [rancher/strongswan-package#1]
• [ ] rancher-net [rancher/rancher-net#13]
• [ ] swarm-agent
• [ ] cadvisor-package [rancher/cadvisor-package#12]
• [ ] rancher-metadata [rancher/rancher-metadata#18]
• [ ] websocket-proxy [rancher/websocket-proxy#38]
• [ ] rancher-catalog-service [rancher/rancher-catalog-service#48]
• [ ] rancher-dns [rancher/rancher-dns#32]
• [ ] agent-instance [rancher/agent-instance#18]
• [x] rancher-compose
• [x] python-agent (no changes needed, so no PR)
• [ ] rancher [#5217]
• [ ] go-machine-service [rancher/go-machine-service#103]
• [ ] host-api [rancher/host-api#77]
• [ ] go-rancher-metadata
• [ ] machine-package
• [ ] docker/machine (no PR. https://github.com/imikushin/machine/releases/tag/v0.7.0_arm)
• [ ] vmware/govmomi (no PR. https://github.com/imikushin/govmomi/releases/tag/v0.2.0_arm)
• [ ] rancher-docker-api-proxy
• [ ] runc (share folder)
• [ ] agent-binaries (This is _kind of_ needed: not if ubuntu:16.04 is the base image for agent and agent-instance)

Additional Features Added

• [ ] convoy
• [ ] convoy-agent
• [ ] kubernetes
• [ ] kubectld
• [ ] kubernetes-agent
• [ ] kubernetes-model
• [ ] rancher-ingress
• [ ] swarm
• [ ] vm
• [ ] giddyup (no ARM-specific PR. https://github.com/imikushin/giddyup/releases/tag/v0.10.1_arm)
• [ ] cluster-manager (no PR. https://github.com/imikushin/cluster-manager/releases/tag/v0.1.4_arm)

[faddat]

That's one awe-inspiring checklist :).

How will you be going about porting each component? Is it just a matter of getting an arm build for each to work? That, I can help with.

[nunofgs]

Can't wait for this 😄

[teleki]

Just assembled a 4 unit ODROID C2 cluster and can't wait to get Rancher going on it.

[imikushin]

@teleki you can! Just build rancher/server and rancher/agent for ARM from #4704, rancher/agent-instance from rancher/agent-instance#18

[teleki]

@imikushin Thank you for the advice. I tried building rancher/server, but I am getting errors. Here is the output of running as docker ./build-image.arm.sh in the rancher/server folder:

++ dirname ./build-image.arm.sh
+ cd .
+ . ../.docker-env.arm
./build-image.arm.sh: line 6: ../.docker-env.arm: No such file or directory
+ echo 'Need to source .docker-env.arm to access Docker on ARM'
Need to source .docker-env.arm to access Docker on ARM
+ '[' '!' -e target/arm/.done ']'
+ mkdir -p target/arm
++ cat /dev/urandom
++ env LC_CTYPE=C tr -cd a-f0-9
++ head -c 16
+ S6_BUILDER=0484087bb24a011b
+ trap 'docker rm -fv 0484087bb24a011b 2>/dev/null || :' EXIT
+ docker run --name=0484087bb24a011b rancher/s6-builder:v2.2.4.3_arm /opt/build.sh
Unable to find image 'rancher/s6-builder:v2.2.4.3_arm' locally
Pulling repository docker.io/rancher/s6-builder
docker: Tag v2.2.4.3_arm not found in repository docker.io/rancher/s6-builder.
See 'docker run --help'.
+ docker rm -fv 0484087bb24a011b
+ :

Here is the output of git status:

On branch pr/4704
Your branch is up-to-date with 'origin/pr/4704'.
nothing to commit, working directory clean

docker version

Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.6.1
 Git commit:   20f81dd
 Built:        Wed, 20 Apr 2016 14:19:16 -0700
 OS/Arch:      linux/arm64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.6.1
 Git commit:   20f81dd
 Built:        Wed, 20 Apr 2016 14:19:16 -0700
 OS/Arch:      linux/arm64

Here is the output of docker images:

REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
<none>                    <none>              a4ed6a02234b        About an hour ago   122 MB
ubuntu                    16.04               2fa927b5cdd3        27 hours ago        122 MB
armv7/armhf-ubuntu_core   latest              2c357551076b        7 months ago        109.5 MB

The Linux version message is:

Welcome to Ubuntu 16.04 LTS (GNU/Linux 3.14.29-56 aarch64)

This is my first attempt to venture into this type of stuff, so I am rather green. I wonder if I am missing some additional configuration settings... Your help is appreciated.

[teleki]

@imikushin: update: Successfully built rancher/agent and rancher/agent-instance from the respective PRs. However, rancher/server still has the same error as mentioned above.

[alphawolf1988]

@imikushin Succesfully built rancher/agent and rancher/server.

@teleki You require to:

1. git clone https://github.com/imikushin/s6-builder
2. ./build.arm.sh

Then you should have the required s6-builder docker image :) and everything should build just fine.
I can run the rancher/server on my Raspberry Pi 3, but it takes hours to start. One cpu is up to 100% all the time (java process), while starting.

@imikushin Could it be, that the process still uses OpenJDK instead of OracleJDK?

[teleki]

@imikushin Thank you for the help; it is much appreciated. I have successfully built s6-builder.

However, now I am getting a different error as I am trying to build rancher/server. The error says that there is no "/usr/bin/java" which is probably true, since based on the logs, it seems that Java is in "/usr/lib/jvm". Here are the relevant snippets of the output:

....
Connecting to download.oracle.com (download.oracle.com)|204.237.184.213|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 78321279 (75M) [application/x-gzip]
Saving to: 'jdk-8u91-linux-arm64-vfp-hflt.tar.gz'
...
2016-06-01 02:48:01 (1.51 MB/s) - 'jdk-8u91-linux-arm64-vfp-hflt.tar.gz' saved [78321279/78321279]

Download done.
Removing outdated cached downloads...
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/java to provide /usr/bin/java (java) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/jjs to provide /usr/bin/jjs (jjs) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/keytool to provide /usr/bin/keytool (keytool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/orbd to provide /usr/bin/orbd (orbd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/pack200 to provide /usr/bin/pack200 (pack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/policytool to provide /usr/bin/policytool (policytool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/rmid to provide /usr/bin/rmid (rmid) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/rmiregistry to provide /usr/bin/rmiregistry (rmiregistry) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/servertool to provide /usr/bin/servertool (servertool) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/tnameserv to provide /usr/bin/tnameserv (tnameserv) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/bin/unpack200 to provide /usr/bin/unpack200 (unpack200) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/jre/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/appletviewer to provide /usr/bin/appletviewer (appletviewer) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/extcheck to provide /usr/bin/extcheck (extcheck) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/idlj to provide /usr/bin/idlj (idlj) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jar to provide /usr/bin/jar (jar) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jarsigner to provide /usr/bin/jarsigner (jarsigner) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javac to provide /usr/bin/javac (javac) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javadoc to provide /usr/bin/javadoc (javadoc) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javah to provide /usr/bin/javah (javah) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/javap to provide /usr/bin/javap (javap) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jcmd to provide /usr/bin/jcmd (jcmd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jconsole to provide /usr/bin/jconsole (jconsole) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jdb to provide /usr/bin/jdb (jdb) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jdeps to provide /usr/bin/jdeps (jdeps) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jhat to provide /usr/bin/jhat (jhat) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jinfo to provide /usr/bin/jinfo (jinfo) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jmap to provide /usr/bin/jmap (jmap) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jps to provide /usr/bin/jps (jps) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jrunscript to provide /usr/bin/jrunscript (jrunscript) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jsadebugd to provide /usr/bin/jsadebugd (jsadebugd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jstack to provide /usr/bin/jstack (jstack) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jstat to provide /usr/bin/jstat (jstat) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/jstatd to provide /usr/bin/jstatd (jstatd) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/native2ascii to provide /usr/bin/native2ascii (native2ascii) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/rmic to provide /usr/bin/rmic (rmic) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/schemagen to provide /usr/bin/schemagen (schemagen) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/serialver to provide /usr/bin/serialver (serialver) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/wsgen to provide /usr/bin/wsgen (wsgen) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/wsimport to provide /usr/bin/wsimport (wsimport) in auto mode
update-alternatives: using /usr/lib/jvm/java-8-oracle/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode
Oracle JDK 8 installed

...
 ---> 7dd5b4837334
Removing intermediate container d1461dd50c46
Step 33 : RUN cd / && mkdir -p $CATTLE_HOME && /usr/share/cattle/cattle.sh extract
 ---> Running in 437df28b035c
/usr/share/cattle/cattle.sh: line 197: /usr/bin/java: No such file or directory
The command '/bin/sh -c cd / && mkdir -p $CATTLE_HOME && /usr/share/cattle/cattle.sh extract' returned a non-zero code: 1
+ docker rm -fv bd1ba640d63ce3d8
bd1ba640d63ce3d8

Apparently somehow java is not in the container. However, if I look at the log is seems to show that Oracle Java was installed.

Any ideas where I'm going wrong?

[alphawolf1988]

@teleki Could you show us your current used server Dockerfile.arm and are you using the build-image.arm.sh for building?

[teleki]

@alphawolf1988 The Dockerfile.arm is the one I got from PR 4704 referenced a few messages above by @imikushin. I only have added one file, ".docker-env.arm", because it was not in the original code, so I got that from here: https://github.com/rancher/os-images/blob/master/.docker-env.arm.sample but changed the IP to my machine's IP.

Here is the my rancher/server/Dockerfile.arm:

FROM armhf/ubuntu:16.04

ENV TERM xterm
RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends openjdk-8-jre-headless \
                    curl \
                    mysql-server \
                    xz-utils \
                    keychain \
                    unzip \
                    openssh-client \
                    iptables \
                    iproute2 \
                    git \
                    redis-server \
                    zookeeper \
                    spiped
RUN mkdir -p /var/run/mysqld && chmod -R 777 /var/run/mysqld

RUN apt-get install -y --no-install-recommends software-properties-common
RUN add-apt-repository -y ppa:webupd8team/java && apt-get update
RUN echo 'yes' | DEBIAN_FRONTEND=Teletype apt-get install -y --no-install-recommends oracle-java8-installer

ADD https://github.com/imikushin/giddyup/releases/download/v0.10.1_arm/giddyup /usr/bin/
ADD https://github.com/imikushin/cluster-manager/releases/download/v0.1.4_arm/cluster-manager /usr/bin/
RUN chmod +x /usr/bin/giddyup /usr/bin/cluster-manager
COPY bin/* /usr/bin/

ENV CATTLE_HOME /var/lib/cattle
ENV DEFAULT_CATTLE_API_UI_INDEX //releases.rancher.com/ui/1.0.5
ENV CATTLE_DB_CATTLE_DATABASE mysql
ENV CATTLE_USE_LOCAL_ARTIFACTS true
ENV no_proxy localhost,127.0.0.1,localaddress,.localdomain.com
ADD artifacts /usr/share/cattle

ADD service /service
ENV S6_SERVICE_DIR /service

COPY target/arm/*static.tar.gz /s6-statics/

EXPOSE 8080
ENV CATTLE_HOST_API_PROXY_MODE embedded
ENV CATTLE_RANCHER_SERVER_IMAGE v1.0.1_arm
ENV CATTLE_RANCHER_COMPOSE_VERSION v0.7.4
ENV DEFAULT_CATTLE_RANCHER_COMPOSE_LINUX_URL   https://releases.rancher.com/compose/${CATTLE_RANCHER_COMPOSE_VERSION}/rancher-compose-linux-amd64-${CATTLE_RANCHER_COMPOSE_VERSION}.tar.gz
ENV DEFAULT_CATTLE_RANCHER_COMPOSE_DARWIN_URL  https://releases.rancher.com/compose/${CATTLE_RANCHER_COMPOSE_VERSION}/rancher-compose-darwin-amd64-${CATTLE_RANCHER_COMPOSE_VERSION}.tar.gz
ENV DEFAULT_CATTLE_RANCHER_COMPOSE_WINDOWS_URL https://releases.rancher.com/compose/${CATTLE_RANCHER_COMPOSE_VERSION}/rancher-compose-windows-386-${CATTLE_RANCHER_COMPOSE_VERSION}.zip
ENV DEFAULT_CATTLE_CATALOG_URL="library=https://github.com/rancher/rancher-catalog.git,community=https://github.com/rancher/community-catalog.git"

EXPOSE 3306
ENV CATTLE_CATTLE_VERSION v0.159.7-arm-4
ADD https://github.com/imikushin/cattle/releases/download/${CATTLE_CATTLE_VERSION}/cattle.jar /usr/share/cattle/

RUN cd /usr && for i in $(ls /s6-statics/*static.tar.gz);do tar -zxvf $i;done && rm -rf /s6-statics/*static.tar.gz
RUN cd / && mkdir -p $CATTLE_HOME && /usr/share/cattle/cattle.sh extract
RUN curl -sLf https:${DEFAULT_CATTLE_API_UI_INDEX}.tar.gz | tar xvzf - -C /usr/share/cattle/war --strip-components=1
RUN /usr/share/cattle/install_cattle_binaries && \
    cd $CATTLE_HOME && export IFS="," && \
    for i in $DEFAULT_CATTLE_CATALOG_URL; do rancher-catalog-service -validate -catalogUrl=$i;done

VOLUME /var/lib/mysql /var/log/mysql /var/lib/cattle

ENV DEFAULT_CATTLE_MACHINE_EXECUTE true
ENV DEFAULT_CATTLE_COMPOSE_EXECUTOR_EXECUTE true
ENV DEFAULT_CATTLE_CATALOG_EXECUTE true

CMD ["/usr/bin/s6-svscan", "/service"]

Thank you!

[alphawolf1988]

@teleki please try:

FROM armhf/ubuntu:16.04

ENV TERM xterm

RUN apt-get update && apt-get install -y --no-install-recommends software-properties-common
RUN add-apt-repository -y ppa:webupd8team/java && apt-get update
RUN echo 'yes' | DEBIAN_FRONTEND=Teletype apt-get install -y --no-install-recommends oracle-java8-installer

RUN java -version

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
                    curl \
                    mysql-server \
                    xz-utils \
                    keychain \
                    unzip \
                    openssh-client \
                    iptables \
                    iproute2 \
                    git \
                    redis-server \
                    zookeeper \
                    spiped
RUN mkdir -p /var/run/mysqld && chmod -R 777 /var/run/mysqld

ADD https://github.com/imikushin/giddyup/releases/download/v0.10.1_arm/giddyup /usr/bin/
ADD https://github.com/imikushin/cluster-manager/releases/download/v0.1.4_arm/cluster-manager /usr/bin/
RUN chmod +x /usr/bin/giddyup /usr/bin/cluster-manager
COPY bin/* /usr/bin/

ENV CATTLE_HOME /var/lib/cattle
ENV DEFAULT_CATTLE_API_UI_INDEX //releases.rancher.com/ui/1.0.5
ENV CATTLE_DB_CATTLE_DATABASE mysql
ENV CATTLE_USE_LOCAL_ARTIFACTS true
ENV no_proxy localhost,127.0.0.1,localaddress,.localdomain.com
ADD artifacts /usr/share/cattle

ADD service /service
ENV S6_SERVICE_DIR /service

COPY target/arm/*static.tar.gz /s6-statics/

EXPOSE 8080
ENV CATTLE_HOST_API_PROXY_MODE embedded
ENV CATTLE_RANCHER_SERVER_IMAGE v1.0.1_arm
ENV CATTLE_RANCHER_COMPOSE_VERSION v0.7.4
ENV DEFAULT_CATTLE_RANCHER_COMPOSE_LINUX_URL   https://releases.rancher.com/compose/${CATTLE_RANCHER_COMPOSE_VERSION}/rancher-compose-linux-amd64-${CATTLE_RANCHER_COMPOSE_VERSION}.tar.gz
ENV DEFAULT_CATTLE_RANCHER_COMPOSE_DARWIN_URL  https://releases.rancher.com/compose/${CATTLE_RANCHER_COMPOSE_VERSION}/rancher-compose-darwin-amd64-${CATTLE_RANCHER_COMPOSE_VERSION}.tar.gz
ENV DEFAULT_CATTLE_RANCHER_COMPOSE_WINDOWS_URL https://releases.rancher.com/compose/${CATTLE_RANCHER_COMPOSE_VERSION}/rancher-compose-windows-386-${CATTLE_RANCHER_COMPOSE_VERSION}.zip
ENV DEFAULT_CATTLE_CATALOG_URL="library=https://github.com/rancher/rancher-catalog.git,community=https://github.com/rancher/community-catalog.git"

EXPOSE 3306
ENV CATTLE_CATTLE_VERSION v0.159.7-arm-4
ADD https://github.com/imikushin/cattle/releases/download/${CATTLE_CATTLE_VERSION}/cattle.jar /usr/share/cattle/

RUN cd /usr && for i in $(ls /s6-statics/*static.tar.gz);do tar -zxvf $i;done && rm -rf /s6-statics/*static.tar.gz
RUN cd / && mkdir -p $CATTLE_HOME && /usr/share/cattle/cattle.sh extract
RUN curl -sLf https:${DEFAULT_CATTLE_API_UI_INDEX}.tar.gz | tar xvzf - -C /usr/share/cattle/war --strip-components=1
RUN /usr/share/cattle/install_cattle_binaries && \
    cd $CATTLE_HOME && export IFS="," && \
    for i in $DEFAULT_CATTLE_CATALOG_URL; do rancher-catalog-service -validate -catalogUrl=$i;done

VOLUME /var/lib/mysql /var/log/mysql /var/lib/cattle

ENV DEFAULT_CATTLE_MACHINE_EXECUTE true
ENV DEFAULT_CATTLE_COMPOSE_EXECUTOR_EXECUTE true
ENV DEFAULT_CATTLE_CATALOG_EXECUTE true

CMD ["/usr/bin/s6-svscan", "/service"]

[alphawolf1988]

Successfully registered agent :)

[teleki]

@alphawolf1988 Thank you for the help. Still no love. Here is what I get when I run the build-image.arm.sh:

...
Oracle JDK 8 installed
 ---> 2bbfa3297100
Removing intermediate container 4784336319ff
Step 6 : RUN java -version
 ---> Running in b2a6a91defee
/bin/sh: 1: java: not found
The command '/bin/sh -c java -version' returned a non-zero code: 127

As I was Googling around, some messages were pointing me to think that the container may have a mixture of 32-bit and 64-bit libraries somehow. However, I am unable to assess if this is true or not.

I logged into the container:

docker run -ti 2bbfa3297100 /bin/bash
root@7c150d68f7f9:/# java -version
bash: /usr/bin/java: No such file or directory

Then I navigated to the Java directory. Here is what I got there:

root@7c150d68f7f9:/# cd /usr/lib/jvm/java-8-oracle/bin/
root@7c150d68f7f9:/usr/lib/jvm/java-8-oracle/bin# ls -la java*
-rwxr-xr-x 1 root root 7791 Jun  2 02:27 java
-rwxr-xr-x 1 root root 1809 Jun  2 02:27 java-rmi.cgi
-rwxr-xr-x 1 root root 8006 Jun  2 02:27 javac
-rwxr-xr-x 1 root root 8014 Jun  2 02:27 javadoc
-rwxr-xr-x 1 root root 8006 Jun  2 02:27 javah
-rwxr-xr-x 1 root root 8006 Jun  2 02:27 javap
root@7c150d68f7f9:/usr/lib/jvm/java-8-oracle/bin# ./java
bash: ./java: No such file or directory

Here is the puzzle: I see the file, it is there and when I run "file" on it, this what I get:

root@7c150d68f7f9:/usr/lib/jvm/java-8-oracle/bin# file ./java
./java: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=c46954c0ae919fe0f71b35f6716d88a811eaa92e, not stripped

Any more ideas?

[alphawolf1988]

@teleki Are you using a raspberry pi 3 or 2 and what OS distribution (and version) is used ?

[teleki]

@alphawolf1988 I am attempting to build a 4 unit ODROID C2 cluster. The specs are:

• Amlogic S905 ARM® Cortex®-A53 (ARMv8) 2GHz quad core CPU
• 2GB DDR3 SDRAM
• 32 GB eMMC 5.0 HS400 Flash Storage
• ARM Mali™-450 MP3 GPU (OpenGL ES 2.0/1.1 for Linux)
• HDMI 2.0 4K/60Hz display
• H.265 4K/60FPS and H.264 4K/30FPS capable VPU
• Gigabit Ethernet
• Ubuntu 16.04 LTS based on Kernel 3.14 LTS (GNU/Linux 3.14.65-66 aarch64)

[teleki]

@alphawolf1988 @imikushin unfortunately, still no love. I started from scratch again. It stopped with the following error:

...
Step 35 : RUN cd / && mkdir -p $CATTLE_HOME && /usr/share/cattle/cattle.sh extract
 ---> Running in 03035cafc513
/usr/share/cattle/cattle.sh: line 197: /usr/bin/java: No such file or directory
The command '/bin/sh -c cd / && mkdir -p $CATTLE_HOME && /usr/share/cattle/cattle.sh extract' returned a non-zero code: 1

Attached the entire output. build-image.arm.out.txt

Also attached the Dockerfile.arm file. Dockerfile.arm.txt

It seems that the problem is that java won't run, possibly due to a mixture of 64 bit and 32 bit code.
Any more ideas? Thanks in advance!

[lavvy]

@teleki did you manage to get it run on the C2

[lavvy]

Or trying other OS like Debian

[teleki]

@lavvy not yet. I am stuck for now. I have only tried it with the default OS that came on C2.

[faddat]

This comment comes from a sadly underinformed place, but overall I've got
to observe that it seems that Java is the enemy of this quest.

Jacob Gadikian
E-mail: [email protected]
SKYPE: faddat
Phone/SMS: [phone]

On Tue, Jun 14, 2016 at 9:36 PM, teleki [email protected] wrote:

@lavvy https://github.com/lavvy not yet. I am stuck for now. I have
only tried it with the default OS that came on C2.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/4703#issuecomment-225901581,
or mute the thread
https://github.com/notifications/unsubscribe/AGz6iSZvdFvcmF7oRrJSK3OZGI4UmanSks5qLrx6gaJpZM4IYe7D
.

[teleki]

@faddat I'm afraid you are right. I installed Oracle Java directly onto the C2 and that worked fine. For some reason it doesn't work the way it is installed in the container. So far I have not had a chance to figure it out why.

[withinboredom]

For anyone who wants to give this a go... I have prebuilt images and instructions on my blog ... even with the incredible work that's been done so far, there's still a fair bit of manual work to bring up a node -- I didn't build rancher/server, just the things to run rancher/agent on arm while rancher/server is on amd64.

[faddat]

Have you attempted the Ur-Power heterogeneous cluster yet? Clearly on the
server/agent level you have, but how might we make it work with some x86
agents and some ARM agents?

I'm thinking labels and careful rules, maybe? (obviously anything using
ARM at this stage is a ways from production ready, but... I think
specialized compute is one of few lifelines Moore's Law has remaining.

:)

-Jake

Jacob Gadikian
E-mail: [email protected]
SKYPE: faddat
Phone/SMS: [phone]

On Sun, Jun 26, 2016 at 3:06 AM, Rob Landers [email protected]
wrote:

For anyone who wants to give this a go... I have prebuilt images and
instructions on my blog
https://www.withinboredom.info/post/rancher-on-pi/ ... even with the
incredible work that's been done so far, there's still a fair bit of manual
work to bring up a node -- I didn't build rancher/server, just the things
to run rancher/agent on arm while rancher/server is on amd64.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rancher/rancher/issues/4703#issuecomment-228568071,
or mute the thread
https://github.com/notifications/unsubscribe/AGz6iSoNilX1Vc81w1J3cY15mu79629wks5qPYo4gaJpZM4IYe7D
.

[withinboredom]

@faddat Eventually (one day) rancher might read the metadata on the image and see what architecture it's built for and only deploy to the architecture its built for ;) Docker is kinda already working on that where pulling ubuntu:16.04 on x86 will pull the appropriate image, and will pull the equivelant on an arm architecture.

Until then, complex rules will be the norm in heterogeneous clusters.

Actually, it would be super spiffy for rancher to implement the manifest thing currently, it would probably save a headache of changing a bunch of code. Specialized utils could exist in a container instead of downloaded from the rancher server....

[lavvy]

+1

[lavvy]

what is the progress of this issue, is there any other place we can be tracking it from, or is it in a little halt.
what about the agent , can't we be able to add an arm node at least, before the server comes.

[jgato]

following @withinboredom blog I am trying to run add a Raspberry Pi 2 as a host, but it is failing to me

INFO: ENV: RANCHER_AGENT_IMAGE=rancher/agent:v1.0.2
Template parsing error: template: :1:8: executing "" at <.Volumes>: map has no entry for key "Volumes"
dirname: missing operand
Try 'dirname --help' for more information.
dirname: missing operand
Try 'dirname --help' for more information.
dirname: missing operand
Try 'dirname --help' for more information.
docker: Error response from daemon: Invalid volume spec ":": volumeinvalid: Invalid volume specification: ':'.
See 'docker run --help'. 

Which distribution do you suggest me to use in the raspberry pi?

[zerodivide1]

Hi, I'm having the same issue that @jgato is having. I am using Hypriot as the host OS:

pirate@black-pearl:~$ uname -a; docker version
Linux black-pearl 4.4.15-hypriotos-v7+ #1 SMP PREEMPT Mon Jul 25 08:46:52 UTC 2016 armv7l GNU/Linux
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:31:15 2016
 OS/Arch:      linux/arm

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:31:15 2016
 OS/Arch:      linux/arm

[zerodivide1]

Doing a little digging and I found the line causing the error message @jgato and I were seeing:

https://github.com/rancher/rancher/blob/207b67ef6d5ff91db0e6dbde33a82a1c0288eb44/agent/run.sh#L193

When I run the command, I see the same error, but when I run the following, it seems to return fine:

pirate@black-pearl:~$ echo $(dirname $(dirname $(dirname $(docker inspect -f '{{index .Config.Volumes "/var/lib/cattle"}}' rancher-agent-state))))
.

[jgato]

@zerodivide1 how could I try it? Do I need to build my own agent images changing this line?

[jerryz1982]

@jgato the problem with the arm image is that the docker client in it is a new version which has a different output format than the old version which run.sh is compatible with.
there are other issues even with the work around by @zerodivide1
https://www.adelton.com/docs/docker/docker-inspect-volumes-mounts#zero-value

The pulling request to build arm agent images is still in progress.

[natami]

Any progress in this task?

[tritao]

Same question as @natami. Any status to when we can expect to get working Rancher ARM agent binaries?

[fredVT]

@teleki : did you finally succeed to install rancher (server & agent) on your odroid-c2 cluster ? I would like to do the same thing, so I'm interested...

[truedat101]

@mflor35 and I are also trying this with a PicoCluster running PINE64 boards. We didn't discover this issue until after we started. I'll post my results here.

[teleki]

@fredVT Sorry. I haven't had a chance to get back to this since last year, so no progress on it.

[solarlancer]

Yes, Rancher ARM agent is highly appreciated. Lots of RPi 3 idling ;-)

[kingprimex]

I subscribe on this issue as well.

[andrewscul]

I am new to Rancher and trying to test things on a Pi cluster therefore I am also looking for the rancher server docker image for Pi 3 in order to run an independent Pi cluster so that it is mobile and not dependent on a linux VM running the server. It sounds like it would be good to have a lightweight version of rancher server which does not overload the Pi 3.

[mvbjrn]

I managed to execute the rancher/agent:v1.2.2 image on a Raspberry Pi but ran into the next problem. The server runs on an AMD64 machine and it seems like the image is downloading the (amd64-) agent from there!? Is it even possible to have an amd64-server and arm-hosts conceptionalwise?

This is the output of docker logs of the container:

INFO: Starting agent for DCA8E1DACFF85C6C62F2
INFO: Access Key: DCA8E1DACFF85C6C62F2
INFO: Config URL: http://rancher-server:8080/v1
INFO: Storage URL: http://rancher-server:8080/v1
INFO: API URL: http://rancher-server:8080/v1
INFO: IP: 192.168.178.63
INFO: Port:
INFO: Required Image: rancher/agent:v1.2.2
INFO: Current Image: rancher/agent:v1.2.2
INFO: Using image rancher/agent:v1.2.2
INFO: Downloading agent http://rancher-server:8080/v1/configcontent/configscripts
INFO: Updating configscripts
INFO: Downloading http://rancher-server:8080/v1//configcontent//configscripts current=
INFO: Running /var/lib/cattle/download/configscripts/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/apply.sh
INFO: Sending configscripts applied 1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319
INFO: Updating pyagent
INFO: Downloading http://rancher-server:8080/v1//configcontent//pyagent current=
INFO: Running /var/lib/cattle/download/pyagent/pyagent-1-9e3dcab12585149a65d818082279708965cd826df1937a55b24f48428f8f0c0644fc1033b5be15c88dcf9540b1c04c67/apply.sh
INFO: Updating host-api
INFO: Downloading http://rancher-server:8080/v1//configcontent//host-api current=1-9608ead804f8cebbfe95b9f114701d8a5ac4244d603d196068043c902b1a0bdf
INFO: Already up to date
INFO: Sending host-api applied 1-9608ead804f8cebbfe95b9f114701d8a5ac4244d603d196068043c902b1a0bdf
INFO: Updating python-agent
INFO: Downloading http://rancher-server:8080/v1//configcontent//python-agent current=1-ba59beaa71a9d74e52708cff1beb0a6fd138b11c050cbdd33def3e7391bd696f
INFO: Already up to date
INFO: Sending python-agent applied 1-ba59beaa71a9d74e52708cff1beb0a6fd138b11c050cbdd33def3e7391bd696f
INFO: Sending pyagent applied 1-9e3dcab12585149a65d818082279708965cd826df1937a55b24f48428f8f0c0644fc1033b5be15c88dcf9540b1c04c67
INFO: Starting agent /var/lib/cattle/pyagent/apply.sh
INFO: Executing /var/lib/cattle/pyagent/agent
INFO: Updating host-config
INFO: Downloading http://rancher-server:8080/v1//configcontent//host-config current=host-config-1-0501df8804d09befe0190816322288c7413dfbd9631634a133aa87c2ae2fccc4
INFO: Already up to date
INFO: Sending host-config applied 1-0501df8804d09befe0190816322288c7413dfbd9631634a133aa87c2ae2fccc4
/var/lib/cattle/pyagent/apply.sh: line 82: /var/lib/cattle/pyagent/agent: cannot execute binary file: Exec format error
/var/lib/cattle/pyagent/apply.sh: line 82: /var/lib/cattle/pyagent/agent: Success

These are the modified Dockerfiles of
agent-base:

# VERSION               0.3.1

FROM armv7/armhf-ubuntu:16.04
LABEL Version="0.3.1"

COPY ./rancher-entrypoint.sh /

ENV SSL_SCRIPT_COMMIT 98660ada3d800f653fc1f105771b5173f9d1a019

RUN apt-get update && \
        apt-get install --no-install-recommends -y \
        arptables \
        bridge-utils \
        ca-certificates \
        conntrack \
        curl \
        ethtool \
        git \
        iproute2 \
        iptables \
        iputils-ping \
        jq \
        kmod \
        openssl \
        passwd \
        pkg-config \
        psmisc \
        python2.7 \
        tcpdump \
        vim-tiny

# install cattle
RUN rm -rf /var/lib/apt/lists/*
RUN mkdir -p /var/lib/cattle /var/lib/rancher
RUN ln -s /usr/bin/python2.7 /usr/bin/python
RUN curl -s https://bootstrap.pypa.io/get-pip.py | python2.7
RUN pip install cattle

# add the pre-compiled share-mnt to the image
ADD share-mnt /usr/bin/share-mnt
RUN  chmod +x /usr/bin/share-mnt

# setup rancher/weave
RUN curl -sL https://github.com/rancher/weave/releases/download/r-v0.0.4/r > /usr/bin/r
RUN chmod +x /usr/bin/r

# setup docker arm 17.03.0-ce
RUN curl -fsSLO https://get.docker.com/builds/Linux/armel/docker-17.03.0-ce.tgz && tar --strip-components=1 -xvzf docker-17.03.0-ce.tgz -C /usr/bin/
RUN chmod +x /usr/bin/docker

RUN rm /var/run
RUN mkdir /var/run
RUN curl -sLf https://raw.githubusercontent.com/rancher/rancher/${SSL_SCRIPT_COMMIT}/server/bin/update-rancher-ssl > /usr/bin/update-rancher-ssl
RUN chmod +x /usr/bin/update-rancher-ssl

ENTRYPOINT ["/rancher-entrypoint.sh"]

and agent:

FROM rancher/agent-base:v0.3.1
COPY register.py resolve_url.py run.sh /

RUN apt-get update && \
        apt-get install --no-install-recommends -y \
    build-essential \
    git \
    wget

RUN wget https://storage.googleapis.com/golang/go1.7.5.linux-armv6l.tar.gz
RUN tar -xvf go1.7.5.linux-armv6l.tar.gz -C /usr/local
ENV PATH /usr/local/go/bin:$PATH
RUN mkdir /go
ENV GOROOT /usr/local/go
ENV GOPATH /go
ENV PATH $GOPATH/bin:$GOROOT/bin:$PATH
RUN go get github.com/rancher/agent
RUN cd $GOPATH/src/github.com/rancher/agent && go build && go install

ENTRYPOINT ["/run.sh"]
LABEL "io.rancher.container.system"="rancher-agent"
ENV RANCHER_AGENT_IMAGE rancher/agent:v1.2.2

I am not sure where to put the go build agent from the last image to not trigger the download from the server.

if you want to try this out, you have to compile share-mnt on your Raspberry Pi. Just check out the rancher/runc repository, switch to the jazzhands branch and compile.

[brunograz]

We have been doing some work on bringing newer versions of rancher-agent to aarch64 systems, it is still quite experimental but has been tested in a Cattle environment with rancher server 1.5.x in a raspberry pi 3.

For anyone interested to give this a try, these images and instructions to set it up are available on dockerhub.

[solarlancer]

@brunograz thanks for your information! i just started to do some research how to build the newer agent microservices on arm platform and it's not completely clear to me how you did the docker image build for the aarch64 microservices (metadata, dns, healthcheck, etc.).
i assume you created a docker image for the aarch64 dapper build wrapper, cloned the microservices source repos, added a Dockerfile.dapper for your platform and used the dapper image to build each of the services.
am i pointing to the right direction?

[dominictayloruk]

@brunograz Any links to the sources, i could do with armhf. Asus Tinkerboard.

[brunograz]

@LarsVader666 Sorry for the late reply. The approach I followed might not be the best practice for this case - as I did not use any of the dapper stuff - but it involved building binaries and adding them directly to the Dockerfile for each of the services (modifications in the Dockerfile were also required to make sure it is compatible with the aarch64 architecture).

• First I built the agent-base container (the base container for most of the services)

• Then you can build the agent container and the rest of the services. (healthcheck, scheduler, network-manager, dns, metadata, net...)

The biggest issue with dapper imo is that it does not download the proper binaries inside the Dockerfile of each service. I might be wrong about this, just don't have enough experience with it.

EDIT: I meant here services that download binaries within the Dockerfile. e.g. agent-base container.

@dominictayloruk We don't have the source code published yet (my bad!) but will do some cleaning in code and publish it sometime soon.

EDIT2: I put together a repo with the code we used to build those images. The Dockerfiles are in the 'package' folder - after building the binaries I also was placing them there.

[CodingSpiderFox]

What's the status?

[ulm0]

Once the images are built, manifest-tool and/or manifest-list could be, so it'll download the image based on the architecture

[CodingSpiderFox]

I think some images have already been built for arm but it seems like the list in the first post has not been updated, for example:

host-api [rancher/host-api#77]

There also still is at least one open pull request about the issue and I wonder why this hasn't been merged yet:

agent-instance [rancher/agent-instance#18]

[cdknudsen8]

anyone have a copy of the docker-env.arm sample? it is not available at the link above anymore and can't seem to find one. thx

[blackholegalaxy]

Any news on that? ARM support, with IoT world growing everyday, would be a killer feature for Rancher. Imagine upgrading images on dozens of Raspberries agents... With Kubernetes supporting ARM64 natively, I think it could be a quick win for Rancher 2.0

[jmreicha]

Anybody had any luck getting rancher-server/rancher-agent containers to work on Rancher 2.0?

[ags131]

@jmreicha I have, its actually fairly easy to do, there is a gotcha in that you have to use the View in API option to change a cluster setting in Rancher. https://gist.github.com/ags131/7bdde11c932ef7a54f44c6decbfd88b8

[blackholegalaxy]

@deniseschannon @smw355 could this be reopened?

[soupli]

deniseschannon closed this 8 hours ago

@deniseschannon So we can state this is never going to happen, or...?

[blackholegalaxy]

No it's an automatic close