Trident: Trident image for Linux on ppc64le

[arsenalzp]

Hello,
I'm unable to use Trident on RHEL 7.5 ppc64le.
Docker image was builded well:

$docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
trident 18.10.0-custom ef12a3f561db 2 hours ago 164 MB

$file bin/tridentctl
bin/tridentctl: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 3.2.0, BuildID[sha1]=122143f6151328a8e8fab4d93b213633e0eafa64, not stripped

$ file bin/trident_orchestrator
bin/trident_orchestrator: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 3.2.0, BuildID[sha1]=9074f26f64a3cf3f0ffcf25e3dd21b7957ca612a, not stripped

Try to start:

$kubectl run trident --image=trident:18.10.0-custom

$ kubectl get pod
NAME READY STATUS RESTARTS AGE
trident-4031422826-65odh 0/1 RunContainerError 1 16s

Checking logs:

$ kubectl logs trident-4031422826-65odh
container_linux.go:247: starting container process caused "exec: \"/usr/bin/env -i PATH='/netapp:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' /usr/local/bin/$BIN -port $PORT -etcd_v3 $ETCDV3 -k8s_api_server $K8S\": stat /usr/bin/env -i PATH='/netapp:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' /usr/local/bin/$BIN -port $PORT -etcd_v3 $ETCDV3 -k8s_api_server $K8S: no such file or directory"

Something went wrong...
Could somebody help me ?

My environment:

$ docker version
Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-74.git6e3bb8e.el7.ppc64le
Go version: go1.9.2
Git commit: 6e3bb8e/1.13.1
Built: Wed Aug 1 17:04:05 2018
OS/Arch: linux/ppc64le

Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-74.git6e3bb8e.el7.ppc64le
Go version: go1.9.2
Git commit: 6e3bb8e/1.13.1
Built: Wed Aug 1 17:04:05 2018
OS/Arch: linux/ppc64le
Experimental: false

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"1.4.7", GitCommit:"92b4f971662de9d8770f8dcd2ee01ec226a6f6c0", GitTreeState:"clean", BuildDate:"2017-02-07T17:25:56Z", GoVersion:"go1.7.5 ibm-1.7", Compiler:"gc", Platform:"linux/ppc64le"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"1.4.7", GitCommit:"92b4f971662de9d8770f8dcd2ee01ec226a6f6c0", GitTreeState:"clean", BuildDate:"2017-02-07T17:25:56Z", GoVersion:"go1.7.5 ibm-1.7", Compiler:"gc", Platform:"linux/ppc64le"}

[ntap-rippy]

The Trident container is based on x86-64 alpine:
https://github.com/NetApp/trident/blob/0f168eef71dd978df90130fb19a83f35add54c60/Dockerfile#L1

Running file against the binaries inside shows x86-64

$ docker run --rm -it netapp/trident:18.10.0 ash -c 'apk add file; file /usr/bin/coreutils'
fetch http://dl-1.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
...
(1/2) Installing libmagic (5.32-r0)
(2/2) Installing file (5.32-r0)
Executing busybox-1.28.4-r1.trigger
OK: 27 MiB in 44 packages
/usr/bin/coreutils: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, stripped

[arsenalzp]

@ntap-rippy, thank for response!
Does it mean I can't build and use Trident on Linux ppc64le in no way ?

[adkerr]

@arsenalzp Since you're building a custom version anyways, you can try adjusting the Dockerfile to use a PPC based image, however that may also require changing the libraries that are installed into the image as well, but keep in mind you're venturing into unsupported and untested territory here

[adkerr]

With that said, if the x86 container will run you may not even need a PPC compiled trident since it's fully containerized. But again, we have done no testing with PPC hosts.

[kangarlou]

I don't think an x86 binary or Docker image can run on PPC. Docker doesn't provide any sort of virtualization/binary translation for you. You have to build your own Trident image on PPC. Also, you shouldn't be running Trident v18.10 with k8s 1.4. k8s 1.4 was supported by Trident v1.0, which was released more than 2 years ago.

[arsenalzp]

@kangarlou yes, I know. I tried to clone Trident repo and then build from source. But something went wrong with Docker image.

[arsenalzp]

So, I changed _Dockerfile_ and updated k8s to v1.11.1+icp ( IBM Cloud Private-ce )

was:
FROM alpine:3.8

became:
FROM ppc64le/alpine:3.8

Also, I add comment to the following line:

RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2

Try to build with _make trident_build_

Seems, everything is OK:

Successfully built 20317c166482

In result, two binaries were created and placed into bin/ directory:

$file bin/*
bin/tridentctl: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 3.2.0, BuildID[sha1]=b0351a7ff7a31003c9d288ac85176e5d0445e833, not stripped
bin/trident_orchestrator: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 3.2.0, BuildID[sha1]=33006488377a4856e39e188d92531ca6e563e8ab, not stripped

Docker image was created also:

$ docker images|grep trident
trident 18.10.0-custom 6cdbb24d162e About a minute ago 165 MB

Try to start:

$kubectl run trident --image=trident:18.10.0-custom

And I'm getting the same error:

$ kubectl logs trident-5b49fcff86-xgz2p
container_linux.go:247: starting container process caused "exec: \"/usr/bin/env -i PATH='/netapp:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' /usr/local/bin/$BIN -port $PORT -etcd_v3 $ETCDV3 -k8s_api_server $K8S\": stat /usr/bin/env -i PATH='/netapp:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' /usr/local/bin/$BIN -port $PORT -etcd_v3 $ETCDV3 -k8s_api_server $K8S: no such file or directory"

Which directive I should add to Dockerfile to get success ?
P.S.
That's not obviously which file/directory does not exist...

Dockerfile.txt

[kangarlou]

You can't just do kubectl run to install Trident. Installing Trident entails installing a service account, ClusterRoles, ClusterRoleBindings, etc, which happen when you run tridentctl install (see the documentation). This means you should use the installer bundle (the .tar.gz file generated as part of the build) to install Trident.

[arsenalzp]

@kangarlou
Yes, I read Docs and I tried to run _tridentctl install_
Here is results:

$ bin/tridentctl install --dry-run -n default
INFO Created installer service account. serviceaccount=trident-installer
INFO Waiting for object to be created. objectName=clusterRole
INFO Created installer cluster role. clusterrole=trident-installer
INFO Waiting for object to be created. objectName=clusterRoleBinding
INFO Created installer cluster role binding. clusterrolebinding=trident-installer
INFO Created installer configmap. configmap=trident-installer
INFO Waiting for object to be created. objectName=installerPod
INFO Created installer pod. pod=trident-installer
INFO Waiting for Trident installer pod to start.

$ kubectl get pod
NAME READY STATUS RESTARTS AGE
trident-installer 0/1 ErrImagePull 0 36s

$ kubectl describe pod trident-installer
Name: trident-installer
Namespace: default
Priority: 0
PriorityClassName:
Node: 10.1.2.125/10.1.2.125
Start Time: Thu, 10 Jan 2019 18:59:22 +0200
Labels: app=trident-installer.netapp.io
Annotations: kubernetes.io/psp=default
Status: Pending
IP:
Containers:
trident-installer:
Container ID:
Image: netapp/trident:18.10.0-custom
Image ID:
Port:
Host Port:
Command:
tridentctl
install
--volume-size
2Gi
--k8s-timeout
3m0s
--namespace
default
--dry-run
--pvc
trident
--pv
trident
--volume-name
trident
--trident-image
netapp/trident:18.10.0-custom
--etcd-image
quay.io/coreos/etcd:v3.3.9
--in-cluster=false
State: Waiting
Reason: ErrImagePull
Ready: False
Restart Count: 0
Environment:
Mounts:
/setup from setup-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from trident-installer-token-4svnl (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
setup-dir:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: trident-installer
Optional: false
trident-installer-token-4svnl:
Type: Secret (a volume populated by a Secret)
SecretName: trident-installer-token-4svnl
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 1m default-scheduler Successfully assigned default/trident-installer to 10.1.2.125
Normal Pulling 1m kubelet, 10.1.2.125 pulling image "netapp/trident:18.10.0-custom"
Warning Failed 1m kubelet, 10.1.2.125 Failed to pull image "netapp/trident:18.10.0-custom": rpc error: code = Unknown desc = manifest for docker.io/netapp/trident:18.10.0-custom not found
Warning Failed 1m kubelet, 10.1.2.125 Error: ErrImagePull
Normal SandboxChanged 2s (x19 over 1m) kubelet, 10.1.2.125 Pod sandbox changed, it will be killed and re-created.

Why installer tries to obtain image _netapp/trident:18.10.0-custom_ and how to define local image instead ?

[kangarlou]

You either need to push your custom image to a registry and use tridentctl install --trident-image <IMAGE> or docker load your custom image on every node in your k8s cluster.

[arsenalzp]

@kangarlou
Thank you for quick response!
Seems, the same error:

$./tridentctl install --trident-image trident:18.10.0-custom -n default
INFO Created installer service account. serviceaccount=trident-installer
INFO Waiting for object to be created. objectName=clusterRole
INFO Created installer cluster role. clusterrole=trident-installer
INFO Waiting for object to be created. objectName=clusterRoleBinding
INFO Created installer cluster role binding. clusterrolebinding=trident-installer
INFO Created installer configmap. configmap=trident-installer
INFO Waiting for object to be created. objectName=installerPod
INFO Created installer pod. pod=trident-installer
INFO Waiting for Trident installer pod to start.
INFO Trident installer pod started. namespace=default pod=trident-installer
> INFO standard_init_linux.go:178: exec user process caused "no such file or directory" caused= exec= process= standard_init_linux.go:178:= user=
INFO Waiting for Trident installer pod to finish.
INFO Trident installer pod finished. namespace=default pod=trident-installer
WARN Installer pod status is Failed. Use 'kubectl describe pod trident-installer -n default' for more information. pod=trident-installer
INFO In-cluster installation completed.
INFO Deleted installer cluster role binding.
INFO Deleted installer cluster role.
INFO Deleted installer service account.

$ kubectl logs trident-installer
standard_init_linux.go:178: exec user process caused "no such file or directory"

[kangarlou]

I would bring up the Trident pod with a command like while true; do echo hi && sleep 1; done and inspect the running pod for the Trident binary and try running the Trident command using kubectl exec.

[arsenalzp]

@kangarlou
I added commet to the following line:

CMD ["/usr/bin/env -i PATH='/netapp:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' /usr/local/bin/$BIN -port $PORT -etcd_v3 $ETCDV3 -k8s_api_server $K8S"]

and put new one:

ENTRYPOINT while true; do echo hi && sleep 1; done

Let's try to run:

$kubectl run trident --image=trident:18.10.0-custom

$ kubectl get pod
NAME READY STATUS RESTARTS AGE
trident-5b49fcff86-947d7 1/1 Running 1 5m

$ kubectl exec trident-5b49fcff86-947d7 /usr/local/bin/tridentctl
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "no such file or directory"

command terminated with exit code 126

$ kubectl exec trident-5b49fcff86-947d7 file /usr/local/bin/tridentctl
/usr/local/bin/tridentctl: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, interpreter /lib64/ld64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=64d79bd71b3db664a43a9dce8c71efa90d7d6658, with debug_info, not stripped

[arsenalzp]

Seems, there are a lack of some libraries:

$ kubectl exec trident-5b49fcff86-947d7 ldd /usr/local/bin/tridentctl
/lib64/ld64.so.2 (0x3fffb7480000)
libpthread.so.0 => /lib64/ld64.so.2 (0x3fffb7480000)
libc.so.6 => /lib64/ld64.so.2 (0x3fffb7480000)

$kubectl exec trident-5b49fcff86-947d7 ls /lib64/ld64.so.2
ls: cannot access '/lib64/ld64.so.2': No such file or directory
command terminated with exit code 2

[arsenalzp]

I tried the following:

$ readelf -a /usr/local/bin/tridentctl|grep interp
[ 1] .interp PROGBITS 0000000010000238 00000238
[Requesting program interpreter: /lib64/ld64.so.2]
01 .interp
02 .interp .note.ABI-tag .note.go.buildid .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt .init .text .fini .rodata .typelink .itablink .gopclntab .eh_frame

$_mkdir /lib64
$ln -s /lib/libc.musl-ppc64le.so.1 /lib64/ld64.so.2

In result I have segfault:

$ strace -vd /usr/local/bin/tridentctl
strace: ptrace_setoptions = 0x51
strace: new tcb for pid 711, active tcbs:1
strace: [wait(0x80137f) = 711] WIFSTOPPED,sig=SIGSTOP,EVENT_STOP (128)
strace: pid 711 has TCB_STARTUP, initializing it
strace: [wait(0x80057f) = 711] WIFSTOPPED,sig=SIGTRAP,EVENT_STOP (128)
strace: [wait(0x00127f) = 711] WIFSTOPPED,sig=SIGCONT
strace: [wait(0x00857f) = 711] WIFSTOPPED,sig=133
execve("/usr/local/bin/tridentctl", ["/usr/local/bin/tridentctl"], ["BIN=trident_orchestrator", "HOSTNAME=trident-5b49fcff86-jq4z"..., "CLI_BIN=tridentctl", "KUBERNETES_PORT_443_TCP_PROTO=tc"..., "KUBERNETES_PORT_443_TCP_ADDR=192"..., "KUBERNETES_PORT=tcp://192.168.10"..., "PWD=/", "HOME=/root", "KUBERNETES_SERVICE_PORT_HTTPS=44"..., "KUBERNETES_PORT_443_TCP_PORT=443", "PORT=8000", "TRIDENT_IP=localhost", "KUBERNETES_PORT_443_TCP=tcp://19"..., "TERM=xterm", "K8S=", "SHLVL=1", "KUBERNETES_SERVICE_PORT=443", "ETCDV3=http://localhost:8001", "PATH=/usr/local/sbin:/usr/local/"..., "KUBERNETES_SERVICE_HOST=192.168."..., "_=/usr/bin/strace"]strace: [wait(0x04057f) = 711] WIFSTOPPED,sig=SIGTRAP,EVENT_EXEC (4)
strace: [wait(0x00857f) = 711] WIFSTOPPED,sig=133
) = 0
strace: [wait(0x00857f) = 711] WIFSTOPPED,sig=133
set_tid_address(0x3fff79832880strace: [wait(0x00857f) = 711] WIFSTOPPED,sig=133
) = 711
strace: [wait(0x00857f) = 711] WIFSTOPPED,sig=133
mprotect(0x11d00000, 65536, PROT_READstrace: [wait(0x00857f) = 711] WIFSTOPPED,sig=133
) = 0
strace: [wait(0x000b7f) = 711] WIFSTOPPED,sig=SIGSEGV
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
strace: [wait(0x06057f) = 711] WIFSTOPPED,sig=SIGTRAP,EVENT_EXIT (6)
strace: [wait(0x00008b) = 711] WIFSIGNALED,core,sig=SIGSEGV
+++ killed by SIGSEGV (core dumped) +++
strace: mmap_cache_delete: tgen=0, ggen=0, tcp=0x106109d80, cache=0, caller=droptcb
strace: dropped tcb for pid 711, 0 remain
Segmentation fault

[arsenalzp]

Attempted to build Trident on ppc64le/ubuntu and, seems, it's getting better : Trident pod has started and is running!

[kangarlou]

Closed per https://github.com/NetApp/trident/issues/205.