Kiali: Feature request: Build ARM/ARM64 images

[wxdao]

It would be great to have kiali/kiali on docker hub to work on ARM machines.

[pilhuhn]

Could you help us getting that done?
Do you have arm hardware to test? I know AWS has Arm virtual machines, but setting all that up is not trivial

[wxdao]

I know shippable.com claims to provide free ARM/ARM64 machines for public repos. However I haven't had a chance to try their service yet.

I'd like to help you with this if possible. I just filed a contact to shippable.com to request free ARM machines, waiting for their response now.

[pilhuhn]

That's cool - thanks!

$GOACH=arm64
$GOOS=linux

This post nicely explains how multi-arch images could then work. But I guess for the start we should only try to have the arm one available for people to explicitly select.

[wxdao]

Hi @pilhuhn. I managed to setup an arm pipeline using drone cloud. You can have a look at https://cloud.drone.io/wxdao/kiali/8 & https://github.com/wxdao/kiali.

Should I create a PR into the project if that looks right to you?

[israel-hdez]

Personally, I was expecting doing cross compilation using our current setup.
Currently, the docker images are built in a Jenkins box, so not sure how we could integrate drone into the build process.

But if this is OK for @pilhuhn, I guess you can create a PR.

[wxdao]

Personally, I was expecting doing cross compilation using our current setup.
Currently, the docker images are built in a Jenkins box, so not sure how we could integrate drone into the build process.

But if this is OK for @pilhuhn, I guess you can create a PR.

I agree that cross compilation is enough to build ARM docker images. I think what @pilhuhn wanted is to actually run tests in ARM machines before publishing ARM docker images, so you may not need to integrate drone to your internal jenkins box. Just checking the tests results from drone linked to this public repo before triggering a internal build should be enough.

[israel-hdez]

@wxdao Ok, so I understood a different thing
What I understood is to have an arm machine to test that the images are built correctly... A machine to download the arm images from dockerhub and try them (do a "smoke test").

So, let's wait for him to clarify ;)

[kiali-workflow[bot]]

This issue has been automatically marked as stale because it has not had activity withint the last 60 days. It will be closed if no further activity occurs in the next 7 days. Thank you for your contributions.

[israel-hdez]

@wxdao are you still interested in this?

[wxdao]

Yes I am. However I'm not sure how I can help.

[israel-hdez]

Perhaps, as was briefly discussed before, just finish the CI setup, so that tests are run on each PR. So, you could just finish the initial effort and open a PR.
Once that is done, we can check how to cross-compile and publish the images to DockerHub.

[cfcosta]

@wxdao did you test running Istio/Grafana/Prometheus/Jaeger on ARM? Because we currently depend on those. There should be no reason for Kiali not to work on ARM, but if everything else is not working, then there's no point.

Also, testing it on k3s would be amazing.

[lucasponce]

This issue has no activity for months.
It sounds interesting as a community effort, so, if someone would like to help, feel free to ping us and we can re-open it.

[MrXinWang]

@lucasponce @wxdao @israel-hdez @pilhuhn Hi all, is this arm64 supporting stuff still on going? If it is, can I help in pushing things forward? I do have arm64 machines so I think building and testing should be ok for me.

[MrXinWang]

So here is my initial plan:

(1) I will firstly try to build binary and images on my arm64 machine to make it buildable. If successful I will try to post PR to fix this first.
Update:

• About the building (seems to work :P I will post a PR firstly):

root@<hostname>:/go/src/github.com/kiali/kiali# make build
Building...
GOOS=linux GOARCH=arm64 CGO_ENABLED=0  go build \
        -o /go/bin/kiali -ldflags "-X main.version=v1.15.0-SNAPSHOT -X main.commitHash=3820fd7207a8c85907f73f4bee16f63be9f6d6d1"

### Run the binary ###
root@<hostname>:/go/src/github.com/kiali/kiali# kiali
I0316 15:15:48.375320    3716 kiali.go:66] Kiali: Version: v1.15.0-SNAPSHOT, Commit: 3820fd7207a8c85907f73f4bee16f63be9f6d6d1
I0316 15:15:48.375395    3716 kiali.go:77] No configuration file specified. Will rely on environment for configuration.
F0316 15:15:48.375475    3716 kiali.go:83] server static content root directory does not exist: /opt/kiali/console

• About the testing (on baremetal only by just running make test):

root@<hostname>:/go/src/github.com/kiali/kiali# make test
Running tests, excluding third party tests under vendor
go test github.com/kiali/kiali github.com/kiali/kiali/appstate github.com/kiali/kiali/business github.com/kiali/kiali/business/checkers github.com/kiali/kiali/business/checkers/authorization github.com/kiali/kiali/business/checkers/destinationrules github.com/kiali/kiali/business/checkers/gateways github.com/kiali/kiali/business/checkers/meshpolicies github.com/kiali/kiali/business/checkers/policies github.com/kiali/kiali/business/checkers/serviceentries github.com/kiali/kiali/business/checkers/services github.com/kiali/kiali/business/checkers/sidecars github.com/kiali/kiali/business/checkers/virtual_services github.com/kiali/kiali/config github.com/kiali/kiali/config/security github.com/kiali/kiali/graph github.com/kiali/kiali/graph/api github.com/kiali/kiali/graph/config/cytoscape github.com/kiali/kiali/graph/telemetry github.com/kiali/kiali/graph/telemetry/istio github.com/kiali/kiali/graph/telemetry/istio/appender github.com/kiali/kiali/graph/telemetry/istio/util github.com/kiali/kiali/handlers github.com/kiali/kiali/jaeger github.com/kiali/kiali/jaeger/jaegertest github.com/kiali/kiali/kubernetes github.com/kiali/kiali/kubernetes/cache github.com/kiali/kiali/kubernetes/kubetest github.com/kiali/kiali/ldap github.com/kiali/kiali/log github.com/kiali/kiali/models github.com/kiali/kiali/prometheus github.com/kiali/kiali/prometheus/internalmetrics github.com/kiali/kiali/prometheus/prometheustest github.com/kiali/kiali/routing github.com/kiali/kiali/server github.com/kiali/kiali/status github.com/kiali/kiali/tests/data github.com/kiali/kiali/util github.com/kiali/kiali/util/httputil github.com/kiali/kiali/util/intutil
ok      github.com/kiali/kiali  0.028s
?       github.com/kiali/kiali/appstate [no test files]
ok      github.com/kiali/kiali/business 0.221s
ok      github.com/kiali/kiali/business/checkers        0.030s
ok      github.com/kiali/kiali/business/checkers/authorization  0.040s
ok      github.com/kiali/kiali/business/checkers/destinationrules       0.035s
ok      github.com/kiali/kiali/business/checkers/gateways       0.044s
ok      github.com/kiali/kiali/business/checkers/meshpolicies   0.031s
ok      github.com/kiali/kiali/business/checkers/policies       0.035s
ok      github.com/kiali/kiali/business/checkers/serviceentries 0.036s
ok      github.com/kiali/kiali/business/checkers/services       0.042s
ok      github.com/kiali/kiali/business/checkers/sidecars       0.033s
ok      github.com/kiali/kiali/business/checkers/virtual_services       0.045s
ok      github.com/kiali/kiali/config   0.023s
ok      github.com/kiali/kiali/config/security  0.017s
?       github.com/kiali/kiali/graph    [no test files]
ok      github.com/kiali/kiali/graph/api        0.141s
ok      github.com/kiali/kiali/graph/config/cytoscape   0.027s
?       github.com/kiali/kiali/graph/telemetry  [no test files]
?       github.com/kiali/kiali/graph/telemetry/istio    [no test files]
ok      github.com/kiali/kiali/graph/telemetry/istio/appender   0.082s
?       github.com/kiali/kiali/graph/telemetry/istio/util       [no test files]
ok      github.com/kiali/kiali/handlers 0.182s
?       github.com/kiali/kiali/jaeger   [no test files]
?       github.com/kiali/kiali/jaeger/jaegertest        [no test files]
ok      github.com/kiali/kiali/kubernetes       0.282s
ok      github.com/kiali/kiali/kubernetes/cache 0.028s
?       github.com/kiali/kiali/kubernetes/kubetest      [no test files]
?       github.com/kiali/kiali/ldap     [no test files]
?       github.com/kiali/kiali/log      [no test files]
ok      github.com/kiali/kiali/models   0.047s
?       github.com/kiali/kiali/prometheus       [no test files]
ok      github.com/kiali/kiali/prometheus/internalmetrics       1.021s
ok      github.com/kiali/kiali/prometheus/prometheustest        0.100s
ok      github.com/kiali/kiali/routing  0.042s
ok      github.com/kiali/kiali/server   1.879s
ok      github.com/kiali/kiali/status   0.045s
?       github.com/kiali/kiali/tests/data       [no test files]
ok      github.com/kiali/kiali/util     0.005s
?       github.com/kiali/kiali/util/httputil    [no test files]
?       github.com/kiali/kiali/util/intutil     [no test files]

(2) Then I will try to discuss with the community in order to update the manifest to make the image multiarch.
Update:

• The image kiali/kiali:dev can be built on arm64 with some small changes of the Dockerfile, but the operator-sdk(make/Makefile.container.mk#L20) is currently problematic as it only has x86_64 releases, we can build that from source code on arm.

@israel-hdez @pilhuhn @lucasponce Wondering if the community has any comments (highly welcomed!) Also I am not very sure about if I have run the test correctly, any comments are also highly welcomed. Thanks!

[israel-hdez]

@MrXinWang If you are willing to help, we can re-open this issue and assign it to you.

Our main concern was how to run the unit tests on CI for every PR to ensure that builds won't get broken. However, I just saw this post in Travis's blog which may make this easier: https://blog.travis-ci.com/2019-10-07-multi-cpu-architecture-support

Our second concern was how to build and push the multi-arch images to Quay.io and DockerHub. The box building the releases doesn't have access to ARM hardware, so we would need to do cross-compiling and so.

We are not sure how hard this is to do. But if you are willing to help, the team will be happy to assist you and bring guidance :-)

[MrXinWang]

Hi @israel-hdez ! Thanks for your reply and the team‘s kindly assistance. Discussed with my manager and yes we would love to work on this issue :))) You can re-open it and assign it to me now.

Also completely agree with the CI stuff. I think not only we need to keep the consistancy of the current CI, also we need to possibly add CI for arm to test its functionality on arm.

About the multiarch image, I dont know how the community build the image now, but it should be not that difficult, as <1> if the community has arm64 machine (either a server or a raspberry pi 4), the image can be directly build (I need to propose some PRs first, also I have access to arm hardware); <2> if the community only has x86_64 machines, we can also cross-compile arm64 image by using docker buildx.

What do you think? Maybe I can propose a PR to fix the Dockerfile and Makefile on arm64 first? Any other suggestions?

[israel-hdez]

@MrXinWang Thanks for taking this issue!
I reopened and assigned it to you.

Doing the changes to Dockerfile and Makefile to have a container image for arm64 sounds a good first step. Once you have that, I guess it shouldn't be too hard to have Travis running the unit tests for arm64 (I guess this could be a good 2nd step). So, go ahead with the PR :-)

Then, perhaps adding some instructions on README.md to build Kiali on arm64 would be great. This will ensure that knowledge is shared.

Once there is a way to manually build the arm64 container and CI in place, we can start looking at how to push the arm64 container do DockerHub & Quay.io.

[MrXinWang]

@israel-hdez PR #2532 proposed. First step done, although I am not sure if the operator image should also be enabled...

For the operator image, if it is required, do you suggest we ask the open-operator-sdk to propose arm releases or we build the binary from source?

As suggested, I am then going to try travis. As dependancies in current .travis.yml are swagger and lint, which support arm64 already, I would expect what you have proposed in https://blog.travis-ci.com/2019-10-07-multi-cpu-architecture-support can work directly :P Any comments do let me know please :)))

[israel-hdez]

For the operator image, if it is required, do you suggest we ask the open-operator-sdk to propose arm releases or we build the binary from source?

The operator is the preferred way to install the latest Kiali, so I think it's better to ask. But it may take time until they add ARM64 support. In the meanwhile, building from source would be quite fine because I think we don't want the installation experience to be different between x86_64 vs arm. Updating two different install instructions is cumbersome.

Otherwise, if we skip the operator, how do you think people will install Kiali on ARM?

[MrXinWang]

Hi @israel-hdez ! Thanks for letting me know. I have proposed operator-sdk/issues#2686, can you please also leave some comments there to push this process? If they release that it will be super convenient for us to enable the arm64 stuff.

Also I saw the comments you left in the PR, if we really need the operator image, the conditional used currently can be removed. I will firstly work on the source code building.

[israel-hdez]

Re-opening, because there is still work to do :)

[jmazzitelli]

Re-opening, because there is still work to do :)

What is the remaining work? Is it the Kiali Operator build?

[MrXinWang]

@israel-hdez @jmazzitelli Yep, the main part is the operator image, or the operator-sdk arm64 binary in a way...

Also I think we need to add travis support for arm, as @israel-hdez mentioned here https://blog.travis-ci.com/2019-10-07-multi-cpu-architecture-support

[BlackMix]

Good job 👏👏
I liked this system and wanted to use it on my
raspberry pi 4b 4gb, I'm looking forward to testing it

[israel-hdez]

I see progress for the operator-sdk support for arm64 :smiley:
Nice to see this is progressing.

Thanks for this @BlackMix @MrXinWang

[BlackMix]

@MrXinWang can u test? operator-sdk-arm64
I did build and posted, let me know

[MrXinWang]

@BlackMix Perfect! Awesome work!

root@<hostname>:~# uname -a
Linux <hostname> 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:41:03 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

root@<hostname>:~# ./operator-sdk-v0.16.0-27-g3687cc1a-dirty-aarch64-linux-gnu
An SDK for building operators with ease

Usage:
  operator-sdk [command]

Available Commands:
  add             Adds a controller or resource to the project
  build           Compiles code and builds artifacts
  bundle          Work with operator bundle metadata and bundle images
  cleanup         Delete and clean up after a running Operator
  completion      Generators for shell completions
  generate        Invokes a specific generator
  help            Help about any command
  migrate         Adds source code to an operator
  new             Creates a new operator application
  olm             Manage the Operator Lifecycle Manager installation in your cluster
  print-deps      Print Golang packages and versions required to run the operator
  run             Run an Operator in a variety of environments
  scorecard       Run scorecard tests
  test            Tests the operator
  version         Prints the version of operator-sdk

Flags:
  -h, --help      help for operator-sdk
      --verbose   Enable verbose logging

Use "operator-sdk [command] --help" for more information about a command.

@israel-hdez And now I can propose another PR to enable the operator image on arm. However, I guess we have another choice to make: As @BlackMix proposed his patch already to the operator-sdk community, shall we wait for the operator-sdk official release or we use @BlackMix's release in his own repository in the building script (also the versions of these operator-sdk are different)? Any comments? By any means I will try to propose this PR today :)))

[BlackMix]

so kiali need istio right? is istio for arm64? or I can use without istio, never tesed it first time xD

[MrXinWang]

@BlackMix Some of my colleagues are working on that, should be out soon :))

[BlackMix]

so I'll wait everyone official finish it,
kiali and operator are build 100% success and working for me tho.
I just need istio now :)

[jmazzitelli]

As @BlackMix proposed his patch already to the operator-sdk community, shall we wait for the operator-sdk official release or we use @BlackMix's release in his own repository in the building script

@MrXinWang you can create a draft PR to get started on it. But we will not merge into master anything using a personal repo (such as @BlackMix )... but get the draft PR working and when operator-sdk releases an official ARM release, we can update the draft PR and then promote it and test/merge it.

But for sure you can feel free to start this work now and submit a draft PR for the kiali team to start looking at and reviewing.

[MrXinWang]

@jmazzitelli PR #2581 created as a draft.

The only aim for proposing this PR such early is to test the travis and see the building log (sorry for that but cannot think of a better way)...

Great thanks for the support from the community! ;P

[BlackMix]

test build passed from operator all builds success.
just waiting merger now.

[lucasponce]

cc @jmazzitelli to take a look and update/finish this one.

[jmazzitelli]

We have travis running ARM64 builds for both kiali and the operator:

https://github.com/kiali/kiali/blob/master/.travis.yml#L5
https://github.com/kiali/kiali-operator/blob/master/.travis.yml#L6

[jmazzitelli]

NOTE: the Kiali team is not prepared to support Kiali or the Kiali operator on ARM64 hardware. So while we can say it builds (Travis CI confirms this), we cannot say anything as to its functionality nor can we support it. So, right now, we are not shipping any images to Quay.io's kiali or kiali-operator repositories. People can build them locally on their own ARM64 hardware using Makefiles if they want to test it out.

Most likely you will need ARM64 binaries for Istio as well. Looks like those will be available with Istio 1.6 (I see them being shipped with 1.6.RC2)

[israel-hdez]

We can use docker's buildx and golang cross-compiling to build the arm64 container image. But docker's buildx is an experimental feature. Not sure if we will be able to have it in the Jenkins server.

[lucasponce]

It's not clear what should be the action from here.
To make travis to push these arch as best effort ?

[jmazzitelli]

All the work is done EXCEPT we are pushing arm64 binaries to quay.io. We would have to do something in travis.

If you want arm64 support, you can build the binaries locally on your arm64 machine. We do not do anything with cross-compiling.

[israel-hdez]

Moving out of the sprint, because this looks stalled.

[morlay]

I have build oci image for 1.20.0 https://hub.docker.com/r/querycap/kiali/tags by https://github.com/querycap/kiali/blob/master/Dockerfile

but have some issue for pr fix this issue.

@israel-hdez a question.

why amd64 use registry.access.redhat.com/ubi7-minimal arm64 use registry.access.redhat.com/ubi8-minimal

https://github.com/kiali/kiali/tree/master/deploy/docker

[jmazzitelli]

why amd64 use registry.access.redhat.com/ubi7-minimal arm64 use registry.access.redhat.com/ubi8-minimal

Because only UBI 8 supports arm64. Containers for arm64 are not supported on RHEL 7 or UBI 7.

See:

• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/getting_started_with_containers/get_started_with_linux_containers#supported_architectures_for_containers_on_rhel
• https://bugzilla.redhat.com/show_bug.cgi?id=1728771

[morlay]

@jmazzitelli got it.

but why not upgrade to ubi8 both for arm64 & amd64 ?

[jmazzitelli]

but why not upgrade to ubi8 both for arm64 & amd64 ?

Most UBI8 images will work on a RHEL7 bare metal host, however those images are not officially supported by RHEL. So if there is a problem the responsibility falls on the container owner to work it out.

This means we currently are not guaranteed that an image based on UBI8 will work 100% on RHEL7 machines (and we do not want to break those users whose bare metal machines are RHEL7).

[morlay]

@jmazzitelli Thanks.

for now, have to build multi platforms image unofficially

[morlay]

@israel-hdez @jmazzitelli could you check this PR https://github.com/kiali/kiali/pull/3405

I find a way to avoid change base image of amd64, but could add arm64 image support.

If no other concerns, we could setup buildx in CI, and publish multi-arch images