etcd doesn't run on arm64

[glevand]

Just updated to v2.3.1 and got this: unsupported architecture; unreliable, unstable. This makes CoreOS unusable for arm64 users. Please think about the ramifications next time you make this kind of change.

[luxas]

https://github.com/coreos/etcd/issues/5027#issuecomment-212481532

[xiang90]

I believe @glevand wants etcd to start on arm64 by default. https://github.com/coreos/etcd/pull/5028 wont solve it. We are working on CI for arm64.

[luxas]

Yes, both I and @glevand wants to start etcd on arm64 without blockers.
But with v2.3.1, it's impossible without hacking the code.
I obviously know that #5028 doesn't solve it, but at least it's possible without code changes,

If you want help with CI for arm, arm64 and ppc64le, just ping me

[glevand]

So what do we need to do to make checkSupportArch() return success for arm64?

[xiang90]

@glevand

1. Setup CI to build/test etcd for every pull request.
2. Continuously running etcd functional testing on arm64.

Step1 should be enough to say we support arm64 in a limited way.

[glevand]

@xiang90 OK, @lupan2005 has some patches to add arm64 build support to the travis.ci gimme script which he is preparing to submit: https://github.com/lupan2005/travis-ci-gimme. This will only support building though. We will see how those work with etcd PR's. He is also working to bring up a Jenkins installation for arm64 testing.

[xiang90]

@glevand Awesome! Thanks for the help on arm64.

[steveeJ]

to be officially supported on CoreOS, which means that every system application needs to catch up. Besides the missing CI integration, what is the official state of the build scripts and code WRT arm64 support?

[heyitsanthony]

@steveeJ arm64 builds (but not tests) already go through CI, but there hasn't been much testing. @mischief did some initial testing with CoreOS+arm64 using #5088

[glevand]

@lupan2005 has been working on our local Jenkins setup. Currently, some of the fmt_tests and dep_tests fail on arm64, but all the unit_tests pass. He plans to go through the tests and get them working. It seems to be more of a tools problem than the etcd code.

[steveeJ]

@glevand sounds like a tooling problem to me too since the most important part passes, which is good to hear. Are there also any integration/functional tests in place to test the binary execution?

[glevand]

@steveeJ well, we have arm64 clusters here running fleet and flannel. That's what we've been using as a test... It would be nice if we did something more to stress it.

[xiang90]

@glevand We listed a few things that need to be done to make arm64 officially supported here: https://github.com/coreos/etcd/blob/master/Documentation/op-guide/supported-platform.md

For arm64, we need a maintainer and set up integration test CI. Do you want to help on that?

[xiang90]

also @luxas

[glevand]

@xiang90 Sure, I can act as arm64 maintainer. We have our local Jenkins system doing the etcd tests, which are generally working OK, but those results are not available publicly. In the next week or so I will do some rework of the tests and work toward having public test results.

I'll see if the tests run using qemu and the foundation model, which would allow anyone to run them. So far we have just run them with real hardware.

[xiang90]

@glevand Is it possible to hook the testing infra with github? So we can run the tests for each pull request?

[glevand]

@xiang90 The problem is that we need to run on an arm64 system. We may be able to use qemu on x86 machines to emulate arm64, but qemu may not run fast enough for the tests to succeed without timeouts. The other option is to run the tests on real arm64 hardware, but it needs to be looked into how to this with github.

[heyitsanthony]

@glevand qemu is definitely too slow for the tests-- @mischief tested arm64 etcd on qemu a few months ago and got a bunch of timeouts.

[luxas]

I wouldn't either go the qemu route. Qemu has quite poor Go runtime support.

Here's an article about Github and Jenkins, might be interesting: https://www.terlici.com/2015/09/29/automated-testing-with-node-express-github-and-jenkins.html

[xiang90]

@glevand Any progress on CI testing on arm64?

[glevand]

@xiang90 still working on the arm64 CI, but nothing to report yet.

[xiang90]

@glevand Is it possible to finish this up before Oct? Or I have to move this to 3.2 milestone.

[glevand]

@xiang90 Our goal is to have it for v3.1.0. I'll bump the priority.

[xiang90]

@glevand Thanks!

[xiang90]

@glevand Kindly remind you that 3.1 rc will be out in mid of Sept.

[xiang90]

@glevand any update on this one? what is your next step?

[glevand]

@xiang90 All tests now run successfully on arm64.
I'm now trying to figure out if we can somehow run arm64 hosted tests from github PRs.

[xiang90]

@glevand OK. That would be great. It is also the last missing part to say we officially support arm64 I guess.

[xiang90]

@glevand Any progress update on this one?

[xiang90]

@glevand I am going to move this out of 3.1 milestone. We are working on an etcd Jenkins. If the happens, we can simply add a arm64 slave to finish the remaining part.

[mkumatag]

@xiang90 we can add ppc64le slave as well to the same jenkins server

[vielmetti]

What's the distance right now between "unsupported" and "supported" arm64 etcd?

I don't see any evidence of unusual behavior on that platform reported recently.

(cc @glevand @xiang90 )

[glevand]

@vielmetti The Container Linux kola test suite does etcd tests, and those have been passing, so I think etcd itself is working OK on arm64.

There have been some improves made to the project's CI/testing recently, and that needs to be updated to work on arm64. My feeling is that in the least the arm64 equivalents of the linux-amd64-* targets of the project's travis file should be working. That would need:

• [ ] Get the build script and test script to run on arm64.
• [ ] Get the etcd-test docker image to build on/for arm64.
• [ ] Get the same test script PASSES that the travis file runs for amd64 to work on arm64.
• [ ] Setup an arm64 Jenkins project that has an etcd github hook to run the tests. See: https://jenkins-etcd-public.prod.coreos.systems/job/etcd-ci-ppc64/.

[mmarmm]

@glevand Hi, could you run CI on Shippable or CodeFresh or does it need to be on Travis? If a machine is supplies via the WorksOnArm project would that be an alternative if the first two don't work? Thank you for your help trying to get etcd more available on Arm64.

http://docs.shippable.com/platform/tutorial/workflow/run-ci-builds-on-arm/ (three free parallel builds) https://thenewstack.io/shippable-brings-continuous-integration-and-delivery-to-arm-servers/

CodeFresh Beta signup: https://goo.gl/forms/aDhlk56jZcblYokj1

Scaleway does also offer paid Arm64 resources: https://www.scaleway.com/virtual-cloud-servers/#anchor_arm

[glevand]

@mmarmm Travis could currently only be used for cross building and static code checks for arm64. Any run-time testing would need to be done through the project's Jenkins system (on an arm64 Jenkins node). My mention of Travis above is only to illustrate what tests in the test script need to be working, not that those tests should be run through Travis.

[jpbetz]

arm64 is included etcd releases, closing this