Swarmkit: "no suitable node available for task" on arm64 due to non-normalized platform name

[vielmetti]

[updated with current information]

arm64 jobs fail to schedule on Swarm, because the system identifies itself as "aarch64". A fix at #2411 in manager/scheduler/filter.go normalizes these names for the 64-bit Arm architecture.

Original report

The issue simply put is that Swarm doesn't always work on arm64 nodes, but sometimes it does. However the issue is sufficiently complicated that I don't yet have a minimum test case or a suggested line of code to fix.

There's information about this issue in a lot of places on the net, and this tracking issue attempts to sort out what works, what's not working, and a minimum suitable test to fix it.

The symptom is this kind of error message in the log file:

DEBU[2017-10-09T20:36:22.336625100Z] no suitable node available for task           module=node node.id=xl2vxso5gh504rosy2u9nyqkc task.id=yj0vos4kyogoklebm3dvw50xb

and a current state of "PENDING".

Some related issues:

• 17.06 won't deploy stack anymore. no suitable node (unsupported platform on 3 nodes) on armhf docker cluster #2294
• State 'Pending' is too vague when a constraint condition is not meet https://github.com/moby/moby/issues/33807
• Platform Filter for Service Scheduling #1981
• Adding Platforms field to TaskSpec Placement https://github.com/moby/moby/pull/33144
• Multi-architecture support in Swarm mode https://github.com/moby/moby/issues/31348
• cross build and architecture labeling https://github.com/moby/moby/issues/35158

There may be additional details hiding elsewhere; I'll track those down as they are identified.

I'm sure there's a file somewhere that has mappings from variant names to canonical names - otherwise how would this work at all anywhere - but I haven't tracked down that file yet.

[alexellis]

Agreed - we should be able to find out what the difference is between expected and current state. These breaking changes came in with the 17.06 release

[vielmetti]

This issue is referred to in https://blog.hypriot.com/post/setup-simple-ci-pipeline-for-arm-images/ especially at https://blog.hypriot.com/post/setup-simple-ci-pipeline-for-arm-images/#comment-3551354550 - attn @StefanScherer

[StefanScherer]

@vielmetti I only had problems with images that are cross built on Circle or Travis or another Intel machine. This worked in the past up to 17.05. It worked for with a Docker image that I built on an ARM64 platform.
Do you see problems with images built on ARM64?

[justincormack]

I can imagine there could be a bug with cross building, if we could verify that is the issue it should be trackable.

[vielmetti]

One specific image that @alexellis was having problems with is this one:

ed$ docker run --rm mplatform/mquery luxas/prometheus-arm64:v1.5.2
Manifest List: No
 Supports: amd64/linux

It looks like @luxas has a prometheus-arm64 image that advertises that it supports amd64/linux, which is unsurprising that it doesn't work. The question is how that image was built. The code is from https://github.com/luxas/kubeadm-workshop

[justincormack]

Yes, that seems to be cross built here https://github.com/luxas/kubeadm-workshop/blob/master/images/prometheus/Makefile

[justincormack]

Currently there is no way to label something as a different architecture if you cross build it. That missing feature is a moby/moby issue rather than a swarmkit one (not sure if there is an existing issue).

[vielmetti]

Thanks @justincormack - issue opened on moby/moby - I looked and didn't find an existing issue, but happy to close if it really turns out to be a duplicate.

[vielmetti]

Still puzzled.

The specific image I'm trying to work on is on

https://hub.docker.com/r/vielmetti/multiarch-prometheus/

which I constructed from the manifest at

https://github.com/vielmetti/multiarch-aarch64/blob/master/manifest/multiarch-prometheus-1.5.2.yml

When I start from the command line as

# docker run --rm vielmetti/multiarch-prometheus:1.5.2
time="2017-10-12T19:54:53Z" level=info msg="Starting prometheus (version=1.5.2, branch=master, revision=bd1182d29f462c39544f94cc822830e1c64cf55b)" source="main.go:75" 

it starts just fine.

Instead, when I start as

# docker service create --name test2_prometheus -p 19090:9090 vielmetti/multiarch-prometheus:1.5.2
e5c69pjghnogpusrqhekitte4
overall progress: 0 out of 1 tasks 
1/1: pending   

$ docker run --rm mplatform/mquery vielmetti/multiarch-prometheus:1.5.2
Manifest List: Yes
Supported platforms:
 - amd64/linux
 - arm64/linux (variant: undefined)

Just to make things very curious, when I set up the same service through Portainer, it works just fine.

[vielmetti]

With @crunchywelch 's change at https://github.com/docker/swarmkit/pull/2411 the problem of failing to schedule on arm64 is resolved.

[vielmetti]

The change at #2411 references arm64 as the normalized name of aarch64, in the same way that amd64 is a normalized name of x86_64.

[vielmetti]

We have this running with no ill effects in a Swarm cluster that has both arm64 and amd64 systems. What would be necessary to get this patch into the next version?

[stevvooe]

@vielmetti I've LGTM'd the PR.

@thaJeztah Can this make 17.11?

[nishanttotla]

Closing this issue since https://github.com/docker/swarmkit/pull/2411 was merged. This change will be part of 17.11: https://github.com/docker/swarmkit/pull/2427

[kremerol]

@vielmetti - @nishanttotla
I think I am able to reproduce this issue on newer docker versions. I did a dedicated branch to highlight the pb: https://github.com/biarms/mysql/tree/feature/reproduce-swarmkit-issue-2401.
What is very strange is that I can reproduce the pb every time on CircleCI, while the same code never fails on Travis:

• Logs on CircleCI: https://app.circleci.com/pipelines/github/biarms/mysql/59/workflows/2eb9a84b-d312-4cfa-90e7-83f8bd9dd6d1/jobs/57/steps
• Logs on Travis: https://travis-ci.org/github/biarms/mysql/builds/690571420