Cli: `docker service ls` fails in swarm mode on amd64 after upgrade to version 20.10.0

[tlex]

Description
On Ubuntu Focal, after upgrade to 20.10.0, docker service ls fails with:

Error response from daemon: rpc error: code = Unimplemented desc = unknown method ListServiceStatuses for service docker.swarmkit.v1.Control

Steps to reproduce the issue:

1. On Ubuntu Focal install the latest docker-ce==5:20.10.0~3-0~ubuntu-focal, docker-ce-cli==5:20.10.0~3-0~ubuntu-focal and containerd.io==1.4.3-1 versions
2. Ensure the node is part of a swarm (in my case, a master)
3. Issue docker service ls

Describe the results you received:

Error response from daemon: rpc error: code = Unimplemented desc = unknown method ListServiceStatuses for service docker.swarmkit.v1.Control

Describe the results you expected:
I expected to see a list of services deployed in swarm

Additional information you deem important (e.g. issue happens only occasionally):
Happens every single time. Tried downgrade, upgrade, reboot even.

Output of docker version:

$ docker version
Client: Docker Engine - Community
 Version:           20.10.0
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        7287ab3
 Built:             Tue Dec  8 18:59:40 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       eeddea2
  Built:            Tue Dec  8 18:57:45 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.4.2-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: sgd7uzk37qfjzg4vnie9pj7gd
  Is Manager: true
  ClusterID: jyxeo7f0s7k209xy1544mylz6
  Managers: 4
  Nodes: 4
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.188.10.1
  Manager Addresses:
   10.188.10.1:2377
   10.188.10.2:2377
   10.188.10.3:2377
   10.188.10.4:2377
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-56-generic
 Operating System: Ubuntu 20.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.81GiB
 Name: docker-a
 ID: EFKT:JEKQ:JUQX:MZWR:XGP5:X7AE:CFVV:4Q4H:VNXQ:NXFZ:GG4A:ND6R
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio weight support
WARNING: No blkio weight_device support

Additional environment details (AWS, VirtualBox, physical, etc.):
Running on bare metal with up to date Ubuntu Focal and a 5.4 kernel:

$ uname -a
Linux docker-a 5.4.0-56-generic #62-Ubuntu SMP Mon Nov 23 19:20:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

My workaround for now is to pin:

docker-ce=5:19.03.14~3-0~ubuntu-focal
docker-ce-cli=5:19.03.14~3-0~ubuntu-focal
containerd.io=1.3.9-1

[tlex]

Note, on Ubuntu Groovy on aarch64(linux/arm64) - with a single node swarm - it works as intended.

[janul]

I had a similar issue. upgrade on majority of nodes in a cluster solves that.

[thaJeztah]

The ListServiceStatuses API option was added in API v1.41, so if you're running mixed versions of docker in the swarm cluster (or have an override in place (DOCKER_API_VERSION) to use an older API version, that would explain yes.

@tlex are all your nodes updated to 20.10?

[tlex]

Indeed, I only updated 2/4 nodes. I'll check again and let you know

[tlex]

I can confirm, after updating a majority of nodes, everything works as expected.

Thanks

[thaJeztah]

Thanks! That's good to hear!

Let me close this ticket, as it looks like the problem is resolved, but feel free to continue the conversation

[xavier-lr]

This is happening to me with a majority of nodes upgraded. Even the majority of managers are upgraded. The "leader" remains in version 19, is this the root cause?

centos@pro-01:~$ docker service ls
Error response from daemon: rpc error: code = Unimplemented desc = unknown method ListServiceStatuses for service docker.swarmkit.v1.Control
centos@pro-01:~$ docker node ls
ID                            HOSTNAME                                  STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
leo4yiyft7pq0m1tky8c3rsh4     cluster-pro-api-20200416_154121           Ready     Active                          19.03.8
a4tmqpyj2p6nrd8sa00yxng05     cluster-pro-api-20210114_094535           Ready     Active                          20.10.2
w09pu8q60r92lm5io66l9wc92     cluster-pro-batch-20210126_175214         Ready     Active                          20.10.2
54lhps735rhfjm1bxwy6ah1cs     cluster-pro-fastcompute-20201228_101827   Ready     Active                          19.03.8
u28t42lzthhbi63aoqpxx03tp *   pro-01                                    Ready     Active         Reachable        20.10.2
1g0rrtaqch7z5u2gczhmuofyi     pro-02                                    Ready     Active         Reachable        20.10.2
q14d9ue7dcjxr98kjqw164zpx     pro-03                                    Ready     Active         Leader           19.03.5

[thaJeztah]

Even though the "leader" is on an older version, in this case it matters which manager you're connected with. In your case it looks like you're connected to pro-01, which is already on Docker 20.10.2.

Note that running a "mixed version" cluster is not supported, and should only be a temporary situation (while upgrading the cluster); during that period, it's not recommended to make changes to the cluster state (deploy/update services or other swarm-managed objects).

[xavier-lr]

Even though the "leader" is on an older version, in this case it matters which manager you're connected with. In your case it looks like you're connected to pro-01, which is already on Docker 20.10.2.

Note that running a "mixed version" cluster is not supported, and should only be a temporary situation (while upgrading the cluster); during that period, it's not recommended to make changes to the cluster state (deploy/update services or other swarm-managed objects).

In fact I'm connected to pro-01 wich fails to list services, as it does pro-02. The only remaining manager with old version (pro-03) is the only one being able to list services.
That's why I'm now afraid that if I proceed to upgrade pro-03, I won't be able to list the services anywhere... or then it will start working on all managers? That's my main concern and the only reason I did not upgrade remaining nodes yet..

[thaJeztah]

"pro-01" and "pro-02" both are on 20.10; the reason they can't list nodes is that they attempt to use an option that's introduced in 20.10, so when trying to get information from the nodes that are _not_ updated ("cluster-pro-fastcompute-20201228_101827" and "pro-03"), it fails.

[belfo]

Even though the "leader" is on an older version, in this case it matters which manager you're connected with. In your case it looks like you're connected to pro-01, which is already on Docker 20.10.2.
Note that running a "mixed version" cluster is not supported, and should only be a temporary situation (while upgrading the cluster); during that period, it's not recommended to make changes to the cluster state (deploy/update services or other swarm-managed objects).

In fact I'm connected to pro-01 wich fails to list services, as it does pro-02. The only remaining manager with old version (pro-03) is the only one being able to list services.
That's why I'm now afraid that if I proceed to upgrade pro-03, I won't be able to list the services anywhere... or then it will start working on all managers? That's my main concern and the only reason I did not upgrade remaining nodes yet..

Did you upgrade latest node?
I have exact same config / issue
Want to be sure it will work once the latest is update.

[xavier-lr]

Hello, yes, crossed fingers and updated remaining nodes. Once ALL nodes (not just majority as said at some point and caused the confusion to me) were upgraded, then it all start working again.

[belfo]

ok thx :)