Swarmkit: no suitable node (missing plugin on 3 nodes)
Hello,
I've upgraded docker to 17.09.0~ce-0~ubuntu. My containers remain in "pending" mode.
When I look inside one task, I have the following error :
no suitable node (missing plugin on 3 nodes)
Docker info :
Containers: 7
Running: 0
Paused: 0
Stopped: 7
Images: 12
Server Version: 17.09.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: uf876bw2ggt1r660790mlmpwv
Is Manager: true
ClusterID: 0p1x95aocg644bnz2v7xfdeyb
Managers: 1
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: xxxxx
Manager Addresses:
xxxx:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-96-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 14.34GiB
Name: ovh-prod-1
ID: TXNV:VBKT:MFJD:4K7W:G3JH:WDB7:LZZF:3SWD:LVQO:6SFP:WOPE:AN7E
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: nelioapp
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
docker service inspect
[
{
"ID": "mxw7zx956dv6achg5zuzotaoe",
"Version": {
"Index": 14473
},
"CreatedAt": "2017-10-03T00:02:36.445118651Z",
"UpdatedAt": "2017-10-03T02:21:39.660220753Z",
"Spec": {
"Name": "tech_blog",
"Labels": {
"com.docker.stack.image": "ghost:alpine",
"com.docker.stack.namespace": "tech"
},
"TaskTemplate": {
"ContainerSpec": {
"Image": "ghost:alpine@sha256:b7c4fcb9e78518ea85fcba80969f445301b38f76833a6668eb268da7b6e7f3a4",
"Labels": {
"com.docker.stack.namespace": "tech"
},
"Privileges": {
"CredentialSpec": null,
"SELinuxContext": null
},
"Mounts": [
{
"Type": "bind",
"Source": "/home/nelio/tech.nelio.io/",
"Target": "/var/lib/ghost/content"
}
],
"StopGracePeriod": 10000000000,
"DNSConfig": {}
},
"Resources": {},
"RestartPolicy": {
"Condition": "any",
"Delay": 5000000000,
"MaxAttempts": 0
},
"Placement": {
"Constraints": [
"node.role == manager"
],
"Platforms": [
{
"Architecture": "amd64",
"OS": "linux"
}
]
},
"Networks": [
{
"Target": "tpp0sufypym3nhup5zaul1ppw",
"Aliases": [
"blog"
]
}
],
"LogDriver": {
"Options": {
"max-size": "50m"
}
},
"ForceUpdate": 0,
"Runtime": "container"
},
"Mode": {
"Replicated": {
"Replicas": 1
}
},
"UpdateConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"RollbackConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"EndpointSpec": {
"Mode": "vip"
}
},
"PreviousSpec": {
"Name": "tech_blog",
"Labels": {
"com.docker.stack.image": "ghost:alpine",
"com.docker.stack.namespace": "tech"
},
"TaskTemplate": {
"ContainerSpec": {
"Image": "ghost:alpine@sha256:b7c4fcb9e78518ea85fcba80969f445301b38f76833a6668eb268da7b6e7f3a4",
"Labels": {
"com.docker.stack.namespace": "tech"
},
"Privileges": {
"CredentialSpec": null,
"SELinuxContext": null
},
"Mounts": [
{
"Type": "bind",
"Source": "/home/nelio/tech.nelio.io/",
"Target": "/var/lib/ghost/content"
}
]
},
"Resources": {},
"Placement": {
"Constraints": [
"node.role == manager"
],
"Platforms": [
{
"Architecture": "amd64",
"OS": "linux"
}
]
},
"Networks": [
{
"Target": "tpp0sufypym3nhup5zaul1ppw",
"Aliases": [
"blog"
]
}
],
"LogDriver": {
"Options": {
"max-size": "50m"
}
},
"ForceUpdate": 0,
"Runtime": "container"
},
"Mode": {
"Replicated": {
"Replicas": 1
}
},
"EndpointSpec": {
"Mode": "vip"
}
},
"Endpoint": {
"Spec": {
"Mode": "vip"
},
"VirtualIPs": [
{
"NetworkID": "tpp0sufypym3nhup5zaul1ppw",
"Addr": "10.0.0.6/24"
}
]
}
}
]
mlescaudron
All 31 comments
Hi,
I have same issue
Docker version 17.09.0-ce, build afdb6d4
petromatviichuk
on 10 Oct 2017
Same problem here I had a running 17.06 cluster with several stacks and after upgrading to 17.06 nothing cannot be scheduled on it and the reason is just a generic missing plugin on 3 nodes
vide
on 10 Oct 2017
@MLescaudron Ok I think I've got what happened here: you (and me, too) updated the nodes without draining them before. That leaves the cluster in an unusable state, and the only solution I found in my tests were to completely shut down the cluster, downgrade the version to the same it was running before (17.06 in my case) and then start all the docker daemons again. This will let you schedule again containers on that swarm.
Now, to properly upgrade the cluster from 17.06 to 17.09, drain the node you are going to update before updating, just like in https://docs.docker.com/engine/swarm/swarm-tutorial/drain-node/ and when you will activate again the updated node, it will start running containers.
EDIT: I spoke too early :( it worked with the first 2 nodes but after draining/updating/activating the last one (the 3rd manager), I cannot schedule any new container any more
vide
on 10 Oct 2017
@MLescaudron your issue seems to be coming from the plugin filter, which checks whether or not a given plugin is present on a node that a task is scheduled on. During upgrading, is it the case that some plugin didn't install or wasn't reported? Can you inspect each node in the cluster and make sure that all plugins are visible, the way you expect them?
nishanttotla
on 11 Oct 2017
@nishanttotla I have the same issue and checked the plugin list before and after upgrade and it's exactly the same (btw I've just the standard plugins that come with docker CentOS' package)
vide
on 16 Oct 2017
@nishanttotla just installed a fresh 17.09 cluster from scratch, and the problem persists. I don't know if maybe is something related to the images we are building but it's really strange.
EDIT: well it cannot find a suitable node even for the public nginx image, so I guess it's not related to my private images.
vide
on 16 Oct 2017
My docker node inspect output:
[
{
"ID": "xsam8fhtu852rmsztfv7ex8bv",
"Version": {
"Index": 49
},
"CreatedAt": "2017-10-16T14:29:17.449628404Z",
"UpdatedAt": "2017-10-16T14:49:05.428620458Z",
"Spec": {
"Labels": {},
"Role": "manager",
"Availability": "active"
},
"Description": {
"Hostname": "swarm01.domain.com",
"Platform": {
"Architecture": "x86_64",
"OS": "linux"
},
"Resources": {
"NanoCPUs": 16000000000,
"MemoryBytes": 33536512000
},
"Engine": {
"EngineVersion": "17.09.0-ce",
"Plugins": [
{
"Type": "Log",
"Name": "awslogs"
},
{
"Type": "Log",
"Name": "fluentd"
},
{
"Type": "Log",
"Name": "gcplogs"
},
{
"Type": "Log",
"Name": "gelf"
},
{
"Type": "Log",
"Name": "journald"
},
{
"Type": "Log",
"Name": "json-file"
},
{
"Type": "Log",
"Name": "logentries"
},
{
"Type": "Log",
"Name": "splunk"
},
{
"Type": "Log",
"Name": "syslog"
},
{
"Type": "Network",
"Name": "bridge"
},
{
"Type": "Network",
"Name": "host"
},
{
"Type": "Network",
"Name": "macvlan"
},
{
"Type": "Network",
"Name": "null"
},
{
"Type": "Network",
"Name": "overlay"
},
{
"Type": "Volume",
"Name": "local"
}
]
},
"TLSInfo": {
"TrustRoot": "-----BEGIN CERTIFICATE-----\nMIIBazCCARCgAwIBAgIUN/3LCRv3Dy7WTpJdh3SrANs0OVUwCgYIKoZIzj0EAwIw\nEzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMTcxMDE2MTQyNDAwWhcNMzcxMDExMTQy\nNDAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH\nA0IABG18C33dIsZSyNdXMwCBGQEbvb9FbzBagF1c3w+RO57SNwn+uodBE3mgjQus\nOcMA8DBWePCu1pGdHfxgMou1kDyjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB\nAf8EBTADAQH/MB0GA1UdDgQWBBTWCJshR4e0CV8EJnfIjQ/Y+06XxjAKBggqhkjO\nPQQDAgNJADBGAiEAlKgJHvgXhaYRUbZLYu9g2gKcQIic5cHcFufqBDayFckCIQCN\n1PFVQ8K3ClPOWAf+unwAPENp1ID7goVEJwTqml16Bg==\n-----END CERTIFICATE-----\n",
"CertIssuerSubject": "MBMxETAPBgNVBAMTCHN3YXJtLWNh",
"CertIssuerPublicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEbXwLfd0ixlLI11czAIEZARu9v0VvMFqAXVzfD5E7ntI3Cf66h0ETeaCNC6w5wwDwMFZ48K7WkZ0d/GAyi7WQPA=="
}
},
"Status": {
"State": "ready",
"Addr": "192.168.10.202"
},
"ManagerStatus": {
"Reachability": "reachable",
"Addr": "192.168.10.202:2377"
}
}
]
Is there a way to know which plugin is it complaining about?
vide
on 17 Oct 2017
@vide hmm, I think the error should be reporting which plugin it is failing on, but right now it doesn't. It seems that docker node inspect gets the list of plugins rightly. Which are the plugins that you are trying to use, and do you see them in the list?
nishanttotla
on 19 Oct 2017
@nishanttotla I'm not using any "special" plugin, it's just a basic docker swarm installation (so, overlay, bridge, host for networking) and I'm also using the syslog plugin to dispatch logs remotely.
vide
on 20 Oct 2017
Same for me ... had the beegfs in place...
morph027
on 29 Oct 2017
Same here. I only use the default docker-ce.
This happens in live service server.
The project of my team will move on live service state in three weeks, but I've found a ton of critical Docker bugs which cannot be accepted on live service.
Especially, most bugs are related to swarm mode network, discovery, or something like this bug.
This is a stupid bug. If you don't have any good programmer in kernel or system side, why don't you hire ppl?
It seems docker swarm mode is not matured yet. Maybe it can be so next 2 or 3 years.
This is the last line my team can make a decision. With high possibility, we will go back to docker swarm.
With COMPLETLY Clean Ubuntu 16.0x, installed docker-ce 17.09, not upgraded.
22 Nodes (Backend)
"Status": {
"Timestamp": "2017-11-02T03:59:39.190016Z",
"State": "pending",
"Message": "missing plugin on 1 node",
"ContainerStatus": {},
"PortStatus": {}
},
yunghoy
on 2 Nov 2017
@yunghoy can you post the output of docker service inspect for the service you're running? It seems like this might be a plugin filter issue like I suspected earlier. While I couldn't see an obvious reason at first, this may be related to the fact that @MLescaudron's service had a bind mount. I would like to see the details of your service to confirm this.
nishanttotla
on 7 Nov 2017
@yunghoy @MLescaudron would you also be able to provide specifically how you created the service (as in the docker service create command that you used)?
nishanttotla
on 7 Nov 2017
@vide I don't see any reference to the syslog driver in your service inspect, and as a result, the swarm plugin filter is unable to find it. It is possible that you're running into a bug which is coming up because I believe older daemon versions didn't report log driver names.
cc @thaJeztah
nishanttotla
on 7 Nov 2017
@nishanttotla bootstrapping a test cluster right now for debugging...you'll ask, i'll debug and answer ;)
Cluster will be fresh from the scratch Ubuntu 16.04 LTS with latest docker-ce
morph027
on 7 Nov 2017
Setup
root@docker-test-1:~# docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
u1p6llojg47n84wpesvqif1cw * docker-test-1 Ready Active Leader
qpwj6u7l4mopmyapk28u5jdq8 docker-test-2 Ready Active Reachable
0jzucmx0dz2msgg2ctmbequv3 docker-test-3 Ready Active Reachable
root@docker-test-1:~# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 17.09.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: u1p6llojg47n84wpesvqif1cw
Is Manager: true
ClusterID: rdezeqpgm4d386lqtf3r8mwcl
Managers: 3
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.0.0.147
Manager Addresses:
10.0.0.147:2377
10.0.0.148:2377
10.0.0.152:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-98-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.998GiB
Name: docker-test-1
ID: 7AWB:E76K:Y7FX:5SZ7:GG5O:VCQ5:CHAY:U5CR:WSN4:U32A:C2S7:WLPM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
root@docker-test-1:~# docker stack deploy -c visualizer.yml visualizer
Creating network visualizer_default
Creating service visualizer_visualizer
Test service with bind mounts (works)
version: '3.1'
services:
visualizer:
image: "dockersamples/visualizer:latest"
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/var/run/docker.sock
deploy:
placement:
constraints: [node.role == manager]
ports:
- 8080:8080
root@docker-test-1:~# docker stack deploy -c visualizer.yml visualizer
root@docker-test-1:~# docker service inspect visualizer_visualizer
[
{
"ID": "ahralnuc46z68vwyojvgl6lss",
"Version": {
"Index": 27
},
"CreatedAt": "2017-11-07T08:18:58.301396081Z",
"UpdatedAt": "2017-11-07T08:18:58.317939323Z",
"Spec": {
"Name": "visualizer_visualizer",
"Labels": {
"com.docker.stack.image": "dockersamples/visualizer:latest",
"com.docker.stack.namespace": "visualizer"
},
"TaskTemplate": {
"ContainerSpec": {
"Image": "dockersamples/visualizer:latest@sha256:3136807ad7b92fddc5eb45689df93c45907bb9047b52f5f5960a9605ed7536d1",
"Labels": {
"com.docker.stack.namespace": "visualizer"
},
"Privileges": {
"CredentialSpec": null,
"SELinuxContext": null
},
"Mounts": [
{
"Type": "bind",
"Source": "/etc/localtime",
"Target": "/etc/localtime",
"ReadOnly": true
},
{
"Type": "bind",
"Source": "/var/run/docker.sock",
"Target": "/var/run/docker.sock"
}
],
"StopGracePeriod": 10000000000,
"DNSConfig": {}
},
"Resources": {},
"RestartPolicy": {
"Condition": "any",
"Delay": 5000000000,
"MaxAttempts": 0
},
"Placement": {
"Constraints": [
"node.role == manager"
],
"Platforms": [
{
"Architecture": "amd64",
"OS": "linux"
}
]
},
"Networks": [
{
"Target": "erx5tuu11rjy003uvh6w6sbn6",
"Aliases": [
"visualizer"
]
}
],
"ForceUpdate": 0,
"Runtime": "container"
},
"Mode": {
"Replicated": {
"Replicas": 1
}
},
"UpdateConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"RollbackConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"EndpointSpec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 8080,
"PublishedPort": 8080,
"PublishMode": "ingress"
}
]
}
},
"Endpoint": {
"Spec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 8080,
"PublishedPort": 8080,
"PublishMode": "ingress"
}
]
},
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 8080,
"PublishedPort": 8080,
"PublishMode": "ingress"
}
],
"VirtualIPs": [
{
"NetworkID": "0nbrcqx4wrunt3nmc19p9ofzx",
"Addr": "10.255.0.5/16"
},
{
"NetworkID": "erx5tuu11rjy003uvh6w6sbn6",
"Addr": "10.0.0.2/24"
}
]
}
}
]
Service with logging options which does not work
version: "3.2"
services:
proxy:
image: vfarcic/docker-flow-proxy:latest
ports:
- 80:80
- 443:443
networks:
- proxy
volumes:
- /etc/localtime:/etc/localtime:ro
environment:
LISTENER_ADDRESS: "swarm-listener"
MODE: "swarm"
BIND_PORTS: "80,443"
deploy:
placement:
constraints: [node.role == manager]
logging:
options:
max-size: "50k"
swarm-listener:
image: vfarcic/docker-flow-swarm-listener:latest
networks:
- proxy
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
DF_NOTIFY_CREATE_SERVICE_URL: "http://proxy:8080/v1/docker-flow-proxy/reconfigure"
DF_NOTIFY_REMOVE_SERVICE_URL: "http://proxy:8080/v1/docker-flow-proxy/remove"
deploy:
placement:
constraints: [node.role == manager]
logging:
options:
max-size: "50k"
networks:
proxy:
external: true
root@docker-test-1:~# docker stack ps proxy
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
nxuv6pzsss66 proxy_swarm-listener.1 vfarcic/docker-flow-swarm-listener:latest Running Pending about a minute ago
bfhikxcambdc proxy_proxy.1 vfarcic/docker-flow-proxy:latest Running Pending about a minute ago
root@docker-test-1:~# docker service inspect proxy_swarm-listener
[
{
"ID": "rw7fnvo6tljp3rn1qtvjd8unk",
"Version": {
"Index": 43
},
"CreatedAt": "2017-11-07T08:25:33.884502748Z",
"UpdatedAt": "2017-11-07T08:25:33.898319569Z",
"Spec": {
"Name": "proxy_swarm-listener",
"Labels": {
"com.docker.stack.image": "vfarcic/docker-flow-swarm-listener:latest",
"com.docker.stack.namespace": "proxy"
},
"TaskTemplate": {
"ContainerSpec": {
"Image": "vfarcic/docker-flow-swarm-listener:latest@sha256:b2b741bc771b77f7eec7089beb8fcdd2666ad2ff0dfd6e8c150aae87ebc77e31",
"Labels": {
"com.docker.stack.namespace": "proxy"
},
"Env": [
"DF_NOTIFY_CREATE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/reconfigure",
"DF_NOTIFY_REMOVE_SERVICE_URL=http://proxy:8080/v1/docker-flow-proxy/remove"
],
"Privileges": {
"CredentialSpec": null,
"SELinuxContext": null
},
"Mounts": [
{
"Type": "bind",
"Source": "/var/run/docker.sock",
"Target": "/var/run/docker.sock"
}
],
"StopGracePeriod": 10000000000,
"DNSConfig": {}
},
"Resources": {},
"RestartPolicy": {
"Condition": "any",
"Delay": 5000000000,
"MaxAttempts": 0
},
"Placement": {
"Constraints": [
"node.role == manager"
],
"Platforms": [
{
"Architecture": "amd64",
"OS": "linux"
}
]
},
"Networks": [
{
"Target": "zfzpjffjhxgxcp6i7ob98ltlf",
"Aliases": [
"swarm-listener"
]
}
],
"LogDriver": {
"Options": {
"max-size": "50k"
}
},
"ForceUpdate": 0,
"Runtime": "container"
},
"Mode": {
"Replicated": {
"Replicas": 1
}
},
"UpdateConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"RollbackConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"EndpointSpec": {
"Mode": "vip"
}
},
"Endpoint": {
"Spec": {
"Mode": "vip"
},
"VirtualIPs": [
{
"NetworkID": "zfzpjffjhxgxcp6i7ob98ltlf",
"Addr": "10.0.1.4/24"
}
]
}
}
]
root@docker-test-1:~# docker service inspect proxy_proxy
[
{
"ID": "urgdd540ei8kyv0zuwqolt29n",
"Version": {
"Index": 38
},
"CreatedAt": "2017-11-07T08:25:31.055906108Z",
"UpdatedAt": "2017-11-07T08:25:31.068960768Z",
"Spec": {
"Name": "proxy_proxy",
"Labels": {
"com.docker.stack.image": "vfarcic/docker-flow-proxy:latest",
"com.docker.stack.namespace": "proxy"
},
"TaskTemplate": {
"ContainerSpec": {
"Image": "vfarcic/docker-flow-proxy:latest@sha256:88f01513409307b13017099c1b10b8a99e64807a7ea3b01efe3e37c05cd3afef",
"Labels": {
"com.docker.stack.namespace": "proxy"
},
"Env": [
"BIND_PORTS=80,443",
"LISTENER_ADDRESS=swarm-listener",
"MODE=swarm"
],
"Privileges": {
"CredentialSpec": null,
"SELinuxContext": null
},
"Mounts": [
{
"Type": "bind",
"Source": "/etc/localtime",
"Target": "/etc/localtime",
"ReadOnly": true
}
],
"StopGracePeriod": 10000000000,
"DNSConfig": {}
},
"Resources": {},
"RestartPolicy": {
"Condition": "any",
"Delay": 5000000000,
"MaxAttempts": 0
},
"Placement": {
"Constraints": [
"node.role == manager"
],
"Platforms": [
{
"Architecture": "amd64",
"OS": "linux"
}
]
},
"Networks": [
{
"Target": "zfzpjffjhxgxcp6i7ob98ltlf",
"Aliases": [
"proxy"
]
}
],
"LogDriver": {
"Options": {
"max-size": "50k"
}
},
"ForceUpdate": 0,
"Runtime": "container"
},
"Mode": {
"Replicated": {
"Replicas": 1
}
},
"UpdateConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"RollbackConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"EndpointSpec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 80,
"PublishedPort": 80,
"PublishMode": "ingress"
},
{
"Protocol": "tcp",
"TargetPort": 443,
"PublishedPort": 443,
"PublishMode": "ingress"
}
]
}
},
"Endpoint": {
"Spec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 80,
"PublishedPort": 80,
"PublishMode": "ingress"
},
{
"Protocol": "tcp",
"TargetPort": 443,
"PublishedPort": 443,
"PublishMode": "ingress"
}
]
},
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 80,
"PublishedPort": 80,
"PublishMode": "ingress"
},
{
"Protocol": "tcp",
"TargetPort": 443,
"PublishedPort": 443,
"PublishMode": "ingress"
}
],
"VirtualIPs": [
{
"NetworkID": "0nbrcqx4wrunt3nmc19p9ofzx",
"Addr": "10.255.0.7/16"
},
{
"NetworkID": "zfzpjffjhxgxcp6i7ob98ltlf",
"Addr": "10.0.1.2/24"
}
]
}
}
]
If i remove the logging options it's starting up.
morph027
on 7 Nov 2017
Also, if i define the driver, it works too.
logging:
driver: "json-file"
options:
max-size: "50k"
@morph027 thanks for the debugging test, I'm using syslog as logging driver in my clusters so it's definitely related to the logging plugin.
@nishanttotla But I configured syslog logging driver in my docker daemon local config, and it does show up in the node inspect
vide
on 7 Nov 2017
@morph027 @vide thanks for your responses. It indeed seems like the issue is with the logging driver. More particularly, from looking at the plugin filter, it seems like the issue is the missing name. The way the plugin filter works for log drivers is
if f.t.Spec.LogDriver != nil {
// If there are no log driver types in the list at all, most likely this is
// an older daemon that did not report this information. In this case don't filter
if typeFound, exists := f.pluginExistsOnNode("Log", f.t.Spec.LogDriver.Name, nodePlugins); !exists && typeFound {
return false
}
}
So it seems like f.t.Spec.LogDriver.Name which is supposed to be "syslog" is being sent as an empty field. This is clear from the service spec.
cc @cpuguy83 @vieux does this need fixing on the daemon side?
nishanttotla
on 7 Nov 2017
@nishanttotla I don't see any Name field in the LogDriver in the inspects pasted above, is it normal ?
vieux
on 7 Nov 2017
@vieux that's my question too. The object does have a Name field, but clearly it's not being populated in the inspects above. The service spec comes from the daemon, which is why I'm wondering why the name isn't being included there.
nishanttotla
on 7 Nov 2017
oh ok I get it, I'll take a look.
vieux
on 7 Nov 2017
I think swarm should handle this case.
This is basically delegating the log driver to whatever the default on the target node is.
Having dockerd set this default for the service spec doesn't seem quite right to me.
cpuguy83
on 8 Nov 2017
Specifically, the filter should have a special case for an empty driver name.
cpuguy83
on 8 Nov 2017
@cpuguy83 thanks for clarifying that, this is something I was wondering about. I'm not sure why we have the filter designed as it is for log drivers, but I'll submit a PR to take care of empty driver names.
nishanttotla
on 8 Nov 2017
@nishanttotla
I'm not sure why we have the filter designed as it is for log drivers
Because we support plugins for log drivers.
cpuguy83
on 8 Nov 2017
I have a windows server 2019 Docker installation with experimental LCOW based Linux containers setup. When I run the application with Docker compose up it runs fine on all the machine including the manager machine. But when I deploy to stack the nodes do not start. Looking at the stack logs I see no suitable node (unsupported platform on all nodes). Does this mean on the compose file some kind of environment is needed to be set?
Here is a link to the repository
https://github.com/shakeelosmani/dockermultiplewebapps/blob/master/docker-compose-stack-deploy.yml
shakeelosmani
on 22 Apr 2020
Related issues
dongluochen
路
10Comments
allencloud
路
7Comments
nishanttotla
路
6Comments
inetdavid
路
4Comments
vieux
路
9Comments
Most helpful comment
Same here. I only use the default docker-ce.
This happens in live service server.
The project of my team will move on live service state in three weeks, but I've found a ton of critical Docker bugs which cannot be accepted on live service.
Especially, most bugs are related to swarm mode network, discovery, or something like this bug.
This is a stupid bug. If you don't have any good programmer in kernel or system side, why don't you hire ppl?
It seems docker swarm mode is not matured yet. Maybe it can be so next 2 or 3 years.
This is the last line my team can make a decision. With high possibility, we will go back to docker swarm.
With COMPLETLY Clean Ubuntu 16.0x, installed docker-ce 17.09, not upgraded.
22 Nodes (Backend)
"Status": {
"Timestamp": "2017-11-02T03:59:39.190016Z",
"State": "pending",
"Message": "missing plugin on 1 node",
"ContainerStatus": {},
"PortStatus": {}
},