Libnetwork: starting container failed: No such network
Tried setting up docker service in a docker swarm consisting of three nodes. After deploying service does not start with docker inspect showing error message given as subject.
OS: Ubuntu 16.04.3
docker version:
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:18 2017
OS/Arch: linux/amd64
Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:40:56 2017
OS/Arch: linux/amd64
Experimental: false
docker info:
Containers: 179
Running: 0
Paused: 0
Stopped: 179
Images: 5
Server Version: 17.09.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: x7d8n8g52ag6tf07camgh6sb9
Is Manager: true
ClusterID: b8dke0zbjde7dl2k0eqe9roib
Managers: 3
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 88.99.123.129
Manager Addresses:
138.201.189.209:2377
88.99.123.129:2377
94.130.185.164:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-98-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.765GiB
Name: node3
ID: NRAH:VWTG:NUSQ:NFB3:L2KS:O7YT:6SRW:FQYV:OO62:DSXA:KD3K:IGZ6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
docker node ls:
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
cpsi2ewjcx7qlu2lpjw9fsb2h node1 Ready Active Leader
pq7v3piqgqzkj3l7kvx0w7uzi node2 Ready Active Reachable
x7d8n8g52ag6tf07camgh6sb9 * node3 Ready Active Reachable
docker network ls:
NETWORK ID NAME DRIVER SCOPE
bf95d5d3539b bridge bridge local
mto9swbvn9v2 cepharum_ssh_default overlay swarm
0478c8f99b75 docker_gwbridge bridge local
0771dc108ce1 host host local
qo5dp6dr295m ingress overlay swarm
1e50251be4d4 none null local
docker service ps cepharum_ssh_ssh:
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
npuf3t78r9qu cepharum_ssh_ssh.1 cepharum/ssh:latest node3 Ready Ready 4 seconds ago
zrlxkn7dfeca \_ cepharum_ssh_ssh.1 cepharum/ssh:latest node1 Shutdown Failed 23 minutes ago "starting container failed: No…"
zz6rpl53fm2u \_ cepharum_ssh_ssh.1 cepharum/ssh:latest node1 Shutdown Complete about an hour ago
zx8pwdbcxgcf \_ cepharum_ssh_ssh.1 cepharum/ssh:latest node2 Shutdown Failed 24 minutes ago "starting container failed: No…"
zz4sdsei67ay \_ cepharum_ssh_ssh.1 cepharum/ssh:latest node3 Shutdown Complete 2 hours ago
docker inspect zrlxkn7dfeca:
[
{
"ID": "zrlxkn7dfecak4v74tuaw2pqs",
"Version": {
"Index": 49140
},
"CreatedAt": "2017-11-19T18:30:45.871194263Z",
"UpdatedAt": "2017-11-19T18:30:51.310932181Z",
"Labels": {},
"Spec": {
"ContainerSpec": {
"Image": "cepharum/ssh:latest@sha256:72fbbe17b489dd1f6f15ccdd1c8d6ba5d24cdac067b1c12380f5652865cedfdd",
"Labels": {
"com.docker.stack.namespace": "cepharum_ssh"
},
"Privileges": {
"CredentialSpec": null,
"SELinuxContext": null
},
"Mounts": [
{
"Type": "volume",
"Source": "cepharum_ssh_users",
"Target": "/home",
"VolumeOptions": {
"Labels": {
"com.docker.stack.namespace": "cepharum_ssh"
}
}
},
{
"Type": "volume",
"Source": "cepharum_ssh_registry",
"Target": "/etc/registry",
"ReadOnly": true,
"VolumeOptions": {
"Labels": {
"com.docker.stack.namespace": "cepharum_ssh"
}
}
}
]
},
"Resources": {},
"Placement": {
"Platforms": [
{
"Architecture": "amd64",
"OS": "linux"
}
]
},
"Networks": [
{
"Target": "mto9swbvn9v26bwbzuxbjled1",
"Aliases": [
"ssh"
]
}
],
"ForceUpdate": 0
},
"ServiceID": "454caf7y28a6ec1r17bxqvddn",
"Slot": 1,
"NodeID": "cpsi2ewjcx7qlu2lpjw9fsb2h",
"Status": {
"Timestamp": "2017-11-19T18:30:51.043942362Z",
"State": "failed",
"Message": "starting",
"Err": "starting container failed: No such network: cepharum_ssh_default",
"ContainerStatus": {
"ContainerID": "de15d63f987a49a68c5d12d15b4c65d1468d566346160391a96a7200103d1296",
"ExitCode": 128
},
"PortStatus": {}
},
"DesiredState": "shutdown",
"NetworksAttachments": [
{
"Network": {
"ID": "qo5dp6dr295mfag9ef0r65rdt",
"Version": {
"Index": 48611
},
"CreatedAt": "2017-10-30T12:05:33.398078736Z",
"UpdatedAt": "2017-11-19T18:25:24.111819005Z",
"Spec": {
"Name": "ingress",
"Labels": {},
"DriverConfiguration": {},
"Ingress": true,
"IPAMOptions": {
"Driver": {},
"Configs": [
{
"Subnet": "10.255.0.0/16",
"Gateway": "10.255.0.1"
}
]
},
"Scope": "swarm"
},
"DriverState": {
"Name": "overlay",
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4096"
}
},
"IPAMOptions": {
"Driver": {
"Name": "default"
},
"Configs": [
{
"Subnet": "10.255.0.0/16",
"Gateway": "10.255.0.1"
}
]
}
},
"Addresses": [
"10.255.0.6/16"
]
},
{
"Network": {
"ID": "mto9swbvn9v26bwbzuxbjled1",
"Version": {
"Index": 48612
},
"CreatedAt": "2017-11-19T15:11:13.266600832Z",
"UpdatedAt": "2017-11-19T18:25:24.117529331Z",
"Spec": {
"Name": "cepharum_ssh_default",
"Labels": {
"com.docker.stack.namespace": "cepharum_ssh"
},
"DriverConfiguration": {
"Name": "overlay"
},
"Scope": "swarm"
},
"DriverState": {
"Name": "overlay",
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
}
},
"IPAMOptions": {
"Driver": {
"Name": "default"
},
"Configs": [
{
"Subnet": "10.0.0.0/24",
"Gateway": "10.0.0.1"
}
]
}
},
"Addresses": [
"10.0.0.3/24"
]
}
]
}
]
excerpt from syslog:
syslog-excerpt.txt
ifconfig:
docker0 Link encap:Ethernet HWaddr 02:42:af:7d:e5:8e
inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
docker_gwbridge Link encap:Ethernet HWaddr 02:42:b3:3e:db:c8
inet addr:172.18.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
inet6 addr: fe80::42:b3ff:fe3e:dbc8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:648 (648.0 B)
eth0 Link encap:Ethernet HWaddr 96:00:00:03:22:ed
inet addr:88.99.123.129 Bcast:88.99.123.129 Mask:255.255.255.255
inet6 addr: 2a01:4f8:1c17:419d::1/64 Scope:Global
inet6 addr: fe80::9400:ff:fe03:22ed/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:53329 errors:0 dropped:0 overruns:0 frame:0
TX packets:52317 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:14672174 (14.6 MB) TX bytes:11342796 (11.3 MB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:6648 errors:0 dropped:0 overruns:0 frame:0
TX packets:6648 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:1011784 (1.0 MB) TX bytes:1011784 (1.0 MB)
veth8e7a84d Link encap:Ethernet HWaddr 7a:96:ab:a0:92:fc
inet6 addr: fe80::7896:abff:fea0:92fc/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:1296 (1.2 KB)
soletan
👍2
All 10 comments
fixed by: https://github.com/moby/moby/pull/35634
fcrisciani
on 29 Nov 2017
👍1
save issue
swarm mode
3 nodes
3 services using same overlay network
reboot two and reboot the last
two services are running and one down
llitfkitfk
on 1 Dec 2017
I've encountered the same issue in the past week all of a sudden. One service has been frequently returning starting container failed: No such network error on startup. This is happening in a production environment, so any immediate advice would be appreciated.
Docker version
Containers: 25
Running: 6
Paused: 0
Stopped: 19
Images: 53
Server Version: 17.09.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: awslogs
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: [redacted]
Is Manager: true
ClusterID: [redacted]
Managers: 3
Nodes: 8
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: [redacted]
Manager Addresses:
[redacted]
[redacted]
[redacted]
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.49-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.293GiB
Name: [redacted]
ID: [redacted]
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 163
Goroutines: 442
System Time: 2017-12-07T22:46:52.476390285Z
EventsListeners: 2
Username: [redacted]
Registry: https://index.docker.io/v1/
Labels:
os=linux
region=us-east-1
availability_zone=us-east-1c
instance_type=c4.xlarge
node_type=manager
Experimental: true
Insecure Registries:
[redacted]
Live Restore Enabled: false
docker service ps [servicename]
stanleegoodspeed
on 8 Dec 2017
👍6
I'm having a similar issue, I think. I'm going to attach my docker.log, but below are some excerpts. For me, Docker crashes and I can't use any of my containers right now.
docker version:
Client:
Version: 17.11.0-ce
API version: 1.34
Go version: go1.9.2
Git commit: 1caf76c
Built: Fri Dec 15 17:57:57 2017
OS/Arch: linux/amd64
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
docker info:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
docker.log excerpt:
time="2017-12-20T10:54:15.511016846-05:00" level=info msg="libcontainerd: started new docker-containerd process" pid=4052
time="2017-12-20T10:54:15-05:00" level=info msg="starting containerd" module=containerd revision=992280 version="1.0.0_beta2_p20171019"
time="2017-12-20T10:54:15-05:00" level=info msg="changing OOM score to -500" module=containerd
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.content.v1.content"..." module=containerd type=io.containerd.content.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.btrfs"..." module=containerd type=io.containerd.snapshotter.v1
time="2017-12-20T10:54:15-05:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.btrfs" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." module=containerd type=io.containerd.snapshotter.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.metadata.v1.bolt"..." module=containerd type=io.containerd.metadata.v1
time="2017-12-20T10:54:15-05:00" level=warning msg="could not use snapshotter btrfs in metadata plugin" error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." module=containerd type=io.containerd.differ.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." module=containerd type=io.containerd.grpc.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." module=containerd type=io.containerd.grpc.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." module=containerd type=io.containerd.grpc.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." module=containerd type=io.containerd.grpc.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." module=containerd type=io.containerd.grpc.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." module=containerd type=io.containerd.grpc.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." module=containerd type=io.containerd.grpc.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." module=containerd type=io.containerd.grpc.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." module=containerd type=io.containerd.monitor.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." module=containerd type=io.containerd.runtime.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." module=containerd type=io.containerd.grpc.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." module=containerd type=io.containerd.grpc.v1
time="2017-12-20T10:54:15-05:00" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." module=containerd type=io.containerd.grpc.v1
time="2017-12-20T10:54:15-05:00" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
time="2017-12-20T10:54:15-05:00" level=info msg=serving... address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
time="2017-12-20T10:54:15-05:00" level=info msg="containerd successfully booted in 0.015213s" module=containerd
time="2017-12-20T10:54:15.777009796-05:00" level=warning msg="devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man dockerd` to refer to dm.thinpooldev section."
time="2017-12-20T10:54:15.804027843-05:00" level=warning msg="devmapper: Base device already exists and has filesystem ext4 on it. User specified filesystem will be ignored."
time="2017-12-20T10:54:15.815641948-05:00" level=info msg="[graphdriver] using prior storage driver: devicemapper"
time="2017-12-20T10:54:15.938943462-05:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2017-12-20T10:54:15.940030623-05:00" level=info msg="Loading containers: start."
time="2017-12-20T10:54:16.502948095-05:00" level=info msg="Removing stale sandbox a4dc0ff5d82c9e0588333558d73277cb2711b8f7ab43f766202ae91744969b6f (portainer_default-sbox)"
time="2017-12-20T10:54:16.511059543-05:00" level=error msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint ee3b279aafb28d44c8a8d67240f5220b853201b89dca14af7a96bbdbb3555463 a6b99a4e36d53a39f42591d743b7772a207f5d6854796041a4c173d7a707dc73], retrying...."
time="2017-12-20T10:54:16.646603313-05:00" level=info msg="Removing stale sandbox ingress_sbox (ingress-sbox)"
time="2017-12-20T10:54:16.654128511-05:00" level=error msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint ee3b279aafb28d44c8a8d67240f5220b853201b89dca14af7a96bbdbb3555463 9f059b1c9b2af8998c4f48edd6709c217e991cc25fa7defd513532a746594556], retrying...."
time="2017-12-20T10:54:16.670891455-05:00" level=info msg="There are old running containers, the network config will not take affect"
time="2017-12-20T10:54:16.710816259-05:00" level=info msg="Loading containers: done."
time="2017-12-20T10:54:16.731909156-05:00" level=info msg="Docker daemon" commit=1caf76c graphdriver(s)=devicemapper version=17.11.0-ce
time="2017-12-20T10:54:16.776642585-05:00" level=info msg="Listening for local connections" addr=/var/run/docker/swarm/control.sock module=node node.id=xyvuvw9yaohqzivpynjs2tmlz proto=unix
time="2017-12-20T10:54:16.776676666-05:00" level=info msg="Listening for connections" addr="[::]:2377" module=node node.id=xyvuvw9yaohqzivpynjs2tmlz proto=tcp
time="2017-12-20T10:54:17.047781669-05:00" level=info msg="5d0b3d8eaf5aa84e became follower at term 18" module=raft node.id=xyvuvw9yaohqzivpynjs2tmlz
time="2017-12-20T10:54:17.047826483-05:00" level=info msg="newRaft 5d0b3d8eaf5aa84e [peers: [5d0b3d8eaf5aa84e], term: 18, commit: 15789, applied: 10000, lastindex: 15789, lastterm: 18]" module=raft node.id=xyvuvw9yaohqzivpynjs2tmlz
(REPEATED 5786 TIMES with different request IDs)
time="2017-12-20T10:54:17.050897928-05:00" level=error msg="wait not found for raft request id 5fa69e586f59" raft_id=5d0b3d8eaf5aa84e
time="2017-12-20T10:54:18.126705871-05:00" level=info msg="5d0b3d8eaf5aa84e is starting a new election at term 18" module=raft node.id=xyvuvw9yaohqzivpynjs2tmlz
time="2017-12-20T10:54:18.127148310-05:00" level=info msg="5d0b3d8eaf5aa84e became candidate at term 19" module=raft node.id=xyvuvw9yaohqzivpynjs2tmlz
time="2017-12-20T10:54:18.127168728-05:00" level=info msg="5d0b3d8eaf5aa84e received MsgVoteResp from 5d0b3d8eaf5aa84e at term 19" module=raft node.id=xyvuvw9yaohqzivpynjs2tmlz
time="2017-12-20T10:54:18.127184985-05:00" level=info msg="5d0b3d8eaf5aa84e became leader at term 19" module=raft node.id=xyvuvw9yaohqzivpynjs2tmlz
time="2017-12-20T10:54:18.127195550-05:00" level=info msg="raft.node: 5d0b3d8eaf5aa84e elected leader 5d0b3d8eaf5aa84e at term 19" module=raft node.id=xyvuvw9yaohqzivpynjs2tmlz
time="2017-12-20T10:54:18.127944325-05:00" level=error msg="failed to create predefined network bridge" error="name conflicts with an existing object" module=node node.id=xyvuvw9yaohqzivpynjs2tmlz
time="2017-12-20T10:54:18.127967742-05:00" level=error msg="failed to create predefined network host" error="name conflicts with an existing object" module=node node.id=xyvuvw9yaohqzivpynjs2tmlz
time="2017-12-20T10:54:18.842425882-05:00" level=info msg="Initializing Libnetwork Agent Listen-Addr=0.0.0.0 Local-addr=192.168.1.145 Adv-addr=192.168.1.145 Data-addr= Remote-addr-list=[] MTU=1500"
time="2017-12-20T10:54:18.843979912-05:00" level=info msg="Node b93b772bb179/192.168.1.145, joined gossip cluster"
time="2017-12-20T10:54:18.844040243-05:00" level=info msg="Node b93b772bb179/192.168.1.145, added to nodes list"
time="2017-12-20T10:54:18.848346640-05:00" level=info msg="Daemon has completed initialization"
time="2017-12-20T10:54:18.855379855-05:00" level=info msg="API listen on /var/run/docker.sock"
time="2017-12-20T10:54:19-05:00" level=error msg="enabling default vlan on bridge br0 failed open /sys/class/net/br0/bridge/default_pvid: permission denied"
time="2017-12-20T10:54:19.081869918-05:00" level=error msg="reexec to set bridge default vlan failed exit status 1"
time="2017-12-20T10:54:19.137420003-05:00" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 74f1647fe75c7aace0b862eb5073b733ca6c6471dc20dd73e814cec25512a61e hzbseotexo3aqqybmnf5od166 err:cannot delete entry as the entry in table endpoint_table with network id hzbseotexo3aqqybmnf5od166 and key 74f1647fe75c7aace0b862eb5073b733ca6c6471dc20dd73e814cec25512a61e does not exist"
time="2017-12-20T10:54:19.203104358-05:00" level=error msg="ddb12b3e2a093d63a1367a32f8927588260f7f64592aa523364ef176eff9aec3 cleanup: failed to delete container from containerd: no such container"
time="2017-12-20T10:54:19.203162961-05:00" level=error msg="fatal task error" error="starting container failed: No such network: portainer_default" module=node/agent/worker/taskmanager node.id=xyvuvw9yaohqzivpynjs2tmlz service.id=7pu7mjbiqdn5iqbjbuytul97s task.id=vgohg01cjjggcpqm2aykqhev0
time="2017-12-20T10:54:19-05:00" level=error msg="enabling default vlan on bridge br0 failed open /sys/class/net/br0/bridge/default_pvid: permission denied"
time="2017-12-20T10:54:19.663888260-05:00" level=error msg="reexec to set bridge default vlan failed exit status 1"
(REPEATED 23 times)
time="2017-12-20T10:54:21.946773618-05:00" level=error msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint p0mfnt3ome6iqb7ejffkbpbm9 e26d55c388be9f1187e2d3b0cf57a42e90604732312437f8af395e6121be5fe1], retrying...."
time="2017-12-20T10:54:24.423117637-05:00" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 2ee71844dadab80ccd11d3928bdb916b0d72752a0ddc1713970aed29d5f53b35 hzbseotexo3aqqybmnf5od166 err:cannot delete entry as the entry in table endpoint_table with network id hzbseotexo3aqqybmnf5od166 and key 2ee71844dadab80ccd11d3928bdb916b0d72752a0ddc1713970aed29d5f53b35 does not exist"
time="2017-12-20T10:54:24.423171297-05:00" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster portainer_portainer 2ee71844dadab80ccd11d3928bdb916b0d72752a0ddc1713970aed29d5f53b35 aborted c.serviceBindings[skey] !ok"
time="2017-12-20T10:54:24.484201822-05:00" level=error msg="85b5a0be2f7fb7e3e69f3af576f1be672d8bff66d7eacf7d3099b24215fa5828 cleanup: failed to delete container from containerd: no such container"
time="2017-12-20T10:54:24.484298902-05:00" level=error msg="fatal task error" error="starting container failed: No such network: portainer_default" module=node/agent/taskmanager node.id=xyvuvw9yaohqzivpynjs2tmlz service.id=7pu7mjbiqdn5iqbjbuytul97s task.id=k3rr51ugm0wxg3e6zmt36bkpn
time="2017-12-20T10:54:24-05:00" level=error msg="enabling default vlan on bridge br0 failed open /sys/class/net/br0/bridge/default_pvid: permission denied"
time="2017-12-20T10:54:24.872893783-05:00" level=error msg="reexec to set bridge default vlan failed exit status 1"
unexpected fault address 0x40fe83f8
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40fe83f8 pc=0xfe83e7]
johnmanko
on 20 Dec 2017
Similar problem here. First it was because of IP address reuse, so we upgraded from 17.09 to 17.10, and now that problem is solved but we're still having issues with overlay networks created for stacks.
deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 59943759fe38b7ff7fcddb722ea740089303f698555a7f48163f32bb70fcbb1f adwz1i3k94yb9d076a4qdss9e err:cannot delete entry as the entry in table endpoint_table with network id adwz1i3k94yb9d076a4qdss9e and key 59943759fe38b7ff7fcddb722ea740089303f698555a7f48163f32bb70fcbb1f does not exist
Dec 29 16:58:12 kafka0 dockerd[1097]: time="2017-12-29T16:58:12.489914938Z" level=error msg="fatal task error" error="starting container failed: No such network: kafka_default" module=node/agent/taskmanager node.id=ecnbx5evaqnipyklmee3ramc8 service.id=4b3est31dqbzza6jkw146orb4 task.id=ygfbav2fetlt8jru34cul0lji
Output from journalctl
Dec 29 16:58:09 kafka0 dockerd[1097]: time="2017-12-29T16:58:09.051766020Z" level=info msg="Node join event for e4ce31a6d326/172.31.100.143"
Dec 29 16:58:12 kafka0 dockerd[1097]: time="2017-12-29T16:58:12.403667722Z" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 59943759fe38b7ff7fcddb722ea740089303f698555a7f48163f32bb70fcbb1f adwz1i3k94yb9d076a4qdss9e err:cannot delete entry as the entry in table endpoint_table with network id adwz1i3k94yb9d076a4qdss9e and key 59943759fe38b7ff7fcddb722ea740089303f698555a7f48163f32bb70fcbb1f does not exist"
Dec 29 16:58:12 kafka0 dockerd[1097]: time="2017-12-29T16:58:12.403712495Z" level=warning msg="rmServiceBinding deleteServiceInfoFromCluster kafka_kafka-1 59943759fe38b7ff7fcddb722ea740089303f698555a7f48163f32bb70fcbb1f aborted c.serviceBindings[skey] !ok"
Dec 29 16:58:12 kafka0 dockerd[1097]: time="2017-12-29T16:58:12.489914938Z" level=error msg="fatal task error" error="starting container failed: No such network: kafka_default" module=node/agent/taskmanager node.id=ecnbx5evaqnipyklmee3ramc8 service.id=4b3est31dqbzza6jkw146orb4 task.id=ygfbav2fetlt8jru34cul0lji
Dec 29 16:58:12 kafka0 dockerd[1097]: time="2017-12-29T16:58:12.842323231Z" level=warning msg="rmServiceBinding handleEpTableEvent kafka_kafka-2 3c6136f784f4636e3a8e9e66a5c17a04999772883012aa7ccc7cb138924534ca aborted c.serviceBindings[skey] !ok"
Dec 29 16:58:12 kafka0 dockerd[1097]: time="2017-12-29T16:58:12.842405469Z" level=warning msg="rmServiceBinding handleEpTableEvent kafka_kafka-2 40a7c8d1fa96a493f07f09a21ece8b6d017478efc590d87bc18b4935089c550b aborted c.serviceBindings[skey] !ok"
Dec 29 16:58:12 kafka0 dockerd[1097]: time="2017-12-29T16:58:12.874109224Z" level=warning msg="rmServiceBinding handleEpTableEvent kafka_kafka-2 00523277e680880b8c37b809925d5f8c9b5b29d1f13c87760de867d29e8efcf5 aborted c.serviceBindings[skey] !ok"
Dec 29 16:58:12 kafka0 dockerd[1097]: time="2017-12-29T16:58:12.875150026Z" level=warning msg="rmServiceBinding handleEpTableEvent kafka_kafka-2 3c6136f784f4636e3a8e9e66a5c17a04999772883012aa7ccc7cb138924534ca aborted c.serviceBindings[skey] !ok"
Dec 29 16:58:12 kafka0 dockerd[1097]: time="2017-12-29T16:58:12.875386070Z" level=warning msg="rmServiceBinding handleEpTableEvent kafka_kafka-2 40a7c8d1fa96a493f07f09a21ece8b6d017478efc590d87bc18b4935089c550b aborted c.serviceBindings[skey] !ok"
Dec 29 16:58:12 kafka0 dockerd[1097]: time="2017-12-29T16:58:12.875979208Z" level=warning msg="rmServiceBinding handleEpTableEvent kafka_kafka-2 614765452a142cff786619fd8e5abeafe283703ebee01d6cf658cd5ebe536cc0 aborted c.serviceBindings[skey] !ok"
Dec 29 16:58:12 kafka0 dockerd[1097]: time="2017-12-29T16:58:12.877931833Z" level=warning msg="rmServiceBinding handleEpTableEvent kafka_kafka-2 cd76ecd09516b2b4e8981dee7cb123b1bc9ac1a9945a31c2e57aa26da79687f1 aborted c.serviceBindings[skey] !ok"
The strange thing is that this network ID adwz1i3k94yb9d076a4qdss9e is our ingress network and it does exist. So does the kafka_default network. Because network ls on that particular node, outputs...
ubuntu@kafka0:~$ sudo docker network ls
NETWORK ID NAME DRIVER SCOPE
c4a8def3d7d8 bridge bridge local
e54b62f8cfd0 docker_gwbridge bridge local
1a577598008e host host local
adwz1i3k94yb ingress overlay swarm
38x8evaixyyp kafka_default overlay swarm
a0555d01cda4 none null local
gaui
on 29 Dec 2017
@gaui can you try with 17.12?
fcrisciani
on 29 Dec 2017
Considering there is no more updates, will close this one. 17.12 should be fine.
Feel free to reopen if there is other occurrences on 17.12 or above
fcrisciani
on 29 Jan 2018
@fcrisciani I would say close. This resolved itself for us upon system reboot.
johnmanko
on 31 Jan 2018
I'm experiencing a similar problem on 18.06.1, some services began to fail occasionally upon redeploy with:
starting container failed: failed to get network during CreateEndpoint: network sa8m3w8pqi0tqzro06naftwbu not found
This is docker log from a worker node, the 4th line looks weird, as it indicates some network removal operations that should not happen:
time="2019-02-22T18:50:12.298220240+08:00" level=info msg="NetworkDB stats worker-04(03ad71380621) - netID:sa8m3w8pqi0tqzro06naftwbu leaving:false netPeers:6 entries:77 Queue qLen:0 netMsg/s:0"
time="2019-02-22T18:55:12.498417669+08:00" level=info msg="NetworkDB stats worker-04(03ad71380621) - netID:sa8m3w8pqi0tqzro06naftwbu leaving:false netPeers:6 entries:61 Queue qLen:0 netMsg/s:0"
time="2019-02-22T18:57:15.646497195+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint sa8m3w8pqi0tqzro06naftwbu 400a306d13ff57f7c0b773f692fb4c1399500c821a9fd2f7b7172fcbbebb903d], retrying...."
time="2019-02-22T18:57:15.728256035+08:00" level=error msg="network xxx_default remove failed: unknown network xxx_default id sa8m3w8pqi0tqzro06naftwbu" module=node/agent node.id=xpoaymvry45sb5q888o1jpoi8
time="2019-02-22T18:57:15.728298569+08:00" level=error msg="remove task failed" error="unknown network xxx_default id sa8m3w8pqi0tqzro06naftwbu" module=node/agent node.id=xpoaymvry45sb5q888o1jpoi8 task.id=5mav3lpsdc7gk8hpwgnyvtbr4
time="2019-02-22T18:57:15.768893162+08:00" level=error msg="fatal task error" error="starting container failed: failed to get network during CreateEndpoint: network sa8m3w8pqi0tqzro06naftwbu not found" module=node/agent/taskmanager node.id=xpoaymvry45sb5q888o1jpoi8 service.id=2x43w0s1gvxkzu4b8qw4wk3y2 task.id=kdyljmb8isyp02cqaq5dqsdnu
time="2019-02-22T18:57:16.045314219+08:00" level=error msg="fatal task error" error="starting container failed: failed to get network during CreateEndpoint: network sa8m3w8pqi0tqzro06naftwbu not found" module=node/agent/taskmanager node.id=xpoaymvry45sb5q888o1jpoi8 service.id=vdputsk4p8vf6a3klda75od44 task.id=j3l9cc5job0ih3g7eokgp73mk
time="2019-02-22T18:57:18.861072689+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint sa8m3w8pqi0tqzro06naftwbu 1c327c91e28c1024b23384b52d16861b8593c002c0312975d4ebac6e5953a962], retrying...."
time="2019-02-22T18:57:40.752379324+08:00" level=warning msg="deleteServiceInfoFromCluster NetworkDB DeleteEntry failed for 655f0010bee9de626f131bfbc629da4d0b79c84227709ac924921c8e7b900c58 sa8m3w8pqi0tqzro06naftwbu err:cannot delete entry endpoint_table with network id sa8m3w8pqi0tqzro06naftwbu and key 655f0010bee9de626f131bfbc629da4d0b79c84227709ac924921c8e7b900c58 does not exist or is already being deleted"
time="2019-02-22T19:00:12.699101399+08:00" level=info msg="NetworkDB stats worker-04(03ad71380621) - netID:sa8m3w8pqi0tqzro06naftwbu leaving:false netPeers:6 entries:64 Queue qLen:0 netMsg/s:0"
ushuz
on 22 Feb 2019
Does this have something to do with SSL certificates' hosts IP addresses getting in the way? Maybe specifying a network IP address on the Docker bridge when it shouldn't, specifically when adding IPs to the extfile.cnf?
ghost
on 26 Jul 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
bauerm97
·
5Comments
kvaps
·
3Comments
gauravphoenix
·
5Comments
Grimeton
·
7Comments
TheCatFelix
·
4Comments
Most helpful comment
I've encountered the same issue in the past week all of a sudden. One service has been frequently returning
starting container failed: No such networkerror on startup. This is happening in a production environment, so any immediate advice would be appreciated.Docker version
docker service ps [servicename]
