Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
If container is created using API with name and image name as parameters, container fails to start with an error: args must not be empty: OCI runtime error.
Steps to reproduce the issue:
sudo podman pull testcontainersofficial/ryuk:0.3.0
curl -X POST -H "Content-Type: application/json" 'http://127.0.0.1:1213/containers/create?name=tester' --data '{"name":"tester","Image":"testcontainersofficial/ryuk:0.3.0"}'
sudo podman start -a
Compare results to:
sudo podman create --name tester2 testcontainersofficial/ryuk:0.3.0
sudo podman start -a
Describe the results you received:
Error: unable to start container 2fb9584fcc8e089e9dc95e38ff79e2b9d5036f0f34ec45a1e885d5a3a8862bab: args must not be empty: OCI runtime error
Describe the results you expected:
Container starts successfully (or fails to missing docker socket on start up, if you use ryuk)
Additional information you deem important (e.g. issue happens only occasionally):
Podman inspect shows differences in configuration when comparing container created with API to one created using podman create command.
Output of podman version:
Version: 2.0.0
API Version: 1
Go Version: go1.13.8
Built: Thu Jan 1 02:00:00 1970
OS/Arch: linux/amd64
Output of podman info --debug:
host:
arch: amd64
buildahVersion: 1.15.0
cgroupVersion: v1
conmon:
package: 'conmon: /usr/libexec/podman/conmon'
path: /usr/libexec/podman/conmon
version: 'conmon version 2.0.18, commit: '
cpus: 4
distribution:
distribution: ubuntu
version: "20.04"
eventLogger: file
hostname: ubuntu
idMappings:
gidmap: null
uidmap: null
kernel: 5.4.0-37-generic
linkmode: dynamic
memFree: 1835118592
memTotal: 8348520448
ociRuntime:
name: runc
package: 'containerd.io: /usr/bin/runc'
path: /usr/bin/runc
version: |-
runc version 1.0.0-rc10
commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
spec: 1.0.1-dev
os: linux
remoteSocket:
path: /run/podman/podman.sock
rootless: false
slirp4netns:
executable: ""
package: ""
version: ""
swapFree: 1551257600
swapTotal: 1964396544
uptime: 94h 2m 36.41s (Approximately 3.92 days)
registries:
search:
- docker.io
- quay.io
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 4
paused: 0
running: 0
stopped: 4
graphDriverName: overlay
graphOptions: {}
graphRoot: /var/lib/containers/storage
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 1
runRoot: /var/run/containers/storage
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 1
Built: 0
BuiltTime: Thu Jan 1 02:00:00 1970
GitCommit: ""
GoVersion: go1.13.8
OsArch: linux/amd64
Version: 2.0.0
Inspect results from image created with API
[
{
"Id": "2fb9584fcc8e089e9dc95e38ff79e2b9d5036f0f34ec45a1e885d5a3a8862bab",
"Created": "2020-06-27T17:40:09.128835551+03:00",
"Path": "",
"Args": null,
"State": {
"OciVersion": "1.0.2-dev",
"Status": "configured",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "0001-01-01T00:00:00Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Healthcheck": {
"Status": "",
"FailingStreak": 0,
"Log": null
}
},
"Image": "344c0229fe447209d01c388035dfe01ca38f5289d99c7e14febbcb9a0b128775",
"ImageName": "testcontainersofficial/ryuk:0.3.0",
"Rootfs": "",
"Pod": "",
"ResolvConfPath": "/var/run/containers/storage/overlay-containers/2fb9584fcc8e089e9dc95e38ff79e2b9d5036f0f34ec45a1e885d5a3a8862bab/userdata/resolv.conf",
"HostnamePath": "/var/run/containers/storage/overlay-containers/2fb9584fcc8e089e9dc95e38ff79e2b9d5036f0f34ec45a1e885d5a3a8862bab/userdata/hostname",
"HostsPath": "/var/run/containers/storage/overlay-containers/2fb9584fcc8e089e9dc95e38ff79e2b9d5036f0f34ec45a1e885d5a3a8862bab/userdata/hosts",
"StaticDir": "/var/lib/containers/storage/overlay-containers/2fb9584fcc8e089e9dc95e38ff79e2b9d5036f0f34ec45a1e885d5a3a8862bab/userdata",
"OCIConfigPath": "/var/lib/containers/storage/overlay-containers/2fb9584fcc8e089e9dc95e38ff79e2b9d5036f0f34ec45a1e885d5a3a8862bab/userdata/config.json",
"OCIRuntime": "runc",
"LogPath": "/var/lib/containers/storage/overlay-containers/2fb9584fcc8e089e9dc95e38ff79e2b9d5036f0f34ec45a1e885d5a3a8862bab/userdata/ctr.log",
"LogTag": "",
"ConmonPidFile": "/var/run/containers/storage/overlay-containers/2fb9584fcc8e089e9dc95e38ff79e2b9d5036f0f34ec45a1e885d5a3a8862bab/userdata/conmon.pid",
"Name": "tester",
"RestartCount": 0,
"Driver": "overlay",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "container-default",
"EffectiveCaps": [
"CAP_AUDIT_WRITE",
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_MKNOD",
"CAP_NET_BIND_SERVICE",
"CAP_NET_RAW",
"CAP_SETFCAP",
"CAP_SETGID",
"CAP_SETPCAP",
"CAP_SETUID",
"CAP_SYS_CHROOT"
],
"BoundingCaps": [
"CAP_AUDIT_WRITE",
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_MKNOD",
"CAP_NET_BIND_SERVICE",
"CAP_NET_RAW",
"CAP_SETFCAP",
"CAP_SETGID",
"CAP_SETPCAP",
"CAP_SETUID",
"CAP_SYS_CHROOT"
],
"ExecIDs": [],
"GraphDriver": {
"Name": "overlay",
"Data": {
"LowerDir": "/var/lib/containers/storage/overlay/1b4924ccd698eb58028bfd99609323eb7fb9b70d38b8de5fb10214b20b512748/diff:/var/lib/containers/storage/overlay/5a11a53324980b0c6a0d80b8f4de19d14e7b06310ea5d3b9e4334752c8aa02fa/diff:/var/lib/containers/storage/overlay/89ae5c4ee501a09c879f5b58474003539ab3bb978a553af2a4a6a7de248b5740/diff",
"UpperDir": "/var/lib/containers/storage/overlay/fc1187b92975d7f7339f0e63f57fff99d5979466df7c24b5ec62de913f7b9b24/diff",
"WorkDir": "/var/lib/containers/storage/overlay/fc1187b92975d7f7339f0e63f57fff99d5979466df7c24b5ec62de913f7b9b24/work"
}
},
"Mounts": [],
"Dependencies": [],
"NetworkSettings": {
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": [],
"SandboxKey": ""
},
"ExitCommand": [
"/usr/bin/podman",
"--root",
"/var/lib/containers/storage",
"--runroot",
"/var/run/containers/storage",
"--log-level",
"debug",
"--cgroup-manager",
"systemd",
"--tmpdir",
"/var/run/libpod",
"--runtime",
"runc",
"--events-backend",
"file",
"container",
"cleanup",
"2fb9584fcc8e089e9dc95e38ff79e2b9d5036f0f34ec45a1e885d5a3a8862bab"
],
"Namespace": "",
"IsInfra": false,
"Config": {
"Hostname": "2fb9584fcc8e",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"HOSTNAME=2fb9584fcc8e",
"HOME=/root",
"container=libpod"
],
"Cmd": [
"/app"
],
"Image": "testcontainersofficial/ryuk:0.3.0",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": "",
"OnBuild": null,
"Labels": null,
"Annotations": {
"io.container.manager": "libpod",
"io.kubernetes.cri-o.Created": "2020-06-27T17:40:09.128835551+03:00",
"io.podman.annotations.autoremove": "FALSE",
"io.podman.annotations.init": "FALSE",
"io.podman.annotations.privileged": "FALSE",
"io.podman.annotations.publish-all": "FALSE",
"org.opencontainers.image.stopSignal": "15"
},
"StopSignal": 15,
"CreateCommand": [
"podman",
"system",
"service",
"--cgroup-manager=systemd",
"--log-level=debug",
"--time=0",
"tcp:127.0.0.1:1213"
]
},
"HostConfig": {
"Binds": [],
"CgroupMode": "host",
"ContainerIDFile": "",
"LogConfig": {
"Type": "",
"Config": null
},
"NetworkMode": "",
"PortBindings": {},
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": [],
"CapDrop": [],
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": [],
"GroupAdd": [],
"IpcMode": "private",
"Cgroup": "",
"Cgroups": "default",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "private",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [],
"Tmpfs": {},
"UTSMode": "private",
"UsernsMode": "",
"ShmSize": 0,
"Runtime": "oci",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": 0,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": [
{
"Name": "RLIMIT_NOFILE",
"Soft": 1048576,
"Hard": 1048576
},
{
"Name": "RLIMIT_NPROC",
"Soft": 4194304,
"Hard": 4194304
}
],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
}
}
]
Inspect results from image created with cli
[
{
"Id": "f0f7615fa1840722b0f81d84fe112f6ea9d39e405b86ea77adec803fc8f6fba2",
"Created": "2020-06-27T17:45:45.193607419+03:00",
"Path": "/app",
"Args": [
"/app"
],
"State": {
"OciVersion": "1.0.2-dev",
"Status": "configured",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "0001-01-01T00:00:00Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Healthcheck": {
"Status": "",
"FailingStreak": 0,
"Log": null
}
},
"Image": "344c0229fe447209d01c388035dfe01ca38f5289d99c7e14febbcb9a0b128775",
"ImageName": "docker.io/testcontainersofficial/ryuk:0.3.0",
"Rootfs": "",
"Pod": "",
"ResolvConfPath": "",
"HostnamePath": "",
"HostsPath": "",
"StaticDir": "/var/lib/containers/storage/overlay-containers/f0f7615fa1840722b0f81d84fe112f6ea9d39e405b86ea77adec803fc8f6fba2/userdata",
"OCIRuntime": "runc",
"LogPath": "/var/lib/containers/storage/overlay-containers/f0f7615fa1840722b0f81d84fe112f6ea9d39e405b86ea77adec803fc8f6fba2/userdata/ctr.log",
"LogTag": "",
"ConmonPidFile": "/var/run/containers/storage/overlay-containers/f0f7615fa1840722b0f81d84fe112f6ea9d39e405b86ea77adec803fc8f6fba2/userdata/conmon.pid",
"Name": "tester2",
"RestartCount": 0,
"Driver": "overlay",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "container-default",
"EffectiveCaps": [
"CAP_AUDIT_WRITE",
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_MKNOD",
"CAP_NET_BIND_SERVICE",
"CAP_NET_RAW",
"CAP_SETFCAP",
"CAP_SETGID",
"CAP_SETPCAP",
"CAP_SETUID",
"CAP_SYS_CHROOT"
],
"BoundingCaps": [
"CAP_AUDIT_WRITE",
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_MKNOD",
"CAP_NET_BIND_SERVICE",
"CAP_NET_RAW",
"CAP_SETFCAP",
"CAP_SETGID",
"CAP_SETPCAP",
"CAP_SETUID",
"CAP_SYS_CHROOT"
],
"ExecIDs": [],
"GraphDriver": {
"Name": "overlay",
"Data": {
"LowerDir": "/var/lib/containers/storage/overlay/1b4924ccd698eb58028bfd99609323eb7fb9b70d38b8de5fb10214b20b512748/diff:/var/lib/containers/storage/overlay/5a11a53324980b0c6a0d80b8f4de19d14e7b06310ea5d3b9e4334752c8aa02fa/diff:/var/lib/containers/storage/overlay/89ae5c4ee501a09c879f5b58474003539ab3bb978a553af2a4a6a7de248b5740/diff",
"UpperDir": "/var/lib/containers/storage/overlay/66cda93cba810a10d53592c03e49af0ff03a2b8de803dc0f5579ab17e278533a/diff",
"WorkDir": "/var/lib/containers/storage/overlay/66cda93cba810a10d53592c03e49af0ff03a2b8de803dc0f5579ab17e278533a/work"
}
},
"Mounts": [],
"Dependencies": [],
"NetworkSettings": {
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": [],
"SandboxKey": ""
},
"ExitCommand": [
"/usr/bin/podman",
"--root",
"/var/lib/containers/storage",
"--runroot",
"/var/run/containers/storage",
"--log-level",
"error",
"--cgroup-manager",
"systemd",
"--tmpdir",
"/var/run/libpod",
"--runtime",
"runc",
"--events-backend",
"file",
"container",
"cleanup",
"f0f7615fa1840722b0f81d84fe112f6ea9d39e405b86ea77adec803fc8f6fba2"
],
"Namespace": "",
"IsInfra": false,
"Config": {
"Hostname": "f0f7615fa184",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"container=podman",
"HOSTNAME="
],
"Cmd": [
"/app"
],
"Image": "docker.io/testcontainersofficial/ryuk:0.3.0",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": "",
"OnBuild": null,
"Labels": null,
"Annotations": {
"io.kubernetes.cri-o.TTY": "false",
"io.podman.annotations.apparmor": "container-default",
"io.podman.annotations.autoremove": "FALSE",
"io.podman.annotations.init": "FALSE",
"io.podman.annotations.privileged": "FALSE",
"io.podman.annotations.publish-all": "FALSE"
},
"StopSignal": 0,
"CreateCommand": [
"podman",
"create",
"--name",
"tester2",
"testcontainersofficial/ryuk:0.3.0"
]
},
"HostConfig": {
"Binds": [],
"CgroupMode": "host",
"ContainerIDFile": "",
"LogConfig": {
"Type": "k8s-file",
"Config": null
},
"NetworkMode": "bridge",
"PortBindings": {},
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": [],
"CapDrop": [],
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": [],
"GroupAdd": [],
"IpcMode": "private",
"Cgroup": "",
"Cgroups": "default",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "private",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"apparmor=container-default"
],
"Tmpfs": {},
"UTSMode": "private",
"UsernsMode": "",
"ShmSize": 65536000,
"Runtime": "oci",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": 0,
"OomKillDisable": false,
"PidsLimit": 4096,
"Ulimits": [
{
"Name": "RLIMIT_NOFILE",
"Soft": 1048576,
"Hard": 1048576
},
{
"Name": "RLIMIT_NPROC",
"Soft": 4194304,
"Hard": 4194304
}
],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
}
}
]
Diagnosed this further.
When container is created with podman create, following metadata (container inspect) is populated:
"Path": "/app",
"Args": [
"/app"
],
When container is created using API result is:
"Path": "",
"Args": null,
If I specify a command using Cmd, Args is set to value from command.
This is certainly a blocker for those who want to use docker api
@baude @jwhonce PTAL
I had some time to look through code paths that libpod and compat handlers use for creating container.
It seems that compat doesn't use specgen.generate.MakeContainer. This function seems to be responsible for choosing correct parameters when generating container.
Compat handler implementation seems to be a bit messy. I don't get why some of logic is packed to utils for compat while libpod logic is neatly packed in single file
I'm working on converting the compat create endpoint to use the same codepath that libpod does, but it's slow going - there are a lot of parameters that need to be handled.
I'd imagine that resolves issue with publish all ports flag not publishing exposed ports.
Btw. I noted some differences in host port randomization. Docker seems to randomize ports on start while podman does it on create. I can see benefits in both approaches. I think that docker's solution is less likely to fail when starting container. If port is chosen on start, it's possible to ensure that host port is free.
I didn't check podman code yet, just noted that port information is available as soon as container had been created - docker required a start to reveal port. Is this behavioral difference worth reporting an issue or can you just discuss it internally?
That's correct - Podman does precompute port mappings at create time,
rather than generating them at runtime. This may be slightly less reliable
than Docker, but it's been done this way for three years without a bug
report about it, so I don't think changing this is a priority unless this
breaks something critical.
On Fri, Jul 3, 2020, 18:32 Sami Korhonen notifications@github.com wrote:
I'd imagine that resolves issue with publish all ports flag not publishing
exposed ports.Btw. I noted some differences in host port randomization. Docker seems to
randomize ports on start while podman does it on create. I can see benefits
in both approaches. I think that docker's solution is less likely to fail
when starting container. If port is chosen on start, it's possible to
ensure that host port is free.I didn't check podman code yet, just noted that port information is
available as soon as container had been created - docker required a start
to reveal port. Is this behavioral difference worth reporting an issue or
can you just discuss it internally?—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/6799#issuecomment-653689455,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3AOCDMAJXQB3FXEL6PO6LRZZL7HANCNFSM4OKCATAQ
.
I guess probability of a conflict depends heavily on amount of activity on host. Tools such as testcontainers might behave a little differently than normal container load. Tc creates a short lived containers with port mapping. I'd imagine that most short lived containers are used for run once type of jobs that do not use port mappings.
I'll try to find time to simulate this once other blocking issues are resolved. It should be relatively easy to create a test to ger rate (# of containers per minute) when too many conflicts start to occur. It would be ideal that conflicts never happened considering tc is used testing - but I think people can live with random failures, if they just don't happen every other day
Podman does precompute port mappings at create time,
rather than generating them at runtime
Unless Podman "holds" the assigned ports (even before the container is started), such behaviour would be very easy to break as soon as you have other, non-Podman processes on the host system.
And, if it "reserves" the port, this sounds like a port leak to me, because it is not safe to assume that the time between create & start is short. In fact, I've seen systems that may stop containers (to free up some resources) for hours.
same problem #5825 which was closed without receiving notification from Github...
I believe we should do this at start time as well. Makes little sense to reserve random ports. Which is what we are doing. I don't think changing the default would break anyone's expectations and existing containers would continue to work, since they have the ports reserved.
Porting the code to run at start time instead of create time is a nontrivial effort, and thus far I haven't seen a compelling argument to make the switch. There's certainly the potential for issues in the current model, but only in cases where containers are turned off for a significant period of time and then restarted, and I don't know how common that actually is.
Also, it might not be bad idea to add podman specific configuration that allows overriding system defaults for range used by port randomization.
Docker is running as daemon, so one range is enough for it. Podman can be used by multiple accounts on single host. Would be nice addition, if ranges could be controlled by user account. This would allow using random ports with per user host firewall rules
Most helpful comment
I guess probability of a conflict depends heavily on amount of activity on host. Tools such as testcontainers might behave a little differently than normal container load. Tc creates a short lived containers with port mapping. I'd imagine that most short lived containers are used for run once type of jobs that do not use port mappings.
I'll try to find time to simulate this once other blocking issues are resolved. It should be relatively easy to create a test to ger rate (# of containers per minute) when too many conflicts start to occur. It would be ideal that conflicts never happened considering tc is used testing - but I think people can live with random failures, if they just don't happen every other day