Openwhisk: docker for mac or CoreOS does not have runc and causes invoker errors

[rabbah]

observed while investigating separate issue:

[2017-09-20T23:59:44.351Z] [INFO] [#sid_102] [RuncClient] running /usr/bin/docker-runc pause 98a22ef65d5aac2abdee8313f3216112356c7c01bda18a38367cab05baa09f79 [marker:invoker_runc.pause_start:170209551] 
[2017-09-20T23:59:44.359Z] [ERROR] [#sid_102] [RuncClient] code: 1, stdout: , stderr: json: cannot unmarshal object into Go value of type []string [marker:invoker_runc.pause_error:170209558:8] 

This causes a container to be disposed and hence there is no container reuse.

[alexkli]

Also seeing no container reuse with Docker for Mac, with slightly different errors in the controller logs:

[2017-09-20T20:12:45.464Z] [INFO] [#sid_102] [RuncClient] running /usr/bin/docker-runc pause b9cbc7880603ae16b3178fa8f8e29d36e99415d262e0350562637cb81c677603 [marker:invoker_runc.pause_start:8052637] 
[2017-09-20T20:12:45.514Z] [ERROR] [#sid_102] [RuncClient] code: 1, stdout: , stderr: open /run/runc/b9cbc7880603ae16b3178fa8f8e29d36e99415d262e0350562637cb81c677603/state.json: no such file or directory [marker:invoker_runc.pause_error:8052687:50] 

There is no docker-runc binary on my system.

[rabbah]

An error in the container life cycle will dispense with it. So every activation will end up with a new container. We need to switch the setting for docker-for-mac to use docker pause/unpause instead.

FYI @csantanapr @markusthoemmes

[ddragosd]

maybe DockerContainer.scala can be told by InvokerReactive if it should use runc or not ? IIRC there used to be a config property for runc ?

[rabbah]

We used to in fact have this setting as a deployment parameters (use-runc or not) - seems we lost it as part of invoker reactive replacing the previous invoker implementation.

[ddragosd]

Or, alternatively, when a new Invoker instance starts, it can check if runc client is installed, and then populate the config with that information. In this way invokers may run in hybrid envs.

[rabbah]

+1, i like the self-discovery.

[ddragosd]

I was about to push a PR with this, but /usr/bin/docker-runc actually exists on Invoker's docker image, making _self-discovery_ tricky...

[csantanapr]

Why are their problems with docker-runc on Mac? it's because docker engine is at very latest version and docker-runc client is on old version 1.12?

Also how is determine today if it's going to use docker or docker-runc ?

[ddragosd]

Why are their problems with docker-runc on Mac?

Docker for Mac doesn't come with runc ( at least I didn't find it anywhere )

Also how is determine today if it's going to use docker or docker-runc ?

Ror pause/resume runc it's being used implicitly, and for the other operations docker is being used.

[ddragosd]

So far I have been testing with these changes: https://github.com/ddragosd/openwhisk/commit/9211bef3ce223f9597ac981ec46503e996f4ef6f that basically check if docker-runc exists. Given that it actually exists on the Invoker image, to test it locally, I had to make sure it's deleted 😈 by updating the start command to : command: /bin/sh -c "rm /usr/bin/docker-runc && /invoker/bin/invoker 0 "

[ddragosd]

I just realized that this doesn't happen only for Mac, but also on some Linux machines like CoreOS:

[2017-09-08T14:23:15.247Z] [ERROR] [#sid_102] [RuncClient] code: 1, stdout: , stderr: open /run/runc/89c6c3f13a181606aaeb36bb8ee1700aad2a3f1a500e1d52a64046495d739a30/state.json: no such file or directory [marker:invoker_runc.pause_error:63599:45] 

 $ uname -a
Linux ip-10-0-0-64.us-west-1.compute.internal 4.7.3-coreos-r3 #1 SMP Thu Feb 23 02:16:16 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz GenuineIntel GNU/Linux

[csantanapr]

yeah the problem is that like you discovered on Mac the image will have docker-runc installed.

[csantanapr]

@ddragosd what's the version of docker engine on that CoreOS system? run docker version

[ddragosd]

on CoreOS:

Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   d5236f0
 Built:        Thu Feb 23 02:17:18 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   d5236f0
 Built:        Thu Feb 23 02:17:18 2017
 OS/Arch:      linux/amd64

[ddragosd]

1.12.6 seems to be the latest docker version for CoreOS as of Sept 6th, the Stable channel - https://coreos.com/releases/#1465.7.0

[csantanapr]

but from inside the invoker you will get 1.12.0
do docker exec invoker0 docker version

but your engine is set to 1.12.6 which should be closer to 1.12.0

[csantanapr]

I rebuild the invoker with latest docker client 17.06.2-ce
and on docker for Mac I get docker re-use no more docker-runc errors in the logs

🍔  $ docker exec invoker0 docker version
Client:
 Version:      17.06.2-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 19:57:21 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.2-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 19:59:19 2017
 OS/Arch:      linux/amd64
 Experimental: true

[csantanapr]

[2017-09-21T03:42:27.683Z] [INFO] [#tid_46] [RuncClient] [marker:invoker_runc.resume_finish:153:8]
[2017-09-21T03:42:27.764Z] [INFO] [#sid_102] [RuncClient] running /usr/bin/docker-runc pause 387ebb93d8b5f40247707b6e91157c043f17a3ca0ad7524e8f5249ac1f59ee75 [marker:invoker_runc.pause_start:1186462]
[2017-09-21T03:42:27.776Z] [INFO] [#sid_102] [RuncClient] [marker:invoker_runc.pause_finish:1186475:13]
[2017-09-21T03:42:28.636Z] [INFO] [#tid_47] [RuncClient] running /usr/bin/docker-runc resume 387ebb93d8b5f40247707b6e91157c043f17a3ca0ad7524e8f5249ac1f59ee75 [marker:invoker_runc.resume_start:19]
[2017-09-21T03:42:28.648Z] [INFO] [#tid_47] [RuncClient] [marker:invoker_runc.resume_finish:33:14]
[2017-09-21T03:42:28.724Z] [INFO] [#sid_102] [RuncClient] running /usr/bin/docker-runc pause 387ebb93d8b5f40247707b6e91157c043f17a3ca0ad7524e8f5249ac1f59ee75 [marker:invoker_runc.pause_start:1187422]
[2017-09-21T03:42:28.739Z] [INFO] [#sid_102] [RuncClient] [marker:invoker_runc.pause_finish:1187438:16]

[csantanapr]

upgrading would be tricky for us, as we got burned in the past and a lot of systems to update and coordinate deployments.
Eventually I think we should update also doing some performance testing to make sure there are no regressions.

maybe best to move forward would be to go back to be an invoker config value, and we override in ansible where we can detect Mac and set it to not use docker-runc

[rabbah]

Some history of note on upgrading docker: from 1.9 we skipped 1.10 and 1.11 - under load testing we consistently found the daemon would lock up, or end up causing kernel panics. These were traced to synchronization bugs in docker. We picked up the fixes in 1.12. We’d have to stress test any upgrades, or pin docker versions to deployments, which could be challenging to manage and test though.

[markusthoemmes]

FWIW I had the same error on docker-machine recently, Carlos‘ suggestion fixed it. We can try if the client is backwards compatible while leaving the server alone?

[markusthoemmes]

Alternatively: We can go back to just mounting the docker and runc binary into the invoker (we mount the runc runtime env anyway). That automatically gives server/client parity.

[markusthoemmes]

I suppose this won't work for Docker on Mac per https://docs.docker.com/docker-for-mac/osxfs/#performance-issues-solutions-and-roadmap

By default, you can share files in /Users/, /Volumes/, /private/, and /tmp directly

[csantanapr]

From a quick read the problem is around performance os io transfer between the filesystems, in this case we just want to expose the docker executable from the host to the container.

I like the idea of parity between client and engine to be on the same version.

[csantanapr]

@markusthoemmes

We can go back to just mounting the docker and runc binary into the invoker (we mount the runc runtime env anyway).

1. How did we use to mount you have link to the old code?
2. Why did we move away from mounting? Any problems going back to mounting?
3. How do we mount the runc? I only see in the Dockerfile the docker-runc binary extracted from the tgz https://github.com/apache/incubator-openwhisk/blob/master/core/invoker/Dockerfile#L9

[ddragosd]

It's interesting to note that the problem of updating docker engine becomes transparent with OW using Mesos and Kube APIs, which are not as sensitive to upgrades as Docker Engine is. But for now we need to focus on solving this problem with Docker, b/c the other APIs integration is in the works ATM.

[ddragosd]

@markusthoemmes

By default, you can share files in /Users/, /Volumes/, /private/, and /tmp directly

This is related to files that are on the MacOS disks. Docker still runs in a VM ( native for Mac or docker machine). I think we can grab the docker binaries b/c, for instance, with docker-compose, we can mount /var/run/docker.sock ( b/c this is not on OSX, but in the VM that runs docker ... it's confusing for me as well at times to think about these 2 layers ).

[markusthoemmes]

@ddragosd you're right, I'm assembling the PR I had in mind.

[ddragosd]

I've fixed the issue with docker for mac in the PR: https://github.com/apache/incubator-openwhisk-devtools/pull/56

By overwriting the docker and runc binaries with the ones from the host, no changes were required to the existing code base. At the same time, once https://github.com/apache/incubator-openwhisk/pull/2793 is in, the Invoker image won't contain the docker binaries, which means that it should work with any docker version installed on the host.