Openwhisk: Invoker use outdated docker API to pause container for docker version 18.09

[axiqia]

Environment details:

• local deployment,native ubuntu
• docker18.09, ubuntu16.04

Steps to reproduce the issue:

wsk action invoke /whisk.system/utils/echo -p message hello --result

Provide the actual results and outputs:

[2019-04-06T07:06:53.148Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [InvokerReactive]  [marker:invoker_activation_start:33]
[2019-04-06T07:06:53.149Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [WhiskAction] [GET] serving from datastore: CacheKey(guest/sleepy) [marker:database_cacheMiss_count:34]
[2019-04-06T07:06:53.149Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [CouchDbRestStore] [GET] 'whisk_local_whisks' finding document: 'id: guest/sleepy, rev: 1-40f8848133d2e3b1855d5542b701104e' [marker:#
[2019-04-06T07:06:53.157Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [CouchDbRestStore]  [marker:database_getDocument_finish:42:8]
[2019-04-06T07:06:53.162Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [WhiskAction] write initiated on existing cache entry, invalidating CacheKey(guest/sleepy), tid 9fd179fb06e529c352277487058e6acf, st#
[2019-04-06T07:06:53.162Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [WhiskAction] write all done, caching CacheKey(guest/sleepy) Cached
[2019-04-06T07:06:53.162Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [ContainerPool] containerStart containerState: prewarmed container: Some(ContainerId(db8ea42a0a32fcf559acf0a2c549581e97ba6c5a312cb0a#
[2019-04-06T07:06:53.163Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [DockerContainer] sending initialization to ContainerId(db8ea42a0a32fcf559acf0a2c549581e97ba6c5a312cb0a23be1e7e874fa8af5) ContainerA#
[2019-04-06T07:06:53.164Z] [INFO] [#tid_sid_invokerWarmup] [DockerClientWithFileAccess] running /usr/bin/docker run -d --cpu-shares 93 --memory 256m --memory-swap 256m --network bridge -e __OW_API_HOST=http#
[2019-04-06T07:06:53.207Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [DockerContainer] initialization result: ok [marker:invoker_activationInit_finish:92:44]
[2019-04-06T07:06:53.221Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [DockerContainer] sending arguments to /guest/sleepy at ContainerId(db8ea42a0a32fcf559acf0a2c549581e97ba6c5a312cb0a23be1e7e874fa8af5#
[2019-04-06T07:06:53.248Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [DockerContainer] running result: ok [marker:invoker_activationRun_finish:133:41]
[2019-04-06T07:06:53.248Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [ContainerProxy]  [marker:invoker_collectLogs_start:133]
[2019-04-06T07:06:53.249Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [ContainerProxy]  [marker:invoker_collectLogs_finish:134:1]
[2019-04-06T07:06:53.250Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [CouchDbRestStore] [PUT] 'whisk_local_activations' saving document: 'id: guest/dd6dc05cf0ce4d15adc05cf0ce3d15b3, rev: null' [marker:#
[2019-04-06T07:06:53.250Z] [INFO] [#tid_sid_dbBatcher] [CouchDbRestStore] 'whisk_local_activations' saving 1 documents [marker:database_saveDocumentBulk_start:83756362]
[2019-04-06T07:06:53.252Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [InvokerReactive] posted completion of activation dd6dc05cf0ce4d15adc05cf0ce3d15b3
**[2019-04-06T07:06:53.321Z] [INFO] [#tid_sid_invokerNanny] [RuncClient] running /usr/bin/docker-runc pause db8ea42a0a32fcf559acf0a2c549581e97ba6c5a312cb0a23be1e7e874fa8af5 (timeout: 10 seconds) [marker:invok#**
**[2019-04-06T07:06:53.362Z] [ERROR] [#tid_sid_invokerNanny] [RuncClient] info: command was unsuccessful, code: 1 (unsuccessful), stdout: , stderr: open /run/runc/db8ea42a0a32fcf559acf0a2c549581e97ba6c5a312cb#**
[2019-04-06T07:06:53.363Z] [INFO] [#tid_sid_invokerNanny] [DockerClientWithFileAccess] running /usr/bin/docker rm -f db8ea42a0a32fcf559acf0a2c549581e97ba6c5a312cb0a23be1e7e874fa8af5 (timeout: 1 minute) [mar#
[2019-04-06T07:06:53.726Z] [INFO] [#tid_sid_dbBatcher] [CouchDbRestStore]  [marker:database_saveDocumentBulk_finish:83756838:476]
[2019-04-06T07:06:53.726Z] [INFO] [#tid_9fd179fb06e529c352277487058e6acf] [CouchDbRestStore]  [marker:database_saveDocument_finish:611:476]

The docker- prefix has been removed from these binaries; see the release notes: https://github.com/docker/docker-ce/releases/tag/v18.09.0

Remove 'docker-' prefix for containerd and runc binaries docker/engine#61 / moby/moby#37907, docker-ce-packaging#241

[style95]

I would handle this issue in https://github.com/apache/incubator-openwhisk/pull/4430

[axiqia]

Thank you for your effort. @style95 I have read your PR changes, is that mean the docker server version in my host is 18.09, but inside invoker container, its own docker client is 1.12?

[style95]

Yes, but actually, that problem comes from the invoker codes as runc client is hardcoded in the code.
https://github.com/apache/incubator-openwhisk/blob/master/core/invoker/src/main/scala/org/apache/openwhisk/core/containerpool/docker/RuncClient.scala#L58

[axiqia]

Much clear! Real appreciate for your help.

[rabbah]

fixed by https://github.com/apache/openwhisk/pull/4430.