Origin: oc cluster up fails on mac with "error creating node config"

[knrc]

When running 'oc cluster up' on a mac, based on the tag v3.11.0, the

Version

oc v3.11.0+0cbc58b
kubernetes v1.11.0+d4cacc0
features: Basic-Auth

Steps To Reproduce

Run 'oc cluster up' on mac

Current Result

Getting a Docker client ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
Creating shared mount directory on the remote host ...
Determining server IP ...
Checking if OpenShift is already running ...
Checking for supported Docker version (=>1.22) ...
Checking if insecured registry is configured properly in Docker ...
Checking prerequisites for port forwarding ...
Checking if required ports are available ...
Checking if OpenShift client is configured properly ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
Starting OpenShift using openshift/origin-control-plane:v3.11 ...
I1012 16:42:36.567938   25836 config.go:40] Running "create-master-config"
I1012 16:42:38.246052   25836 config.go:46] Running "create-node-config"
I1012 16:42:39.653261   25836 flags.go:30] Running "create-kubelet-flags"
I1012 16:42:40.684323   25836 run_kubelet.go:49] Running "start-kubelet"
Error: error creating node config: could not create OpenShift configuration: cannot start container 12c1d210ad5040275c66065e9659ec8a7856b6f9fa822d133a4cb5f678814e1e; caused by: cannot start container 12c1d210ad5040275c66065e9659ec8a7856b6f9fa822d133a4cb5f678814e1e

Expected Result

Successful startup of 3.11 cluster

Getting a Docker client ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
Creating shared mount directory on the remote host ...
Determining server IP ...
Checking if OpenShift is already running ...
Checking for supported Docker version (=>1.22) ...
Checking if insecured registry is configured properly in Docker ...
Checking prerequisites for port forwarding ...
Checking if required ports are available ...
Checking if OpenShift client is configured properly ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
Starting OpenShift using openshift/origin-control-plane:v3.11 ...
I1012 16:46:37.989224   26159 config.go:40] Running "create-master-config"
I1012 16:46:39.707623   26159 config.go:46] Running "create-node-config"
I1012 16:46:40.856186   26159 flags.go:30] Running "create-kubelet-flags"
I1012 16:46:41.925547   26159 run_kubelet.go:48] Running "start-kubelet"
I1012 16:46:42.307358   26159 run_self_hosted.go:181] Waiting for the kube-apiserver to be ready ...
I1012 16:46:57.459015   26159 interface.go:26] Installing "kube-proxy" ...
I1012 16:46:57.459045   26159 interface.go:26] Installing "kube-dns" ...
I1012 16:46:57.459057   26159 interface.go:26] Installing "openshift-service-cert-signer-operator" ...
I1012 16:46:57.459065   26159 interface.go:26] Installing "openshift-apiserver" ...
I1012 16:46:57.459094   26159 apply_template.go:81] Installing "openshift-apiserver"
I1012 16:46:57.459097   26159 apply_template.go:81] Installing "kube-dns"
I1012 16:46:57.459097   26159 apply_template.go:81] Installing "kube-proxy"
I1012 16:46:57.459107   26159 apply_template.go:81] Installing "openshift-service-cert-signer-operator"
I1012 16:46:59.584384   26159 interface.go:41] Finished installing "kube-proxy" "kube-dns" "openshift-service-cert-signer-operator" "openshift-apiserver"
I1012 16:48:06.686751   26159 run_self_hosted.go:242] openshift-apiserver available
I1012 16:48:06.686785   26159 interface.go:26] Installing "openshift-controller-manager" ...
I1012 16:48:06.686803   26159 apply_template.go:81] Installing "openshift-controller-manager"
I1012 16:48:08.735196   26159 interface.go:41] Finished installing "openshift-controller-manager"
Adding default OAuthClient redirect URIs ...
Adding registry ...
Adding sample-templates ...
Adding centos-imagestreams ...
Adding router ...
Adding persistent-volumes ...
Adding web-console ...
I1012 16:48:08.798854   26159 interface.go:26] Installing "openshift-image-registry" ...
I1012 16:48:08.798867   26159 interface.go:26] Installing "sample-templates" ...
I1012 16:48:08.798873   26159 interface.go:26] Installing "centos-imagestreams" ...
I1012 16:48:08.798878   26159 interface.go:26] Installing "openshift-router" ...
I1012 16:48:08.798883   26159 interface.go:26] Installing "persistent-volumes" ...
I1012 16:48:08.798889   26159 interface.go:26] Installing "openshift-web-console-operator" ...
I1012 16:48:08.798951   26159 interface.go:26] Installing "sample-templates/mysql" ...
I1012 16:48:08.798969   26159 interface.go:26] Installing "sample-templates/postgresql" ...
I1012 16:48:08.798975   26159 interface.go:26] Installing "sample-templates/cakephp quickstart" ...
I1012 16:48:08.798981   26159 interface.go:26] Installing "sample-templates/django quickstart" ...
I1012 16:48:08.798986   26159 interface.go:26] Installing "sample-templates/rails quickstart" ...
I1012 16:48:08.798989   26159 apply_list.go:67] Installing "centos-imagestreams"
I1012 16:48:08.798992   26159 interface.go:26] Installing "sample-templates/jenkins pipeline ephemeral" ...
I1012 16:48:08.799067   26159 apply_list.go:67] Installing "sample-templates/rails quickstart"
I1012 16:48:08.799075   26159 interface.go:26] Installing "sample-templates/mongodb" ...
I1012 16:48:08.799085   26159 interface.go:26] Installing "sample-templates/mariadb" ...
I1012 16:48:08.799084   26159 apply_list.go:67] Installing "sample-templates/postgresql"
I1012 16:48:08.799106   26159 interface.go:26] Installing "sample-templates/dancer quickstart" ...
I1012 16:48:08.799086   26159 apply_list.go:67] Installing "sample-templates/mysql"
I1012 16:48:08.799113   26159 apply_list.go:67] Installing "sample-templates/cakephp quickstart"
I1012 16:48:08.799108   26159 apply_list.go:67] Installing "sample-templates/django quickstart"
I1012 16:48:08.799114   26159 interface.go:26] Installing "sample-templates/nodejs quickstart" ...
I1012 16:48:08.799181   26159 apply_list.go:67] Installing "sample-templates/mariadb"
I1012 16:48:08.799208   26159 apply_list.go:67] Installing "sample-templates/jenkins pipeline ephemeral"
I1012 16:48:08.799273   26159 apply_template.go:81] Installing "openshift-web-console-operator"
I1012 16:48:08.799305   26159 interface.go:26] Installing "sample-templates/sample pipeline" ...
I1012 16:48:08.799347   26159 apply_list.go:67] Installing "sample-templates/dancer quickstart"
I1012 16:48:08.799382   26159 apply_list.go:67] Installing "sample-templates/mongodb"
I1012 16:48:08.799413   26159 apply_list.go:67] Installing "sample-templates/sample pipeline"
I1012 16:48:08.799435   26159 apply_list.go:67] Installing "sample-templates/nodejs quickstart"
I1012 16:48:11.607054   26159 interface.go:41] Finished installing "sample-templates/mysql" "sample-templates/postgresql" "sample-templates/cakephp quickstart" "sample-templates/django quickstart" "sample-templates/rails quickstart" "sample-templates/jenkins pipeline ephemeral" "sample-templates/mongodb" "sample-templates/mariadb" "sample-templates/dancer quickstart" "sample-templates/nodejs quickstart" "sample-templates/sample pipeline"
I1012 16:48:28.145835   26159 interface.go:41] Finished installing "openshift-image-registry" "sample-templates" "centos-imagestreams" "openshift-router" "persistent-volumes" "openshift-web-console-operator"
Login to server ...
Creating initial project "myproject" ...
Server Information ...
OpenShift server started.

The server is accessible via web console at:
    https://127.0.0.1:8443

You are logged in as:
    User:     developer
    Password: <any value>

To login as administrator:
    oc login -u system:admin

Additional Information

The failure is caused by

Error response from daemon: Mounts denied:
The path /var/lib/kubelet/device-plugins
is not shared from OS X and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> File Sharing.
See https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.

and the introduction of the '/var/lib/kubelet/device-plugins' mount in the following commit

https://github.com/openshift/origin/commit/10560a82888f0566a98edef6daafb97e58d6ad33

[cunningt]

I got by this error by adding /var/lib/kubelet/device-plugins to the list of bind mounted directories in Docker->File Sharing. I had to type the the directory name, not use Finder that comes up with the + window, which resulted in an error.

[knrc]

@cunningt I took a look at the commit and it's purpose is vague, since we are building our own version of oc with an extension for installing Istio I went ahead and removed the offending line.

[B1raj]

Any fix for this issue yet?

[tech-mint]

having same issue

[kylearctiq]

also having this issue still

[Dimss]

The following worked for me on

• MacBookPro Mojave 10.14.1
• Docker Version 18.06.1-ce-mac73 (26764)

by executing following steps:

1. mkdir -p /var/lib/kubelet/device-plugins
2. sudo chown -R your_user /var/lib/kubelet/device-plugins
3. restart docker engine
4. run oc cluster --base-dir=/tmp/new-oc-run (set --base-dir to ensure fresh installation)

[Robb-Greathouse]

Just FYI. I tried the above; but still getting the same error.
Error: error creating node config: could not create OpenShift configuration: cannot start container 701e2ec8659b0d79c2643b3d8a6ed8d22ccb0acc734bf177d592867db1429055; caused by: cannot start container 701e2ec8659b0d79c2643b3d8a6ed8d22ccb0acc734bf177d592867db1429055

[knrc]

@Robb-Greathouse If you want to try our version then checkout https://github.com/Maistra/origin/releases, it's the oc command with the addition of an istio operator.

[Dimss]

@knrc what do you mean by support for Istio operator? Is it means that in that version I don’t need to patch master-config.yaml anymore? And is the Istio already installed in the cluster? Or only Istio operator already there and only CRD is needed to install Istio?

[knrc]

@Dimss It's part of the maistra configuration so if you run 'oc cluster up' you will see the operator deployed into the istio-operator namespace. You would then need to deploy the installation CR into the istio-operator namespace, details of which can be found in our docs

[Robb-Greathouse]

Ok. Does this work on Mac?

On Tue, Nov 27, 2018 at 12:33 PM Kevin Conner [e-mail]
wrote:

@Robb-Greathouse https://github.com/Robb-Greathouse If you want to try
our version then checkout https://github.com/Maistra/origin/releases,
it's the oc command with the addition of an istio operator.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/origin/issues/21246#issuecomment-442187554,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AX3GuIB1YTQdPHNofhCgneyiSnO5w6Ipks5uzZOcgaJpZM4XaV3Y
.

--
Robb Greathouse
Middleware BU
505-507-4906

[Robb-Greathouse]

I'll give it a try.

On Tue, Nov 27, 2018 at 3:40 PM Robb Greathouse [e-mail] wrote:

Ok. Does this work on Mac?

On Tue, Nov 27, 2018 at 12:33 PM Kevin Conner [e-mail]
wrote:

@Robb-Greathouse https://github.com/Robb-Greathouse If you want to try
our version then checkout https://github.com/Maistra/origin/releases,
it's the oc command with the addition of an istio operator.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/origin/issues/21246#issuecomment-442187554,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AX3GuIB1YTQdPHNofhCgneyiSnO5w6Ipks5uzZOcgaJpZM4XaV3Y
.

--
Robb Greathouse
Middleware BU
505-507-4906

--
Robb Greathouse
Middleware BU
505-507-4906

[mrdoodles]

I needed to include /var/lib/kubelet/device-plugins in the docker shared folders as mentioned by cunningt above
Also I needed to install and start docker-machine
All working now on OSX High Sierra

[kareiva]

Related to https://github.com/openshift/origin/issues/20633 ?

[Avdhey]

is this related to https://github.com/openshift/origin/issues/21512#issuecomment-441664590 ?
cluster up for 3.11 is not supported on macOS directly, you need mini-shift to run 3.11 clusters.

[knrc]

@Avdhey We've had no problems running 'oc cluster up' with previous releases and the only issue we have discovered on MacOS has been this one with the 3.11 tool. The cause was the introduction of a very small, linux specific change being introduced to the codebase. We publish our own version of 'oc' so just patched it ourselves.

Given 'oc cluster up' and minishift are being replaced with 4.x this issue can likely be closed as "Won't Fix"

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[murphye]

For those that may still care about running 3.11 on a Mac, I can confirm that https://github.com/Maistra/origin/releases is a workaround for this problem. It does not seem to have the error documented here, but is using 3.11 images for oc cluster up.

However, I had problems getting authentication working with registry.redhat.io to pull down images. You will not be able to deploy any of the apps from the Catalog. This relates to the openshift/imagestreamsecret needed to authenticate. I updated my ~/docker/config.json to include the need auth info and reinstalled the imagestreamsecret secret with no success. I am sure there is a way to get it working, however.

A workaround for that issue would be to use the alternate centos7 image stream:
oc apply -n openshift -f https://raw.githubusercontent.com/openshift/origin/master/examples/image-streams/image-streams-centos7.json

Please be aware there is now CodeReady Containers for OpenShift 4.2 (https://github.com/code-ready/crc) but it's pretty memory intensive on a Mac. That being said the version of oc cluster up for Maistra is eating a lot of CPU for some reason.