Origin: cluster up fails to push to the registry on Docker for Mac CE 17.12.0 and newer

Created on 13 Feb 2018 · 46Comments · Source: openshift/origin

Docker for Mac CE 17.12.0 introduced an internal proxy for Docker that prevents pushes into the local OpenShift registry when running a cluster with cluster up

You also get a warning when starting cluster up like:

WARNING: An HTTP proxy (docker.for.mac.http.internal:3128) is configured for the Docker daemon, but you did not specify one for cluster up
WARNING: An HTTPS proxy (docker.for.mac.http.internal:3129) is configured for the Docker daemon, but you did not specify one for cluster up
WARNING: A proxy is configured for Docker, however 172.30.1.1 is not included in its NO_PROXY list.
   172.30.1.1 needs to be included in the Docker daemon's NO_PROXY environment variable so pushes to the local OpenShift registry can succeed.

For now, the workaround is to downgrade Docker to a version prior to 17.12.0. You can find download links to previous releases in their release note page:
https://docs.docker.com/docker-for-mac/release-notes/

Open Issues:
https://github.com/docker/for-mac/issues/2506
https://github.com/docker/for-mac/issues/2470
https://github.com/docker/for-mac/issues/2467
https://github.com/docker/for-mac/issues/2386

componencli kinbug lifecyclrotten prioritP2 simaster

Source

csrwng

👍20 😕3

All 46 comments

@mfojtik @deads2k
/cc @mpbarrett

csrwng on 13 Feb 2018

I would hold on fixing this until @deads2k finish the internal refactoring of cluster up.

mfojtik on 14 Feb 2018

👍1

same here, is there any other workaround for this? I don't want to downgrade...
adding 172.30.1.1 to no_proxy in docker prefs does not help
docker Version 18.02.0-ce-mac53 (22617)

riuvshin on 23 Feb 2018

👍1

I haven't be able to find a work around.. No proxy doesn't seem to work.

alberttwong on 2 Mar 2018

Docker Community Edition 18.03.0-ce-mac59 2018-03-26
Fix for the HTTP/S transparent proxy when using “localhost” names (e.g. host.docker.internal).

This does not fix the issue. So don't upgrade.

mikz on 6 Apr 2018

😕4

Confirming that I'm still seeing this with
2018-04-30 18.03.1-ce-mac65

cmcconnell1 on 30 May 2018

😕2

So decided to try this again and still broken with docker version 18.03.1-ce. At least it's nice to know it's a verified issue/bug. Bummed there is no resolution besides using an older version of docker though.

Valien on 19 Jun 2018

Apologies, I previously automated workarounds for our team shortly after my posts above and then I forgot about this issue. I'm new to Minishift/Openshift (coming from deis workflow && looking at OS Origin to provide a solid PaaS solution), and not sure if this would work for everyone.

TL;DR:
Here is my initial POC (with docker for mac minishift workaround) for a local minishift cluster:
minishift-local-poc
Its automated and allows our distributed teams to quickly config and deploy our platform within a few minutes.

Details

For our use case, we're working locally with minishift, pulling from remote private AWS ECR repos, tagging and pushing our images to the remote (Openshift / Minishift registry--AFAIK, minishift still considers this to be a remote) docker registry.
I had found another blog post from last year where they needed to use secrets else they were getting the permission denied errors (as was I) so set those up, and automated that process as well. I'm not sure if that's the correct "Openshift/Minishift way," but it works for us so I went with it.
My solution configures a minishift cluster with requisite configs such that future starts can just be minishift start and it will use the default profile configurations specified in the script. Then the parent script spawns child script/processes, which each configure requisite apps for our platform in separate minishift project/namespaces--with each app pulling/tagging their requisite docker images from private AWS ECR repo and pushing into the Minishift docker registry--i.e.:

docker push 172.30.1.1:5000/myapp-local/myapp

Another bit of advice from my experience was that the virtualbox driver just worked--('seem to recall some other drivers missing plugins, configurations, etc.) so went with VB driver and that = profit for me. JIC, my script will install the requisite minishift plugins, enable them, and also set admin creds--this should only be used in local/dev env's as it is not secure. So it will configure the driver to be virtualbox as the default, but you can easily change that as you wish.
Notes
I'm currently Not running local docker, but using Minishift docker context--will that work for you?
This parent script sets the Docker context to the Minishift Registry and thus, I'm not even running Docker at all on my OSX laptop/workstation
- Note: see the above noted git repo for the minishift deployment script which pushes, to Minishift docker registry without any local docker for mac running.

$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

AIR, you Can, have local docker running with this script without issues too, since the script will set the requisite context.

DOCKER CONTEXT
# switch to minishift context
eval $(minishift docker-env)
#eval $(minikube docker-env) # if we were using minikube

env | grep -i docker
DOCKER_HOST=tcp://192.168.99.100:2376
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/Users/cmcc/.minishift/certs

# switch back to your machines context do:
eval $(docker-machine env -u)

I hope this helps others and would love to hear how folks are automating their docker images in Openshift Origin in H/A production environments, as I'm currently rolling my own scripts ATM to facilitate app creation, deployment, rollback, etc.
I'm curious if the masses would recommend embracing using Openshift templates? This seems very cool to be able to export any/all requisite NS/Project artifacts and codify/promote/manage them, etc.

-cmcc

cmcconnell1 on 19 Jun 2018

@cmcconnell1 I just settled on oc cluster up --create-machine.

mikz on 20 Jun 2018

👍1

I am seeing this issue on docker version 18.06.1-ce-mac73 as well. Had to downgrade to 17.09 as a work around

patelbharat on 3 Sep 2018

@patelbharat You can just dooc cluster up --create-machine.

fg78nc on 3 Sep 2018

@fg78nc that is gone in 3.10.

mikz on 3 Sep 2018

👍1

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     true

$ oc version
oc v3.10.0+dd10d17
kubernetes v1.10.0+b81c8f8
features: Basic-Auth

Server https://127.0.0.1:8443
openshift v3.10.0+e3465d0-44
kubernetes v1.10.0+b81c8f8

Still same issue.

$ oc cluster up --create-machine
Error: unknown flag: --create-machine

bappr on 14 Sep 2018

Confirmed working with

docker version
Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:22:25 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:28:28 2017
 OS/Arch:      linux/amd64
 Experimental: true

bappr on 14 Sep 2018

I still have this problem with docker q8.06.01-ce-mac73 and 3.10. Only updated as i thought this was fixed.

rebeccaSimmonds19 on 10 Oct 2018

FYI OpenShift 4.0 won't run "bare" on macOS, period. Minishift is the only viable path to running a local cluster on macOS at this point.

@csrwng once minishift supports 4.0 we can probably close this issue.

adambkaplan on 11 Oct 2018

😕4

@adambkaplan minishift with hyperkit runner is basically the same as Docker for macOS. And that works. So, why it would not run on plain Docker for mac? Just because of some proxy setting?

mikz on 11 Oct 2018

@adambkaplan Is your comment summing this up as a docker for mac issue and not an OpenShift issue at this point (due to the proxy settings being ignored by Docker for Mac)?

Could you elaborate by what you mean by "bare" on Mac OS? You are just reiterating some form of virtualization is required to provide a Linux kernel?

leetrout on 11 Oct 2018

@leetrout @mikz for OpenShift 3.11, the barrier is the outstanding docker for mac issue, which has been there for quite some time. Running "bare" via oc cluster up (which is what I meant) will fail if you're using Docker for Mac > 17.09.

OpenShift 4.0 (which is currently alpha.0) is a whole different ballgame. We don't have support for running local clusters on a Mac at present.

adambkaplan on 11 Oct 2018

😕1

@adambkaplan what about Linux? would it still be possible to run oc cluster up ?
that is a significant change that will break a lot of CI testing pipelines where products are tested on local openshift clusters. For testing purposes there is no way to run whole proper OS cluster and in the same time you can't run minishift with VM backend on CI infras due to nested virtualization issues.

riuvshin on 11 Oct 2018

👍1

@riuvshin cluster up works on RHEL/CentOS/Fedora, and in principle any other Linux distro that supports docker and/or cri-o.

FYI cluster up is going away with 4.0. I'd refer you to openshift/installer if you want to follow how 4.0 installations will work.

adambkaplan on 11 Oct 2018

😕3

@adambkaplan would it be possible to run local cluster as a set of docker containers with https://github.com/openshift/installer?

riuvshin on 12 Oct 2018

Hi.

I have same issue with docker 18.06.1-ce-mac73 (26764)

I try running a cluster with oc cluster up --loglevel=5 --http-proxy=gateway.docker.internal:3128 --https-proxy=gateway.docker.internal:3129 and I'm getting this warning.

WARNING: A proxy is configured for Docker, however 172.30.1.1 is not included in its NO_PROXY list.
172.30.1.1 needs to be included in the Docker daemon's NO_PROXY environment variable so pushes to the local OpenShift registry can succeed.

The ~/Library/Group\ Containers/group.com.docker/http_proxy.json file has the following content:

{
  "exclude": "172.30.1.1",
  "transparent_http_ports": [
    80
  ],
  "transparent_https_ports": [
    443
  ]
}

Does this problem have a solution other than downgrade?

hakdogan on 5 Nov 2018

@hakdogan

Still using origin v3.6 but using docker-machine works for me

$ eval $(docker-machine env openshift) 
# needs pre created docker-machine, in my was case it was created by oc cluster up --create-machine

$ env | grep -i docker
# just to check that we are referencing the docker machine

$ oc cluster up 
# will create cluster in docker-machine

Note: I don't have a good way of persisting data. Stopping the docker-machine will wipe openshift cluster data

uLan08 on 20 Nov 2018

If you head over to okd.io and download minishift, it uses the latest version of Openshift Origin...

$ minishift start
$ minishift oc-env
$ oc login etc

Hate having the VM but it works.

bobbydeveaux on 20 Nov 2018

😕2

@uLan08
Hi Patrick

Unfortunately this didn't work for me. I got following error when executing the command.

oc cluster up --create-machine --loglevel=5

...
FAIL
   Error: could not create OpenShift configuration
   Caused By:
     Error: cannot start container f40c3852875bacceac537247005a516b1a9754ed0a5f10f88ed4600707f52a70
     Caused By:
       Error: Error response from daemon: {"message":"linux mounts: path /var/lib/origin/openshift.local.volumes is mounted on / but it is not a shared or slave mount"}

@bobbydeveaux

Hi Bobby,

Thank you for your advice but I don't want to use minishift.

hakdogan on 20 Nov 2018

Issue is still here((( When it will be fixed? Minishift haven't got actual documentation. All example explained via oc cluster up. I can't complete my research about ASB because oc doens't work and documentation for minishift doesn't work too or it's 404 on pages!

@mfojtik Hi! Do you have any additional information?

Asgoret on 26 Nov 2018

@Asgoret I used to downgrade docker for mac to get this worked...

riuvshin on 26 Nov 2018

@Asgoret I used to downgrade docker for mac to get this worked...

@riuvshin I upgrade Docker for macOS today....so which version i work?

Asgoret on 26 Nov 2018

@alberttwong one question. If oc cluster up won't work on macOS is that mean, that documentation will be corrected for minishift in short time? At this moment there is no actual documentation for large part of OKD in minishift. For example ASB installer documentation shows 404 error and minishift addons doesn't updates for 7 months or more time.

Asgoret on 26 Nov 2018

In addition to the thought of @Asgoret, I think that also the OKD documentation is not enough. There is no explanation for Mac users about this problem. I used to downgrade because there are no other options.

hakdogan on 26 Nov 2018

@Asgoret I was using this version https://download.docker.com/mac/stable/18433/Docker.dmg
this is 17.06.0-ce-mac18

riuvshin on 26 Nov 2018

still an issue with docker for mac and oc cluster up:
Docker version 18.09.1, build 4c52b90
will try to downgrade and check if it works.

rb2010 on 15 Feb 2019

Any updates?

benweizhu on 6 May 2019

I downgrade to Docker version 17.09.1-ce, build 19e2cf6, it works~

benweizhu on 6 May 2019

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot on 4 Aug 2019

/remove-lifecycle stale

infinitewarp on 8 Aug 2019

Issues go stale after 90d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot on 6 Nov 2019

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot on 6 Dec 2019

/remove-lifecycle rotten

infinitewarp on 16 Dec 2019

Issues go stale after 90d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot on 16 Mar 2020

/remove-lifecycle stale

infinitewarp on 25 Mar 2020

Issues go stale after 90d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

openshift-bot on 23 Jun 2020

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot on 23 Jul 2020

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

openshift-bot on 22 Aug 2020

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.