Origin: oc cluster up fails on new docker check (boot2docker uses 17.03 now)

[eschabell]

On osX using oc to install a version v3.4 OSE image for OpenShift Container Platform and it fails due to the image being spun up using boot2docker now contains newest version of Docker v17.03.x. The oc command checks agains the minor number, failing due to the leading zero (string comparison versus integer comparison fails I think).

Version

$ oc version
oc v3.4.1.2
kubernetes v1.4.0+776c994features: Basic-Auth

$ docker version
Client:
Version: 1.13.1
API version: 1.26
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 08:47:51 2017
OS/Arch: darwin/amd64

Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 08:47:51 2017
OS/Arch: linux/amd64
Experimental: true

Steps To Reproduce

1. 'oc cluster up --image=registry.access.redhat.com/openshift3/ose --version=v3.4.1.5 --create-machine'
2. fails with error:
   -- Checking OpenShift client ... OK
   -- Create Docker machine ...
   Creating docker-machine openshift
   -- Checking Docker client ... OK
   -- Checking Docker version ... FAIL
   Error: Minor number must not contain leading zeroes "03"

Current Result

Error: Minor number must not contain leading zeroes "03"

Expected Result

Successful install of cluster OCP

Additional Information

See attached screenshot where I watch the virtualBox booting with boot2docker and the version is circled and my error in the console is circled.... should be clear enough?

Question: can I get a new OC or point me to the source to build a new oc?

[stevekuznetsov]

Building a new oc won't help here -- I think we are using a semantic version parsing utility ... and this version is not semantic, so we'll need to change that.

[stevekuznetsov]

That being said, I don't think we support bleeding edge Docker anyway, 1.5 release will be using docker-1.12

[csrwng]

We have a fix going in today for this
https://github.com/openshift/origin/pull/13201

[eschabell]

https://github.com/openshift/origin/blob/85eb37b34f0657631592356d020cef5a58470f8e/vendor/github.com/blang/semver/semver.go#L232

The exact error... oef!

[eschabell]

I am using 1.13 locally, but can't prevent whomever updated the boot2docker.... so what can I use to avoid the latest update?

[eschabell]

@csrwng : I see you put in the fix on #13201, where will that oc appear for download so I can test it for you?

[csrwng]

@eschabell you can build it yourself locally :-)

oc new-app -f https://raw.githubusercontent.com/csrwng/build-origin/master/origin-builder.yaml -p SOURCE_URL=https://github.com/csrwng/origin.git -p SOURCE_REF=docker_version_warning

[eschabell]

@csrwng just waiting for the patch to merge... thanks for the quick turnaround.

One question, how to run oc without a running cluster, as when I do this now I am getting:

"Unable to connect to the server: dial tcp 192.168.99.100:8443: connect: host is down"

[stevekuznetsov]

What actions are you expecting to take without a cluster?

[csrwng]

@eschabell sorry you do need a cluster to build the oc client

[SillyMoo]

Hi, I don't suppose this fix will be back ported to older openshift versions (i.e on a new minor release) will it?

[csrwng]

@SillyMoo generally no ... with very few exceptions only security fixes are backported

[eschabell]

Fix worked for me.... but the tag is a bit community, "v1.5.0-alpha.3+b6fbeef-176", when and where will this be added to the online openshift tools downloads?

[eschabell]

Also note, this could have been prevented if the .docker/machine/cache/boot2docker.iso was used if it was in place, but it is downloaded each time you run oc cluster up.... seems a waste to cache it at all if that is how it works....

[ppatierno]

@eschabell ... I had same error and I used minishift for the cluster. I run the command for building the oc tool but now I can't find the compiled version. Where is it supposed to be ?

[eschabell]

@ppatierno There is a route created by the container doing the build..... it's in the top right corner... it exposes it via web server I think and you can download it. Btw, if you can't build locally like I had, you can let some Linux user do it, still a Darwin binary. ;-)

[ppatierno]

@eschabell yes I realized about that after writing you ... btw ...

I think I have another problem. I had your same error on Fedora not OSX. The above command rebuild oc for Darwin so it can't work on my Fedora. Now ?

[eschabell]

@csrwng can you provide a build 'oc new-app' for @ppatierno like you did for Darwin, but he needs to have Linux binary?

[ppatierno]

@eschabell I'm trying to do it by myself changing this value in the provided yaml file :

- name: PLATFORM
  description: "The platform of the oc binary to build. Valid values: linux/amd64, darwin/amd64, windows/amd64"
  displayName: "Platform"
  value: darwin/amd64  #--> to linux/amd64

I'll let you know !

[ppatierno]

@eschabell it worked ! I can use my oc now ;)

[danwinship]

Building a new oc won't help here -- I think we are using a semantic version parsing utility ... and this version is not semantic, so we'll need to change that.

In upstream kubernetes code this should just be a matter of switching from "k8s.io/kubernetes/pkg/util/version".ParseSemantic() to .ParseGeneric(). In origin I think there are still places that are using semver directly, but it's an easy port to utilversion.

[ppatierno]

The oc tool works, at least I don't receive the parsing error and I can start some pods. One pod doesn't start due to following error :

2017-03-03 13:30:07 WARN Config:298 - Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
Error starting address controller: /var/run/secrets/kubernetes.io/serviceaccount/token

[lukaskk]

windows 10

:\Users\praca>oc cluster up --version=v1.3.0
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... FAIL
Error: Minor number must not contain leading zeroes "03"

[ppatierno]

My summary is that ....

Updated Docker 17.03 CE breaks my oc tool 1.4.1-rc1 with above error.

Error: Minor number must not contain leading zeroes "03"

I re-built the oc tool with above patch having 1.5.0-alpha3. The above break is fixed but if a pod running on openshift needs secrets I have following error :

2017-03-03 13:30:07 WARN Config:298 - Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
Error starting address controller: /var/run/secrets/kubernetes.io/serviceaccount/token

My only way to have all working was to come back to Docker 1.13 and oc tool 1.4.1-rc1 :-(

I hope a better solution will be available soon.

[jarcher]

I've gone back to the CDK for now...saw the Alpha release for the devtools on mac - one item I was trying to determine was how to get a hold of older version of docker.

[eschabell]

Here the docs point to the downloads: https://docs.docker.com/v1.13/

[spicysomtam]

Ho hum. Thought this was fixed in 3.6.0-alpha tools client. cluster up works fine, but I get it creating a pod:

44s 12s 3 {kubelet localhost} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: docker: failed to parse docker version \"17.04.0-ce\": illegal zero-prefixed version component \"04\" in \"17.04.0-ce""

Am I on the right issue or has someone raised another issue for this?

[tlawrence]

I'm having the same issue @spicysomtam. Read elsewhere that this is fixed in master but don't really want to use an untagged release.
Looks like downgrading Docker is the best answer but can't find decent instructions to do that on a Mac. Any idea?

[tlawrence]

from @cloakedcode in another issue:

There are two ways to downgrade:

1. Recreate the URL

If you happen to know the build number (e.g. 14937) -- not the git commit you see when running docker --version or docker version -- you can form the URL:

https://download.docker.com/mac/{{ beta | stable}}/{{ version }}.{{ build number }}/Docker.dmg

example:

https://download.docker.com/mac/stable/1.12.6.14937/Docker.dmg
(Note: the build number shows up in "About Docker" from the menulet menu.)

screen shot 2017-01-23 at 2 12 56 pm

Of course, you'd need to know the build number, which is hard to do if you've already upgraded.

1. Pull it from Trash

After upgrading, the old Docker.app gets put in the Trash (e.g. Docker (14937)). Uninstall the current version, Preferences... -> Uninstall / Reset -> Uninstall; delete /Applications/Docker; copy the old version from the Trash into /Applications; rename to Docker; and launch Docker to begin the install process.

[spicysomtam]

I use ubuntu, so I downgraded docker-engine:

 sudo apt-get remove docker-engine
 sudo apt-get install docker-engine=1.13.1-0~ubuntu-xenial

Go and edit your docker systemd start script and add the insecure registry again.

sudo vi /lib/systemd/system/docker.service
sudo systemctl daemon-reload
sudo systemctl restart docker

[mcrobbjohn]

How do I get a fix for this without downgrading docker.

[spicysomtam]

From what i understand you either build from the latest commits or you wait until a release that fixes it. Since the latest releases of docker are about about having community and enterprise releases of docker, not much has changed in docker, so its easier to downgrade docker.

[eschabell]

This is happening again for the OCP 3.5:

On start-build for a new container, it gives this in the build events log:

Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: docker: failed to parse docker version \"17.04.0-ce\": illegal zero-prefixed version component \"04\" in \"17.04.0-ce""
20 times in the last 4 minutes

Using OSX and OC v3.5.5.5-2.

How is anyone using OCP 3.5?

[stevekuznetsov]

As per the docs:

OpenShift Container Platform 3.5 requires Docker 1.12.

[eschabell]

@stevekuznetsov, this version I believe is in the container build tooling (boot2docker), not on my local machine. I am using 1.13 and it references the latest docker engine (17.04.0-ce) as that's what you get when it tries to use boot2docker in the container build start up.

[kanedafromparis]

Hi,

I think the workaround is explain into this thread, nevertheless I had a hard time to deduct it properly, so here is my2ç (on mac osx) :

docker-machine create --driver driver-name \
          --engine-insecure-registry 172.30.0.0/16 \
          --virtualbox-boot2docker-url  https://github.com/boot2docker/boot2docker/releases/download/v1.13.1/boot2docker.iso \
        my-docker-machine

then

 oc cluster up --metrics=false --create-machine=false \
          --docker-machine='my-docker-machine'

Which allow me to :

    docker-machine create --driver virtualbox \
     --virtualbox-cpu-count "4" 
     --virtualbox-memory "8192"
     --engine-insecure-registry 172.30.0.0/16 \
     --virtualbox-boot2docker-url  https://github.com/boot2docker/boot2docker/releases/download/v1.13.1/boot2docker.iso \
      ose13-cb &&
    oc cluster up --metrics=false \
       --use-existing-config=false 
       --version="v1.3.3" --create-machine=false \
       --docker-machine='ose13-cb'

(instead, you probably can use : --virtualbox-import-boot2docker-vm to use an existing vm)