Origin: oc cluster up does not work in a proxied environment

[jorgemoralespou]

In an environment with a proxy oc cluster up can start, if the proxy is configured for Docker daemon but builds (as an example) will not work.
Provide a way to configure a proxy in the config generated by oc cluster up. And if it is currently possible, please, document.

[jorgemoralespou]

@csrwng This short of creates many dependencies in minishift. Do you have an initial idea on what would be required to support this?

[csrwng]

@jorgemoralespou thank you for reminding me about this. At the very least we should document the configuration. We need to do some testing to see what it will take, but basically we need to configure the default proxy settings for builds. We'd also need to configure the proxy for registry access when importing image streams. I'll definitely take a look on Wednesday

[LalatenduMohanty]

@csrwng This is also required from CDK/minishift side. As from our experience we have seen many users using CDK behind proxy environment.

[pittar]

@csrwng have you been able to look into configuration documentation by any chance? We are also behind a corporate proxy and can't create image streams when using "oc cluster up".

Environment:
Windows 7
Docker Toolbox (VirtualBox)
OpenShift CLI 1.3.2

Using Docker Toolbox we can pull images, and we can start OpenShift with "oc cluster up", but OpenShift itself can't pull any images.

Somehow, I need to figure out how to set HTTP_PROXY and HTTPS_PROXY for OpenShift itself.

Thanks.

[csrwng]

@pittar we are working on adding a flag to cluster up to let you specify the proxy server to use. However, if you want HTTP_PROXY and HTTPS_PROXY environment variables to be set for origin, you can use the -e flag on cluster up currently.

[pittar]

Thanks @csrwng. I gave it a shot, unfortunately it seems I'm now running into https://github.com/openshift/origin/issues/9565.

Any other suggestions? Once I set the HTTP_PROXY and HTTPS_PROXY, I need to be able to add the clusterNetworkCIDR AND the serviceNetworkCIDR to the NO_PROXY list, so I need a comma.

[csrwng]

@pittar you can specify multiple -e arguments

[jorgemoralespou]

@csrwng will multiple -e concatenate the ENV?
If he needs NO_PROXY=a,b will that be achieved by -e NO_PROXY=a -e
NO_PROXY=b?

[csrwng]

@jorgemoralespou yes it will

[csrwng]

@jorgemoralespou sorry, I spoke too quickly ... actually no

[pittar]

@jorgemoralespou
Yes, that's the problem I'm running into. I need to add three entries (I believe) to my NO_PROXY variable, but only the 1st one works. If I add three -e NO_PROXY arguments, then only one of those is actually set.

I wish we didn't have to deal with a proxy! This all works so nicely on my laptop at home ;)

[csrwng]

@pittar did you try the workaround suggested in the issue you linked to?
--flag='"foo,bar"'

[pittar]

Hi @csrwng, I have tried:
`oc cluster up --env=HTTP_PROXY=proxy.host,HTTPS_PROXY=proxy.host,NO_PROXY="10.128.0.0/14,172.30.0.0/16,192.168.0.0/16"

oc cluster up --env HTTP_PROXY=proxy.host --env HTTPS_PROXY=proxy.host --env NO_PROXY="10.128.0.0/14,172.30.0.0/16,192.168.0.0/16"

oc cluster up --env HTTP_PROXY=proxy.host --env HTTPS_PROXY=proxy.host --env='"NO_PROXY,10.128.0.0/14,172.30.0.0/16,192.168.0.0/16"'

`oc cluster up --env HTTP_PROXY=proxy.host --env HTTPS_PROXY=proxy.host --env='"NO_PROXY,"10.128.0.0/14,172.30.0.0/16,192.168.0.0/16""'

The examples with --flag='"foo,bar"' seem strange, since I need to properly nest:
--env='"key,value'" where value is another quoted string with commas.

I'm probably doing something dumb, but it's hard to see.

[csrwng]

@pittar how about:
oc cluster up --env 'HTTP_PROXY="proxy.host"' --env 'HTTPS_PROXY="proxy.host"' --env 'NO_PROXY="10.128.0.0/14,172.30.0.0/16,192.168.0.0/16"'

[pittar]

@csrwng

NO_PROXY isn't set with that option:

`$ docker exec -it origin bash
[root@default origin]# echo $HTTP_PROXY
proxy.host
[root@default origin]# echo $HTTPS_PROXY
proxy.host
[root@default origin]# echo $NO_PROXY

`

I'll keep trying different combos. I'm sure one will work eventually. If I figure it out, I'll post the answer.

[pittar]

@csrwng @jorgemoralespou

Ok, I got it to work!

oc cluster up --env 'HTTP_PROXY=proxy.host' --env 'HTTPS_PROXY=proxy.host' --env '"NO_PROXY=10.128.0.0/14,172.30.0.0/16,192.1 68.0.0/16"'

When I bash into the "origin" container, I now see a properly set HTTP_PROXY, HTTPS_PROXY, and the full NO_PROXY list.

Thanks for your help.

[jorgemoralespou]

@csrwng is that enough to have proxies working? There's no need to add
anything to master-config.yaml to enable proxies to work in builds?

[pittar]

@jorgemoralespou
This config now lets me run images from Docker Hub. I haven't tried builds yet.
When I bash into the "origin" container, I see properly set HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. Two of the three CIDRS in my NO_PROXY list above are from the master-config.yaml file as specified in this doc: https://docs.openshift.org/latest/install_config/http_proxies.html

[csrwng]

@jorgemoralespou you still need to configure the build defaults as explained here: https://docs.openshift.org/latest/install_config/build_defaults_overrides.html#manually-setting-global-build-defaults

[jorgemoralespou]

@pittar this means that for building you're going to have some additional
work to do that is not easy.

Out of curiosity, would you expect to use the same cluster in different
locations, with and without proxy? Meaning that you create the oc cluster
without proxy, work, and keeping the config would you go to your office,
start the cluster with the proxy work more, go back home and work more
without proxy and so on? Or would you expect to create a cluster for each
location, home and work? Or would you create a cluster everytime not
keeping the config as you move?

Thanks for the answer.

[pittar]

@jorgemoralespou we're new to OpenShift and trying to figure out the best way to setup developer work stations.

I was experimenting with creating templates at home on my personal laptop using "oc cluster up" (MacOS, no proxy). That was working great.

I had trouble (as you can see) getting the same setup running at work. Here, we're on Windows 7 (so only Docker Toolbox) and behind a corporate proxy. This gave me some fits!

Now that OpenShift is running with "oc cluster up" (thanks @csrwng), I've been able to import the same templates and start up a few of our apps. We already had some apps on Docker Hub. This part is now working.

I haven't tried builds, and it's not on my near-term list. I still want to try the CDK (mentioned here) to see if that's more appropriate for developer workstations.

To more directly answer your questions:
1) No, we don't plan on using the same cluster in different locations. The end result will be each developer has his/her own cluster on their local PC (behind the proxy).
2) No, there wouldn't be any copying of config.
3) Each location (in this case, each developer work station) would have it's own single-node OpenShift for dev.
4) We might create a cluster every time. The idea of a clean env whenever you want it does sound nice. I think we'll need to see how that goes, though.

I hope that helps!

[adamjk-dev]

+1 here. We need to be able to bring a cluster up on a node behind a corporate proxy. I am not having a bunch of luck with what is in these comments. It seems like however I set the environment variables something won't work (either oc cluster up won't work with those proxy vars set, or when I bring the cluster up and set them manually on origin, I am not able to pull things down via the registry).

From my end, it has been an issue getting something to work behind a corporate proxy (CDK and/or oc cluster up). It has been a battle to try to get something to work. We really just need an easy way to get developers local access to OSE so they can vet code/apps before promoting and running in a real environment.

[csrwng]

@adamjk-dev we're working on adding arguments to let you specify the proxy.
https://github.com/openshift/origin/pull/12483

[lucastheisen]

Perhaps this is the wrong place to ask, but when running oc cluster up behind my corporate proxy, it fails with:

PS C:\Users\ltheisen> oc cluster up
Starting OpenShift using openshift/origin:v3.6.0 ...
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v3.6.0 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ...
   Using Docker shared volumes for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ...
   Using 10.0.75.2 as the server IP
-- Starting OpenShift container ...
   Creating initial OpenShift configuration
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
FAIL
   Error: cannot access master readiness URL https://10.0.75.2:8443/healthz/ready
   Details:
     No log available from "origin" container

   Caused By:
     Error: Get https://10.0.75.2:8443/healthz/ready: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

This makes sense because 10.0.75.2 is being _proxied_... If i add that IP to my no_proxy in my console, then it works:

PS C:\Users\ltheisen> $env:no_proxy="$env:no_proxy,10.0.75.2"
PS C:\Users\ltheisen> oc cluster up
Starting OpenShift using openshift/origin:v3.6.0 ...
OpenShift server started.

The server is accessible via web console at:
    https://10.0.75.2:8443

You are logged in as:
    User:     developer
    Password: <any value>

To login as administrator:
    oc login -u system:admin

WARNING: An HTTP proxy (http://gatekeeper-w.mitre.org:80) is configured for the Docker daemon, but you did not specify one for cluster up
WARNING: An HTTPS proxy (http://gatekeeper-w.mitre.org:80) is configured for the Docker daemon, but you did not specify one for cluster up
WARNING: A proxy is configured for Docker, however 172.30.1.1 is not included in its NO_PROXY list.
   172.30.1.1 needs to be included in the Docker daemon's NO_PROXY environment variable so pushes to the local OpenShift registry can succeed.

The problem is that the IP is dynamically allocated by docker... So this is a bit of chicken and egg... Is there a suggested approach to get around this? Or am i missing something obvious?

[teotia39]

Guys , If i follow openshift origin advanced installation @
https://docs.openshift.org/latest/install_config/install/advanced_install.html ,

then post installation do i still need to execute "oc cluster up" command.

Second question while creating inventory files , we have set of variables like "openshift_http_proxy" "openshift_https_proxy" and even no_proxy variables which we can set , Does this variables will make sure that proxy at my machine level , docker level will be set automatically and i will be able to pull required images fro router and registry from internet and y cluster will start working , As of now none of my pods i.e. registry pod and router pod is not running , am getting error syncing pod . I tried to set proxy post advanced installation @ https://docs.openshift.com/enterprise/3.0/admin_guide/http_proxies.html

but then the defualt pod's goes off and then i get pod syning failed.

Should i set my office proxy at docker level and machine level separately.

[jorgemoralespou]

Have you tried setting those parameters in "oc cluster up"?

If you do "oc cluster up -h" you'll see there's 3 parameters related to proxies:

      --http-proxy='': HTTP proxy to use for master and builds
      --https-proxy='': HTTPS proxy to use for master and builds
      --no-proxy=[]: List of hosts or subnets for which a proxy should not be used

Maybe you should try to set those directly to the "oc cluster up" command.

[teotia39]

Should i execute this after executing the openshift ansible playbook for openshift origin installation ???????

[jorgemoralespou]

@teotia39 since this issue is for "oc cluster up" not sure why you're running the openshift installation. My reply was to @lucastheisen.

"oc cluster up" is to run openshift locally on your laptop. Not for when you do a full install of a cluster with ansible.
To your second question, those ansible variables will apply to the nodes where you're installing openshift with ansible, not to your client machine.

[lucastheisen]

@jorgemoralespou , I do set those, but this issue is not _inside_ of openshift... Its when oc cluster up is processing. It looks like oc cluster up runs the origin container, then tries to connect to it at the IP assigned to the container by docker (_Error: cannot access master readiness URL https://10.0.75.2:8443/healthz/ready_). So its not outbound container traffic (which is working fine), its console oc command traffic (from my host to the origin container). And because I don't know the IP of the origin container until it is run, I don't know what I have to add to my host no_proxy. Does that clarify?

[cliffbdf]

I get exactly the same behavior as lucastheisen. My organization also has a proxy. I am setting both HTTP_PROXY and HTTPS_PROXY.