Kubespray: proxies break due to changes in 2.13.0

[billykwooten]

Environment:

Cloud provider or hardware configuration:

ESX 7.0

OS:

Linux 4.15.0-99-generic x86_64
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Version of Ansible (ansible --version):
ansible 2.9.6
config file = /root/kubespray/kubespray/ansible.cfg
configured module search path = ['/root/kubespray/kubespray/library']
ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]

Version of Python (python --version):

Python 3.6.9

Kubespray version (commit) (git rev-parse --short HEAD):
01dbc909

Network plugin used:
Calico

Command used to invoke ansible:
ansible-playbook -i inventory/homedepot/hosts.yml cluster.yml --flush-cache

Output of ansible run:

This is the error I am getting on my run:

2020-05-08 07:28:27,441 p=9123 u=root n=ansible | TASK [download : download_file | Download item] ********************************
2020-05-08 07:28:27,443 p=9123 u=root n=ansible | fatal: [ldk00474.homedepot.com -> 10.16.34.255]: FAILED! => {"attempts": 4, "changed": false, "dest": "/tmp/releases/kubeadm-v1.17.5-amd64", "elapsed": 10, "msg": "Request failed: <urlopen error [Errno 99] Cannot assign requested address>", "url": "https://storage.googleapis.com/kubernetes-release/release/v1.17.5/bin/linux/amd64/kubeadm"}
2020-05-08 07:28:27,531 p=9123 u=root n=ansible | fatal: [ldk00473.homedepot.com -> 10.16.36.139]: FAILED! => {"attempts": 4, "changed": false, "dest": "/tmp/releases/kubeadm-v1.17.5-amd64", "elapsed": 10, "msg": "Request failed: <urlopen error [Errno 99] Cannot assign requested address>", "url": "https://storage.googleapis.com/kubernetes-release/release/v1.17.5/bin/linux/amd64/kubeadm"}

Anything else do we need to know:

I have 3 hosts that run Ubuntu 18.04, I ran the playbook from host #1 to configure itself and the other 2 hosts with kubespray. When setting the proxies I get the above error.

I saw that in v2.13.0 we moved proxies to cluster.yml (see here)

In older versions proxy config was done via kubespray-defaults/main.yaml see here

On v2.13.0 I get the above error, however if I go delete kubespray-defaults/tasks/no_proxy.yml and remove it's task from kubespray-defaults/tasks/main.yml, delete the new proxy stuff from cluster.yml, then go into kubespray-defaults/defaults/main.yaml and add back in the following:
```

Set no_proxy to all assigned cluster IPs and hostnames

no_proxy: >-
{%- if http_proxy is defined or https_proxy is defined %}
{%- if loadbalancer_apiserver is defined -%}
{{ apiserver_loadbalancer_domain_name| default('') }},
{{ loadbalancer_apiserver.address | default('') }},
{%- endif -%}
{%- for item in (groups['k8s-cluster'] + groups['etcd'] + groups['calico-rr']|default([]))|unique -%}
{{ hostvars[item]['access_ip'] | default(hostvars[item]['ip'] | default(fallback_ips[item])) }},
{%- if item != hostvars[item].get('ansible_hostname', '') -%}
{{ hostvars[item]['ansible_hostname'] }},
{{ hostvars[item]['ansible_hostname'] }}.{{ dns_domain }},
{%- endif -%}
{{ item }},{{ item }}.{{ dns_domain }},
{%- endfor -%}
{%- if additional_no_proxy is defined -%}
{{ additional_no_proxy }},
{%- endif -%}
127.0.0.1,localhost,{{kube_service_addresses}},{{kube_pods_subnet}}
{%- endif %}
proxy_env:
http_proxy: "{{ http_proxy| default ('') }}"
HTTP_PROXY: "{{ http_proxy| default ('') }}"
https_proxy: "{{ https_proxy| default ('') }}"
HTTPS_PROXY: "{{ https_proxy| default ('') }}"
no_proxy: "{{ no_proxy| default ('') }}"
NO_PROXY: "{{ no_proxy| default ('') }}"
```

So basically, revert to the old method. The error goes away completely and the playbook works as expected. My proposition here is to go back to the old method of using proxies via kubespray-defaults, i've had no issues once I modified the playbook back to what it was, for proxies.

[billykwooten]

https://github.com/kubernetes-sigs/kubespray/compare/master...billykwooten:master

This would be my proposal, the old method works just fine. The new method I get the error.

[codyseavey]

I ran into this today as well and it seems to have been caused by this task in roles/kubespray-defaults/tasks/no_proxy.yml:

- name: Populates no_proxy to all hosts
  set_fact:
    no_proxy: "{{ hostvars.localhost.no_proxy_prepare }}"
    proxy_env:
      no_proxy: "{{ hostvars.localhost.no_proxy_prepare }}"
      NO_PROXY: "{{ hostvars.localhost.no_proxy_prepare }}"

This block unsets http_proxy, https_proxy, HTTP_PROXY and HTTPS_PROXY due to ansibles DEFAULT_HASH_BEHAVIOR being to replace the proxy_env variable rather than to merge it. My proposal would be to use a combine for the task mentioned above rather than relying on the hash_behavior to be set.

[billykwooten]

I can confirm @codyseavey has a more elegant fix. I confirmed it does fix the issue by combining the variables vs overwriting the proxy_env variable. Closing my PR #6111 .

Please merge #6112 as it does fix this issue.

[asubb]

I had the same issue as mentioned but the fix didn't help me. The use case is a little bit different -- I'm trying to install only Helm, so running with tag -t helm.

One more thing needs to be added on top of the fix I believe so "proxy_env" always runs:

diff --git a/cluster.yml b/cluster.yml
index bf93e1bb..763beaa6 100644
--- a/cluster.yml
+++ b/cluster.yml
@@ -14,6 +14,8 @@
           HTTPS_PROXY: "{{ https_proxy | default ('') }}"
           no_proxy: "{{ no_proxy | default ('') }}"
           NO_PROXY: "{{ no_proxy | default ('') }}"
+      tags:
+        - always
       no_log: true

 - hosts: bastion[0]