Charts: OpenVPN connection is not established properly

[kc004]

Is this a request for help?: Yes

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): No

Version of Helm and Kubernetes: Kubernetes 1.10.1
Helm 2.9.1

What happened: I am not able to connect kubernetes services using OpenVPN kubernetes chart.

What you expected to happen: On my local pc i can access kubernetes services locally using VPN

How to reproduce it (as minimally and precisely as possible): Deploy openVPN using helm and generate key to connect to the cluster.

Anything else we need to know:
The issue is i am not able to call kubernetes services locally using VPN and also my internet is stopped working after connecting to VPN.

Here are some routes logs:

With VPN connection
/ # nslookup google.com
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 10.43.0.10 {{{ kube-dns IP address}}

DNS request timed out.
timeout was 2 seconds.

Without VPN connection
/ # nslookup google.com
Server: UnKnown
Address: 192.168.0.1

Non-authoritative answer:
Name: google.com
Addresses: 2404:6800:4009:809::200e
216.58.203.206

In Kubernetes OpenVPN pod console
/ # nslookup google.com
nslookup: can't resolve '(null)': Name does not resolve
Name: google.com
Address 1: 173.194.66.101 qo-in-f101.1e100.net
Address 2: 173.194.66.100 qo-in-f100.1e100.net
Address 3: 173.194.66.113 qo-in-f113.1e100.net
Address 4: 173.194.66.102 qo-in-f102.1e100.net
Address 5: 173.194.66.139 qo-in-f139.1e100.net
Address 6: 173.194.66.138 qo-in-f138.1e100.net
Address 7: 2607:f8b0:400d:c01::8b qo-in-x8b.1e100.net

/ # nslookup kubernetes.default.svc.cluster.local
nslookup: can't resolve '(null)': Name does not resolve
Name: kubernetes.default.svc.cluster.local
Address 1: 10.43.0.1 kubernetes.default.svc.cluster.local

It can resolve my service DNS and give my IP address of that service but not able to find nameserver as kube-dns.kube-system.svc.cluster.local

Route of OpenVPN pod console
/ # iproute
default via 169.254.1.1 dev eth0
10.240.0.0/16 via 10.240.0.2 dev tun0
10.240.0.2 dev tun0 src 10.240.0.1
169.254.1.1 dev eth0

[jmymy]

What version of this chart are you using? Latest? Are you using any custom options in values.yml ?

[kc004]

@jmymy I am using official kubernetes chart. https://github.com/kubernetes/charts/tree/master/stable/openvpn

[jannishuebl]

I have the same problem. @kc004 could you fix it in the meantime?

@jmymy can I provide you any log outputs?

I am also running Kubernets 1.10.1, using RKE for deployment.

From my local maschine I can ping the Openvpn pod, the 10.240.0.1 and the 10.42.. address of the openvpn pod. But services or pods beside are not working.

ping 10.42.2.9
PING 10.42.2.9 (10.42.2.9): 56 data bytes
64 bytes from 10.42.2.9: icmp_seq=0 ttl=64 time=23.704 ms
64 bytes from 10.42.2.9: icmp_seq=1 ttl=64 time=23.684 ms

ping 10.42.1.16
PING 10.42.1.16 (10.42.1.16): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

[kc004]

@jannishuebl still problem exist.

[jmymy]

Can you provide your openvpn server and client config please?

[gregkeys]

Im having the same problem, Im trying to provide access outside of the kubernetes network.
We have an aws VPC VPN connection connected for secure access to a remote API we need to be able to develop against that ip which is propagated into our VPC, we also need to be able to access the internet while on the VPN which is currently not working either

openvpn.conf

server 10.240.0.0 255.255.0.0
verb 3
key /etc/openvpn/certs/pki/private/server.key
ca /etc/openvpn/certs/pki/ca.crt
cert /etc/openvpn/certs/pki/issued/server.crt
dh /etc/openvpn/certs/pki/dh.pem

key-direction 0
keepalive 10 60
persist-key
persist-tun

proto tcp
port  443
dev tun0
status /tmp/openvpn-status.log

user nobody
group nogroup
duplicate-cn

push "route NETWORK 255.255.240.0"
push "route "172.31.0.0" 255.0.0.0"
push "route "192.168.0.0" 255.0.0.0"
OVPN_K8S_SEARCH
push "dhcp-option DNS OVPN_K8S_DNS"

configure.sh

#!/bin/sh
/etc/openvpn/setup/setup-certs.sh
iptables -t nat -A POSTROUTING -s 10.240.0.0/255.255.0.0 -o eth0 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 172.31.0.0/255.255.0.0 -o eth0 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 192.168.0.0/255.255.0.0 -o eth0 -j MASQUERADE
mkdir -p /dev/net
if [ ! -c /dev/net/tun ]; then
    mknod /dev/net/tun c 10 200
fi

if [ "$DEBUG" == "1" ]; then
    echo ========== ${OVPN_CONFIG} ==========
    cat "${OVPN_CONFIG}"
    echo ====================================
fi
IP=$(ip route get 8.8.8.8 | awk '/8.8.8.8/ {print $NF}')
BASEIP=`echo $IP | cut -d"." -f1-3`
NETWORK=`echo $BASEIP".0"`
DNS=$(cat /etc/resolv.conf | grep -v '^#' | grep nameserver | awk '{print $2}')
SEARCH=$(cat /etc/resolv.conf | grep -v '^#' | grep search | awk '{$1=""; print $0}')
FORMATTED_SEARCH=""
for DOMAIN in $SEARCH; do
  FORMATTED_SEARCH="${FORMATTED_SEARCH}push \"dhcp-option DOMAIN-SEARCH ${DOMAIN}\"\n"
done
cp -f /etc/openvpn/setup/openvpn.conf /etc/openvpn/
sed 's|OVPN_K8S_SEARCH|'"${FORMATTED_SEARCH}"'|' -i /etc/openvpn/openvpn.conf
sed 's|OVPN_K8S_DNS|'"${DNS}"'|' -i /etc/openvpn/openvpn.conf
sed 's|NETWORK|'"${NETWORK}"'|' -i /etc/openvpn/openvpn.conf

openvpn --config /etc/openvpn/openvpn.conf

[jmymy]

Did you use custom values in the helm chart? i.e. values.yaml?

[gregkeys]

@jmymy Yes and I also manually updated the configmap and restarted the container, thats how I added the extra routes, I also manually edited the configure.sh entry in the configmap

[kc004]

@jmymy i also did same as @gregkeys above comment but no luck.

[gregkeys]

i have a sneaky suspicion it might need some custom iptable rules and ip forwarding e.g. https://www.linode.com/docs/networking/vpn/tunnel-your-internet-traffic-through-an-openvpn-server/ i toyed with it a little with no luck though

[neyz]

I have the same problem using it.

Running on core OS with canal CNI, no network policies activated.

I just tweaked the OVPN_K8S_POD_NETWORK and OVPN_K8S_POD_SUBNET=255.255.0.0 to match with my installation.

From the pod I can access internet / services but when connected with the openvpn client I can't reach anything at all.

any help appreciated

[lnikell]

same issue, did someone solve this issue?

[neyz]

I was hoping NET_ADMIN would be enough to handle this but it seems not.

I managed a workaround by enabling ip_forward=1 in sysctl.

I don't know if it's the cleanest method, if it's related to CoreOS or Kubernetes in general but it's a real pain to configure because you need to whitelist the sysctl's on each node..

Again, there probably is a better way to do things but at least I got it to work. On GKE it seems ip_forward is already set to 1 so it's not a problem.

hope it helps !

[sonoroot]

@neyz - Where did you set the IP forwarding, within the POD itself? - Thanks
I see, you mention node - so I guess I have my answer...

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[peterbosalliandercom]

I had the same problem and fixed it with a temporary workaround (privilleged container, setting sysctl ip_forward = 1 within the container. related to this issue https://github.com/kubernetes/kops/issues/6068. Maybe this could help with a more structural solution: https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/#setting-sysctls-for-a-pod

[jonstelly]

I'm about to submit a pull request to add this issue (and a few other changes) to the openvpn chart readme.

One fix that would probably work would be to use a privileged security policy but I have some concern about adding that to this chart. I don't have a good handle on the security ramifications for openvpn which is probably used as a bastion/jump server.

Workarounds seem to be different depending on how you deployed your kubernetes instance. A quick-ish hack/workaround that works on my RKE environment (and hopefully others):

The following assumes you have ssh access to the nodes. After deploying the openvpn chart, run the following:

Find which nodes openvpn was deployed to:

kubectl get pod -o wide

For each of those nodes run:
WARNING: this is calling sudo, carefully review before you run it

ssh -t <NODE> 'sudo nsenter -t $(docker inspect --format {{.State.Pid}} $(docker ps -f ''name=k8s_openvpn'' -q))' -n sysctl -w net.ipv4.ip_forward=1

[Tarjei400]

I was hoping NET_ADMIN would be enough to handle this but it seems not.

I managed a workaround by enabling ip_forward=1 in sysctl.

I don't know if it's the cleanest method, if it's related to CoreOS or Kubernetes in general but it's a real pain to configure because you need to whitelist the sysctl's on each node..

Again, there probably is a better way to do things but at least I got it to work. On GKE it seems ip_forward is already set to 1 so it's not a problem.

hope it helps !

Did anything change in that matter? For me ip_forward seems to do a trick aswell, however it looks like it requires aswell for pod to have privilaged securityContext :/

[gboor]

I just ran into this issue as well. I can bash into the pod and connect to the whole network, all the routes are in place, but I cannot contact any of the services from the client (it does connect to openvpn fine).

I am running Kubernetes on Google Cloud. I have 3 clusters and the other 2 work fine, but this one did not. It is the newest, so perhaps it's some new security settting?

I set the pod to privileged and changed net.ipv4.ip_forward to 1 and then it started working. So that workaround does the trick. The question, however, is now how I can set this in the deployment yaml. I tried the following on the container spec;

        securityContext:
          privileged: true
          sysctls:
            - name: net.ipv4.ip_forward
              value: 1

but when purging and reinstalling with my custom chart, net.ipv4.ip_forward remains 0 until I set it manually.

Has anyone managed to toggle this from the deployment directly?

[theAkito]

I'm not really sure if my problem is connected with this. I hope someone can give me a lead on my problem. Any help is appreciated!

I am getting

Wed Jan 15 15:42:27 2020 TCP connection established with [AF_INET]127.0.0.1:44171
Wed Jan 15 15:42:27 2020 127.0.0.1:44171 Connection reset, restarting [0]
Wed Jan 15 15:42:27 2020 127.0.0.1:44171 SIGUSR1[soft,connection-reset] received, client-instance restarting
Wed Jan 15 15:42:32 2020 TCP connection established with [AF_INET]127.0.0.1:36487
Wed Jan 15 15:42:32 2020 127.0.0.1:36487 Connection reset, restarting [0]
Wed Jan 15 15:42:32 2020 127.0.0.1:36487 SIGUSR1[soft,connection-reset] received, client-instance restarting
Wed Jan 15 15:42:37 2020 TCP connection established with [AF_INET]127.0.0.1:33075
Wed Jan 15 15:42:37 2020 127.0.0.1:33075 Connection reset, restarting [0]
Wed Jan 15 15:42:37 2020 127.0.0.1:33075 SIGUSR1[soft,connection-reset] received, client-instance restarting
Wed Jan 15 15:42:42 2020 TCP connection established with [AF_INET]127.0.0.1:36003
Wed Jan 15 15:42:42 2020 127.0.0.1:36003 Connection reset, restarting [0]
Wed Jan 15 15:42:42 2020 127.0.0.1:36003 SIGUSR1[soft,connection-reset] received, client-instance restarting

after setting up the OpenVPN pod. Tried many different configuration options, but none work. The only thing that I see is that the time may be wrong, according to the location's proper timezone. Could this be a problem? If yes, how can I set up the pod to be created with custom commands run on startup, when it is created by this chart?

[jonstelly]

I'm pretty sure those log entries are just the readiness probe. Notice how the times are all 5 second apart (the default interval for the readiness probe). I'm pretty sure if you set readinessProbe.initialDelaySeconds to 10 seconds, and readinessProbe.periodSeconds to 15 seconds, you'd see these messages every 10 seconds on pod startup until it got 2 successes, then you'd see the events every 15 seconds.

Are you saying you're also unable to connect? You may want to open a new issue and check out here on some troubleshooting / log collection help.

[theAkito]

Greetings @jonstelly,
thank you for your help!
I tried out connecting with a cert and it indeed worked, after all, thank you!

Problem now is that the canal workload in the kube-system namespace somehow has trouble setting up on the node that connected through OpenVPN. The ip address of the node that is recognized is the one that is given by OpenVPN.
Now I am not sure if this has something to do with wrong or missing OpenVPN chart configuration options or if this is rather related to calico/flannel...