Kops: Add support for overlay2 Docker storage driver

[kaazoo]

The Docker documentation recommends OverlayFS 2 (overlay2) instead of OverlayFS (overlay):
https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/

It seems like Kops always configures Docker for overlay, even if you set overlay2, see https://github.com/kubernetes/kops/blob/8fb99a87e2eeac1f55ccaeb5c50ceb934f7cedd8/upup/pkg/fi/nodeup/command.go#L356

[kaazoo]

See also https://github.com/kubernetes/minikube/issues/1380

[aledbf]

@kaazoo overlay2 is not recommended in the current docker version (< 1.13) https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#external-dependency-version-information

[jgmize]

If docker 1.13 lands in k8s 1.7 then this will be blocked by https://github.com/kubernetes/kops/issues/2715 but based on https://github.com/kubernetes/kubernetes/issues/42926#issuecomment-311365261 this may have to wait for a later release.

[chrislovecnm]

K8s 1.7 is not running docker err Moby 1.13. Still on 1.12

[imsky]

@kaazoo i'm not sure https://github.com/kubernetes/kops/blob/8fb99a87e2eeac1f55ccaeb5c50ceb934f7cedd8/upup/pkg/fi/nodeup/command.go#L356 is actually resetting the storage, it just seems to pass the overlay2->overlay alias to kernelHasFilesystem.

fwiw here's the docker info from one of our nodes, we've been running overlay2 for months now:

# docker info
Containers: 29
 Running: 25
 Paused: 0
 Stopped: 4
Images: 144
Server Version: 1.12.3
Storage Driver: overlay2
...

[kaazoo]

@imsky Thanks for the feedback! I was able to set the following in the cluster config via 'kops edit cluster':

  docker:
    bridge: ""
    ipMasq: false
    ipTables: false
    logDriver: json-file
    logLevel: warn
    logOpt:
    - max-size=10m
    - max-file=5
    storage: overlay2
    version: 1.12.6

OverlayFS 2 is now used for Docker:

# docker info
Containers: 16
 Running: 16
 Paused: 0
 Stopped: 0
Images: 9
Server Version: 1.12.6
Storage Driver: overlay2
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.76-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.795 GiB
Name: ip-172-20-115-134
ID: 7T7D:MK6J:HNEQ:6AOY:CQNI:4QCG:NDYF:VMQE:XLLR:UFYY:IDIN:IZWE
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

By the way, I installed 'kernel-lt' package from the 'elrepo' Yum repository for CentOS 7, as you need kernel 4.x for OverlayFS 2.

[chrislovecnm]

Can we close this?

[geekofalltrades]

Note for Googlers: #4899 is now open for supporting overlay2 and possibly making it the default in Kops 1.10.