Mysql: MySQL fails to run on Github Actions using Kind

[tamalsaha]

I am reposting this from Kubernetes kind rpeo.
xref: https://github.com/kubernetes-sigs/kind/issues/1179

What happened:
We are trying to run an official MySQL pod inside a Kind cluster on GitHub actions. Here is our demo repo: https://github.com/tamalsaha/mysql-kind-demo

The problem is mysql pod fails to run with the error message:

+ mysqld --verbose --help
mysqld: Can't read dir of '/etc/mysql/conf.d/' (OS errno 13 - Permission denied)
mysqld: [ERROR] Fatal error in defaults handling. Program aborted!

You can see the full log here: https://github.com/tamalsaha/mysql-kind-demo/commit/dfef4e188accdb6334b75f4658bd0fc1dd35c732/checks?check_suite_id=364071606

The puzzling this is that if I just run docker run mysql:8.0.14 mysqld --verbose --help it works! You can see full log here:
https://github.com/tamalsaha/mysql-kind-demo/commit/d452e672fa2219fb5ea5174bd99108d6d0b2728a/checks?check_suite_id=364034120

When I try these on my Ubuntu 18.04 Desktop (using pod in a Kind cluster and Direct Docker), both modes work.

You can see the diff between pod log from my Desktop (left) and Github action (right). They mostly look identical: https://www.diffchecker.com/D64wMscf

Can you help my understand why mysqld --verbose --help fails when run as a pod inside a Kind v1.16.3 cluster on GitHub actions?

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Here is our demo repo: https://github.com/tamalsaha/mysql-kind-demo

If you want to run locally, try this:

kind create cluster
kubectl apply -f https://github.com/tamalsaha/mysql-kind-demo/raw/master/hack/kubernetes/mysql.yaml

Anything else we need to know?:

One hack we found is that if we mount an emptyDir to /etc/mysql/conf.d folder, mysqld --verbose --help command starts working.

Environment:

• kind version: (use kind version): v0.6.1
• Kubernetes version: (use kubectl version): v1.16.3
• Docker version: (use docker info):

Client:
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.3.1)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 13
 Server Version: 3.0.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.0.0-1027-azure
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 6.783GiB
 Name: fv-az86
 ID: KRSI:74QW:YCD2:XEK7:4WO2:DOAO:FRYG:MP4B:J2YH:N63L:5SGR:JV5A
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

• OS (e.g. from /etc/os-release):

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

[tamalsaha]

GitHub builds their VM from https://github.com/actions/virtual-environments/tree/master/images/linux using packer.

[wglambert]

Using your custom image tigerworks/mysql:8.0.18 I wasn't able to reproduce

$ docker run -d --rm -e MYSQL_ROOT_PASSWORD=root --name mysql tigerworks/mysql:8.0.18
845aa47f6a67d825aae760a5b799d1bb1f10e88d2e569f0a20f9a8b66c764a95

$ docker logs mysql 2>&1 | tail -n 2
2019-12-19T18:06:15.878473Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.18'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
2019-12-19T18:06:16.025879Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 33060

$ docker exec mysql mysqld --verbose --help | tail -n 10                             
updatable-views-with-limit                                   YES
upgrade                                                      AUTO
validate-config                                              FALSE
validate-user-plugins                                        TRUE
verbose                                                      TRUE
wait-timeout                                                 28800
windowing-use-high-precision                                 TRUE

To see what values a running MySQL server is using, type
'mysqladmin variables' instead of 'mysqld --verbose --help'.

$ docker exec -it mysql mysql -uroot -proot
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 12
Server version: 8.0.18 MySQL Community Server - GPL

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.00 sec)

So the issue you're having is probably related to the environment, you could try asking over at the Docker Community Forums, Docker Community Slack, or Stack Overflow.

[tamalsaha]

@wglambert , I don't think you tried with Kind. If you read my original post, docker run is also working for me on GitHub action. It does not work when I deploy as a Pod in a Kind cluster in GitHub actions.

[wglambert]

Testing your own image in kind isn't in our realm; these repos are for issues with the image itself, but I'll try it anyway out of curiosity.

I removed the privileged policy

$ ./kind-linux-amd64 create cluster                                                                                         
Creating cluster "kind" ...                                                                                                                       
 ✓ Ensuring node image (kindest/node:v1.16.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"                                                                                                                
You can now use your cluster with:                                                                                                                

kubectl cluster-info --context kind-kind

Have a nice day!

$ kubectl apply -f <(cat << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: stash-e2e
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 128Mi
  storageClassName: standard

---
apiVersion: v1
kind: Secret
metadata:
  name: stash-e2e
  namespace: default
type: Opaque
data:
  username: cm9vdA==
  password: c3Rhc2gtZTJlLWNwcWR0eg==

---

apiVersion: v1
kind: Pod
metadata:
  name: stash-e2e
  namespace: default
spec:
  containers:
  - env:
    - name: MYSQL_ROOT_PASSWORD
      valueFrom:
        secretKeyRef:
          key: password
          name: stash-e2e
    image: tigerworks/mysql:8.0.18
    imagePullPolicy: IfNotPresent
    name: mysql
    command: ["/bin/bash"]
    args:
    - -c
    - "set -x; echo 'options overlay metacopy=off redirect_dir=off' > /etc/modprobe.d/disable_overlay_redirect_dir.conf;  whoami; id; cat /etc/os-release; uname -a; cat /etc/passwd; cat /proc/mounts; mount; echo; echo; ls -lah /etc/; ls -lah /etc/mysql; ls -lah /etc/mysql/conf.d; touch /etc/mysql/conf.d/test.cnf; ls -lah /etc/mysql/conf.d; ls -lah /usr/sbin/mysqld; stat /etc/mysql/conf.d/; chmod a+rwx /etc/mysql/conf.d/; stat /etc/mysql/conf.d/; stat $(which mysqld); strace mysqld --verbose --help"
    ports:
    - containerPort: 3306
      name: mysql
      protocol: TCP
    volumeMounts:
    - mountPath: /var/lib/mysql
      name: stash-e2e
  restartPolicy: Never
  volumes:
  - name: stash-e2e
    persistentVolumeClaim:
      claimName: stash-e2e
EOF
)
persistentvolumeclaim/stash-e2e created
secret/stash-e2e created
pod/stash-e2e created

$ kubectl get po
NAME        READY   STATUS      RESTARTS   AGE
stash-e2e   0/1     Completed   0          42s

$ kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
stash-e2e   Bound    pvc-c3d30111-1f52-46f9-9f11-2cc5efda95b5   128Mi      RWO            standard       53s

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-c3d30111-1f52-46f9-9f11-2cc5efda95b5   128Mi      RWO            Delete           Bound    default/stash-e2e   standard                56s

$ kubectl logs stash-e2e 2>&1 | tail -n 10
updatable-views-with-limit                                   YES
upgrade                                                      AUTO
validate-config                                              FALSE
validate-user-plugins                                        TRUE
verbose                                                      TRUE
wait-timeout                                                 28800
windowing-use-high-precision                                 TRUE

To see what values a running MySQL server is using, type
'mysqladmin variables' instead of 'mysqld --verbose --help'.

[tamalsaha]

Thanks @wglambert ! I am not really using any custom image. I only added strace to see why it is failing. https://github.com/tamalsaha/mysql/commit/fcdbc4ae4887b91bdd33b15d2a34a7a61d7c1717

It seems you tried it on a local machine. My problem is this just fails on GitHub actions. Works fine on local machine for me, too.

[tamalsaha]

Here is my GitHub actions workflow: https://github.com/tamalsaha/mysql-kind-demo/blob/master/.github/workflows/kind-master.yml

You can see the result: https://github.com/tamalsaha/mysql-kind-demo/commit/476ba9f4388712de9be94a4353372498b28e2d2e/checks?check_suite_id=366005008#step:5:914

This is why this issue is so confusing. :(

[yosifkit]

Possibly the same as #330 (specifically the last comment) and #133?

[tamalsaha]

Thank you very much @yosifkit ! I see that there is an apparmor profile for mysql in enforced mode.

+ sudo aa-status
apparmor module is loaded.
17 profiles are loaded.
17 profiles are in enforce mode.
   /sbin/dhclient
   /usr/bin/lxc-start
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/sbin/mysqld
   /usr/sbin/tcpdump
   docker-default
   lxc-container-default
   lxc-container-default-cgns
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting
   man_filter
   man_groff
0 profiles are in complain mode.
1 processes have profiles defined.
1 processes are in enforce mode.
   /usr/sbin/mysqld (1622) 
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

+ sudo cat /etc/apparmor.d/usr.sbin.mysqld
# vim:syntax=apparmor
# Last Modified: Tue Feb 09 15:28:30 2016
#include <tunables/global>

/usr/sbin/mysqld {
  #include <abstractions/base>
  #include <abstractions/nameservice>
  #include <abstractions/user-tmp>
  #include <abstractions/mysql>
  #include <abstractions/winbind>

# Allow system resource access
  /proc/*/status r,
  /sys/devices/system/cpu/ r,
  /sys/devices/system/node/ r,
  /sys/devices/system/node/** r,
  capability sys_resource,
  capability dac_override,
  capability dac_read_search,
  capability setuid,
  capability setgid,

# Allow network access
  network tcp,

  /etc/hosts.allow r,
  /etc/hosts.deny r,

# Allow config access
  /etc/mysql/** r,

# Allow pid, socket, socket lock file access
  /var/run/mysqld/mysqld.pid rw,
  /var/run/mysqld/mysqld.sock rw,
  /var/run/mysqld/mysqld.sock.lock rw,
  /run/mysqld/mysqld.pid rw,
  /run/mysqld/mysqld.sock rw,
  /run/mysqld/mysqld.sock.lock rw,

# Allow systemd notify messages
  /{,var/}run/systemd/notify w,

# Allow execution of server binary
  /usr/sbin/mysqld mr,
  /usr/sbin/mysqld-debug mr,

# Allow plugin access
  /usr/lib/mysql/plugin/ r,
  /usr/lib/mysql/plugin/*.so* mr,

# Allow error msg and charset access
  /usr/share/mysql/ r,
  /usr/share/mysql/** r,

# Allow data dir access
  /var/lib/mysql/ r,
  /var/lib/mysql/** rwk,

# Allow data files dir access
  /var/lib/mysql-files/ r,
  /var/lib/mysql-files/** rwk,

# Allow keyring dir access
  /var/lib/mysql-keyring/ r,
  /var/lib/mysql-keyring/** rwk,

# Allow log file access
  /var/log/mysql.err rw,
  /var/log/mysql.log rw,
  /var/log/mysql/ r,
  /var/log/mysql/** rw,

# Allow read access to OpenSSL config
  /etc/ssl/openssl.cnf r,
  # Site-specific additions and overrides. See local/README for details.
  #include <local/usr.sbin.mysqld>
}

This profile seem to give READ access to /etc/mysql/**. I am new to apparmor. So, I may be missing something here. Can you please help me understand why mysqld fails to read the /etc/mysql/conf.d/ directory in Dind container when apparmor gives it access?

# Allow config access
  /etc/mysql/** r,

[yosifkit]

My guess is that the host apparmor doesn't work correctly for the mount namespace of the container and so denies access from mysqld (but somehow works for earlier files?).

I think the easiest workaround is to just disable apparmor before starting kind and see if that fixes it.

[BenTheElder]

:wave: Thanks @yosifkit -- https://github.com/docker-library/mysql/issues/617#issuecomment-567721583 has some very handy pointers.
kind is something of a hack, in general apparmor or selinux causing issues is not surprising :grimacing:
Definitely think disabling apparmor (at least for mysql on this CI host...) is the way to go here :+1:

[BenTheElder]

@tamalsaha if you can confirm this we'll get it documented under https://kind.sigs.k8s.io/docs/user/known-issues/ for the next person :upside_down_face:

[tamalsaha]

@BenTheElder , I think tried to disable apparmor but it still failing.

doc: https://help.ubuntu.com/community/AppArmor#Disable_AppArmor_framework

https://github.com/tamalsaha/mysql-kind-demo/commit/3fb9a5683536e88823b13ba9b5058b2f570d6aa3

https://github.com/tamalsaha/mysql-kind-demo/commit/3fb9a5683536e88823b13ba9b5058b2f570d6aa3/checks?check_suite_id=369417185

Am I missing anything here?

[wglambert]

I'm going to close since https://github.com/kubernetes-sigs/kind/issues/1179#issuecomment-571381576 was closed with an addition to the docs https://kind.sigs.k8s.io/docs/user/known-issues/#apparmor

And given that this is an environmental issue with Github Actions there's nothing we can foreseeably change in the image to alleviate this.
https://github.com/actions/runner looks like the place to file this, if not then one of these repos https://github.com/actions

[BenTheElder]

sorry for the noise and thanks for help investigating!