Virtual-environments: MySQL 8 Installation Failure

[browner12]

Describe the bug

I install MySQL 8 as a step of my Action to run my tests. The install script is more or less replicated from Laravel Homestead, with some minor tweaks. On Jan 12 the installation succeeded, and on Jan 13 the installation failed, with no changes to the script.

End of Success Log:

done!
update-alternatives: using /var/lib/mecab/dic/ipadic-utf8 to provide /var/lib/mecab/dic/debian (mecab-dictionary) in auto mode
Setting up mysql-community-server (8.0.18-1ubuntu18.04) ...
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Created symlink /etc/systemd/system/multi-user.target.wants/mysql.service → /lib/systemd/system/mysql.service.
Setting up mysql-server (8.0.18-1ubuntu18.04) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...

End of Failure Log:

done!
update-alternatives: using /var/lib/mecab/dic/ipadic-utf8 to provide /var/lib/mecab/dic/debian (mecab-dictionary) in auto mode
Setting up mysql-community-server (8.0.19-1ubuntu18.04) ...
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Created symlink /etc/systemd/system/multi-user.target.wants/mysql.service → /lib/systemd/system/mysql.service.
Unsafe symlinks encountered in /var/run/mysqld, refusing.
Unsafe symlinks encountered in /var/run/mysqld, refusing.
Unsafe symlinks encountered in /var/run/mysqld, refusing.
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
Setting up mysql-server (8.0.19-1ubuntu18.04) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.

As you can see the installation is failing to make symlinks in /var/run/mysqld, which leads me to believe there may be some permissions issue.

While trying to debug I came across the following resource which suggested a change of ownership of the root directory:

https://askubuntu.com/questions/1094776/systemd-tmpfiles-wont-start

This led me to throw some ls -al s into my install script. The one thing that stands out is the / directory is owned by 501:staff. Unfortunately I cannot compare that with the ownership prior to the breakage of the install.

drwxr-xr-x   26  501 staff       4096 Feb 18 16:24 .
drwxr-xr-x   26  501 staff       4096 Feb 18 16:24 ..
-rw-r--r--    1 root root    29844013 Feb 13 13:13 agpm-osx-x64-1.151.0.tar.gz
drwxr-xr-x    2  501 staff      20480 Apr 26  2019 bin
drwxr-xr-x    4 root root        4096 Feb 12 09:08 boot
drwxr-xr-x    4 root root        4096 Feb 18 16:24 data
drwxr-xr-x   15 root root        3800 Feb 18 16:24 dev
drwxr-xr-x  128 root root       12288 Feb 13 13:13 etc
drwxr-xr-x+   4 root root        4096 Feb 13 13:11 home
drwxr-xr-x    2 root root        4096 Feb 12 11:56 imagegeneration
drwxr-xr-x   20 root root        4096 Feb 12 09:14 lib
drwxr-xr-x    2 root root        4096 Feb 12 09:14 lib32
drwxr-xr-x    2 root root        4096 Feb  8 15:49 lib64
drwx------    2 root root       16384 Feb  8 15:54 lost+found
drwxr-xr-x    2 root root        4096 Feb  8 15:49 media
drwxr-xr-x    3 root root        4096 Feb 18 16:24 mnt
drwxrwxrwx+  13 root root        4096 Feb 13 13:13 opt
dr-xr-xr-x  146 root root           0 Feb 18 16:24 proc
drwx------    5 root root        4096 Feb 12 11:40 root
drwxr-xr-x   22 root root         840 Feb 18 16:24 run
drwxr-xr-x    2 root root        4096 Feb  8 16:06 sbin
drwxr-xr-x    2 root root        4096 Feb 12 09:07 snap
drwxr-xr-x    2 root root        4096 Feb  8 15:49 srv

This same install script will work on a local version of Ubuntu 18.04.

I'm at a little bit of a loss whether this is a mistake on my end, or whether it is a bug and something changed on Actions that caused this issue. Any help/insight you could provide would be greatly appreciated.

Area for Triage:

MySQL

Question, Bug, or Feature?:

Possible Bug

Virtual environments affected

• [ ] macOS 10.15
• [ ] Ubuntu 16.04 LTS
• [x] Ubuntu 18.04 LTS
• [ ] Windows Server 2016 R2
• [ ] Windows Server 2019

Expected behavior

I expect MySQL 8 to install successfully.

Actual behavior

Description above.

[browner12]

I created a dummy public repo with an identical MySQL8 install script, and it is succeeding.

The interesting thing to note is now the / directory is owned by root:root instead of 501:staff, which leads me to believe this is the cause of the issue.

https://github.com/browner12/mysql8

[browner12]

The plot thickens. The first dummy repo I created was in my personal account, and that ran fine.

I just created another dummy repo in my organization account (the same as where the original error occurred) and it failed again. It also had the / directory owned by 501:staff.

Relevant settings between my organization and personal account seem to be the same.

As a temporary work around, I am starting the script by running chown root:root /, which fixes the permissions and allows the MySQL 8 installation to complete successfully.

[miketimofeev]

@browner12 I've noticed MySQL version has changed between successful and failed builds
8.0.18-1ubuntu18.04 -> 8.0.19-1ubuntu18.04
Could you try to install the same 8.0.18, please? I wonder if there are some changes in MySQL itself.

[browner12]

The MySQL version difference is not relevant to the failure. With the DEB package you only select the major version, and whatever the most recent version in the major release is downloaded and installed. I can get failures and successes with version 8.0.19 depending on whether I chown the / directory or not.

MySQL overall is irrelevant in this issue. I've just duplicated the failure by installing Blackfire and running into the same problem.

The ownership of / by 501:staff seems to be the culprit, so we need to figure out why that is occurring even though both workflows are configured the same.

[miketimofeev]

@browner12 Got it. I was able to repro. On one of the projects, it shows 501:staff and on another root:root on the same ubuntu:latest.
Will try to figure out what goes wrong.

[svpernova09]

@miketimofeev are the runs which show 501:staff running on MacOS hosts? As opposed to whatever native host is powering the ubuntu containers? I've run across this 501:staff issue before with virtual machines when the image wasn't cleaned up properly before being tagged and shared.

I would assume GH would only run on MacOS hosts if you've specified to run there since they're billed at a higher rate.

[miketimofeev]

@browner12 @svpernova09 we managed to find the root cause in our VM initialization scripts and we are working on fixing them now.

[miketimofeev]

@browner12 the fix has been deployed. Could you check it, please?

[browner12]

It is still failing unfortunately.

https://github.com/kennedycomm/mysql8/runs/468257867?check_suite_focus=true

How about I check tomorrow morning in case things are taking a little bit to propagate out?

[miketimofeev]

@browner12 thanks for sharing the link and sorry for the inconvenience. We will investigate whether the environment, where the run is being executed, has already updated or not.

[browner12]

The run this morning passed and the permissions look good.

https://github.com/kennedycomm/mysql8/runs/470030693?check_suite_focus=true

Maybe it just took a bit to propagate out?

[miketimofeev]

@browner12 yes, the fix doesn't propagate in a tick but I supposed it was already deployed. I'm going to close the issue if everything works for you now. Feel free to reopen if you see it again.
Thank you!

[browner12]

Thanks for getting this fixed.

Were the fixes in a public repo we can see? Or something private to Github?

Was the problem similar to what @svpernova09 was describing?

[miketimofeev]

@browner12 yeah, it's under-the-hood internal provisioning scripts to get VMs ready for action(private repo), so it's a bit different from @svpernova09 assumption.