Mysql: Cannot run mysql:5.7.28 with '-t' option

[m15o]

Cannot run mysql:5.7.28 with '-t' option:
docker run --rm -t -e MYSQL_ALLOW_EMPTY_PASSWORD=true mysql:5.7.28

2019-10-16 06:25:00+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.28-1debian9 started.
2019-10-16 06:25:00+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2019-10-16 06:25:00+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.28-1debian9 started.
2019-10-16 06:25:00+00:00 [Note] [Entrypoint]: Initializing database files
2019-10-16T06:25:00.882890Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-10-16T06:25:01.081984Z 0 [Warning] InnoDB: New log files created, LSN=45790
2019-10-16T06:25:01.122587Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2019-10-16T06:25:01.182468Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: aed9b346-efdd-11e9-bac1-0242ac110004.
2019-10-16T06:25:01.186354Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2019-10-16T06:25:01.922428Z 0 [Warning] CA certificate ca.pem is self signed.
2019-10-16T06:25:02.027645Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2019-10-16 06:25:04+00:00 [Note] [Entrypoint]: Database files initialized
2019-10-16 06:25:04+00:00 [Note] [Entrypoint]: Starting temporary server
Please enable --log-error option or set appropriate redirections for standard output and/or standard error in daemon mode.
2019-10-16 06:25:04+00:00 [ERROR] [Entrypoint]: Unable to start server.

mysql:5.7.27 has no problem with '-t' option:
docker run --rm -t -e MYSQL_ALLOW_EMPTY_PASSWORD=true mysql:5.7.27

[m15o]

Cannot run on circleci either:

2019-10-16 04:35:18+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.28-1debian9 started.
2019-10-16 04:35:18+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2019-10-16 04:35:18+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.28-1debian9 started.
2019-10-16 04:35:18+00:00 [Note] [Entrypoint]: Initializing database files
2019-10-16T04:35:18.322128Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-10-16T04:35:18.488295Z 0 [Warning] InnoDB: New log files created, LSN=45790
2019-10-16T04:35:18.517011Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2019-10-16T04:35:18.572621Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 5b4f2653-efce-11e9-9e26-0242c0a8e003.
2019-10-16T04:35:18.572869Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2019-10-16T04:35:19.184439Z 0 [Warning] CA certificate ca.pem is self signed.
2019-10-16T04:35:19.317383Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2019-10-16 04:35:21+00:00 [Note] [Entrypoint]: Database files initialized
2019-10-16 04:35:21+00:00 [Note] [Entrypoint]: Starting temporary server
Please enable --log-error option or set appropriate redirections for standard output and/or standard error in daemon mode.
2019-10-16 04:35:21+00:00 [ERROR] [Entrypoint]: Unable to start server.

Exited with code 1

[ruudk]

Thanks for posting this as I just discovered the same issue on our CircleCI. It picked up 5.7.28 overnight.

[ruudk]

I'm unable to reproduce this on macOS with the same MySQL version. I find this interesting:

Please enable --log-error option or set appropriate redirections for standard output and/or standard error in daemon mode.

Could it be that CircleCI runs the container in a different way?

[m15o]

I am also running docker on macOS.

It runs perfectly without '-t' or '--tty' option like this:
docker run --rm -e MYSQL_ALLOW_EMPTY_PASSWORD=true mysql:5.7.28

But it dont with '-t' option:
docker run --rm -t -e MYSQL_ALLOW_EMPTY_PASSWORD=true mysql:5.7.28

[tgandrews]

I get the same issue on Windows. It is not just Linux

[raptor235]

Same issue here... all our builds started failing this morning.

[DannyvdSluijs]

Same issue here. Sadly no additional information on the why. Setting the mysql image from 5.7 to 5.7.27 makes the builds work again.

[felicianotech]

I can confirm that this is affecting many CircleCI customers using the 5.7 tag.

For CircleCI users seeing this, temp workaround listed here.

[ltangvald]

The 5.7.28 image is the first built from a fairly big refactoring effort done, so might be related to that. Will look into this.

[tianon]

@ltangvald I've confirmed it's related to the new use of --daemonize -- the old script on 5.7 didn't use --daemonize :disappointed:

Do we need to set --log-error explicitly to /dev/stderr or something?

[ltangvald]

Yup, just found the same thing :) Yeah, I think that should work.

[ltangvald]

That should work for 8.0 too (just isn't required), so don't think we need to add any extra version checks.

[tianon]

Hopefully you've got a better handle on how to make that work because my testing is coming up empty. :sweat_smile:

Doing --log-error=/dev/stderr adds .err on the end, and explicitly creating a symlink with an existing extension to avoid that added suffix leads to things like Could not open file '/var/log/mysql/error.log' for error logging: Permission denied. :confused:

[ltangvald]

I'll give it a try. If not, we could fall back to the 5.6-way of doing it for 5.7 as well, just to get this fixed quickly, i.e. change the test to only 8.0 using daemonize for the temporary startup.

[ltangvald]

Yeah, I couldn't find a quick fix for it, so I made pr #601 to revert to the 5.6 way for now

[tianon]

Thanks @ltangvald :heart:!

This is fixed by #601 (will update on Hub via https://github.com/docker-library/official-images/pull/6797, per https://github.com/docker-library/faq#an-images-source-changed-in-git-now-what). :+1:

[ltangvald]

One additional note: daemonize can't handle writing to stdout|stderr when tty is enabled. This works in 8.0 because it explicitly handles this by writing to a file, i.e:

2019-10-17 06:51:20+00:00 [Note] [Entrypoint]: Starting temporary server
mysqld will log errors to /var/lib/mysql/40d0991fa3b6.err
mysqld is running as pid 97
2019-10-17 06:51:21+00:00 [Note] [Entrypoint]: Temporary server started.

Not a huge issue for the temporary startup, but if it fails for whatever reason it will make it harder to get the reason. Maybe do --log-error to some temporary file and then cat it afterward?

[tianon]

Hmm, not crazy about logging to a file which we then cat because it makes the failure cases a lot harder to debug (since often the failure cases will cause the cat to not execute properly). :disappointed:

[ltangvald]

Heh, good point. If the server startup fails it will (unless we start adding even more hacks) not get to the point where it prints the log. Maybe we should just drop the usage of daemonize for now, since we have to keep the wait-loop logic for 5.6 anyway.

[tianon]

Yeah, it's a shame but sounds like the most reliable way to keep our error logs. :disappointed:

Maybe we can file this as a feature request for --daemonize's future? :innocent:

[ltangvald]

I don't really understand the technicalities of it myself, but I've been told that disconnecting from stdout is the basis for the entire daemonize functionality, so might not be possible. May be other ways to get the same benefit in this case, though.