Nextcloud-snap: Install of 15.0.4snap1 fails (mysql setup problem)

[mikesi]

Running as root:
nextcloud.manual-install adminuser somepassword
the command hangs on:
Waiting for PHP...

Checking the system logs:
Feb 23 19:10:57 nextcloud nextcloud.php-fpm[12802]: Waiting for MySQL...
Feb 23 19:10:57 nextcloud nextcloud.mysql[12779]: Starting MySQL
Feb 23 19:10:57 nextcloud nextcloud.apache[12849]: Waiting for PHP...
Feb 23 19:10:59 nextcloud nextcloud.renew-certs[12794]: Saving debug log to /var/snap/nextcloud/current/certs/certbot/logs/letsencrypt.log
Feb 23 19:10:59 nextcloud nextcloud.renew-certs[12794]: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Feb 23 19:10:59 nextcloud nextcloud.renew-certs[12794]: No renewals were attempted.
Feb 23 19:10:59 nextcloud nextcloud.renew-certs[12794]: No hooks were run.
Feb 23 19:10:59 nextcloud nextcloud.renew-certs[12794]: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Feb 23 19:11:02 nextcloud mysqld_safe[14025]: Starting mysqld daemon with databases from /var/snap/nextcloud/11688/mysql
Feb 23 19:11:02 nextcloud mysqld[14027]: /snap/nextcloud/11688/bin/mysqld (mysqld 5.7.22-log) starting as process 14027 ...
Feb 23 19:11:02 nextcloud mysqld[14027]: failed to set datadir to /var/snap/nextcloud/11688/mysql/
Feb 23 19:11:02 nextcloud mysqld[14027]: Aborting
Feb 23 19:11:02 nextcloud mysqld[14027]: Binlog end
Feb 23 19:11:02 nextcloud mysqld[14027]: /snap/nextcloud/11688/bin/mysqld: Shutdown complete
Feb 23 19:11:02 nextcloud mysqld_safe[14033]: mysqld from pid file /tmp/pids/mysql.pid ended
Feb 23 19:11:03 nextcloud nextcloud.mysql[12779]: ...... * The server quit without updating PID file (/tmp/pids/mysql.pid).
Feb 23 19:11:03 nextcloud nextcloud.mysql[12779]: sed: can't read /var/snap/nextcloud/11688/mysql/root.ini: No such file or directory
Feb 23 19:11:03 nextcloud nextcloud.mysql[12779]: Waiting for MySQL...

Seems that mysql setup fails (can't set owner of data directory) and the install process just hangs and waits.

OS:
Ubuntu 18.04.2 LTS

Snap info:
installed: 15.0.4snap1 (11688) 196MB -

[BernieO]

Even as root you need to run this command with sudo.
Can you try again with a prefixed sudo?
Ref:
https://github.com/nextcloud/server/issues/14359#issuecomment-466746435

[mikesi]

Hi @BernieO
Running with sudo produces the same result.
One note it's in a LXD container.

Feb 24 09:22:01 nextcloud nextcloud.nextcloud-fixer[79039]: Waiting for Apache...
Feb 24 09:22:03 nextcloud nextcloud.apache[78985]: Making sure nextcloud is setup...
Feb 24 09:22:03 nextcloud nextcloud.mysql[78999]: 2019-02-24T09:22:03.702197Z 0 [Warning] The syntax '--log_warnings/-W' is deprecated and will be removed in a future release. Please use '--log_error_verbosity' instead.
Feb 24 09:22:03 nextcloud nextcloud.mysql[78999]: 2019-02-24T09:22:03.708108Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
Feb 24 09:22:03 nextcloud nextcloud.mysql[78999]: 2019-02-24T09:22:03.717772Z 0 [Note] Ignoring --secure-file-priv value as server is running with --initialize(-insecure) or --bootstrap.
Feb 24 09:22:03 nextcloud nextcloud.mysql[78999]: 2019-02-24T09:22:03.727624Z 0 [Note] mysqld (mysqld 5.7.22-log) starting as process 79216 ...
Feb 24 09:22:03 nextcloud nextcloud.mysql[78999]: 2019-02-24T09:22:03.844621Z 0 [Note] Creating the data directory /var/snap/nextcloud/11688/mysql/
Feb 24 09:22:03 nextcloud nextcloud.mysql[78999]: 2019-02-24T09:22:03.994866Z 0 [ERROR] Can't change data directory owner to mysql
Feb 24 09:22:03 nextcloud nextcloud.mysql[78999]: 2019-02-24T09:22:03.995150Z 0 [ERROR] Aborting
Feb 24 09:22:04 nextcloud nextcloud.mysql[78999]: 2019-02-24T09:22:04.011140Z 0 [Note] Binlog end
Feb 24 09:22:04 nextcloud nextcloud.mysql[78999]: 2019-02-24T09:22:04.037854Z 0 [Note] mysqld: Shutdown complete
Feb 24 09:22:05 nextcloud nextcloud.php-fpm[79016]: Waiting for MySQL...
Feb 24 09:22:05 nextcloud nextcloud.mysql[78999]: Starting MySQL
Feb 24 09:22:06 nextcloud nextcloud.apache[78985]: Waiting for PHP...
Feb 24 09:22:07 nextcloud nextcloud.renew-certs[79023]: Saving debug log to /var/snap/nextcloud/current/certs/certbot/logs/letsencrypt.log
Feb 24 09:22:07 nextcloud nextcloud.renew-certs[79023]: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Feb 24 09:22:07 nextcloud nextcloud.renew-certs[79023]: No renewals were attempted.
Feb 24 09:22:07 nextcloud nextcloud.renew-certs[79023]: No hooks were run.
Feb 24 09:22:07 nextcloud nextcloud.renew-certs[79023]: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Feb 24 09:22:11 nextcloud mysqld_safe[80194]: Starting mysqld daemon with databases from /var/snap/nextcloud/11688/mysql
Feb 24 09:22:11 nextcloud mysqld[80196]: /snap/nextcloud/11688/bin/mysqld (mysqld 5.7.22-log) starting as process 80196 ...
Feb 24 09:22:11 nextcloud mysqld[80196]: failed to set datadir to /var/snap/nextcloud/11688/mysql/
Feb 24 09:22:11 nextcloud mysqld[80196]: Aborting
Feb 24 09:22:11 nextcloud mysqld[80196]: Binlog end
Feb 24 09:22:11 nextcloud mysqld[80196]: /snap/nextcloud/11688/bin/mysqld: Shutdown complete
Feb 24 09:22:11 nextcloud mysqld_safe[80204]: mysqld from pid file /tmp/pids/mysql.pid ended
Feb 24 09:22:12 nextcloud nextcloud.mysql[78999]: ...... * The server quit without updating PID file (/tmp/pids/mysql.pid).
Feb 24 09:22:12 nextcloud nextcloud.mysql[78999]: sed: can't read /var/snap/nextcloud/11688/mysql/root.ini: No such file or directory
Feb 24 09:22:12 nextcloud nextcloud.mysql[78999]: Waiting for MySQL...

[m-nogues]

I am having the same problem on debian 9.

The installation just hangs on the "Waiting for PHP..." message.

[cestpasphoto]

Exactly same problem here with Nextcloud 15.0.5snap1 on Debian.
If you check issues #819 #745 and #733, it seems that nextcloud uses mysql from system instead of the container version. Deleting mysql from system solves the issue, but it is not possible in my case.

[kyrofa]

Yeah there seems to be a snapd bug causing the snap to run mysql from the host instead of out of the snap. I've logged that bug here.

[cestpasphoto]

Thanks @kyrofa : do you need additional details like snapd version?

[zyga]

Hello. I'll have a look. Is it fair to say that Debian 9 is sufficient to reproduce the issue?

[cestpasphoto]

I personally am on Debian Sid

[patman-cp]

Any known workarounds for this?

I hit this running Fedora 29.

It looks like this has been a problem since Jan 2018.

I don't see any fixes in snapd, per https://bugs.launchpad.net/snapd/+bug/1819734

[mhdion]

$lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.11 (stretch)
Release: 9.11
Codename: stretch

I can confirm that, after removing and purging every myslq and mariadb packages, I was able to install and configure nextcloud using snap.

[cestpasphoto]

Did you remove _mysql_ and _mariadb_ packages on the host, or on the snap? The goal is that snap packages should work, whatever the packages are installed on host side.

[mhdion]

I removed, and purged, old mysql packages and latest mariadb packages.

After that, I re-installed the nextcloud snap and it worked as expected.

By the way, I also removed apache2 in order to free the 80 and 443 TCP
ports. After getting nextcloud up and running, I installed apache2 again
and changed it ports.

Now, It's working fine.

I have not make the test of installing again mariadb and to check if it
interferes with nextcloud's snap

El 7/10/2019 a las 02:45, cestpasphoto escribió:
>

Did you remove /mysql/ and /mariadb/ packages on the host, or on the
snap? The goal is that snap packages should work, whatever the
packages are installed on host side.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/nextcloud-snap/issues/913?email_source=notifications&email_token=AAN7GLA36P5KZEJLNBDP6UTQNLEGTA5CNFSM4GZZUOT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAPCXBI#issuecomment-538848133,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAN7GLCCXOOLD5DWXSLNIC3QNLEGTANCNFSM4GZZUOTQ.

[xkl]

folks, a dirty workaround that works for me (running bullseye):

$ mv /etc/mysql /etc/mysql.tmp
$ snap restart nextcloud
$ mv /etc/mysql.tmp /etc/mysql

-xkl

[kyrofa]

folks, a dirty workaround that works for me (running bullseye):

Note that you will need to do that every time the snap updates or the machine restarts.

[xkl]

folks, a dirty workaround that works for me (running bullseye):

Note that you will need to do that every time the snap updates or the machine restarts.

just dont ask, how I handle this, it's not getting cleaner....

[kyrofa]

just dont ask, how I handle this, it's not getting cleaner....

Hahaha :see_no_evil:

[kyrofa]

@zyga any progress on this?

[zyga]

@kyrofa I noticed this but I don't know what the problem is really.

• nextcloud is not using classic confinement, that's good
• mysql inside the snap and on the host are sharing all IPC primitives so perhaps one is substituted for the other for connection requests
• host's /etc is visible for snaps so perhaps the presence of mysql config file causes issues for the next cloud snap

Perhaps the expectations are wrong? Could we have a chat about this when you are online?

[maxbritto]

I can confirm that removing mysql and mariadb from the host worked for me :
sudo apt remove mariadb
sudo apt remove mysql
sudo apt autoremove
sudo apt purge

Then run the snap install of nextcloud and it should work.
I'm on a fresh raspbian lite

[kyrofa]

• host's /etc is visible for snaps so perhaps the presence of mysql config file causes issues for the next cloud snap

@zyga that doesn't seem true on Ubuntu, at least not entirely. I can cat /etc/hosts, for example, but:

$ sudo snap run --shell nextcloud.mysql <<EOF
cat /etc/mysql/my.cnf
EOF
cat: /etc/mysql/my.cnf: Permission denied

That's accompanied by the following in the syslog:

Oct 22 16:25:45 protege kernel: [339197.020803] audit: type=1400 audit(1571786745.884:1806): apparmor="DENIED" operation="open" profile="snap.nextcloud.mysql" name="/etc/mysql/my.cnf.fallback" pid=12091 comm="cat" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

On the host, of course:

$ cat /etc/mysql/my.cnf
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# <snip all the comments>

!includedir /etc/mysql/conf.d/

Or are you saying that this IS actually how it works on Debian? The entire host's /etc is bind-mounted into the core snap? If that's true, and comments like @xkl's suggest that it is, then you're probably right. However, that's not something individual snaps should have to work around, because it means that the snaps operate completely differently across distributions, which still seems like a snapd bug.

I can ping you on IRC in the morning, if you like.

[zyga]

@kyrofa permissions are another layer. My point is that /etc is _there_ and it's the host's /etc/, yes it is bind-mounted.

On Debian confinement is not enabled and you will actually successfully read the host's /etc/mysql/* files.

[kyrofa]

That's almost certainly the issue, then. It seems MySQL parses all the configuration files it can find, starting with /etc. On Debian, snaps are not properly isolated and it ends up reading the configuration from the host. I will look into ways to disable that behavior, but I will admit that playing cross-distro whack-a-mole is not my highest priority. If anyone has any ideas please propose them.

[zyga]

@kyrofa do a layout of tmpfs over /etc

[efrasyad]

Same issue when upgrading from 15 to 16, solved using @xkl workaround.
Will there be permanent fix for future upgrade?

Thanks.

[rbwrightjr91]

I can confirm that removing mysql and mariadb from the host worked for me :
sudo apt remove mariadb
sudo apt remove mysql
sudo apt autoremove
sudo apt purge

Then run the snap install of nextcloud and it should work.
I'm on a fresh raspbian lite

This worked for me, Ubuntu 18.04 LTS

[gerhartob]

folks, a dirty workaround that works for me (running bullseye):

$ mv /etc/mysql /etc/mysql.tmp
$ snap restart nextcloud
$ mv /etc/mysql.tmp /etc/mysql

-xkl

I don't know if this is better or even more ugly, but at least it's automated...
I've created an systemd overlay unit file and a script for the snap.nextcloud.mysql.service.
This way the SQL instance of the snap is started after the host SQL damon and the host config is moved into a temp dir before starting it and after starting it the config is restored.

systemctl edit snap.nextcloud.mysql.service
add the following lines:
[Unit]
After=mysqld.service
[Service]
ExecStartPre=+/usr/local/bin/nextcloud-snap-mysql-fix enable 2
ExecStartPost=+/usr/local/bin/nextcloud-snap-mysql-fix undo 10

and put this script into /usr/local/bin/nextcloud-snap-mysql-fix:
snap-nextcloud-mysql-fix.txt

It's still pretty messy but it works for me even after a reboot.

[patman-cp]

I'm no longer hitting this on Fedora 31. I haven't tried in a while and don't know how this was fixed.

I Fedora 31 up to date, though I'm using a recent git kernel to test some other problems on my system but that should not change anything:

# snap --version
snap 2.43.3-1.fc31
snapd 2.43.3-1.fc31
series 16
fedora 31
kernel 5.7.0-rc2+

# snap list nextcloud
Name Version Rev Tracking Publisher Notes
nextcloud 18.0.4snap1 20498 stable nextcloud✓ -

[xkl]

status report, unfortunately not solved on the following setup, and still using the dirty workaround:

# . /etc/os-release; echo $PRETTY_NAME
Debian GNU/Linux 10 (buster)

# snap --version
snap    2.45
snapd   2.45
series  16
debian  10
kernel  4.19.0-9-amd64

# snap list nextcloud
Name       Version      Rev    Tracking       Publisher   Notes
nextcloud  19.0.0snap1  21796  latest/stable  nextcloud✓  -

[Dinsmoor]

Hey this is still a problem on Debian, installing mysql-common will kill the nextcloud snap. This issue has been open for a long time. What can we do to help?

[kyrofa]

Yes, snapd considers it not their problem, I'm afraid. Someone encountering this error, can you please provide the output of the following commands?

$ snap run --shell nextcloud.mysql -c 'mysqld --print-defaults'
$ snap run --shell nextcloud.mysql -c 'mysqld --defaults-file=$SNAP/my.cnf --print-defaults'

[kyrofa]

Also, if you have this problem, try the beta/issue-913 channel and let me know if it fixes it:

$ sudo snap install nextcloud --channel=latest/beta/issue-913

Or if you already have it installed:

$ sudo snap refresh nextcloud --channel=latest/beta/issue-913

[xkl]

Here you are!

# snap run --shell nextcloud.mysql -c 'mysqld --print-defaults'
mysqld would have been started with the following arguments:
--user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock
--port=3306 --basedir=/usr --datadir=/var/lib/mysql --tmpdir=/tmp --lc-messages-dir=/usr/share/mysql
--skip-external-locking --bind-address=127.0.0.1 --key_buffer_size=16M --max_allowed_packet=16M
--thread_stack=192K --thread_cache_size=8 --myisam_recover_options=BACKUP --query_cache_limit=1M
--query_cache_size=16M --log_error=/var/log/mysql/error.log --expire_logs_days=10
--max_binlog_size=100M --character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci 

and

# snap run --shell nextcloud.mysql -c 'mysqld --defaults-file=$SNAP/my.cnf --print-defaults'
mysqld would have been started with the following arguments:
--user=root --max_allowed_packet=100M --secure-file-priv=NULL --skip-networking

I will try the beta/issue-913 channel later today or tomorrow evening (CEST), and report the results.

Thanks for revisiting this issue.

-xkl

[DrOwl]

For me:

# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 9.11 (stretch)
Release:        9.11
Codename:       stretch

# snap run --shell nextcloud.mysql -c 'mysqld --print-defaults'
mysqld would have been started with the following arguments:
--user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --basedir=/usr --datadir=/var/lib/mysql --tmpdir=/tmp --lc-messages-dir=/usr/share/mysql --skip-external-locking --bind-address=127.0.0.1 --key_buffer_size=16M --max_allowed_packet=16M --thread_stack=192K --thread_cache_size=8 --myisam-reco

# snap run --shell nextcloud.mysql -c 'mysqld --defaults-file=$SNAP/my.cnf --print-defaults'
mysqld would have been started with the following arguments:
--user=root --max_allowed_packet=100M --secure-file-priv=NULL --skip-networking 



md5-4755f1dd2037fdda82736b0f0b5bd3fe




#  snap remove nextcloud                                                                                                               
nextcloud removed

#  snap saved
Set  Snap       Age    Version      Rev    Size    Notes
6    nextcloud  9.96s  19.0.1snap1  22327   2468B  auto

# snap forget 6
Snapshot #6 forgotten.

# snap install nextcloud --channel=latest/beta/issue-913
nextcloud (beta/issue-913) 19.0.1snap1+git1.f4c9f02 from Nextcloud✓ installed



md5-647d2a1b1694e1fbef81dbcc2750352d




# snap start nextcloud
Started.

# snap logs -f  nextcloud                                                                                                              
2020-07-22T11:07:21Z nextcloud.mysql[32451]: 2020-07-22T11:07:21.675196Z 0 [ERROR] Aborting
2020-07-22T11:07:21Z nextcloud.mysql[32451]: Starting MySQL
2020-07-22T11:07:23Z nextcloud.renew-certs[32488]: Saving debug log to /var/snap/nextcloud/current/certs/certbot/logs/letsencrypt.log
2020-07-22T11:07:23Z nextcloud.renew-certs[32488]: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2020-07-22T11:07:23Z nextcloud.renew-certs[32488]: No renewals were attempted.
2020-07-22T11:07:23Z nextcloud.renew-certs[32488]: No hooks were run.
2020-07-22T11:07:23Z nextcloud.renew-certs[32488]: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2020-07-22T11:07:27Z nextcloud.mysql[32451]: ...... * The server quit without updating PID file (/tmp/pids/mysql.pid).
2020-07-22T11:07:27Z nextcloud.mysql[32451]: sed: can't read /var/snap/nextcloud/22369/mysql/root.ini: No such file or directory
2020-07-22T11:07:27Z nextcloud.mysql[32451]: Waiting for MySQL...



md5-647d2a1b1694e1fbef81dbcc2750352d




#  snap run --shell nextcloud.mysql -c 'mysqld --print-defaults'
mysqld would have been started with the following arguments:
--user=mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --basedir=/usr --datadir=/var/lib/mysql --tmpdir=/tmp --lc-messages-dir=/usr/share/mysql --skip-external-locking --bind-address=127.0.0.1 --key_buffer_size=16M --max_allowed_packet=16M --thread_stack=192K --thread_cache_size=8 --myisam-recover=BACKUP --query_cache_limit=1M --query_cache_size=16M --log_error=/var/log/mysql/error.log --expire_logs_days=10 --max_binlog_size=100M 


# snap run --shell nextcloud.mysql -c 'mysqld --defaults-file=$SNAP/my.cnf --print-defaults'
mysqld would have been started with the following arguments:
--user=root --max_allowed_packet=100M --secure-file-priv=NULL --skip-networking 

[kyrofa]

Ah, missed one, but okay now I'm pretty confident in this. See #1414, and test it out using the latest/beta/pr-1414 branch.

[Dinsmoor]

@kyrofa
The latest/beta/pr-1414 branch works for me.
I can have both mariadb-server (mysql) running and also have the nextcloud snap working correctly.
It does seem slower? May be just my perception, but at least this giant bug has a fix.
Thank you very much, now I can actually store my kids' photos and run other applications.
-Fellow Dad

[kyrofa]

The latest/beta/pr-1414 branch works for me.

Excellent, I'll ship it then, thank you.

It does seem slower? May be just my perception, but at least this giant bug has a fix.

Slower than not working at all :stuck_out_tongue: ? Or slower than when you didn't have mariadb-server installed, and thus could use the snap? If the latter, I can't explain it other than perception, because it should have ended up doing the same thing as it's doing now. In fact, I would expect it to start up a smidge faster now since it's not looking for other files at all anymore.