Server: All users' avatar pictures missing after NC 10.0.3 → 11.0.1
Hi everybody,
I've migrated my ownCloud instance to Nextcloud (path: ownCloud 9.0.8 → Nexcloud 9.0.56) and then updated to the current Nexcloud version 11.0.1 (path: Nextcloud 9.0.56 → 10.0.3 → 11.0.1). In this last upgrade from 10.0.3 to 11.0.1, all Nexcloud users' avatar pictures went missing.
Expected behaviour
Nextcloud users' avatars should be preserved.
Actual behaviour
Nextcloud users' avatars are gone and not returning (even after a considerable amount of time that a background job may have taken to convert them).
Server configuration
Operating system: Debian Jessie 8.7.0 (x64) on an Intel J1900 with 16 GB RAM
Web server: Apache/2.4.10 with mod_fcgid (formerly known as 'FastCGI') and suEXEC-custom (its docroot configured to /srv/www)
PHP version: PHP 7.0.15-1~dotdeb+8.1 from the Dotdeb repository plus the Redis-Cache compiled and activated, basically following the steps of @enoch85 's script, modifying the module's activation in fcgid, of course.
Database: 5.5.54-0+deb8u1
Nextcloud version: 11.0.1 (stable)
Updated from an older Nextcloud/ownCloud or fresh install: Migrated and updated (ownCloud 9.0.8 → Nextcloud 9.0.56 → Nextcloud 10.0.3 → Nextcloud 11.0.1
Where did you install Nextcloud from: manually from tarball (https://download.nextcloud.com/server/releases/nextcloud-11.0.1.tar.bz2).
Signing status: No errors have been found.
List of activated apps:
App list
Enabled:
- activity: 2.4.1
- admin_audit: 1.1.0
- apporder: 0.3.3
- calendar: 1.5.0
- comments: 1.1.0
- contacts: 1.5.2
- dav: 1.1.1
- federatedfilesharing: 1.1.1
- federation: 1.1.1
- files: 1.6.1
- files_external: 1.1.2
- files_pdfviewer: 1.0.1
- files_sharing: 1.1.1
- files_texteditor: 2.2
- files_versions: 1.4.0
- files_videoplayer: 1.0.0
- firstrunwizard: 2.0
- gallery: 16.0.0
- logreader: 2.0.0
- lookup_server_connector: 1.0.0
- nextcloud_announcements: 1.0
- notifications: 1.0.1
- password_policy: 1.1.0
- provisioning_api: 1.1.0
- serverinfo: 1.1.1
- sharebymail: 1.0.1
- survey_client: 0.1.5
- systemtags: 1.1.3
- tasks: 0.9.4
- templateeditor: 0.2
- theming: 1.1.1
- twofactor_backupcodes: 1.0.0
- updatenotification: 1.1.1
- workflowengine: 1.1.1
Disabled: - encryption
- external
- files_accesscontrol
- files_automatedtagging
- files_retention
- files_trashbin
- user_external
- user_ldap
- user_saml
The content of config/config.php:
Config report
{
"system": {
"instanceid": "REMOVED SENSITIVE VALUE",
"passwordsalt": "REMOVED SENSITIVE VALUE",
"datadirectory": "REMOVED SENSITIVE VALUE",
"tempdirectory": "REMOVED SENSITIVE VALUE",
"dbtype": "mysql",
"version": "11.0.1.2",
"dbname": "cloud",
"dbhost": "localhost",
"dbtableprefix": "oc_",
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"installed": true,
"maintenance": false,
"forcessl": true,
"theme": "",
"overwritewebroot": "",
"overwrite.cli.url": "",
"trusted_domains": [
"REMOVED SENSITIVE VALUE"
],
"log_type": "owncloud",
"logfile": "REMOVED SENSITIVE VALUE",
"loglevel": "1",
"logdateformat": "F d, Y H:i:s",
"logtimezone": "Europe\/Berlin",
"log_query": false,
"cron_log": true,
"mail_smtpmode": "smtp",
"mail_smtpsecure": "tls",
"mail_from_address": "cloud",
"mail_domain": "REMOVED SENSITIVE VALUE",
"mail_smtpauthtype": "LOGIN",
"mail_smtphost": "REMOVED SENSITIVE VALUE",
"mail_smtpauth": 1,
"mail_smtpname": "REMOVED SENSITIVE VALUE",
"mail_smtpport": "587",
"mail_smtppassword": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"forceSSLforSubdomains": true,
"memcache.local": "\OC\Memcache\Redis",
"filelocking.enabled": "true",
"memcache.distributed": "\OC\Memcache\Redis",
"memcache.locking": "\OC\Memcache\Redis",
"redis": {
"host": "localhost",
"port": 6379,
"timeout": 0,
"dbindex": 0
}
}
}
Are you using external storage, if yes which one: no
Are you using encryption: no
Are you using an external user-backend, if yes which one: no
If you need more information, don't hesitate to ask.
And yes, I've read #2399, but as already described: In my case, avatar pictures are not returning.
HackintoshHD
All 10 comments
They arein a wrong location. Once the background job runs successfully they should show up again.
Anything in your log file related to avatars?
nickvergessen
on 8 Feb 2017
@nickvergessen A grep -i 'avatar' owncloud.log returns:
{"reqId":"…","remoteAddr":"","app":"updater","message":"\\OC\\Repair::step: Repair step: Fix permissions so avatars can be stored again","level":1,"time":"February 04, 2017 21:58:20","method":"--","url":"--","user":"--","version":"9.1.3.2"}
{"reqId":"…","remoteAddr":"","app":"updater","message":"\\OC\\Repair::startProgress: Starting ... Fix permissions so avatars can be stored again (2)","level":1,"time":"February 04, 2017 21:58:20","method":"--","url":"--","user":"--","version":"9.1.3.2"}
{"reqId":"…","remoteAddr":"","app":"updater","message":"\\OC\\Repair::step: Repair step: Add move avatar background job","level":1,"time":"February 04, 2017 21:58:20","method":"--","url":"--","user":"--","version":"9.1.3.2"}
I'm a bit surprised to see "version":"9.1.3.2", I'm absolutely sure I made the upgrade step via NC 10.0.3 (is this by any chance the internal version number?).
HackintoshHD
on 9 Feb 2017
yes, 9.1 is the internal number for 10.0
There is no Started migrating avatars to AppData folder in the log? Then it looks like the avatars are not moved yet. Can you post a dump of your oc_jobs table?
nickvergessen
on 10 Feb 2017
@nickvergessen Thanks a lot for your feedback and for sheding some light on the internal version numbers!
No, there's no
Started migrating avatars to AppData folderin the log.As requested, here's my oc_jobs table - I've anonymised my users' names as _XY_name_changed_ and deleted the list of _File_Versions_ entries for the same reason, which I hope is okay:
mysql> select * from oc_jobs;
+------+----------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------+-------------+
| id | class | argument | last_run | last_checked | reserved_at |
+------+----------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------+-------------+
| 3 | OCA\Activity\BackgroundJob\EmailNotification | null | 1486236199 | 0 | 0 |
| 4 | OCA\Activity\BackgroundJob\ExpireActivities | null | 1486234067 | 0 | 0 |
| 3865 | OCA\DAV\CardDAV\SyncJob | null | 1486234027 | 0 | 0 |
| 3866 | OCA\Federation\SyncJob | null | 1486234002 | 0 | 0 |
| 3867 | OCA\Files\BackgroundJob\ScanFiles | null | 1486236028 | 0 | 0 |
| 3868 | OCA\Files\BackgroundJob\DeleteOrphanedItems | null | 1486233902 | 0 | 0 |
| 3869 | OCA\Files\BackgroundJob\CleanupFileLocks | null | 1486236602 | 0 | 0 |
| 3871 | OCA\Files_Versions\BackgroundJob\ExpireVersions | null | 1486235978 | 0 | 0 |
| 3872 | \OC\Authentication\Token\DefaultTokenCleanupJob | null | 0 | 1486237214 | 0 |
| 3873 | OC\Settings\RemoveOrphaned | null | 0 | 1486237215 | 0 |
| 3874 | OCA\UpdateNotification\Notification\BackgroundJob | null | 0 | 1486237215 | 0 |
| 3875 | OC\Migration\BackgroundRepair | {"app":"dav","step":"OCA\\DAV\\Migration\\GenerateBirthdays"} | 0 | 1486237243 | 0 |
| 3876 | OCA\Files_Sharing\DeleteOrphanedSharesJob | null | 0 | 1486237244 | 0 |
| 3877 | OCA\Files_Sharing\ExpireSharesJob | null | 0 | 1486237244 | 0 |
| 3878 | OCA\Survey_Client\BackgroundJobs\AdminNotification | null | 0 | 1486237262 | 0 |
| 3879 | OCA\DAV\Migration\ValueFix | {"user":"01_name_replaced"} | 0 | 1486241892 | 0 |
| 3880 | OCA\DAV\Migration\ValueFix | {"user":"02_name_replaced"} | 0 | 1486241892 | 0 |
| 3881 | OCA\DAV\Migration\ValueFix | {"user":"03_name_replaced"} | 0 | 1486241892 | 0 |
| 3882 | OCA\DAV\Migration\ValueFix | {"user":"04_name_replaced"} | 0 | 1486241892 | 0 |
| 3883 | OCA\DAV\Migration\ValueFix | {"user":"05_name_replaced"} | 0 | 1486241893 | 0 |
| 3884 | OCA\DAV\Migration\ValueFix | {"user":"06_name_replaced"} | 0 | 1486241893 | 0 |
| 3885 | OCA\DAV\Migration\ValueFix | {"user":"07_name_replaced"} | 0 | 1486241893 | 0 |
| 3886 | OCA\DAV\Migration\ValueFix | {"user":"08_name_replaced"} | 0 | 1486241893 | 0 |
| 3887 | OCA\DAV\Migration\ValueFix | {"user":"09_name_replaced"} | 0 | 1486241893 | 0 |
| 3888 | OCA\DAV\Migration\ValueFix | {"user":"10_name_replaced"} | 0 | 1486241893 | 0 |
| 3889 | OCA\DAV\Migration\ValueFix | {"user":"11_name_replaced"} | 0 | 1486241893 | 0 |
| 3890 | OCA\DAV\Migration\ValueFix | {"user":"12_name_replaced"} | 0 | 1486241893 | 0 |
| 3891 | OCA\NextcloudAnnouncements\Cron\Crawler | null | 0 | 1486241900 | 0 |
| 3892 | OC\Repair\NC11\MoveAvatarsBackgroundJob | null | 0 | 1486241900 | 0 |
| 3893 | OC\Repair\NC11\CleanPreviewsBackgroundJob | {"uid":"01_name_replaced"} | 0 | 1486241900 | 0 |
| 3894 | OC\Repair\NC11\CleanPreviewsBackgroundJob | {"uid":"02_name_replaced"} | 0 | 1486241900 | 0 |
| 3895 | OC\Repair\NC11\CleanPreviewsBackgroundJob | {"uid":"03_name_replaced"} | 0 | 1486241900 | 0 |
| 3896 | OC\Repair\NC11\CleanPreviewsBackgroundJob | {"uid":"04_name_replaced"} | 0 | 1486241900 | 0 |
| 3897 | OC\Repair\NC11\CleanPreviewsBackgroundJob | {"uid":"05_name_replaced"} | 0 | 1486241900 | 0 |
| 3898 | OC\Repair\NC11\CleanPreviewsBackgroundJob | {"uid":"06_name_replaced"} | 0 | 1486241900 | 0 |
| 3899 | OC\Repair\NC11\CleanPreviewsBackgroundJob | {"uid":"07_name_replaced"} | 0 | 1486241900 | 0 |
| 3900 | OC\Repair\NC11\CleanPreviewsBackgroundJob | {"uid":"08_name_replaced"} | 0 | 1486241900 | 0 |
| 3901 | OC\Repair\NC11\CleanPreviewsBackgroundJob | {"uid":"09_name_replaced"} | 0 | 1486241900 | 0 |
| 3902 | OC\Repair\NC11\CleanPreviewsBackgroundJob | {"uid":"10_name_replaced"} | 0 | 1486241900 | 0 |
| 3903 | OC\Repair\NC11\CleanPreviewsBackgroundJob | {"uid":"11_name_replaced"} | 0 | 1486241900 | 0 |
| 3904 | OC\Repair\NC11\CleanPreviewsBackgroundJob | {"uid":"12_name_replaced"} | 0 | 1486241900 | 0 |
| 3905 | OC\Command\CommandJob | "O:33:\"OCA\\Files_Versions\\… [anonymised]
[…]
| 4011 | OC\Command\CommandJob | "O:33:\"OCA\\Files_Versions\\… [anonymised] | 0 | 1487015794 | 0 |
+------+----------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------+-------------+
148 rows in set (0.00 sec)
OC\Repair\NC11\MoveAvatarsBackgroundJob is still there, but lastrun is 0. That's odd.
The only thing I could imagine is that either cron is executed with a wrong user, or the directories are not writable by the user executing the cron.
nickvergessen
on 20 Feb 2017
Thanks, that sounds very plausible indeed. - The user name for my Nextcloud instance is _www-cloud_. The status for cron execution in Nextcloud's web interface admin section is green:
Excerpt from crontab of _www-cloud_ (domain anonymised):
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
*/15 * * * * php-cgi -f /srv/www/cloud.my-domain.tld/htdocs/cron.php
I'll check over the days to come whether I find any signs of a failing cron execution despite the "all green" status above.
As for directory permissions: Which directories are the relevant here for avatars?
Thanks a lot!
HackintoshHD
on 20 Feb 2017
data/appdata_*/avatar/* and of course all parent folders if the children do not exist.
nickvergessen
on 22 Feb 2017
Permissions on data/appdata_$some_number/avatar seem correct as far as I can see:
drwxr-xr-x 6 www-cloud www-cloud 4096 Feb 4 21:59 .
drwxrwx--- 17 www-cloud www-data 4096 Feb 4 21:58 ..
drwxr-xr-x 2 www-cloud www-cloud 4096 Feb 4 22:01 appstore
drwxr-xr-x 16 www-cloud www-cloud 4096 Feb 4 22:03 avatar
drwxr-xr-x 70 www-cloud www-cloud 4096 Feb 20 22:20 preview
drwxr-xr-x 3 www-cloud www-cloud 4096 Feb 4 21:59 theming
Okay, as promised above, I've had a look for possible signs of a failing cronjob execution despite the "all green" status for it in Nextcloud's web interface.
Executing the command of my crontab entry for the relevant user www-cloud _manually_ returns the following (anonymised) output:
myserver:/var/log $ sudo -u www-cloud php-cgi -f /srv/www/cloud.my-domain.tld/htdocs/cron.php
PHP Notice: Undefined index: SCRIPT_FILENAME in /srv/www/cloud.my-domain.tld/htdocs/lib/base.php on line 133
PHP Notice: Undefined index: SCRIPT_NAME in /srv/www/cloud.my-domain.tld/htdocs/lib/base.php on line 140
PHP Notice: Undefined index: SCRIPT_FILENAME in /srv/www/cloud.my-domain.tld/htdocs/lib/base.php on line 141
PHP Notice: Undefined index: REQUEST_URI in /srv/www/cloud.my-domain.tld/htdocs/lib/base.php on line 177
{"data":{"message":"Backgroundjobs are using system cron!"},"status":"error"}
At first sight, I'm inclined to assume something goes wrong in Nextcloud 11 when executing the cronjob under my suexec-separated, dedicated user www-cloud.
Could this in any way be related to the old ownCloud bug report 11551?
HackintoshHD
on 26 Feb 2017
Good news, problem solved! - Turns out I had committed two mistakes:
- cron.php must be executed using php (or for some php-cli), but definitely _not_ using php-cgi as above, of course (strikethrough: "php
-cgi", domain anonymised):
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command
*/15 * * * * /usr/bin/php -f /srv/www/cloud.my-domain.tld/htdocs/cron.php
- When setting up the dedicated mysql user for ownCloud, I had been way too conservative with the user's mysql privileges: I had so to speak granted a 'whitelist' of privileges, i.e.
grant select,index,insert,update,delete,create,drop,alter on cloud.* to cloud@localhost identified by 'secret';
rather than executing agrant all privileges on cloud.* to cloud@localhost identified by 'secret';
as the Nextcloud documentation would suggest.
This had left me with a Nextcloud mysql user missing at least thelock tablesprivilege, thus incapable of locking theoc_jobstable, which would apparently prevent scheduled jobs in it from being executed completely. - _Outch!_
Both is fixed now and the formerly missing avatars have returned immediately.
Thanks a lot, @nickvergessen, for your continued support!
HackintoshHD
on 12 Mar 2017
Related issues
ghost
·
3Comments
jancborchardt
·
3Comments
mfechner
·
3Comments
rullzer
·
3Comments
georgehrke
·
3Comments
Most helpful comment
Good news, problem solved! - Turns out I had committed two mistakes:
-cgi", domain anonymised):grant select,index,insert,update,delete,create,drop,alter on cloud.* to cloud@localhost identified by 'secret';rather than executing a
grant all privileges on cloud.* to cloud@localhost identified by 'secret';as the Nextcloud documentation would suggest.
This had left me with a Nextcloud mysql user missing at least the
lock tablesprivilege, thus incapable of locking theoc_jobstable, which would apparently prevent scheduled jobs in it from being executed completely. - _Outch!_Both is fixed now and the formerly missing avatars have returned immediately.
Thanks a lot, @nickvergessen, for your continued support!