Server: Following symlinks on external storage causes loops

[garblixa]

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Steps to reproduce

Each of the approx. 600 LDAP users has symbolic links in his home directory, depending on the group membership, as follows:

+all -> /home/all
+allteachers -> /home/groups/TEACHERS
+classes -> /home/classes
+groups -> /home/groups
+software -> /home/software

When a user uses the Nextcloud (desktop) app to synchronize external shares, there is an endless loop.
Also the Nextcloud cronjob runs without end and fills the table oc_filecache by scanning the same files over and over again. The serverload is increased by the php nextcloud cronjob and mysql.

For example a user teacher1 within the group TEACHERS accesses:
/home/teacher1/+classes/1A/student1/+groups/TEACHERS...
...
/home/teacher1/+classes/12A/student12/+groups/TEACHERS...
and also the same share:
/home/groups/TEACHERS...
etc...

Expected behaviour

1. Symbolic links on external shares should not be followed, or it should be configurable.
2. If several users share the same external directory, these files should only be indexed once by occ files:scan to keep the size of the database small and to reduce the server load

Actual behaviour

1. Symbolic links on the external drives are followed and the same files are synchronized/indexed again and again
2. The same files are indexed by the cronjob, I guess "occ files:scan" for all users, although all users, except the home directory, have the same files.

Server configuration

Operating system:
Linux cloud 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64 GNU/Linux

Web server:
apache 2.4.38-3+deb10u3
Database:
mariadb-10.3

PHP version:
7.3.14-1~deb10u1

Nextcloud version: (see Nextcloud admin page)
18.0.3

Updated from an older Nextcloud/ownCloud or fresh install:
Fresh install

Where did you install Nextcloud from:
https://download.nextcloud.com/server/releases/nextcloud-18.0.3.zip

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:

• accessibility: 1.4.0
• activity: 2.11.0
• bruteforcesettings: 1.5.0
• calendar: 2.0.2
• cloud_federation_api: 1.1.0
• comments: 1.8.0
• contacts: 3.2.0
• dav: 1.14.0
• deck: 0.8.0
• federatedfilesharing: 1.8.0
• federation: 1.8.0
• files: 1.13.1
• files_antivirus: 2.2.1
• files_external: 1.9.0
• files_pdfviewer: 1.7.0
• files_rightclick: 0.15.2
• files_sharing: 1.10.1
• files_trashbin: 1.8.0
• files_versions: 1.11.0
• files_videoplayer: 1.7.0
• firstrunwizard: 2.7.0
• logreader: 2.3.0
• lookup_server_connector: 1.6.0
• nextcloud_announcements: 1.7.0
• notes: 3.2.0
• notifications: 2.6.0
• oauth2: 1.6.0
• onlyoffice: 4.1.4
• password_policy: 1.8.0
• polls: 1.3.0
• privacy: 1.2.0
• provisioning_api: 1.8.0
• serverinfo: 1.8.0
• settings: 1.0.0
• sharebymail: 1.8.0
• spreed: 8.0.5
• support: 1.1.0
• survey_client: 1.6.0
• systemtags: 1.8.0
• terms_of_service: 1.4.0
• text: 2.0.0
• theming: 1.9.0
• twofactor_backupcodes: 1.7.0
• updatenotification: 1.8.0
• user_ldap: 1.8.0
• viewer: 1.2.0
• workflowengine: 2.0.0
  Disabled:
• admin_audit
• encryption
• photos
• recommendations
  

Nextcloud configuration:

Config report

{
"system": {
"instanceid": "REMOVED SENSITIVE VALUE",
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"REMOVED SENSITIVE VALUE"
],
"datadirectory": "REMOVED SENSITIVE VALUE",
"dbtype": "mysql",
"version": "18.0.3.0",
"overwrite.cli.url": "REMOVED SENSITIVE VALUE",
"dbname": "REMOVED SENSITIVE VALUE",
"dbhost": "REMOVED SENSITIVE VALUE",
"dbport": "",
"dbtableprefix": "oc_",
"mysql.utf8mb4": true,
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"installed": true,
"memcache.locking": "\OC\Memcache\Redis",
"memcache.local": "\OC\Memcache\Redis",
"redis": {
"host": "REMOVED SENSITIVE VALUE",
"port": 6379
},
"mail_smtpmode": "smtp",
"mail_sendmailmode": "smtp",
"skeletondirectory": "",
"ldapIgnoreNamingRules": false,
"ldapProviderFactory": "OCA\User_LDAP\LDAPProviderFactory",
"ldapUserCleanupInterval": 30,
"loglevel": 2,
"maintenance": false,
"upgrade.disable-web": true,
"theme": "",
"lost_password_link": "disabled"
}
}

Are you using external storage, if yes which one: local/smb/sftp/...

| 1 | /all | SMB / CIFS | Log-in credentials, save in database | host: "pdc-server", share: "all", root: "", domain: "", show_hidden: false, timeout: "" | | All | |
| 2 | /allteachers | SMB / CIFS | Log-in credentials, save in database | host: "pdc-server", share: "allteachers", root: "", domain: "", show_hidden: false, timeout: "" | | | TEACHERS |
| 3 | /groups | SMB / CIFS | Log-in credentials, save in database | host: "pdc-server", share: "groups", root: "", domain: "", show_hidden: false, timeout: "" | | All | |
| 4 | /software | SMB / CIFS | Log-in credentials, save in database | host: "pdc-server", share: "software", root: "", domain: "", show_hidden: false, timeout: "" | | All | |
| 5 | /homes | SFTP | Log-in credentials, save in database | host: "schooladmin", root: "\/home\/teachers\/$user" | | | TEACHERS |
| 6 | /homes | SFTP | Log-in credentials, save in database | host: "schooladmin", root: "\/home\/sysadmins\/$user" | | | SYSADMINS |
| 7 | /homes | SFTP | Log-in credentials, save in database | host: "schooladmin", root: "\/home\/students\/$user" | | | STUDENTS |

Are you using encryption: yes/no
no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
LDAP

LDAP configuration (delete this part if not used)

LDAP config

+-------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Configuration | s01 |
+-------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| hasMemberOfFilterSupport | 1 |
| homeFolderNamingRule | attr:uid |
| lastJpegPhotoLookup | 0 |
| ldapAgentName | |
| ldapAgentPassword | * |
| ldapAttributesForGroupSearch | |
| ldapAttributesForUserSearch | |
| ldapBackupHost | |
| ldapBackupPort | |
| ldapBase | dc=schule,dc=edu |
| ldapBaseGroups | ou=group,dc=schule,dc=edu |
| ldapBaseUsers | ou=people,dc=schule,dc=edu |
| ldapCacheTTL | 600 |
| ldapConfigurationActive | 1 |
| ldapDefaultPPolicyDN | |
| ldapDynamicGroupMemberURL | |
| ldapEmailAttribute | |
| ldapExperiencedAdmin | 0 |
| ldapExpertUUIDGroupAttr | gidNumber |
| ldapExpertUUIDUserAttr | uid |
| ldapExpertUsernameAttr | uid |
| ldapExtStorageHomeAttribute | uid |
| ldapGidNumber | gidNumber |
| ldapGroupDisplayName | cn |
| ldapGroupFilter | (&(|(objectclass=SchoolGroup))(!(|(cn=ADMINISTRATION)(cn=DOMAINUSERS)(cn=TEMPLATES)(cn=WORKSTATIONS)))) |
| ldapGroupFilterGroups | ADMINISTRATION;DOMAINUSERS;STUDENTS;TEMPLATES;WORKSTATIONS |
| ldapGroupFilterMode | 1 |
| ldapGroupFilterObjectclass | SchoolGroup |
| ldapGroupMemberAssocAttr | member |
| ldapHost | schooladmin |
| ldapIgnoreNamingRules | |
| ldapLoginFilter | (&(&(|(objectclass=SchoolAccount))(|(memberof=cn=STUDENTS,ou=group,dc=schule,dc=edu)(memberof=cn=SYSADMINS,ou=group,dc=schule,dc=edu)(memberof=cn=TEACHERS,ou=group,dc=schule,dc=edu)))(uid=%uid)) |
| ldapLoginFilterAttributes | |
| ldapLoginFilterEmail | 0 |
| ldapLoginFilterMode | 0 |
| ldapLoginFilterUsername | 1 |
| ldapNestedGroups | 1 |
| ldapOverrideMainServer | |
| ldapPagingSize | 500 |
| ldapPort | 389 |
| ldapQuotaAttribute | |
| ldapQuotaDefault | |
| ldapTLS | 0 |
| ldapUserAvatarRule | default |
| ldapUserDisplayName | addressBookCN |
| ldapUserDisplayName2 | |
| ldapUserFilter | (&(|(objectclass=SchoolAccount))(|(memberof=cn=STUDENTS,ou=group,dc=schule,dc=edu)(memberof=cn=SYSADMINS,ou=group,dc=schule,dc=edu)(memberof=cn=TEACHERS,ou=group,dc=schule,dc=edu))) |
| ldapUserFilterGroups | STUDENTS;SYSADMINS;TEACHERS |
| ldapUserFilterMode | 0 |
| ldapUserFilterObjectclass | SchoolAccount |
| ldapUuidGroupAttribute | auto |
| ldapUuidUserAttribute | auto |
| turnOffCertCheck | 1 |
| turnOnPasswordChange | 0 |
| useMemberOfToDetectMembership | 1 |
+-------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Logs

Web server access log

Web server acces log

xxx.xxx.xxx.xxx - dub [26/Mar/2020:17:15:59 +0100] "PROPFIND /remote.php/dav/files/dub/homes/+classes/10AM/user1/+groups/TEACHERS/test.txt HTTP/1.1" 207 105075 "-" "Mozilla/5.0 (Windows) mirall/2.6.4stable-Win64 (build 20200303) (Nextcloud)"

[flammekueche]

Same problem on NC 19.0.0 (docker version).
PHP 7.4.7, Mariadb 10.4.13
Im not using LDAP, just a symlnks outside data is enough to cause the loop.

[flammekueche]

Huge increase of sql requests respond time, as well as database size.
Everywhere a symlnks exists, entries with path included "//" are recursively added.

SELECT storage,path,name FROM oc_filecache WHERE path LIKE '%//%' limit 1000;
+---------+----------------------------------------------------------------+---------------------------+
| storage | path | name |
+---------+----------------------------------------------------------------+---------------------------+
| 3 | files/Documents/Synology/Adblock//ad-blocker | ad-blocker |
| 3 | files/Documents/Synology//spamassassin | spamassassin |
| 3 | files/Documents/Synology//diskstation | diskstation |
| 3 | files/Documents/Synology//synoreport | synoreport |
...
| 3 | files/Documents/Synology/owncloud | owncloud |
| 3 | files/Documents/Synology//owncloud | owncloud |
| 3 | files/Documents/Synology///owncloud | owncloud |
| 3 | files/Documents/Synology////owncloud | owncloud |
| 3 | files/Documents/Synology/////owncloud | owncloud |
| 3 | files/Documents/Synology//////owncloud | owncloud |
| 3 | files/Documents/Synology///////owncloud | owncloud |
...
+---------+----------------------------------------------------------------+---------------------------+

Temporary workaround:
DELETE FROM oc_filecache WHERE path LIKE '%//%';

[adepertat]

I have the exact same symptoms except for the fact that I do not use symlinks. Rather my files are on a an NFS mount.

Found the same workaround as @flammekueche but it is unsatisfactory because the tablespace grows anyway. I can recover the space with ALTER TABLE oc_filecache FORCE; but if I'm not careful, the table will grow too large, the partition will fill up, and I am forced to TRUNCATE oc_filecache to recover the space.

NextCloud 19.0.0 official Docker image / MariaDB 10.4.13

[adepertat]

Updated to 19.0.1 today, issue persists.

[pohuing]

This also has the funny error that the scanner eventually fails and produces a log. Since we're dealing with infinite recursion here we get a logfile that eats up all the drive's space and then just stops. I've got 40+GB of trying to scan one directory right now.

[3v1n0]

Got a 122GB oc_filecache db because of this... And those are relative links that could be followed properly.