Server: lookup_server_connector creates a lot of oc_jobs entries, high DB load during cron, and tries to sync even if disabled in options

Created on 6 Jan 2019 · 12Comments · Source: nextcloud/server

Steps to reproduce

Upgraded to NC 14 some days ago
All Federation options are disabled
Users synced from LDAP (ca. 5k)

Expected behaviour

There should be no connections to external servers. The system should run stable.

Actual behaviour

A lot of traffic to lookup.nextcloud.com. Increasing, high DB load. Leak of personal information (usernames) to nextcloud.com without permission.

It looks like lookup_server_connector app is trying to update lookup.nextcloud.com on ervery user change, also if users are synced via LDAP (and nothing is changed), even if the global adressbook and all other federation options are disabled.
This often fails (maybe server fail / network / proxy timeout) and creates a entry in oc_jobs for retrying during cronjob.
I got about 300k of lookup retries in the oc_jobs table, The cron job has no chance to get this done, so all other jobs (mails) are not processed.

My temporary solution was to delete the lookup_server_connector app completely (occ app:disable didn't work) and deleting all entries for this class in oc_jobs.

Server configuration

Operating system:
Debian 9
Web server:
Apache 2.4
Database:
MariaDB (10.1.37)
PHP version:
7.2
Nextcloud version: (see Nextcloud admin page)
14.0.4.2
Updated from an older Nextcloud/ownCloud or fresh install:
Updated

Signing status:

Signing status

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

No errors have been found.

Nextcloud configuration:

Config report

{
"system": {
"instanceid": "REMOVED SENSITIVE VALUE",
"passwordsalt": "REMOVED SENSITIVE VALUE",
"secret": "REMOVED SENSITIVE VALUE",
"trusted_domains": [
"",
"c"
],
"datadirectory": "REMOVED SENSITIVE VALUE",
"dbtype": "mysql",
"version": "14.0.4.2",
"dbname": "REMOVED SENSITIVE VALUE",
"dbhost": "REMOVED SENSITIVE VALUE",
"dbport": "",
"dbtableprefix": "oc_",
"mysql.utf8mb4": true,
"dbuser": "REMOVED SENSITIVE VALUE",
"dbpassword": "REMOVED SENSITIVE VALUE",
"dbdriveroptions": {
"1002": "SET wait_timeout = 28800"
},
"installed": true,
"default_language": "de",
"updatechecker": true,
"has_internet_connection": true,
"memcached_servers": [
[
"localhost",
11211
]
],
"appstoreenabled": true,
"ldapIgnoreNamingRules": false,
"ldapProviderFactory": "\OCA\User_LDAP\LDAPProviderFactory",
"loglevel": 1,
"trusted_proxies": "REMOVED SENSITIVE VALUE",
"forwarded_for_headers": [
"HTTP_X_FORWARDED_FOR"
],
"overwritehost": ",
"overwriteprotocol": "https",
"overwrite.cli.url": "",
"mail_from_address": "REMOVED SENSITIVE VALUE",
"mail_smtpmode": "smtp",
"mail_domain": "REMOVED SENSITIVE VALUE",
"skeletondirectory": "\/data\/nextcloud_skel",
"theme": "tui",
"activity_expire_days": 14,
"integrity.check.disabled": true,
"memcache.local": "\OC\Memcache\APCu",
"memcache.distributed": "\OC\Memcache\Redis",
"memcache.locking": "\OC\Memcache\Redis",
"redis": {
"host": "REMOVED SENSITIVE VALUE",
"port": 6379,
"password": "REMOVED SENSITIVE VALUE"
},
"knowledgebaseenabled": false,
"defaultapp": "apporder",
"sharing.minSearchStringLength": 4,
"sharing.maxAutocompleteResults": 5,
"maintenance": false,
"debug": false,
"htaccess.RewriteBase": "\/",
"proxy": "*",
"mail_smtphost": "REMOVED SENSITIVE VALUE",
"mail_smtpport": "25",
"updater.release.channel": "production"
},
"apps": {
"QuotaAdmin": {
"enabled": "no",
"installed_version": "0.0.1",
"types": ""
},
"accessibility": {
"enabled": "yes",
"installed_version": "1.0.1",
"types": ""
},
"activity": {
"enabled": "yes",
"installed_version": "2.7.0",
"notify_email_announcementcenter": "1",
"notify_email_calendar": "0",
"notify_email_calendar_event": "0",
"notify_email_calendar_todo": "0",
"notify_email_circles_as_member": "0",
"notify_email_circles_as_moderator": "1",
"notify_email_comments": "0",
"notify_email_favorite": "0",
"notify_email_file_changed": "0",
"notify_email_file_created": "0",
"notify_email_file_deleted": "0",
"notify_email_file_downloaded": "0",
"notify_email_file_restored": "0",
"notify_email_public_links": "0",
"notify_email_remote_share": "0",
"notify_email_shared": "0",
"notify_email_systemtags": "0",
"notify_setting_batchtime": "86400",
"notify_setting_self": "1",
"notify_setting_selfemail": "0",
"notify_stream_calendar": "1",
"notify_stream_calendar_event": "1",
"notify_stream_calendar_todo": "1",
"notify_stream_circles_as_member": "1",
"notify_stream_circles_as_moderator": "1",
"notify_stream_comments": "1",
"notify_stream_favorite": "1",
"notify_stream_file_changed": "1",
"notify_stream_file_created": "1",
"notify_stream_file_deleted": "1",
"notify_stream_file_downloaded": "1",
"notify_stream_file_favorite": "0",
"notify_stream_file_restored": "1",
"notify_stream_public_links": "1",
"notify_stream_remote_share": "1",
"notify_stream_shared": "1",
"notify_stream_systemtags": "1",
"types": "filesystem"
},
"admin_audit": {
"enabled": "no",
"installed_version": "1.2.0",
"types": "logging"
},
"announcementcenter": {
"enabled": "yes",
"installed_version": "3.3.1",
"types": "logging"
},
"apporder": {
"enabled": "yes",
"hidden": "[]",
"installed_version": "0.6.0",
"order": "[\"\/apps\/files\/\",\"\/apps\/activity\/\",\"\/apps\/calendar\/\",\"\/apps\/tasks\/\",\"\/apps\/circles\/\",\"\/apps\/spreed\/\",\"\/apps\/notes\/\",\"\/apps\/contacts\/\",\"\/apps\/gallery\/\",\"\/apps\/announcementcenter\/\",\"\/apps\/gpxedit\/\",\"\/apps\/quotaadmin\/\"]",
"types": ""
},
"audioplayer": {
"enabled": "no",
"installed_version": "2.1.0",
"types": "filesystem"
},
"backgroundjob": {
"lastjob": "515531"
},
"bookmarks": {
"enabled": "no",
"installed_version": "0.10.1",
"types": ""
},
"bruteForce": {
"whitelist_1": ""
},
"bruteforcesettings": {
"enabled": "yes",
"installed_version": "1.3.0",
"types": ""
},
"calendar": {
"enabled": "yes",
"installed_version": "1.6.4",
"types": ""
},
"circles": {
"allow_federated": "0",
"allow_linked_groups": "0",
"enabled": "yes",
"installed_version": "0.15.2",
"members_limit": "1000",
"types": ""
},
"cloud_federation_api": {
"enabled": "yes",
"installed_version": "0.0.1",
"types": "filesystem"
},
"comments": {
"enabled": "yes",
"installed_version": "1.4.0",
"types": "logging"
},
"contacts": {
"enabled": "yes",
"installed_version": "2.1.8",
"types": ""
},
"core": {
"backgroundjobs_mode": "cron",
"installed.bundles": "[\"CoreBundle\"]",
"installedat": "1506428024.7302",
"lastcron": "1546800408",
"lastupdateResult": "[]",
"lastupdatedat": "1546799081",
"moveavatarsdone": "yes",
"previewsCleanedUp": "1",
"public_files": "files_sharing\/public.php",
"public_webdav": "dav\/appinfo\/v1\/publicwebdav.php",
"scss.variables": "385ed5e08336aefb25c8b98f0028dbe6",
"shareapi_allow_group_sharing": "yes",
"shareapi_default_permission_canshare": "no",
"shareapi_default_permissions": "15",
"umgmt_show_backend": "false",
"umgmt_show_email": "false",
"umgmt_show_last_login": "false",
"updater.secret.created": "1545907978",
"vendor": "nextcloud"
},
"dashboard": {
"enabled": "no",
"installed_version": "4.0.5",
"types": ""
},
"dav": {
"buildCalendarSearchIndex": "yes",
"enabled": "yes",
"installed_version": "1.6.0",
"types": "filesystem"
},
"drop_account": {
"enabled": "yes",
"installed_version": "0.0.13",
"types": ""
},
"federatedfilesharing": {
"enabled": "yes",
"installed_version": "1.4.0",
"types": ""
},
"federation": {
"enabled": "yes",
"installed_version": "1.4.0",
"types": "authentication"
},
"files": {
"cronjob_scan_files": "500",
"default_quota": "10 GB",
"enabled": "yes",
"installed_version": "1.9.0",
"types": "filesystem"
},
"files_clipboard": {
"enabled": "no",
"installed_version": "0.7.1",
"types": ""
},
"files_downloadactivity": {
"enabled": "yes",
"installed_version": "1.3.0",
"types": "filesystem"
},
"files_external": {
"allow_user_mounting": "no",
"enabled": "no",
"installed_version": "1.3.0",
"types": "filesystem",
"user_mounting_backends": "ftp,dav,owncloud,sftp,amazons3,dropbox,googledrive,swift,\OC\Files\Storage\SFTP_Key"
},
"files_fulltextsearch": {
"enabled": "yes",
"files_audio": "0",
"files_encrypted": "0",
"files_external": "0",
"files_federated": "0",
"files_group_folders": "0",
"files_image": "0",
"files_local": "1",
"files_office": "1",
"files_pdf": "1",
"files_size": "5",
"installed_version": "1.1.1",
"types": "filesystem"
},
"files_markdown": {
"enabled": "yes",
"installed_version": "2.0.5",
"types": ""
},
"files_pdfviewer": {
"enabled": "yes",
"installed_version": "1.3.2",
"types": ""
},
"files_retention": {
"enabled": "yes",
"installed_version": "1.3.1",
"types": "filesystem"
},
"files_rightclick": {
"enabled": "yes",
"installed_version": "0.8.4",
"types": ""
},
"files_sharing": {
"enabled": "yes",
"incoming_server2server_share_enabled": "no",
"installed_version": "1.6.2",
"lookupServerUploadEnabled": "no",
"outgoing_server2server_share_enabled": "no",
"types": "filesystem"
},
"files_texteditor": {
"enabled": "yes",
"installed_version": "2.6.0",
"types": ""
},
"files_trashbin": {
"enabled": "yes",
"installed_version": "1.4.1",
"types": "filesystem,dav"
},
"files_versions": {
"enabled": "yes",
"installed_version": "1.7.1",
"types": "filesystem,dav"
},
"files_videoplayer": {
"enabled": "yes",
"installed_version": "1.3.0",
"types": ""
},
"firstrunwizard": {
"enabled": "no",
"installed_version": "2.1",
"types": "logging"
},
"fulltextsearch": {
"app_navigation": "0",
"cron_err_reset": "1546796703",
"enabled": "yes",
"installed_version": "1.1.0",
"provider_indexed": "{\"files\":\"1\",\"test_provider\":\"0\"}",
"search_platform": "OCA\FullTextSearch_ElasticSearch\Platform\ElasticSearchPlatform",
"types": ""
},
"fulltextsearch_elasticsearch": {
"analyzer_tokenizer": "standard",
"elastic_host": "",
"elastic_index": "my_index",
"enabled": "yes",
"installed_version": "1.0.2",
"types": ""
},
"gallery": {
"enabled": "yes",
"installed_version": "18.1.0",
"types": ""
},
"gpxedit": {
"enabled": "yes",
"installed_version": "0.0.10",
"types": ""
},
"groupfolders": {
"enabled": "no",
"installed_version": "1.1.0",
"types": "filesystem"
},
"guests": {
"allow_external_storage": "false",
"enabled": "yes",
"group": "guests",
"installed_version": "1.0.0",
"types": "authentication",
"usewhitelist": "true",
"whitelist": "activity,apporder,circles,files_markdown,files_sharing,files_texteditor,files_trashbin,files_versions,gallery,notifications,spreed"
},
"logreader": {
"enabled": "yes",
"installed_version": "2.0.0",
"levels": "00111",
"ocsid": "170871",
"relativedates": "1",
"types": ""
},
"lookup_server_connector": {
"enabled": "yes",
"installed_version": "1.2.0",
"types": "authentication"
},
"metadata": {
"enabled": "yes",
"installed_version": "0.8.0",
"types": ""
},
"nextcloud_announcements": {
"enabled": "no",
"installed_version": "1.1",
"types": "logging"
},
"notes": {
"enabled": "yes",
"installed_version": "2.5.1",
"types": ""
},
"notifications": {
"enabled": "yes",
"installed_version": "2.2.1",
"types": "logging"
},
"oauth2": {
"enabled": "yes",
"installed_version": "1.2.1",
"types": "authentication"
},
"password_policy": {
"enabled": "no",
"installed_version": "1.2.2",
"types": ""
},
"provisioning_api": {
"enabled": "yes",
"installed_version": "1.4.0",
"types": "prevent_group_restriction"
},
"quota_warning": {
"alert_email": "yes",
"enabled": "yes",
"installed_version": "1.3.0",
"repeat_warning": "14",
"types": "filesystem",
"warning_email": "yes"
},
"quotaadmin": {
"enabled": "yes",
"installed_version": "0.0.1",
"types": ""
},
"serverinfo": {
"enabled": "yes",
"installed_version": "1.4.0",
"types": ""
},
"sharebymail": {
"enabled": "no",
"installed_version": "1.2.0",
"types": "filesystem"
},
"spreed": {
"enabled": "yes",
"installed_version": "4.0.2",
"signaling_ticket_secret": *
"stun_servers": "[\"stun.nextcloud.com:443\"]",
"types": "prevent_group_restriction"
},
"support": {
"enabled": "no",
"installed_version": "1.0.0",
"types": ""
},
"survey_client": {
"enabled": "no",
"installed_version": "1.0.0",
"types": ""
},
"systemtags": {
"enabled": "yes",
"installed_version": "1.4.0",
"types": "logging"
},
"tasks": {
"enabled": "yes",
"installed_version": "0.9.8",
"types": ""
},
"theming": {
"enabled": "no",
"installed_version": "1.3.0",
"types": "logging"
},
"twofactor_backupcodes": {
"enabled": "yes",
"installed_version": "1.3.1",
"types": ""
},
"twofactor_totp": {
"enabled": "yes",
"installed_version": "1.5.0",
"types": ""
},
"updatenotification": {
"enabled": "yes",
"installed_version": "1.4.1",
"types": "",
"update_check_errors": "0"
},
"user_ldap": {
"_lastChange": "1545915889",
"background_sync_interval": "1800",
"background_sync_offset": "0",
"background_sync_prefix": "",
"cleanUpJobOffset": "4600",
"enabled": "yes",
"has_memberof_filter_support": "0",
"home_folder_naming_rule": "attr:uid",
"installed_version": "1.4.0",
"last_jpegPhoto_lookup": "0",
"ldap_agent_password": "REMOVED SENSITIVE VALUE",
"ldap_attributes_for_group_search": "",
"ldap_attributes_for_user_search": "sn\nuid\nmail",
"ldap_backup_host": "",
"ldap_backup_port": "",
"ldap_base": "ou=user,o=uni",
"ldap_base_groups": "ou=user,o=uni",
"ldap_base_users": "ou=user,o=uni",
"ldap_cache_ttl": "3600",
"ldap_configuration_active": "1",
"ldap_default_ppolicy_dn": "",
"ldap_display_name": "sn",
"ldap_dn": "cn=proxy-nextcloud,i",
"ldap_dynamic_group_member_url": "",
"ldap_email_attr": "mail",
"ldap_experienced_admin": "1",
"ldap_expert_username_attr": "uid",
"ldap_expert_uuid_group_attr": "",
"ldap_expert_uuid_user_attr": "uid",
"ldap_gid_number": "gidNumber",
"ldap_group_display_name": "cn",
"ldap_group_filter": "",
"ldap_group_filter_mode": "0",
"ldap_group_member_assoc_attribute": "uniqueMember",
"ldap_groupfilter_groups": "",
"ldap_groupfilter_objectclass": "",
"ldap_host": "ldap:\/\/localhost",
"ldap_login_filter": "",
"ldap_login_filter_mode": "0",
"ldap_loginfilter_attributes": "mail",
"ldap_loginfilter_email": "0",
"ldap_loginfilter_username": "1",
"ldap_nested_groups": "0",
"ldap_override_main_server": "",
"ldap_paging_size": "0",
"ldap_port": "389",
"ldap_quota_attr": "",
"ldap_quota_def": "",
"ldap_tls": "0",
"ldap_turn_off_cert_check": "0",
"ldap_turn_on_pwd_change": "0",
"ldap_user_display_name_2": "mail",
"ldap_user_filter_mode": "0",
"ldap_userfilter_groups": "",
"ldap_userfilter_objectclass": "inetOrgPerson",
"ldap_userlist_filter": "**",
"types": "authentication",
"use_memberof_to_detect_membership": "1"
},
"workflowengine": {
"enabled": "yes",
"installed_version": "1.4.0",
"types": "filesystem"
}
}
}

Are you using external storage, if yes which one: local/smb/sftp/...
no
Are you using encryption: yes/no
no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
LDAP

LDAP configuration (delete this part if not used)

LDAP config

</details>

### Client configuration
**Browser:**
not relevant
**Operating system:**

### Logs
#### Web server error log
<details>
<summary>Web server error log</summary>
no errors
</details>

#### Nextcloud log (data/nextcloud.log)
<details>
<summary>Nextcloud log</summary>
no errors
</details>

#### Browser log
<details>
<summary>Browser log</summary>

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...
```

0. Needs triage bug

Source

alx-tuilmenau

All 12 comments

Right. I see what is happening. The lookup server is indeed disabled. But since we do not have the state we try to tell it to remove all the data.

@schiessle any idea how to solve this?

rullzer on 7 Jan 2019

Ok I think I have a solution in mind.

The moment that the lookupserver uploads get disabled. We should insert a background job. This job should

Iterate over all users that have logged in
Check if any of their properties are marked as public
- if so insert a job to remove this info from the server
- if not skip the user

rullzer on 7 Jan 2019

@alx-tuilmenau @rullzer One workaround is to set an empty value for the lookup_server in config.php. The background jobs will not be rescheduled then, if the lookup server isn't reachable.

juliushaertl on 7 Jan 2019

One workaround is to set an empty value for the lookup_server in config.php. The background jobs will not be rescheduled then, if the lookup server isn't reachable.

yes, this would be a quick workaround to solve the problem.

As a more general solution, I wounder if we should introduce a "dsiable_lookup_server" config.php switch which would be:

more obvious then setting 'lookup_server' to an empty string
could skip the process of updating the lookupserver right at the beginning
would allow us to hide the additional visibility settings on the personal page if the lookup server is disabled.

schiessle on 7 Jan 2019

👍1

In my opinion, if the "global adressbook" is not activated in the admin options, there should be no requests at all. Noone expects the leak of the usernames to an external server if the option is turned off, it's also a GDPR problem.
Of course, on turning off the option it's ok to remove users, but this should happen once on turning off and not on every user change and not if the option was never turned on.

If I set lookup_server to an empty value and re-activate the app, a lot of jobs are created very fast again, it's the same problem (it retries 5 times via cron). Maybe it depends on the configuration here, because outgoing http(s) is done via proxy. It looks like every LDAP sync, maybe on every login, will trigger the lookup_server sync. Setting lookup_server to an empty string is not a solution.

alx-tuilmenau on 7 Jan 2019

@alx-tuilmenau ah, now I saw that a setting already exists "Allow users to publish their data to a global and public address book", I missed it when I looked at the settings matrix the first time.

Yes, we could use this to disable the connection to the lookup server completely which would come with the draw back that already published data would stay at the lookup server. So we would probably need something like suggested from @rullzer https://github.com/nextcloud/server/issues/13396#issuecomment-451844152

If I set lookup_server to an empty value and re-activate the app, a lot of jobs are created very fast again, it's the same problem (it retries 5 times via cron).

With a quick look at the code I confirm this behavior. The connector will try to update the lookup server, will fail and create the background job, the background job will disappear after first try. But of course on every sync with the LDAP we will create them again and again. So setting the lookup_server to '' will at least help in that regard that the background jobs are cleaned up again.

I will work on a fix for this part, this should be straight forward.

schiessle on 7 Jan 2019

We can see the same behaviour after upgrading from NC13 to NC14. Unfortunately the enormous job queue kills the whole server after a while.

Is there something we can do right now to resolve the issue? (empty vaule for lookup_server parameter didn't help)

jngeiger on 9 Jan 2019

👍1

@jngeiger See my first post, because I don't use federation, I moved apps/lookup_server_connector folder to a different location (outside nextcloud/docroot) and deleted all entries with class like %LookupServer% from oc_jobs table.

alx-tuilmenau on 9 Jan 2019

I'm also seeing this on my company server with AD-backed authentication. I noticed CPU load slowly creeping up after upgrading to 14, and finally tracked it back to this issue. Removing lookup_server_connector to mitigate the issue has it under control for now, but I'd like to not have to do that after every upgrade.

bremensaki on 25 Feb 2019

Is this fix likely to get a backport to 15.x?

bremensaki on 5 Mar 2019

There is a backport for NC 15:
https://github.com/nextcloud/server/pull/14696

tflidd on 27 Jun 2019

Hi,

I also had this bug that generated enormous amount of UPDATE and INSERT each 15 minutes. Each cron job had a duration of >10m with taking 100% CPU.

I was running 16.0.1 and I believe that the fix is included in that version. If yes, it didn't work for me.

Since we don’t use federation currently, the fix was to delete all those pointless jobs in the DB (14000+) :

DELETE FROM `nextcloud`.`oc_jobs` WHERE `oc_jobs`.`class` LIKE 'OCA\\LookupServerConnector\\BackgroundJobs\\RetryJob';

And removing the offending app

mkdir -p /root/backup_NC_lookup_server_issue/usr/share/nextcloud/apps/lookup_server_connector
mv /usr/share/nextcloud/apps/lookup_server_connector/ /root/backup_NC_lookup_server_issue/usr/share/nextcloud/apps/lookup_server_connector/

Just wanted to report that.