Server: Files are not deleted from S3 (primary)

Created on 6 Apr 2020 · 13Comments · Source: nextcloud/server

How to use GitHub

Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

Steps to reproduce

Set S3 as primary storage
Upload, say, 2000 files into a folder (JPGs here)
Delete that folder, and try to empty trashbin

Expected behaviour

Trashbin should be empty correctly

Actual behaviour

After some time, an error appears (Error while empty trash). Reloading the page shows no more files either in the Files or Trashbin.

But files still on the Object Storage, here OBJECTS and SIZE:

Before these test operations (upload, delete...)

Following commands had been executed (after):

sudo -u testing php occ files:scan test

Starting scan for user 1 out of 1 (test)
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 0       | 0     | 00:00:00     |
+---------+-------+--------------+

sudo -u testing php occ files:cleanup

0 orphaned file cache entries deleted

sudo -u www-data php occ trashbin:cleanup --all-users

Remove deleted files for all users
Remove deleted files for users on backend Database
   test

One user has reported that interface show he is "using" 1,9Gb of storage, but he has NO FILES or FOLDERS at all, either in FILES or TRASHBIN in a production instance.

Server configuration

Operating system: Ubuntu 18.04

Web server: Nginx 17

Database: MariaDB 10.4

PHP version: 7.3

Nextcloud version: (see Nextcloud admin page) 18.0.3

Updated from an older Nextcloud/ownCloud or fresh install: Fresh install

Where did you install Nextcloud from: Official sources

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

Enabled:
  - accessibility: 1.4.0
  - admin_audit: 1.8.0
  - announcementcenter: 3.7.0
  - apporder: 0.9.0
  - cloud_federation_api: 1.1.0
  - dav: 1.14.0
  - external: 3.5.0
  - federatedfilesharing: 1.8.0
  - files: 1.13.1
  - files_accesscontrol: 1.8.1
  - files_automatedtagging: 1.8.2
  - files_pdfviewer: 1.7.0
  - files_rightclick: 0.15.2
  - files_sharing: 1.10.1
  - files_trashbin: 1.8.0
  - files_versions: 1.11.0
  - files_videoplayer: 1.7.0
  - groupfolders: 6.0.3
  - impersonate: 1.5.0
  - logreader: 2.3.0
  - lookup_server_connector: 1.6.0
  - notifications: 2.6.0
  - oauth2: 1.6.0
  - password_policy: 1.8.0
  - privacy: 1.2.0
  - provisioning_api: 1.8.0
  - settings: 1.0.0
  - sharebymail: 1.8.0
  - theming: 1.9.0
  - theming_customcss: 1.5.0
  - twofactor_backupcodes: 1.7.0
  - viewer: 1.2.0
  - workflow_script: 1.3.1
  - workflowengine: 2.0.0

Nextcloud configuration:

Config report

{
    "system": {
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "testing.example.com",
                "autocreate": true,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***",
                "hostname": "10.1.0.2",
                "port": 8080,
                "use_ssl": false,
                "region": "fr-par",
                "use_path_style": true
            }
        },
        "log_type": "file",
        "logfile": "\/var\/log\/nextcloud\/testing.example.com-nextcloud.log",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "testing.example.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "18.0.3.0",
        "overwrite.cli.url": "https:\/\/testing.example.com",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "3306",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "dbdriveroptions": {
            "1009": "\/etc\/ssl\/mysql\/ca-cert.pem",
            "1008": "\/etc\/ssl\/mysql\/client-cert.pem",
            "1007": "\/etc\/ssl\/mysql\/client-key.pem",
            "1014": false
        },
        "installed": true,
        "skeletondirectory": "",
        "default_language": "fr",
        "default_locale": "fr_FR",
        "activity_expire_days": 30,
        "auth.bruteforce.protection.enabled": false,
        "blacklisted_files": [
            ".htaccess",
            "Thumbs.db",
            "thumbs.db"
        ],
        "htaccess.RewriteBase": "\/",
        "integrity.check.disabled": false,
        "knowledgebaseenabled": false,
        "logtimezone": "Europe\/Paris",
        "maintenance": false,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "updatechecker": false,
        "appstoreenabled": false,
        "upgrade.disable-web": true,
        "filelocking.enabled": false,
        "overwriteprotocol": "https",
        "preview_max_scale_factor": 1,
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 2.5,
            "dbindex": 2,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "quota_include_external_storage": false,
        "theme": "",
        "trashbin_retention_obligation": "auto, 7",
        "updater.release.channel": "stable",
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "tls",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "overwritehost": "testing.example.com",
        "preview_max_x": "1280",
        "preview_max_y": "800",
        "jpeg_quality": "70",
        "loglevel": 2,
        "enabledPreviewProviders": [
            "OC\\Preview\\PNG",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\GIF",
            "OC\\Preview\\BMP",
            "OC\\Preview\\XBitmap"
        ],
        "apps_paths": [
            {
                "path": "\/var\/www\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/custom",
                "url": "\/custom_apps",
                "writable": true
            }
        ]
    }
}

Logs are completely empty (we have just fired up a test instance, and test this use case).

0. Needs triage bug object storage

Source

acsfer

👍8

Most helpful comment

I can always confirm this with v19.0.5.
My test instance is completely empty, no files at all, trashbin cleaned, but mc outputs this:

./mc du minio/bucket
2.7GiB

and ./mc ls minio/bucket lists hundreds of files from my different tests.
Some of the files were created more than one month ago in the bucket.

These are clearly not images previews as i have big files in the bucket:

./mc ls minio/bucket
...
[2020-11-24 20:16:54 CET] 115KiB urn:oid:50503
[2020-10-07 10:10:30 CEST] 1.1KiB urn:oid:15082
[2020-11-24 20:38:10 CET]  99KiB urn:oid:55762
[2020-10-07 10:09:26 CEST] 5.5MiB urn:oid:14773
[2020-11-24 20:38:09 CET] 192KiB urn:oid:55750
[2020-10-07 09:59:00 CEST]  26KiB urn:oid:11050
[2020-10-07 10:10:27 CEST]   110B urn:oid:15034
[2020-10-06 11:21:30 CEST] 360KiB urn:oid:883
[2020-11-24 20:21:33 CET] 271KiB urn:oid:54307
[2020-10-07 09:59:52 CEST]  25MiB urn:oid:11158
...

Summary: an empty instance, and a bucket with 2.87GB used and 3685 objects in it.
😮

acsfer on 24 Nov 2020

👍3

All 13 comments

Your issue is likely filelocking. Disable it in the nextcloud config, disable redis filelocking. Restart PHP-FPM and try reproducing this again.

In my case by disabling filelocking all of my issues related to deletion were resolved. I just let the S3 backend handle the filelocking now.