Server: Nextcloud and S3

Created on 25 Oct 2017  ·  63Comments  ·  Source: nextcloud/server

Nextcloud version: 12.0.0.3

Nextcloud is connected to S3. When copying files using aws s3 php sdk in a directory bucket,Nextcloud does not see them. But if you look through the content of S3 Browser, everything is there. If the files to burn through nextcloud, no problem. What could be the problem? Some kind of analogue occ filesystem_external:scan ?

0. Needs triage bug external storage
Source
picture ldapomni
👍9

Most helpful comment

This issue needs to be resolved ASAP, I can't even see any file even when I erase and re.init the whole database. S3 delete readd on NC web ui doesn't work.

picture ctenuun on 8 Apr 2018
👍6

All 63 comments

ps: Premissions is full.
ps2: Directory and files visible if create path with step by step
league\flysystem adapter s3

$fs_s3->createDir("../dir1");
$fs_s3->createDir("../dir1/subdir2");
$fs_s3->createDir("../dir1/subdir2/subdir3/");
$fs_s3->write("../dir1/subdir2/subdir3/test.txt");
all well and nextcloud displays file

if create file as $fs_s3->write("../dir2/subdir5/subdir6/test.txt")
the file is created but nextcloud does not display (s3 browser sees it)

ldapomni picture ldapomni on 25 Oct 2017

So you are uploading files directly to S3 and not through Nextcloud?

If so, in the options of the external storage definition there is a dropdown with different options:

bildschirmfoto vom 2017-10-25 16-28-15

nickvergessen picture nickvergessen on 25 Oct 2017

Options "Check for changes" set.
also php function aws s3 sdk $cont=$fs_s3->ListContents('',true); work fine and return list of items.

when NC refers to the directory in ceph radosgw log shows that reading the contents of the directory is, but for some reason, NC doesn't display them

ldapomni picture ldapomni on 26 Oct 2017

For test: added
file_put_contents ('dir to file test',print_r($object,true),FILE_APPEND); 240 line opendir() function
/apps/file_external/lib/Lib/Storage/AmazonS3.php.
result - directory present. NC doesn't display them.

Array
(
    [Key] => file1.txt
    [LastModified] => 2017-10-26T05:34:39.735Z
    [ETag] => "355884b5c38c11828b51aafb7c543581"
    [Size] => 8
    [StorageClass] => STANDARD
    [Owner] => Array
        (
            [ID] => rados
            [DisplayName] => S3 cloud
        )

)
Array
(
    [Key] => test_20171026093814.txt
    [LastModified] => 2017-10-26T05:38:14.789Z
    [ETag] => "23a398553ab3b06032dc7badcca0eaf4"
    [Size] => 3508
    [StorageClass] => STANDARD
    [Owner] => Array
        (
            [ID] => rados
            [DisplayName] => S3 cloud
        )

)
Array
(
    [Prefix] => dir4/
)
Array
(
    [Prefix] => test1/
)
Array
(
    [Prefix] => test2/
)

dir4 and subdir write external scripts

ldapomni picture ldapomni on 26 Oct 2017

for some reason, the function filetype($path) AmazonS3.php return false on 'dir4'
$this->getConnection()->doesObjectExist($this->bucket,$path.'/') return false, but true for subdir and dir created with NC

NC log:
Debug OC\Files\Cach.. !!! Path 'dir4' is not accessible or present !!!
Why?

ldapomni picture ldapomni on 26 Oct 2017

If use code from league\flysystem AwS3Adapter.php to determine directory - 

 protected function doesDirectoryExist($location)
    {
        // Maybe this isn't an actual key, but a prefix.
        // Do a prefix listing of objects to determine.
        $command = $this->s3Client->getCommand(
            'listObjects',
            [
                'Bucket' => $this->bucket,
                'Prefix' => rtrim($location, '/') . '/',
                'MaxKeys' => 1,
            ]
        );

        try {
            $result = $this->s3Client->execute($command);

            return $result['Contents'] || $result['CommonPrefixes'];
        } catch (S3Exception $e) {
            if ($e->getStatusCode() === 403) {
                return false;
            }

            throw $e;
        }
    }

and change AmazonS3.php function filetype (file/dir)
work is ok.

ldapomni picture ldapomni on 26 Oct 2017

I'm having the exact same issue (NC 12.0.3). I do have the Check for changes: Once every direct access like @nickvergessen mentioned, which makes access much slower, but I do get to see the files/dirs I create straight in S3.

However, a whole tree that I synced (i.e. aws s3 sync ...) from a different bucket goes unnoticed in NC.

  • If I add new files to one of those ignored folders, nothing changes.
  • If I move one of those folders under another one previously created from S3 (and present in NC), still not visible.

    - If I copy files inside one of the invisible folders into a visible one, they show up fine.

@ldapomni I've spotted the filetype() function in file /var/www/nextcloud/apps/files_external/lib/Lib/Storage/AmazonS3.php

288         public function filetype($path) {
289                 $path = $this->normalizePath($path);
290 
291                 if ($this->isRoot($path)) {
292                         return 'dir';
293                 }
294 
295                 try {
296                         if ($this->getConnection()->doesObjectExist($this->bucket, $path)) {
297                                 return 'file';
298                         }
299                         if ($this->getConnection()->doesObjectExist($this->bucket, $path.'/')) {
300                                 return 'dir';
301                         }
302                 } catch (S3Exception $e) {
303                         \OCP\Util::logException('files_external', $e);
304                         return false;
305                 }
306 
307                 return false;
308         }

However, I'm not sure which changes I should made based on your prior comment to test. Also, how to debug this further and gather the needed info?

default50 picture default50 on 19 Nov 2017
👍1

cc @icewind1991

MorrisJobke picture MorrisJobke on 18 Dec 2017

I can't reproduce the problem with master (to be NC13) and amazon S3, when creating a folder and adding a file to it trough the aws console, and running occ files:scan it properly detects the newly added folder and file.

Can anyone experiencing this problem try either the lastest NC13 beta (as always when using beta software: backup) or 12.0.4 which should include the same s3 changes.

icewind1991 picture icewind1991 on 19 Dec 2017

@icewind1991
I have the same
If you copy files to the bucket via aws-cli you cant see them in NC, but if you create folders from S3 console - you can.
https://help.nextcloud.com/t/cant-see-files-copied-to-s3-bucket/26680

```Exception during scan: Invalid argument supplied for foreach()

0 /usr/share/nginx/nextcloud/apps/files_external/lib/Lib/Storage/AmazonS3.php(291): OCA\Files\Command\Scan->exceptionErrorHandler(2, 'Invalid argumen...', '/usr/share/ngin...', 291, Array)

1 /usr/share/nginx/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(97): OCA\Files_External\Lib\Storage\AmazonS3->opendir('')

2 /usr/share/nginx/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php(106): OC\Files\Storage\Wrapper\Wrapper->opendir('')

3 /usr/share/nginx/nextcloud/lib/private/Files/Storage/Wrapper/Encoding.php(158): OC\Files\Storage\Wrapper\Availability->opendir('')

4 /usr/share/nginx/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(97): OC\Files\Storage\Wrapper\Encoding->opendir('')

5 /usr/share/nginx/nextcloud/lib/private/Files/Cache/Scanner.php(359): OC\Files\Storage\Wrapper\Wrapper->opendir('')

6 /usr/share/nginx/nextcloud/lib/private/Files/Cache/Scanner.php(410): OC\Files\Cache\Scanner->getNewChildren('')

7 /usr/share/nginx/nextcloud/lib/private/Files/Cache/Scanner.php(390): OC\Files\Cache\Scanner->handleChildren('', true, 3, '56419', true, 0)

8 /usr/share/nginx/nextcloud/lib/private/Files/Cache/Scanner.php(324): OC\Files\Cache\Scanner->scanChildren('', true, 3, '56419', true)

9 /usr/share/nginx/nextcloud/lib/private/Files/Utils/Scanner.php(242): OC\Files\Cache\Scanner->scan('', true, 3)

10 /usr/share/nginx/nextcloud/apps/files/lib/Command/Scan.php(161): OC\Files\Utils\Scanner->scan('/nextcloudadmin')

11 /usr/share/nginx/nextcloud/apps/files/lib/Command/Scan.php(234): OCA\Files\Command\Scan->scanFiles('nextcloudadmin', '/nextcloudadmin', false, Object(Symfony\Component\Console\Output\ConsoleOutput), false)

12 /usr/share/nginx/nextcloud/3rdparty/symfony/console/Command/Command.php(256): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

13 /usr/share/nginx/nextcloud/core/Command/Base.php(161): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

14 /usr/share/nginx/nextcloud/3rdparty/symfony/console/Application.php(818): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

15 /usr/share/nginx/nextcloud/3rdparty/symfony/console/Application.php(186): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

16 /usr/share/nginx/nextcloud/3rdparty/symfony/console/Application.php(117): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

17 /usr/share/nginx/nextcloud/lib/private/Console/Application.php(170): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

18 /usr/share/nginx/nextcloud/console.php(100): OC\Console\Application->run()

19 /usr/share/nginx/nextcloud/occ(11): require_once('/usr/share/ngin...')

20 {main}

```
after occ files:scan

sporykhin picture sporykhin on 2 Feb 2018
👍2

@icewind1991 Any reaction?

sporykhin picture sporykhin on 8 Feb 2018

After update to NC13, problem still exists.
@sporykhin some error after occ files:scan.

ldapomni picture ldapomni on 14 Feb 2018

@nickvergessen Any chance to find @icewind1991 here?

sporykhin picture sporykhin on 20 Feb 2018

Same problem here, I work normally with S3 folder/files created through NC or directly via AWS S3 GUI, but not with files uploaded with the aws cli. I believe it's somehow connected to the error I keep seeing in the log:

image

velimirg picture velimirg on 21 Feb 2018

I have a very similar error when trying to delete a folder from an S3 bucket within NC.

  • Invalid argument supplied for foreach() at /snap/nextcloud/5132/htdocs/apps/files_external/lib/Lib/Storage/AmazonS3.php#291
  • [Delete] is missing and is a required parameter

Nextcloud 12.0.5


"Exception": "InvalidArgumentException", "Message": "Found 1 error while validating the input provided for the DeleteObjects operation:\\n[Delete] is missing and is a required parameter",
#0 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/Middleware.php(79): AwsApiValidator->validate('DeleteObjects', Object(AwsApiStructureShape), Array)
#1 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php(421): AwsMiddleware::Aws{closure}(*** sensitive parameters replaced ***)
#2 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php(444): AwsS3S3Client::AwsS3{closure}(*** sensitive parameters replaced ***)
#3 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php(378): AwsS3S3Client::AwsS3{closure}(*** sensitive parameters replaced ***)
#4 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/S3Client.php(397): AwsS3S3Client::AwsS3{closure}(*** sensitive parameters replaced ***)
#5 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/Middleware.php(55): AwsS3S3Client::AwsS3{closure}(*** sensitive parameters replaced ***)
#6 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/S3/SSECMiddleware.php(59): AwsMiddleware::Aws{closure}(*** sensitive parameters replaced ***)
#7 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/IdempotencyTokenMiddleware.php(77): AwsS3SSECMiddleware->__invoke(Object(AwsCommand), NULL)
#8 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php(65): AwsIdempotencyTokenMiddleware->__invoke(Object(AwsCommand))
#9 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php(59): AwsAwsClient->executeAsync(Object(AwsCommand))
#10 /snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/AwsClientTrait.php(78): AwsAwsClient->execute(Object(AwsCommand))
#11 /snap/nextcloud/5132/htdocs/apps/files_external/lib/Lib/Storage/AmazonS3.php(255): AwsAwsClient->__call('deleteObjects', Array)
#12 /snap/nextcloud/5132/htdocs/apps/files_external/lib/Lib/Storage/AmazonS3.php(224): OCAFiles_ExternalLibStorageAmazonS3->batchDelete('2018/02')
#13 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(87): OCAFiles_ExternalLibStorageAmazonS3->rmdir('2018/02')
#14 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/PermissionsMask.php(106): OCFilesStorageWrapperWrapper->rmdir('2018/02')
#15 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(87): OCFilesStorageWrapperPermissionsMask->rmdir('2018/02')
#16 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Availability.php(95): OCFilesStorageWrapperWrapper->rmdir('2018/02')
#17 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Encryption.php(298): OCFilesStorageWrapperAvailability->rmdir('2018/02')
#18 [internal function]: OCFilesStorageWrapperEncryption->rmdir('2018/02')
#19 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Storage.php(258): call_user_func_array(Array, Array)
#20 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Storage.php(189): OCAFiles_TrashbinStorage->doDelete('2018/02', 'rmdir')
#21 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Encryption.php(617): OCAFiles_TrashbinStorage->rmdir('2018/02')
#22 /snap/nextcloud/5132/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(569): OCFilesStorageWrapperEncryption->moveFromStorage(Object(OCAFiles_TrashbinStorage), '2018/02', 'files_trashbin/...')
#23 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Trashbin.php(251): OCFilesStorageWrapperWrapper->moveFromStorage(Object(OCAFiles_TrashbinStorage), '2018/02', 'files_trashbin/...')
#24 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Storage.php(247): OCAFiles_TrashbinTrashbin::move2trash('AWS/2018/02', false)
#25 /snap/nextcloud/5132/htdocs/apps/files_trashbin/lib/Storage.php(189): OCAFiles_TrashbinStorage->doDelete('2018/02', 'rmdir')
#26 /snap/nextcloud/5132/htdocs/lib/private/Files/View.php(1159): OCAFiles_TrashbinStorage->rmdir('2018/02')
#27 /snap/nextcloud/5132/htdocs/lib/private/Files/View.php(348): OCFilesView->basicOperation('rmdir', '/AWS/2018/02', Array)
#28 /snap/nextcloud/5132/htdocs/apps/dav/lib/Connector/Sabre/Directory.php(303): OCFilesView->rmdir('/AWS/2018/02')
#29 /snap/nextcloud/5132/htdocs/3rdparty/sabre/dav/lib/DAV/Tree.php(179): OCADAVConnectorSabreDirectory->delete()
#30 /snap/nextcloud/5132/htdocs/3rdparty/sabre/dav/lib/DAV/CorePlugin.php(287): SabreDAVTree->delete('AWS/2018/02')
#31 [internal function]: SabreDAVCorePlugin->httpDelete(Object(SabreHTTPRequest), Object(SabreHTTPResponse))
#32 /snap/nextcloud/5132/htdocs/3rdparty/sabre/event/lib/EventEmitterTrait.php(105): call_user_func_array(Array, Array)
#33 /snap/nextcloud/5132/htdocs/3rdparty/sabre/dav/lib/DAV/Server.php(479): SabreEventEventEmitter->emit('method:DELETE', Array)
#34 /snap/nextcloud/5132/htdocs/3rdparty/sabre/dav/lib/DAV/Server.php(254): SabreDAVServer->invokeMethod(Object(SabreHTTPRequest), Object(SabreHTTPResponse))
#35 /snap/nextcloud/5132/htdocs/apps/dav/appinfo/v1/webdav.php(76): SabreDAVServer->exec()
#36 /snap/nextcloud/5132/htdocs/remote.php(162): require_once('/snap/nextcloud...')
#37 {main}
"File": "/snap/nextcloud/5132/htdocs/3rdparty/aws/aws-sdk-php/src/Api/Validator.php", "Line": 65
kruffin picture kruffin on 9 Mar 2018

13.0.1
Still not working

sporykhin picture sporykhin on 23 Mar 2018
👍3

@kruffin I have the same

ldapomni picture ldapomni on 6 Apr 2018

This issue needs to be resolved ASAP, I can't even see any file even when I erase and re.init the whole database. S3 delete readd on NC web ui doesn't work.

ctenuun picture ctenuun on 8 Apr 2018
👍6

@nickvergessen
please help with this 😄

sporykhin picture sporykhin on 24 Apr 2018
👍3

This is sad, but 13.0.2 - still not working
Externally uploaded files to the bucket cant be viewed in NC interface

sporykhin picture sporykhin on 31 May 2018
👍3

I have the same problem. Snap NextCloud 13.0.2 on Ubuntu 16.04
Also if I use the command: sudo nextcloud.occ files:scan lays147 --all
(obs.: If the content below should be in another issue, pls let me know. But since is related to S3 use, I think this should work. )
I have the following exception:
`Exception during scan: Invalid argument supplied for foreach()

0 /snap/nextcloud/6916/htdocs/apps/files_external/lib/Lib/Storage/AmazonS3.php(295): OCA\Files\Command\Scan->exceptionErrorHandler(2, 'Invalid argumen...', '/snap/nextcloud...', 295, Array)

1 /snap/nextcloud/6916/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(101): OCA\Files_External\Lib\Storage\AmazonS3->opendir('')

2 /snap/nextcloud/6916/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(101): OC\Files\Storage\Wrapper\Wrapper->opendir('')

3 /snap/nextcloud/6916/htdocs/lib/private/Files/Storage/Wrapper/Availability.php(109): OC\Files\Storage\Wrapper\Wrapper->opendir('')

4 /snap/nextcloud/6916/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(101): OC\Files\Storage\Wrapper\Availability->opendir('')

5 /snap/nextcloud/6916/htdocs/lib/private/Files/Storage/Wrapper/Wrapper.php(101): OC\Files\Sorage\Wrapper\Wrapper->opendir('')

6 /snap/nextcloud/6916/htdocs/lib/private/Files/Cache/Scanner.php(372): OC\Files\Storage\Wraper\Wrapper->opendir('')

7 /snap/nextcloud/6916/htdocs/lib/private/Files/Cache/Scanner.php(423): OC\Files\Cache\Scannr->getNewChildren('')

8 /snap/nextcloud/6916/htdocs/lib/private/Files/Cache/Scanner.php(403): OC\Files\Cache\Scannr->handleChildren('', true, 3, '146', true, 0)

9 /snap/nextcloud/6916/htdocs/lib/private/Files/Cache/Scanner.php(337): OC\Files\Cache\Scannr->scanChildren('', true, 3, '146', true)

10 /snap/nextcloud/6916/htdocs/lib/private/Files/Utils/Scanner.php(244): OC\Files\Cache\Scaner->scan('', true, 3)

11 /snap/nextcloud/6916/htdocs/apps/files/lib/Command/Scan.php(161): OC\Files\Utils\Scanner-scan('/lays147')

12 /snap/nextcloud/6916/htdocs/apps/files/lib/Command/Scan.php(234): OCA\Files\Command\Scan-scanFiles('lays147', '/lays147', false, Object(Symfony\Component\Console\Output\ConsoleOutput, false)

13 /snap/nextcloud/6916/htdocs/3rdparty/symfony/console/Command/Command.php(264): OCA\Files\ommand\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Componnt\Console\Output\ConsoleOutput))

14 /snap/nextcloud/6916/htdocs/core/Command/Base.php(161): Symfony\Component\Console\CommandCommand->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Consle\Output\ConsoleOutput))

15 /snap/nextcloud/6916/htdocs/3rdparty/symfony/console/Application.php(874): OC\Core\Comman\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Consol\Output\ConsoleOutput))

16 /snap/nextcloud/6916/htdocs/3rdparty/symfony/console/Application.php(228): Symfony\Componnt\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\ComponentConsole\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

17 /snap/nextcloud/6916/htdocs/3rdparty/symfony/console/Application.php(130): Symfony\Componnt\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfoy\Component\Console\Output\ConsoleOutput))

18 /snap/nextcloud/6916/htdocs/lib/private/Console/Application.php(173): Symfony\Component\Cnsole\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Compnent\Console\Output\ConsoleOutput))

19 /snap/nextcloud/6916/htdocs/console.php(90): OC\Console\Application->run()

20 /snap/nextcloud/6916/htdocs/occ(11): require_once('/snap/nextcloud...')

21 {main}`

Any workarounds for this?

lays147 picture lays147 on 3 Jun 2018

I am facing the same issue.
Version: Nextcloud 13.0.4

The only workaround I found is to just create the directory via Nextcloud UI and the files will then appear.

Eg.

S3            |     Nextcloud UI
dir1/a.txt    |     dir1/a.txt
dir1/b.txt    |     dir1/b.txt
dir2/c.txt    |
dir2/d.txt    |

Create the folder dir2 via the Nextcloud UI,

S3            |     Nextcloud UI
dir1/a.txt    |     dir1/a.txt
dir1/b.txt    |     dir1/b.txt
dir2/c.txt    |     dir2/c.txt
dir2/d.txt    |     dir2/d.txt

One weird thing I noticed is that my logs are not showing any error at all, no warning, no error.

Also, occ:files:scan --all ran successfully and didn't show any error

yashmehrotra picture yashmehrotra on 24 Jun 2018
👍1

The workaround made by @yashmehrotra worked for me. Thanks

lays147 picture lays147 on 2 Jul 2018

Same issue. Nextcloud 13.0.5.2. PHP 7.0.30. I'm using the S3 service from Digital Ocean, called "Spaces."

The workaround mentioned by @yashmehrotra did not work for me. I can see the folder I "created" through the Nextcloud UI, but nothing under that folder, and occ files:scan generates the errors in the log.

Note: I used s3cmd to create a top level folder called "books," many folders below that, and many files in each folder. In the log, the errors look like this:

{"reqId":"...","level":0,"time":"2018-07-25T11:32:24-07:00","remoteAddr":"...","user":"...","app":"OC\Files\Cache\Scanner","method":"PROPFIND","url":"/remote.php/webdav/archive/books","message":"!!! Path 'books/bbc4' is not accessible or present !!!","userAgent":"...","version":"13.0.5.2"}

There is no such error for the "books" folder. Only the subfolders.

sbw picture sbw on 25 Jul 2018

Same issue. A files:scan does not help in my case.

This same issues exists with ownCloud, however Pydio seems to handle the situation properly.

This likely stems from the fact that S3 stores everything as objects and "paths" are only implicit in that the object names have forward slashes ("/") in them. Therefore, some trickery is needed to emulate standard directory hierarchy. NC is obviously aware of this, but it sounds like some small piece of the puzzle is missing.

piranhaphish picture piranhaphish on 10 Aug 2018

Seeing the same issue after a fresh install of NC 14.0 on Docker

sjwoodard picture sjwoodard on 8 Sep 2018

Could it be related to versioning being enabled on S3? I can't reproduce the issue on buckets without versioning. Ignore that comment, I reproduced it with and without versioning enabled.

sjwoodard picture sjwoodard on 8 Sep 2018

The bucket is created automatically by nextcloud when connecting to S3, so i think that uses the default setup for a bucket...

lays147 picture lays147 on 8 Sep 2018

Thanks for suggesting versioning. For what it's worth, I'm using S3 storage provided by DigitalOcean, which does not yet support versioning. I have not reproduced the problem since my last comment. And I have not reproduced the problem on DigitalOcean and Amazon simultaneously to see if there is any difference. I hope to get back to that soon. It'd also be interesting to reproduce it with Storj or some other S3 service providers.

sbw picture sbw on 8 Sep 2018

I have found a workaround when all else fails...

I wrote a C# script to copy the S3 directory structure (i.e. not download the files) to my computer. Then I used the Windows NC client (2.3.3) to sync the folder structure onto the NC server. Once you add the folders, the client will start showing the files, but it will also start syncing the files to you computer. To avoid downloading all the files, I added extensions like *.jpg, *.mpg, etc. to the ignored files list in the NC client.

Here's the C# script to sync the S3 directory structure to your computer:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Amazon;
using Amazon.S3.Model;

namespace SyncS3DirectoryStructure
{
    class Program
    {
        static void Main(string[] args)
        {
            // CHANGE THESE
            var awsId = "AWS_ID";
            var awsKey = "AWS_SECRET";
            var bucket = "BUCKET_NAME";
            var region = RegionEndpoint.USEast1;
            var outDir = @"c:\LOCAL_DIR";

            using (var s3 = new Amazon.S3.AmazonS3Client(awsId, awsKey, region))
            {
                var request = new ListObjectsRequest
                {
                    BucketName = bucket
                };
                var dirs = new List<string>();
                do
                {
                    var response = s3.ListObjectsAsync(request).Result;
                    var folders = response.S3Objects;
                    folders.ToList().ForEach(x => {
                        if (!x.Key.Contains('/'))
                        {
                            return;
                        }
                        var dir = x.Key.Substring(0, x.Key.LastIndexOf('/') + 1);
                        if (!dirs.Contains(dir))
                        {
                            dirs.Add(dir);
                            Console.WriteLine("Found new dir: " + dir);
                        }
                    });
                    if (response.IsTruncated)
                    {
                        request.Marker = response.NextMarker;
                    }
                    else
                    {
                        request = null;
                    }
                } while (request != null);

                foreach (var newDir in dirs)
                {
                    var fullDir = Path.Combine(outDir, newDir.Replace('/', '\\'));
                    Console.WriteLine("Creating " + fullDir);
                    Directory.CreateDirectory(fullDir);
                }
            }
        }
    }
}
sjwoodard picture sjwoodard on 9 Sep 2018

I have the same issue here.

Files uploaded via AWS CLI are not appearing on Nextcloud.

All permissions are set correctly, not sure how to debug this too.

MickaelTH picture MickaelTH on 27 Sep 2018

Hey guys

Ive found the only one way to implement this as soon as functionality is
completely broken :
S3fs mount to NC host
"Local" external mount
Occ files:scan by Cron job

Ugly workaround but it works

чт, 27 сент. 2018 г., 14:58 Mickael notifications@github.com:

I have the same issue here.

Files uploaded via AWS CLI are not appearing on Nextcloud.

All permissions are set correctly, not sure how to debug this too.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/server/issues/6954#issuecomment-425063278,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABZKEOuz7iV35q3-uJq2SpaF87US3S-yks5ufL16gaJpZM4QF5QF
.

sporykhin picture sporykhin on 28 Sep 2018

Has anyone moved on to version 14 and still have this issue?

dmcwhinnie picture dmcwhinnie on 14 Dec 2018

Has anyone moved on to version 14 and still have this issue?

I am using nextcloud 15 and the issue still exists.

aeltawela picture aeltawela on 1 Jan 2019
👍1

You could try https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3 (patch file). Like @ldapomni discovered the way a directory is validated is wrong.

kesselb picture kesselb on 1 Jan 2019
👍3

I'm still seeing this issue with 14.0.4.

sbw picture sbw on 2 Jan 2019

You could try https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3 (patch file). Like @ldapomni discovered the way a directory is validated is wrong.

i am using 15.02 and this fixed it for me.
to reproduce the issue i used aws-cli

when uploading a file using below command you will see file in nextcloud:
aws s3 cp /path/to/file s3://bucket-name

but when uploading like below (such as what occurs when uploading nested folders) you will not see the file in nextcloud:
aws s3 cp /path/to/file s3://bucket-name/folder/file

i replaced my "AmazonS3.php" at "nextcloud/apps/files_external/lib/Lib/Storage/AmazonS3.php" with the patched file below:
https://github.com/nextcloud/server/blob/759714eadec507dfa7893a8c05632c5262117025/apps/files_external/lib/Lib/Storage/AmazonS3.php

i then ran the occ scan command: sudo -u apache php occ files:scan --all and was very happy to see my files start appearing in nextcloud.

(edited to add nextcloud version)

itzjustinn picture itzjustinn on 21 Jan 2019
🎉3 👍1

Can this be pulled into the next version plz?

dimm0 picture dimm0 on 4 Feb 2019
👍1

You could try https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3 (patch file).

I applied that a few days ago to 15.0.2, and it solved the problem for me. S3 external storages remain excruciatingly slow (at least using Digital Ocean Spaces), but new stuff does show up as expected after the patch.

sbw picture sbw on 4 Feb 2019
👍1

I think it now needs a button to manually trigger files list reload... Updating the list on every view can be taking half an hour for large buckets, and turning off refresh on view leaves new files not visible

dimm0 picture dimm0 on 4 Feb 2019
👍1

Hi,

i agree that a sync button should be usefull ... i used S3 upload API for init reason ... but after i want only use NC.

yroffin picture yroffin on 30 Mar 2019

@kesselb Many people are saying that this patch(https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3) fixed the problem, will you be merging this into master ?

yashmehrotra picture yashmehrotra on 30 Mar 2019

@yroffin If you apply my patch https://docs.nextcloud.com/server/stable/admin_manual/configuration_server/occ_command.html#scan should work.

@yashmehrotra Yes.

kesselb picture kesselb on 30 Mar 2019
👍2

I'm trying to work with an S3 storage through nextcloud, and both options it provides currently don't work well:
If I use the "scan always" option, it's constantly scanning the S3 storage on every folder browse, and only displays the folder size when I come after some time. Then it's always Pending. The new directories appear, but I suppose it's always busy scanning... Especially when there are lots of folders
If I use "scan never" option, obviously the new folders don't appear.

So the solution might be either manual rescan, or tunable period of rescan, like 10 mins or half an hour, and no full rescans between those.

dimm0 picture dimm0 on 11 Apr 2019

Pick "scan never" and setup https://docs.nextcloud.com/server/stable/admin_manual/configuration_server/occ_command.html#scan as cron could work for your use case.

kesselb picture kesselb on 11 Apr 2019

Would that be a recommended option then? Most people who use S3 have a bunch of files in it, that's the reason to use it

dimm0 picture dimm0 on 11 Apr 2019

Would that be a recommended option then

No. It could work for your use case. I think the cleanest way is to write the files to s3 through nextcloud. In this case you can use scan never.

kesselb picture kesselb on 12 Apr 2019

For some reason I did not have this issue in NC 15.0.5. However, once I upgraded to 15.0.7 I began to experience this issue. (I cannot say about 15.0.6)
This issue was caused by:

  • Files I copied locally up to S3 bucket.
  • Files I copied from one S3 bucket to another S3 bucket.

I was unable to see these files from NC. I applied patch https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3 (Here is the file: https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3.patch) and it did correct this issue.

Thank you!

wanchic picture wanchic on 14 Apr 2019

Like wanchic's reports, my Nextcloud has again lost access to my S3-compatible DigitalOcean Spaces external storage. I had installed that patch some time ago, and it worked. Since then, I have upgraded to 15.0.7, and my S3 external storage is again empty. Sorry, I can't say which version broke it. I'll try applying the patch again.

I had understood the patch was adopted into the released Nextcloud. I wonder whether it got backed out somehow? I'll try to look through the source code history if I have time.

Off topic: S3 external storages are basically unusable, even when they're working. They're just too slow. I tried setting "Check for Changes: Never," but it was still too slow to use. I'm no Nextcloud expert, but I have the impression Nextcloud keeps a database of metadata for known files, updated by "occ scan all." So my expectation is that Nextcloud will quickly display information from that database, whether or not it "checks for changes" in the background, even for S3 external storages. It appears instead it's crawling the directory every time I load the web page. That's a serious flaw that makes external storages completely unusable for me. I should search for that as a bug report and report it if I can't find anything.

sbw picture sbw on 14 Apr 2019

I had understood the patch was adopted into the released Nextcloud. I wonder whether it got backed out somehow? I'll try to look through the source code history if I have time.

Not yet :( There is a open pull request but i was a bit late for Nextcloud 16 code freeze. It might be classified as bugfix and backported.

kesselb picture kesselb on 14 Apr 2019

Off topic: S3 external storages are basically unusable, even when they're working. They're just too slow.

I had the same issue with S3 DigitalOcean Spaces & Nextcloud, so to make Nextcloud efficient again, I had to switch to DO Block Storage.

HLFH picture HLFH on 14 Apr 2019

Unfortunately, I don't have a choice in the platform I use. Two weeks ago my client was mandated to use Heroku-Shield in order to meet HIPPA Compliance. Since Heroku uses a ephemeral disk platform, S3 was our only storage option that met HIPPA Compliancy for our client.

wanchic picture wanchic on 15 Apr 2019
👍1

@kesselb

I think the cleanest way is to write the files to s3 through nextcloud. In this case you can use scan never.

Tried copying relatively large dirs from local disk to S3 with nextcloud. Half of folders are not visible, although copied fine. Need to enable rescan again.

So it's not a solution...

dimm0 picture dimm0 on 16 Apr 2019

@dimm0 This might be something for another issue. The case here with the directory support will be fixed soon.

@sbw Sounds good to me. Would you mind to create a new enhancement issue for this? (I think it should be like that maybe something broken).

kesselb picture kesselb on 16 Apr 2019

I applied patch https://github.com/nextcloud/server/compare/bugfix/6954/scan-external-s3 and Fixed my Issue. I Running Nextcloud 15.0.7

Thank you

JarbyDev picture JarbyDev on 13 May 2019

still broken in 16.0.1 it seems but the patch still fixes it :(

itzjustinn picture itzjustinn on 21 May 2019

Same here :

  • Folder created by Scaleway Object Storage explorer are seen by NC
  • Folder created by mc (Minio Client) can't been seen by NC but are browsable by SC OB explorer or s3fs
Loxeno picture Loxeno on 22 May 2019

I can confirm the patch mentioned above resolved the issue for me. I'm running nextcloud 16.0.1 and using Digital Ocean spaces for external storage, with folders/files populated using rclone directly to DO spaces. I basically just downloaded the modified AmazonS3.php file into apps/files_external/lib/Lib/Storage/ and then re-ran php occ files:scan <user> --path=/<user>/files/<name of the external folder>. I think php occ files:scan --all might have worked as well though. Before doing that I had the same issue as many others, where none of the directories populated by rclone were showing up in nextcloud.

alexklibisz picture alexklibisz on 27 May 2019
❤1

@kesselb wrote, "@sbw Sounds good to me. Would you mind to create a new enhancement issue for this? (I think it should be like that maybe something broken)."

Sorry, I can't tell whether that pertains to the original issue (S3 external storage stuff never shows up in Nextcloud) or my off-topic observation that "S3 external storages are basically unusable, even when they're working. They're just too slow."

I hope it's the latter. I hope you're suggesting I create an "enhancement issue" to make S3 external storage fast enough to be usable. If that's what you're suggesting, I'll do that when I have time. (It's not an enhancement, really, but a defect, in my opinion: I believe Nextcloud is meant to immediately display the information it already has from a previous scan. I don't know what Nextcloud is doing instead, but whatever it is, it can't be the intended behavior.)

sbw picture sbw on 30 May 2019
👍1

hey @alexklibisz i was trying to follow what you did but I couldn't modify or download updated AmazonS3.php due to read-only file system. How did you bypass it?

hasangokdag picture hasangokdag on 22 Jun 2019

S3 external storages are basically unusable, even when they're working. They're just too slow.

@sbw this one.

hey @alexklibisz i was trying to follow what you did but I couldn't modify or download updated AmazonS3.php due to read-only file system. How did you bypass it?

@hasangokdag If you are using the snap package applying this patch is not possible.

kesselb picture kesselb on 22 Jun 2019
😕2

I am running nextcloud in a docker container, so I have root access inside
of it by default.

On Sat, Jun 22, 2019, 06:42 Daniel Kesselberg notifications@github.com
wrote:

S3 external storages are basically unusable, even when they're working.
They're just too slow.

@sbw https://github.com/sbw this one.

hey @alexklibisz https://github.com/alexklibisz i was trying to follow
what you did but I couldn't modify or download updated AmazonS3.php due to
read-only file system. How did you bypass it?

@hasangokdag https://github.com/hasangokdag If you are using the snap
package applying this patch is not possible.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/server/issues/6954?email_source=notifications&email_token=AB5E27FS73YARJP7HA6SH2LP3X6Z7A5CNFSM4EAXSQC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYKGWDA#issuecomment-504654604,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB5E27DY7ANRGFACFPB4YRTP3X6Z7ANCNFSM4EAXSQCQ
.

alexklibisz picture alexklibisz on 22 Jun 2019

For those like me using snap and impatient, you can upgrade to Nextcloud 17:

sudo snap refresh nextcloud --channel=17/edge
sudo snap disable nextcloud && sudo snap enable nextcloud
crypdick picture crypdick on 31 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

rullzer picture rullzer  ·  3Comments
mfechner picture mfechner  ·  3Comments
ThomasLeister picture ThomasLeister  ·  3Comments
e-alfred picture e-alfred  ·  3Comments
MariusBluem picture MariusBluem  ·  3Comments