Server: Folders/Files of external storage aren´t appearing in Nextcloud web/client

[fpPierre]

Good Morning everybody,

I do have a issue, with a integrated external storage of my Nextcloud environment. The storage was integrated at the admin page by using ftp.
Thus it isn´t related to a user but to the system.

The configuration of this storage is set to encrypted and without rescan (in the beginning it was activated to check at every access) the external storage.

All data (folders as well as files) on that storage was upload by nextcloud (no direct uploads to the ftp storage).

Suddenly the contents of that storage disappears (not all at the same time but in differnet steps until the folder became empty) on nextcloud web and the clients removed the files from lokal sync folders.

To reintegrate the "lost" data I started a "files:scan --all" but except a couple of short appearings of some folders, the main folder remains as an empty one.

I found out that suspiciously I can browse into all folders by typing in the directory within the url. This is also possible for the files.
After I done this for a particular file or folder, it appears in the filebrowser of nextcloud web as well as in the clients.

While I was doing that I monitored the database and the servers` data folder but wasn´t able to encounter differences.

Unfortunately by somehow the files disappear randomly hours later. Maybe by some sync tasks of further processes.

Under the bottom I do have the question what I´m able to do for a complete reintegration including all folders and files without losing the encryption keys to get access to my data.

Environment:
VM on CentOS HOST with SolusVM

VM resources:
OS - Ubuntu 16.04.1 LTS
CPU - 14 cores (have to look for the particular cpu)
memory - 23,54 GB working memory
swp - 16GB
space / disc - 3,1tb ssd RAID 5
PHP 7.08
MYSQL 5.5.37

Nextcloud: 10.0.2 (9.1.2.2)

encountered log entries:
fseek() - stream does not seeking
and
stat() ssl handshake timed out / stat failed for ftps<- the integration is with ftp and not with sftp

Cheers,
Pierre

[fpPierre]

additional information:
https://help.nextcloud.com/t/folders-files-of-external-storage-aren-t-appearing-in-nextcloud-web-client/5961/7

[fpPierre]

Is there any possibility to copy the data onto the server or another ftp storage and integrate it including the encryption keys?
I would like to exclude the current FTP drive as the source of this issue.

[fpPierre]

not sure whether this issue from some notification cronjob collaborates with my problem. Maybe it avoids that a required cronjob for the file listing will not be started?:

Error while running background job (class: OCA\Activity\BackgroundJob\EmailNotification, arguments: ): {"Exception":"Doctrine\DBAL\Exception\SyntaxErrorException","Message":"An exception occurred while executing 'DELETE FROM oc_activity_mq WHERE amq_timestamp <= ? AND amq_affecteduser IN ()' with params [1481368169]:\n\nSQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')' at line 1","Code":0,"Trace":"#0 \/var\/www\/nextcloud\/3rdparty\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/DBALException.php(116): Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\DBAL\Driver\PDOException))\n#1 \/var\/www\/nextcloud\/3rdparty\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/Statement.php(174): Doctrine\DBAL\DBALException::driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Driver\PDOException), 'DELETE FROM `oc...', Array)\n#2 \/var\/www\/nextcloud\/apps\/activity\/lib\/MailQueueHandler.php(293): Doctrine\DBAL\Statement->execute(Array)\n#3 \/var\/www\/nextcloud\/apps\/activity\/lib\/BackgroundJob\/EmailNotification.php(145): OCA\Activity\MailQueueHandler->deleteSentItems(Array, 1481368169)\n#4 \/var\/www\/nextcloud\/apps\/activity\/lib\/BackgroundJob\/EmailNotification.php(99): OCA\Activity\BackgroundJob\EmailNotification->runStep(25, 1481368169)\n#5 \/var\/www\/nextcloud\/lib\/private\/BackgroundJob\/Job.php(53): OCA\Activity\BackgroundJob\EmailNotification->run(NULL)\n#6 \/var\/www\/nextcloud\/lib\/private\/BackgroundJob\/TimedJob.php(54): OC\BackgroundJob\Job->execute(Object(OC\BackgroundJob\JobList), Object(OC\Log))\n#7 \/var\/www\/nextcloud\/cron.php(145): OC\BackgroundJob\TimedJob->execute(Object(OC\BackgroundJob\JobList), Object(OC\Log))\n#8 {main}","File":"\/var\/www\/nextcloud\/3rdparty\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/Driver\/AbstractMySQLDriver.php","Line":90}

[LukasReschke]

cc @nickvergessen

[fpPierre]

In addition to the main issue.

It should avoid that files from clients desktop folders (desktop client) will removed as far as such issues occurs.
The disappeared files from the ftp storage aren´t marked as deleted files/folders within the databases and not at the nextcloud trash (of course) but why will this files removed from lokal desktop nextcloud folders as far as there isn´t any explicit remove/delete instruction within the database?

To avoid data loss it should arise a a conflict file / folder on desktop folder or something similar to confirm the removals

[nickvergessen]

The activity thing is fixed in https://github.com/nextcloud/activity/pull/94

[Yomark1]

I also have this problem, but not with FTP, but with SMB(password saved in database, check every time on direct access).
What solves it temporary, is to just place a new file in the root of the share. The rest also reappears then after a refresh in the browser.
But after a while other(not always the same files) files are missing again. I see no relevant errors for this in the logs.
(Using NextCloud 11, SMB share is Windows 2012R2 server)

[nickvergessen]

cc @icewind1991

[Xuanwo]

I do have this problem too, in S3.

[nathank1989]

I, too, still have this issue. I have a link to an external (self-hosted) FTP server, and the folder appear, but files do not. I upload files to the FTP folder, and try to refresh nextcloud and nothing happens. The reverse, uploading in NC to the FTP works as expected, the files shows in the FTP folder immediately.

[electrointellect]

I have found a few workarounds for this problem that work to varying degrees.

First if you have to use FTP and do what I was doing which is drop large amounts of data in one large burst rather then slowly over time is to put all the data you want on the FTP server. Then login to your account that has the external FTP shared to it. Then login to the admin account in a different window (you may have to make use of private browser or both chrome and firefox to login to two at once). Remove the external FTP storage, then in the account it was shared with refresh the page so the external storage disappears. Now add back the external FTP storage like it was before and go back to the account it is again shared with and refresh. When you click on the shared folder it should refresh with the new files and folders. This is a pain to do all the time but in large bursts from time to time is tolerable atleast till they fix it (if they fix it).

Second if you are willing to upload to your external FTP folder not via something like filezilla but by syncing an external folder with your Nextcloud desktop client and let it put files (it tends to be slower then other methods sometimes.) then they will always show up once added. I mixed the first method to put most the data then this method to add the new things gradually and it works good enough other then the desktop client uses 20-50% of my CPU during folder rescans.

Third if you dislike the desktop client you can upload the files via the Nextcloud web GUI and they always show up afterwards. It tends to be slower then some methods and any time your connection cuts of for a second you have to start over. This method works good if your dropping a single large file at a time or many small files. Just dropping a whole massive folder did not work so well in my experience.

Fourth which is the best method I have found so far. If you are able to install OpenSSH on your server then I have found that often times the files you add directly will show up on Nextcloud. You add external OpenSSH through the protocol of SFTP, SSH File Transfer Protocol (not to be mistaken with FTPS which is FTP with SSL). Then you simply add accounts directly to your linux server and can login to them via SFTP. Filezilla also works with SFTP if you put sftp:// before your server IP/URL. If you connect to your server via windows you can always download a program called SFTP Net Drive to map a drive letter to your SFTP server which is convenient. Sometimes it is still slow to show up on Nextcloud but it often times does on its own when you click on the folders via the web GUI or your mobile device app.

The first method works most times to force a refresh manually. I have found that if you do not login to the account its shared on and refresh the page before and after removing the external storage it just assumes nothing changed. You must refresh the page so the shared external folder disappears then when you add it again it sees it as a new folder and refreshes. There may be a cron task you can setup that would refresh the SFTP automatically on a timed interval in the background but I have not found one that works yet. SFTP works alot better with Nextcloud then FTP in my testing of all the external storage methods. Samba, FTP, WebDAV, etc did all not work as good as SFTP. My external storage server shared via SFTP is running FreeNAS Corral (aka FreeNAS10) and my main Nextcloud server is running Ubuntu Server 16.10 in case you were wondering. The other advantage when added via SFTP it actually refreshes the file and folder sizes. It never did that with Samba and was slow to do it with FTP. Hopefully this helps some of you having this or a similar issue as there has been no efforts I have seen to fix the issue yet. Unfortunately this issue was around long ago when it was still the OwnCloud project and was never fixed then either. I know the issue is because the protocols do not support certain needed functions except for OpenSSH which does which is why SFTP often works. Anyway enjoy and good luck.

[Malapeno]

I'm experiencing the same issue as @nathank1989 using a local Synology DSM FTP server. Also I have noticed in my case this issue whereby ->

1. Attaching a user to the external storage (FTP) through the admin Nextcloud account.
2. Uploading files as this user from Nextcloud web into their home directory on the FTP server storage.
3. Then as admin removing this user's attached external storage (FTP).
4. Re-attaching this users external storage (FTP) once again as admin.

This results in empty folders where the user had previously uploaded files to through the Nextcloud web interface. Yet logging into the FTP server directly reveals these previously uploaded files still exist.

I have setup a cron background job as indicated in the Nextcloud Defining Background Tasks instructions to scan for file related changes every 15 minutes. I assume Nextcloud is capable of detecting existing files on my FTP server upon being enabled as an external storage device.

Has any headway been made on this issue?

Thanks for any help/suggestions you can provide

[mr-bolle]

Hi,

i use a external share as well (local Folder) - and after creating this share everything works prodigious.
After few weeks i establish, that some Folder empty or have only a singe text file in it.

If i checkt the folder all files availabe (with right permission www-data:www-data 755).
Yesterday i delete this external share, and create a new share with the same settings.

And into the Web all files are now available. I use the default share settings (Preview active & Check for changes: once every direct access)

I will update this Post, if this issue appear again

[ghost]

I had the same issue as @mr-bolle. Local folder. It is not a cache problem: I tried different browsers at different hours, I tried some reboots, nothing. But recreating the share (delete and add), solved it (... is this temporarily? we will see).

[stuckj]

I just noticed I also have this problem on a couple external shares I have setup with SFTP. However, rather than deleting / re-adding the share I found a simpler fix was to just touch the external folder (i.e., from a terminal outside next cloud) to update the last modified date to the current time. Go back to that folder in nextcloud and all the files appear with the subdirs showing the size as pending (as it scans them presumably).

My guess is that the external folder code is counting on the modified dates being propagated up the filesystem hierarchy to the external folder to determine if it should refresh the folder contents or not.

Either way, that's a workaround. Though it's not ideal.

[itsthejb]

Hi all,

I also have this issue: simple local FTP shares that work perfectly fine with a normal FTP client, but when added as Nextcloud external storage they are empty. Error in the logs I get is:

Exception during scan: opendir(ftp://[email protected]/): failed to open dir: operation failed
#0 [internal function]: OCA\Files\Command\Scan->exceptionErrorHandler(2, 'opendir(ftp://....', '/config/www/nex...', 66, Array)
#1 /config/www/nextcloud/apps/files_external/lib/Lib/Storage/StreamWrapper.php(66): opendir('ftp://itsthejb....')
#2 /config/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(102): OCA\Files_External\Lib\Storage\StreamWrapper->opendir('')
#3 /config/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(102): OC\Files\Storage\Wrapper\Wrapper->opendir('')
#4 /config/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php(109): OC\Files\Storage\Wrapper\Wrapper->opendir('')
#5 /config/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(102): OC\Files\Storage\Wrapper\Availability->opendir('')
#6 /config/www/nextcloud/lib/private/Files/Cache/Scanner.php(375): OC\Files\Storage\Wrapper\Wrapper->opendir('')
#7 /config/www/nextcloud/lib/private/Files/Cache/Scanner.php(426): OC\Files\Cache\Scanner->getNewChildren('')
#8 /config/www/nextcloud/lib/private/Files/Cache/Scanner.php(406): OC\Files\Cache\Scanner->handleChildren('', true, 3, '101068', true, 0)
#9 /config/www/nextcloud/lib/private/Files/Cache/Scanner.php(338): OC\Files\Cache\Scanner->scanChildren('', true, 3, '101068', true)
#10 /config/www/nextcloud/lib/private/Files/Utils/Scanner.php(245): OC\Files\Cache\Scanner->scan('', true, 3)
#11 /config/www/nextcloud/apps/files/lib/Command/Scan.php(145): OC\Files\Utils\Scanner->scan('/itsthejb', true, NULL)
#12 /config/www/nextcloud/apps/files/lib/Command/Scan.php(201): OCA\Files\Command\Scan->scanFiles('itsthejb', '/itsthejb', Object(Symfony\Component\Console\Output\ConsoleOutput), false, true, false)
#13 /config/www/nextcloud/3rdparty/symfony/console/Command/Command.php(255): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /config/www/nextcloud/core/Command/Base.php(166): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /config/www/nextcloud/3rdparty/symfony/console/Application.php(901): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /config/www/nextcloud/3rdparty/symfony/console/Application.php(262): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /config/www/nextcloud/3rdparty/symfony/console/Application.php(145): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /config/www/nextcloud/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#19 /config/www/nextcloud/console.php(97): OC\Console\Application->run()
#20 /config/www/nextcloud/occ(11): require_once('/config/www/nex...')
#21 {main}

I tried a few recommended PHP settings such as allow_url_fopen = 1 and allow_url_include = On, but no difference. Plus this issue seems to be stale....

[ronaldscott]

I just ran into this problem with a local directory mounted as an external storage. I was in the process of renaming one of the files in the external storage using the NextCloud web interface. When I was done, every single file and directory except the one I renamed was missing from NextCloud, though the files were still present on disk. I tried restarting the NextCloud server to no avail. I didn't want to remove and re-add the external storage because this is my family's shared photo directory. There are thousands of images in it and I was afraid that NextCloud would want to do preview generation all over again (it took four days the last time.)

The workaround from @stuckj (I just touched the root folder) resolved my problem as well.

[skjnldsv]

As the version of the software you've reported this for has reached end of life, I will close this ticket. If this is still happening after an upgrade to the latest version, feel free to reopen

[hardhub]

@skjnldsv I think you can reopen it if it is only about version because I just reproduced it in NC 18.0.4

It is not displayed until nginx and php-fpm are restarted.
After that it is shown in web.
I am not sure if such bahviour is normal so I have reported it here.