@icewind1991

nickvergessen on 22 Feb 2017

I have the same issue.
Any idea to fix it ?

Update : I can not see the reference of the new file in the table oc_filecache

ymolinet on 20 Mar 2017

This issue is a major problem for us because file change by a local SMB Share are not propagate to the nextcloud client. We can help you to fix this issue if we can understand how the sync process works and how the SMB notify works in Nextcloud.

ymolinet on 22 Mar 2017

@icewind1991

nickvergessen on 23 Mar 2017

up ?

ymolinet on 27 Mar 2017

I'm working on this issue, and I have a question.
In apps\files_external\lib\Command\Notify.php, I can found the function that it call by the notify command.
In the execute function (at line 90), I can found only the request to update a files already register in file_cache table, but no method to add or remove files....

ymolinet on 28 Mar 2017

If I understand the process correctly, another issue is possible.
External_Storage are register as mount point in oc_mounts.
Each user that use an external storage refer it by a storage id.
The storage id is use to refer the file in oc_filecache.

So, if a user push a file (from web ou client), it is refer in his storage, but all other are not updated ...

ymolinet on 29 Mar 2017

Anyone could explained how NextCloud get file data to store it in the filecache table?
especially mimetype list, mimepart, etag, permissions

ymolinet on 29 Mar 2017

The notify command doesn't write the new info to the filecache itself.

It marks a folder as out-of-date (setting it's size to -1), the background scanner (cron) will then detect that and handle updating the cache.

If you're running into issues with the cache not being updated while the notify command works fine, please verify that nextcloud's cron job is running correctly

icewind1991 on 29 Mar 2017

So notify works only if a file or folder is changed, what's happen if we add a file ? In addition, scan must have the credential to access to the share especially if we use credential session (ldap user).

ymolinet on 29 Mar 2017

For new files the folder is also marked as outdated and the scanner will find the new files.

session credentials might indeed cause issues, not sure, will investigate later

icewind1991 on 29 Mar 2017

If we use the session credential, scan can occur only if the user is connected to the web interface.
It could be interresting to allow scan when the user is connected throw the windows client (could be an option to enable in the admin panel).

ymolinet on 31 Mar 2017

If I correctly understand, the windows client use WebDAV to connect with NextCloud.
it could be interresting to invoque the cron (like ajax cron) when a user log on by Nextcloud client.

ymolinet on 4 Apr 2017

is it possible to run two cron jobs? one every minute that only for checking if a folder has been marked as updated? and the other for everything else. I have my cron time set to 15min, so it takes up to 15 minutes before a change is synced.

ariselseng on 7 Apr 2017

I think you can not solve this problem with two cron.
Your second job need to have the user credential store somewhere. If (like me) you used the credential in the session, NextCloud don't have the login/password to connect to the remote share.

ymolinet on 11 Apr 2017

Multiple parallel cron jobs are supported since NC 10 I think

nickvergessen on 11 Apr 2017

Up ?

ymolinet on 23 Apr 2017

I think that the best way to fix this issue is to create a "new" way to scan the files.
A scan method, running with a remote account can read permissions on the remote share and associate it with nextcloud user ... (LDAP Account).

ymolinet on 26 Apr 2017

Is it possible to have informations about the file_cache table (column description and waiting values)

ymolinet on 26 Apr 2017

I think a problem exist with the markParentAsOutdated function because it use dirname() php method.
under Windows, this method use / and \ to parse the directory, but under Linux it use only /.
So when Nextcloud is running under linux and check a Windows path, the parent directory is not correctly parse ...

ymolinet on 22 May 2017

👍1

In addition, path is return from notify with / at the beginning, but files in filecache don't have this (path column).

ymolinet on 22 May 2017

Hi
We are experiencing the same problem. Using v12.04. Update and delete directly made in the filesystem (mounted in nextcloud via CIFS) are not correctly propagated to all users via sync desktop.

philippe-opendsi on 15 Dec 2017

cc @icewind1991

MorrisJobke on 18 Dec 2017

Using 'Login credentials, save in database' instead of 'Login credentials, save in session' should make the file scanner pickup the changes correctly.

As for decreasing the "sync interval" you can run occ files:scan --unscanned --all as a separate cron job to pick up new changes before the "normal cron job" does it.

icewind1991 on 23 Jan 2018

👍1

@icewind1991 Could you add this to the documentation for the notify command? Thanks

MorrisJobke on 23 Jan 2018

https://github.com/nextcloud/documentation/pull/652

icewind1991 on 23 Jan 2018

Hi all,

When running this command, I receive :

Scanning files for 27 users
Starting scan for user 1 out of 27 (05C757BD-241F-466F-83AF-5E2E60B4C881)
Starting scan for user 2 out of 27 (083A35D7-1564-4D62-8F80-724EB2B8DFB6)
Starting scan for user 3 out of 27 (0CE75D09-6809-4F60-ABFE-7B5DEBA854D9)
Starting scan for user 4 out of 27 (103447F3-9BC7-473C-B5AB-682F137BE67D)
Starting scan for user 5 out of 27 (26B2FD3D-293D-4E6F-AABD-D20136CDE394)
Starting scan for user 6 out of 27 (2A844536-1909-429A-A2CD-470421A5FA72)
Starting scan for user 7 out of 27 (509882B0-6A47-471E-BF73-29467E983E14)
Starting scan for user 8 out of 27 (54E0CE78-5AC8-46E5-959D-C7B957E4825C)
Starting scan for user 9 out of 27 (55F3B2B2-AFAE-4CCF-9006-5F948FAFA45C)
Starting scan for user 10 out of 27 (564E63B9-A75F-4995-B5C1-F2DD6EDD3B45)
Starting scan for user 11 out of 27 (56F7356D-A833-48CE-B071-BF7362A27B3B)
Starting scan for user 12 out of 27 (7BF3D783-7BA2-4177-9EA8-AE1C173F640B)
Starting scan for user 13 out of 27 (7C32C66D-D520-4510-B93D-EF67EBE951B8)
Starting scan for user 14 out of 27 (7E72AC07-2501-4749-89CB-B25EBCBD4908)
PHP Fatal error: Wrong parameters for Exception([string $exception [, long $code [, Exception $previous = NULL]]]) in /var/www/clients/client1/web124/web/apps/files_external/3rdparty/icewind/smb/src/Exception/Exception.php on line 17

Nextcloud 12.0.4

Thanks,
Yannick

De : Robin Appelman [mailto:[email protected]]
Envoyé : mardi 23 janvier 2018 13:40
À : nextcloud/server server@noreply.github.com
Cc : Yannick MOLINET yannick@adn-systemes.fr; Comment comment@noreply.github.com
Objet : Re: [nextcloud/server] occ files_external:notify does not propogate changes to clients. (#3524)

Using 'Login credentials, save in database' instead of 'Login credentials, save in session' should make the file scanner pickup the changes correctly.

As for decreasing the "sync interval" you can run occ files:scan --unscanned --all as a separate cron job to pick up new changes before the "normal cron job" does it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/nextcloud/server/issues/3524#issuecomment-359778300, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADSbFB0JNq42Rm9fioMaBAr6I0D6i2sIks5tNdMfgaJpZM4MEIjK.

ymolinet on 24 Jan 2018

Does it work with version 12? I tested with version 13.0.0 and getting the same issue.

In that thread @cowai said version 11.0.x but does someone know if it works with version 12.0.x?
Thanks

jclgs on 27 Mar 2018

I have the same propagation issue. If I create a file in the external storage it doesn't propagate to desktop client with version 13.0.1

tinoper on 27 Mar 2018

same for me (NC 13 / debian stretch)

it's a big problem for an organization with internal network. It's really great to get network shares in the NC instance but with this bug, it's really not possible to deploy NC… and I'm sad :(

julpec on 4 Apr 2018

If you're experiencing this issue please give some details about the way you have the SMB external storage setup and how you're running the notify command.

icewind1991 on 5 Apr 2018

On my side ....

SMB is configured with Save Credential in Database. The mount is on a Windows 2008 R2 file server.

I'm running the Notify command by command line for testing. ./occ files_external:notify -u myuser -p mypassword ID_Of_The Share.

After that, cron.php running every 15 minutes. Even if I'm running the Files:scan -Unscanned, nothing change.

Do you need more information?

jclgs on 5 Apr 2018

Can you try adding the -v flag to the notify command to see if it receives the expected changes

icewind1991 on 5 Apr 2018

Yes, I added the -v flag and the Notify command is seeing every changes. New File, New Folder, change file, change folder.

jclgs on 5 Apr 2018

Hi, when I try to execute the following command:
sudo -u www-data php occ files_external:notify --user=myUserName --password=myPassword 4 where 4 is the connection id

We get a connection error mesage
[Icewind\SMB\Exception\ConnectionException]
Connection not valid

myUserName is a ldap user and I also try to add the domain name but maybe some kind of / vs \ issue doesn't allow to do this or at least I can't

Let me know if you need more information or if I need to try something else.

Exception trace: () at /var/www/nextcloud/apps/files_external/3rdparty/icewind/smb/src/Connection.php:59 Icewind\SMB\Connection->read() at /var/www/nextcloud/apps/files_external/3rdparty/icewind/smb/src/NotifyHandler.php:76 Icewind\SMB\NotifyHandler->listen() at /var/www/nextcloud/apps/files_external/lib/Lib/Notify/SMBNotifyHandler.php:74 OCA\Files_External\Lib\Notify\SMBNotifyHandler->listen() at /var/www/nextcloud/apps/files_external/lib/Command/Notify.php:141 OCA\Files_External\Command\Notify->execute() at /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php:264 Symfony\Component\Console\Command\Command->run() at /var/www/nextcloud/core/Command/Base.php:161 OC\Core\Command\Base->run() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:874 Symfony\Component\Console\Application->doRunCommand() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:228 Symfony\Component\Console\Application->doRun() at /var/www/nextcloud/3rdparty/symfony/console/Application.php:130 Symfony\Component\Console\Application->run() at /var/www/nextcloud/lib/private/Console/Application.php:173 OC\Console\Application->run() at /var/www/nextcloud/console.php:90 require_once() at /var/www/nextcloud/occ:11

tinoper on 5 Apr 2018

Hello

For me, SMB is configured with Save Credential in Database. The mount is on a Linux Debian server with samba4.

For testing, I have run the occ command like this : ./occ files_external:notify -v -u myuser -p mypassword <ID_Of_The Share>

Just after that it gave me Self-test successful. When I create a new file on the SMB storage, I have this in the output : modified test4. So, it detect changes on the smb mount !

After that, I don't have the file synchronized with my client but if I run files:scan --unscanned for the same user, it is working, the files are synchronized.

Another way to make the changes synchronize is to navigate in the Web interface to the folder where changes were made. The Web instance is scanning the files and populate the database, so the client side get the information to synchronize the files… wright ?

So, there is something I don't understand : is it necessary to run the files_external:notify for each users ? I have check this option on the external mount, so it should run this for each user no ? Here is an example of my external mount :
| 13 | /<Name of mount> | SMB / CIFS | Identifiants de connexion, sauvegarder dans la base de données | host: "172.16.0.11", share: "home", root: "Docs", domain: "lycee" | enable_sharing: true | | Group1 |

How to know if the cron.php is running all wright ?

julpec on 11 Apr 2018

You should only need to run one notify command for each configured mount point as long as all users "see" the same files on the share.

icewind1991 on 11 Apr 2018

@tinoper Which version of smbclient do you use? I have problems on the wonderfall/nextcloud which uses a very recent smbclient. On older versions it works great.

Edit: I use 4.7.6 on alpine linux 3.7 and it does not work.

ariselseng on 11 Apr 2018

@icewind1991 , but the notify command expect a user and a password !! Here is the result of sudo -u www-data php occ files_external:notify 13 :
Error while trying to create storage Username and/or password required

The users doesn't see the same files, it's a smb sharing of the home directory on the network.

@tinoper , smbclient version 4.5.12. What version should I use ?

julpec on 11 Apr 2018

If there are different files for each user then you need a seperate notify command for each user

icewind1991 on 11 Apr 2018

@cowai and @julpec , I'm using smbclient 4.3.11-Ubuntu
@icewind1991 , files_external:notify is working if user is on Windows LDAP ?
I have a wrong user password issue on the following command: sudo -u www-data php occ files_external:notify --user=myUserName --password=myPassword 4
I also try with DOMAIN/user syntax because myUserName is authenticating with Windows LDAP

Regards

tinoper on 12 Apr 2018

With latest nextcloud official docker image (13.0.1) and added smbclient,
This does not work reliably.

$ docker-compose exec --user www-data app php -d memory_limit=4G occ files_external:notify 9 -vvv
Error while running self-test, change is subfolder not detected
added /Markedsavdeling/Ari/10
added /Markedsavdeling/Ari/10/20
added /Markedsavdeling/Ari/10/20/test1.txt
modified /Markedsavdeling/Ari/10/20/test1.txt
modified /Markedsavdeling/Ari/10/20/test1.txt
added /Markedsavdeling/Ari/10/20/30
added /Markedsavdeling/Ari/10/20/30/40
added /Markedsavdeling/Ari/10/20/30/50
removed /Markedsavdeling/Ari/10/20/30/50
added /Markedsavdeling/Ari/10/20/30/40/50
added /Markedsavdeling/Ari/10/20/30/40/50/test2.txt
modified /Markedsavdeling/Ari/10/20/30/40/50/test2.txt
modified /Markedsavdeling/Ari/10/20/30/40/50/test2.txt
added /Markedsavdeling/Ari/10/20/30/40/50/60
added /Markedsavdeling/Ari/10/20/30/40/50/60/test3.txt
modified /Markedsavdeling/Ari/10/20/30/40/50/60/test3.txt

All these changes were noticed by the command, but when I run:

$ docker-compose exec --user www-data app php -d memory_limit=4G occ files:scan --unscanned --all  -vv
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 0       | 0     | 00:00:00     |
+---------+-------+--------------+

I have the Markedsavdeling/Ari folder synced on a local client and the /Markedsavdeling/Ari/10/20/30/40/50 folder got synced, but the containing "60" folder is missing.

This problem has been going on for more than a year now and Nextcloud is quite unusable for people with an external Windows share that is frequently accessed remotely.

ariselseng on 13 Apr 2018

I think the reason for the first error in the notify command is because of the user do not have access to the root of the share.
Edit: Confirmed to be true.

ariselseng on 13 Apr 2018

@icewind1991 Maybe the problem lays around this function https://github.com/nextcloud/server/blob/stable13/apps/files_external/lib/Command/Notify.php#L149
It seems that paths that are too deep have less change of getting found/detected with files:scan --unscanned, even though they are detected with the files_external:notify.

ariselseng on 13 Apr 2018

On my side, I have a different opinion. I think the Notify is working because it detects the changes. I saw them when I run -vvv options. But this is the process after the detection that is not working.

If Notify detect the change, after the change is detected, it should force the cache to be refresh and it looks like it does not. If I'm going to the web interface, it refresh the cache and my NextCloud Client on Windows 10 is seeing the changes. If I'm not going on the web page, the Nextcloud client on windows is never seeing the change, even if Notify command found it.

So I think that the trigger that the Notify Command should do when he detect changes, is to trigger the update of the cache like done in the Web Interface

jclgs on 13 Apr 2018

@jclgs You describe exactly my findings. occ files_external:notify detects changes, but occ files:scan --unscanned does not pick them up.

ariselseng on 13 Apr 2018

@cowai I'm not sure that the depth is in relation. Furthermore, my user have write access on the root of the share : it's his home directory !
@jclgs I'm totaly agree with you. That's exactly what I noticed.

julpec on 14 Apr 2018

@icewind1991 any development related to that? 26 days without any news. It can be cool if this can be included in the next release. I'm planning to upgrade to version 13 soon, but wait to get this solves.

If it works, it can be an interesting feature for people using it beucause right now, it looks not working like it should.

jclgs on 10 May 2018

Hello
Is it working well in NC12 ?
I'm still waiting for that and I think I will install NC12 in order to get the smb modifications propagate to clients
@icewind1991 do you have any news of this bug ?

Thanks for all

julpec on 29 May 2018

We improved the overall SMB compatibility a lot recently. Please give 13.0.4 a good try it holds already a lot of improvements.

I would like to close this ticket here and would ask you to open new tickets with specific scenarios where the detection doesn't work.

Thanks

MorrisJobke on 3 Jul 2018

@MorrisJobke I will test when I can. AFAIK there is no scenario where it actually works. I feel I have tried everything.

ariselseng on 3 Jul 2018

@MorrisJobke I will test when I can. AFAIK there is no scenario where it actually works. I feel I have tried everything.

:/ Sorry to hear that. We have quite some big setups where it works just fine. Have you SMBv2 or v3? Because it only works in v2 AFAIK.

MorrisJobke on 3 Jul 2018

@MorrisJobke I have tried both. The changes show up in the notify command but do not trigger changes when doing a scan.

ariselseng on 3 Jul 2018

The changes show up in the notify command but do not trigger changes when doing a scan.

The scan is not needed then. If they show up in the notify command then they are properly recognized and are put into the file index

MorrisJobke on 3 Jul 2018

Then the etag of the parent folders should change and trigger the clients to sync that folders/files.

MorrisJobke on 3 Jul 2018

You could check the DB table oc_filecache for the files you update and there the etag as well as all etags of the parents should change. If this is the case it worked properly.

MorrisJobke on 3 Jul 2018

@MorrisJobke I thought I needed to do occ:files scan --unscanned on interval.

When I opened this issue the folder with the changes did not get synced to my nextcloud client. Even after waiting hours. Only after I opened the folder manually in the web interface did the changes get noticed.
I will try with latest release today anyways and check.

ariselseng on 3 Jul 2018

Run occ files_external:notify -v

This needs to run the whole time as it gets pushed new changes actively from SMB and then there is no need to periodically scan , because the changes are pushed to Nextcloud ;) See https://docs.nextcloud.com/server/13/admin_manual/configuration_files/external_storage/smb.html#smb-update-notifications

MorrisJobke on 3 Jul 2018

@MorrisJobke I thought I needed both. I am also using the occ files_external:notify -v continuously without changes ever showing up.
I have used the SMB package from @icewind1991 and that works manually with PHP. I can see the changes happening. Just as in the occ files_external:notify -v , where I see the changes listed but in nextcloud clients the changes does not propogate.
I will try again now with 13.0.4 check one more time.

ariselseng on 3 Jul 2018

@MorrisJobke Now it works :) I still need both commands. So running "php occ files:scan --unscanned [mount_user]" each minute will do.
Note to others: You nowadays need to force smb to use NT1 in smb.conf.

ariselseng on 3 Jul 2018

@cowai @MorrisJobke for me, it doesn't work. The "php occ files:scan --unscanned" command needs the user id as argument. The same applies to the'occ files_external:notify -v' command. When I run this command as is, it returns this to me:
Error while trying to create storage Username and/or password required
I have over 1000 users in my ldap database and I can't put this command for every user in a cron.

Do you have an idea to fix this without using user-specific commands?

julpec on 3 Jul 2018

In addition, notifications do not go up for changes on samba sharing.
When I make a change, I run the command occ files:scan --unscanned <user_id> in the process but nothing goes up on the nextcloud client, even if I run the command occ files_external:notify -u <user-id> -p <pass> -v 13 during the change (I don't see any notification).
The only thing that works is when I access the directory via the web interface...

julpec on 3 Jul 2018

I've upgraded now from 13.0.1 to 13.0.4 and it's still not working!
I have one process with occ files_external:notify -v and one with occ files:scan --unscanned --all
The changes are correct recognized by files_external:notify.
But changes are not recognized by windows client.
Only the web interface is working if I enter the affected folder.

theroch on 10 Jul 2018

I've checked the oc_filecache table while I create some files on the SMB share.
Running files_external:notify recognized the change correct, but updates not the oc_filecache table.
Execution of occ files:scan --unscanned <user-id> also updates not the oc_filecache table.
Only if I run occ files:scan <user-id> the oc_filecache is updated, but the nextcloud client still not recognize the change.

theroch on 10 Jul 2018

Ok I've tested a little bit around.
If I set the column 'size' of the parent element in oc_filecache table to -1 (like in the updateQuery). Running then occ files:scan --unscanned <user-id> the parent and all of it's childrens are correctly recognized but the nextcloud client is still not informed about the changes!

[update]
~Can please someone explain how the nextcloud client is informed about updates?~
I didn't regard the storage id.

Some more explanation:
We mount the share global with session credentials.
One process runs with occ files_external:notify -u <user-id> -p <pass> -v <mount-id>
One process runs with occ files:scan --unscanned --all
I've tested with 2 Level folders whats means: mount/Level1/Level2/File

The process with files_external:notify recognizes all the changes correctly but doesn't seems to update the oc_filecache table.

If I now change the oc_filecache table manual by setting the etag to null, I have to set the etag of Level2 and Level1 to null.
It is not sufficient to set only the etag of Level2 to null!

If I run now occ files:scan --unscanned --all or occ files:scan --unscanned <user-id> the changes are correctly recognized by the nextcloud client.
Notice: The <user-id> have to be same as from the user of files_external:notify process.

So maybe @cowai put his finger on the wound:

@icewind1991 Maybe the problem lays around this function https://github.com/nextcloud/server/blob/stable13/apps/files_external/lib/Command/Notify.php#L149
It seems that paths that are too deep have less change of getting found/detected with files:scan --unscanned, even though they are detected with the files_external:notify.

[update2]
Strange behavior it seems to be not sufficient to set size to -1 that doesn't trigger the nextcloud client.
Instead you have to set the etag along the path to null.
Can anyone else confirm?

theroch on 10 Jul 2018

I've created a patch to workaround this issue:

--- Notify.php  2018-07-10 18:58:03.440317527 +0200
+++ Notify.php  2018-07-10 20:11:41.306307145 +0200
@@ -54,7 +54,7 @@
                $this->connection = $connection;
                // the query builder doesn't really like subqueries with parameters
                $this->updateQuery = $this->connection->prepare(
-                       'UPDATE *PREFIX*filecache SET size = -1
+                       'UPDATE *PREFIX*filecache SET size = -1, etag = NULL
                        WHERE `path` = ?
                        AND `storage` IN (SELECT storage_id FROM *PREFIX*mounts WHERE mount_id = ?)'
                );
@@ -156,10 +156,13 @@

        private function markParentAsOutdated($mountId, $path) {
                $parent = dirname($path);
-               if ($parent === '.') {
-                       $parent = '';
-               }
+               $parent = trim($parent, "\\/");
+               if (empty($parent) || $parent === '.' || $parent === '\\' || $parent === '/') {
+           $this->updateQuery->execute(['', $mountId]);
+                       return;
+           }
                $this->updateQuery->execute([$parent, $mountId]);
+               $this->markParentAsOutdated($mountId, $parent);
        }

        private function logUpdate(IChange $change, OutputInterface $output) {

I've tested the workaround and it seems to work correctly.
It also seems to be that the execution of occ files:scan --unscanned <user-id>is no more needed.

theroch on 10 Jul 2018

@theroch yeaaaaah !! great ! It finally works perfectly!
I had found a workaround by scanning regularly in a cron the only activated users in nextcloud but it was an additional load on the server and the synchronizations were only done every 10 minutes. Now everything is immediate, it's really great!
Thanks to you @theroch .
This patch should now be worn on the next version of NC !

julpec on 15 Jul 2018

@theroch ho no !!!
It worked for some files but now it doesn't work anymore. I think it was my cron that was still working...
I created new directories and files, they were synchronized. Subsequently, I deleted them and the deletion did not spread. I then created a new file in the same directory and nothing... then a new file in another directory and still nothing...
@theroch my samba editing is done with the connection identifiers saved in the database. I checked in the oc_filecache table and my new files do not appear.

julpec on 15 Jul 2018

@theroch sorry for my last post ! I hadn't applied the patch properly. Now everything works very well even file and directory deletions.

julpec on 15 Jul 2018

@julpec Did you use the patch from the comment in this issue or from the pull request?
We use the patch from the pull request now for several days and it seems to work properly.
Perhaps a few other people can also confirm this?

theroch on 16 Jul 2018

Hi
Unfortunately it doesn't work correctly. I mean partly it does but when I change permission (adding access or removing) for particular folder or sub-folder, the change is nicely recognized in "occ files_external:notify -v" (run with admin user which has access to whole share) but nothing change in sync folder or website access. Also "occ files:scan ---unscanned --all" scan doesn't help. Only way to get changes synchronized is to run "occ files:scan --all" - can I run it every 15 minutes or it is not recommended?
I have active directory with 200 users, nextcloud smb is configured with storing password in database.

PiotrIr on 2 Aug 2018

@PiotrIr you're right, it doesn't work for me either. Besides, I have no return in "occ files_external:notify -v". I really don't understand anything, sometimes it works and other times there is no synchronization as long as I haven't created a file or directory from my client or if I haven't accessed the web interface. @PiotrIr you can try to run "occ files:scan --all" every 15min but I think it will take a very very long time with 200 users. For me, with 10 users activated, it takes more than 10min.

Also, it seems that the patch works better than the pull request but in any case the synchronization time is quite long: it takes several minutes before the modification is propagated to the client and I don't know why…

julpec on 2 Aug 2018

Julpec - I just only wander if this is safe to run "occ files:scan --all" every 15 minutes. Somebody said on another post, this is not for regular run, only when there are issues.

PiotrIr on 2 Aug 2018

@PiotrIr It should okay if you have a small enough data set. For me doing a scan takes about 1 hour. Also it is quite resource intensive.
Also, with this approach changes would take at least 15 minutes to propogate. And that increases the chance that 2 people are working on the same document without knowing it.

ariselseng on 2 Aug 2018

cowai, - thank you for your reply. I just wander - is any better fix for this? Is it possible to scan only this particular shared folder instead of all files?

PiotrIr on 2 Aug 2018

yes. You can use --path for that I think it is --path "/USER/files/FOLDER".

ariselseng on 2 Aug 2018

Thank you cowai, However I can't find the way to use --path in my scenario because how to scan this shared folder for all users? It looks you need to give particular user name and then the path seems to be local (not \file_server\shared_folder). Did I missed something?

PiotrIr on 2 Aug 2018

My example only works if you have a global mount in nextcloud. Then you can scan with a user that has access to the path, in my case, "admin". Yes, the path should be local and exact path in nextcloud.

ariselseng on 2 Aug 2018

Hi cowai, I'm trying to use it but it doesn't recognize active directory user - only local one. Is it possible to use AD user?

PiotrIr on 2 Aug 2018

@PiotrIr You need to mount the samba share inside nextcloud, as a global external storage mount. Here is where you define the samba/AD user.
In the occ:files scan command you define a nextcloud user that has access to the path where you mounted the external storage.

ariselseng on 2 Aug 2018

Ok, now I understand. In this case the -path wan't do for me as I need to use NTFS permissions for users.

PiotrIr on 2 Aug 2018

The problem with scanning files is that, in my case, each user has access to a personal directory so I have to run a scan for each user which becomes a real problem beyond 10 users: it takes time and resources.
I have continued to test and it seems that it is the deletion that does not work for me. A deletion is not propagated on the customer. Here is an example of what I did :

creating folder "test1" at the root of the share + creating file "testfile1" inside --> sync ok
creating file "testfile2" at the root of the share --> sync ok
removing "testfile1" and "testfile2" --> sync not ok
After that, every file or directory created is not sync. The only way to get a fresh new sync is to create a file from the client or to access the folder in the web interface.
@theroch it's the same with the patch or with the pull request.

julpec on 2 Aug 2018

This is kindly similar to my case I believe. In your case I would probably try to create user which has access to all home folders, attach root folder (above home folders) and scan the root folder using this user. Disadvantage will be - all users will see names of other users folder's but they shouldn't be able to access them

Any way, I just wander how to make sure, the "occ files_external:notify" will start automatically and when it crashes will restart? Any advise?

PiotrIr on 2 Aug 2018

@PiotrIr I have no idea. For me, it's a mysterious problem because I don't have enough knowledge and I don't have the possibility to fully immerse myself in the CN code. I can only pray that this issue is really resolved.
Other information after my last post : it's the same problem with moving file. This "breaks the system" and after that, no synchronization works.
Furthermore, if I edit directly from the samba share, nothing is returned in the occ files_external:notify -v. The only modifications indicated are those made from the client.

julpec on 2 Aug 2018

@PiotrIr

Any way, I just wander how to make sure, the "occ files_external:notify" will start automatically and when it crashes will restart? Any advise?

Here is a sample systemd unit to run occ files_external as service and restart on failure or exit.
You have to replace the <name> strings with your data.

[Unit]
Description=Nextcloud external file watch
After=network.target

[Service]
Type=simple
User=<web-user>
Group=<web-group>
WorkingDirectory=/var/www/nextcloud
ExecStart=/var/www/nextcloud/occ files_external:notify -u <user-name> -p <password> <share-id>
RemainAfterExit=no
#on-failure or always, on-abort, etc
Restart=always
RestartSec=120s

[Install]
WantedBy=multi-user.target
Alias=nextcloud_external_notify.service
Alias=occ_files_external.service

theroch on 2 Aug 2018

@julpec

creating folder "test1" at the root of the share + creating file "testfile1" inside --> sync ok

creating file "testfile2" at the root of the share --> sync ok

removing "testfile1" and "testfile2" --> sync not ok
After that, every file or directory created is not sync. The only way to get a fresh new sync is to create a file from the client or to access the folder in the web interface.

We very rarely delete data, so this case it not really tested by us.
I'll run some tests and check the database to see what happens.

theroch on 2 Aug 2018

theroch - thank you so much, it is working!

PiotrIr on 2 Aug 2018

well, it was working to the moment when I disabled smb1 on windows server and now I'm getting notification "Notify is not supported with the used smb version". Is Notify supported only on SMB1?

PiotrIr on 3 Aug 2018

does somebody use SMB2 with Notify?

PiotrIr on 3 Aug 2018

I came across an interesting circumstance with the Nexcloud desktop client during my tests. Although I changed files on the SMB share and the notifier correctly changed the ETAGs along the path, the client did not detect any changes. The client only ran through a small selection of my folders!?
I wondered how the client would be informed about changes to the server and then took a closer look at the Developer Docs for the Nextcloud desktop client:
The client runs through the paths locally and compares the ETAG of each element with that of the element on the server. If the ETAGs differ, a sync is triggered.
In detail, the documentary even says:
"Note on remote discovery: Since a change to a file on the server causes the etags of all parent folders to change, folders with an unchanged etag can be read from the journal directly and don't need to be walked into."

Resetting the ETAGs along the path through my patch seems to be the correct approach.

Now it must be further investigated why the client does not find any changes even though the ETAGs have changed, i.e. now debugging is necessary. Unfortunately, C++ is not my strong suit, so it will probably take a while.

theroch on 24 Aug 2018

Is this working now with official nextcloud docker image + smbclient?

ariselseng on 25 Feb 2019

@theroch Is things working for you with your patch now?
I am now trying the official dockeri image + smbclient and when testing, and it seems some files do not get synced, even though they are listed during occ files_external:notify, but not during occ files:scan --unscanned.

ariselseng on 25 Feb 2019

@cowai I'm using now nextcloud 15.04 with desktop client 2.5.1. The sync works in this versions now much better, so I didn't test it with my patch.
Which files do you have problems with, in what depth of subdirectories do they lie?

theroch on 25 Feb 2019

@theroch Same versions here.
It seems to be quite random. In some tests its the file which resides in the 9th level that gets synced. Next time its the first level. I am testing by appending text to 3 files. Why Nextcloud still has problems with this after years is beyond me.
Try this a couple of times:
$ date >> SHARE/1th/2th/3th/4th/5th/testfile1.txt ;sleep 1;date >> SHARE/1th/2th/3th/testfile2.txt ;sleep 1;date >> SHARE/1th/testfile3.txt

Do each of the files show up in your client? For mine its totally random.

ariselseng on 25 Feb 2019

@MorrisJobke is this a known working feature now?

ariselseng on 25 Feb 2019

@MorrisJobke is this a known working feature now?

Yep - we have quite some instances where this is actively used for big external storages and there it propagates properly the changed files :/

MorrisJobke on 26 Feb 2019

@MorrisJobke @icewind1991 I have tried multiple times the last couple of years. Maybe there is something wrong/different with my setup?

The smb server is using Windows Server 2012 R2 Standard.

The main bug/problem I have is this:

The "occ files:scan --unscanned --all" command does not always list every change that the occ files_external:notify command did.

This is the procedure to reproduce:

use latest 15.0.4-apache official docker image
enable external storage
mount the smb storage on a machine for testing. ( a local cifs mount)
add a global smb mount
run php occ files_external:notify 1 -vv
setup a desktop client to sync a specific folder on smb. Let it finish syncing.
run this to check for unscanned:

$ while true;do php occ files:scan --unscanned --all;sleep 15;done

run this on the machine where you have the cifs share locally mounted:

$ mkdir -p SHARE/1th/2th/3th/4th/5th
$ date >> SHARE/1th/2th/3th/4th/5th/testfile1.txt && sleep 1 && date >> SHARE/1th/2th/3th/testfile2.txt 
 && sleep 1 && date >> SHARE/1th/testfile3.txt

watch as the occ files_external:notify command lists the changes successfully.
wait a couple of minutes.
check that the file tree on the share and on the desktop client is the same. It will not be.

ariselseng on 26 Feb 2019

@icewind1991 @MorrisJobke
I have confirmed that after doing some changes on smb the parent folders size is set to -1.
But if there is more than 1 file changed that shares a subset of the path tree, only one of the files is scanned correctly. This must be a bug. Note, I am running latest official docker image.

In my test 33 bytes are added to testfile1.txt, testfile2.txt and testfile3.txt
Here is output from sql query:

select path,size from oc_filecache where path like "SHARE/%"

before doing changes:

after changes, parent folders are marked with size -1 correctly:

after running occ files:scan --unscanned --all, note wrong sizes for testfile3.txt and testfile2.txt:

after full scan, now have correct sizes:

ariselseng on 26 Feb 2019

@icewind1991 @MorrisJobke
Sorry for the spam here.

FINALLY I got to the bottom of this.
When I make getIncomplete() in /lib/private/Files/Cache/Cache.php return an array with all entries with -1 as size instead of just 1, and then for loop over each one in backgroundScan() in /lib/private/Files/Cache/Scanner.php, it works fine!

I got the hint here where there was a "FIXME" line: https://github.com/nextcloud/server/blob/master//lib/private/Files/Cache/Scanner.php#L523

It seems that for some reason the sorting of getIncomplete makes a difference. I believe this bug only happens when the entry with the highest fileid is further down in the tree then the other changes.
After backgroundScan() has processed the folders with -1 size that was not returned in getIncomplete() suddenly has its old wrong size back.

@icewind1991 I am not sure if making getIncomplete() returning an array is the right solution to all this. I am willing to help if you have any thoughts. If you are not convinced this is not a bug or I am not making myself understood clearly, please tell :)

ariselseng on 26 Feb 2019

Okay so the bug only presents it self if there is more than folder with size -1 + that they are in the same tree + that the folder created last (highest id) and is the one deepest into the tree.

Like test1/test.txt and test1/subfolder/test2.txt
Where "subfolder" has a higher id than "test1".

A quick change of ordering in getIncomplete() fixes the problem it seems.
@icewind1991 @MorrisJobke Is this a safe fix? Is the "order by fileid DESC" worth it?
``diff --git a/Cache.php b/nextcloud/lib/private/Files/Cache/Cache.php index 43e48e5..7d5ed90 100644 --- a/Cache.php +++ b/nextcloud/lib/private/Files/Cache/Cache.php @@ -841,7 +841,7 @@ class Cache implements ICache { */ public function getIncomplete() { $query = $this->connection->prepare('SELECTpathFROMPREFIXfilecache' - . ' WHEREstorage= ? ANDsize= -1 ORDER BYfileidDESC', 1); + . ' WHEREstorage= ? ANDsize= -1 ORDER BYpath` ASC', 1);
$query->execute([$this->getNumericStorageId()]);
if ($row = $query->fetch()) {
return $row['path'];

ariselseng on 27 Feb 2019

👍1

@icewind1991 @MorrisJobke Is this a safe fix? Is the "order by fileid DESC" worth it?

Looking at the code it shouldn't make a difference. But maybe @icewind1991 knows more?

MorrisJobke on 27 Feb 2019

There could also be a potential fix related to and in calculateFolderSize() if it somehow did not calculate size for parent folders more than 1 level up, that also had size -1. That is the core of the issue but I am not sure how to fix that. Just returning a sorted result in getIncomplete() where top most folder is returned it also avoids the issue with wrong calculated folder sizes.

ariselseng on 27 Feb 2019

I think I remembered the reasoning behind the DESC, if it uses ASC it means that when the background scanner runs, it's will run on the top most folder that is marked incomplete (since child id's are generally higher that the parent one), this is also likely the folder that will take the longest to scan.

sorting DESC makes it so we run the background scanner in the smallest "chunks" possible.

Now I'm not sure if this is still a relevant "advantage", I remember there being logic somewhere that ran the background scanner one "chunk" at a time, but I can't find it so it might have been removed.

I think switching to ASC will be safe to try for 16

icewind1991 on 27 Feb 2019

👍1

@theroch
Care to checkout these changes? And do a proper test to try and get sync issues? Like do changes and check with rsync --stats --dry-run --checksum or something.

A couple of PRs here will probably help for the issues listed in this thread:

https://github.com/nextcloud/server/pull/14425 (Fixes where some folders to not get scanned properly)
https://github.com/nextcloud/server/pull/14456 (Fixes where no folders get scanned)

@MorrisJobke
@icewind1991

There is still also this issue that makes the very first registered change in occ files_external:notify not be registered: https://github.com/nextcloud/server/issues/14441

With all these bugs it makes me wonder if I am the only one actually using the SMB share outside of Nextcloud :D

ariselseng on 1 Mar 2019

To avoid running into performance issues we definitely need a db index on path, which is used in the ORDER BY clause.

klada on 4 Mar 2019

👍1

@klada You're right.
I have updated my PR to avoid this though. https://github.com/nextcloud/server/pull/14425

ariselseng on 4 Mar 2019

@cowai I'will try your changes this week.

theroch on 4 Mar 2019

Server: occ files_external:notify does not propogate changes to clients.

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

Client configuration

All 107 comments

before doing changes:

after changes, parent folders are marked with size -1 correctly:

after running occ files:scan --unscanned --all, note wrong sizes for testfile3.txt and testfile2.txt:

after full scan, now have correct sizes:

Related issues