Server: Files from external storage (Dropbox) never get updated.

Created on 23 Jan 2017 · 18Comments · Source: nextcloud/server

Steps to reproduce

connect Dropbox as external storage
sync the folder with the desktop client
files get downloaded once but never updated again, update also does not happen in web interface

Expected behaviour

Files should update.

Actual behaviour

Files are not updated.

Server configuration

Operating system: Linux (Ubuntu 16.10)

Web server: Apache

Database: MySQL (5.7.16)

PHP version: 7.0.13

Nextcloud version: 11.0.1

Updated from an older Nextcloud/ownCloud or fresh install: upgraded from NC 10

Where did you install Nextcloud from: from NC homepage

Signing status:

Signing status

No errors have been found.

Are you using encryption: no

bug external storage

Source

powerpaul17

Most helpful comment

Workaround/Script for shell-users. No warranty, tested on NC12.

#!/bin/bash

PHP="/usr/bin/php"
OCC="/var/www/nextcloud/occ"


# Admin Mountpoints
${PHP} ${OCC} files_external:list | grep "| [0-9]" | while read MOUNTLINE ; do
    TYPE=$(echo "${MOUNTLINE}" | awk -F'[ \t]*[|][ \t]*' '{ print $4 }')
    if [ "$TYPE" == "Dropbox" ] ; then
        MOUNTPOINT=$(echo "${MOUNTLINE}" | awk -F'[ \t]*[|][ \t]*' '{ print $3 }')
        # Applicable Users
        echo "${MOUNTLINE}" | awk -F'[ \t]*[|][ \t]*' '{ print $8 }' | while IFS=', ' read -ra USERNAMES ; do
            for USERNAME in "${USERNAMES[@]}" ; do
                ${PHP} ${OCC} files:scan --path="${USERNAME}/files${MOUNTPOINT}" --quiet
            done
        done
        # Applicable Groups
        echo "${MOUNTLINE}" | awk -F'[ \t]*[|][ \t]*' '{ print $9 }'| while IFS=', ' read -ra GROUPNAMES ; do
            for GROUPNAME in "${GROUPNAMES[@]}" ; do
                ${PHP} ${OCC} group:list | awk "/${GROUPNAME}:/{flag=1;next}/:/{flag=0}flag" | cut -c 7- | while read USERNAME ; do
                    ${PHP} ${OCC} files:scan --path="${USERNAME}/files${MOUNTPOINT}" --quiet
                done
            done
        done
    fi
done

# User Mointpoints
${PHP} ${OCC} user:list | awk -F'[ \t]*[-:][ \t]*' '{ print $2 }' | while read USERNAME ; do
    ${PHP} ${OCC} files_external:list $USERNAME | grep "| [0-9]" | while read MOUNTLINE ; do
        TYPE=$(echo "${MOUNTLINE}" | awk -F'[ \t]*[|][ \t]*' '{ print $4 }')
        if [ "$TYPE" == "Dropbox" ] ; then
            MOUNTPOINT=$(echo "${MOUNTLINE}" | awk -F'[ \t]*[|][ \t]*' '{ print $3 }')
            ${PHP} ${OCC} files:scan --path="${USERNAME}/files${MOUNTPOINT}" --quiet
        fi
    done
done

Currently user:list and group:list only lists 500 (default), change that to your needs.

ManOki on 13 Jul 2017

👍3

All 18 comments

@icewind1991 what was the name of the config for discovering changes?

nickvergessen on 23 Jan 2017

Interesting. Just noticed the same thing myself. I'll follow this thread

philsson on 19 Feb 2017

The same happens to me. It seems if run occ files:scan --all it updates, but the process takes like 3 hours...

borekon on 7 Mar 2017

Also following.

I tried to move some files to a subfolder in Dropbox, then came back to nextcloud to find that nextcloud did not detect the changes. When I tried to move the files from nextcloud, I get an error saying that the files already exist in the directory, but no files are visible.

heskja on 1 Jul 2017

Doing some further research yielded this Dropbox developer blog post: https://blogs.dropbox.com/developers/2013/11/low-latency-notification-of-dropbox-file-changes/

Seems like long polling with some implementation of the delta API calls would solve this. Not sure where in nextcloud code this should be added though.

heskja on 2 Jul 2017

👍2

Workaround/Script for shell-users. No warranty, tested on NC12.

#!/bin/bash

PHP="/usr/bin/php"
OCC="/var/www/nextcloud/occ"


# Admin Mountpoints
${PHP} ${OCC} files_external:list | grep "| [0-9]" | while read MOUNTLINE ; do
    TYPE=$(echo "${MOUNTLINE}" | awk -F'[ \t]*[|][ \t]*' '{ print $4 }')
    if [ "$TYPE" == "Dropbox" ] ; then
        MOUNTPOINT=$(echo "${MOUNTLINE}" | awk -F'[ \t]*[|][ \t]*' '{ print $3 }')
        # Applicable Users
        echo "${MOUNTLINE}" | awk -F'[ \t]*[|][ \t]*' '{ print $8 }' | while IFS=', ' read -ra USERNAMES ; do
            for USERNAME in "${USERNAMES[@]}" ; do
                ${PHP} ${OCC} files:scan --path="${USERNAME}/files${MOUNTPOINT}" --quiet
            done
        done
        # Applicable Groups
        echo "${MOUNTLINE}" | awk -F'[ \t]*[|][ \t]*' '{ print $9 }'| while IFS=', ' read -ra GROUPNAMES ; do
            for GROUPNAME in "${GROUPNAMES[@]}" ; do
                ${PHP} ${OCC} group:list | awk "/${GROUPNAME}:/{flag=1;next}/:/{flag=0}flag" | cut -c 7- | while read USERNAME ; do
                    ${PHP} ${OCC} files:scan --path="${USERNAME}/files${MOUNTPOINT}" --quiet
                done
            done
        done
    fi
done

# User Mointpoints
${PHP} ${OCC} user:list | awk -F'[ \t]*[-:][ \t]*' '{ print $2 }' | while read USERNAME ; do
    ${PHP} ${OCC} files_external:list $USERNAME | grep "| [0-9]" | while read MOUNTLINE ; do
        TYPE=$(echo "${MOUNTLINE}" | awk -F'[ \t]*[|][ \t]*' '{ print $4 }')
        if [ "$TYPE" == "Dropbox" ] ; then
            MOUNTPOINT=$(echo "${MOUNTLINE}" | awk -F'[ \t]*[|][ \t]*' '{ print $3 }')
            ${PHP} ${OCC} files:scan --path="${USERNAME}/files${MOUNTPOINT}" --quiet
        fi
    done
done

Currently user:list and group:list only lists 500 (default), change that to your needs.

ManOki on 13 Jul 2017

👍3

Hi, i would like to ask, if this script is working to refresh the content of the integrated dropbox account.
In my mind, the connector don't refresh the file listing from the dropbox data.
This is also a linked problem: https://help.nextcloud.com/t/desktop-client-delta-updates-to-files/7790/6
Does your script activate the dropbox connector itself, of does is only do a local filescan? (In my mind you do a local filesync for nextcloud, but not reading nor activating the dropbox connector, right?

Thank you very much.

towerlexa on 15 Aug 2017

I have this problem with an admin-added external folder. I've tried switching to a user-added external storage, hoping that until this is resolved, at least @ManOki's script will help (it doesn't appear to work for admin-added folders).

paour on 18 Sep 2017

👍1

Hi Paour,
thanks for your answer. I'll try it again as a user added external storage!
Cool!
regards, Axel

towerlexa on 18 Sep 2017

My solution, inspired by @ManOki is as follows, not that I'm on Manjaro and Nextcloud is running on Docker.

~ >>> cat /etc/systemd/system/dropbox.service

[Unit]
Description=Scan for new files on Dropbox

[Service]
Type=simple
ExecStart=/usr/bin/docker exec -i nextcloud_server_1 occ files:scan --path="/conor/files/Dropbox"

~ >>> cat /etc/systemd/system/dropbox.timer

[Unit]
Description=Scrape Dropbox Hourly

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target

ConorIA on 18 Sep 2017

👍2

@paour what are admin-added external folders? are those directly shared to all users? are they added through a plugin another app?

maybe just a hint, how this script works:
first line, all usernames are queried from nextcloud by occ user:list
second line, all mountpoint for that user are queried from nextcloud by occ files_external:list $USERNAME
for each of those user mountpoints, an refresh is triggered via occ files:scan --path="${USERNAME}/files${MOUNTPOINT}"

so if the admin-added mountpoints are not listed by the nextcloud commands mentioned above, the script will fail. but maybe we can add some lines to complete it.

EDIT: @ConorIA you hardcoded your path --path="/conor/files/Dropbox", but this is not generic for all and only supports single user

ManOki on 19 Sep 2017

@ManOki yes, the admin can allow users to add their own mappings, or can set mappings for one or several users from the admin console. These do not appear in occ files_external:list $USERNAME.

paour on 19 Sep 2017

@paour can you show me such a command line for the admin console to set a mapping, please.

ManOki on 19 Sep 2017

I'm not aware of a command-line to do this, but you can do it in the web admin console:

admin_-_nextcloud

paour on 20 Sep 2017

Thanks for the hint, i wasn't aware of that. Admin mountpoints are listed without any user_id, just occ files_external:list.
I will update my comment with the script...just checking, how to update the path without a valid username

ManOki on 20 Sep 2017

Aha, then my issue may have been that NC forgot my mountpoint: it wasn't in the list when I ran that command. It still appeared in the web UI though.

paour on 20 Sep 2017

Ok, i updated the script to support also admin mountpoints. I didn't test, if it is valid to just trigger a single update on any user. current state is, that all applicable users and groups with their users get updated, which might cause a lot of traffic (to dropbox).

@paour Maybe your mountpoint was defined a long time ago with an old owncloud/nextcloud instance, so those information are not stored properly. Now the script should work, if occ files_external:list list your mountpoint.

ManOki on 20 Sep 2017

Dropbox support was moved to a separate app, please report issues with this over there: https://github.com/icewind1991/files_external_dropbox

Please try this new app, because it uses updated libraries that support newer Dropbox APIs.

The app works with Nextcloud 12+.