Maps: manual scanphoto terminated

[nextgen-networks]

Expected:
manual Scan adds photos to map

actual:
scan & add is termininated with error

Setup:

• Debian 18.04. with with official updates until 2019-10-14
• PHP 7.3.10
• MariaSQL 10.1.41
• Webserver Apache & nginx proxy
• NC: 16.05
• 8 CPU Cores
• 16 GB RAM

Used CMD:
occ $ maps:scan-photos

CMD Output:
...
[1274] Photo "/USERNAME1/files/IMG_20180723_203516.jpg" added
[1275] Photo "/USERNAME1/files/IMG_20180723_203539.jpg" added
[1276] Photo "/USERNAME1/files/IMG_20180723_203555.jpg" added
[1277] Photo "/USERNAME1/files/IMG_20180723_203616.jpg" added
======== User USERNAME2 ========
[1] Photo "/USERNAME2/files/Photos/Coast.jpg" added
[2] Photo "/USERNAME2/files/Photos/Hummingbird.jpg" added
[3] Photo "/USERNAME2/files/Photos/Nut.jpg" added
======== User USERNAME3 ========
"\n\u001b[33mIn Availability.php line 90:\u001b[39m\n\u001b[37;41m \u001b[39;49m\n\u001b[37;41m [OCP\Files\StorageNotAvailableException (1)] \u001b[39;49m\n\u001b[37;41m \u001b[39;49m\n\n\u001b[32mmaps:scan-photos [\u001b[39m\u001b[32m\u003Cuser_id\u003E\u001b[39m\u001b[32m]\u001b[39m\n\n"

NexcloudLog:
[PHP] Error: Cannot modify header information - headers already sent by (output started at /var/www/vhosts/SUBSCRIPTIONNAME1/DOMAINNAME1/apps/maps/lib/Command/RescanPhotos.php:75) at /var/www/vhosts/SUBSCRIPTIONNAME1/DOMAINNAME1/lib/private/AppFramework/Http/Output.php#69

POST /index.php/apps/occweb/cmd
from MYipADRESS1 by USERNAME3 at 2019-10-15T11:25:32+00:00

[vwbusguy]

I'm seeing this now, but only on the user with MFA enabled.
With -vvvv:

In Storage.php line 341:

  [OCP\Files\StorageNotAvailableException (1)]  


Exception trace:
  at /srv/nextcloud/apps/files_sharing/lib/External/Storage.php:341
 OCA\Files_Sharing\External\Storage->getShareInfo() at /srv/nextcloud/apps/files_sharing/lib/External/Storage.php:211
 OCA\Files_Sharing\External\Storage->checkStorageAvailability() at n/a:n/a
 call_user_func_array() at /srv/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php:506
 OC\Files\Storage\Wrapper\Wrapper->__call() at n/a:n/a
 call_user_func_array() at /srv/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php:506
 OC\Files\Storage\Wrapper\Wrapper->__call() at /srv/nextcloud/apps/files_sharing/lib/External/Scanner.php:89
 OCA\Files_Sharing\External\Scanner->scanAll() at /srv/nextcloud/apps/files_sharing/lib/External/Scanner.php:44
 OCA\Files_Sharing\External\Scanner->scan() at /srv/nextcloud/lib/private/Files/Cache/Watcher.php:105
 OC\Files\Cache\Watcher->update() at /srv/nextcloud/lib/private/Files/View.php:1344
 OC\Files\View->getCacheEntry() at /srv/nextcloud/lib/private/Files/View.php:1445
 OC\Files\View->getDirectoryContent() at /srv/nextcloud/lib/private/Files/Node/Folder.php:100
 OC\Files\Node\Folder->getDirectoryListing() at /var/lib/nextcloud/apps/maps/lib/Service/PhotofilesService.php:322
 OCA\Maps\Service\PhotofilesService->gatherPhotoFiles() at /var/lib/nextcloud/apps/maps/lib/Service/PhotofilesService.php:332
 OCA\Maps\Service\PhotofilesService->gatherPhotoFiles() at /var/lib/nextcloud/apps/maps/lib/Service/PhotofilesService.php:67
 OCA\Maps\Service\PhotofilesService->rescan() at /var/lib/nextcloud/apps/maps/lib/Command/RescanPhotos.php:80
 OCA\Maps\Command\RescanPhotos->rescanUserPhotos() at /var/lib/nextcloud/apps/maps/lib/Command/RescanPhotos.php:71
 OCA\Maps\Command\RescanPhotos->execute() at /srv/nextcloud/3rdparty/symfony/console/Command/Command.php:255
 Symfony\Component\Console\Command\Command->run() at /srv/nextcloud/3rdparty/symfony/console/Application.php:915
 Symfony\Component\Console\Application->doRunCommand() at /srv/nextcloud/3rdparty/symfony/console/Application.php:272
 Symfony\Component\Console\Application->doRun() at /srv/nextcloud/3rdparty/symfony/console/Application.php:148
 Symfony\Component\Console\Application->run() at /srv/nextcloud/lib/private/Console/Application.php:214
 OC\Console\Application->run() at /srv/nextcloud/console.php:99
 require_once() at /srv/nextcloud/occ:11

maps:scan-photos [<user_id>]

I think that the auth is failing for the user with MFA enabled:
``` public function getShareInfo() {
$remote = $this->getRemote();
$token = $this->getToken();
$password = $this->getPassword();

            // If remote is not an ownCloud do not try to get any share info
            if(!$this->remoteIsOwnCloud()) {
                    return ['status' => 'unsupported'];
            }

            $url = rtrim($remote, '/') . '/index.php/apps/files_sharing/shareinfo?t=' . $token;

            // TODO: DI
            $client = \OC::$server->getHTTPClientService()->newClient();
            try {
                    $response = $client->post($url, [
                            'body' => ['password' => $password],
                            'timeout' => 10,
                            'connect_timeout' => 10,
                    ]);
            } catch (\GuzzleHttp\Exception\RequestException $e) {
                    if ($e->getCode() === Http::STATUS_UNAUTHORIZED || $e->getCode() === Http::STATUS_FORBIDDEN) {
                            throw new ForbiddenException();
                    }
                    if ($e->getCode() === Http::STATUS_NOT_FOUND) {
                            throw new NotFoundException();
                    }
                    // throw this to be on the safe side: the share will still be visible
                    // in the UI in case the failure is intermittent, and the user will
                    // be able to decide whether to remove it if it's really gone
                    throw new StorageNotAvailableException();
            }

            return json_decode($response->getBody(), true);
    }

```

[tacruc]

@nextgen-networks is this only happening when using occ-web? Is it crashing, if you run it from the comand line?

[vwbusguy]

Yes, it crashes from command-line, but only for the user with MFA. The logs from my previous comment were from running sudo -u apache php ./occ maps:scan-photos

[tacruc]

Are you using some kind of external storages, or group folders?
As I cannot reporduce the error.
I assume MFA = Multi factor authentification?

[vwbusguy]

I'm not using external storages. There are shared folders between instance users.
Yes, multi-factor authentication. The reason why I ask is that the only account it's failing on is the only one that has MFA enabled and the error where it fails indicates and authentication failure on the API call.

[vwbusguy]

I thought I found a potential clue. The database had an old Google Drive connection back from several years ago that didn't show up in the web interface. I removed that orphaned storage and the references to it in the filecache table and re-ran occ files:cleanup, but even after doing that, it didn't fix the error. Nothing was returned with occ files_external:list -a before or after doing this, so this might be a red herring.

What's weird is that the maps app has no problem getting location data for newly added photos, but I'm unable to get the maps app to scan the existing photos for my user.

[tacruc]

Do you still get the same error?

[vwbusguy]

Yes. Still getting it as of NextCloud 19.0.1 and maps 0.1.6.

Newly uploaded photos get properly added, but scan-photos fails without adding any.

[root@Tauriel nextcloud]# sudo -u apache php ./occ maps:scan-photos scott -vvv
======== User scott ========

In Storage.php line 342:

  [OCP\Files\StorageNotAvailableException (1)]  


Exception trace:
  at /srv/nextcloud/apps/files_sharing/lib/External/Storage.php:342
 OCA\Files_Sharing\External\Storage->getShareInfo() at /srv/nextcloud/apps/files_sharing/lib/External/Storage.php:212
 OCA\Files_Sharing\External\Storage->checkStorageAvailability() at n/a:n/a
 call_user_func_array() at /srv/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php:507
 OC\Files\Storage\Wrapper\Wrapper->__call() at n/a:n/a
 call_user_func_array() at /srv/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php:507
 OC\Files\Storage\Wrapper\Wrapper->__call() at /srv/nextcloud/apps/files_sharing/lib/External/Scanner.php:90
 OCA\Files_Sharing\External\Scanner->scanAll() at /srv/nextcloud/apps/files_sharing/lib/External/Scanner.php:45
 OCA\Files_Sharing\External\Scanner->scan() at /srv/nextcloud/lib/private/Files/Cache/Watcher.php:106
 OC\Files\Cache\Watcher->update() at /srv/nextcloud/lib/private/Files/View.php:1343
 OC\Files\View->getCacheEntry() at /srv/nextcloud/lib/private/Files/View.php:1444
 OC\Files\View->getDirectoryContent() at /srv/nextcloud/lib/private/Files/Node/Folder.php:101
 OC\Files\Node\Folder->getDirectoryListing() at /var/lib/nextcloud/apps/maps/lib/Service/PhotofilesService.php:322
 OCA\Maps\Service\PhotofilesService->gatherPhotoFiles() at /var/lib/nextcloud/apps/maps/lib/Service/PhotofilesService.php:332
 OCA\Maps\Service\PhotofilesService->gatherPhotoFiles() at /var/lib/nextcloud/apps/maps/lib/Service/PhotofilesService.php:67
 OCA\Maps\Service\PhotofilesService->rescan() at /var/lib/nextcloud/apps/maps/lib/Command/RescanPhotos.php:80
 OCA\Maps\Command\RescanPhotos->rescanUserPhotos() at /var/lib/nextcloud/apps/maps/lib/Command/RescanPhotos.php:71
 OCA\Maps\Command\RescanPhotos->execute() at /srv/nextcloud/3rdparty/symfony/console/Command/Command.php:255
 Symfony\Component\Console\Command\Command->run() at /srv/nextcloud/3rdparty/symfony/console/Application.php:1012
 Symfony\Component\Console\Application->doRunCommand() at /srv/nextcloud/3rdparty/symfony/console/Application.php:272
 Symfony\Component\Console\Application->doRun() at /srv/nextcloud/3rdparty/symfony/console/Application.php:148
 Symfony\Component\Console\Application->run() at /srv/nextcloud/lib/private/Console/Application.php:215
 OC\Console\Application->run() at /srv/nextcloud/console.php:100
 require_once() at /srv/nextcloud/occ:11

maps:scan-photos [<user_id>]

[vwbusguy]

I had a chance today to look at it and I was able to hack together a workaround. It should probably minimally have some kind of logger warning instead, but this seems to have fixed it:

In gatherPhotoFiles() in lib/Service/PhotofilesService.php:

                try{
                        $notes = array_merge($notes, $this->gatherPhotoFiles($node, $recursive));
                }catch (\OCP\Files\StorageNotAvailableException $e){
                        continue;
                }catch (Exception $e){
                        continue;
                }
                 continue;

[vwbusguy]

If you want, I can submit a PR for it?

[vwbusguy]

Let me know if I can be of any more help?

[tacruc]

Yeah, the PR would be great. Sorry, I was cycling the last two week's and haven't got though all emails yet.

[vwbusguy]

I'm actually away on vacation this week. It might be next week before I can get one tested, but I'm happy to submit one when I get back.

[vwbusguy]

It looks like there's a similar issue with importing tracks (not sure if there's a separate bug for it) but I should be able to patch both with the same logic.

[vwbusguy]

It looks like gatherTrackFiles() in lib/Service/TracksService.php is where the other is happening. In both cases, we just need to log the exception as a warning and move on.

[vwbusguy]

This fixes it in TracksService.php (Note that the SotrageNotAvailableException seems to need to be explicitly added. It's been a minute since I've hacked PHP and I'm not used to PHP7 namespaces.)

                try {
                        $notes = array_merge($notes, $this->gatherTrackFiles($node, $recursive));
                        continue;
                }catch (\OCP\Files\StorageNotAvailableException $e){
                        continue; 
                }catch (Exception $e){
                        continue;
                }

[vwbusguy]

I think I have it pretty well condensed. I'll work on porting this to a PR for both:

        foreach ($nodes as $node) {
            if ($node->getType() === FileInfo::TYPE_FOLDER AND $recursive) {
                try {
                        $notes = array_merge($notes, $this->gatherTrackFiles($node, $recursive));
                }catch (\OCP\Files\StorageNotAvailableException | \Exception $e){
                        $msg = "WARNING: Could not access " . $node->getName();
                        echo($msg . "\n");
                        $this->logger->error($msg);
                }
                continue;
            }
            if ($this->isTrack($node)) {
                $notes[] = $node;
            }
        }