Video-hub-app: Use fs.Dirent objects, and extract metadata in the background

[cal2195]

Currently, the app freeze while performing the first phase of the import (listing all the files recursively).

A large portion of this delay in phase 1 is due to stating all the files to see if they are directories. Alternatively using fs.Dirent objects, the following code would be much better:

// Recursively walk through a directory compiling ImageElements
  const walkSync = (dir, filelist) => {
    const files = fs.readdirSync(dir, {encoding: 'utf8', withFileTypes: true});

    files.forEach(function (file) {
      if (file.name) {
        if (!fileSystemReserved(file.name)) {
          // if the item is a _DIRECTORY_
          if (file.isDirectory()) {
            filelist = walkSync(path.join(dir, file.name), filelist);
          } else {
            const extension = file.name.split('.').pop();
            if (acceptableFiles.includes(extension.toLowerCase())) {
              // before adding, remove the redundant prefix: sourceFolderPath
              const partialPath = dir.replace(sourceFolderPath, '');
              // fil finalArray with 3 correct and 5 dummy pieces of data
              finalArray[elementIndex] = [partialPath, file.name, cleanUpFileName(file.name), '', 0, '', 0, 0];
              elementIndex++;
            }
          }
        }
      }
    });

See https://nodejs.org/api/fs.html#fs_fs_readdirsync_path_options

This will be a drastic improvement in speed, especially over networked file systems. Unfortunately, this is only added in node v10.10.0, which will need to be updated first.

Additionally, it would be much better if the second phase (metadata extraction) was performed in the background like phase 3 (thumbnail generation). This is especially important with the addition of file hashes, which will require reading every file.

[whyboris]

Thank you!

I'm happy to update to _node v10_ and to this new approach -- will do so along with my _angular7_ branch 👍

ps -- some users have requested we ignore files that start with . and with _ which we'll do as well 👌

[whyboris]

_master_ has been updated with _node 10.14_ so this is ready for a PR -- whenever you'd like 🥇

[cal2195]

Darn, looks like electron 3.0.10 only has node v10.2.0 - https://electronjs.org

Unless I'm wrong, this will have to wait until electron 4?

[whyboris]

I think you're right - https://electronjs.org/releases#3.0.0 electron 3 comes with Node v10.2.0
I suspect _electron 4_ is coming out soon - before we have our next release 👌

[whyboris]

A better way to see which _node_ version is in which _Electron_ version:
https://github.com/electron/releases

Looking at how the v3.0.0 was released, once they are done with _beta_ versions, they have a few _nightly_ versions and then release it for good. Feels like we'll get v4.0.0 around New Year's 🎄

[cal2195]

Nice! 🎉

I noticed you tried to use v4 in your original upgrade - was there any problems, and would it be possible to have a beta branch to test it out?

[whyboris]

It's worth a try -- feel free to give it a go: electron 4 beta 10 came out 30 minutes ago:
https://github.com/electron/electron/releases/tag/v4.0.0-beta.10
I tried it before but _something_ wasn't working (could have been something else), so I just rolled back to decrease the number of moving pieces, fixed the problem, and never tried with the version 4 beta 🐟

edit: tried the beta 10 and the app wouldn't compile:

internal/validators.js:125
    throw new ERR_INVALID_ARG_TYPE(name, 'string', value);
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "file" argument must be of type string. Received type object
    at validateString (internal/validators.js:125:11)

I'm happy to just wait for the non-beta release 👌

[whyboris]

When working on this, should be easy to close this issue too: https://github.com/whyboris/Video-Hub-App/issues/24 👍

[cal2195]

I just tried beta 11 there on master and it works! 🎉

[whyboris]

Electron v4.0 is out as of 1 hour ago 🎉
https://github.com/electron/electron/releases/tag/v4.0.0
We'll upgrade in a few days 👍 -- or feel free to open up this change as a PR with the version bump 💯

[cal2195]

_A wild PR has appeared!_ 🍃 #33