Emscripten: Using fetch to directly write a file into a mounted volume doesn’t work

[diegurrio]

When we fetch to a file the contents are not written to our mounted volume. What we see is that it in fact gets written to another (default?) volume. As a workaround, what we are doing is fetching to memory, and then writing it out which ends up in our volume. It would be great to be able to fetch directly to our mounted volume. We have attached the file that isolates and duplicates the issue we're seeing.

Example:

// Mount volume
EM_ASM(
  FS.mkdir('/test');
  // Then mount with IDBFS type
  FS.mount(IDBFS, {}, '/test');
  // Then sync
  FS.syncfs(true, function (err) {
  // Error
  assert(!err);
  });
);


emscripten_fetch_attr_t attr;
emscripten_fetch_attr_init(&attr);
strcpy(attr.requestMethod, "GET");
attr.attributes = EMSCRIPTEN_FETCH_PERSIST_FILE | EMSCRIPTEN_FETCH_LOAD_TO_MEMORY | EMSCRIPTEN_FETCH_SYNCHRONOUS;
attr.destinationPath = "/test/fetch.h";
emscripten_fetch_t* fetch1 = emscripten_fetch(&attr, "http://raw.githubusercontent.com/emscripten-core/emscripten/master/system/include/emscripten/fetch.h"); // Blocks here until the operation is complete.
if (fetch1->status == 200) {
    printf("Finished downloading");
        std::ifstream myFile ("/test/fetch.h");
        if (myFile.good()) {
        printf("File exists");
        } else {
        printf("File does not exists");
        }
} else {
    printf("Downloading %s failed, HTTP failure status code: %d.\n", fetch1->url, fetch1->status);
}
emscripten_fetch_close(fetch1);

[pmarchini]

@diegurrio from the code above you are expecting to fetch and directly write into /test/fetch.h file into your volume, am I right?

[diegurrio]

@PietroMarchini94 you are correct. I was hoping that to be the behavior of the API. Instead the file ends up in a different volume (maybe a default?).

[pmarchini]

@diegurrio atm fetch library is puting the cached data into the IndexedDB directly from JS (Fetch.js)

  try {
    var transaction = db.transaction(['FILES'], 'readwrite');
    var packages = transaction.objectStore('FILES');
    var putRequest = packages.put(data, destinationPathStr);
    putRequest.onsuccess = function(event) {
#if FETCH_DEBUG
      console.log('fetch: Stored file "' + destinationPathStr + '" to IndexedDB cache.');
#endif
      HEAPU16[fetch + {{{ C_STRUCTS.emscripten_fetch_t.readyState }}} >> 1] = 4; // Mimic XHR readyState 4 === 'DONE: The operation is complete'
      HEAPU16[fetch + {{{ C_STRUCTS.emscripten_fetch_t.status }}} >> 1] = 200; // Mimic XHR HTTP status code 200 "OK"
      stringToUTF8("OK", fetch + {{{ C_STRUCTS.emscripten_fetch_t.statusText }}}, 64);
      onsuccess(fetch, 0, destinationPathStr);
    };

For this reason file system is not getting involved.
In fact the file is beign saved into default FILES indexDB

It seems, more than a bug, a normal behavior of the API and the cleanest way to save into FS seems to be to write out the file into the FS onsuccess as you're doing.
If I'm right the only way to achive a direct write from the API it's to add a new attribute and a new feature to the emiscripten_fetch.cpp/fetch.js library.

[diegurrio]

Yes, we figured it's the normal behavior. It would be extremely useful to do direct writes, even if we need to specify a new argument. :)

[pmarchini]

@kripken Do you think that fetch directly to FS could be an improvement?
In case, if you don't mind, I'd like to take a look.

[kripken]

@pmarchini In general yes, but it might not be the best time for that - I believe @juj has plans to work soon on fetch on top of a new backend, which might mean some large changes. Could be best to wait for that, or coordinate to make sure the two wouldn't conflict.

[juj]

Indeed Fetch does not operate together with the old JavaScript-based MEMFS (src/library_fs.js, src/library_memfs.js etc.), but the design of that was to phase out MEMFS, so the Fetch API would operate orthogonally/independently of MEMFS.