Emscripten: Cannot access packaged files after memory growth

[kichikuou]

test.c:

#include <stdio.h>
#include <stdlib.h>

main() {
  FILE *fp;
  int c;

  malloc(20000000);  // Enlarge memory

  fp = fopen("test.c", "r");
  while ((c = fgetc(fp)) != EOF)
    putchar(c);
}

$ emcc -s WASM=1 -s ALLOW_MEMORY_GROWTH=1 --preload-file test.c -o test.html test.c

This throws "TypeError: attempting to access detached ArrayBuffer" when attempted to read from test.c.

It seems packaged files are stored in the heap. Probably memfs continues to hold a reference to the old heap after the memory grows?

Emscripten 1.37.9 / Firefox 53.0

[kripken]

Yes, that's correct, it does avoid copies, but looks like we didn't realize that doesn't mix well with memory growth.

It's possible that building with --no-heap-copy would make this work, in which case, we might want to just auto-enable that for memory growth.

[olafmaas]

Just ran into the same problem. Building with --no-heap-copy solved the problem for me.

Chrome will output Cannot perform TypedArrayCreate on a detached ArrayBuffer (Just mentioning this to make searching a bit more easy for other Chrome users with the same problem. :stuck_out_tongue: )

[nstoddard]

I just ran into this, and adding --no-heap-copy fixes it for me. Any chance this can be enabled by default when memory growth is enabled?

[juj]

Very good find. I'm a bit surprised as to why I haven't run into this before, even though I've used these in conjunction a lot. Though I think I may have been passing a maximum size always, which probably reserves the address space up front and does not end up detaching the buffer, so this issue is avoided then.

Proposed fix in above PR to use --no-heap-copy in that scenario.

[ztl8702]

I encountered a situation where even with --no-heap-copy enabled this error can still happen.

I was trying to port librime, a C++ library to WebAssembly. In my particular use case, because the code was never optimised for the browser, it used a lot of memory (> 128MB). I initially compiled it with -s ALLOW_MEMORY_GROWTH=1 and observed the memory size grew 4 times, from 16MB to 256MB, doubling each time. TypeError: attempting to access detached ArrayBuffer occurred after the last growth.

The codebase made use of Boost.Interprocess, and around the time that TypeError was thrown it was attempting to create a file mapping using Boost. The related code is:

file_.reset(new boost::interprocess::file_mapping(file_name.c_str(), file_mapping_mode));
region_.reset(new boost::interprocess::mapped_region(*file_, file_mapping_mode));

ref: https://github.com/rime/librime/blob/8d8d2e6c2725041552bb17c787978cf319460d44/src/rime/dict/mapped_file.cc#L54

Below is part of the logs when the error happened. _I am looking for ways to reproduce this with in simpler codebase._

Interestingly, everything works fine when I added -s TOTAL_MEMORY=256MB to the compile options. So I think it likely has to do with memory growth.

Software versions:

em++ 1.37.35
Firefox 59.0 (64-bit)
Ubuntu 17.10 64-bit

[meldron]

Hey guys,

so I had the same problem as @ztl8702 with opus/speedexdsp.

The third memory growth failed:

enlarged memory arrays from 16777216 to 33554432, took 26 ms (has ArrayBuffer.transfer? false)
enlarged memory arrays from 33554432 to 67108864, took 67 ms (has ArrayBuffer.transfer? false)
OpusEncoderWorker.js:1 Uncaught TypeError: Cannot perform %TypedArray%.prototype.set on a detached ArrayBuffer

Software Versions.

emcc 1.37.33 & 1.37.39
Chrome 66.0.3359.139
Ubuntu 16.04

So if I used -s TOTAL_MEMORY=256MB, everything works fine.

Are there any drawbacks for using such a big TOTAL_MEMORY?

[donthomasitos]

Same problem here. I get

"TypeError: Cannot perform Construct on a detached ArrayBuffer"

with ALLOW_MEMORY_GROWTH, --no-heap-copy does not change anything. It occurs on a file read operation on a packaged file. Any idea of what I could try?

[kripken]

@Inreal The stack trace might help figure that out - the question is where a reference to a detached ArrayBuffer has remained.

[donthomasitos]

Does that help? That's what i get on -g3 in Google Chrome console. @kripken

TypeError: Cannot perform Construct on a detached ArrayBuffer
at new Uint8Array ()
at Uint8Array.subarray ()
at Object.read (enscape.js:2633)
at Object.read (enscape.js:4343)
at Object.doReadv (enscape.js:5169)
at ___syscall145 (enscape.js:5246)
at ___stdio_read (wasm-function[6869]:119)
at _fread (wasm-function[7038]:201)
at __ZNSt3__213basic_filebufIcNS_11char_traitsIcEEE9underflowEv (wasm-function[3020]:370)
at dynCall_ii (wasm-function[7952]:13)

enscape.html:179 TypeError: Cannot perform Construct on a detached ArrayBuffer
at new Uint8Array ()
at Uint8Array.subarray ()
at Object.read (enscape.js:2633)
at Object.read (enscape.js:4343)
at Object.doReadv (enscape.js:5169)
at ___syscall145 (enscape.js:5246)
at ___stdio_read (wasm-function[6869]:119)
at _fread (wasm-function[7038]:201)
at __ZNSt3__213basic_filebufIcNS_11char_traitsIcEEE9underflowEv (wasm-function[3020]:370)
at dynCall_ii (wasm-function[7952]:13)
printErr @ enscape.html:179

enscape.html:179 exception thrown: abort({}) at Error
at jsStackTrace (http://localhost:6931/enscape.js:1040:13)
at stackTrace (http://localhost:6931/enscape.js:1057:12)
at abort (http://localhost:6931/enscape.js:14545:44)
at ___syscall145 (http://localhost:6931/enscape.js:5248:69)
at ___stdio_read (wasm-function[6869]:119)
at _fread (wasm-function[7038]:201)
at __ZNSt3__213basic_filebufIcNS_11char_traitsIcEEE9underflowEv (wasm-function[3020]:370)
at dynCall_ii (wasm-function[7952]:13)
at Object.Module.dynCall_ii (http://localhost:6931/enscape.js:13989:38)
at invoke_ii (http://localhost:6931/enscape.js:10747:32)

Uncaught abort({}) at Error
at jsStackTrace (http://localhost:6931/enscape.js:1040:13)
at stackTrace (http://localhost:6931/enscape.js:1057:12)
at abort (http://localhost:6931/enscape.js:14545:44)
at ___syscall145 (http://localhost:6931/enscape.js:5248:69)
at ___stdio_read (wasm-function[6869]:119)
at _fread (wasm-function[7038]:201)
at __ZNSt3__213basic_filebufIcNS_11char_traitsIcEEE9underflowEv (wasm-function[3020]:370)
at dynCall_ii (wasm-function[7952]:13)
at Object.Module.dynCall_ii (http://localhost:6931/enscape.js:13989:38)
at invoke_ii (http://localhost:6931/enscape.js:10747:32)

[kripken]

Thanks, I think the interesting part is at at Object.read (enscape.js:2633). What function is on that line?

(Also maybe at Object.read (enscape.js:4343) matters too.)

[donthomasitos]

Ok, these ones are the code snippets:

Line 4343 links to that one:

Thanks for having a look @kripken :)

[kripken]

Thanks @Inreal. My best guess is that we keep a reference to the main buffer through the canOwn parameter being true in a call to write in library_memfs.js. We should probably not allow that to be true in memory growth.

I'll run the test suite to see if we have coverage for that code path - if not, we'll need to create a testcase for it. If you have a simple one (in the format of the existing tests in the test suite) that would be very useful.

[kripken]

There was indeed a relevant test. I extended it and opened a PR with a fix, #7142.