Flow: Crash on Apple Silicon

[STRML]

Flow version: 0.138

Expected behavior

Flow should run as normal under Rosetta 2.

Actual behavior

$ flow check
Out of shared memory:
Raised by primitive operation at file "src/heap/sharedMem.ml", line 55, characters 25-49

This may have to do with a quirk in Rosetta 2 / shared memory. I am not familiar enough with the codebase to know for sure, but would be happy to investigate if given some pointers.

Building a proper aarch64 version is stalled in https://github.com/facebook/flow/pull/8418/files which seems to simply need a review. If that can be done, we can get this into flow_bin and perhaps the problem will be solved.

Edit:

Fix from @atulsmadhugiri:

[options]
sharedmemory.heap_size=3221225472

[alexgurr]

This is a huge blocker for our team, running new macs. Would be great to see this resolved.

[DMXL]

Same here running Flow for VSCode, this is the log:

usage: sudo dmesg
Dec 11 13:15:00.778 [error] Flow server (pid 45797) exited with code Out_of_shared_memory (15)
Dec 11 13:15:00.778 [info] Creating a new Flow server
Dec 11 13:15:00.782 [info] Spawned server #109 (pid=45801)
Dec 11 13:15:00.831 [error] Connection 'server #109' was closed from the other side
Unable to obtain kernel buffer: Operation not permitted

Flow version: 0.135/0.140

[gkz]

We're looking into getting Macs with Apple silicon for the team (they are not yet standard issue for us).

[STRML]

If anyone is committed to fixing this, I'll happily ship them one. Email me. See my contact info at STRML.net.

[Jasperrr91]

Same here, impossible to get this working

Out of shared memory: Raised by primitive operation at file "src/heap/sharedMem.ml", line 43, characters 6-40

[BotaniQue]

Same, really waiting for an update...
The project is unsupportable on M1 Macs

[STRML]

Bounty is still claimable...

On Sat, Jan 2, 2021 at 5:26 PM Evgeny Korotenko [e-mail]
wrote:

Same, really waiting for an update...
The project is unsupportable on M1 Macs

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/facebook/flow/issues/8538#issuecomment-753537615, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJEKP7FTJC4R74VLAOVFULSX6MSNANCNFSM4T3QWJDQ
.

[chapati23]

There's no way to get around this? e.g. it's possible to run commands under rosetta with arch -x86_64 /bin/bash -c "yarn flow" — which works for most of my CLIs (e.g. some homebrew packages) that aren't yet natively M1-compatible

...except for flow 😞

[chapati23]

It's not even working inside a Docker container, really scratching my head about this 🤔

https://code.visualstudio.com/docs/remote/containers

[atulsmadhugiri]

I was able to get around the Out of shared memory issue by setting sharedmemory.heap_size to 3-4GB in .flowconfig. (The default is 25GB according to the docs)

It ended up looking something like:

[options]
sharedmemory.heap_size=3221225472

Hope this works for others.

[jtomaszewski]

It seems you're right @atulsmadhugiri, it works for me as well!

Although it didn't work on flow 0.90 . After installing the recent version (0.141.0), it started working. So beware, you might need to update your flow to make it happen.

EDIT: It works on flow 0.105 too. So somewhere between 0.90-0.105 there's a commit that makes it work on Macbooks M1 (with the sharedmemory.heap_size line in the flowconfig).

[STRML]

How about that, it works. Nice find @atulsmadhugiri!

Interestingly, 3GB seems to be the magic number. 4GB creates the same error. Unsure how if this will cause problems on larger codebases, I will try.

[Jasperrr91]

Confirmed working on 0.124.0. Great find @atulsmadhugiri, much appreciated!

[maxsalven]

Interestingly, 3GB seems to be the magic number. 4GB creates the same error. Unsure how if this will cause problems on larger codebases, I will try.

sharedmemory.heap_size=4000000000 works on my 16GB M1 MBP with a non trivial codebase.

I managed to get a native binary of flow built as well, it took my cold flow start time down from 56s to 45s.

Have attached the binary for Flow 0.141.0 if anyone is interested.
flow.zip

[radex]

@maxsalven Hi! Do you have a quick tip on how to get around OCaml shouting that it doesn't support arm64, and if you change the ocaml version, then that beta versions of ocaml are not supported? (FWIW, I know how to compile Flow for x86_64, but have ~zero familiarity with OCaml tooling… But would like to be able to build it myself until official builds are available)

[Jasperrr91]

4221225472

Interestingly, 3GB seems to be the magic number. 4GB creates the same error. Unsure how if this will cause problems on larger codebases, I will try.

sharedmemory.heap_size=4000000000 works on my 16GB M1 MBP with a non trivial codebase.

I managed to get a native binary of flow built as well, it took my cold flow start time down from 56s to 45s.

Have attached the binary for Flow 0.141.0 if anyone is interested.
flow.zip

4221225472 seems to be the limit on my 16GB, from 43... upwards it crashes.

[maxsalven]

@maxsalven Hi! Do you have a quick tip on how to get around OCaml shouting that it doesn't support arm64, and if you change the ocaml version, then that beta versions of ocaml are not supported? (FWIW, I know how to compile Flow for x86_64, but have ~zero familiarity with OCaml tooling… But would like to be able to build it myself until official builds are available)

Similarly I've never used OCaml. I was messing around with builds a few weeks ago to try get Flow to work on the M1. Unfortunately I can't recall everything I did, but it looks like I ended up having the following installed via homebrew:

ocaml HEAD-f9d3c21
opam 2.0.7

And then I had to make a few changes to the flow repo:
0001-Fix-build.patch.txt

The build instructions are here:
https://github.com/facebook/flow#building-flow-from-source

Let me know if you can't get it working, and then I'll try create some exact reproduction steps.

[EduardoRFS]

If you all pin OCaml to 4.10.2 it should build, there is also an opam version for Darwin ARM64. 4.11 don't have support for Darwin ARM64

[Ashoat]

Does anybody know of a way to set the sharedmemory.heap_size config conditionally for M1 Macs only? I'd like to avoid limiting heap size for everybody on our team.

[mroch]

Would someone mind trying the binary from #8571? https://38546-25880891-gh.circle-artifacts.com/0/flow-osx.zip

I added a bit more logging so we can see which syscall is failing.

Interested in any "Out of shared memory" errors in /tmp/flow/*.log

[jaimeagudo]

@Ashoat you can pass the --sharedmemory-heap-size option to the cli. Check flow --help

[jaimeagudo]

I got [1] 47961 segmentation fault straight away and no log file is created at all.

Restoring

sharedmemory.heap_size=2147483648

to my .flowconfig seems to work though @mroch

[radex]

@mroch this is failing: https://github.com/facebook/flow/blob/master/src/heap/hh_shared.c#L541 with EINVAL (sorry, didn't test your build, but I've already established in my own debugging this earlier, before I gave up). What's weird about it is memfd_init looks to be pretty much an isolated piece of code, but when I compiled it by itself, it ran without any trouble. So something weird seems to be taking place before ftruncate that causes it to fail.

[jaimeagudo]

yeah, my regular flow binary moans about it too @radex @mroch

[2021-01-21 14:30:15.705] argv=./node_modules/flow-bin/flow-osx-v0.115.0/flow start --flowconfig-name .flowconfig --temp-dir /tmp/flow .
[2021-01-21 14:30:15.705] lazy_mode=off
[2021-01-21 14:30:15.705] arch=classic
[2021-01-21 14:30:15.705] abstract_locations=off
[2021-01-21 14:30:15.705] max_workers=8
Unhandled exception: Unix.Unix_error(Unix.EINVAL, "ftruncate", "")
Raised by primitive operation at file "src/heap/sharedMem.ml", line 62, characters 41-90
Called from file "src/heap/sharedMem.ml", line 107, characters 6-40

[smorimoto]

If anyone is interested, I'm going to publish a Flow build for M1 on my fork, is anyone interested in that?

[jaimeagudo]

Great stuff @smorimoto, please share a link when you have it, thanks! 🙇‍♂️

[smorimoto]

I've built a few recent versions, so you can download whatever you want.

• https://github.com/smorimoto/flow/commits/darwin-arm64-v0.143.1
  
  
  
  • https://cirrus-ci.com/task/5010814929207296
  
  
• https://github.com/smorimoto/flow/commits/darwin-arm64-v0.143.0
  
  
  
  • https://cirrus-ci.com/task/4963621694341120
  
  
• https://github.com/smorimoto/flow/commits/darwin-arm64-v0.142.0
  
  
  
  • https://cirrus-ci.com/task/6292228823842816
  
  
• https://github.com/smorimoto/flow/commits/darwin-arm64-v0.141.0
  
  
  
  • https://cirrus-ci.com/task/5283052441042944
  
  
• https://github.com/smorimoto/flow/commits/darwin-arm64-v0.140.0
  
  
  
  • https://cirrus-ci.com/task/5999232463142912
  
  

[smorimoto]

Please let me know if you need a different version. I can build it for you btw.

[smorimoto]

@mroch If the Facebook team needs a test infrastructure for M1, I can lend you my spare infrastructure for free. Please feel free to contact me.

[jaimeagudo]

Thanks again for the contribution @smorimoto ! I work with a huge codebase we can't easily upgrade to any of these, we are stuck on v0.115.0, if you could trigger and share a build on that version I'll buy you a 🍺 😄 .

[petrkrulis]

Nice @smorimoto! We are stuck with 0.80.0 (upgrading means 1-2 weeks of work), so that would be another 🍺😁

[smorimoto]

Note that the problem here seems to be that certain syscalls are not responded correctly from the kernel. This means that even if we use a native build, the exact same problem will continue to occur. I'm going to fix it when I have spare time. (But @mroch knows more about Flow codebase.) And this will probably be fixed in the next kernel update, so it may be fixed over time.

[smorimoto]

@jaimeagudo @peduarte Thanks for your kind offer! I will publish that builds later, but I'm not old enough to legally drink beer in Japan yet :)

[jaimeagudo]

aha, white hacker! 😄 Ok, so you are now invited to 🇪🇸 to do your ocaml stuff and drink 🍺 at the ☀️ 😄 (assuming you are 16+ lol)

[smorimoto]

Oh, I definitely need to go there!

[smorimoto]

I planned to build them but building older versions with newer compiler versions is often tricky. So I'm trying to solve this issue from a different aspect now.

[samwgoldman]

Unfortunately, this appears to be a macOS bug where the POSIX shared memory system didn't handle the change from 4KB to 16KB page size. We've opened a radar with Apple. Until this can be resolved, you can set sharedmemory.heap_size=4294950912 which is the maximum that won't trigger the bug.

[smorimoto]

@samwgoldman Yeah, I've heard that it will be fixed in an upcoming update, but I'm not sure.

[smorimoto]

After all, I haven't had time to try, but I think watching the latest RC releases is something we can do.

[Ashoat]

Unfortunately, this appears to be a macOS bug where the POSIX shared memory system didn't handle the change from 4KB to 16KB page size. We've opened a radar with Apple. Until this can be resolved, you can set sharedmemory.heap_size=4294950912 which is the maximum that won't trigger the bug.

Note that @Jasperrr91 and @atulsmadhugiri have reported that 4221225472 is the max for 16 GiB M1 models (below that it crashes)

[gualopezb]

I'm using flow v0.75.0, and the option for tweaking the shared memory heap size is not available either through the .flowconfig file or as a CLI argument. The minimal version that supports this variable is v0.100.0 but I'm trying to avoid doing so since I'd need to deal with a bunch of errors that are not caught in the v0.75.0. Has anyone run also into this situation?

[petrkrulis]

@gualopezb Yes, that's pretty much our situation. Updating is not an option because of the breaking changes in flowtype.

[STRML]

Not to derail this conversation, but check out https://medium.com/flow-type/upgrading-flow-codebases-40ef8dd3ccd8 to simplify your upgrade.

Otherwise I would suggest you fork, backport the option, and republish to your own registry. Hardly a low-effort fix.

[samwgoldman]

Okay, I've confirmed that this bug is resolved in macOS 11.3 beta 3, which is currently in public beta. Unfortunately, since this is a OS bug, people will need to use the workaround (setting sharedmemory.heap_size) until they are able to upgrade. Thanks for your patience, all.

[radex]

@samwgoldman Thanks for the update! On a related topic, we get an OCaml update merged ( https://github.com/facebook/flow/pull/8616 ) so that we can start making native Flow builds? There doesn't seem to be any issues I could find with arm builds, and I understand if Facebook doesn't have the right CD infra right now to make native builds, but this patch allows Homebrew to build Flow (both from source and via Homebrew's binary bottle service).

On my repo, this makes Flow 35% faster :)

BTW: I published my arm64 builds as @nozbe/flow-bin

[mroch]

On my repo, this makes Flow 35% faster :)

wow, nice!

the problem with our CI/CD is that Circle doesn't offer M1 executors, which makes testing impossible. we're looking into cross-compiling to at least get builds, and are adding a linux-arm64 build as a slightly better proxy. how does homebrew build for M1, I wonder?

[radex]

how does homebrew build for M1, I wonder?

I'm not entirely sure (maybe a Homebrew maintainer could say, @carlocab perhaps?), but I _think_ they use their own Mac Minis (donated by Apple) colocated on MacStadium, and use GitHub Actions' self-hosted runners.

(BTW: I personally highly recommend using GH Actions self-hosted for any team doing a non-trivial project… Unlike something like Jenkins, self-hosting GH Actions is almost trivial, and the benefit of having very, very fast builds and ease of debugging makes the little maintenance pain well worth it!)

[carlocab]

they use their own Mac Minis (donated by Apple) colocated on MacStadium

This is roughly correct. We originally used Apple-provided DTKs, but these were recalled along with everyone else's. Now we use Mac Minis provided by MacStadium.