In the archive - coreutils.nodbg.zip - you will find two files, coreutils.nodbg.wasm and coreutils.nodbg.opt.wasm.
Unfortunately, they're fairly large, but that's as far as I could get for now. Maybe you'll be able to reduce them further and get to the bottom of the issue with wasm-reduce, but I couldn't spend more time on it.
Initially I've built the original source code in a release mode, which, combined with wasm-opt --asyncify -Os, introduced an infinite loop in the resulting WebAssembly, and I couldn't figure out what's going on.
After rebuilding in debug mode, I'm getting a [slightly more helpful] runtime assertion panic somewhere in initialisation of the app (when it tries to allocate data for a HashMap), and that panic is not reproducible by the original app, suggesting that some values were indeed misoptimized.
My initial assumption was that this is some bug specific to Asyncify, but, by slowly eliminating different options, got down to just -O1.
coreutils.nodbg.opt.wasm is produced with
$ wasm-opt.exe coreutils.nodbg.wasm -O1 -g -o coreutils.nodbg.opt.wasm
using wasm-opt from the latest commit on master (https://github.com/WebAssembly/binaryen/commit/cd0cc95e2794375c463a69833b1ccc9cc96d597c), however I could reproduce the issue with older versions as well - should be easy to do so using the command above.
To reproduce the issue, install a WASI-compatible runner, e.g. Wasmtime or Wasmer, and invoke it on both Wasm files.
This is what I'm seeing on the unoptimised file:
$ wasmtime -g coreutils.nodbg.wasm
coreutils.nodbg 0.0.1 (multi-call binary)
Usage: coreutils.nodbg [function [arguments...]]
Currently defined functions/utilities:
base32, base64, basename, cat, cksum, comm, cp, cut, date, df,
dircolors, dirname, echo, env, expand, expr, factor, false, fmt, fold,
hashsum, head, join, link, ln, ls, md5sum, mkdir, mktemp, more, mv, nl,
od, paste, printenv, printf, ptx, pwd, readlink, realpath, relpath, rm,
rmdir, seq, sha1sum, sha224sum, sha256sum, sha3-224sum, sha3-256sum,
sha3-384sum, sha3-512sum, sha384sum, sha3sum, sha512sum, shake128sum,
shake256sum, shred, shuf, sleep, sort, split, sum, tac, tail, tee, test,
tr, true, truncate, tsort, unexpand, uniq, wc, yes
and the optimised one:
$ wasmtime -g coreutils.nodbg.opt.wasm
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: LayoutErr { private: () }', C:\Users\rreverser\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib/rustlib/src/rust\src/libcore/alloc/layout.rs:231:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: failed to run main module `coreutils.nodbg.opt.wasm`
Caused by:
0: failed to invoke command default
1: wasm trap: unreachable
wasm backtrace:
0: 0x55cdd9 - <unknown>!__rust_start_panic
1: 0x55bf3e - <unknown>!rust_panic
2: 0x55bc75 - <unknown>!std::panicking::rust_panic_with_hook::hb8132b4308a71007
3: 0x55baca - <unknown>!rust_begin_unwind
4: 0x568654 - <unknown>!core::panicking::panic_fmt::hdd2ab611a748a491
5: 0x56d5ae - <unknown>!core::option::expect_none_failed::habae0dd01495a6a7
6: 0x339708 - <unknown>!core::result::Result<T,E>::unwrap::h0b29ccc7c9c10d07
7: 0x89b8 - <unknown>!core::alloc::layout::Layout::pad_to_align::hae1edfd09d682fcc
8: 0x5db74 - <unknown>!core::alloc::layout::Layout::array::h630440cfaca600e0
9: 0x57b73 - <unknown>!hashbrown::raw::calculate_layout::h32a6c9dc2acba081
10: 0x594fe - <unknown>!hashbrown::raw::RawTable<T>::new_uninitialized::h92067dc36e22c506
11: 0x59b10 - <unknown>!hashbrown::raw::RawTable<T>::try_with_capacity::h64ecb1c3ee6c48f1
12: 0x5864c - <unknown>!hashbrown::raw::RawTable<T>::resize::h1c5680148fe8f2d3
13: 0x58216 - <unknown>!hashbrown::raw::RawTable<T>::reserve_rehash::h36d80382ea8e9ffc
14: 0x5a96b - <unknown>!hashbrown::raw::RawTable<T>::reserve::h96e5457f885fde23
15: 0x5a872 - <unknown>!hashbrown::raw::RawTable<T>::insert::hd555411ed0fee9f1
16: 0x77f8e - <unknown>!hashbrown::map::HashMap<K,V,S>::insert::h263136bf7b5163b1
17: 0x44e23 - <unknown>!std::collections::hash::map::HashMap<K,V,S>::insert::h659bd993b5cd211f
18: 0xe449 - <unknown>!coreutils::util_map::h81ccd6dc74a16eac
19: 0x7d64 - <unknown>!coreutils::main::h791587a0c1680f0e
20: 0x4cd5c - <unknown>!std::rt::lang_start::{{closure}}::h14ee8eacd196f15d
21: 0x55aa3b - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::hd2bb8386068f691a
22: 0x55c02a - <unknown>!std::rt::lang_start_internal::h66de5b0ec01e6d33
23: 0x4cd33 - <unknown>!std::rt::lang_start::h01445dbe67544740
24: 0x849f - <unknown>!__original_main
25: 0x7cad - <unknown>!_start
Odd, I can't reproduce this. wasm-opt -O1 -g emits a file that still works. It is a different size than yours though, mine is 8963093 and yours is 8931128. So perhaps this is a windows-specific issue?
Would be good if someone with a windows machine can run wasm-reduce on this. Or even quicker, seeing which pass is responsible could help, if you compile with BINARYEN_PASS_DEBUG=3 in the env it will dump .wasm files for all intermediate steps, and you can check those.
So perhaps this is a windows-specific issue?
Huh, weird, but maybe? FWIW I downloaded an artifact built by Github actions as well as tried older ones from Github releases page, so this should be fairly reproducible for others on Windows by trying same binaries.
This is super odd, but on a different machine I can't reproduce this either and getting same size as you :| I'll try again when back at work.
Ah, no, it was just old Binaryen. Can reproduce same size & issue on the latest commit.
Just wondering can you reproduce this before f6eb790eec107064798b9e54552f1ef5966f6359 commit?
Would be good if someone with a windows machine can run wasm-reduce on this. Or even quicker, seeing which pass is responsible could help, if you compile with
BINARYEN_PASS_DEBUG=3in the env it will dump.wasmfiles for all intermediate steps, and you can check those.
> ls byn-*.wasm | foreach -parallel { wasmtime $_ *>$null; @{$_ = $?} } | format-table -autosize
Name Value
---- -----
C:\Users\me\AppData\Local\Temp\byn-004-remove-unused-brs.wasm True
C:\Users\me\AppData\Local\Temp\byn-003-dce.wasm True
C:\Users\me\AppData\Local\Temp\byn-002-memory-packing.wasm True
C:\Users\me\AppData\Local\Temp\byn-001-duplicate-function-elimination.wasm True
C:\Users\me\AppData\Local\Temp\byn-000-before.wasm True
C:\Users\me\AppData\Local\Temp\byn-005-remove-unused-names.wasm True
C:\Users\me\AppData\Local\Temp\byn-006-optimize-instructions.wasm False
C:\Users\me\AppData\Local\Temp\byn-007-precompute.wasm False
C:\Users\me\AppData\Local\Temp\byn-008-simplify-locals-nostructure.wasm False
C:\Users\me\AppData\Local\Temp\byn-009-vacuum.wasm False
C:\Users\me\AppData\Local\Temp\byn-010-reorder-locals.wasm False
C:\Users\me\AppData\Local\Temp\byn-011-remove-unused-brs.wasm False
C:\Users\me\AppData\Local\Temp\byn-012-coalesce-locals.wasm False
C:\Users\me\AppData\Local\Temp\byn-013-simplify-locals.wasm False
C:\Users\me\AppData\Local\Temp\byn-014-vacuum.wasm False
C:\Users\me\AppData\Local\Temp\byn-015-reorder-locals.wasm False
C:\Users\me\AppData\Local\Temp\byn-017-reorder-locals.wasm False
C:\Users\me\AppData\Local\Temp\byn-016-coalesce-locals.wasm False
C:\Users\me\AppData\Local\Temp\byn-019-merge-blocks.wasm False
C:\Users\me\AppData\Local\Temp\byn-018-vacuum.wasm False
C:\Users\me\AppData\Local\Temp\byn-020-remove-unused-brs.wasm False
C:\Users\me\AppData\Local\Temp\byn-023-precompute.wasm False
C:\Users\me\AppData\Local\Temp\byn-021-remove-unused-names.wasm False
C:\Users\me\AppData\Local\Temp\byn-022-merge-blocks.wasm False
C:\Users\me\AppData\Local\Temp\byn-024-optimize-instructions.wasm False
C:\Users\me\AppData\Local\Temp\byn-025-vacuum.wasm False
C:\Users\me\AppData\Local\Temp\byn-028-simplify-globals.wasm False
C:\Users\me\AppData\Local\Temp\byn-027-duplicate-import-elimination.wasm False
C:\Users\me\AppData\Local\Temp\byn-029-remove-unused-module-elements.wasm False
C:\Users\me\AppData\Local\Temp\byn-030-directize.wasm False
C:\Users\me\AppData\Local\Temp\byn-026-duplicate-function-elimination.wasm False
The True is for working program, and False for a crashing one, correspondingly (sorry, I parallelised the check but forgot to add a sort command in PowerShell pipeline, so they're a bit out of order, but you can see that byn-006-optimize-instructions.wasm is the first broken one).
At this point I'll probably leave it to someone on the team as I've already spent more engineering time on digging into this bug than I'm supposed to / allowed to these days 馃槄
My assumption it's could be __popcnt / __popcnt64 which may be UB for x86 without ABM. I'm going to make a fix, but I can鈥檛 check because I am not a windows user.
I believe couple of people on the team have Windows laptops or access to a shared Windows machine, so it shouldn't be hard to check / dig further (not on the weekend, of course).
@RReverser I guess for reproduction of this issue someone should have Windows OS + cpu without SSE4.2 support =) If this is what I think and problem only with __popcnt
Mine is i7-8650U, it definitely has SSE4.2.
That's one of the reasons I want to defer - I've already spent two days (including one on a weekend) trying to guess what's wrong, and slowly got to the repro above. Now, while I could keep trying more options, I'd rather leave it to someone who can get to the bottom of the problem faster by both having access to Windows machine but knowing the codebase at the same time.
While I do have Windows already, trying various solutions on just my computer doesn't scale too well :)
Oh it's really unpleasant to spend so much time.
Alright, Here is what I did:
1) Optimize original wasm on my machine wasm-opt ./coreutils.nodbg.wasm -O1 -g -o ./coreutils.nodbg.opt2.wasm and got 8聽963聽093 bytes wasm file
2) translate ./coreutils.nodbg.opt2.wasm and ./coreutils.nodbg.opt.wasm to wat files via wasm2wat
3) do diff -a -B -I '\s*call' coreutils.nodbg.opt.wat coreutils.nodbg.opt2.wat or diff -a -u -B -I '\s*call' coreutils.nodbg.opt.wat coreutils.nodbg.opt2.wat
And found out that in the first broken file missed bunch of operations like this:
i32.const 1
i32.and
So Binaryen on win32 for some reason mistakenly optimize this:
local.get $0 ;; it could be `call` instruction as well
i32.const 1
i32.and
to this
local.get $0
So my previous assumption still relevant I guess and probably starting from here. So if popcnt broken it will return 0 here which subsequently leads to such a result
Thanks for checking, that sounds pretty reasonable, so I went ahead and downloaded the artifact from Github actions for your 2 commits in the PR... and both fail in exactly same way :(
Seriously, at this point I'll just leave it to someone else to try different options 馃槄
If you don't have access to a Windows machine, you can probably use free VMs from https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/ for testing.
UPD: these ones are even better, as they have Visual Studio preinstalled, so you can not only try prebuilt .exe's, but also build them yourself. They're all free, just with 90 days limitation. https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/
Hmm, in this case it some IB of MSVC. But at least we have a direction
Btw windows build has 38.6 MB but linux / mac only ~3 MB for artifacts. Is it normal?
I agree guessing is not a good use of time.
Given we know which pass this is in, it should be possible to git bisect this on a windows machine pretty quickly.
It seems it automatically closed by #2944. We still hasn't confirmation that PR really fixed this issue, so better keep it open I guess
Oh, was that PR marked as closing this one? Then that's why it automatically closed, yeah. Reopening.
Btw windows build has
38.6 MBbut linux / mac only ~3 MBfor artifacts. Is it normal?
It appears that at some point (I don't know whether intentionally or not) Binaryen builds for Windows stopped including binaryen.dll, and instead each executable statically includes all the relevant code, while Linux executables still use a shared libbinaryen.so, which for some reason isn't even included in Github actions builds.
So no, I'd say it's not normal.
Since this is going to take some time, I meanwhile went through downgrades from Github releases to find a working version and stopped on 92, which produces 8963093 bytes like @kripken said the latest Linux version does, too.
This information should be useful to whoever will be doing bisecting.
Note that version 93 might work as well, but its Windows version on Github releases seems to be a debug build and not a release one, so it was running too long and I've decided to skip it.
I claim that the defect is not windows specific and can be trivially reproduced using WSL2 (ubuntu 20.04 LTS). The steps were identical to the prescribed in the defect
michael@mpwk1:/mnt/c/Users/mp/tmp/2942$ wasmtime -g coreutils.nodbg.wasm
coreutils.nodbg 0.0.1 (multi-call binary)
Usage: coreutils.nodbg [function [arguments...]]
Currently defined functions/utilities:
base32, base64, basename, cat, cksum, comm, cp, cut, date, df,
dircolors, dirname, echo, env, expand, expr, factor, false, fmt, fold,
hashsum, head, join, link, ln, ls, md5sum, mkdir, mktemp, more, mv, nl,
od, paste, printenv, printf, ptx, pwd, readlink, realpath, relpath, rm,
rmdir, seq, sha1sum, sha224sum, sha256sum, sha3-224sum, sha3-256sum,
sha3-384sum, sha3-512sum, sha384sum, sha3sum, sha512sum, shake128sum,
shake256sum, shred, shuf, sleep, sort, split, sum, tac, tail, tee, test,
tr, true, truncate, tsort, unexpand, uniq, wc, yes
michael@mpwk1:/mnt/c/Users/mp/tmp/2942$ ls
coreutils.nodbg.opt.wasm coreutils.nodbg.wasm
michael@mpwk1:/mnt/c/Users/mp/tmp/2942$ wasmtime -g coreutils.nodbg.opt.wasm
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: LayoutErr { private: () }', C:\Users\rreverser\.rustup\toolchains\stable-x86_64-pc-windows-msvc\lib/rustlib/src/rust\src/libcore/alloc/layout.rs:231:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: failed to run main module `coreutils.nodbg.opt.wasm`
Caused by:
0: failed to invoke command default
1: wasm trap: unreachable
wasm backtrace:
0: 0x55cdd9 - <unknown>!__rust_start_panic
1: 0x55bf3e - <unknown>!rust_panic
2: 0x55bc75 - <unknown>!std::panicking::rust_panic_with_hook::hb8132b4308a71007
3: 0x55baca - <unknown>!rust_begin_unwind
4: 0x568654 - <unknown>!core::panicking::panic_fmt::hdd2ab611a748a491
5: 0x56d5ae - <unknown>!core::option::expect_none_failed::habae0dd01495a6a7
6: 0x339708 - <unknown>!core::result::Result<T,E>::unwrap::h0b29ccc7c9c10d07
7: 0x89b8 - <unknown>!core::alloc::layout::Layout::pad_to_align::hae1edfd09d682fcc
8: 0x5db74 - <unknown>!core::alloc::layout::Layout::array::h630440cfaca600e0
9: 0x57b73 - <unknown>!hashbrown::raw::calculate_layout::h32a6c9dc2acba081
10: 0x594fe - <unknown>!hashbrown::raw::RawTable<T>::new_uninitialized::h92067dc36e22c506
11: 0x59b10 - <unknown>!hashbrown::raw::RawTable<T>::try_with_capacity::h64ecb1c3ee6c48f1
12: 0x5864c - <unknown>!hashbrown::raw::RawTable<T>::resize::h1c5680148fe8f2d3
13: 0x58216 - <unknown>!hashbrown::raw::RawTable<T>::reserve_rehash::h36d80382ea8e9ffc
14: 0x5a96b - <unknown>!hashbrown::raw::RawTable<T>::reserve::h96e5457f885fde23
15: 0x5a872 - <unknown>!hashbrown::raw::RawTable<T>::insert::hd555411ed0fee9f1
16: 0x77f8e - <unknown>!hashbrown::map::HashMap<K,V,S>::insert::h263136bf7b5163b1
17: 0x44e23 - <unknown>!std::collections::hash::map::HashMap<K,V,S>::insert::h659bd993b5cd211f
18: 0xe449 - <unknown>!coreutils::util_map::h81ccd6dc74a16eac
19: 0x7d64 - <unknown>!coreutils::main::h791587a0c1680f0e
20: 0x4cd5c - <unknown>!std::rt::lang_start::{{closure}}::h14ee8eacd196f15d
21: 0x55aa3b - <unknown>!std::sys_common::backtrace::__rust_begin_short_backtrace::hd2bb8386068f691a
22: 0x55c02a - <unknown>!std::rt::lang_start_internal::h66de5b0ec01e6d33
23: 0x4cd33 - <unknown>!std::rt::lang_start::h01445dbe67544740
24: 0x849f - <unknown>!__original_main
25: 0x7cad - <unknown>!_start
In addition
michael@mpwk1:/mnt/c/Users/mp/tmp/2942$ wasmtime --version
wasmtime 0.18.0
So it's reproduced with original coreutils.nodbg.wasm (which not touched by binaryen / wasm-opt) right?
So it's reproduced with original
coreutils.nodbg.wasm(which not touched by binaryen / wasm-opt) right?
Ah, I see, so it's still relevant to coreutils.nodbg.opt.wasm
@michaelp could you try to download binaryen from master rebuild it and use via wasm-opt coreutils.nodbg.wasm -O1 -g -o coreutils.nodbg.opt2.wasm and try to launch coreutils.nodbg.opt2.wasm on wasmtime or just compare it with previous coreutils.nodbg.opt.wasm by size?
I claim that the defect is not windows specific and can be trivially reproduced using WSL2 (ubuntu 20.04 LTS).
What was meant by Windows-specific is that this is something specific to Windows version of wasm-opt, not Windows version of wasmtime.
That is, if you use Linux version of wasm-opt, then produced coreutils.nodbg.opt.wasm is different and doesn't exhibit same issue.
I'm getting the same stacktrace as this with a yew, wasm-pack, wasm-bindgen, web-view based project I'm working on, which is running on webkit2gtk on Linux:
panicked at 'called `Option::unwrap()` on a `None` value', src/libstd/panicking.rs:386:15
Stack:
wasm-stub@[wasm code]
<?>.wasm-function[console_error_panic_hook::Error::new::h2949e2ca75b7ecca]@[wasm code]
<?>.wasm-function[console_error_panic_hook::hook_impl::h59023c3dd5d0a768]@[wasm code]
<?>.wasm-function[console_error_panic_hook::hook::hc80dc65d805553a9]@[wasm code]
<?>.wasm-function[core::ops::function::Fn::call::h1efb9c791c1492a7]@[wasm code]
<?>.wasm-function[std::panicking::rust_panic_with_hook::hf2c303df71675c7a]@[wasm code]
<?>.wasm-function[rust_begin_unwind]@[wasm code]
<?>.wasm-function[core::panicking::panic_fmt::hd99935ab1907ce22]@[wasm code]
<?>.wasm-function[core::panicking::panic::h9e1d61e86fb54de0]@[wasm code]
<?>.wasm-function[core::option::Option<T>::unwrap::h82cc8250ba57d0be]@[wasm code]
<?>.wasm-function[rust_begin_unwind]@[wasm code]
<?>.wasm-function[core::panicking::panic_fmt::hd99935ab1907ce22]@[wasm code]
<?>.wasm-function[core::option::expect_none_failed::h0541cb17d8c73a02]@[wasm code]
<?>.wasm-function[core::result::Result<T,E>::unwrap::h25771e5558c24a86]@[wasm code]
<?>.wasm-function[core::alloc::layout::Layout::pad_to_align::h9a582f06a2691e50]@[wasm code]
<?>.wasm-function[core::alloc::layout::Layout::array::h971953ffc59c85a0]@[wasm code]
<?>.wasm-function[hashbrown::raw::calculate_layout::h427fb67acc3840d3]@[wasm code]
<?>.wasm-function[hashbrown::raw::RawTable<T>::new_uninitialized::hc1c280bdc7d87936]@[wasm code]
<?>.wasm-function[hashbrown::raw::RawTable<T>::try_with_capacity::h00fc03242d72d936]@[wasm code]
<?>.wasm-function[hashbrown::raw::RawTable<T>::resize::h3cb30034a06b3681]@[wasm code]
<?>.wasm-function[hashbrown::raw::RawTable<T>::reserve_rehash::h364a21dcbcc80769]@[wasm code]
<?>.wasm-function[hashbrown::raw::RawTable<T>::reserve::he7c16a63a5b19a42]@[wasm code]
<?>.wasm-function[hashbrown::raw::RawTable<T>::insert::hddfbf9828739d805]@[wasm code]
<?>.wasm-function[hashbrown::map::HashMap<K,V,S>::insert::h102a4a8682303e6a]@[wasm code]
<?>.wasm-function[std::collections::hash::map::HashMap<K,V,S>::insert::h9136cd6189c24768]@[wasm code]
<?>.wasm-function[yew::virtual_dom::vtag::VTag::add_attribute::h3052cec2aba4aa7a]@[wasm code]
<?>.wasm-function[<yew_bulma::components::form::input_field::InputField<Key,Type> as yew::html::Component>::view::h10dc72becb5252cf]@[wasm code]
<?>.wasm-function[<COMP as yew::html::Renderable>::render::h5d04215a585a2638]@[wasm code]
<?>.wasm-function[<yew::html::scope::UpdateComponent<COMP> as yew::scheduler::Runnable>::run::h8122a75a500b7c3b]@[wasm code]
<?>.wasm-function[yew::scheduler::Scheduler::start::h36681fba845c2239]@[wasm code]
<?>.wasm-function[yew::html::scope::Scope<COMP>::mount_in_place::h63cf0310bcd87240]@[wasm code]
<?>.wasm-function[yew::app::App<COMP>::mount::h3b6819a69cc192f5]@[wasm code]
<?>.wasm-function[yew::app::App<COMP>::mount_to_body::ha361273556f2268e]@[wasm code]
<?>.wasm-function[frontend::run_app::hde8d2000a0b46e36]@[wasm code]
<?>.wasm-function[run_app]@[wasm code]
wasm-stub@[wasm code]
run_app@[native code]
It appears to be an intermittent issue, and isn't exhibited on Edge on Windows 10.
I don't think I'm using bineryen (I could be wrong, perhaps web-pack still makes use of it with my settings), the project is compiled with web-pack using the https://rustwasm.github.io/wasm-pack/book/commands/build.html web target and the --dev profile. Perhaps the issues are related somehow?
@kellpossible wasm-pack does use wasm-opt under the hood, although I think they're stuck on an old version, and I'm not sure if it already had this bug or not. You can try disabling their wasm-opt via options as per https://rustwasm.github.io/wasm-pack/book/cargo-toml-configuration.html. If it does help, then you can download newest wasm-opt and run optimisations separately.
@RReverser thanks for the heads up. I'll try disabling and see. According to that document you linked, it should be disabled by default with the dev profile, which is the one that I have been using. I'll manually disable it anyway and see what happens.