Wrangler: Segmentation fault after wrangler executes its command

[orium]

🐛 Bug Report

wrangler crashes with a segmentation fault after running the given command. This apparently only happens with release build and not with debug builds. Reproducible with wrangler version 1.10.3 as well as master (as of b601101); also reproducible with both rustc 1.45.0 (5c1f21c3b 2020-07-13) as well as rustc 1.46.0-nightly (346aec9b0 2020-07-11).

Environment

• operating system: Arch linux: Linux 5.7.8-arch1-1 #1 SMP PREEMPT Thu, 09 Jul 2020 16:34:01 +0000 x86_64 GNU/Linux
• output of rustc -V: I've tried multiple version, but including the currect stable: rustc 1.45.0 (5c1f21c3b 2020-07-13)
• output of node -v: not installed
• output of wrangler -V: repoducible in 1.10.3 and master (as of b601101)
• contents of wrangler.toml: empty

Steps to reproduce

You can either cargo install wrangler from crates.io, or in master (as of b601101):

$ cargo build --release
⋮
$ ./target/release/wrangler
The Wrangler Team <[email protected]>

USAGE:
    wrangler [SUBCOMMAND]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

SUBCOMMANDS:
    kv:namespace     Interact with your Workers KV Namespaces
    kv:key           Individually manage Workers KV key-value pairs
    kv:bulk          Interact with multiple Workers KV key-value pairs at once
    route            List or delete worker routes.
    secret           Generate a secret that can be referenced in the worker script
    generate         Generate a new worker project
    init             Create a wrangler.toml for an existing project
    build            Build your worker
    preview          Preview your code temporarily on cloudflareworkers.com
    dev              Start a local server for developing your worker
    publish          Publish your worker to the orange cloud
    config           Set up wrangler with your Cloudflare account
    subdomain        Configure your workers.dev subdomain
    whoami           Retrieve your user info and test your auth config
    tail             Aggregate logs from production worker
    help            Prints this message or the help of the given subcommand(s)
zsh: segmentation fault (core dumped)  ./target/release/wrangler

Running inside lldb:

$ lldb ./target/release/wrangler
(lldb) target create "./target/release/wrangler"
Current executable set to '/home/orium/programming/cloudflare/wrangler/target/release/wrangler' (x86_64).
(lldb) r
Process 550702 launched: '/home/orium/programming/cloudflare/wrangler/target/release/wrangler' (x86_64)
 wrangler 1.10.3
The Wrangler Team <[email protected]>

USAGE:
    wrangler [SUBCOMMAND]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

SUBCOMMANDS:
    kv:namespace     Interact with your Workers KV Namespaces
    kv:key           Individually manage Workers KV key-value pairs
    kv:bulk          Interact with multiple Workers KV key-value pairs at once
    route            List or delete worker routes.
    secret           Generate a secret that can be referenced in the worker script
    generate         Generate a new worker project
    init             Create a wrangler.toml for an existing project
    build            Build your worker
    preview          Preview your code temporarily on cloudflareworkers.com
    dev              Start a local server for developing your worker
    publish          Publish your worker to the orange cloud
    config           Set up wrangler with your Cloudflare account
    subdomain        Configure your workers.dev subdomain
    whoami           Retrieve your user info and test your auth config
    tail             Aggregate logs from production worker
    help            Prints this message or the help of the given subcommand(s)
Process 550702 stopped
* thread #3, name = 'reqwest-interna', stop reason = signal SIGSEGV: invalid address (fault address: 0x18)
    frame #0: 0x00007ffff79bb8e6 libpthread.so.0`__pthread_rwlock_wrlock + 22
libpthread.so.0`__pthread_rwlock_wrlock:
->  0x7ffff79bb8e6 <+22>: movl   0x18(%rdi), %edx
    0x7ffff79bb8e9 <+25>: movl   %fs:0x2d0, %eax
    0x7ffff79bb8f1 <+33>: cmpl   %eax, %edx
    0x7ffff79bb8f3 <+35>: je     0x7ffff79bb960            ; <+144>
(lldb) bt
* thread #3, name = 'reqwest-interna', stop reason = signal SIGSEGV: invalid address (fault address: 0x18)
  * frame #0: 0x00007ffff79bb8e6 libpthread.so.0`__pthread_rwlock_wrlock + 22
    frame #1: 0x00007ffff7bbd02a libcrypto.so.1.1`CRYPTO_THREAD_write_lock + 10
    frame #2: 0x00007ffff7b50ef0 libcrypto.so.1.1`OPENSSL_init_crypto + 800
    frame #3: 0x00007ffff7ce61f2 libssl.so.1.1`OPENSSL_init_ssl + 50
    frame #4: 0x0000555555d4ab8b wrangler`std::sync::once::Once::call_inner::hcff3709ae0293da4 at once.rs:416:21
    frame #5: 0x0000555555be6ac0 wrangler`openssl_sys::init::hb7d4ee155f3460e3 + 64
    frame #6: 0x0000555555be64a3 wrangler`openssl::ssl::connector::ctx::h1740157059bc9ac1 + 19
    frame #7: 0x0000555555be6564 wrangler`openssl::ssl::connector::SslConnector::builder::hbfe4bf78845fde53 + 20
    frame #8: 0x0000555555be241c wrangler`native_tls::imp::TlsConnector::new::h15c5ba45815bd72a + 60
    frame #9: 0x0000555555be2b65 wrangler`native_tls::TlsConnectorBuilder::build::ha2b64693ad06c741 + 21
    frame #10: 0x0000555555b091f4 wrangler`reqwest::connect::Connector::new_default_tls::h602c88545d6f5fda + 52
    frame #11: 0x0000555555ba6427 wrangler`reqwest::async_impl::client::ClientBuilder::build::hb12da7537dfe3e0a + 1223
    frame #12: 0x0000555555b3f008 wrangler`_$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hb8ed2143500ac64b + 88
    frame #13: 0x0000555555b5ea2f wrangler`tokio::macros::scoped_tls::ScopedKey$LT$T$GT$::set::ha65e6a297784a429 + 287
    frame #14: 0x0000555555b8622d wrangler`tokio::runtime::basic_scheduler::BasicScheduler$LT$P$GT$::block_on::hb3a58c78048f2d8b + 285
    frame #15: 0x0000555555b147ca wrangler`tokio::runtime::context::enter::h5e9f6f9d15b6ebb6 + 186
    frame #16: 0x0000555555b9276b wrangler`std::sys_common::backtrace::__rust_begin_short_backtrace::h5d994d9dea05c497 + 1371
    frame #17: 0x0000555555b02505 wrangler`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h4e89c2fa4cd2a1a3 + 101
    frame #18: 0x0000555555d5842a wrangler`std::sys::unix::thread::Thread::new::thread_start::h3b6d8a0cd87a87c6 [inlined] _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hcf205bcf9b46c587 at boxed.rs:1076:9
    frame #19: 0x0000555555d58424 wrangler`std::sys::unix::thread::Thread::new::thread_start::h3b6d8a0cd87a87c6 [inlined] _$LT$alloc..boxed..Box$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::h2d53e2246128f5d8 at boxed.rs:1076
    frame #20: 0x0000555555d5841b wrangler`std::sys::unix::thread::Thread::new::thread_start::h3b6d8a0cd87a87c6 at thread.rs:87
    frame #21: 0x00007ffff79b6422 libpthread.so.0`start_thread + 226
    frame #22: 0x00007ffff7e41bf3 libc.so.6`__clone + 67

Also worth saying that valgrind doesn't see any invalid memory accesses.

[kentonv]

I've noticed this too (on Linux). Seems to be common after error messages, but it doesn't always happen.

[EverlastingBugstopper]

are you able to reproduce this issue if you build with the command cargo build --release --features vendored-openssl or cargo install --features vendored-openssl?

[orium]

are you able to reproduce this issue if you build with the command cargo build --release --features vendored-openssl or cargo install --features vendored-openssl?

Still have a segmentation fault.

[orium]

It seems the issue goes away if I have a ~/.wrangler/version.toml...

[orium]

I could minify the bug to this:

fn main() {
    std::thread::spawn(move || {
        reqwest::blocking::Client::new()
            .get("https://crates.io/api/v1/crates/wrangler")
            .send().unwrap();
    });

    // Just so we write some memory to trigger the bug.
    let mut v = Vec::new();
    for i in 0..100000 {
        v.push(i);
    }
}

Still get the segmentation fault almost all the time. It looks like it might be cause by https://github.com/sfackler/rust-openssl/issues/1293.

Ref https://github.com/rustwasm/wasm-pack/issues/823

[EverlastingBugstopper]

That does seem to be the issue doesn't it - nice job tracking it down! If your PR gets merged upstream and a release is cut we'll be sure to update the dependency. Thanks a bunch @orium!

[nataliescottdavidson]

Hey @orium - I'm not able to reproduce this as is. Could you include your Cargo.toml for the minified example?

[orium]

I think I don't have it anymore, but IIRC it only had a dependency on reqwest and nothing else. It is normal for some systems not to be able to reproduce this very easily. It's a non-deterministic race condition based on thread scheduling and timing.

In any case the issue is fixed upstream (https://github.com/sfackler/rust-openssl/issues/1293) and once we get a new release of rust-openssl (which is a transitive dependency of reqwest) we should be good.

[codenoid]

yes, reproduced with:

> cargo install wrangler
Ubuntu 20.04.1 LTS
 >  cargo --version
cargo 1.45.0 (744bd1fbb 2020-06-15)
 >  rustc --version
rustc 1.45.0 (5c1f21c3b 2020-07-13)

wait, when writing this comment, i got another error message when just executing wrangler command

 >  wrangler
👷 ✨  wrangler 1.11.0
The Wrangler Team <[email protected]>

USAGE:
    wrangler [SUBCOMMAND]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

SUBCOMMANDS:
    kv:namespace    🗂️  Interact with your Workers KV Namespaces
    kv:key          🔑  Individually manage Workers KV key-value pairs
    kv:bulk         💪  Interact with multiple Workers KV key-value pairs at once
    route           ➡️  List or delete worker routes.
    secret          🤫  Generate a secret that can be referenced in the worker script
    generate        👯  Generate a new worker project
    init            📥  Create a wrangler.toml for an existing project
    build           🦀  Build your worker
    preview         🔬  Preview your code temporarily on cloudflareworkers.com
    dev             👂  Start a local server for developing your worker
    publish         🆙  Publish your worker to the orange cloud
    config          🕵️  Authenticate Wrangler with a Cloudflare API Token or Global API Key
    subdomain       👷  Configure your workers.dev subdomain
    whoami          🕵️  Retrieve your user info and test your auth config
    tail            🦚  Aggregate logs from production worker
    login           🔓 Authenticate Wrangler with your Cloudflare username and password
    help            Prints this message or the help of the given subcommand(s)
thread '<unnamed>' panicked at 'Client::new(): reqwest::Error { kind: Builder, source: Normal(ErrorStack([])) }', /home/ken/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.10.8/src/blocking/client.rs:575:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Update

okay, this weird

[orium]

@codenoid I think the

thread '<unnamed>' panicked at 'Client::new(): reqwest::Error { kind: Builder, source: Normal(ErrorStack([])) }', /home/ken/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.10.8/src/blocking/client.rs:575:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

is caused by the same issue, so the upstream fix in rust-openssl should fix that as well.