Materialize: Crash: cannot assign requested address

[cuongdo]

Here was the setup:

• EC2 VM w/ 256 GB of memory
• 1 warehouse
• chbenchmark load generator with 1 transactional thread, severely throttled (1 minute of load, followed by 30 minutes of no load)
• a view with query 1 (#522) and a PEEK q01 loop

Right before the crash, I created a view for query 19 (#542) and PEEKed it a few times. Soon after, materialized crashes with this backtrace:

materialized_1     | [2019-09-26T20:17:25.166860052Z WARN src/materialize/pgwire/codec.rs:180] error for client: Error->sql parser error: Expected SELECT, VALUES, or a subquery in the query body, found: EOF
materialized_1     | [2019-09-26T20:17:45.582359162Z ERROR src/materialize/server/mod.rs:74] error handling request: unexpected eof
materialized_1     | materialized encountered an internal error and crashed.
materialized_1     |
materialized_1     | We rely on bug reports to diagnose and fix these errors. Please
materialized_1     | copy and paste the following details and mail them to [email protected].
materialized_1     | To protect your privacy, we do not collect crash reports automatically.
materialized_1     |
materialized_1     |  thread: worker thread 3
materialized_1     | message: called `Result::unwrap()` on an `Err` value: Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" }
materialized_1     | stack backtrace:
materialized_1     |    0: materialized::handle_panic
materialized_1     |              at src/materialize/bin/materialized.rs:148
materialized_1     |    1: core::ops::function::Fn::call
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libcore/ops/function.rs:69
materialized_1     |    2: std::panicking::rust_panic_with_hook
materialized_1     |              at src/libstd/panicking.rs:481
materialized_1     |    3: std::panicking::continue_panic_fmt
materialized_1     |              at src/libstd/panicking.rs:384
materialized_1     |    4: rust_begin_unwind
materialized_1     |              at src/libstd/panicking.rs:311
materialized_1     |    5: core::panicking::panic_fmt
materialized_1     |              at src/libcore/panicking.rs:85
materialized_1     |    6: core::result::unwrap_failed
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libcore/macros.rs:18
materialized_1     |    7: core::result::Result<T,E>::unwrap
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libcore/result.rs:800
materialized_1     |       dataflow::server::Worker<A>::process_peeks::{{closure}}
materialized_1     |              at workdir/src/dataflow/server.rs:473
materialized_1     |       alloc::vec::Vec<T>::retain::{{closure}}
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/vec.rs:1033
materialized_1     |       <alloc::vec::DrainFilter<T,F> as core::iter::traits::iterator::Iterator>::next
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/vec.rs:2766
materialized_1     |       <&mut I as core::iter::traits::iterator::Iterator>::next
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libcore/iter/traits/iterator.rs:2608
materialized_1     |       core::iter::traits::iterator::Iterator::try_fold
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libcore/iter/traits/iterator.rs:1572
materialized_1     |       core::iter::traits::iterator::Iterator::fold
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libcore/iter/traits/iterator.rs:1685
materialized_1     |       core::iter::traits::iterator::Iterator::for_each
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libcore/iter/traits/iterator.rs:604
materialized_1     |       <alloc::vec::DrainFilter<T,F> as core::ops::drop::Drop>::drop
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/vec.rs:2793
materialized_1     |       core::ptr::real_drop_in_place
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libcore/ptr/mod.rs:197
materialized_1     |       alloc::vec::Vec<T>::retain
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/vec.rs:1033
materialized_1     |    8: dataflow::server::Worker<A>::process_peeks
materialized_1     |              at workdir/src/dataflow/server.rs:417
materialized_1     |       dataflow::server::Worker<A>::run
materialized_1     |              at workdir/src/dataflow/server.rs:315
materialized_1     |    9: dataflow::server::serve::{{closure}}
materialized_1     |              at workdir/src/dataflow/server.rs:149
materialized_1     |       timely::execute::execute_from::{{closure}}
materialized_1     |              at cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/0d5dfae/timely/src/execute.rs:294
materialized_1     |       timely_communication::initialize::initialize_from::{{closure}}
materialized_1     |              at cargo/git/checkouts/timely-dataflow-4c0cc365061cd263/0d5dfae/communication/src/initialize.rs:269
materialized_1     |       std::sys_common::backtrace::__rust_begin_short_backtrace
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libstd/sys_common/backtrace.rs:77
materialized_1     |   10: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libstd/thread/mod.rs:470
materialized_1     |       <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libstd/panic.rs:315
materialized_1     |       std::panicking::try::do_call
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libstd/panicking.rs:296
materialized_1     |   11: __rust_maybe_catch_panic
materialized_1     |              at src/libpanic_unwind/lib.rs:82
materialized_1     |   12: std::panicking::try
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libstd/panicking.rs:275
materialized_1     |       std::panic::catch_unwind
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libstd/panic.rs:394
materialized_1     |       std::thread::Builder::spawn_unchecked::{{closure}}
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libstd/thread/mod.rs:469
materialized_1     |       core::ops::function::FnOnce::call_once{{vtable.shim}}
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/libcore/ops/function.rs:231
materialized_1     |   13: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/boxed.rs:746
materialized_1     |   14: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
materialized_1     |              at rustc/eae3437dfe991621e8afdc82734f4a172d7ddf9b/src/liballoc/boxed.rs:746
materialized_1     |       std::sys_common::thread::start_thread
materialized_1     |              at src/libstd/sys_common/thread.rs:13
materialized_1     |       std::sys::unix::thread::Thread::new::thread_start
materialized_1     |              at src/libstd/sys/unix/thread.rs:79
materialized_1     |   15: start_thread
materialized_1     |   16: __clone
materialized_1     |

[frankmcsherry]

From @benesch: we use TCP connections internally to communicate between pgwire handlers and dataflow workers, with an eye towards distributed execution. At the moment there is no connection pooling, and so you eventually exhaust available ports. Connection pooling seems like the right way to fix this.

[cuongdo]

Thanks for the explanation. This is blocking stability testing at the moment. Let’s file a separate issue and get someone to work on it.