Sled: dependency reduction

[spacejam]

• [x] when 1.35 is released: remove futures, use std futures
• [x] replace hashbrown if it becomes std default
• [x] replace rayon with simple internal threadpool
• [ ] replace serde requirement

[jeehoonkang]

On rayon: it seems sled depends on rayon for its thread pool. Do you intend to implement a lightweight fork of it and use the fork? I'd also like to ask what's the motivation behind this effort. Rayon thread pool is quite good :)

[spacejam]

@jeehoonkang mostly I want to be fairly aggressive about keeping dependencies minimized, and rayon is using an ancient version of crossbeam, and I'd like to be able to reduce this duplicated dependency. I'm already implementing threadpool draining on top of rayon, which is already a good chunk of the total complexity I need in an implementation. I don't need a complex threadpool, but the ability to grow when necessary might be another nice thing to have which rayon probably won't implement. Threadpools aren't very complex, and I can remove a large number of the total dependencies in sled by rolling my own.

[Kerollmops]

Hashbrown is now in the std library, you will need to create a type alias and depend on the rustc-hash crate to get the FxHasher type. It will not really reduce the number of dependencies though :)

https://github.com/rust-lang/rust/pull/58623#issuecomment-486050909

[spacejam]

one down is still nice :]

[Kerollmops]

You will replace hashbrown by rustc-hash actually :) ok, you will no more need to compile hashbrown, good point!

EDIT: you will need to compile hashbrown, in fact.

[kanru]

It looks like the current Futures usage is quite simple. The receiving side is always a blocking wait() on the result (one in pagecache/src/segment.rs and one in subscription.rs). If I understand correctly, they can be replaced by a mpsc async channel.

[spacejam]

@kanru in the near term it's not so hard to just replace it with a mutex with a thin layer on top that sort of acts like a oneshot channel. Since the futures support in std doesn't really work for the sled use case (we don't want to give up control to a specific scheduler that makes a static decision about latency vs throughput)

[spacejam]

rayon and lazy_static are now replaced by simple implementations that don't trigger lots of TSAN false positives

[spacejam]

For now, serde will stay, as we don't need to deserialize anything during recovery anymore (log message headers provide everything we need now)