Vector: Improve vector compile (check) times

[MOZGIII]

Working on vector is difficult, because on every tiny change that I make rls takes a significant amount of time (tens of seconds) to react. This happens because it spends a lot of time recompiling (or, more specifically, checking) the code.
Let's discuss how we can improve the compile/check speed!

[LucioFranco]

I would suggest using rust-analyzer which gives me about a 3-6s response time and can handle the project much better. It basically just runs cargo watch in the background. Beyond that I don't know if there is much we can, we just have a lot of code to compile and check.

[MOZGIII]

@LucioFranco thanks for the suggestion! I'll definitely take a look into rust-analyzer. I thought it was already used by rls, but, apparently, it's a separate thing. Maybe it'll be good enough!

In the meantime, what if we split the code into multiple crates? The codebase of the vector itself is not that huge, however, I suspect a lot of time is spent on linking with all the crates - I know we integrate with more than a handful of big ones. If we make it so that big dependencies are isolated in the relevant crates - they wouldn't have to be rebuilt when we change unrelated parts of the code. For instance, currently, when I edit docker source, the compilation process has to do all the heavy lifting for all the AWS related stuff - i.e aws_s3 sink is rebuilt too. If we split the code into crates by separating the concerns - it'll be much less job to do for rls and rust-analyzer per a single file change - in general.

[binarylogic]

👍 I'm a big fan of this. If a new contributor is finding it difficult to contribute we should take that seriously. I am also curious what improvements we can make here.

[LucioFranco]

@MOZGIII last I heard about splitting things up is that it didn't help that much but it may be worth a shot if we can prove with numbers it helps.

[lukesteensen]

This is definitely a big factor for contributors and something we should look into addressing. Echoing Lucio, I'm not sure that there's a surefire solution here that we can just go off and implement. More likely, we'll need to dedicate some time tackling our overall tech debt, cleaning up dependencies, and exploring some different strategies for this.

[LucioFranco]

I think the biggest win to compilation will be to upgrade everything to futures 0.3 and tokio 0.2, which will allow us to reduce a lot of duplicates.

[MOZGIII]

I'm not certain that splitting the code into multiple crates would give us the desired speed gain. Intuitively, it should, but I don't have a deep enough understanding of rustc and cargo build process to say for sure. So I'll be happy to try different options too.

P.S. I tried rust-analyzer, and it is indeed better than rls. I can't say it's super-fast, but it's fast enough to stop irritating me as rls did. Thanks for suggesting it, @LucioFranco. This is less of a pain point now for me personally.

[a-rodin]

rust-analyzer is awesome, but in order to run tests it still necessary to build them using rustc.

[MOZGIII]

Thx @a-rodin! This should be much better now, looking forward to trying new feature flags!
If I still feel like it's way too slow - I'll reopen this issue. I hope, the changes you introduced will dramatically improve the iteration times, and I'll be happy with it!