Warp: Compile time is in minutes...

[scull7]

I'm seeing compile times on the order of minutes for a small < 10 route service.

    Finished dev [unoptimized + debuginfo] target(s) in 3m 12s
     Running `target/debug/mtbox-warp-admin`

What are some things that I could look for to help shrink this to something more manageable?

[seanmonstar]

Is this also compiling all dependencies, or just the edit-save-compile loop?

[scull7]

This is just the edit-save-compile loop. I've gone through and compiled all of the associated crates and split out a bunch of functionality into separate crates and compiled those separately. Got compile times < 5s for all my other crates including a crate where I've put the handlers. This crate just contains the actual service code.

[scull7]

This is a compile from scratch:

    Finished dev [unoptimized + debuginfo] target(s) in 5m 15s
     Running `target/debug/mtbox-warp-admin`

[seanmonstar]

Yea, when there's a lot of generic code, sometimes the compiler hits pathological cases. You can sprinkle in Filter::boxed(), which returns a BoxedFilter<T>. At the expense of a dynamic call, it can help the compiler no longer need to reason about the types inside it.

[scull7]

I apologize for my lack of understanding. Where should I be sprinkling in the Filter::boxed() calls?

    let employee_search = warp::get()
        .and(warp::path!("employees"))
        .and(db.clone())
        .and(warp::query::raw().and_then(handlers::employee::parse_query))
        .and_then(handlers::employee::search);

    let employee_enroll = warp::post()
        .and(warp::path!("employees" / "enroll"))
        .and(db.clone())
        .and(system_uuid.clone())
        .and(warp::body::content_length_limit(1024 * 16))
        .and(warp::body::json())
        .and_then(handlers::employee::enroll);

    let employee_by_uuid = warp::get()
        .and(warp::path!("employees" / "uuid" / String))
        .and(db.clone())
        .and_then(handlers::employee::get_by_uuid);

    let employee_roles_by_uuid = warp::get()
        .and(warp::path!("employees" / "uuid" / String / "roles"))
        .and(db.clone())
        .and(system_uuid.clone())
        .and_then(handlers::employee::get_roles_by_uuid);

Should it be somewhere in the route definition or within the handler definition?

[seanmonstar]

It's really kind of guess work, since it's hard to understand which types are causing the slow down. I'd just try adding in a bunch of places and see if there's a difference. If so, I'd then slowly start removing them until it slows down significantly.

[scull7]

I've made some headway into the compile-time issue. It seems that a large portion of the time is spent within the linking step. Using the new lld linker
the following times are generally representative of the speed up:

Full Rebuild:

    Finished dev [unoptimized + debuginfo] target(s) in 2m 40s
     Running `target/debug/mtbox-warp-admin`

Partial Rebuild:

    Finished dev [unoptimized + debuginfo] target(s) in 0.20s
     Running `target/debug/mtbox-warp-admin`

This was accomplished by following the instructions here:
https://github.com/rust-lang/rust/issues/39915#issuecomment-538049306

No sprinkling of magic required :)

[koivunej]

Just in case someone comes around and the lld-9 doesn't help enough: I ended up creating a macro to handle orring the filters together and boxing on debug builds, as in https://github.com/ipfs-rust/rust-ipfs/pull/163/commits/ae3306686209afa5911b1ad02170c1ac3bacda7c as an example of sprinkling (a lot of) boxing around suggested earlier. This had a big effect on incremental compilation times (minutes to seconds).

[scull7]

Just in case someone comes around and the lld-9 doesn't help enough: I ended up creating a macro to handle orring the filters together and boxing on debug builds, as in ipfs-rust/rust-ipfs@ae33066 as an example of sprinkling (a lot of) boxing around suggested earlier. This had a big effect on incremental compilation times (minutes to seconds).

Thank you this works fantastically

[stuartcrobinson]

i think this article describes this situation exactly, and gives an example of the BoxedFilter sprinkling:

https://rust-classes.com/chapter_6_3.html#chapter-63---warp-and-slow-compile-times

and here's a minimal demo of @koivunej's macros that also worked for me fantastically 🚀 (from 3 minutes to 20 seconds):

/************ from this: ************/

let routes = about.or(example1).or(example2);

/************* to this: *************/

/// Helper to combine the multiple filters together with Filter::or, possibly boxing the types in
/// the process. This greatly helps the build times for `ipfs-http`. 
/// https://github.com/seanmonstar/warp/issues/507#issuecomment-615974062
macro_rules! combine {
  ($x:expr, $($y:expr),+) => {{
      let filter = boxed_on_debug!($x);
      $( let filter = boxed_on_debug!(filter.or($y)); )+
      filter
  }}
}

#[cfg(debug_assertions)]
macro_rules! boxed_on_debug { ($x:expr) => { $x.boxed() }; }

#[cfg(not(debug_assertions))]
macro_rules! boxed_on_debug { ($x:expr) => { $x }; }

let routes = about.or(combine!(example1, example2));

thank you @scull7 for opening this 🙏

[koivunej]

That's great @stuartcrobinson! A note for you and anyone else finding this issue: Please note that in https://github.com/rs-ipfs/rust-ipfs/pull/422 we transitioned closer to the balanced tree idea for macro_rules! combine from https://github.com/seanmonstar/warp/issues/619.