Cargo: Add the ability to ignore registry-lookup failures

[brson]

I am constantly forced online because Cargo wants to update the registry. I do not like this and consider it a major problem that our build tool does not work without being connected to the Internet.

To give you a concrete example of a thing that happens. Earlier this week I built my project, cargo-crusader. Then I updated my toolchain, as I do nearly every day. Today I continued working on cargo-crusader while enjoying a morning in the refreshing air of a park. My peace was almost immediately disturbed when I discovered that I could no longer build cargo-crusader with the same crate resolution that worked just a few days ago because Cargo wanted to update the registry.

[alexcrichton]

Do you know why Cargo decided to update the registry? Some possible cases could be that the lock file was missing or the lock file was present and a package was missing from the index. In both these cases it's kinda necessary to update the lockfile.

Overall I agree that Cargo should be resilient to update failures, but I'd prefer to have something reproducible locally where it's 100% sure that Cargo doesn't need to update the registry, it's just optimistically doing so.

[sfackler]

Why is it impossible to update the lockfile if Cargo can't talk to the registry?

[alexcrichton]

It's certainly not impossible, there are situations where you can avoid talking to the registry, but many stars have to align:

• We have to discover that the index has already been checked out and is relatively recent
• Using the information in the index we determine that the dependency graph can be satisfied
• All dependencies which were satisfied were then already downloaded

All the error messages along the way, especially in resolution, would also need to indicate that allowing Cargo to talk to the registry might help make the errors go away.

[sfackler]

Having reasonable warning messages is certainly reasonable. However, it is _really important_ that it is possible to do "as much as possible" while not being able to talk to the registry. This definitely includes a project not ending up in a state where it can't build without connecting back to the registry, which definitely happens right now. Ideally, Cargo would even be able to run cargo update, limited to the dependency set it has cached locally when offline.

[alexcrichton]

Do you have an example of a state where cargo can't build without connecting back to the registry? The one I can think of is there's a lock file indicating that a version of a dependency is required but it isn't available locally, but I can't think of many others off the top of my head.

[sfackler]

Adding a dependency that you already have locally forces a registry update for sure. I seem to remember that offline cargo updates would get you stuck with a required registry update, but that doesn't seem to be the case in the version that ships with 1.2.0.

[sfackler]

A --offline flag could be a decent way to deal with this by making people opt-in to a possibly old view of the world.

[alexcrichton]

We discussed this on IRC a bit yesterday.

Some conclusions were:

• @wycats points out that just because one app has populated the cache of crates doesn't always mean that the same cache is useful for another.
• @sfackler points out that if you're working with a tree of path dependencies then the "cache population" done by the top is indeed what's needed for working on the sub-dependencies

I think @sfackler's idea of an --offline flag would work well here. By default Cargo would talk to the network as necessary and then using --offline would indicate that it should use the local cache only, possibly tailoring the error messages appropriately to try not passing the flag if it fails.

[lilydjwg]

(I hope I've found the most related issue to comment.)

@alexcrichton I'd like to have an --offline flag to completely work offline too, using local cache when possible, failing when there aren't satisfying cache.

I've been offline for some days. I tried to write a simple program in those days, but Cargo always want to update its registry and thus failed.

But I did have all my dependencies cached. I had to manually find and extract source files from cached crates, then manually call rustc with appropriate options, then build my little program. It was tedious.

I want to write Rust programs just as I do with Python or C: when I have all necessary dependencies locally available, I can work with them; it only fails when I lack some of them.

[alexcrichton]

@lilydjwg this issue would indeed suffice for that use case!

[llogiq]

Another data point, while I'd welcome a --offline flag, I think we should be able to work offline by default, issuing a suitable warning to the user and optimally setting some kind of flag to warn the user that they may want to update when they next build while online.

[MagaTailor]

Why does this need to go online?

$ cargo clean -p rayon
 Downloading clippy v0.0.64
error: unable to get packages from source

A regular cargo clean doesn't require net access - what's going on?

[benaryorg]

Probably also of interest is #2895 and @aidanhs' comment in particular.

[oconnor663]

Another example use case for this stuff that I run into frequently: I like playing with Hello World examples when I'm trying out a new library, and I have a little shell function for creating them quickly. I frequently get myself in a state where one of my hello-world projects is building fine, but a new one with the same dependencies won't build without network access, probably because it's trying to create a lock file first. (Copying the lock file from the old project doesn't always seem to help, maybe because there's other metadata in there that I'd need to tweak and I just don't know enough to do it right.)

I don't think solving my case wouldn't necessarily require adding flags. Maybe there could be a separate command (separate program entirely?) that knows how to inspect all my caches and then put my project in a state where it's guaranteed to build from those caches. I assume that Cargo's directory layout isn't stable, though, so it might make sense to have a tool like this inside the project? A tool like this might also be useful for people who want to set up their own vendoring, if they have a directory full of libraries and they want to rewrite their Cargo.toml to use all of them by path, though the inputs in that case are pretty different.

[matklad]

All the error messages along the way, especially in resolution, would also need to indicate that allowing Cargo to talk to the registry might help make the errors go away.

Why do we need a special error message here? I think using cached deps from other apps can be viewed as an optimization. What about the following algo for adding a dependency?

1. Try to update the registry. If this fails:
2. Stuff the error somewhere.
3. Try to resolve dependencies using cached registry and already downloaded packages.
4. If succeeded, write the lockfile. It might not be as fresh as theoretically possible, but it will be consistent (like if you have traveled back in time to the previous registry update).
5. If fail, just show the error message from step 2.

[aidanhs]

At minimum I'd expect a warning if cargo has fallen back to trying to resolve using offline packages (though my personal preference is still to have a flag you need to use to opt in, per my comment on the other issue).

[alexcrichton]

@matklad yeah that's one possible route to take here, but it's subtly different enough that we need to be sure to tread lightly. That strategy would change Cargo's behavior around resolution, which runs the risk of causing lots of confusion down the line if you're not always getting the latest version of dependencies.

(just something to keep in mind)

[caipre]

I ran into this last night when I found that the internet wasn't working at my home. No problem, I figured, since I happened to already have the packages I needed in my local cache. In fact, cargo insisted on updating the registry, even when my Cargo.toml pinned the dependencies to versions I had available locally. I was starting a new project, so I assume cargo wouldn't have complained if a Cargo.lock file were present. Echoing the sentiments of others: cargo should just make the Cargo.lock file based on my current known registry.

[kornelski]

I would love ability to generate the Cargo.lock offline. I need to be able to add new dependencies to the project while working offline — I do have all the crates on my disk locally, especially when I split one of my crates in two.

When I add a new dependency with path = "../foo" to my Cargo.toml it's annoying that cargo insists on connecting to the Internet, even to add a private dependency that doesn't exist on crates.io.

[alexcrichton]

@pornel

When I add a new dependency with path = "../foo" to my Cargo.toml it's annoying that cargo insists on connecting to the Internet, even to add a private dependency that doesn't exist on crates.io.

FWIW this was a historical bug in Cargo which I believe was fixed recently. Does nightly Cargo exhibit this behavior for you?

[kornelski]

@alexcrichton Yes! I've tried again with the latest nightly and I was able to add a dependency offline. Great!

[aturon]

Closing in favor of https://github.com/rust-lang/cargo/issues/4686. While "airplane mode" doesn't have quite as nice a workflow, we feel it's the right place to start.