Renovate: Rust (Cargo) support

[cyplo]

This is a:

• [ ] Bug report (non-security related)
• [x] Feature request
• [ ] I'm not sure which of those it is

I'm using:

• [x] The Renovate GitHub App
• [ ] Self-hosted GitHub
• [x] Self-hosted GitLab
• [ ] Self-hosted VSTS

Please describe the issue:

This project seems extremely cool !
Would be even cooler with support for Rust packages (crates) ;)

I am happy to help adding the support if given some guidance/mentorship.

thank you and have a great day !

[rarkins]

@cyplo thanks for the suggestion and offer! I will definitely take you up on it.

What I'll do first is write up a generic doc on how to add any new language/package manager. I'm getting these types of requests a lot recently so it's time I wrote it up.

Then, I can create a new branch for you with the basic structure required to add rust/cargo which would then allow you to take over the branch and fill in the logic (parsing Cargo.toml to extract dependencies, looking up new versions, and patching the new versions into the file).

[rarkins]

Doc: https://github.com/renovateapp/renovate/blob/master/docs/adding-a-package-manager.md
Starting branch for Cargo: https://github.com/renovateapp/renovate/tree/feat/1870-rust-cargo

Here is an example new package manager branch PR that is almost finished: https://github.com/renovateapp/renovate/pull/1874

Cargo will be more complex than the buildkite PR though because of (a) version ranges, and (b) lock files. If you think that supporting exact versions only or non-lockfile support only initially would be OK then we could get the initial support faster.

There are three main parts remaining:

1. (extract.js) Extract all existing dependencies & versions from the package files
2. (pacage.js) Look up if any of those have newer versions available
3. (update.js) Patch the package files before raising PRs

It looks like the following will need to be added as part of this:

• lib/datasource/crates
• lib/util/cargo-semver (maybe)

The semver handling is a lot like node's, but not exactly. We can probably "proxy" a lot to the node.js semver package but write a few special cases for Cargo.

[cyplo]

Wow, that's a lot of work done to help me witch such a quick turnaround, thank you so much for this ! :)
I may have some more code-related questions when I dive more into this.
What would be the best way to ask them and to whom ? Do not want to steal too much time of others :)

[rarkins]

Feel free to ask questions in this issue and I will probably be the one answering. I don’t think it should disturb others.

We have a Gitter channel linked from the README that you may find convenient too, although at some hours there’s no one online.

FYI I’m in GMT+2 Timezone

[cyplo]

Heya ! I typically like to start with writing one test on the outside of the problem to see how to proceed and to guide me what to do next. Where would be a good place for this test to live ? The test could be something like 'given this cargo.toml when I run renovate I expect this crate to be marked as to be upgraded' or some such. Thank you :)

[rarkins]

Because there are so many languages getting added, the code is modular in a way that we don’t really have that type of full end to end test (mocking all the way through would be a pain). We test lib/manager/* inside test/manager/* and break our the extract/lookup/update tests separately

[cyplo]

Coolio ! Will keep you posted :) I know Rust better than JS so this may take some time :) If there is anyone else looking at this and thinking about helping - let me know, we can sync on this :)

[rarkins]

Don’t worry, I’ll be helping, but I don’t know Rust. First step is the extractDependencies function that takes the Cargo.toml content and returns an array of dependencies inside.

[rarkins]

I've added the code "skeleton" necessary for this work to begin. Now work needs to be done to fill in the lib/manager/cargo/* files and associated tests.

[cyplo]

I must admit I totally forgot about this one - sorry about that. Will definitely take a look - but also, for others reading this - please feel free to pick this up as well :)

[nchashch]

Extracting dependencies -- extractPackageFile

Parsing TOML

This should be pretty simple using toml-node library.

Implementation route

Probably the first thing for a stub implementation would be to get a Cargo.toml file, extract the [dependencies] section, and just get a list of name, version pairs.

Then we will need to deal with multiple dependency types and figure out a way to parse version requirements.

Types of dependencies

• [dependencies]
• [dev-dependencies]
• [build-dependencies]
• Platform specific: [target.*.dependencies]
• Feature specific: [dependencies.*]

Combinations are allowed:

• [target.*.dev-dependencies]
• [dev-dependencies.*]
  etc.

Types of version requierments (see reference)

• Caret requirements
• Tilde requirements
• Wildcard requirements
• Inequality requirements
• Multiple requirements

Probably will need to parse these in order to figure out which dependencies have to be updated. Might be possible to reuse existing code if other managers use the same or similar version requirements format.

Reference on specifying dependencies in Cargo

Specifying dependencies

Handling Cargo.lock files

Implementation route

It looks like Cargo.lock can be updated with cargo update command. It also updates local crates.io index.

Although it isn't clear if renovate should do it by default, because if Cargo.lock is checked into a git repository it is supposed to be updated when there is a working version of the code for reproducible builds, and automatically updating dependencies doesn't guarantee that the code will work.

Probably there should be some kind of mechanism for allowing users to choose if they want to update Cargo.lock automatically.

Docs

• Cargo.toml vs. Cargo.lock
• Why do binaries have Cargo.lock in version control but not libraries

[rarkins]

@cyplo thanks for this great summary.

Do the platform-specific and feature-specific dependencies change anything about how we look them up? Or can Renovate be "ignorant" of what these means and simply look up the dependency name in crates.io?

For example, I'll use this from the Cargo docs:

[target."i686/linux.json".dependencies]
openssl = "1.0.1"
native = { path = "native/i686" }

In this case the platform is "i686/linux.json" and the package name is "openssl". When we lookup the crate.io index for openssl, do we need to do any filtering based on platform value? Or we simply take the most recent version of openssl available? I'm wondering if it's possible to publish versions to that support only certain platforms and we need to filter based on both package name as well as which platforms each release supports.

For parsing the Cargo.toml to extract dependencies we should consider whether not to parse it as toml and instead just create a dumb "per line" reader for extracting. The reason is that it makes the update function much easier if you just tell it the line number and new value to update, instead of parsing then replacing and then stringifying (and hoping your stringifying matches the existing format the user prefers). If you look at the Dockerfile and Docker Composer extract functions you can see how they don't fully "parse" the file and instead process it line by line.

If Cargo uses its own syntax for semver ranges then we will probably need to make lib/versioning/semver-cargo. You can see semver-composer as an example of that already.

Finally, for lockfile updating, the usual rule is: if you update the package file, then you need to update the lock file too. The main problem is we want to make sure we only update the package(s) we are updating, and not accidentally update all of them. e.g. we maybe don't run cargo update but instead cargo update [email protected] or something like that.

[nchashch]

Do the platform-specific and feature-specific dependencies change anything about how we look them up? Or can Renovate be "ignorant" of what these means and simply look up the dependency name in crates.io?

No, feature specific dependencies are looked up the same way - same for platform specific dependencies. So renovate can ignore the difference between normal and feature specific dependencies.

For parsing the Cargo.toml to extract dependencies we should consider whether not to parse it as toml and instead just create a dumb "per line" reader for extracting.

The format is simple enough, so a dumb "per line" reader should work.

Finally, for lockfile updating, the usual rule is: if you update the package file, then you need to update the lock file too. The main problem is we want to make sure we only update the package(s) we are updating, and not accidentally update all of them. e.g. we maybe don't run cargo update but instead cargo update [email protected] or something like that.

Renovate can just run

$ cargo update -p dependency_name

for every updated dependency, and it would only updatedependency_name in Cargo.lock.

[rarkins]

This is looking quite "simple" so far. I think we should implement #2780 first, it can be merged to master first and then the lib/manager/cargo/* code added in a later PR.

We also need to decide what to do with git dependencies (e.g. hosted on github.com). I think to begin with we add a skipReason for them and add support in a second phase if Cargo supports git + versioning.

[nchashch]

Cargo also allows users to pull versions of dependencies, which aren't yet published to crates.io from GitHub. This might require some special logic on the renovate end. See reference:
https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#working-with-an-unpublished-minor-version

[nchashch]

And there is this syntax for specifying dependenceis:

[dependencies.awesome]
version = "1.3.5"
default-features = false # do not include the default features, and optionally
                         # cherry-pick individual features
features = ["secure-password", "civet"]

reference: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#choosing-features

It looks like a dependency can either be specified as an inline table:
reference: https://github.com/toml-lang/toml#user-content-inline-table

[dependencies]
libc = "0.2.43"
bitflags = "1.0.4"
pcap-sys = { version = "0.1", path = "pcap-sys" } # Inline table
pnet = { version = "0.21.0", optional = true, default-features = false} # Inline table

Or as a normal table:
reference: https://github.com/toml-lang/toml#user-content-table

[target.'cfg(windows)'.dependencies.winapi]
version = "0.3.6"
features = ["ws2def", "ws2ipdef"]

So the extractPackageFile function will have to handle that.

[rarkins]

@nchashch I think we can defer pre-release github version handling until a later PR, but it sounds like we should handle both types of table from the beginning?

[rarkins]

I'm always OK with deferring features to later if:

• What we do support is useful on its own (i.e. it's OK for some users to support feature A but not B yet)
• We handle the lack of support gracefully (e.g. do not print errors or generate invalid outputs)

[renovate-bot]

:tada: This issue has been resolved in version 14.37.0 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket: