Crates.io: VSCode crates extension could use an API that only returns a crate's versions

[LucianoBestia]

The result of the request
https://crates.io/api/v1/crates/web-sys
is very big > 1MB
because of the many "features": {} nodes repeated for every version node.
Without the "features" nodes it is only 30kB.
Is there a way to get the response without the features nodes?
They are rarely useful when requesting the list of all the versions for a crate.

[jtgeibel]

To add a bit more context, here is what I'm seeing locally in Firefox dev-tools:

• transferred: 107.1KB
• content-encoding: gzip
• uncompressed size: 0.99MB

As far as I can tell, we don't use this information in the frontend anywhere. If we drop the field by default that might break existing 3rd party clients of our API. Alternatively, we could have the frontend pass a query parameter to avoid returning the information (and possibly drop it from the query as well).

I think web-sys is an edge-case here, but this is something we should keep an eye on.

[LucianoBestia]

More details about my problem:
I use VSCode with the extension "crates". This extension suggests the versions of the crate.
Only for web_sys I get this error about the size of the blob:

web-sys: StatusCodeError: 403 -
"{"message":"This API returns blobs up to 1 MB in size.
The requested blob is too large to fetch via the API,
but you can use the Git Data API to request blobs up to 100 MB in size.",
"errors":[{"resource":"Blob","field":"data","code":"too_large"}],
"documentation_url":"https://developer.github.com/v3/repos/contents/#get-contents\"}"

[LucianoBestia]

I found now that there is also this query:
https://crates.io/api/v1/crates/web-sys/versions
But it also has all the "features" bloat.

pass a query parameter to avoid returning the information
I suppose you intended something like this:
https://crates.io/api/v1/crates/web-sys/versions/nofeatures

[carols10cents]

web-sys: StatusCodeError: 403 -
"{"message":"This API returns blobs up to 1 MB in size.
The requested blob is too large to fetch via the API,
but you can use the Git Data API to request blobs up to 100 MB in size.",
"errors":[{"resource":"Blob","field":"data","code":"too_large"}],
"documentation_url":"https://developer.github.com/v3/repos/contents/#get-contents\"}"

That response looks like it's coming from GitHub, not crates.io. I see this issue you raised in the crates extension's repo, and after looking at the extension's code, it looks like the problem might be that the extension is getting this info from the index on GitHub, where the web-sys file is 1.1MB, rather than from the crates.io API.

@serayuzgur what information do you need from crates.io to make the crates extension work? If our current API isn't providing the right information, we'd consider adding a new API response if it would enable new use cases like yours.

[serayuzgur]

@carols10cents Hi, we were using crates.io api before. @sgrif stated too many number of concurrent requests are not allowed so I switched to gihub index. This is the issue .

Api call rates, resource sizes and performance problems will occur if I continue to use APIs as 3rd party.
According to me , I have 3 options;

• crates.io can help me and provide a service which gets all crate names in one request and returns an optimized response.
• I can write a service which gets all crate names in one request and returns an optimized response. (sponsors needed)
• We can switch to Blob api of github and see what will happen next.

[carols10cents]

* crates.io can help me and provide a service which gets all crate names in one request and returns an optimized response.

Do you need just the crate names or do you need crate names and their versions? What would be the ideal JSON format of the response?

[serayuzgur]

@carols10cents A simple service will be enough. I will send you the names array and expect to have a map with the name of the crates with versions (not yanked) in it.
Since query params will be too large for many cases , my suggestion is using PUT instead of GET.

// Request
["web-sys","image"]
// Response with not yanked versions
{
  "web-sys": {
    "versions": [
      "0.3.35",
      "0.3.34",
      "0.3.33",
      "0.3.32"
    ]
  },
  "image": {
    "versions": [
      "0.1.15",
      "0.1.14",
      "0.1.13",
      "0.1.12"
    ]
  }
}

[jtgeibel]

@serayuzgur is it possible to use the local registry maintained by cargo under ~/.cargo/registry/index/github.com-1ecc6299db9ec823/.git? This wouldn't have any size limitations and would allow the extension to work while offline.

That directory contains a bare git repository (a repo with no checked-out working directory), so it is necessary to navigate the tree starting from the origin/master ref. For example, the following command will return the same contents that you're currently obtaining via the GitHub API:

git --no-pager --git-dir=$HOME/.cargo/registry/index/github.com-1ecc6299db9ec823/.git show origin/master:we/b-/web-sys

• --no-pager ensures that git will not automatically use a pager (like less) when running the command from a terminal
• --git-dir= points to the local registry index repository
• show - the sub-command to show the contents of an object.
• origin/master:we/b-/web-sys - specify the object to dump in <rev>:<path> syntax.

There are probably JS packages that can walk the repo directly, but I also think it would be reasonable to shell out to a git command if that's possible to do from a VSCode extension.

[serayuzgur]

@jtgeibel Hi, it seems possible but I fear that if I add this kind of logic in it it will lose stability and become much more dependent on the local machine. Also freshness of the local repository is a question mark for me, when it get updates? What do you think?

[LucianoBestia]

I think the local cache of crates.io index is updated on every cargo build.
So for an active developer it is the actual situation.
I wrote a little cli to return versions from this cache just to test it.

use ansi_term::Colour::{Green, Red, Yellow};
use clap::{App, Arg};
use dirs;
use regex::Regex;
use std::env;
use std::fs;
use std::path::Path;
use unwrap::unwrap;

/// Linux only. Only one argument: crate name.
fn main() {
    let matches = App::new(env!("CARGO_PKG_NAME"))
        .version(env!("CARGO_PKG_VERSION"))
        .author(env!("CARGO_PKG_AUTHORS"))
        .about(env!("CARGO_PKG_DESCRIPTION"))
        .arg(
            Arg::with_name("crate_name")
                .required(true)
                .help("crate name to search")
                .takes_value(true),
        )
        .get_matches();

    let mut crate_name = "";
    if let Some(cn) = matches.value_of("crate_name") {
        crate_name = cn;
        println!("Crate name: {}", Yellow.paint(crate_name));
    }

    if !crate_name.is_empty() {
        let versions = get_versions(&crate_name);
        println!("{:?}", versions);
    }
}

/// get versions from the local cache of the crates.io index
pub fn get_versions(crate_name: &str) -> Vec<String> {
    let mut versions = Vec::new();
    // the linux shell home dir symbol ~ or HOME is not expanded in raw rust
    // I must use the dirs crate
    let mut folder = unwrap!(dirs::home_dir());
    folder.push(".cargo/registry/index/github.com-1ecc6299db9ec823/.cache");
    //interesting rules for the folder structure
    if crate_name.len() == 1 {
        folder.push("1");
    } else if crate_name.len() == 2 {
        folder.push("2");
    } else if crate_name.len() == 3 {
        folder.push("3");
    } else {
        folder.push(&crate_name[0..2]);
        folder.push(&crate_name[2..4]);
    }
    println!("Folder: {:?}", &folder);
    let dir = Path::new(&folder);
    for entry in unwrap!(fs::read_dir(dir)) {
        //crazy amount of unwrap to go from path to a normal string
        let path = unwrap!(entry).path();
        let entry_file_name = unwrap!(unwrap!(path.file_name()).to_str());
        if entry_file_name == crate_name {
            //read the content and find versions
            let file_content = unwrap!(fs::read_to_string(path));
            //I will use regex to find all "vers": "0.3.3",
            let re = unwrap!(Regex::new(r#""vers":"(.*?)".*?"yanked":(.*?)[,}]"#));
            for cap in re.captures_iter(&file_content) {
                println!("version: {} yanked: {}", &cap[1], &cap[2]);
                if &cap[2] == "false" {
                    versions.push(cap[1].to_string())
                }
            }
            break;
        }
    }
    //return
    versions
}

[dependencies]
unwrap = "1.2.1"
clap = "2.33.0"
ansi_term = "0.12.1"
dirs="2.0.2"
regex = "1.3.3"

[LucianoBestia]

Also the command

cargo update --dry-run

updates the local cache of crates.io index.
And writes what crates have new versions for this cargo.toml/cargo.lock.
But because of --dry-run it does not automatically change cargo.toml.

[LucianoBestia]

Just to continue the dialog, I put the code on github.
https://github.com/LucianoBestia/cargo_reg_local

[serayuzgur]

Hi @LucianoBestia , thanks for the code. I have to rewrite it with ts and test if it works but it will take some time since I do not have too much free time these days.