Docs.rs: Add support for blacklisting crates

[pietroalbini]

Some of the crates on docs.rs are blacklisted for legal reasons, but are still up on crates.io. Before #529 was merged we disabled them globally by blocking the route in nginx, but I'd like to move away from that model.

Deleting the old content is already covered by the delete-crate command, but even with that new releases would still be published. We need to add a blacklisted_crates table containing the names of those crates, and change the docbuilder to refuse building those crates.

[pietroalbini]

Here are some mentoring instructions for this issue!

1. A new database migration should be added here, creating the database table.
2. In the build process, use the connection to query the table you just created, and return an error if the crate we're trying to build is blacklisted.
3. Add the commands cratesfyi blacklist add CRATE and cratesfyi blacklist remove CRATE to the CLI. The logic to add or remove a crate should not be on the binary, but on the main docs.rs crate.

You can learn how to contribute to docs.rs in the README, and feel free to ask questions here or on Discord (channel #docs-rs)! If you plan to work on this please leave a comment saying so, so we don't get multiple people working on the same issue.

[kvrhdn]

Hi @pietroalbini, thank you for the clear instructions! I would like to work on this.

[kvrhdn]

I have a question about how the filtering / blacklisting should function: if I understand this issue correctly, we want to prevent new versions of a known 'bad' crate to be built and added to doc.rs.

Q's:

• could there be a case in which only certain versions of a 'bad' crate are blocked and other not?
• do we want to be able to specify a version after which builds are stopped? I.e., "all versions of crate xyz after version 1.2.3".

The simple implementation I have working now only stores a crate name in blacklisted_crates and does not build a crate if the name exists in blacklisted_crates. Is this enough? (For now?)

[jyn514]

Closed in #539