Cirrus-ci-docs: Cirrus CI tasks indexed by search engines

[dotdoom]

Expected Behavior

Tasks are not indexed by search engines. An individual task or build run rarely contains anything useful, and imposes more of a search noise or even attracts unnecessary attention to build.

Real Behavior

• Google: cirrusci osx throttle
  See search results down on the first page, indexed from some issues on GitHub.
• Google: "task scheduling was delayed" to find all builds

[RDIL]

I’ll take care of this. Just to clarify, you don’t want them showing?

[fkorotkov]

Yeah, seems noise. Should add robots.txt with something like:

User-agent: *
Disallow: /task/
Disallow: /build/

Or maybe even

User-agent: *
Disallow: /

🤔

[dotdoom]

@RDIL thank you!

To clarify, I don't mind my own build hanging out there, so it's not worth going through Google's "remove data" process just for me. But I think in general build results should not be indexed. Subject to a debate, of course.

[RDIL]

Fixed now.

[fkorotkov]

Should be deployed in a few minutes