Gitea: Make default robots.txt

[theel0ja]

My gitea instance is seeing ton of requests from Googlebot to /user/login for example.

[lunny]

see https://docs.gitea.io/en-us/customizing-gitea/

[lafriks]

@lunny request was to provide one by default with recommended rules

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[karthanistyr]

I naively deployed a vanilla gitea docker image on a subdomain and lacked a robots.txt.
Bots hence came and hit a high number of /archive URLs, creating loads of artifacts and consuming a large space on the disk.

I think a default robots.txt forbidding access to this sort or URLs (and other sane defaults) would be really useful for ease of deployment.

[Mikaela]

Is this a duplicate of https://github.com/go-gitea/gitea/issues/705 or the other way?

[lunny]

@Mikaela you are right. Let's move to #705 to discuss further.