Mastodon: want to get "Opt-out of search engine"

[osapon]

I would like information of "Opt-out of search engine indexing" to be included in API account entity and JSON of user information.
Even if I want to create an external search engine, I need to obtain from the user page whether it is opted out.
It is easy to use, if I can get it with toots or user information.

---

• [x] I searched or browsed the repo’s other issues to ensure this is not a duplicate.
• [ ] This bug happens on a tagged release and not on master (If you're a user, don't worry about this).

[ghost]

well, such a feature actually exists

i actually don't know what you mean. if the feature is enabled a meta tag called <meta content='noindex' name='robots'> will be placed into the profile pages and status html code. there's no need for an extra api.

[osapon]

It is wasteful communication to acquire the user page again after getting toots.
If the account entity contained in toots contains information, it can be reflected immediately.

[ThisIsMissEm]

Perhaps it'd be an idea to use the X-Robots-Tag header? https://developers.google.com/search/reference/robots_meta_tag

[osapon]

Currently, only HTML has "no-robot" indication.
The account data obtained from the API has no "no-robot" attribute.

I may make application that uses toots and accounts, I can not judge whether or not to show that user to the search engine crawler from the acquired JSON.
After acquiring with JSON, accessing the HTML again only to acquire the "no-robot" attribute also loads the instance.

[ThisIsMissEm]

Right, the X-Robots-Tag header on API responses would include either noindex or another robots option.

[ThisIsMissEm]

See also #7164 — potentially there needs to be a way for a API client to specify it is a bot, without relying on user agent allow, perhaps: X-Requested-With: Bot or X-Requested-With: Crawler and the API would return different or no content based on that?