Wowchemy-hugo-modules: Generate robots.txt automatically
Currently, the robots.txt file is not generated automatically. It can be useful to generate it automatically based on predefined rules with the default being allow all access.
somnathrakshit
All 3 comments
The default intended behavior is for search engines to index all content, hence robots.txt is redundant in the general case.
gcushen
on 22 Oct 2018
That is true but many sites like Yandex prefer a robots.txt for SEO.
somnathrakshit
on 22 Oct 2018
At first I didn't have the robots.txt file and I noticed Google and Bing coming to our department websites and ONLY look for the robots.txt file! Nothing else! So, I guess now it should be implemented, otherwise you not get indexed!
in static/robots.txt put:
User-agent: *
Disallow:
And watch you access.log files!
phiser678
on 15 Mar 2019
👍3
Was this page helpful?
0 / 5 - 0 ratings
Related issues
gcushen
路
4Comments
petzi53
路
4Comments
CommonClimate
路
4Comments
chris-prener
路
3Comments
henningninneh
路
3Comments
Most helpful comment
At first I didn't have the robots.txt file and I noticed Google and Bing coming to our department websites and ONLY look for the robots.txt file! Nothing else! So, I guess now it should be implemented, otherwise you not get indexed!
in static/robots.txt put:
User-agent: *
Disallow:
And watch you access.log files!