Hugo: With defaultContentLanguageInSubdir: true, a robots.txt file should be created at the root

[martignoni]

With defaultContentLanguageInSubdir: true, various robots.txt files are wrongly created in languages subdirs, and no robots.txt is created at the root of the website.

Refs:

• RFC: https://datatracker.ietf.org/doc/draft-koster-rep/
• https://en.wikipedia.org/wiki/Robots_exclusion_standard

See: https://discourse.gohugo.io/t/multilingual-site-always-get-a-noindex-tag-in-root-index-html

[divinerites]

Thanks a lot @martignoni.

And I suspect this will lead to the same issue for 404 page (see https://discourse.gohugo.io/t/404-html-not-generated-on-root-if-defaultcontentlanguageinsubdir-true/14048).

A bundle 😸

EDIT / Done with #5161

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. The resources of the Hugo team are limited, and so we are asking for your help.
If this is a bug and you can still reproduce this error on the master branch, please reply with all of the information you have about it in order to keep the issue open.
If this is a feature request, and you feel that it is still relevant and valuable, please tell us why.
This issue will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

[martignoni]

Still present as of Hugo 0.53.

[davidsneighbour]

I am not sure if I understand that right, but if in Hugo's current configuration there is no robots.txt in the root directory and the language directories have one individual robots.txt each, then this is against the way robots.txt is supposed to work:

https://support.google.com/webmasters/answer/6062596?hl=en

Robots.txt has to be always at the domain root, not in a subfolder. If that is what happens here it should be marked as a bug, not an enhancement.

[ulziibay]

Hello @martignoni, I believe I ran into issue after reading https://discourse.gohugo.io/t/how-to-have-root-static-folder-with-multilingual-setup/16902/2. I have a quesiton,

_with this issue withstanding, how does your toy website at https://github.com/moodlebox/moodlebox.net/blob/master/config.yml, know to put the robots.txt, sitemap.xml at the root of public folder as well as in public/en and public/de and public/fr?_

With my current config below, I am unable to get hugo to copy over the robot.txt file or the sitemap.xml.

At the moment, https://mitpu.org/ DO NOT redirect to https://mitpu.org/en, unliked your website.

Thanks.

DefaultContentLanguage = "en"
defaultContentLanguageInSubdir = true
baseURL = "https://mitpu.org/"
languageCode = "en-us"
title = "MITPU"
theme = "timer-hugo"
disqusShortname = "mitpu"

# Enable robots.txt generation
enableRobotsTXT = true

# We do redirects via Netlify's _redirects file, generated by Hugo (see "outputs" below).
disableAliases = true

# We Used Ionicons Icon font for Icon, for more details check this link - https://ionicons.com/v2/

[languages]
    [languages.en]
        languageName = "English"
        title = "MITPU"
        contentDir = "content/en"
        weight = 2
        baseURL = "https://mitpu.org/en"

        # Site Menus
        [languages.en.menu]
            # Navbar Menus
            [[languages.en.menu.nav]]
                name    = "Hobby Groups"
                url     = "/groups"
                weight  = 2
            [[languages.en.menu.nav]]
                name    = "Podcast"
                url     = "/podcast"
                weight  = 3
            [[languages.en.menu.nav]]
                name    = "Referral"
                url     = "/referral"
                weight  = 4
            [[languages.en.menu.nav]]
                name    = "About"
                url     = "/about"
                weight  = 5
            [[languages.en.menu.nav]]
                name    = "Contact"
                url     = "/contact"
                weight  = 6

        # Site Params
        [params]
            home = "Home"
            dateFormat = "6 January 2006"
            #
            # You can use either an image or text as logo.
            # To use an image use 'logo' (logo itself) and 'logo_alt' (alternative text if image cannot displayed)
            # To use text as logo use 'logo_text'
            logo = "images/logo.png"
            logo_alt = "logo"
            #logo_text = "logo_text"

            # Banner Section
            [params.banner]
                enable  = true
                bgImage = "images/mitpu_background.jpg"
                heading = "Welcome to MITPU & We are a "
                btn     = true
                btnText = "Join now"
                btnURL  = "https://www.facebook.com/groups/582198341944244/"
                [[params.banner.flipText]]
                    title   = "Group of diverse talents"
                [[params.banner.flipText]]
                    title   = "501C certified NPO"
                [[params.banner.flipText]]
                    title   = "Community"

            # Homepage About Section
            [params.about]
                enable  = true
                title   = "MITPU Facts"
                image   = "images/about/mitpu_background.jpg"
                [[params.about.content]]
                    subtitle = "250"
                    description = "members"
                [[params.about.content]]
                    subtitle = "237"
                    description = "active members"
                [[params.about.content]]
                    subtitle = "189"
                    description = "requests that did not meet criteria"
                [[params.about.content]]
                    subtitle = "134"
                    description = "members who completed contact form"
                [[params.about.content]]
                    subtitle = "91"
                    description = "members who introduced #танилцья #introduction"
                [[params.about.content]]
                    subtitle = "190"
                    description = "Total number of posts"
                [[params.about.content]]
                    subtitle = "679"
                    description = "Total number of comments"
                [[params.about.content]]
                    subtitle = "2222"
                    description = "Total number of reactions"
                [[params.about.content]]
                    subtitle = "4"
                    description = "admins"
                [[params.about.content]]
                    subtitle = "5"
                    description = "board members"


            # Portfolio Section On Homepage
            [params.portfolio]
                enable  = true
                title   = "Latest Works"

    [languages.mn]
        languageName = "mongolian"
        baseURL = "https://mitpu.org/mn"
        title = "MITPU"
        contentDir = "content/mn"
        weight = 1
        # Navbar Menus
        [languages.mn.menu]
            [[languages.mn.menu.nav]]
                name    = "Хобби бүлэг"
                url     = "/groups"
                weight  = 2
            [[languages.mn.menu.nav]]
                name    = "Подкаст"
                url     = "/podcast"
                weight  = 3
            [[languages.mn.menu.nav]]
                name    = "Ажлын байрны зууч"
                url     = "/referral"
                weight  = 4
            [[languages.mn.menu.nav]]
                name    = "Бидний тухай"
                url     = "/about"
                weight  = 5
            [[languages.mn.menu.nav]]
                name    = "Холбогдох"
                url     = "/contact"
                weight  = 6

        # Site Params
        [languages.mn.params]
            home = "Нүүр"
            dateFormat = "2006/01/29"
            #
            # You can use either an image or text as logo.
            # To use an image use 'logo' (logo itself) and 'logo_alt' (alternative text if image cannot displayed)
            # To use text as logo use 'logo_text'
            logo = "images/logo.png"
            logo_alt = "logo"
            #logo_text = "logo_text"

            # Banner Section
            [languages.mn.params.banner]
                enable  = true
                bgImage = "images/mitpu_background.jpg"
                heading = "MITPU-д тавтай морил & Бид бол "
                btn     = true
                btnText = "Бидэнтэй нэгдэх"
                btnURL  = "https://www.facebook.com/groups/582198341944244/"
                [[params.banner.flipText]]
                    title   = "Group of diverse talents"
                [[languages.mn.params.banner.flipText]]
                    title   = "501C-р баталгаажсан ашгийн бус байгууллага"
                [[params.banner.flipText]]
                    title   = "Community"

            # Homepage About Section
            [languages.mn.params.about]
                enable  = true
                title   = "MITPU-н үзүүлэлтүүд"
                image   = "images/about/mitpu_background.jpg"
                [[languages.mn.params.about.content]]
                    subtitle = "250"
                    description = "гишүүн"
                [[languages.mn.params.about.content]]
                    subtitle = "237"
                    description = "идэвхтэй гишүүн"
                [[languages.mn.params.about.content]]
                    subtitle = "189"
                    description = "хүсэлтийг шаардлага хангаагүйн улмаас цуцалсан"
                [[languages.mn.params.about.content]]
                    subtitle = "134"
                    description = "гишүүн холбоо барих маягтийг бөглөсөн"
                [[languages.mn.params.about.content]]
                    subtitle = "91"
                    description = "гишүүн өөрсдийгөө танилцуулсан #танилцья"
                [[languages.mn.params.about.content]]
                    subtitle = "190"
                    description = "фб бичвэр/posts"
                [[languages.mn.params.about.content]]
                    subtitle = "679"
                    description = "сэтгэгдэл/comments"
                [[languages.mn.params.about.content]]
                    subtitle = "2222"
                    description = "Хариу үйлдэл / reactions"
                [[languages.mn.params.about.content]]
                    subtitle = "4"
                    description = "админ"
                [[languages.mn.params.about.content]]
                    subtitle = "5"
                    description = "ТУЗ-ийн гишүүн"


            # Portfolio Section On Homepage
            [languages.mn.params.portfolio]
                enable  = true
                title   = "Сүүлийн үед гүйцэтгэсэн ажлууд"


    # Footer section
    [params.footer]
        customized  = true # Set this variable to true if you customized the design.
        copyright_owner = "Themefisher"
        copyright_since = "2018"
        [[params.footer.social]]
            icon = "ion-social-facebook"
            link = "https://www.facebook.com/groups/582198341944244/"
        [[params.footer.social]]
            icon = "ion-social-linkedin"
            link = "https://linkedin.com"
        [[params.footer.social]]
            icon = "ion-social-gitlab"
            link = "https://gitlab.com/mitpu/"

[martignoni]

with this issue withstanding, how does your toy website at https://github.com/moodlebox/moodlebox.net/blob/master/config.yml, know to put the robots.txt, sitemap.xml at the root of public folder as well as in public/en and public/de and public/fr

A sitemap.xml and robots.txt are created automatically by Hugo in all language subfolders.

A sitemap.xml is also created automatically in the root folder, but a robots.txt is not.

[ulziibay]

Thanks for the response!

However, in my case sitemap.xml is not even generated at the root, neither is robots.txt

[martignoni]

Dont know. This is not a support forum. Ask here.

[yktoo]

The issue is still there in v0.68.3, and pressing.

[nicoduesing]

The issue is still there in v0.72.0.

[davidsneighbour]

I can't believe this is still an issue. A robots.txt has to be in the root directory of a website. Where _website_ means domain.

Use case 1: If the language versions are in subdirectories then the (collective) robots.txt has to be in the root directory, not the language directories.

Use case 2: If the language versions are in subdomains the robots.txt needs to be in the root directory of the subdomain. For Hugo this means in the directory where the subdomain of the language is mapped.

There is no connection to the locations where a Sitemap could/should/might be. Please don't connect these two things. The robots.txt for a domain is in the root directory of that domain.

[martignoni]

There is no connection to the locations where a Sitemap could/should/might be. Please don't connect these two things. The robots.txt for a domain is in the root directory of that domain.

Right. I've updated the description issue.

[nicoduesing]

If defaultContentLanguageInSubdir is

• false the 404.html and robots.txt is generated in the root directory and in all language directories.
• true it is only generated in all language directories. (Because there is no language rendered in the root directory.)

So it is wrong in both cases but only gets a problem, if defaultContentLanguageInSubdir is set to true. In the first case, there are _only_ too many robot.txts.

[nicoduesing]

I've created a pull request to fix the robots.txt I'm not sure how hugo should behave with the 404.html. If that should change, too, you can do similar changes for it.

[martignoni]

AFAIK, 404.html behavior is currently OK, since its localisation makes sense.