Magento2: A few issues when you use /pub as DocumentRoot

[hostep]

Preconditions (*)

Steps to reproduce:

robots.txt

Update: The robots.txt part is no longer valid, was fixed in Magento 2.2.0

sitemap.xml

Steps to reproduce:

1. Make sure the Magento docroot is set to the pub directory
2. In the backend, go to Marketing => Site Map
3. Click 'Add Sitemap' (small inconsistency btw, no space here)
4. Enter 'sitemap.xml' into the Filename field, enter '/' into the Path field
5. Click 'Save & generate'

Actual result (*)

Problem: sitemap.xml is created in the Magento root directory and in the column 'Link for Google' it says: http://example.com/sitemap.xml, which is incorrect

Continuing:

1. Click on the sitemap entry in the table
2. Change the Path to '/pub/'
3. Click 'Save & generate'

Problem: sitemap.xml is created in the pub directory, but the column 'Link for Google' now says: http://example.com/pub/sitemap.xml, which is also incorrect

In this case a solution with a redirect/rewrite in the .htaccess/nginx config file isn't really possible as the path and filename of the sitemap are dynamic.

Would be great if Magento (or somebody else) can come up with a way to solve this :)

Thanks!

Expected result (*)

[piotrekkaminski]

@buskamuza is it being addressed?

[gordonknoppe]

+1 this issue needs attention. We should be able to use pub/ to serve from production.

To clarify what I found:

sitemap.xml:

By setting up my sitemap.xml to generate into the pub/ directory I was able to place a file in the sub-directory fine and the resulting XML doesn't have any issues with pub/ in paths the file.

robots.txt:

I see two issues here: Magento will always write to the root folder and there is no way that I can see to change this via configuration. And additionally, if you have set the path of your sitemap.xml to pub/ as above then the robots.txt file created in the root has pub/ in it's "Sitemap: *" path.

[hostep]

I just tested this with Magento 2.0.0 GA, and the issues still remain.

robots.txt

Steps to reproduce:

1. Make sure the Magento docroot is set to the pub directory
2. In the backend, go to Stores => Configuration => General => Design => Search Engine Robots
3. Enter something in the Edit custom instruction of robots.txt File field
4. Save

Problem: the robots.txt file is created in the Magento root directory, not in the pub directory, so it isn't accessible for web requests.

Possible solution for serving Magento from both the root and the pub directory:
Always generate the robots.txt file in the pub directory, and add some line to the .htaccess/nginx config file only in the root directory which redirects you to the robots.txt file in the pub directory when requesting /robots.txt.
Downside: this might confuse shop owners/developers, if they want to add a robots.txt file to the root directory themselves without using the Magento backend, their file won't be used.
What do you think?

sitemap.xml

Steps to reproduce:

1. Make sure the Magento docroot is set to the pub directory
2. In the backend, go to Marketing => Site Map
3. Click 'Add Sitemap' (small inconsistency btw, no space here)
4. Enter 'sitemap.xml' into the Filename field, enter '/' into the Path field
5. Click 'Save & generate'

Problem: sitemap.xml is created in the Magento root directory and in the column 'Link for Google' it says: http://example.com/sitemap.xml, which is incorrect

Continuing:

1. Click on the sitemap entry in the table
2. Change the Path to '/pub/'
3. Click 'Save & generate'

Problem: sitemap.xml is created in the pub directory, but the column 'Link for Google' now says: http://example.com/pub/sitemap.xml, which is also incorrect

In this case a solution with a redirect/rewrite in the .htaccess/nginx config file isn't really possible as the path and filename of the sitemap are dynamic.

Would be great if Magento (or somebody else) can come up with a way to solve this :)

Thanks!

[mcspronko]

Hi @tanya-soroka and @piotrekkaminski

Do you guys have any updates for this ticket?

Thanks,
Max

[mcspronko]

Hi @piotrekkaminski

Do you have any update about this bug?

Thanks,
Max

[Marko-M]

Luckily you can symlink your way around it, but yep, it's an issue.

[mcspronko]

Hi @Marko-M

Imagine every time marketing department will be asking developer to symlink generated sitemap.xml files :)

[Marko-M]

@mcspronko

Yep, an issue.

[mcspronko]

Hi @tanya-soroka

Do you have any update for this issue?
Thanks,
Max

[markalston]

Still broken in 2.1.0

[atIOCrON]

I too hoped this would be resolved in 2.1... we need a robots.txt file accessible on the frontend urgently and don't want to mess around with the hacks suggested above.

[hostep]

@piotrekkaminski: maybe you should change the label from improvement to bug. I assume it will get picked up earlier if this is tagged as a bug, which it clearly is :)
Also I don't see a MAGETWO- reference number, so this isn't tracked in your internal Jira issues?
I do see a MAGETWO-47607 in https://github.com/magento/magento2/issues/2802 which is related to this issue.

[piotrekkaminski]

I expanded scope of MAGETWO-47607 to include also robots issue.

[mcspronko]

Hi @piotrekkaminski

Any update on this issue?

cc: @pboisvert

[piotrekkaminski]

@mcspronko still in the queue to be fixed.

[mcspronko]

@piotrekkaminski it seems like it is not critical to have proper sitemap.xml and robots.txt files for marketing purposes. Let me fix this issue :), might be faster. So reported 2400 websites will benefit from having proper indexation by search engines.

[piotrekkaminski]

@mcspronko researching why it wasn't included in latest patch. I understand the importance of this issue.

[mcspronko]

Hi @piotrekkaminski

Any luck with findings?

[markalston]

I was able to workaround this problem by adding symlinks in pub to ../robots.txt and ../sitemap.xml respectively. Seems to work fine.

[liquidia]

still broken in 2.1.1

[Flamestyle]

My decision: the location of the files is standard in the site root.
Add to host config:

    location = /robots.txt {
        alias /var/www/example.com/html/robots.txt;
    }

    location = /sitemap.xml {
        alias /var/www/example.com/html/sitemap.xml;
    }

[johnhughes1984]

This is causing us a fair bit of hassle too, any update on potential release as still not resolved in 2.1.2

[ihor-sviziev]

Still have this issue in 2.1.2. Any updates on it?

[miketechtwo]

This is still an issue on 2.1.3...

[mcspronko]

Hi @maksek

Any updates for this issue?
Thanks.

[johnhughes1984]

Would like an update on this too please @maksek

[m0zziter]

Also registering interest in this issue.

[mikebranderhorst]

This is still an issue on 2.1.4...

[piotrekkaminski]

The issue is planned for 2.2.0 release (and possibly backport to earlier version in one of the patches) but we don't have a date yet.

[awwea]

I use it like this:

1. Create folders 'sitemaps' and 'robots' in magento web root
2. Setup magento to save sitemap.xml in folder 'sitemaps' (it works with additinal subfolders too)
3. Save robots.txt file in folder 'robots' in format: example.com.txt

Access of robots.txt as example.com/robots.txt
Access sitemap.xml as example.com/sitemaps/sitemap.xml

NGINX:

location /sitemaps {
    alias $MAGE_ROOT/sitemaps/;
}
location ^/sitemaps/\.xml { allow all; log_not_found off; access_log off; try_files $uri =404; }

location /robots {
    alias $MAGE_ROOT/robots/;
}
location ^/robots\.txt$ { allow all; log_not_found off; access_log off; try_files $uri =404; }
rewrite ^/robots\.txt$ /robots/$http_host.txt last;

Enjoy

[DavidLambauer]

@awwea works for me!

[balaganeshg]

I am going to work in group

[mcspronko]

@maksek can you please assign this ticket to me? Thanks.

[maksek]

@mcspronko how is progress on this one?

[mcspronko]

@maksek thanks for asking. I am thinking if only pub directory ahould be supported for robots.txt and sitemap.xml so all merchants might need to reconfigure their document root to be pub. Or having support for magento root and pub directories.

[johnhughes1984]

Hey @mcspronko @maksek

I know it'd be treated as a breaking change, but I'm in favour if dropping support for Magento root and just using pub - however I appreciate I may not be aware of all the knock on effects this would bring. I feel like the original decision to support both has probably got us to the point we're now at.

We all know pub is the recommendation for obvious reasons, but still nearly 18 months in it is painful or requires workarounds to use properly and affects deployment process. Also, it is not just this issue re sitemap.xml / robots.txt, there are other issues working with pub brings that quite frankly should not still be present, another example being you can't use custom error themes for maintenance mode: see #6403

I also believe there are still issues with the component manager not showing as per #7623

Further to this third party extension developers need to consider these different document roots and not all do so. It can get quite confusing working with a feed or export module for example, when the admin says the file is one location and on the server it's in another or hasn't been created.

For me this issue is bigger than just the one highlighted on this ticket and really should have some serious thought put into it by the Magento team.

Max, I'm a big fan our yours and I think it's great your taking this one on as a highly active community member and previous Magento core developer, so I salute your sir :) However, I really think this is such a key issue right now that it needs attention at senior level from Magento regardless of whether the implementation ends up being done by a community member or core developer.

I really don't want this to come across as a rant as I really appreciate the shift that has been made this year and @maksek and his team have done an incredible job to date and the new community powered approach shows how truly amazing the Magento ecosystem is. However as SI partners we can't be explaining to merchants why we have issues with deployment when 2.2 comes out.

cc/ @piotrekkaminski @pboisvert

[gordonknoppe]

Agree with @johnhughes1984, delete /index.php from the root directory and regression test against /pub/index.php. It should never have been there. /pub/index.php should be the only option.

[mcspronko]

@gordonknoppe
This will be a breaking change and will require MINOR version bump. What would be your advice for 99% of existing running Magento 2 websites who use root directory and have everything configured for it?

I think change should support both root and non-root and additional communication with all Magento agencies about pub as a preferable way of setting up Magento 2 servers.

[johnhughes1984]

@mcspronko @gordonknoppe

If it can't be removed (at least for now) due to backward compatibility issues then in my opinion we need to take action or push Magento to make it harder / less obvious at the very least.

It should be as frowned upon as editing the core or using the object manager and for example DevDocs could be changed to only provide guidance on how to install from /pub nor root.

Ideally as you would when thinking about code I think the /index.php approach should be 'deprecated' so to speak in the next release and removed altogether in a further future release - thus giving the community fair warning and time to make the switch.

I think it's important to move towards removing it, not only for the reasons in my previous comment re confusion and issues / errors it has caused, but for the main reason pub/index.php was included in the first place: security

[ihor-sviziev]

I think it would be great to mark /index.php as deprecated with description to use /pub/index.php instead.
And in few next releases we can remove it.

[hardyjohnson]

Is there a reason the Magento application would ever save (write) to the application root (or pub) vs the the writable path (./var)?

A preferable solution would be to enforce writing to the configured writable path and then shipping symlinks in the pub directory to the writable var location.

It seems strange to me that the application root would be assumed to be writable at all in a production environment.

[KevinMace]

I raised the issue of not being able to set a custom maintenance page when using pub as the document root https://github.com/magento/magento2/issues/6403 with EE support over a year ago, yet it's still not been resolved by them.

It's not acceptable that such an issue should still be outstanding over 12 months later, especially when you've got to put your site in to maintenance mode to do a deployment, showing your users an unbranded generic error page.

It certainly doesn't look good on the Magento EE support. I'm pleased to see the community is taking hold of the reins and trying to find a solution to the issue.

Sorry for the rant, I just wanted to share my frustrations and acknowledge the great work being done by the Magento community.

[magento-team]

Internal ticket to track issue progress: MAGETWO-67054

[okorshenko]

Internal tickets: MAGETWO-47607 and MAGETWO-67054 already closed and will be delivered in 2.2.0 release

[magento-engcom-team]

@hostep, thank you for your report.
The issue is already fixed in 2.2.0

[davidbrandastic]

2.2.3 it appears that adding a sitemap in /pub still causes "Sitemap: https://domain/pub/sitemap.xml" to be dynamically appended to the end of /robots.txt despite being an invalid URL. ( though robots.txt does work correctly after switching the docroot to /pub ).

The documentation does not currently advise creating an alias for "/sitemap.xml to /pub/sitemap.xml" and generating it in "/".

http://devdocs.magento.com/guides/v2.2/install-gde/tutorials/change-docroot-to-pub.html

[RishabhRkRai]

I make some folders inside var/ to use for my custom module to use for some purpose in Magento 2.1.7 but these also not works until I create a symlink to that folder inside pub/.

[czettnersandor]

Please note the sitemap url in the robots.txt is still wrong. I added this line to pub/.htaccess below the RewriteEngine on line:

RewriteRule    "^pub/sitemaps/sitemap.xml$"  "/sitemaps/sitemap.xml"  [PT]

Which made it work

[crtl]

Still is present in 2.2.4 CE.
Having pub as the docroot causes multiple issues like with sitemaps, and robots.txt.

[Bonobomagno]

still on 2.2.6. robots is generated right, but sitemap.xml not.

[ihor-sviziev]

Hi @Bonobomagno,
As this issue at tome point was fixed and closed, please create separate issue. If we'll be able to reproduce it - we'll mark it as "acknowledged"

[maderlock]

It was closed, but not fixed.

[ihor-sviziev]

Hi @maderlock,
If this issue still reproducing on latest magento versions - please report separate issue with steps to reproduce, actual and expected results.

[akosglue]

same in 2.3.0

[ci2014]

Still an issue 2.3.0

[ihor-sviziev]

Hi @engcom-backlog-nazar,
Could you re-check if this issue appeared in 2.3.x again? People complaining about it.

[Bartlomiejsz]

@akosglue @ci2014 @ihor-sviziev it seems like, according to https://github.com/magento/magento2/pull/9094 adding:
'directories' => [ 'document_root_is_pub' => true ],
to your app/etc/env.php resolves the issue.
Don't know how is it on clean installation, but installation/update script should probably check and add this entry if needed.

[m2-assistant[bot]]

Hi @engcom-backlog-nazar. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if you want to validate it one more time, please, go though the following instruction:

• [ ] 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 2. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!
  

• [ ] 3. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• [ ] 4. If the issue is not relevant or is not reproducible any more, feel free to close it.

[magento-engcom-team]

@engcom-backlog-nazar Thank you for verifying the issue.

Unfortunately, not enough information was provided to acknowledge ticket. Please consider adding the following:

• [ ] Add "Component: " label(s) to this ticket based on verification result. If uncertain, you may follow the best guess

Once all required information is added, please add label "Issue: Confirmed" again.
Thanks!

[magento-engcom-team]

:white_check_mark: Confirmed by @engcom-backlog-nazar
Thank you for verifying the issue. Based on the provided information internal tickets MAGETWO-99141, MAGETWO-99142 were created

Issue Available: @engcom-backlog-nazar, _You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself._

[ghost]

HI @ihor-sviziev adding this lines to app/etc/env.php solve the problem

'directories' => [ 'document_root_is_pub' => true ],

[hostep]

Maybe Magento's installation procedure should ask if you host from pub or not and then write that config setting to the env.php file? That might be a nice solution here (which only then works for new shops, not for existing shops).

That flag should also be documented somewhere, currently I can't find it on the devdocs site...

[ihor-sviziev]

Hi @hostep,
Good idea for asking for document root during setup process!
I also reported issue for documenting this flag https://github.com/magento/devdocs/issues/4093

[limajasachin]

I am working on this at #dmcdindia19

[magento-engcom-team]

@limajasachin thank you for joining. Please accept team invitation here and self-assign the issue.

[m2-assistant[bot]]

Hi @limajasachin. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if you want to validate it one more time, please, go though the following instruction:

• [ ] 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 2. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!
  

• [ ] 3. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• [ ] 4. If the issue is not relevant or is not reproducible any more, feel free to close it.

[limajasachin]

@magento-engcom-team give me 2.3-develop instance

[magento-engcom-team]

Hi @limajasachin. Thank you for your request. I'm working on Magento 2.3-develop instance for you

[magento-engcom-team]

Hi @limajasachin, here is your Magento instance.
Admin access: https://i-1146-2-3-develop.instances.magento-community.engineering/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

[limajasachin]

This issue is not reproducible.

[limajasachin]

I am working on this at #dmcdindia19

[michaeldyl520]

It can still be reproduced on 2.2.6.

[NetStorm84]

@magento give me 2.3-develop instance

[magento-engcom-team]

Hi @NetStorm84. Thank you for your request. I'm working on Magento 2.3-develop instance for you

[magento-engcom-team]

Hi @NetStorm84, here is your Magento instance.
Admin access: https://i-1146-2-3-develop.instances.magento-community.engineering/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

[victortodoran]

This is still an issue in 2.3.1

[m2-assistant[bot]]

Hi @progreg. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [x] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [x] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [x] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [x] 4. Verify that the issue is reproducible on 2.3-develop branch
  Details

  - Add the comment @magento give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[mcspronko]

Hey, @antonkril is this issue will ever be fixed?
The issue has been opened May 29, 2019, never been fixed and now a Bot requests additional details.

[progreg]

@mcspronko , an issue still reproducible and I have an idea how it can be fixed.

[ihor-sviziev]

Hi @mcspronko @progreg,
This issue actually fixes by adding one line in env.php file.
https://github.com/magento/magento2/issues/1146#issuecomment-482085895
Also I created issue for documenting it in https://github.com/magento/devdocs/issues/4093, but it still wasn’t done.
I think it’s better to document this option and close this issue, or you prefer better solution?

[Bartlomiejsz]

Hi @ihor-sviziev, sorry to interfere but don't you think it would be good idea to at least add new configuration to installation process that would automatically set this line for new instances?

[progreg]

@ihor-sviziev, Yeap you are right, it will fix the problem. Missed up your comment, thanks for the notice, seems this issue no longer need and it can be closed.

[progreg]

@Bartlomiejsz We can open a new issue for this needs as this one have many unneeded details and also touches on Robots module issue that is no longer relevant.