Magento2: Images added to sitemap are "404 Not Found" on Nginx [Magento 2.1.0]

[RG-1]

Steps to reproduce

1. Magento 2.1.0 and Nginx with https://github.com/magento/magento2/blob/2.1.0/nginx.conf.sample
2. Go to Stores > Settings: Configuration > Catalog: XML Sitemap > Products Options
3. Set Add Images into Sitemap to "All"
4. Go to Marketing > SEO & Search: Site Map

5. Add and generate sitemap

   Expected result

6. URL of images shouldn't contain pub directory.

<image:image>
<image:loc>
https://example.com/media/catalog/product/s/a/sample-product-img.jpg
</image:loc>
<image:title>Sample product</image:title>
<image:caption>Sample product</image:caption>
</image:image>
<PageMap xmlns="http://www.google.com/schemas/sitemap-pagemap/1.0">
<DataObject type="thumbnail">
<Attribute name="name" value="Sample product"/>
<Attribute name="src" value="https://example.com/media/catalog/product/s/a/sample-product-img.jpg"/>
</DataObject>
</PageMap>

Actual result

1. Images are "404 Not Found"

<image:image>
<image:loc>
https://example.com/pub/media/catalog/product/s/a/sample-product-img.jpg
</image:loc>
<image:title>Sample product</image:title>
<image:caption>Sample product</image:caption>
</image:image>
<PageMap xmlns="http://www.google.com/schemas/sitemap-pagemap/1.0">
<DataObject type="thumbnail">
<Attribute name="name" value="Sample product"/>
<Attribute name="src" value="https://example.com/pub/media/catalog/product/s/a/sample-product-img.jpg"/>
</DataObject>
</PageMap>

[hostep]

Is not exactly the same issue, but might be related to: https://github.com/magento/magento2/issues/1146

[IlnitskiyArtem]

@RG-1 Unfortunately I could not reproduce this issue. May you please advise if this problem is still relevant for latest release Magento 2?

[hostep]

I haven't verified it it is still true on the latest version, but we definitely had the same problem with Magento 2.1.3 or 2.1.4 if I remember correctly.

We could work around it by setting the configurations:

• General > Web > Base URLs > Base URL for User Media Files to {{unsecure_base_url}}media/
• General > Web > Base URLs (Secure) > Base URL for User Media Files to {{secure_base_url}}media/

By default, these fields are empty and the 'pub' directory part was getting added to the image paths in the sitemap.xml files.

Be aware that you should test this with the 'pub' directory being used as docroot.

@IlnitskiyArtem: not sure if this helps? If you want I can test again using a plain vanilla Magento setup.

[IlnitskiyArtem]

@hostep I verified in 2.1.3 and 2.2.0 (develop) versions and I cannot reproduce this issue.
In the attachments the sitemap files for 2.1.3 and 2.2.0 versions.

2.1.3.txt
2.2.0.txt

[hostep]

@IlnitskiyArtem Ok, I was able to reproduce this again in Magento 2.1.7
I forgot this bug was triggered by the cronjobs. If you generate the sitemap.xml file using the adminhtml, there is no problem, it only happens when you let the sitemap.xml file generate using the cronjobs.

So steps to reproduce:

• Setup a clean Magento 2.1.7 shop and use 'pub' as docroot
• Create one product with one image
• Create a sitemap.xml, verify its content, there is no 'pub' in the image url
• In the configuration, go to: Catalog > XML Sitemap > Generation Settings
• Enable and put the start time one or two minutes from your current time
• Save and flush cache
• Manually run php bin/magento cron:run a couple of times until the generation of the sitemap.xml is triggered by this command
• Check the sitemap.xml file again, it now contains 'pub' in the image url.

Would be great if this could be fixed! The workaround mentioned above gets us around the issue, but not all Magento customers will discover this workaround, so a fix would be appreciated :)

Thanks!

[IlnitskiyArtem]

@hostep
Thanks for the additional information!
We've created internal ticket MAGETWO-70150 to address this issue.

[magento-engcom-team]

@RG-1, thank you for your report.
We've created internal ticket(s) MAGETWO-70150 to track progress on the issue.

[dvershinin]

Why hasn't the issue been fixed in 2 years?

[winterk80]

Still a bug in 2.1.7
1) /Pub/ is still being added to image urls causing SEO problems
2) Sitemap lastmod date does not change when you add products to or edit categories

[kamilszarmach]

distributed-cd

[magento-engcom-team]

@kamilszarmach thank you for joining. Please accept team invitation here and self-assign the issue.

[PascalBrouwers]

After founding the obscure setting 'document_root_is_pub' => true to fix my sitemaps because my site is server from the pub folder I see the images are also starting with 'pub' in the sitemap.xml
I can't believe this still hasn't been fixed.

[PascalBrouwers]

My quick workaround in NGINX:

    RewriteEngine on

    RewriteRule ^pub/(.*) /$1 [PT]`

[Boumii]

Still exist in 2.2.6
Backend generated sitemap.xml contains /pup/ in image src, when shops root is in /pup/.

Workaround by hostep works for me. Set generation time some minutes past to current time and generate sitemap with "bin/magento cron:run" generates sitemap without /pup/ in image path. Then I set frequency to monthly to not overwrite by automation.

[jaywilliams]

Another workaround for those who still are looking for a solution for this is to create a dummy pub symlink in the pub directory, like so:

$ ln -s . pub

[krisdante]

mm19pl

[m2-assistant[bot]]

Hi @krisdante. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if you want to validate it one more time, please, go though the following instruction:

• [ ] 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 2. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!

• [ ] 3. If the issue is not relevant or is not reproducible any more, feel free to close it.

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[krisdante]

This is now (2.3.2) duplicate of #24484 as the reason of 404 is different.
There is no pub directory in the URL anymore as of 2.3.2.

See the sitemap below:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
<url>
<loc>http://creativeshop.me/foo.html</loc>
<lastmod>2019-09-15T11:39:26+00:00</lastmod>
<changefreq>daily</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>
http://creativeshop.me/privacy-policy-cookie-restriction-mode
</loc>
<lastmod>2019-09-15T11:32:03+00:00</lastmod>
<changefreq>daily</changefreq>
<priority>0.2</priority>
</url>
<url>
<loc>http://creativeshop.me/bar.html</loc>
<lastmod>2019-09-15T11:45:00+00:00</lastmod>
<changefreq>daily</changefreq>
<priority>1.0</priority>
<image:image>
<image:loc>
http://creativeshop.me/media/catalog/product/cache/5ac664ada4c750d9cbcf6e03d9aa8773/a/n/animal-racoon.jpg
</image:loc>
<image:title>Bar</image:title>
</image:image>
<PageMap xmlns="http://www.google.com/schemas/sitemap-pagemap/1.0">
<DataObject type="thumbnail">
<Attribute name="name" value="Bar"/>
<Attribute name="src" value="http://creativeshop.me/media/catalog/product/cache/5ac664ada4c750d9cbcf6e03d9aa8773/a/n/animal-racoon.jpg"/>
</DataObject>
</PageMap>
</url>
</urlset>

The 404 problem is fixed by #24605

[magento-engcom-team]

Hi @RG-1. Thank you for your report.
The issue has been fixed in magento/magento2#24605 by @krisdante in 2.3-develop branch
Related commit(s):

• 6ab019b21eb90d9ebe119d8777a111a4b88bfad8
• b7059a615b59b673690f0f3d8187f684213fc72d
• 861e9f9401dd7845a77e63fb6cddd5359064c027

The fix will be available with the upcoming 2.3.4 release.

[mertgokceimam]

@magento-engcom-team You also need to release a patch for Magento 2.2.9 EE as this effects us enterprise customers also...