Docs-website: Sitemap.xml is not populating a site map
Description
GTS uses a search tool to allow our support engineers to search multiple content sources at once. The tool was able to index Product Docs using the sitemap here: https://docs.newrelic.com/sitemap.xml/ but that appears to be gone now.
We likely need to update https://www.gatsbyjs.com/plugins/gatsby-plugin-sitemap/ . Complicating things, it seems like there are some issues with Gatsby sitemaps in general: https://github.com/gatsbyjs/gatsby/issues/31095
Expected
It seems like our current config for the sitemap is broken and doesn't generate the sitemap at https://docs.newrelic.com/sitemap/sitemap-index.xml
Eng work
- add gatsby-plugin-sitemap gatsby-plugin-sitemap
outputoption to themegatsby-config.jsto explicitly set file path. - ensure the
<link rel="sitemap" />in<head>has the correcthreffor the path we set. - ensure / update swiftype UI and google search console have correct sitemap path
- bump all sites to use theme version this is deployed in
bradleycamacho
All 13 comments
Seems like a potentially significant issue for two reasons:
- Potentially blocking another team
- sitemap is a key part of how search engines crawl sites and prioritize content in search, so this may be impacting SEO
austin-schaefer
on 4 May 2021
@austin-schaefer would you say this a P1? There is no work around I can thinking of..
jpvajda
on 4 May 2021
@bradleycamacho Reading through the issue it looks like they changed the default path that they create the sitemap at, I was able to find it here:
https://docs.newrelic.com/sitemap/sitemap-0.xml
The site map index doesn't include the sub-folder that the sitemap files are placed in currently, but there is a PR open to fix that issue:
fix(gatsby-plugin-sitemap): Sitemap path bug #31184
If you are running into issues affected by the above, there seems to be a workaround by setting the output config to /./ referenced here
rudouglas
on 4 May 2021
Oh weird. I tried that path yesterday but didn't see it. Must've typo'ed it. Nice catch, @rudouglas
roadlittledawn
on 4 May 2021
for time being, i updated swiftype UI to use that sitemap path
roadlittledawn
on 4 May 2021
Oh wow, thanks @rudouglas! I'll share that with our stakeholders.
I will keep this issue open as I believe we still need to fix a few of those remaining issues @rudouglas linked to.
bradleycamacho
on 4 May 2021
Ye @roadlittledawn it seems to throw a 404 when you have a trailing slash 馃し
rudouglas
on 5 May 2021
Testing this on Developer site for build times 馃槢:
It looks like setting output: '/' in the gatsby-config results in the index href being //sitemap-index.xml which does not redirect correctly to the sitemap-0.xml. In addition the actual sitemap is still at
localhost:9005/sitemap/sitemap-0.xml
But if we set something specific like output: 'public' it will result in href="/public/sitemap-index.xml", which will redirect correctly to the sitemap-index page, and the sitemap will then be at localhost:9005/sitemap-0.xml which is whats expected i think?
Testing this on the docs site to confirm
rudouglas
on 5 May 2021
@rudouglas @roadlittledawn let me know if we want to ask Gatsby for any insights to this issue. We can drop this in the shared slack room if you need pointers.
jpvajda
on 5 May 2021
Ye @jpvajda that might be the best way to go, it works differently on the docs site than it does on the developer site, which may also be down to versioning, but might save time to ask them to weigh in?
rudouglas
on 5 May 2021
@rudouglas I updated the theme's demo site config to explicitly set output: '/'. i ran yarn workspace demo clean to remove previous build files (i assume that's maybe why you saw the sitemap/sitemap-0.xml as well when you built developer site). it looks like it outputs two files:
sitemap-0.xmlwhich has the expected list of pagessitemap-index.xmlwhich looks like an index pointing to available sitemap files. I assume search engines would know to follow that (we can test that when in production to see if google sees it)
It also looked like the <link rel="sitemap" /> in the <head> still pointed to //sitemap-index.xml.
I updated the package to 4.1.0-next.1 and it fixed that double slash. 馃し Created this PR if you want to take a look.
roadlittledawn
on 6 May 2021
@roadlittledawn it sounds like we may be good on this issue then? 馃グ
jpvajda
on 6 May 2021
@roadlittledawn ok ye sitemap-index.xml is usually used when you have multiple sitemaps, so there's one place for search engines to hit, I ran this locally as well in the theme demo and it does get rid of the double slash 馃槃 not sure why it didn't work in the developer site 馃槥
rudouglas
on 6 May 2021
Related issues
jerelmiller
路
5Comments
wrekone
路
3Comments
jpvajda
路
8Comments
jpvajda
路
7Comments
jpvajda
路
3Comments
Most helpful comment
@rudouglas I updated the theme's demo site config to explicitly set
output: '/'. i ranyarn workspace demo cleanto remove previous build files (i assume that's maybe why you saw thesitemap/sitemap-0.xmlas well when you built developer site). it looks like it outputs two files:sitemap-0.xmlwhich has the expected list of pagessitemap-index.xmlwhich looks like an index pointing to available sitemap files. I assume search engines would know to follow that (we can test that when in production to see if google sees it)It also looked like the
<link rel="sitemap" />in the<head>still pointed to//sitemap-index.xml.I updated the package to
4.1.0-next.1and it fixed that double slash. 馃し Created this PR if you want to take a look.