Next.js: Add a Dynamic Sitemap example that works with the new Dynamic Routes

[creativiii]

Example request

Up until now I've been using a custom Express server with next-routes to handle all my routing. I'm very excited to upgrade and finally move to the Dynamic Routing that was introduced with NextJS 9.

That said, a key part of my site is the Sitemap. At the moment it's very easy for me to have one thanks to my custom server.js and the sitemap package however I am completely lost on how to achieve this with dynamic routing.

I'm not sure if this would be considered out of scope for NextJS, but Sitemaps are essential for websites and its unfortunate that deciding to use a great feature like Dynamic Routing means that I'd have no obvious way of creating one.

An example showing how to use the existing Sitemap NPM Package would be fantastic, but any solution would work.

I've tried my best to get it to work but haven't gotten anywhere so far.

[danielr18]

@creativiii How are you currently using the sitemap package? Would it be possible to continue using it like you do but creating the sitemap files in the public folder with a script?

[creativiii]

@danielr18 I am not generating a static Sitemap on build, which is the suggestion I see a lot when googling, so I don't think this solution would work if that's what you're suggesting.

Here's an amended version of my server.js that's generating my sitemap.

const express = require( 'express' )
const axios = require( 'axios' )
const next    = require( 'next' )
const cacheableResponse = require('cacheable-response')
const sm = require('sitemap')

// Import middleware.
const routes = require( './routes' )

// Setup app.
const app     = next( { dev: 'production' !== process.env.NODE_ENV } )
const handle  = app.getRequestHandler()
const handler = routes.getRequestHandler( app )

const ssrCache = cacheableResponse({
  ttl: 1000 * 60 * 60, // 1hour
  get: async ({ req, res, pagePath, queryParams }) => ({
    data: await app.renderToHTML(req, res, pagePath, queryParams)
  }),
  send: ({ data, res }) => res.send(data)
})

const createSitemap = (res) => {
  let urlRoutes = ['posts', 'pages'];
  let baseUrl = 'https://example.com/'

  let sitemap = sm.createSitemap ({
    hostname: baseUrl,
    cacheTime: 1
  });

  sitemap.add({
    url: baseUrl,
    changefreq: 'daily',
    priority: 1.0
  })

};

app.prepare()
  .then( () => {

    // Create server.
    const server = express();

    server.get('/sitemap.xml', function(req, res) {
      res.header('Content-Type', 'application/xml');
      createSitemap(res);
    });

    // Use our handler for requests.
    server.use( handler );

    // Don't remove. Important for the server to work. Default route.
    server.get( '*', ( req, res ) => {
      ssrCache({ req, res })
    } );

    // Get current port.
    const port = process.env.PORT || 8080;

    // Error check.
    server.listen( port, err => {
      if ( err ) {
        throw err;
      }

      console.log( `> Ready on port ${port}...` );
    } );
  } );

This generates a sitemap with my base url, I then would loop through my desired routes to add other pages. These pages are dynamically generated from the information being pulled from Wordpress so they change fairly often.

[creativiii]

Also since I posted this a user on Spectrum shared their own solution.

I never got around to testing it out since I realised that I also need a way to use cacheable-response which would be impossible with the new dynamic routes. 🤷‍♂️

[dohomi]

I am making use of the new rewrites feature of NextJS v9.1.4

//next.config.js
 experimental: {
      modern: true,
      async rewrites () {
        return [
          {source: '/sitemap.xml', destination: '/api/sitemap'},
        ]
      },
      catchAllRouting: true
    },

Inside of pages/api/sitemap.ts I run the npm module sitemap and build a dynamic sitemap on the fly based on content from a headless API:

import { SitemapStream, streamToPromise } from 'sitemap'
import { IncomingMessage, ServerResponse } from 'http'

export default async function sitemapFunc(req: IncomingMessage, res: ServerResponse) {
  res.setHeader('Content-Type', 'text/xml')
  try {
    const stories =  await fetchContentFromAPI() // call the backend and fetch all stories

    const smStream = new SitemapStream({ hostname: 'https://' + req.headers.host })
    for (const story of stories) {
      smStream.write({
        url: story.full_slug,
        lastmod: story.published_at
      })
    }
    smStream.end()
    const sitemap = await streamToPromise(smStream)
      .then(sm => sm.toString())
    res.write(sitemap)
    res.end()
  } catch (e) {
    console.log(e)
    res.statusCode = 500
    res.end()
  }
}

[sj-log]

I am making use of the new rewrites feature of NextJS v9.1.4

//next.config.js
 experimental: {
      modern: true,
      async rewrites () {
        return [
          {source: '/sitemap.xml', destination: '/api/sitemap'},
        ]
      },
      catchAllRouting: true
    },

Inside of pages/api/sitemap.ts I run the npm module sitemap and build a dynamic sitemap on the fly based on content from a headless API:

import { SitemapStream, streamToPromise } from 'sitemap'
import { IncomingMessage, ServerResponse } from 'http'

export default async function sitemapFunc(req: IncomingMessage, res: ServerResponse) {
  res.setHeader('Content-Type', 'text/xml')
  try {
    const stories =  await fetchContentFromAPI() // call the backend and fetch all stories

    const smStream = new SitemapStream({ hostname: 'https://' + req.headers.host })
    for (const story of stories) {
      smStream.write({
        url: story.full_slug,
        lastmod: story.published_at
      })
    }
    smStream.end()
    const sitemap = await streamToPromise(smStream)
      .then(sm => sm.toString())
    res.write(sitemap)
    res.end()
  } catch (e) {
    console.log(e)
    res.statusCode = 500
    res.end()
  }
}

I wonder why my fetchContentFromAPI gets error Cannot find name 'fetchContentFromAPI'. . Do you know What should I do to find that function? Thank you in advance!

[dohomi]

@sj-log fetchContentFromAPI is a dummy function. You would need to write your own function to call your backend/service to fetch all links you would like to expose as a sitemap

[sj-log]

@dohomi Which function did you use for your case? Should I use 'getInitialProps()' at there?

[dohomi]

@sj-log I think you misunderstand: you can do whatever suits your usecase in this function. It is nothing specific to NextJS, it simply returns an array of items and I just gave it the name fetchContentFromAPI

[sj-log]

@dohomi Well Thank you for the answer, but I really have no idea how to fetch backend or the service as you mentioned.

[dohomi]

@sj-log what backend do you need to call? I am using a headless CMS (Storyblok). You either can use the native fetch or Axios, whatever suits your need

[MihaiWill]

@dohomi Perfect!!! Thanks a lot! What about:

Warning: You have enabled experimental feature(s).
Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use them at your own risk.

[dohomi]

@MihaiWill you will see this warning as soon you enable any of the experimental features of NextJS. As it states: the experimental features are not considered stable and used at your own risk. I am running them very stable so far :-)

[sj-log]

@dohomi Dear friend, Just I've found another sitemap-generator for nextjs.
so I solved this site mapping issue! because I couldn't use any of Story book or Axios. So sorry if I made you a bit frustrating! just I wanted to grab my goal instantly.

• If there is someone who is struggling to build a sitemap in nextjs, please refer to this one as well. I got this.

dohomi's way to implement is quite simple and good, However, If you still feel 'I don't know what to do' status, please check out the above link. The way I linked, is for a total beginner as easy as I can follow up. Good days.

[leerob]

I've spent some time looking into this. I didn't want to mess with a custom server, or utilize rewrites. Here's where I landed.

// pages/sitemap.xml.js
import React from 'react';

const EXTERNAL_DATA_URL = 'https://jsonplaceholder.typicode.com/posts';

const createSitemap = (posts) => `<?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
        ${posts
          .map(({ id }) => {
            return `
                    <url>
                        <loc>${`${EXTERNAL_DATA_URL}/${id}`}</loc>
                    </url>
                `;
          })
          .join('')}
    </urlset>
    `;

class Sitemap extends React.Component {
  static async getServerSideProps({ res }) {
    const request = await fetch(EXTERNAL_DATA_URL);
    const posts = await request.json();

    res.setHeader('Content-Type', 'text/xml');
    res.write(createSitemap(posts));
    res.end();
  }
}

export default Sitemap;

More info here.

[perryraskin]

@leerob took a look at your blog post, really helpful! do you know what I would do to make a dynamic sitemap if my app creates blog post pages on the fly via [slug].tsx? For example, right now https://raskin.me/blog/init(blog) goes to my blog post, but it uses the template [slug].tsx to generate it. Currently the sitemap grabbed the actual "[slug]" name.

[leerob]

@perryraskin I think you would need to slightly modify your script to look at your /posts folder via something like posts/*.md.

js const pages = await globby(['posts/*.md', 'pages/**/*.tsx', '!pages/_*{.jsx,.tsx}']);

That will fetch all your Markdown files. Then, you'd need to probably replace posts with blog in the URL you output to your sitemap.

Give that a shot!

[perryraskin]

@leerob nice, that worked great. the sitemap is still showing the [slug] though, should i remove it? i tried doing sitemap.replace but it does not seem to be having any effect

[leerob]

@perryraskin You'll probably want to exclude the blog folder since you're handling it via posts. This should get it working for ya!

const pages = await globby(['posts/*.md', '!pages/blog', 'pages/**/*.tsx', '!pages/_*{.jsx,.tsx}']);

Another idea to consider: utilize getStaticPaths and getStaticProps that will be released soon to generate a static-site (SSG) instead of server-side rendering (SSR) your blog posts. Here's an example.

Once SSG support lands, I'll probably update the blog post to include an example for that. This isn't directly related to your question about the sitemap, but I just figured I'd mention it 😄

https://github.com/zeit/next.js/discussions/10437#discussioncomment-333

[pspeter3]

getStaticPaths and getStaticProps just launched. Can you use them to generate non HTML content (eg RSS / Sitemap)?

[leerob]

@pspeter3 After reading the new docs, it seems like getStaticProps generates HTML/JSON. To create non-HTML content, I believe you'd need to use getServerSideProps.

export async function getServerSideProps({res}) {
    const request = await fetch(EXTERNAL_DATA_URL);
    const posts = await request.json();

    res.setHeader('Content-Type', 'text/xml');
    res.write(createSitemap(posts));
    res.end();
}

[pspeter3]

@leerob Thanks! I'm able to get that to run but I end up with admin/config.yml.html _(I'm_ trying to generate configuration for Netlify CMS)_. Do you how I can not generate the .html suffix?

[leerob]

@pspeter3 The way I've gotten around that myself is using replace for file extensions. I used this when mapping directly over the pages directory looking at MDX files. I believe a similar approach could work for you to filter out .html files after talking to Netlify CMS.

const sitemap = `
    <?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
        ${pages
            .map((page) => {
                const path = page
                    .replace('pages', '')
                    .replace('.js', '')
                    .replace('.mdx', '');
                const route = path === '/index' ? '' : path;

                return `
                    <url>
                        <loc>${`https://yoursitehere.com${route}`}</loc>
                    </url>
                `;
            })
            .join('')}
    </urlset>
`;

[pspeter3]

Got it. I was actually commenting on something different. I was trying to use Next to generate a static file that does not have a .html ending. I think that really I want the ability to generate static files using the Next infrastructure and be able to set the permalink. Maybe I should open a new issue?

[leerob]

@pspeter3 Ahh, sorry for the confusion there – my bad. I agree, I would open up a new issue and try and provide a solid example 👍

[pspeter3]

Added #11115 as the new issue

[BrunoBernardino]

Here I have a solution that allows you to have a dynamically-generated sitemap that covers dynamic routes and static ones, as an interim solution of sorts (I've already commented on the issue above, but for those who land here and aren't following that one):

It uses a lambda for the sitemap generation, routed via @now/node in vercel.json, with the static pages/routes built with a script before deploy.

Maybe it'll help you. It supports static routes, and allows for newly-dynamic pages to still be valid and added (like for products, in that example).

[t-lochhead]

Thank you @leerob for putting this together. The exact code wasn't working for me, but got it to work through some small edits (found here). Sharing in case anyone else ran into the same problem.

````
// pages/sitemap.xml.js

const EXTERNAL_DATA_URL = "https://jsonplaceholder.typicode.com/posts";

const createSitemap = (posts) => <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> ${posts .map(({ id }) => { return

${${EXTERNAL_DATA_URL}/${id}}

; }) .join("")} </urlset> ;

// remove component
export async function getServerSideProps({ res }) {
const request = await fetch(EXTERNAL_DATA_URL);
const posts = await request.json();

res.setHeader("Content-Type", "text/xml");
res.write(createSitemap(posts));
res.end();
}

// add component here
export default () => null;
````

[pspeter3]

This is a cool hack! Will it work for exporting too?

[t-lochhead]

@pspeter3 Not sure! First time I tried this.

[xaun]

@leerob what's the go with generating a sitemap like that for thousands, or even millions of posts? Would that be recommended or beneficial?

[leerob]

@xaun If there are millions of posts, there are likely _multiple_ sitemaps stitched together.

[BrunoBernardino]

@xaun have you tried https://github.com/vercel/next.js/issues/9051#issuecomment-661110802 ? It should work for your case as well.

[dbertella]

I have tried something very similar to what you are suggesting and succeded to get a sitemap working in my local machine for both dynamic and static routes but in prod it looks like static routes aren't generated in getServerSideProps. I'm using globby package to get the list of the pages but I guess it's not possible to use it in production?

This is my attempt if it can help someone else, https://github.com/dbertella/cascinarampina/commit/efdfea09a91888a59de67d340b0bac025c4d5a7c#diff-2e0db62b38c8e7637d13c40a1a3034cb516d8899478112624a04b6f1f5c0abde
I for tonight added the pages list as a static list for now.

I was thinking will this work in a function at build time and write a static xml file instead of having a ssr page?

[BrunoBernardino]

@dbertella that's what I suggest in https://github.com/vercel/next.js/issues/9051#issuecomment-661110802 if you want to try that out.

[dbertella]

Hey @BrunoBernardino yours is a good idea. I'm not super familiar with the whole node environment but if I understand correctly what you are doing the only difference with my script is that you generate the list of pages at build time and that's cool indeed. Regarding the lamda it seems you are using an older version of next, would it work the same if you put the function in the api folder? Also @now/node seems to be deprecated in case will it work the same with @vercel/node? The real question is if you move the lamda in the api folder would you still need to install the additional package?

I will try to generate the list of static pages at build time btw, see if it work fine with my current implementation and let you know in any case.

EDIT. That seems to works fine! Thanks for the tip. I'm running globby in a prebuild script and generate a page.json static file. Other than that the approach with a ssr page seems to work fine I made a PR with the needed changes here

[BrunoBernardino]

Awesome to hear it worked! I'm sure newer versions might need some tweaks, but should still work fine.

[dbertella]

I'm actually thinking at a different approach now and going static by default. The issue I have is that I'm trying to use es6 imports in a node.js script and it doesn't like it very much, but I'd probably go back to it when I have time to try to figure it out this is my next attempt https://github.com/dbertella/cascinarampina/pull/9/files