Next.js: Add an example how to add robots.txt or sitemap.xml

[sarukuku]

Edit by Maintainers

Root level files are supported in Next.js 9.1 and newer.

Create the robots.txt file in the public/ directory.

For example:

pages/index.js
public/robots.txt

Note the public folder is on the same level as the pages folder, not inside of it!

Read more in the docs: https://nextjs.org/docs/basic-features/static-file-serving

---

There’s a closed issue related to this without a clear answer: https://github.com/zeit/next.js/issues/226

It’s kind of closed in favor of the Programmatic API so I guess I should implement that with custom server?)

[PetrSnobelt]

you can write clear text using component like this:

import React from 'react'
export default class extends React.Component {
  static async getInitialProps ({ req, res }) {
    //res.writeHead(302, { Location: '/redirect' }) //sample how to response custom header
    res.write('clear text')
    res.end()
  }
}

or you can use static assets with custom routing

[spleshka]

Another way of doing this: put robots.txt into /static folder, then add this to your server config:

const options = {
  root: __dirname + '/static/',
  headers: {
    'Content-Type': 'text/plain;charset=UTF-8',
  }
};
server.get('/robots.txt', (req, res) => (
  res.status(200).sendFile('robots.txt', options)
));

[JonAbrams]

Thanks @spleshka, your solution worked for me. I had to make a small change for it to work on Heroku though, I had to use path.join(…)

e.g.

const path = require('path');
const options = {
  root: path.join(__dirname, '/static'),
  headers: {
    'Content-Type': 'text/plain;charset=UTF-8',
  }
};
server.get('/robots.txt', (req, res) => (
  res.status(200).sendFile('robots.txt', options)
));

[schoenwaldnils]

@PetrSnobelt your approach sadly doesn't work with the static export

[kachkaev]

Related PR: https://github.com/zeit/next.js/pull/4163

Example: with-sitemap-and-robots-express-server

[tomsoderlund]

Here’s how I did it, using next-routes:

routes.js:

const routes = require('next-routes')
const routesImplementation = routes()
routesImplementation.add('/sitemap.xml', 'sitemap.xml')

sitemap.xml.js:

import React from 'react'

const sitemapXml = `<?xml version="1.0" encoding="UTF-8"?>
  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url><loc>https://www.MYWEBSITE.COM/</loc><lastmod>2018-09-26</lastmod></url>
  </urlset>`

export default class Sitemap extends React.Component {

  static getInitialProps ({res}) {
    res.setHeader('Content-Type', 'text/xml')
    res.write(sitemapXml)
    res.end()
  }

}

[r0b-]

thanks @tomsoderlund for this 'serverless' solution!
BTW: For robots.txt you need to change the content-type to 'text/plain'...

[MohamedHajr]

@tomsoderlund I Implemented Tom's solution but I didn't need next/routes just adding the react component and serving the file throw it but it only works in my development env, but not in production and by production I mean now serverless deployment

[KingZee]

@tomsoderlund That's hilarious and amazing! By far the best solution for serving static files anywhere instead of a common folder.

[Timer]

Next.js now has a public/ folder that mounts files at the root automatically! Create public/robots.txt and you're done!

[chibicode]

Note: public/ folder is experimentalized:

https://github.com/zeit/next.js/pull/7771
https://github.com/zeit/next.js/pull/7810

[joshiste]

how about a custom server.js with

    server.use('/', express.static('public'));

[gemouhi]

I'm using this solution for sitemap.xml (its working) I have example with appolo
[https://gist.github.com/jfnadev/d879aa022d4d13a6d060c422a04ede54]
But my server.js doesn't work
any examples with now.json & server.js(express) ??

[awitherow]

Is there a best practice on this? I am looking at building up my site SEO, and having these two files would be nice :)

[RayLuxembourg]

how about a custom server.js with

    server.use('/', express.static('public'));

Thanks!

[awitherow]

I have done the following after some painful researching...

pages/sitemap.xml.js

import React, { Component } from "react";
import sanity from "../lib/sanity";

import { links } from "../components/Navigation";

export default class Sitemap extends Component {
  static async getInitialProps({ res }) {
    let xml = '<?xml version="1.0" encoding="UTF-8"?>';
    xml += '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">';

    const SITE_ROOT = "YOUR_ROOT_HERE";

    const routes = [
      {
        url: "",
        changefreq: "daily",
        priority: 1,
      },
    ];

    links.map(({ id }) =>
      routes.push({
        url: `${id}`,
        changefreq: "daily",
        priority: 1,
      }),
    );

    const Posts = getPosts()

    for (let i = 0; i < Posts.length; i += 1) {
      const item = Posts[i];
      routes.push({
        url: `posts/${item.slug.current}`,
        changefreq: "daily",
        priority: 1,
      });
    }

    routes.map(({ url, changefreq, priority }) => {
      xml += "<url>";
      xml += `<loc>${SITE_ROOT}/${url}</loc>`;
      xml += `<changefreq>${changefreq}</changefreq>`;
      xml += `<priority>${priority}</priority>`;
      xml += "</url>";
    });

    xml += "</urlset>";

    res.setHeader("Content-Type", "text/xml");
    res.write(xml);
    res.end();
  }
}

robots.txt.js

import React, { Component } from "react";

export default class Robots extends Component {
  static getInitialProps({ res }) {
    res.setHeader("Content-Type", "text/plain");
    res.write(`User-agent: *
    Disallow:
    Sitemap: YOUR_ROOT_HERE/sitemap.xml`);
    res.end();
  }
}

[mddanishyusuf]

I'm using zeit to host nextjs application. So, it's super easy for.

Just put into pages/ and route in now.json like

{ "src": "/robots.txt", "dest": "/robots.txt" }

[skdigital]

@mddanishyusuf dam that looks simple, so litterally put the robots.txt file straight into the pages folder and then add add a now.json file to the project root and add the above code? Could you post a few screenshots?

Will it work or sitemap, how would that be implemented exactly the same?

[mddanishyusuf]

@skdigital this is working for me. And I also observe that also don't need now.json

https://tweetjobs.dev/robots.txt

note: I'm using zeit hosting.

[timneutkens]

I should note that we're going to add a public directory very soon (in 9.1).

[skdigital]

@mddanishyusuf

yes same here, i am using zeit now hosting....

Its not working for me. I put the robots.txt file directly into the pages folder and then pushed to github. My site auto deployes to zeit now and it deployed without failures. I tried, www.mysite.com/robots.txt and it did not work...

I am lost with the robots.txt. I tried just putting in static and also tried putting in pages, nothing is picking it up.

[skdigital]

@timneutkens would that mean we just put sitemaps and robots.txt files directly in the public folder and things should work out the box??

Is there an ETA for 9.1?

Thanks, Tim...

[mddanishyusuf]

@skdigital don't push to github.

try now cli to push the changes and then check.

[rsilvamanayalle]

If you're using Now 2.0

1. Add robots.txt to static folder.
2. Add follow code innow.jsoninside router array property:

{
      "src": "/robots.txt",
      "dest": "/static/robots.txt"
}

And finally, run with now

[realfresh]

Alternative when using raw HTTP server.

const { createServer } = require("http");
const { parse } = require("url");
const next = require("next");

const dev = process.env.NODE_ENV !== "production";
const app = next({ dev });
const handle = app.getRequestHandler();

app.prepare().then((r) => {
  createServer((req, res) => {
    const parsedUrl = parse(req.url, true);
    const { pathname, query } = parsedUrl;
    if (pathname === "/robots.txt") {
      res.statusCode = 200;
      res.write("User-agent: *\nDisallow: /");
      res.end();
    }
    else {
      handle(req, res, parsedUrl);
    }
  }).listen(process.env.PORT || 3000, (err) => {
    if (err) {
      throw err;
    }
    console.log("> Ready");
  });
});

[reginpv]

We can now create /public dir and files will be rendered as if in root

source: https://nextjs.org/blog/next-9-1

[brandonaaskov]

This may have been closed ages ago, but if you're using Next 9.1 or greater, you should use the public folder as @reginpv links to ☝️(works great)

[ivan-kleshnin]

public seems to be completely broken in latest NextJS & Now. Can't remember having so many bugs in years 😞 Files are (seemingly) random refuse to be served (404 errors).

Switching back to static for now. I can easily create a repo proof of the bug if anybody cares.

[ijjk]

Hi @ivan-kleshnin it sounds like you are using @now/next@canary which is experimental. Can you confirm this? Providing a link to a repo with a reproduction would definitely help 🙏

[ivan-kleshnin]

@ijjk I switched to canary because of bugs in previous versions.
Anyway, I narrowed done one of my issues here https://github.com/zeit/next.js/issues/10256
Thanks for your attention.

[ijjk]

@ivan-kleshnin What are the bugs you are running into on the stable channel of @now/next?

The above issue looks related to now dev and not Next.js. We are currently looking into optimizing the now dev experience to work more seamlessly with Next.js. In the meantime if you aren't using the api folder to write endpoints in languages other than JavaScript you can only use next dev which should work as expected

[ivan-kleshnin]

I reopened the issue at now repo, thank you. It belongs there.

[lupelius]

const routes = require('next-routes')
const routesImplementation = routes()
routesImplementation.add('/sitemap.xml', 'sitemap.xml')

res.write is not a function, when i check typeof to make sure it is a function, I get Error: "Rss.getInitialProps()" should resolve to an object. But found "undefined" instead.

res.write works on dev run but can't export it onto live env onto netlify.