Parcel: How do I include robots.txt / humans.txt in the build?

[samuelgozi]

❔ Question

How do I include robots.txt/humans.txt in the build?
I assume that it's not necessary to explain why this is important.

💻 Code Sample

🌍 Your Environment

| Software | Version(s) |
| ---------------- | ---------- |
| Parcel | 1.9.1
| Node |10.11.0
| npm/Yarn |1.10.1
| Operating System | MacOS Mojave

[jesnil01]

Why so you need parcel for this? You can just copy over your static files into your dist folder if you need them there.

[samuelgozi]

Why so you need parcel for this? You can just copy over your static files into your dist folder if you need them there.

Because parcel is an automating tool, your argument can be applied to any parcel feature.
Also, I don't want to need to explain to people working on the project that in order for it to be ready for deployment they need to manually copy files to the dist folder after successfully "building" it.

[jesnil01]

Parcel is a bundler and you should not have to bundle these kind of files.
And you don't have to copy it manually, make a script that you run after your build has finished.

[samuelgozi]

Parcel is a bundler and you should not have to bundle these kind of files.
And you don't have to copy it manually, make a script that you run after your build has finished.

I strongly disagree, if it was only a bundler then it shouldn't process static files like images, fonts, json files etc. It is supposed to bundle code and static assets(as stated in the website), and I think that it is pretty clear that its goal is to replace Webpack, Browserfy and the rest.

I understand that there might be tasks that parcel should not be doing, but I don't think that copying robots.txt and human.txt one of those tasks, I don't see how this is different from webmanifest or favicons other than the fact that I don't know of a meta tag to include them in the HTML file..

[ManzDev]

Maybe make a parcel-plugin for this? I think that would be interesant.

Parcel could check robots.txt and delete comments, minify or check (lint) syntax for avoid crawler problems.

[samuelgozi]

Maybe make a parcel-plugin for this? I think that would be interesant.

Parcel could check robots.txt and delete comments, minify or check (lint) syntax for avoid crawler problems.

Im making one right now, but I might take it a step further and just create a plugin for copying files.
I know that a plugin already exists for that, but I would like to learn how to build a parcel plugin anyways.

[icopp]

Including static files in general is common enough that I feel like the Parcel team should really maintain an official plugin to do this in 2.0 (they may have already been planning that, I dunno).

A robots.txt-specific plugin with the above functionality would be neat, though.

[samuelgozi]

Including static files in general is common enough that I feel like the Parcel team should really maintain an official plugin to do this in 2.0 (they may have already been planning that, I dunno).

A robots.txt-specific plugin with the above functionality would be neat, though.

I submitted a feature request for that a while ago, I don't know if it's going anywhere, I hope it does.
I have a question about the robots.txt/humans.txt plugin, other than copying it, is there really a need to remove any comments from it? I mean, after all, only crawlers download it.

[ManzDev]

I have a question about the robots.txt/humans.txt plugin, other than copying it, is there really a need to remove any comments from it? I mean, after all, only crawlers download it.

Not necessarily.

Maybe, best feature would be to review robots.txt syntax. Sometimes, mistakes are often serious about SEO.

[llarquere]

waiting for a plugin ...
inserting ''
into the of index.html
does the job

[caffeinum]

+1

Also, a CNAME files for a github pages builds fits this issue

[iainjreid]

I'd also like to chime in and say that this would be a really welcomed feature, but regarding the CNAME support, there could be a bug.

I'm using Parcel to bundle a GitHub pages site at the moment, and whilst it does a fantastic job with the HTML, JS, CSS, etc, the CNAME file is copied incorrectly, as it adds a trailing . to the file name

$ parcel build index.html CNAME
$ ls -1 dist
app.2137714b.js
app.2137714b.js.map
app.868b5728.css
app.868b5728.css.map
CNAME.
favicon.219d5973.png
index.html

[lukepighetti]

I think Parcel should support a generic entrypoint file, where you simply list all of the files you'd like to be bundled.

this index.parcel file might look like this

index.html
robots.txt
CNAME

[kevincox]

Another reason to do this in the bundler is it makes the dev server return the files. Even if they don't need bundling it is nice to have everything there.

[filips123]

Is there any update? I have a similar problem because I want to include/copy some files that aren't included by entry HTML or any other file but still needed for website to function properly.

The only difference, which actually makes things more difficult, is that those files are JSON, so I also can't just include them as additional entrypoints because JSON files will be converted to JS files. Solution for this would be to allow plugins to "force" including some files, even if they aren't included by other files, and also filename-based plugins/transformers/packages that are able to overwrite generic extension-based ones.

[mischnic]

I also can't just include them as additional entrypoints because JSON files will be converted to JS files

I have an implementation for that already, I hope it will be fixed for Parcel 2 in the near future.

[alsoknownasdrew]

Any updates on this?

[iainjreid]

For anybody looking to generate a valid CNAME file, I published a little plugin that uses the homepage field in your project's package file to create the required record.

• Plugin: https://github.com/emphori/parcel-plugin-cname
• Usage: https://github.com/mstksg/corona-charts/blob/master/package.json (not my work)

@mischnic would you like me to port the feature over to Parcel? I could also add support for humans.txt generation from the contributors field in the project's package file too?

[joshbedo]

Anyone have a solution for exposing robots.txt and sitemap.xml? This is super important for SEO and I don't see an easy way to do this with Parcel, React, and Netlify.

EDIT: removed a plugin that destroys the /dist folder every rebuild and just threw the robots.txt file inside the /dist folder. Everything seems to be working :)

[caffeinum]

@joshbedo sounds like a semioptimal solution, what I did is added a build step which copies the file: cp robots.txt dist/robots.txt

[joshbedo]

Yeah build doesn't delete and rebuild all files each time only files that need rebuilt so it works fine. No need to copy it over each time when it never changes.