Gatsby: [gatsby-remark-copy-linked-files] Random errors copying files
Description
When I run gatsby develop or gatsby build I'm detecting randomly thrown errors regarding operation not permitted, followed by no such file or directory in the console. By randomly thrown I mean that with each develop or build command, different files are affected!
As you can see in my gatsby-config.js below, I'm using the plugin gatsby-remark-copy-linked-files to copy over every single file being referenced in my markdown files. As described in the plugin's readme, I set ignoreFileExtensions to an empty array in order to copy images as well.
Due to my further investigations I also found out that one of my GraphQL queries seems to have an effect on that error: I'm using the following query to query the markdown information for a single project:
export const query = graphql`
query ProjectQuery($slug: String!) {
markdownRemark(fields: { slug: { eq: $slug } }) {
htmlAst
fields {
slug
}
frontmatter {
title
date
titleImagePath {
publicURL
}
}
excerpt
}
}
`;
_When I remove excerpt, everything works fine!_
Expected result
I would expect that every single file is present in my public directory.
Actual result
With each develop or build command, different files are missing in the public directory.
Here's an example output of my console (like I said, there are also cases where image-1 is copied without an error):
[...]
success extract queries from components — 0.103 s
⢀ run graphql querieserror copying file { Error: EPERM: operation not permitted, open 'C:\development\web\gatsby\playground\public\image_1-aa655cbbcd4de392c2bc22d7bb49208c.png'
errno: -4048,
code: 'EPERM',
syscall: 'open',
path: 'C:\\development\\web\\gatsby\\playground\\public\\image_1-aa655cbbcd4de392c2bc22d7bb49208c.png' }
[...]
error copying file { Error: ENOENT: no such file or directory, chmod 'C:\development\web\gatsby\playground\public\image_1-aa655cbbcd4de392c2bc22d7bb49208c.png'
errno: -4058,
code: 'ENOENT',
syscall: 'chmod',
path: 'C:\\development\\web\\gatsby\\playground\\public\\image_1-aa655cbbcd4de392c2bc22d7bb49208c.png' }
[...]
// followed by multiple similar errors for different files (images or videos)
Environment
- Gatsby version (
npm list gatsby): [email protected] - gatsby-cli version (
gatsby --version): 1.1.48 - Node.js version: v9.9.0
- Operating System: Windows 10
File contents (if changed):
gatsby-config.js:
module.exports = {
plugins: [
`gatsby-plugin-sass`,
{
resolve: `gatsby-plugin-typography`,
options: {
pathToConfigModule: `src/utils/typography.js`
}
},
{
resolve: `gatsby-source-filesystem`,
options: {
name: `content`,
path: `${__dirname}/content/`,
}
},
{
resolve: `gatsby-transformer-remark`,
options: {
plugins: [
`gatsby-remark-emoji`,
{
resolve: `gatsby-remark-copy-linked-files`,
options: {
ignoreFileExtensions: []
}
},
{
resolve: `gatsby-remark-prismjs`,
options: {
classPrefix: "language-",
}
}
]
}
}
]
};
package.json:
{
"name": "GatsbyTest",
"description": "Test with Gatsby",
"license": "MIT",
"scripts": {
"develop": "gatsby develop",
"build": "gatsby build",
"serve": "gatsby serve"
},
"dependencies": {
"gatsby": "^1.9.241",
"gatsby-link": "^1.6.39",
"gatsby-plugin-sass": "^1.0.24",
"gatsby-plugin-typography": "^1.7.18",
"gatsby-remark-copy-linked-files": "^1.5.30",
"gatsby-remark-emoji": "0.0.1",
"gatsby-remark-images": "^1.5.59",
"gatsby-remark-prismjs": "^1.2.21",
"gatsby-source-filesystem": "^1.5.27",
"gatsby-transformer-remark": "^1.7.37",
"moment": "^2.21.0",
"react-helmet": "^5.2.0",
"react-icons": "^2.2.7",
"react-swipe-events": "^1.0.3",
"react-waypoint": "^8.0.1",
"rehype-react": "^3.0.2",
"typography-theme-lincoln": "^0.15.11",
"typography-theme-stern-grove": "^0.15.10"
},
"devDependencies": {}
}
gatsby-node.js:
const path = require(`path`);
const { createFilePath } = require(`gatsby-source-filesystem`);
const _ = require("lodash");
exports.onCreateNode = ({ node, getNode, boundActionCreators }) => {
const { createNodeField } = boundActionCreators;
let slug;
if (node.internal.type === `MarkdownRemark` && node.frontmatter.layout !== `timeline`) {
if (node.frontmatter.slug) {
slug = `/${node.frontmatter.slug}/`;
} else if (node.frontmatter.title) {
slug = `/${_.kebabCase(node.frontmatter.title)}/`;
} else {
slug = createFilePath({ node, getNode, basePath: `pages` });
}
createNodeField({
node,
name: `slug`,
value: slug,
});
}
};
exports.createPages = ({ graphql, boundActionCreators }) => {
const { createPage } = boundActionCreators;
return new Promise((resolve, reject) => {
graphql(`
{
allMarkdownRemark {
edges {
node {
frontmatter {
layout
}
fields {
slug
}
}
}
}
}
`
).then(result => {
result.data.allMarkdownRemark.edges.forEach(({ node }) => {
if (node.frontmatter.layout === 'post') {
createPage({
path: `blog${node.fields.slug}`,
component: path.resolve(`./src/templates/blog-post.jsx`),
context: {
slug: node.fields.slug,
}
})
} else if (node.frontmatter.layout === 'project') {
createPage({
path: `projects${node.fields.slug}`,
component: path.resolve(`./src/templates/project.jsx`),
context: {
slug: node.fields.slug,
}
})
}
});
resolve()
})
})
};
gatsby-browser.js: not changed / not present
gatsby-ssr.js: not changed / not present
pschild
All 11 comments
Thanks for the detailed writeup. It looks like Windows is complaining about file permissions for some reason. Can you try out gatsby-starter-blog and see if you get the same errors?
m-allanson
on 26 Mar 2018
Thank you for your quick reply!
Indeed I could reproduce the error using gatsby-starter-blog. I did the following steps:
git clone https://github.com/gatsbyjs/gatsby-starter-blog.gitnpm install- Added the video file
test.mp4in the foldersrc/pages/my-second-post - Added HTML code for the video in
src/pages/my-second-post/index.md:
html <video controls> <source src="./test.mp4" type="video/mp4"> </video> gatsby develop=> everything is working- Removing
.cacheandpublicfolder manually for resetting - Added
excerptto the GraphQL query insrc/templates/blog-post.js gatsby develop=>
[...] success extract queries from components — 0.100 s Generating image thumbnails [==========================----] 6/7 0.3 secs 86%error copying file { Error: EPERM: operation not permitted, open 'C:\development\web\gatsby-playground\gatsby-starter-blog\public\test-ba3bf8b32d7d7b182699db99dcafcce6.mp4' errno: -4048, code: 'EPERM', syscall: 'open', path: 'C:\\development\\web\\gatsby-playground\\gatsby-starter-blog\\public\\test-ba3bf8b32d7d7b182699db99dcafcce6.mp4' } error copying file { Error: EPERM: operation not permitted, unlink 'C:\development\web\gatsby-playground\gatsby-starter-blog\public\test-ba3bf8b32d7d7b182699db99dcafcce6.mp4' errno: -4048, code: 'EPERM', syscall: 'unlink', path: 'C:\\development\\web\\gatsby-playground\\gatsby-starter-blog\\public\\test-ba3bf8b32d7d7b182699db99dcafcce6.mp4' } Generating image thumbnails [==============================] 7/7 0.4 secs 100% â run graphql querieserror copying file { Error: ENOENT: no such file or directory, chmod 'C:\development\web\gatsby-playground\gatsby-starter-blog\public\test-ba3bf8b32d7d7b182699db99dcafcce6.mp4' errno: -4058, code: 'ENOENT', syscall: 'chmod', path: 'C:\\development\\web\\gatsby-playground\\gatsby-starter-blog\\public\\test-ba3bf8b32d7d7b182699db99dcafcce6.mp4' } [...]
I also tried to run gatsby develop from an admin console, but that resulted in the same error...
Any ideas what's going on there?
pschild
on 26 Mar 2018
Same errors here randomly appearing with gatsby-source-filesystem and gatsby-transformer-sharp. It doesn't appear in production using Netlify, so it may be a Windows only problem. May it be an antivirus thing ?
sebastienfi
on 26 Mar 2018
I just tested it on a Linux system and it worked.
So yeah, it really seems to be a Windows related error...
Do you have any ideas on how to make it work on Windows? And I'm still wondering why the error only appears if I add excerpt to my GraphQL query 🤔
pschild
on 27 Mar 2018
If this happens only on windows, I will see if I can reproduce it.
pieh
on 27 Mar 2018
Ok, so I did some digging and issue is that gatsby-transformer-remark is generating AST multiple times for same node in parallel which trigger running remark plugins multiple times (which then triggers copying same file multiple times and it seems like windows fs implementation is sometimes wonky with multiple concurent handles to same file). I will have PR up soon, just need to add comment and prettify my fix.
pieh
on 27 Mar 2018
Cool! Thank you very much for your efforts!
pschild
on 27 Mar 2018
@pschild my fix was merged - can you try updating gatsby-transformer-remark to 1.7.39 and let me know if that fixed things for you?
pieh
on 28 Mar 2018
@pieh I just updated and gave it a try. It's working! Nice job, thank you! 👌
pschild
on 28 Mar 2018
I get these errors when I specify destinationDir (I'm running Linux).
In gatsby-config.js:
resolve: 'gatsby-remark-copy-linked-files',
options: {
destinationDir: 'remark-linked-files',
ignoreFileExtensions: []
}
If I remove the destinationDir row I don't see these errors.
Before each build I delete the .cache and public folders. I see this error both on gatsby serve and gatsby build.
In package.json:
"gatsby-source-filesystem": "2.0.12",
"gatsby-remark-copy-linked-files": "2.0.8",
magnusarinell
on 20 Dec 2018
@pieh it appears I'm running into a similar issue with gatsby-transformer-sharp (see https://github.com/gatsbyjs/gatsby/issues/20602)
you might want to weigh in on the approach to resolve the issue for that plugin ? thanks!
denisbabineau
on 14 Jan 2020
Related issues
theduke
·
3Comments
mikestopcontinues
·
3Comments
dustinhorton
·
3Comments
brandonmp
·
3Comments
hobochild
·
3Comments
Most helpful comment
Ok, so I did some digging and issue is that
gatsby-transformer-remarkis generating AST multiple times for same node in parallel which trigger running remark plugins multiple times (which then triggers copying same file multiple times and it seems like windows fs implementation is sometimes wonky with multiple concurent handles to same file). I will have PR up soon, just need to add comment and prettify my fix.