Gatsby: Allow for gatsby-output plugins

[KnisterPeter]

Summary

Integrate an interface into the output chain of gatsby. This would allow to write plugins which could change the output processing of gatsby.

For example:

• Redirect the output to another folder
• Write the output to a remote location like s3 or ssh or ...

Motivation

We are trying to run our gatsby process in an AWS lambda function. Lambdas by default have a read-only filesystem which means gatsby fails to write the site to disk.
Right now we are starting the gatsby process from /tmp which is writable, but thats a workaround.
As the page grows it might overflow the FS available on lambda.

[Ronva]

Could you provide an outline for how you would potentially go about integrating this? It would be great for getting on the same page, even if it's just a bullet point list of your thought process.

[KnisterPeter]

@Ronva @lannonbr
I've not looked into the details of gatsby right now. I guess its a lot more work to do than to define interfaces and we are good to go. The examples above would require some sort of filesystem layer which could be replaced. The from my point of view the outine would similar to:

• Identify ideas and usecases solvable by these output plugins
• Look through sources where these plugins would need to be integrated (where are file accesses to '.cache' or 'public')
• Define common interface for plugins
• Move file system output (.cache and public folders) into a default output plugin

Is this what you need to hear from me, or did I misunderstood you?

[wardpeet]

Moving .cache or public directories isn't something we want to take on at this point. We might want to fix this going forward with gatsby v3. It's not as easy as you describe. We also run webpack & babel which can save files to disk if they want. We won't have control over that so we can't ever guarantee that files are written to the correct locations.

This is what I've done so far on google cloud functions (which also only enables /tmp as a write partition). I copy the whole directory inside /tmp/gatsby. I chdir inside /tmp/gatsby so it's my working directory. From there on I run gatsby build. When that's done I publish the public folder in a gcloud bucket (storage).

If something else needs to happen another gcloud function just picks it up from the gcloud bucket. This way I can have 1 function that does gatsby builds and other functions that do other things, so I keep my functions lean and mean.

[KnisterPeter]

@wardpeet We do exactly what you described above as well but on AWS. But it feels like kind of a hack and having so much copy operations instead of writing directly to the destination is debug intense and error prone. If its setup correctly I think it will work but ... :wink:

Anyway, feel free to keep this as feature request for v3 or close it as you like.

[wardpeet]

More and more people might need this so we might consider this for v3. I'll discuss with the team.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[wardpeet]

It seems like we also have the discussion at https://github.com/gatsbyjs/gatsby/issues/1878 so I'll be closing this one as a duplication.