Netlify-cms: Support local development

[erquhart]

The problem

When a site is being developed locally, Netlify CMS still makes all changes to the remote repository. This creates a few problems:

• you can't use Netlify CMS locally until your site is pushed to a remote
• if you do have a remote, all cms changes are occurring there, which causes the local and remote repo to go out of sync, often unexpectedly
• you have to make decisions about your site, like where to put the source, when you may not be ready to do so

@mjackson summed it up pretty well:

Proposed solution

From a user/developer standpoint, Netlify CMS running a local environment should work like so:

1. Run my site locally
2. Navigate to the CMS on my locally served site
3. The CMS detects that it is running locally
4. The CMS automatically activates local development mode (breaking change)
5. Any work in the CMS affects local files

For most use cases, this should "just work", out of the box.

Technical details

Detect local environment

The CMS can be loaded in many ways, including from a CDN via script tag, even during local development. For this reason, the only generally applicable approach to local env detection is by url. We can provide obvious defaults like localhost and 127.0.0.1 and allow a custom url to be configured.

Working against local files

Netlify CMS always expects to run in a browser with no access to servers, local or otherwise. Backends bridge this gap. To make changes to data, the CMS as a browser app requires a running server somewhere to access the data to be changed, and a backend extension to interface with that server via API calls.

For local development to work a running server must provide a local URL that the CMS can make API calls against. This can happen in at least two ways:

• CLI package
  This is the "universal" approach that will work for any setup - a companion package distributed via npm and run via CLI
• Tool/starter specific
  This approach removes the concern of running a CLI tool from the developer, but requires integration of the local file system server with the developer's tooling. This can be done by providing middleware for specific stacks, (as @talves has done with create-react-app - link), or including middleware in Netlify CMS starters that run with the development server.

UI

This could change, but the UI will likely consist of a conspicuous switch in the top right corner, near the user avatar, perhaps showing green when live and red when local. To make local development mode more conspicuous, another indicator like a color bar and notice at screen top may also be added.

This switch will only be visible if local_backend is configured, which will happen by default only for users of the netlify-cms package.

It will probably happen often that a user hits the local development switch and the server isn't running. We should anticipate this by first ensuring the server is reachable when the switch is flipped, and if it isn't, switch back off and provide notification that the local development server isn't reachable.

Configuration

Sites will often have a live backend and a local backend that have nothing to do with one another, and as backends are allowed to have arbitrary configuration schemas, the local backend configuration should be a separate object with the same allowances. A sample configuration might look like this:

backend:
  name: github
  repo: my/repo

local_backend:
  name: file-system
  url: https://random.local.url:2345

Note that the url key would not be required by the cms, but could be required by the backend in use.

Default behavior

Default behavior will be provided through the netlify-cms package. If no local backend is configured, the netlify-cms package should register and configure the file system backend.

Local development should be switched on automatically when a local environment is detected, but this would be a breaking change, so if this feature is included in a minor release, the switch will remain in "live" mode by default, and local-by-default can be added in the next major release.

Related issues

Resolves:

• #329
• #1085
• #2236

Somewhat related to:

• #1795

Coincides with:

• #2333

Paves the way for:

• #2251

[talves]

In #2333, I outline the first phase of the configuration defining a development backend and having it trigger a default backend when in development mode. I am not sure if this would meet the Default behavior section above, but we can discuss further if needed. Once a default development backend can be configured in the CMS, we can concentrate on how we solve the adding of a file-system cli to expose the fs-api without a manual configuration.

As mentioned above, if you have a Gatsby or CRA application, it is fairly easy to configure the fs backend if you need/want it during development. The readme's will get you started in those cases.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[designermonkey]

Hi guys, how's the progress with this issue?

[erquhart]

It isn’t in active development, still roadmap. But it’s almost hit the top of our most requested issues in a short period of time, so likely to be prioritized soon. Never stale, this one.

_{Sent with GitHawk}

[benoitb85]

Just switched from Wordpress to self-hosted Hugo on my home personal server. I'd like to use netlify CMS locally.

[AChris07]

Bumping this for good measure. I'm currently in the same boat, configuring Netlify CMS for my Gatsby site, and I'd like to have a valid option to test locally first, rather than having to test remotely.

[erezrokah]

Best way to move this forward is up vote the issue. This is currently number 3 on our list.

[iredmedia]

I'm excited for this, if I'm editing my admin (cms) config.yml to add new fields, i cannot test. Hopefully this upcoming change fixes this!

[erezrokah]

Update:
We have the docs up https://www.netlifycms.org/docs/beta-features/#working-with-a-local-git-repository if anyone would like to try it out and give us some feedback.

We're still thinking on how to due it more streamlined (probably SSG plugins to start the server automatically and have the CMS detect that it's running).

[ndalliard]

I use this feature with great success. Thanks!

One thing I run into today: When I tried to upload an image (size 442K) during the creation of a post, I got PayloadTooLargeError: request entity too large at readStream at the console where I run npx netlify-cms-proxy-server and Failed to persist entry: SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data on the Netlify CMS Frontend.

The error on the Frontend didn't make any sense. After some minutes I checked the console log. I guess the image is too large and also that this happens only locally. Not sure if this is intended behaviour.

[erezrokah]

Thanks @ndalliard a fix for that was merged today https://github.com/netlify/netlify-cms/pull/3229/files#diff-216dd90ce734cc01217c9959f0b240d2R14 and will be available in the next release

Apparently express's default for JSON body is 100kb

Update: Now released in version [email protected]

[smashercosmo]

It would be nice to have it also available as a middleware. So it could be easily attached to, for example, Gatsby dev server. Without starting a separate process.

[smashercosmo]

Also, when developing locally you don't usually want to make actual commits when editing content via cms. What you want is just writing changes to local files. What netlify-cms-backend-fs does.

[erezrokah]

Also, when developing locally you don't usually want to make actual commits when editing content via cms. What you want is just writing changes to local files. What netlify-cms-backend-fs does.

That's actually the new default with the latest version of the proxy server

[erezrokah]

It would be nice to have it also available as a middleware. So it could be easily attached to, for example, Gatsby dev server. Without starting a separate process.

Our next step is to integrate it into the different SSG plugins

[gileshinchcliff]

I'm new to react and front-end tech so bear with me, but I'm having some issues following your guidelines. My config looks like this:
```
backend:
name: git-gateway
repo: rebelastronaut/trinity

local_backend: true
media_folder: src/pages/images
public_folder: src/pages/images

and I run the proxy as stated and it's at the newest version ( 1.1.4) However when I try to publish I get this error coming out of the proxy server:

"encoding":"base64"}],"options":{"newEntry":false,"parsedData":{"title":"testing","description":"No Description!"},"commitMessage":"Update static_pages “about”","collectionName":"Static Pages","useWorkflow":false,"unpublished":false,"status":"draft"}}}: Cannot read property 'mkdir' of undefined
```

I've tried a few workarounds but it seems like I'm going round in circles. Any suggestions on how I can learn and develop locally.

[erezrokah]

@gileshinchcliff can you please open a new issue and share your config.yml and possibly the repo?

[gileshinchcliff]

@erezrokah sure.

[erquhart]

I think we can safely consider this issue satisfied, remaining points will be covered under #3285.

Docs for current support: https://www.netlifycms.org/docs/beta-features/#working-with-a-local-git-repository

[neokree]

Hi, if the local_backend: true is active in a live website this can cause problems?
There are any security issues in that case?

[erezrokah]

Hi @neokree, the CMS ignores that flag when not running on localhost:
https://github.com/netlify/netlify-cms/blob/72b26f4435b428f79805920566678627e60fffaa/packages/netlify-cms-core/src/actions/config.js#L178

So it should not cause any issues.

[idrisadetunmbi]

Hi, I may be wrong, but this does not seem to work with the editorial_workflow

[erezrokah]

@idrisadetunmbi, you're correct, by default the local mode works directly on the file system so it has not way of tracking unpublished entries branches, status, etc.
You can set it up to use a local git client using MODE=git npx netlify-cms-proxy-server which will give you editorial workflow support, but requires not having any pending changes in your repo.

[franva]

hi @erezrokah I hit the same need to run admin locally and need to test the Editorial Workflow, which I followed here.
The reason why I need to try Editorial workflow is: the Editorial Workflow doesn't work on my live website and the Error message is very generic and not helpful.

[alexboots]

I appears using MODE=git does make the editorial workflow show up but, it doesn't work correctly - it breaks if you try and save anything. Not sure why that is the case.

[erezrokah]

Hi @alexboots can you please open a new issue for that and add supply the information in the issue template?