Dvc.org: dev mode improvements

[jorgeorpinel]

• [x] Easy: Can we name the yarn script just dev like before? faster to type

• [x] Reasonable: Can it have watch/autoreload? Or perhaps allow reloading the current page by clicking on the current nav item (used to work).

• [x] Changing sudebar.json requires stopping yarn develop and running it again (this was automatic before with Next.js)

• [ ] Maybe hard: Can dev mode somehow purge the Gatsby cache automatically?
  There are times when I make a change to the code or contents, I simply don't get the changes reflected. Even stopping yarn develop and running it isn't enough. I have to stop it, rm -rf .cache/, and run it again.
  E.g. this happens when changing redirects, even if the browser cache is disabled.

[iAdramelk]

CC @pavelgrinchenko

Some quick answers:

1. Can it have watch/autoreload? Or perhaps allow reloading the current page by clicking on the current nav item (used to work).
   E.g. changing sudebar.json requires stopping yarn develop and running it again (this was automatic before with Next.js)

It watch/autoreload for the most part, but right now sidebar.json is an exception to some degree. Changing titles and katacoda links should still autoreload, but changing slug part will not automatically create page on the new url because this change didn't start page rebuild process (it will probably update menu, but actual page will not be created).

We can either fix it by implementing this issue https://github.com/iterative/dvc.org/issues/1044 (renaming md file or creating new md file will automatically trigger this page build). Or we can look for a way to say to gatsby the it should rebuild all of the doc pages on the sidebar.json changes. Not sure that it possible, but there may be a hook for that.

1. Maybe hard: Can dev mode somehow purge the Gatsby cache automatically?
   There are times when I make a change to the code or contents, I simply don't get the changes reflected. Even stopping yarn develop and running it isn't enough. I have to stop it, rm -rf .cache/, and run it again.
   E.g. this happens when changing redirects, even if the browser cache is disabled.

This one is a little trickier. Always removing cache is not a good think because it will make running dev server significantly slower. But there some changes that we can make that will not automatically update .cache and will cause described issues.

It shouldn't be a problem with Gatsby's code because Gatsby track and update it, but nodejs code like prism plugins and Express middlewares can cause this issue: If we run gatsby's plugin and change some of their indirect dependencies like we edit highlight syntax in prismjs plugin or update redirects config without changing middleware itself for example.

We can try to fix it, but maybe it's ok to leave ti as is, because it only reveal itself in very specific situations and we didn't change redirects and prism plugins that often.

[jorgeorpinel]

OK, I'm OK with adapting to these situations if needed. Let's just try (not huge effort) to make some of the improvements in a small PR that closes this issue?

[pavelgrinchenko]

1. It can be done easily
2. Gatsby don't have any hooks for creating pages in the development mode. So we can't make Gatsby known that it should rebuild site pages.
3. I agree with Alexey that it's ok to leave it as is. Because after merging this PR into the master branch you will need rebuild all images cache every run and it's the long task. Concerning browser redirects caching - it doesn't depend on Gatsby at all. You can try to add random query string to the URL, to ignore cache. ?a, ?aa, etc.

[jorgeorpinel]

Concerning browser redirects caching - it doesn't depend on Gatsby at all

I wasn't talking about browser caching, that can be disabled from Dev Tools in Chrome for example. I was talking about the redirects-list.json file.

[jorgeorpinel]

Hey guys I ran into another problem but I don't want to open yet another issue so here it is:

I'm on master branch, install node deps and run the dev server. In another tab, I try to git checkout refactor/get-started-2 and get:

fatal: cannot create directory at 'content/docs/tutorials/get-started': Permission denied

The folder already exists BTW so I don't understand what Git is talking about... git status shows a bunch of Changes not staged for commit (interrupted checkout).

I have to go to the first tab, stop the dev server, and then I can checkout to my feature branch normally. Hot reload issue?

I'm still using Cmder (Git Bash) on Windows.

[pavelgrinchenko]

Seems file watchers on Windows somehow lock files

[jorgeorpinel]

Maybe hard: Can dev mode somehow purge the Gatsby cache automatically?
There are times when I make a change to the code or contents, I simply don't get the changes reflected. Even stopping yarn develop and running it isn't enough. I have to stop it, rm -rf .cache/, and run it again.
E.g. this happens when changing redirects, even if the browser cache is disabled.

I found another case where this happens: when you rename a md file in contents and update the sidebar.json accordingly. The sidebar reflects the change, but trying to load it yields a 404. I have to stop the dev server, run gatsby clean, and yarn develop again which takes several minutes to regenerate the thumbnails...

[shcheklein]

making this p0. dev experience is terrible with Gatsby so far. @iAdramelk @pavelgrinchenko @fabiosantoscode any ideas at least theoretically how to improve this?

[iAdramelk]

@shcheklein to make sure, are we talking about adding/removing pages and removing it?

If yes, then the problem as I wrote above it that now we get doc page urls from the sidebar.json at the build time, but then we change sidebar.json pages are not rebuild because they don't know they they depend on the sidebar.json.

Basically the part that causes the problem is this: https://github.com/iterative/dvc.org/blob/master/gatsby-node.js#L9-L19

It's only set once then we create node, and because md file didn't change, it didn't updates then sidebar.json is updated.

So there are two things that we can possible do here:

1. Check if there is a way to update all md nodes on sidebar.json changes. Probably we can use createContentDigest for this.
2. Decouple page slugs from the sidebar.json values. Basically we can still use json for the menu structure oand ordering, but actual page urls and the list of pages to generate should come from the md files themselves and their frontmatter (for example taking slug from the file path or frontmatter instead of sidebar.json would solve it automatically).

[shcheklein]

@iAdramelk yes, any structural changes, like @jorgeorpinel mentioned above. I think, I even can expect that it's hard to make it hot-reloadable, but this

gatsby clean, and yarn develop again which takes several minutes to regenerate the thumbnails...

is definitely too much.

Check if there is a way to update all md nodes on sidebar.json changes. Probably we can use createContentDigest for this.

sounds worth trying. I hope that it will update all of them but will detect that most of them haven't changed.

[jorgeorpinel]

UPDATE: The renaming md file issue is solved in #1145 BTW. And the thumbnail generation is much faster in that branch.

[rogermparent]

I can take this one on. I think the main problem is that we're requireing JSON files instead of going through a plugin like gatsby-source-json. I know it's kinda clunky to pull JSON through GraphQL instead of require, but Gatsby's hot reloading is a function of its GraphQL Node system so we've got to go through the proper Gatsby channels to get HMR.

My PR #1145 changes doc page behavior to get its slugs from filenames, and I can add frontmatter-based slug override functionality if we think it'll be needed.
I think this is a good base to start on when tackling the sidebar issue, as it means Gatsby doesn't rely on the JSON file for page generation and the now less-complex sidebar can be fed through gatsby-source-json more easily.

I'm torn between this and the deploy issue right now, I'm much more proficient in what it takes to fix the former but the latter is overall more important. We'll have to decide tomorrow, because it's already way too late for me here. Good night, everyone!

[shcheklein]

@jorgeorpinel PTAL again and feel free to reopen if still see some problems here.