Cli: Netlify Dev incorrectly rewrites requests for files that *don't* exist

[edaemon]

I think this is the right place to open this; https://github.com/netlify/netlify-dev-plugin/issues/155 and https://github.com/netlify/netlify-dev-plugin/issues/181 seem related but don't quite capture the issue.

There seems to be a difference in the way netlify dev processes redirects compared to the Netlify platform. (Note: I'm not certain both/either use node-redirects-parser, just assuming.) In a CRA app using react-router, you need a public/_redirects file like this to use any routes aside from the base path:

/*    /index.html    200

That works fine on Netlify itself; requests to custom routes are properly rewritten without interfering with requests for files that actually exist. However netlify dev seemingly applies this rule to all requests, similar to the way Netlify would if you appended an exclamation point to the rewrite (200!). This means requests for scripts like /static/js/bundle.js and other basic assets are rewritten to /index.html. The rewritten requests get a 200 response but obviously it serves the HTML, so JS processing will immediately throw errors like Uncaught SyntaxError: Unexpected token <.

This breaks Netlify Dev, which has otherwise been immensely useful; I'm searching for a workaround/fix.

[sw-yx]

i have run into this exact problem myself but never figured it out!!! cc @biilmann

yes i can confirm netlify dev uses node-redirects-parser. but the real underlying library is netlify-redirector which is the closed source WASM compilation we are using (and in theory should have absolutely zero difference between netlify dev and production netlify). however we might be introducing this error in node-redirects-parser itself so there are two possible sources of error.

i'd glance through the code in this repo to see if theres some easy fix we overlooked.

[sw-yx]

p.s. i have renamed this repo to netlify-redirect-parser to avoid the confusing name mismatch between github an npm

[apappu9]

hey, i think the issue is in @netlify/rules-proxy. This package seems to be checking if there's already a static file with the following function:

function notStatic(pathname) {
  const maybePublicPath = path.resolve(
    config.publicFolder,
    pathname.slice(1)
  );
  return !fs.existsSync(maybePublicPath);
}

I don't think this works (at least in my case) since webpack-dev-server is generating all the static assets in a virtual location, so they would never actually exist in the file system.

I'm not too experienced with how this works, so please correct me if i'm wrong :)

[edaemon]

Thanks for finding that @apappu9! I shorted the notStatic() function in rules-proxy like this:

  function notStatic(pathname) {
    const maybePublicPath = path.resolve(
      config.publicFolder,
      pathname.slice(1)
    );
    //return !fs.existsSync(maybePublicPath);
    return false;
  }

That yields some strange results. It works, since the requests for the files that only exist in memory are treated as if they exist on disk. The strange part is that requests that use react-router work even though they aren't being rewritten to index.html (at least no Rewrote URL to... message appears). It's a nice surprise but I'm wary of spooky fixes.

FYI, CRA has an open PR to allow use of webpack-dev-server's writeToDisk option: https://github.com/facebook/create-react-app/pull/6144 (which hopefully works with Netlify Dev)

I suppose this issue's title is a bit off, though -- Netlify Dev is rewriting the request because the files don't exist, at least not on disk, and I'm not sure if it's feasible to check for them any other way.

[sw-yx]

cool investigations! i still havent had time to figure this out tho but if anyone wants to take a crack at this you are very welcome.

[jgierer12]

@edaemon I think that's because webpack-dev-server itself handles the redirect in that case. Netlify Dev is just a proxy for the webpack-dev-server, so every request that it gets (and doesn't rewrite itself) it just passes on to WDS, which then handles the rewriting. So that part will only ever work in development where the dev-server is running, but not in production where Netlify provides its own server.

IMO the best way to use Netlify Dev, in order to make it as close as possible to the real Netlify environment, is to not use any dev servers at all. If possible, configure your tools to just watch the files and write the changes to disk, and let Netlify Dev handle the serving.

[sw-yx]

we think this issue probably lies in https://github.com/netlify/netlify-rules-proxy/blob/master/index.js#L170 instead of this repo/library

[edaemon]

I'm not able to access that repo, @sw-yx, but that file is the one I modified. This issue may belong there.

And you're right, @jgierer12, having the dev server involved here breaks prod parity a bit. Looking over the docs, though, it seems like Netlify Dev requires a dev server to be running; can it serve files standalone?

[sw-yx]

yes it can serve files standalone - maybe this helps? https://www.netlify.com/blog/2019/04/24/zero-config-yet-technology-agnostic-how-netlify-dev-detectors-work/

sorry if the readme doesnt explain it well, happy to take suggestions for improvement

[edaemon]

I'm probably missing something obvious here as I can't get it to serve standalone. Seemingly, if it detects create-react-app (or maybe any SSG/build tool) then netlify dev will wait for a dev server to respond. Is there an option in netlify.toml to adjust that?

[sw-yx]

yes that what it is designed to do.

sorry i think i dont really understand the reason you want to serve files standalone. you're using CRA but want it to serve standalone because of this /* /index.html 200 redirect bug? lets solve it there rather than hack around this. for the time being, we acknowledge /* /index.html 200 doesn't work well and will have to fix it. but is there anything else, anything worse, that you're facing?

[edaemon]

The redirect issue is the only one I'm facing.

The standalone serving isn't all that important. I was only pursuing it because this redirect issue is a result of the difference between Netlify Dev and the Netlify platform; I was hoping to minimize that difference.

[sw-yx]

me too. me too. leave it as a known bug for now - we will fix this when we can.

[jgierer12]

Correct me if I'm wrong but isn't this caused by the indifference between Netlify Dev and Prod? Prod wouldn't redirect if it doesn't encounter a file at that path, so should Dev really do it?

I guess it becomes a philosophical question, do you want Netlify Dev to be a day-to-day development tool that puts convenience first, or should it be a accurate representation of Netlify Prod to be used in the testing/pre-staging phase. I personally would more appreciate the latter, since webpack-dev-server etc. already work great as dev tools.

[sw-yx]

ah. yes, it should be an accurate representation of Netlify Prod.

[RaeesBhatti]

This has been fixed by ignoring /* /index.html 200 rule in netlify dev. If this pops up again, please open a new ticket.
Ref: https://github.com/netlify/cli/blob/fa86a6179419bf38022f4bcc36643b758d059736/src/utils/rules-proxy.js#L69