Vscode-js-debug: js-debug should follow redirects when requesting sourcemaps

[gkaindl]

In my setup, the vscode debugger extension can't load the source map for the browser app bundle anymore, even though it always used to work in previous versions of vscode. The error message given is Could not read source map for http://127.0.0.1:8080/App.d4a65e86.js: Unexpected end of JSON input.

Trace log is attached: vscode-debugadapter-1.json.gz

launch.json entry (probably not related, though) – chrome-helper.sh just launches a specific version of Chromium with a set of debugging extensions installed, so that we can share the setup more easily across the team. The path override is just necessary to map the files in the source map to local files, but it's most definitely unrelated to this issue.

{
      "name": "Debug in Chromium",
      "type": "chrome",
      "request": "launch",
      "url": "http://127.0.0.1:8080/",
      "runtimeExecutable": "${workspaceRoot}/.vscode/chrome-helper.sh",
      "webRoot": "${workspaceRoot}/src",
      "smartStep": true,
      "sourceMaps": true,
      "sourceMapPathOverrides": {
        "../*": "${webRoot}/*"
      }
    }

VS Code Version: 1.47.1 on macOS 10.15.4

Additional info: The source map itself is fine, valid JSON, and works in the browser itself, and it used to work well in older versions of vscode, too. The error message given is Could not read source map for http://127.0.0.1:8080/App.d4a65e86.js: Unexpected end of JSON input.

I suspect it is because the source map URL is a redirect to the actual URL, which the new debugger can't seem to deal with. Here's the output of curl when you request the source map, as the debugger sees it.

As you can see, it requests source map at /, but gets redirected to /app. This should, however, be allowed, I think, and works fine with browsers and older versions of the debugger, so I have a strong suspicion that maybe the redirect is what causes the confusion, and the "Unexpected end of JSON input" error is bogus, because the redirect response does not contain a body.

Unfortunately, I can't share the source map itself (since the app is not open-source), but please take my word that the source map itself is perfectly valid :-)

gck@tenenbaum$ curl -Ivvv 'http://127.0.0.1:8080/App.d4a65e86.js.map'
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> HEAD /App.d4a65e86.js.map HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< content-length: 0
content-length: 0
< location: /app/App.d4a65e86.js.map
location: /app/App.d4a65e86.js.map
< connection: keep-alive
connection: keep-alive
< date: Wed, 22 Jul 2020 09:34:07 GMT
date: Wed, 22 Jul 2020 09:34:07 GMT

< 
* Connection #0 to host 127.0.0.1 left intact
* Closing connection 0

[connor4312]

Thanks for the good issue! Indeed I don't think we follow redirects for sourcemaps.

[gkaindl]

Thanks for putting it into the milestone! :-)

In the mean time, I've adapted the debug build of my backend to deliver the source map without a redirect, and I can confirm that this was/is the problem, so following redirects should fix it!

[connor4312]

Should be fixed in the next nightly

[gkaindl]

Unfortunately, it is not working with the nightly. There is now the following error reported in the debug console:

Could not read source map for http://127.0.0.1:8080/App.d4a65e86.js: Unexpected 200 response from http://127.0.0.1:8080/App.d4a65e86.js.map

So the redirect is followed correctly, there's then a problem further down the request pipeline. Also, the 200 response can't be coming from the URL http://127.0.0.1:8080/App.d4a65e86.js.map, as the error message suggests, but from http://127.0.0.1:8080/app/App.d4a65e86.js.map, where the redirect leads to.

For reference, here's the corresponding curl output:

gck@tenenbaum$ curl -I 'http://127.0.0.1:8080/App.d4a65e86.js.map'
HTTP/1.1 301 Moved Permanently
content-length: 0
location: /app/App.d4a65e86.js.map
connection: keep-alive
date: Thu, 23 Jul 2020 09:25:21 GMT

gck@tenenbaum$ curl -I 'http://127.0.0.1:8080/app/App.d4a65e86.js.map'
HTTP/1.1 200 OK
etag: 1595453717.3509617-3426733
content-length: 3426733
connection: keep-alive
date: Thu, 23 Jul 2020 09:25:29 GMT

Maybe an additional hint: The corresponding javascript bundle to the source map http://127.0.0.1:8080/App.d4a65e86.js is also a redirect (to http://127.0.0.1:8080/app/App.d4a65e86.js) in my setup. The source map location in the bundle is only App.d4a65e86.js.map (no path components), so if the debugger was aware of the actual URL from which it got the js bundle, the source map URL should be deduced to be http://127.0.0.1:8080/app/App.d4a65e86.js.map directly. Thus, maybe the problem is also about the debugger not being aware that a redirect has been followed to load the actual JS bundle itself?

[connor4312]

Okay, that's interesting. Setting up a redirect locally, works. If you use got to make a request against http://127.0.0.1:8080/App.d4a65e86.js.map, does that work? In the linked commit I just moved us from our custom http client wrapper to using Got with followRedirect = true.

[gkaindl]

Okay, I think I've figured it out – it's not the redirect, but got seems to be stricter about decoding deflate-compressed responses (see https://github.com/sindresorhus/got/issues/1124), and that's what's causing the issue now.

I'm using Swift Vapor as a backend framework, which is in turn based on SwiftNIO and uses their response compressor from NIO-Extras, and apparently, there's an incompatibility between the two about the zlib compression (gzip compression seems to work, "deflate" does not).

I think I'll open an issue with Vapor and see what they say.

If you want to "fix" it in the debugger, it works to set the "Accept-Encoding" header to "identity" for the got request. I'm not sure if it's sensible to do so or not – source maps tend to be large on one hand, but on the other, there are quite a few issues in the got github about it failing to decode a compressed response, so it might be better to be on the safe side?

[connor4312]

Thanks for the investigation, those are some good findings.

I'm not sure I want to disable compressed responses by default, because as you note (particularly over the public internet) using compressed responses can speed things up quite a bit and using them was one of the reasons I wanted to move to got. It's understandable, but somewhat disappointing, that got adheres to the strict specification rather than doing extra work for compatibility.

I'd like to wait a little bit and see what Vapor says, and also see if we get any further issues coming in while this is on nightly. If Vapor is amenable to fixing it and we don't hear anything else, then I'd like to keep allowing compressed responses. If not then perhaps a setting to disable it would be appropriate/

[gkaindl]

Just a quick update: It turned out that there was a bug in SwiftNIO that would cause a part of the compressed response to be missing in a specific code path, and that would cause a non-stream-oriented decompressor to choke when decompressing it – which is what happened to got.

The issue has been resolved now, so I don't think an option to load source maps with Accept-encoding: identity is needed anymore!

Thank you for your help in looking into this!

[connor4312]

Good to hear, thanks for the update!

[roblourens]

Should I just verify the redirect response part or does the second part need verification (or can @gkaindl confirm that everything now works as expected)?

[gkaindl]

@roblourens Thanks for asking! Yes, everything works as expected now!

It was a combination of issues – the vscode-js-debug issue was not following the redirects for source maps (this is definitely working fine now after switching to the got library to fetch them), but that then triggered a bug in Swift-NIO's response compressor, which was causing the follow-up issue and confusion. The SwiftNIO problem has been fixed in the mean time, too.

I'm running the nightly build and vscode "insiders" now, and everything works as expected!

[connor4312]

Thanks for confirming!