Haul: Source-maps missing in dev builds

[ScottPierce]

Current Behavior

I'm testing out building an app with Kotlin, and compiling sources to JS. Everything seems to be working really well with haul, except for stack-traces. I can debug kotlin sources, and everything. I just see this "can't symbolocate error" every time I see a crash:

GET /index.android.bundle 200 11019287 - 12.494 ms
POST /symbolicate 404 151 - 0.214 ms
POST /symbolicate 404 151 - 0.141 ms

If I watch the logs, I see the following:

01-20 22:31:23.555  1925 15179 E ReactNativeJS: TypeError: Object.freeze can only be called on Objects.
01-20 22:31:23.555  1925 15179 E ReactNativeJS: 
01-20 22:31:23.555  1925 15179 E ReactNativeJS: This error is located at:
01-20 22:31:23.555  1925 15179 E ReactNativeJS:     in HelloWorld (at renderApplication.js:35)
01-20 22:31:23.555  1925 15179 E ReactNativeJS:     in RCTView (at View.js:71)
01-20 22:31:23.555  1925 15179 E ReactNativeJS:     in View (at AppContainer.js:102)
01-20 22:31:23.555  1925 15179 E ReactNativeJS:     in RCTView (at View.js:71)
01-20 22:31:23.555  1925 15179 E ReactNativeJS:     in View (at AppContainer.js:122)
01-20 22:31:23.555  1925 15179 E ReactNativeJS:     in AppContainer (at renderApplication.js:34)
01-20 22:31:23.567  1925 15179 E ReactNativeJS: TypeError: Object.freeze can only be called on Objects.
01-20 22:31:23.567  1925 15179 E ReactNativeJS: 
01-20 22:31:23.567  1925 15179 E ReactNativeJS: This error is located at:
01-20 22:31:23.567  1925 15179 E ReactNativeJS:     in HelloWorld (at renderApplication.js:35)
01-20 22:31:23.567  1925 15179 E ReactNativeJS:     in RCTView (at View.js:71)
01-20 22:31:23.567  1925 15179 E ReactNativeJS:     in View (at AppContainer.js:102)
01-20 22:31:23.567  1925 15179 E ReactNativeJS:     in RCTView (at View.js:71)
01-20 22:31:23.567  1925 15179 E ReactNativeJS:     in View (at AppContainer.js:122)
01-20 22:31:23.567  1925 15179 E ReactNativeJS:     in AppContainer (at renderApplication.js:34)
01-20 22:31:23.575  1925 15179 W ReactNativeJS: Unable to symbolicate stack trace: JSON Parse error: Unrecognized token '<'
01-20 22:31:23.577  1925 15179 W ReactNativeJS: Unable to symbolicate stack trace: JSON Parse error: Unrecognized token '<'

Expected Behavior

My stack-traces would be able to be symbolicated just fine.

Haul Configuration (webpack.haul.js)

module.exports = ({ platform }, defaults) => ({
  entry: `./index.js`,
  module: {
    ...defaults.module,
    rules: [
      ...defaults.module.rules,
      {
        test: /\.js$/,
        use: ["source-map-loader"],
        enforce: "pre"
      }
    ]
  }
});

Your Environment

| software | version
| ---------------- | -------
| Haul | 1.0.0-beta.12
| react-native | 0.52
| node | 9.4.0
| npm or yarn | yarn 1.3.2

[Krizzu]

Hey, thanks for bringing this up. I've noticed the same warning in my current project, we'll try to investigate it.

[ScottPierce]

Just saw the following error while debugging an exception I created to repro this issue:

 WARN  Unable to find source map.
POST /symbolicate 404 151 - 0.455 ms

 WARN  Unable to find source map.
POST /symbolicate 404 151 - 0.437 ms

The sourcemaps are definitely there. I'm able to see all the kotlin sources in chrome, and also set breakpoints, even on the line that throws an exception. I'll work on creating a reproducible example project.

[ScottPierce]

I've created a sample repo to reproduce this issue. The CrashExample branch purposefully throws an exception to show off the issue: https://github.com/ScottPierce/kotlin-react-native/tree/CrashExample

[simonbuchan]

Not worth a separate issue, but RN will sometimes ask to symbolicate an empty stack (I think for console.error(), e.g. caught by an error boundary), it's a little confusing that returns a 404 too, even if it has the same behavior.

[ScottPierce]

I added log to simbolicateMiddleware.js to log the body of the response and this is what I see:

{"stack":[{"file":"http://10.0.3.2:8081/index.bundle?platform=android&dev=true&minify=false","methodName":"RuntimeException_init_0","lineNumber":3845,"column":24},{"file":"http://10.0.3.2:8081/index.bundle?platform=android&dev=true&minify=false","methodName":"HelloWorld.render","lineNumber":1,"column":1361},{"file":"http://10.0.3.2:8081/index.bundle?platform=android&dev=true&minify=false","methodName":"finishClassComponent","lineNumber":1,"column":129987},{"file":"http://10.0.3.2:8081/index.bundle?platform=android&dev=true&minify=false","methodName":"updateClassComponent","lineNumber":1,"column":129489},{"file":"http://10.0.3.2:8081/index.bundle?platform=android&dev=true&minify=false","methodName":"beginWork","lineNumber":1,"column":137366},{"file":"http://10.0.3.2:8081/index.bundle?platform=android&dev=true&minify=false","methodName":"performUnitOfWork","lineNumber":1,"column":178002},{"file":"http://10.0.3.2:8081/index.bundle?platform=android&dev=true&minify=false","methodName":"workLoop","lineNumber":1,"column":179134},{"file":"http://10.0.3.2:8081/index.bundle?platform=android&dev=true&minify=false","methodName":"Object._invokeGuardedCallback","lineNumber":1,"column":3344},{"file":"http://10.0.3.2:8081/index.bundle?platform=android&dev=true&minify=false","methodName":"invokeGuardedCallback","lineNumber":1,"column":2153},{"file":"http://10.0.3.2:8081/index.bundle?platform=android&dev=true&minify=false","methodName":"renderRoot","lineNumber":1,"column":180554}]}

Is a url for the file acceptable in this case? This seems like it could be why the file can't be found:
"file":"http://10.0.3.2:8081/index.bundle?platform=android&dev=true&minify=false"

[simonbuchan]

Given that it doesn't look like .file is used at all except for sniffing the running platform for the --platform all case, that shouldn't be the problem.

The error comes from symbolicateMiddleware.js:56, in createSourceMapConsumer() before it even starts mapping frames. Essentially looks like the middleware couldn't find the webpack output file in the webpack file system, which seems odd. Perhaps the structure changed in some later version of webpack? Seems like webpack should have a simpler method to get the source map of a file.

Note, you can debug with node --inspect-brk node_modules/haul/bin/cli.js --platform android (for node 6, --inspect --debug-brk), which will give you a chrome inspector link you can open. Or, if you're using a Jetbrains IDE (possibly including Android Studio?) you should be able to just point a node debug run config at the cli.js (you can do it with the npm script, but it's more complicated)

[ScottPierce]

@simonbuchan Thanks, that was helpful, although I didn't get the --inspect-brk to work. Not sure why.

The problem seems to be that it's looking at the following path: <project directory>/dist, and ./dist doesn't exist in my project directory. Where is the actual webpack output directory, and why would it be looking in the wrong place?

[simonbuchan]

--inspect-brk is a shorthand for --inspect --debug-brk added in Node 8, so most likely you're using an earlier node if that errors. If the link doesn't work, it must be a chrome-like browser, firefox, IE etc don't implement the chrome-devtools:/// protocol.

It's in an in-memory FS, just like usual webpack-dev-server, see: https://webpack.js.org/api/node/#custom-file-systems so you will need a debugger or a lot of painful messing with debugging statements.

[ScottPierce]

So the problem is with symbolicateMiddleware.js line 48:

const base = delve(compiler.outputFileSystem.data, hops);

base has 2 keys. An empty string, and index.android.bundle. It then looks for index.android.bundle.map and it's not there.

When I run yarn haul bundle --platform android, the index.android.bundle.map file is generated, along with the index.android.bundle file.

How would you recommend debugging this further? Why would the file not be generated in the in-memory filesystem? Do I need to use the devtool setting maybe?

[simonbuchan]

Hmm, could be the source maps are not getting generated externally in haul start - either not at all or they are generated inline, and for some reason haul bundle build is. I suspect it's due to the --dev or --minify defaults being different, so you could try haul bundle --dev true --minify false which should be pretty much the same output as haul start

[ScottPierce]

You are right. Running haul bundle --dev true --minify false only generates the index.android.bundle file. Would this have anything to do with me using the source-map-loader?

module.exports = ({ platform }, defaults) => {
    return ({
      entry: `./index.js`,
      module: {
        ...defaults.module,
        rules: [
          ...defaults.module.rules,
          {
            test: /\.js$/,
            use: ["source-map-loader"],
            enforce: "pre"
          }
        ]
      }
    });
}

What else would cause the map file not to be generated?

[simonbuchan]

Yeah, looking at it, the dev mode source maps are created by: https://github.com/callstack/haul/blob/master/src/utils/makeReactNativeConfig.js#L134-L136 which is equivalent to devtool: 'module-eval', which will not generate .map files.

Blame points at #289 which claims source maps still work with those changes, but that is devtools, while symbolicate is needed for redboxes? @jukben any ideas how to resolve both these? Use both eval and external source maps? Sounds expensive.

[ScottPierce]

Is there any way that I could override this for now and get this working?

[ScottPierce]

I undid the change from that PR locally, and stacktraces are working now; however debugging is no longer working. I can't see my sources in chrome anymore.

I had to make one minor modification to get stacktraces with proper lines and files: Here's the old setting for devtool: devtool: bundle ? 'source-map' : 'eval-source-map',, and I wasn't seeing filenames or correct line numbers with that. When I changed it to devtool: 'source-map',, the stacktrace seems fine.

[simonbuchan]

Hmm, devtool: 'eval-*' is equivalent to the current plugins: [new EvalSourceMapDevToolPlugin()]. It looks like the root of the issue is that before #289 non-bundle builds generated both eval and external source maps - eval required for remote debugging, external for symbolicate. This isn't an ideal state either, but see below. (On the other hand, it probably shouldn't be adding SourceMapDevToolPlugin and devtool: 'source-map when bundling, since the latter adds the former)

When remote debugging, the code is actually running in the debugger: e.g. Chrome or React Native Debugger, all the JS<->Native communication is remoted over the debugger connection(!), and chrome can look up source maps when generating an error stack itself. But when running inside RN's embedded JavaScriptCore runtime, it doesn't have access to source maps, so it delegates to the packager via symbolicate. So whatever is done needs to work in both. I assume you don't get stacks when not remote debugging because of this issue: the symbolicate call is 404ing? But be aware console.errors() will redbox without a stack, and I think that's used by some of the react-native polyfill / shims?

I'm not part of the team, but I would guess a PR should have:

• JS errors display correct stack in redbox whether remote debugging or not
• When remote debugging, console, breakpoints, etc. show correct source file and location, with original source.
• haul bundle size does not increase dramatically

So I guess start experimenting, reading up on https://webpack.js.org/plugins/source-map-dev-tool-plugin/ etc.? This stuff is tricky.

Don't focus on this, but it would be cool if chrome workspace support worked too - for that the moduleFilenameTemplate should create files like file:///C:/path/to/file on windows, file:///path/to/file on nix..., see how create-react-app does it in dev - though in prod they make the url relative for I assume consistency across build machines.

[simonbuchan]

I've needed to re-implement this for a work project, since I'm dynamically loading sources at runtime, so I have multiple bundles. Here's the relevant code that pulls out source maps and looks them up:

const { SourceMapConsumer } = require('source-map');
const sourceMapConsumers = new Map();

// MultiCompiler.apply() is a no-op for some reason!?
(compiler.compilers || [compiler]).forEach(compiler => {
  compiler.plugin('compilation', compilation => {
    console.log('compilation', compilation.name, 'created');
    // Enable module source mapping if it wasn't already
    compilation.plugin("build-module", module => {
      module.useSourceMap = true;
    });
    compilation.plugin('after-optimize-assets', assets => {
      sourceMapConsumers.clear();
      Object.keys(assets).forEach(file => {
        const asset = assets[file];
        // TODO: Don't try to get source maps for assets?
        // TODO: Only re-create if hash has changed
        const map = asset.map();
        if (!map) {
          // TODO: should be using compilation.emitWarning()(?)
          console.warn('  -', file, 'has no source map');
          return;
        }
        sourceMapConsumers.set(file, new SourceMapConsumer(map));
      });
    });
  });
});

const Url = require('url');
app.post('/symbolicate', (req, res) => {
  const chunks = [];
  req.on('data', chunk => chunks.push(chunk));
  req.on('end', () => {
    const body = JSON.parse(Buffer.concat(chunks).toString());
    // stack is an array of:
    //   { file: "http://10.0.2.2:8081/index.bundle?...",
    //     methodName: string,
    //     lineNumber: number,
    //     column: number }
    body.stack.forEach(frame => {
      const { pathname } = Url.parse(frame.file);
      const file = pathname.substr(1); // skip leading /
      const fileSourceMapper = sourceMapConsumers.get(file);
      if (!fileSourceMapper) {
        console.warn(`${file}:${frame.lineNumber}:${frame.column} symbolicate failed: no source map`);
        console.warn('  valid files:', ...sourceMapConsumers.keys());
        return;
      }
      const position = fileSourceMapper.originalPositionFor({
        line: frame.lineNumber,
        column: frame.column,
      });
      if (position.line === null) {
        console.warn(`${file}:${frame.lineNumber}:${frame.column} symbolicate failed: no mapping found`);
        return;
      }
      frame.file = position.source;
      frame.lineNumber = position.line;
      frame.column = position.column;
    });
    res.end(JSON.stringify(body));
  });
});

If nobody else picks this up, I'll see if I can cook up a PR for this.

[simonbuchan]

Note that if you are actually using multiple bundles, you need to do some magic to affect the file for external files, the simplest is eval(source + '\n//# sourceURL=' + url), but that only works for remote debugging. JavaScriptCore will simply not include eval-sourced frames, which is a pain. I suspect if I expose the native JSEvaluateScript in a RN plugin I could get that working, and be more efficient to boot, but this is fine for now.
Also note that in my experimentation, any other devtool plugin than new SourceMapDevToolPlugin() (= devtool: 'inline-module-source-map') will either break this symbolication or not have correct source maps in remote debugger. I'm pretty sure I can fix that eventually, but it'll take a bit of time.

[jukben]

Good to hear that you have some progress with it! I'm keeping my fingers crossed. 💪

[ScottPierce]

@simonbuchan Would this provide both types of source-maps for both production and dev builds? Where would this code end up going? The app.post('/symbolicate', (req, res) => { is clearly in the simbolicateMiddleware.js. What about the (compiler.compilers || [compiler]).forEach(compiler => { code block?

[ScottPierce]

@simonbuchan Also, I don't understand your comment about multiple bundles. Can you explain that better? I'm not sure I have any context for what the "multiple bundles" would be.

[simonbuchan]

This is symbolicate only, but I mention how it interacts with existing source map plugins. I wrote this as part of a standalone haul equivalent, so no places for things to go strictly (which is part of why this isn't a PR yet), the first is a webpack plugin (without the standard class wrapper) that grabs the source maps from assets as they are generated, including updates during watch, the second is the middleware that uses the output of that.

Webpack can output multiple bundles, either by having multiple entries (not one entry with multiple modules), or passing multiple configurations, hence the comment about MultiCompiler.

Usage is as I described in the second comment, at the moment it's eval().

[zamotany]

This might be relevant:

339 not only will remove problems with MultiCompiler mode, but it will use both eval source mas and source maps for respectively debugger and symbolicate middleware. Unfortunatelly it has some performance cost, but we will try to minimize it in follow-up PRs.

[simonbuchan]

I've found inline source maps work fine for remote debugging, so my current setup is (basically) new SourceMapDevToolPlugin({ filename: bundle && '[file].map' }), which is equivalent to devtool: bundle ? 'module-source-map' : 'inline-module-source-map'. This is slower for dev, but I haven't gotten cheap to work properly yet for me. This will always set the asset.__SourceMapDevToolData[sourceMapFilename] to the source map, if I've been reading the source correctly, so it could be pulled off of there.

[goodmind]

@simonbuchan Can you explain? I can't get sourcemaps for index.ios.bundle while running haul bundle --dev true --minify false neither with module-source-map nor inline-module-source-map

[simonbuchan]

That command generates the source maps on disk the same as they are used by haul start by default - it's for debugging why it's not working, not for fixing it.
That stack looks correctly source mapped anyway, otherwise the lines would all be on line one of main.jsbundle. The fact that it's a useless stack is on the RN runtime - try enabling "break on exceptions" in the debugger (the stop sign on the top right)

[goodmind]

@simonbuchan it's actually caught exception as explained in ExceptionsManager

// Workaround for reporting errors caused by `throw 'some string'`
  // Unfortunately there is no way to figure out the stacktrace in this
  // case, so if you ended up here trying to trace an error, look for
  // `throw '<error message>'` somewhere in your codebase.

and then there's console.error :\

[simonbuchan]

As I said,

The fact that it's a useless stack is on the RN runtime

[goodmind]

@simonbuchan I mean I can't pause on it because it's caught exception

[simonbuchan]

Well then, it's a good thing I didn't suggest that!

try enabling "break on exceptions" in the debugger (the stop sign on the top right)

[zamotany]

No longer an issue with revamped version of Haul on next branch.