Google-cloud-node: Publish sources and source maps to npm

[carnesen]

@JustinBeckwith @kinwa91 Yesterday the @google-cloud projects were modified to remove source maps from their npm packages with the stated rationale, "When a user runs code in an environment that enables source maps their stack traces end in code that doesn't exist." It seems to me there are two solutions to this issue
1) Remove the source maps from the packages
2) Include both source maps and sources in the packages

Unless I'm missing something, solution 2 is way better, and in my experience including sources and maps is a standard best practice when publishing compiled npm packages. I know my workflow depends on both being present.

[mattwelke]

Solution 2 does sound reasonable. In the pre-TS world, we published "source" code too. Publishing both source maps and source code would enable debugging in the TS source code, good stacktraces, etc. I remember reading something from Microsoft recently about them recognizing the need to end up in TS source code when debugging code that calls NPM package code. I'll try to find it.

[carnesen]

Related: We should prolly publish the "declaration source maps" too https://github.com/Microsoft/TypeScript/issues/14479

[JustinBeckwith]

@bcoe this all started when some joker said we shouldn't publish source maps with our modules 🤣 Mind chatting with @soldair on this one? I have no opinion one way or the other, and trust whatever y'all decide.

[soldair]

There is no reasonable way for end users to modify and compile the typescript in dependencies. The published packages dont have the config files and dev dependencies to make a useful debugging experience.

Having traces that go through code you cant compile makes it easier to link lines of code into issues on github but makes it grep luck to insert a console.log in the correct place and debug yourself. This is why we decided to go with no source maps. We had already stopped shipping source and extra files some time ago to reduce the download size of published tarballs.

Now if stack traces included the line numbers and file paths to the source and the compiled output this wouldn't be an issue and might be kinda cool.

[bcoe]

had a good conversation with @soldair about this today. I think inline source-maps (--inlineSourceMap) would be a good solution:

• this reduces the number of files that ship with your project, while still providing source-map functionality.
• tools exist today that automate the prettification of stack traces, e.g., https://www.npmjs.com/package/source-map-support

Where things get more exiting though, are plans that are partially in the works to add source-map support into Node.js itself (and potentially V8) -- the first pass at this design will likely favor inline source-maps.

[justingrant]

plans that are partially in the works to add source-map support into Node.js itself (and potentially V8) -- the first pass at this design will likely favor inline source-maps.

Hi @bcoe - Better platform support for sourcemaps would be hugely useful. Is there a repo or wiki for this work? I'd like to follow progress.

I'm unfamiliar with inline sourcemaps. Is that where the original source is appended (as a data URI) to the end of the transpiled bundle js file? Is there a way to avoid downloading the sourcemap when not debugging?

[bcoe]

@justingrant I'm working with a mentee on the Node.js project to experiment with adding some initial source-map support to Node.js, our work is going to be logged here.

[sghosh-discovery]

Also, could you please remove references to source maps from the code when it does not exist? Otherwise, the parcel build complains about it. It still has:
//# sourceMappingURL=index.js.map

[bcoe]

👋 returning to this conversation after a long break 😆I've come around to @soldair's point of view (at least partially), with the realization that it source maps would increase the size of yargs by 70%.

I am of the mind that source-maps are useful in environments where the source doesn't exist, because a lot of the time a library maintainer simply wants a user to deliver a stack trace (they don't need the upstream user to have the source). In fact, Node.js just got fancy source-map support, making it quite valuable to ship source maps for the purposes of stack traces.

My current thinking, Is I'd like to come up with a convention for releasing debug versions of libraries, which include source-maps, while continuing to release the main version without (cutting down on library size significantly).

[justingrant]

Hi @bcoe, thanks for all the work you're doing to make sourcemaps and debugging better!

Hey, is there a prioritized list of user-facing problems and use-cases that need to be solved via improving library sourcemaps? My sense is that one source of disagreement in the community has stemmed from different ideas about what users need most. For example, if your conception of debugging is "fire up an IDE debugger, set breakpoints/logpoints, and then single-step through code to figure out the root cause of an error or unexpected result" then you'll have different instincts for what's important vs. if your preferred debugging strategy for library dependencies is "insert console.log statements in library code".

IMHO, the prioritized requirements are something like this:

1. Remove concerns that have prevented many maintainers from including sourcemaps and sources in their published packages
2. Make it easy for Node & browsers to report call stacks that include original-source row/col
3. Make it easy for devs using browser devtools and IDEs to view read-only original sources during a debug session, and to set breakpoints in, step into, inspect call stacks, and inspect local variables in the original source of library code
4. Reduce install time and disk space usage of node_modules
5. Don't add significant build-time complexity or introduce new classes of build bugs
6. Install-time solutions must be easy to use for both top-level and nested dependencies
7. Reduce download time of npm packages, esp. for initial installs or CI builds
8. Make it easy for IDE users to view, grep, and set breakpoints in library code even when a debug session is not running
9. Reduce aggregate bandwidth consumption from installing npm packages

Does that match what you're thinking? If not, how would you change or re-order that list, and why?

I'd like to come up with a convention for releasing debug versions of libraries, which include source-maps, while continuing to release the main version without (cutting down on library size significantly).

I have some questions & concerns with this convention:

• Wouldn't It add significant work and complexity for app devs? First a lengthy install (often >20s) before debugging. Then you need to remember to switch it back, and wait for another install to complete. Seems painful, esp. if you need to repeat a few times.
• How would you know ahead of time which packages have a debug version? Would you have to either look on GitHub or try it in the CLI and wait a few seconds for it to fail to find the @debug variant?
• Discoverability (knowing that this convention exists) might be a challenge, both for maintainers and for app devs, esp. newbies.
• Build tooling is already complicated and rickety. Asking package maintainers to add debug/release publishing seems like it'd add a lot of work and new bugs for the ecosystem, esp. on top of existing ESM vs. CJS vs. UMD complexity.
• Would this require double builds for every version, double test runs, etc.? (Dev vs release builds already exist, but AFAIK those are usually either/or, not both at once)
• npm i foo@debug would work OK for top-level dependencies, but wouldn't it require fairly advanced npm/yarn skills to handle nested dependencies? If yes that seems like a high bar to debug, esp. for newbies.

My sense is that an easier approach might be to extend package managers (npm, yarn), bundlers (webpack, rollup, parcel, etc), and debuggers (VSCode, Chrome, Firefox, Sentry, etc.) with small but globally-applicable tooling changes that would still allow debug files to be removed but wouldn't add as much extra work for maintainers and for library consumers.

Here's one idea about how this could work. I'm curious to know what you think!

• specify a debugFiles config option for package.json with the same semantics as files.
  
  
  
  • Default could be *.map or *.map; ./src, or just ./src, but devs can override to add other file patterns or even individual code files that are only imported in debug builds.
  
  
  • Bundlers and compilers could maybe fill debugFiles automatically because they know exactly which folders & files are being transpiled & source-mapped.
  
  
  • Docs should encourage using separate .map files because with inline sourcemaps there's nothing to put into debugFiles.
  
  
  • Should debugFiles be a subset of files or mutually exclusive? Once this rule is decided, npm publish should enforce it by failing if a package has debug files that don't follow the rule.
  
  
• npm i and yarn add would take a new option (e.g. --dontSaveDebugFiles) to prevent debugFiles matches from being copied to disk during install. This doesn't address the bandwidth issue but would improve disk usage and install perf.
• Debuggers like VSCode or browser devtools (or anything that needs info from original source, like call-stack display or Sentry) could JIT-download and cache packages during debugging to pull out sourcemaps or sources that are missing at runtime.
  
  
  
  • As an optimization, the NPM registry could auto-generate debugFiles-only tarballs which would be faster to JIT-download.
  
  
  • How would debuggers know if they need to go online to find debug files or should expect to find them on disk? Should package_lock.json and yarn.lock be extended with a new "noDebugFiles": true flag for this purpose?
  
  
• Download size could be addressed via the npm registry building and exposing package tarballs with debugFiles removed. An opt-in CLI option (e.g. npm i --dontDownloadDebugFiles) could let clients pull only debug-free tarballs where available, and like --dontSaveDebugFiles should recursively apply to nested dependencies.

Would something like this be workable?

[bcoe]

@justingrant sorry for the slow reply:

Here's one idea about how this could work. I'm curious to know what you think!

I think you have some good ideas. It's similar to what my colleague @murgatroid99 suggested, the difference being that he suggested a debug package that lives beside your library, and can be installed automatically by the tooling _(not dissimilar to how DefinitelyTyped has introduced types for a variety of libraries in the community)._

_The advantage of having things in their own package is that npm delivers your module as one tgz asset (_at least when I worked there this is how it worked_), so it would be hard to dynamically only deliver some of the files._

I think a good place to continue this conversation might be the Node.js package maintenance group, and/or the npm RFCs repo.

---

If we can come up with a better community practice, than I would happily follow it with the libraries I maintain for GCP.