Snowpack: Support for npm/yarn linked package updates

[FredKSchott]

Original Discussion: https://www.pika.dev/npm/snowpack/discuss/311
/cc @weyert, @FredKSchott

Developers will work on dependencies in separate repos, use npm link/yarn link to install them local to the Snowpack app, and then want to see those changes updated live in their Snowpack app.

Right now, we don't support this, and you essentially need to restart the server to detect these changes.

Option 1

• Run npm ls --link=true at dev start time, to get a list of all npm linked deps
• If one of those linked directories is used in the application, start watching it for changes
• on change to a watched, linked dependency, reinstall deps

Without some extra smarts around detecting whether imports/exports changed, we'll have to reinstall the entire set of dependencies for now.

Option 2

Could we treat linked packages as source code instead? As long as they are ESM, we could pass them through our source code build workflow instead of the package build workflow, and then no reinstalls needed. I think this would be worth trying first since it would be so much better if it works.

This would only need to run during dev. At build, the normal install should get the latest linked package changes.

We could use the same npm ls --link=true output scanner as option 1, to get the proper file path redirects.

[DorianBlues]

Great project!
I want to contribute to snowpack for these open issues, but I notice that @pika/pack doesn't support watch mode for typescript building. Now I have to rebuild snowpack every time I change the source code.
Is there any plan to support watch mode for @pika/pack?

[FredKSchott]

Thanks for your interest! There's an open issue for that support in the @pika/pack repo, but unfortunately a --watch command doesn't exist today.

[drwpow]

Note: it may be possible to also add support for local modules (discussion here: https://www.pika.dev/npm/snowpack/discuss/347). At the very least, it should be investigated.

[drwpow]

Added in #470.

We ended up going with something like Option 1. We discovered that snowpack build actually works already with linked packages! It was just the dev server that didn’t rerun snowpack install whenever those files changed (you could stop the server and run snowpack dev --reload but that wasn’t obvious). So the first reason was that option 1 required the fewest changes to Snowpack itself.

The other reason is that while it’s tempting to treat linked packages like local source code, the reality is that it’s not local source code. There’s a reason why that code is imported through npm and not a part of the project itself. So there may be other complicated implications around treating that code as if it exists in an environment it doesn’t. It felt safer to treat packages like packages, even if they’re on your local filesystem.

We did minimize the impact of this change by only listening to local linked packages and not every dependency you import. We can possibly add that in the future, but before we do, we wanted to make sure that didn’t have performance implications.

[airhorns]

I think that having picked option 1 over option 2 here makes a lot of sense except in the case of yarn workspaces, like what we're working through here: https://www.pika.dev/npm/snowpack/discuss/286

I think you're right that it's reasonable to expect code imported from npm / or via node_modules to be built already and not need further transformation, except for yarn workspaces, which use node_modules links to expose each of the workspace packages to the others. Developers are likely working in a workspace in order to make changes to multiple packages simultaneously and share code between packages without a build step being necessary, so if there is some way to go with Option 2 for packages inside a yarn workspace that'd be super duper!

[FredKSchott]

For now, I think it's out of scope for us to support compiling packages from source and watching them for changes. But, you should be able to accomplish something very similar by just running your package build step in some sort of watch mode. In that world, the compiled package code is updated on every change, and then snowpack will detect the change and reinstall from the compiled code on every change.

[airhorns]

I get that. I used a different workaround which is to put all my shared code in the web package that Snowpack is building and then source it from the api package (and/or the other ones) in my workspace. The other tools I'm using, like ts-node (and ts-node-dev) as well as VSCode are fine with the cross workspace imports going the other way, so the Snowpack built package inside the workspace just kind of sucks in all the code that it needs to require.

I totally understand why supporting cross-package builds is annoying. It violates the way that normal NPM packages are supposed to work, and in the general case I think it's fine to not build stuff from node_modules. It also probably poses significant challenges to implement internally. It's just that for workspaces, it sucks as a developer experience because now you need to run one watch process per package in the workspace. You pay a productivity penalty for trying to do things "right" and re-use code or properly factor things. It's back to the non-workspace developer experience of many repos with many build toolchains and many fiddly bits that all have to agree.

So I'd ask if you'd reconsider. If you zoom out one level, "compiling packages from source and watching them for changes" is kind of Snowpack's reason to exist (as a humble user who didn't build it of course). It's just that the package in question happens to be in a sibling folder instead of a nested folder.

Also, FWIW, this is possible to do with Webpack, you just have to deal with all the other bungus that Webpack makes you deal with and Snowpack doesn't.

Also, one final thing, because of Snowpack's terminal clearing / owning behaviour, it doesn't play nice with other tools sharing the same stdout, so it's kind of a weird experience to run it alongside other watcher tools like you might with concurrently. Even if I wanted to have one master command that ran a bunch of watcher processes all at once Snowpack sucks away all their output each time it reloads, so it's hard to catch if the other tools had an error anything like that.

[FredKSchott]

Thanks for the extra context, I can definitely sympathize and want to solve for this use-case, but you rightfully pointed out that it would require a BIG change to how Snowpack works internally. I'll keep this in mind, and it could make sense as a major feature to tackle later on down the road.

[brunolemos]

I totally agree with all points raised by @airhorns.

I got super excited about Snowpack on my initial tests. Super fast DX, great work!
I was making a repo comparing its speed against alternatives like next.js, create-react-app and vite to share the results on social media when I noticed the lack of monorepo support.

Unfortunately 99% of my projects use yarn workspaces and I'd lose the main benefit if I use snowpack.
Here's some problems I faced:

1. When changing the content of my-package/App, the page is doing a full reload
2. After the full reload, it's showing the content of the previous compilation instead of the current one _(it's probably reloading before my tsc -w finishes)_
3. import MyPackage from 'my-package/App' works only if App file is an already-transpilled .js file; fails if it contains JSX code, for example, or if the extension is .jsx, .tsx, etc
   

@drwpow: It felt safer to treat packages like packages, even if they’re on your local filesystem

I totally understand the rationale here and it makes sense. But like @airhorns said, for Yarn Workspace projects we wanna treat them as source files, so the Option 2 makes more sense in that use case.

Maybe if we don't wanna enable that by default, we could have an option to opt-in to that behavior?

@FredKSchott: it would require a BIG change to how Snowpack works internally

Sad to read that :( I was hoping I could make a small patch on a fork to get that working.
But I really hope it gets tracked, I believe it's worth it.
Monorepos are becoming more common each day.

[FredKSchott]

Just for anyone else reading, I do want to clarify that Snowpack does support monorepos: your project dependencies are getting correctly loaded from the sibling directories without having to npm install them directly. (besides your 2. above, that could be a bug! @drwpow may be worth taking a look, if we're not clearing the cache after a dependency re-install)

If I'm understanding this thread right, what IS still missing is the ability to support a monorepo where parts of your application exist as seperate packages, outside of your main application, and you want to do active development on them without waiting for a re-install after every change. We should definitely support this! But the big problem is that it messes with our idea of what's "source code" and what's "dependency code", which are handled in different ways today.

[brunolemos]

what IS still missing is the ability to support a monorepo where parts of your application exist as separate packages, outside of your main application, and you want to do active development on them without waiting for a re-install after every change

yes, exactly. for example, this folder structure:

• /packages
  
  
  
  • core
  
  
  • components
  
  
  • web
  
  
  • mobile
  
  

The web uses both the core and components packages. There's active development on all three, so the expectation is that they're treated as source files with all the benefits like instant change.

When using create-react-app, for example, they also only have basic support for monorepo, but adding a few lines overriding their webpack config is enough to get the desired behavior:

https://github.com/devhubapp/devhub/blob/d1ee4eb517a1217c95a50d41603976cfa79c0e6e/packages/web/config-overrides.js#L11-L32

[FredKSchott]

That would probably be the best that we could do: if you opt-in, we treat your linked, monorepo packages as if they were source files, using the exact same build as your application instead of any build defined in the package itself. That means we'd need a way to scan THEM for dependencies of their own, and somehow add those to the set of dependencies that you need to install to run your application.

Question: would it be acceptable to the source code inside those packages the same way that you treat your application source code? We could go a step further and use the actual source code within those packages instead of the built package themselves, although most packages don't define a "source code" entrypoint so you'd need to add that to your package.json.

[brunolemos]

@FredKSchott sounds good. At least on my projects that's the case, the build step is only defined in the main package and the other packages are dumb source files.

I suppose some projects need different babel config on their packages but dunno how common is that though.

I use slightly different tsconfigs but that's already supported nowadays by snowpack ("run:tsc": "tsc -b" + adding references to the main tsconfig + composite": true to the other tsconfigs).

[MatejBransky]

@FredKSchott

Question: would it be acceptable to the source code inside those packages the same way that you treat your application source code? We could go a step further and use the actual source code within those packages instead of the built package themselves, although most packages don't define a "source code" entrypoint so you'd need to add that to your package.json.

I think that Parcel.js v1 already does that. It looks for source attribute in package.json of the dependency and if it finds it then it compiles the source code of this dependency. 🙂 We already use this approach in companys monorepos.

Related merged PR in Parcel.js: https://github.com/parcel-bundler/parcel/pull/1101

[brunolemos]

If someone could give me some directions about which files/methods would need to be updated I'm willing to take a look.

• I tried messing with onDepWatchEvent
• mounted the workspace package using something like "mount:../components": "mount ../components --to /web_modules/components" so it would get to onWatchEvent
• removed node_modules from ALWAYS_EXCLUDE
• commented out config.exclude just in case
• replaced some reload broadcast message with update+url

and some other crazy changes but wasn't able to get the instant-update behavior.

[geoyws]

FWIW I'm using https://github.com/cleric-sh/repro 's method for my Yarn Workspaces monorepo and it works.
Unfortunately, I'm still trying to get it to import JSX from the imported repo via snowpack build with no success. I'm working around a Cannot find module... error where it keeps looking for index.js in /build/src/index.js when it lives in /_dist_/index.js.

[chopfitzroy]

Just chiming in here, above linked "source" field that is used in parcel is the same field that microbundle uses to build from.

It seems to have become something of a standard among bundlers, would it be possible to evaluate this field, see if the file exists, and then build from it, and watch if possible?

That being said I have experienced issues in the past if the source code has custom syntax (i.e using symbols like @ to define the project root in a Webpack config) so it might be worth being able to disable?

[joaotavora]

Hi, I'll just take the liberty of pointing you to what I think is a problem with this feature: https://github.com/snowpackjs/snowpack/discussions/1167#discussioncomment-234008

[joaotavora]

I'm sorry for beating this horse, but after spending many hours debugging this (and diving into the Snowpack code), it seems that this feature was completely broken by #762, which includes this change to dev.ts.

commit 610fb9d28452bf24758387a00fe81a4ed103e503
Author: Drew Powers <[email protected]>
Date:   Thu Aug 13 15:53:52 2020 -0600

    Dev server improvements (#762)

    * Improve dev server output

    * fred attempt

    * fix bad spacing on http url line of header

    * add proxy message

    * add build watch message similar to dev

    * updates

    Co-authored-by: Fred K. Schott <[email protected]>

diff --git a/packages/snowpack/src/commands/dev.ts b/packages/snowpack/src/commands/dev.ts
--- a/packages/snowpack/src/commands/dev.ts
+++ b/packages/snowpack/src/commands/dev.ts
@@ -954,3 +880,3 @@
   function onDepWatchEvent() {
-    reinstallDependencies().then(() => hmrEngine.broadcastMessage({type: 'reload'}));
+    hmrEngine.broadcastMessage({type: 'reload'});
   }

So this explains why one changes a dependency, the browser gets updated , but nothing really gets re-installed.

I can't tell from the commit message why this was removed (was it the "fred attempt"?). Anyway I would think that it would be possible, looking at that part of the code, to extract only the dependency that needs to be reinstalled and perform that. Is there a function that one can call that re-installs a single dependency? Then, presumably snowpack would start working again with npm link. @FredKSchott please advise.

[FredKSchott]

Oh interesting! You're right, that was a regression incorrectly merged in that PR. Will create a prioritized issue to fix

[brunolemos]

@FredKSchott where can we follow the current discussions about treating linked packages as source files, specially in yarn workspace environment? (since this and some other related issues are closed)

[Jack-Works]

I'm having this problem now. Is there a quick way to resolve this? For example, hard code my linked dependencies names in snowpack.config.js or somewhere

[stephtr]

I'm having this problem now. Is there a quick way to resolve this? For example, hard code my linked dependencies names in snowpack.config.js or somewhere

In order to circumvent the issue, I made a symlink pointing to the linked dependency's location. In the snowpack config I added:

mount: {
    'linked-dependency': '/linked-dependency',
},
alias: {
        'linked-dependency': './linked-dependency',
},

Despite not working perfectly, I at least get hot-reloading for my linked dependency while developing.