Nx: package.json per app

[comunitius]

First of all,. just like to say thank you! I love nrwl nx and how well organized the code looks like when using nx! Also, it's awesome you folks are investing in React!

Prerequisites

Please answer the following questions for yourself before submitting an issue.
YOU MAY DELETE THE PREREQUISITES SECTION.

• [X] I am running the latest version
• [X] I checked the documentation and found no answer
• [X] I checked to make sure that this issue has not already been filed
• [X] I'm reporting the issue to the correct repository (not related to Angular, AngularCLI or any dependency)

Expected Behavior

Is there a way we could have a package.json per app? If you are using docker, and would like, let's say, have an angular app running in one container, and the nest api into another container, then both containers will have to run npm install over the all the npm packages even when angular packages are only a subset of all the packages (and viceversa with nest). Additionally, if we do npm install inside the containers, huge packages like cypress will be downloaded, even if they have nothing to do with the api (nest) build. This take its toll especially when running a CI build in travis, circleci, etc (yeah, I know there is cache, but it's not 100% reliable).

Please describe the behavior you are expecting:

Would love to see every app having different package.json. Plus I think it is cleaner. Why having one package,json to rule them all? Seems very tightly coupled.

Current Behavior

What is the current behavior?
There is only one package.json for all apps using nx

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Context

Please provide any relevant information about your setup:

• version of Nx used: 8.4.13

[the-ult]

A related question/problem we are facing with using a node graphql-api is:

• how to create small node_modules just for production and just for the node application.
  Since otherwise node_modules will include the nx, angular, cypress libraries, etc

To solve this, we are using a separate (specific) package.json in the apps/graphql-server.

However, this results in other problems like:

• nx affected:build does not work for the graphql-server (needs to be started from it's own folder)
• npm install order/dependency between main package.json and apps/grahpql-server/package.json
• etc

So the question really is:

• Can we prevent (needing) multiple package.json files when using node applications?

_N.B._

• could not find anything in the docs
• We are using separate docker containers as well

[vsavkin]

Thanks for submitting the issue.

You could create a package.json per project, and for instance use something like yarn workspaces to make npm installs relatively fast. We used to do it for one of our internal projects.

We believe in the single-version policy, where, for instance, all applications using NestJS, use the same version of NestJS. Even though it looks a bit crazy at first, we think it is beneficial. Companies like Google do it. So having one node_modules at the root is very similar to what Google does (with a single third_party).

I'll try to find sometime this week to author a guide on single-version policy, where I'll also cover how to add multiple package.json files if you decide that single-version policy doesn't work for you.

[comunitius]

Nice thanks @vsavkin ! In the meanwhile while you create the documentation, do you have an example of a project with different package.jsons?. I just want to decouple my react app from my nest app for now

[the-ult]

Thanks @vsavkin

We love the ONE package.json to rule them all approach and rather have one package.json instead of multiple.

The only reason why we have a separate package.json for our Express (GraphQL-server) app is that we need a small production node_modules for the graphql-server, without all angular dependencies etc.

Would be great if that could be part of the guide.

(each app lives in it's own docker container)

[kylecordes]

Single version policy:

Fantastic, great path for most projects.

Single list of packages:

This is a little harder to deal with as @the-ult suggested. I would love to see something per-app/library that limits which packages (name only, not repeating or varying the version!) the app/library can depend on. This data could also fuel a mechanism to generate a minimal production-only package.json algorithmically when needed.

(I wonder if this might be possible today by omitting package.json from implicitDependencies in nx.dev...)

[wvanderdeijl]

I totally agree with the previous suggestion of generating a minimalistic package.json for production. However, we would also need a package-lock.json to prevent the runtime production app from fetching different versions than the nx workspace.

We mainly need this to get a production app that is as small as possible since it is deployed as a Google Cloud Function

[comunitius]

I'm starting to see the point about one package.json. I've had to deal with multiple lodash versions in the past and it's a pain. Having said that, just like the previous comments, I need a small package.json in production (with the package-lock.json as well) so I can build and deploy multiple small docker images/containers in production.

[aRusenov]

I'm following this discussion and I understand that single-version policy may be an issue for some. But for minimal node server deployments (i.e. not deploying the entire node_modules) why not simply bundle the node app?

[vsavkin]

Quick update: We are working on a course that will cover things like single-version policy. Stay tuned.

[josesaranda]

I think having one package.json per application is useful if you want to publish your libraries or applications separately in the future. Is there anyone who has solved that problem?

[WangZishi]

I'm following this discussion and I understand that single-version policy may be an issue for some. But for minimal node server deployments (i.e. not deploying the entire node_modules) why not simply bundle the node app?

I think it is mainly because there still tons of issues with bundle apps using nx with webpack, nestjs, typeorm, and docker.

https://github.com/nrwl/nx/issues/1518
https://github.com/nrwl/nx/issues/1393
https://github.com/nrwl/nx/issues/1010
https://github.com/nrwl/nx/issues/803
https://github.com/nrwl/nx/issues/633

Each of them is great. But using those significant parts together is terrible.

[mderoche]

Reading through this because I have a use-case with several "sub-applications" that are lazily loaded into a single core Angular application, and they need to be developed by independent teams. I haven't seen that particular wrench thrown in this thread yet.

While the single package.json strategy works great for a single team, I am struggling with how these large companies handle the seemingly incredible cooperation costs of having multiple teams operate with a single package.json.

I am looking for an architecture where we have a core Angular application that is comprised of several sub-applications (which is now facilitated by Nx, and previously by the vanilla Angular CLI.) I wanted each sub-application to have a private package.json to define dependencies specific to that project, while keeping globally shared dependencies (such as Angular itself) in the root-level package.json.

This dramatically cuts the cooperation costs between teams; if Project A needs [email protected], and Project B needs [email protected], then Project B doesn't need to petition and wait for Project A to upgrade their application to support [email protected].

Of course, we still have the issues of "Okay, everyone upgrade to Angular N+1 on mm/dd/yyyy!", but I think having private package.jsons can remove a vast majority of related versioning issues.

This all works fine for JIT compilation. Issues arise when you try and build it with AoT.

Is there a recommendation on how to handle this type of situation, with multiple teams working on individual sub-projects under the same global project?

[demisx]

I like the idea of having a single master package.json file for enforcing versions across the entire repo, but I’m still struggling to understand how one creates a docker image for each app with the package.json being a subset from the master file? How do you derive an appropriate version of package.json per each app? 🤷🏼‍♂️ Obviously, I don’t need angular packages in my NestJs container and vice versa.

[kraiz]

I use generate-package-json-webpack-plugin to solve this problem.
But I would like to generate a yarn/npm lockfile instead to tighten the dependencies but didnt find a solution for this yet.

[demisx]

@kraiz Thank you for the tip. Do you know if this plugin can be used for Angular apps as well to weed out the backend (NestJs) dependencies? In other words, is it smart enough to parse Angular source code and build a list npm packages that app needs?

[kraiz]

You don't need it for angular because it's webpack configuration is already making sure you only get the stuff you need into your bundle files without the need for an external runtime dependency folder (node_modules).

[demisx]

@kraiz Got it. That makes sense.

[bennymeg]

I use generate-package-json-webpack-plugin to solve this problem.
But I would like to generate a yarn/npm lockfile instead to tighten the dependencies but didnt find a solution for this yet.

@kraiz ,
Is this what you are looking for?

[kraiz]

Ahh, didn't know that the lockfile can be a superset. Thx, will try that!

[MaximeBernard]

How does Google manage to update all (i can imagine they have hundred/thousands) apps at once?

From my experience, you always get:

• one app that requires too much effort/cost/time to be updated (i.e. that won't happen)
• one app that needs to do something before that (i.e. delaying the update for months/years)
• one app that requires to stay at some package version (angular users are often locked in older versions of TS, or some webpack users often rely on some webpack plugin which has not been migrated yet (or never will))

I'm pretty sure I didn't cover all the scenarios but they all lead to late migration / no migration at all.

Even though I agree it's important to encourage every app/team to stay up to date with the latest releases and use the same version, it becomes more and more complex over time until it's not even possible anymore.

One strategy here for nx could be to offer both options:

• 1 package.json (current way)
• multiple package.json (yarn workspaces way)

That being said, it's probably a lot of work to maintain both possibilities and much more work for many features.

On the other hand, keeping the yarn workspaces way and doing some checks when running nx could offer both worlds features with a limited maintenance overhead.

Whatever you decide, keep up the good work 👍

[mderoche]

@MaximeBernard Are you using Angular with multiple package.jsons? If so, how do you manage to get AoT compilation to work with multiple package.jsons?

[mikkhait]

Just following a thread and wonder if any progress has been made on this front.
We are architecting a set of microservices (NestJS, React) and would love to have a package.json for each one of them as well as have a Dockerfile for each app.

Would love to hear how others have solved that?

[MaximeBernard]

Currently, I just use yarn workspaces.

I'm missing the whole part about changes detection, deps graph and retesting/rebuilding only what's affected but I can afford to build the whole monorepo.

Also, using "paths" (e.g. https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#onpushpull_requestpaths) I should be able to smartly mitigate the issue.

[gavindoughtie]

How does Google manage to update all (i can imagine they have hundred/thousands) apps at once?

From my experience, you always get:

• one app that requires too much effort/cost/time to be updated (i.e. that won't happen)
• one app that needs to do something before that (i.e. delaying the update for months/years)
• one app that requires to stay at some package version (angular users are often locked in older versions of TS, or some webpack users often rely on some webpack plugin which has not been migrated yet (or never will))

I'm pretty sure I didn't cover all the scenarios but they all lead to late migration / no migration at all.

Even though I agree it's important to encourage every app/team to stay up to date with the latest releases and use the same version, it becomes more and more complex over time until it's not even possible anymore.

One strategy here for nx could be to offer both options:

• 1 package.json (current way)
• multiple package.json (yarn workspaces way)

It seems to me that you could have an automated package-pruning step using something like https://www.npmjs.com/package/depcheck, and generate per-deployment package.json files at will. My ideal flow would be the current nx approach while developing, but then when I create a docker image to deploy some app, it would traverse the dependencies and produce a minimal package.json for that app image.

[angelnikolov]

Has anyone found any clean solution on this? It's the only thing stopping from fully updating the monorepo approach :(
I realize the potential and benefits the single repository gives us, however installing dependencies for 10 separate projects on every docker build is getting slower and slower..

[atifsyedali]

We replaced the default @nrwl/node builder with our own webpack-based node builder, which includes generating a package.json file. It's config-compatible with the @nrwl/node builder. To use:

yarn add @apployees-nx/node

Then find-and-replace @nrwl/node in workspace.json with @apployees-nx/node.

Run a build nx build myproject. Check your dist folder for package.json.

Also, if you have a package.json in the app you are building, it will be merged.

See https://github.com/Apployees/apployees-nx/#enhanced-node-builder----apployees-nxnode

[vajahath]

I was also quite concerned (scared) about the single version policy, initially. But after some read, experiments and thought, I feel like we should follow this single version approach. As the maintainers said, this is like how Google operates and there are valid reasons for it.

But this doesn't mean we've to install every single dependency specified under this repo. Rather, we should use a single version of a package/code at a time across the repo for easy maintenance in long run (how to do it is another problem). This reduces complexity, increases code sharing and tackles many other problems like diamond dependency problem(see below).

This article is a good read. In it they say:

A single repository provides unified versioning and a single source of truth. There is no confusion about which repository hosts the authoritative version of a file. If one team wants to depend on another team's code, it can depend on it directly. The Google codebase includes a wealth of useful libraries, and the monolithic repository leads to extensive code sharing and reuse.

The Google build system5 makes it easy to include code across directories, simplifying dependency management. Changes to the dependencies of a project trigger a rebuild of the dependent code. Since all code is versioned in the same repository, there is only ever one version of the truth, and no concern about independent versioning of dependencies.

Most notably, the model allows Google to avoid the "diamond dependency" problem (see Figure 8) that occurs when A depends on B and C, both B and C depend on D, but B requires version D.1 and C requires version D.2. In most cases it is now impossible to build A. For the base library D, it can become very difficult to release a new version without causing breakage, since all its callers must be updated at the same time. Updating is difficult when the library callers are hosted in different repositories.

This feel like the article not only stresses the point of single version policy, but also adding it as a point in the definition of good monorepos.

When thinking about dependency problem in monorepo, we've to consider code sharing in the first place. Single version policy encourages it.

At the same time, we need to find a way to reduce node_modules size per project while deploying as well as developing. Thoughts?

[MaximeBernard]

Don't get me wrong, I'm deeply in favor of a single repository. This does provide a unified versioning between internal dependencies. But that does not mean you need to have a unified version for your external dependencies.

You should strive for it but the bigger your repository gets, the harder it becomes until it's not even thinkable. And I'm pretty sure Google has already hit that point.

Yarn workspaces and Lerna do just that and do it well.

I'm just suggesting nx to take that approach as well so it meets everyone's needs.

[manju-reddys]

@atifsyedali You done great job, your module (schematic) does extract what the runtime needs after prod build of an app.

[tmtron]

I am a bit confused about this. It seems that different things are mixed up in this discussion.

I'll try to sort my thoughts - please comment when I got something wrong:

Monorepo: everyone here agrees a monorepo is a good thing and this is why we use nx

Dependencies: we have 2 kinds of dependencies:

1. internal dependencies: apps and libraries in the monorepo that reference each other
   
   
   
   • with nx we can directly reference the libs. Then the nx builder will take care to (re-)build what is required
     
     
     
     
     
     • i.e. we don't need to manually find the referenced libs, run the build process and install the lib in a local (or external) npm repository
     
     
     
   
   
   • nx also allows us to enforce dependency constraints between our libs
   
   
   • we always have only up-to-date version of internal libs (i.e. no version numbers needed)
   
   

1. external dependencies: downloaded from the npm registry (when we execute npm install)
   
   
   
   • they are currently defined in a single package.json file at the project root
   
   
   • external dependencies can have different versions: see version management below
   
   

Version management
I think we have 2 groups of users here. some who want to;

1. enforce that all code in the monorepo MUST use the same version of each dependency (what nx offers now)
2. have defaults versions, but still be able to override those for some libs

External dependency constraints

The features that I am missing in the current nx implementation:

• we have no way to specify constraints for external dependencies: I'd like to constrain the use of external libraries
  i.e. currently every lib can easily import any external dependency and there's no way to forbid this.
  So a client-lib (e.g. for angular) can accidentally import a huge lib which is only meant to be used on the server-app (e.g. node-js).
• another disadvantage of the single package.json file is that the IDEs will always suggest
  all code in every lib: e.g. for auto-import, code-completion, etc.
  
  
  
  • if we had multiple package.json files (e.g. the lerna way), then the IDE will find the nearest
    
    package.json file and only suggest the "allowed" code
  
  

I must admit that I don't have too much experience with nx, npm, yarn, lerna, etc..
But ideally, we'd have one configuration to specify all external dependencies and their versions and per lib a config to specify the allowed external dependencies for this lib (version should be optional: when not set, use the default, otherwise override it).
Then maybe the package.json files can be generated from this config - or maybe we can have an EsLint rule to enforce it?

BTW: Just a hint to the some people who mentioned that they want to have multiple package.json files to reduce the production node_modules folder: you may want to check out the ncc compiler: it can build a single js file which includes your code and only the required parts of the node_modules (and you can optionally minify the output to make it even smaller)

[admir86]

I'll try to find sometime this week to author a guide on single-version policy, where I'll also cover how to add multiple package.json files if you decide that single-version policy doesn't work for you.

hey @vsavkin, any updates on this? any example how to add multiple package.json?

[tomepejo]

@admir86, you can try using yarn workspaces. That worked for me.

You can read a bit here and follow this simple example to do the setup.
Then in your Docker file, you should have something like this:

ARG APP_NAME

ENV SERVICE_DIR /usr/src/service/apps/${APP_NAME}
ENV ROOT_DIR /usr/src/service

# Copy root package json
COPY ./package.json ./yarn.lock ${ROOT_DIR}/

# Copy package json of specific service
COPY ./apps/${APP_NAME}/package.json ${SERVICE_DIR}/

WORKDIR ${SERVICE_DIR}

RUN yarn install --focus --production

The last command will install the needed dependencies only for the target application, thus avoiding huge node_modules folder inside your Docker image.

I hope this tip will be helpful for you.

[admir86]

hey @tomepejo thx. the part "Dealing With Version Conflicts" is exactly what I'm looking for. Have you already tried it within a nx workspace? I think I would missing some good features from nx if I had to migrate to a "pure" yarn workspace.

[tomepejo]

Have you already tried it within a nx workspace?

Yes, it works all together. Just give it a try, actually it's quite easy to configure.

[guibulator]

I was actually looking for a solution to a similar issue. I am currently building a single-version policy mono-repo with angular and nestjs with some azure functions created as lib that I wanted to be self-contained so that they can be deployed as the smallest package possible. I don't know why it was not yet mentioned in this thread but this solution is quite easy to use. It leverages the @nrwl/node:builder using a custom web pack config and the GeneratePackageJsonPlugin.

You can follow this example posted by Thomas-jakemeyn

[tomepejo]

@guibulator, that was my first implementation as well, however there are some downsides that lead me to try out the yarn workspaces solution.

• With the package json generate plugin, you are not aware which packages are used by which of your apps, until you trigger the build and check what was generated in the dist folder.
• It works in 90% of the cases, but if some implicit dependency wasn't automatically picked-up, you need to manually add it in the webpack configuration (mysql for example).

Having one root package.json with the common dependencies and separate one in each of your apps (with yarn workspaces) is much more obvious and clean solution. You can be always sure, which apps will be affected if you, add, remove, or update a dependency.

However, the best approach will probably be to try both solutions yourself and decide whatever works best in your case.

[METACEO]

Ahh, I was not aware of this issue/thread last week when I asked my question during the Nx Office Hours: https://youtu.be/6nTroiS3tZA?t=3199
I'll have to iterate and throw some ideas here as this is looking to be a growing issue.

[METACEO]

How in any way could this issue relate with #2483? If we're defining what specific deps affect our apps or libs then could this same definition be used to build custom package.json files?

For example, if I have the below package.json file, would an nx.json file like below help build separate package.json files or identifying runtime dependencies?

While this may require some manual defining by the developers, it would only be whenever a new app, lib, or dependency is introduced. Similar I supposed to Bazel's build files - though if the code could be statically analyzed then this could be automated?

{
  "name": "app-monorepo",
  "dependencies": {
    "@angular/common": "9.0.0",
    "@nestjs/common": "7.0.0",
    "rxjs": "6.5.0",
    "uuid": "7.0.0"
  }
}

{
  "npmScope": "app-monorepo",
  "projects": {
    "app-frontend": {
      "dependencies": {
        "package.json": {
          "dependencies": [
            "@angular/common",
            "rxjs"
          ]
        }
      }
    },
    "app-backend": {
      "dependencies": {
        "package.json": {
          "dependencies": [
            "@nestjs/common",
            "rxjs",
            "uuid"
          ]
        }
      }
    }
  }
}

[sketchnotes]

@METACEO, imho, if it's supposed to be a different file nx.json maybe it might be less redundant and a bit easier to scale up? The Gradle is not totally appropriate way to give the examples because of the different ecosystem but its way to manage the dependencies is rather elegant and scalable. Maybe there's sth to borrow...

[tmtron]

Maybe it should also support js/ts files, so that we can run arbitrary code to build the config/package.json?
This is used e.g. by EsLint and works well.

When you only have a declarative json/yml, ..whatever file, you are very limited - we can see the same problems in angular.json which just grows out of bounds, has lots of duplication and becomes unreadable and unmanageable for larger projects.

[METACEO]

Hey @sketchnotes, regarding the nx.json file I proposed above, I'm not sold on it or tied to it and I remain open to anything that helps solve the original issue (I again like the "unit" approach that Bazel takes!) If I need to throw out questionable ideas to keep the conversation going, then I just may! ;)

I am however not fully understanding your question, could you elaborate?

[westmark]

For what it's worth, I leveraged the fact that all the information you need is located in dist/nxdeps.json and put together this quick-n-dirty script

https://gist.github.com/westmark/9bd341c0bffdf0136969734da7b176e4

For an app located under dist/my-app

> tsc
> node nx-generate-deps.js --app my-app

Anyone using it would (probably) have to change the paths (root mainly). I'm quite new to NX and haven't explored much of what the CLI is capable of doing, but this works beautifully for me in any case.

One obvious downside is the lack of lock files. I have no idea, and haven't tried, if one can just copy whatever is in the project root and use it, even if it will contain extraneous package definitions.

[billyjov]

@tomepejo can you actually pls share an example how to use it in the nx context ?

[ivawzh]

We are hesitated to migrate to NX from generic Yarn workspaces set up due to this issue.
Hope in the future, there will be a solution for NX to isolate app dependencies during per-app build.
I don't think bundle will help us, because we are using NextJs with dynamic imports and many static assets.

[METACEO]

One obvious downside is the lack of lock files. I have no idea, and haven't tried, if one can just copy whatever is in the project root and use it, even if it will contain extraneous package definitions.

@westmark I agree, I'd prefer not to get into the business of parsing locks files myself!.. but what you're pointing out shows there's a lot of data in the nxdeps.json file to overlap with our package files. It'd be great for more Nrwl input here as we can look at and interpret the nxdeps.json file ourselves, but we may not have the full vision. Ultimately I think there's a lot of data and a lot of opportunities here!

[optimasquad]

@METACEO Do We have any example to follow for per app package.json file in Nx workspace?

[joshmeads]

Building upon @westmark's great work:
https://gist.github.com/joshmeads/8740641292d6d336ea3b791a18282c7b

• Fixes a change in recent NX versions where dep.target is prefixed with npm:
• Changes the CLI usage slightly.
  
  
  
  • --appFolder path/within/apps
  
  
  • --appVersion 1.0.2 - Optional, sets the package.json version
  
  
  • --rootPath . - Optional, sets the root path for the project
  
  
  • --docker true - Optional, turns on a flag to change some settings for docker build stages
  
  

Usage CLI:
npx ts-node ./tools/nx-generate-deps.ts --project ./tools/tsconfig.tools.json --appFolder api/user --appVersion 1.0.2

Usage Docker:

# Root monorepo context

WORKDIR /app
COPY ./dist/apps/$APP_FOLDER ./dist
COPY ./dist/nxdeps.json ./tools/docker/tsconfig.json ./tools/nx-generate-deps.ts ./
COPY package.json ./package-base.json
RUN npx ts-node ./nx-generate-deps.ts --project ./tsconfig.json --appFolder $APP_FOLDER --appVersion $APP_VERSION --rootPath /app --docker true

# Output with package.json will be in /app/dist for the next build stage to copy over.

All of this was built just for my own use case, but hopefully, it helps someone else too. Similar limitations around the lock files though.

[westmark]

Awesome @joshmeads! Maybe this thing deserves its own repo and npm cli package to make it easier to find.

[RogierdeRuijter]

I would like to share the (sub-optimal) solution I found to the problem of keeping the docker images small.

Angular

For Angular I used a multi-stage build to keep the image small.

NestJS

For Nest, it was more difficult because you need to keep the node_modules folder in the image, but you would like to remove all of the Angular dependencies.

To do this I used 2 techniques:

• depcheck with npm uninstall
• npm prune --production

With depcheck you can find all the dependencies that are not used by your code. When you have the list of unused dependencies you can remove them using npm uninstall. I used the following script for this

_remove-unused-deps.sh_

#!/bin/bash
depcheck > data.txt

cat data.txt 

cat data.txt | (while read F
do
    if [[ ${F:0:1} == "*" ]]; then 
        OUTPUT="$(printf "$F" | cut -c 3-)"
        foo="${foo} ${OUTPUT}"
        # echo "pruned ${OUTPUT} from node modules"
    fi
done && npm uninstall $foo);

After running this script in the docker image I run npm prune --production. To remove all the dev dependencies from the project.

This makes the images smaller, but not as small as if you would have separate package.json files.

Hope this helps someone :)

Cheers

[ivawzh]

@RogierdeRuijter very cool approach, thanks for sharing. And I have a few questions -

Comparing your remove-unused approach to @joshmeads 's pick-used approach. I am assuming the remove-unused approach can support lock file properly, am I right?

This makes the images smaller, but not as small as if you would have separate package.json files.

What is the reason of that ☝️ ?

Does depcheck support peerDependencies removal/keep properly?

---

By the way, I think type: docs is a wrong label for this issue, we want an officially supported feature to keep app size scalable. As @METACEO suggests

I'd prefer not to get into the business of parsing locks files myself!.. but what you're pointing out shows there's a lot of data in the nxdeps.json file to overlap with our package files. It'd be great for more Nrwl input here as we can look at and interpret the nxdeps.json file ourselves, but we may not have the full vision.

[RogierdeRuijter]

@ivawzh

I am assuming the remove-unused approach can support lock file properly, am I right?

Yes, this does support the lock file. You will run a normal npm install (or npm ci) command to install your node_modules. It is only after the installation of your node modules, that you will start to mess with them.

What is the reason of that ☝️ ?

I am not 100% sure why this is the case, but my best guess is that some peerDependencies are not uninstalled.

Does depcheck support peerDependencies removal/keep properly?

Well, the removal of the node modules is not done by depcheck itself. Depcheck only returns a list of unused packages in your package.json. Just to make it explicit ill give a small example:

If you have a package.json that looks something like this

{
  ...,
  "@angular/cli": "9.0.0",
  "@nestjs/core" : "6.0.0",
  ...
}

And in your nest src folder you run depcheck it will return you the following list:

["@angular/cli"]

You will then use npm uninstall to actually uninstall the dependencies that are listed by depcheck. Therefore it is 'npm uninstall' its job to remove the peerDependencies, but I do not think it is doing that fully. Maybe there is some additional clean up command that could be run after npm prune that checks for orphan peerDepedencies, but I am not sure about this.

So currently, the nestjs docker image is 450mb and I believe it used to be (before I migrated to an nx monorepo) around 250 mb.

Cheer

[Brian-McBride]

There is another project that aligns with the needs expressed here.
https://github.com/vercel/ncc

Simple CLI for compiling a Node.js module into a single file, together with all its dependencies, gcc-style.

Basically, it's using webpack to bundle as much as it can into a single file. Similar to what I see Nx doing. However, packages that it can't bundle, it creates a new node_modules with only those dependences inside. I have a team using it on one project, and so far it has been solid. It's a pretty simple set of microservices though.

[IO89]

We have recently tried to Dockerize Node app for our project, we have encountered same issue with one package.json file that would install dependencies from all other apps.

After looking at schema.json in nx/packages/node/src/builders/build/ folder we found externalDependencies is set to all which would keep external dependencies out of the bundle. Which make sense and helps to produce smaller builds.

We have decided to tweak workspace.json file in our project:
"configurations": { "production": { "optimization": true, "extractLicenses": true, "inspect": false, "fileReplacements": [ { "replace": "apps/api/src/environments/environment.ts", "with": "apps/api/src/environments/environment.prod.ts" } ], "externalDependencies": "none" } }

By running nx build <your name of node app> --prod would bundle dependencies of Node app in one file main.js. Then we would copy this build into our container and run node process on it.

[AlonMiz]

That is the main point now that blocks us from moving to nx.
We have a monorepo using lerna and yarn workspaces with multiple package.json's,
around 15 projects: backend (node.js), frontend (react) and e2e (puppeteer)

while POCing one package json, the docker image of node microservices goes from 200MB to 750MB, which is not reasonable for us.

my suggestion is to use multiple package json while restricting the single version policy
that can be achieved in several ways, i have two from the top of my head:

1. defining module resolutions per module that will override the version across all apps and libs
2. running some kind of constraint check before each install or build. (EDIT: seems like yarn v2 has the exact ability) that will find violations of using different versions across packages.

such that, we would enjoy both worlds. basically we'll have separation of concern regarding package dependencies and docker image size, and on the other hand we'll enforce the single version policy

[AlonMiz]

@vsavkin i would love to hear your thoughts here 🙏

[vordimous]

@tmtron @Brian-McBride
I implemented the ncc compiler for all of the apps in the dist folder. I can now compile runnable apps that don't need node modules installed.

vendor/index.js

const fs = require('fs');
const resolve = require('path').resolve;
const join = require('path').join;
const { exec } = require("child_process");

const APPS_DIR = '../dist/apps/';
const MAIN_FILENAME = 'main.js';
const OUTPUT_DIR = 'vendorApp';

function execShellCommand(cmd) {
  return new Promise((r, _) => {
    exec(cmd, (error, stdout, stderr) => {
      if (error) {
        console.warn(error);
      }
      r(stdout? stdout : stderr);
    });
  });
}

async function nccDirs(appsDist) {
  for (const dir of fs.readdirSync(appsDist)) {
    const appDir = join(appsDist, dir);
    const appMain = join(appDir, MAIN_FILENAME);

    if (fs.existsSync(appMain)) {
      console.log('ncc for:', appMain);
      const result = await execShellCommand(`ncc build ${appMain} -m -C -o ${join(appDir, OUTPUT_DIR)}`);
      console.log(result);
    }
  }
}

// get library path
nccDirs(resolve(__dirname, APPS_DIR));

npm scripts

    "vendor": "node vendor/index.js",
    "vendor:all": "rm -rf dist && nx run-many --target=build --all --sourceMap=false && npm run vendor",

[Brian-McBride]

@tmtron @Brian-McBride
I implemented the ncc compiler for all of the apps in the dist folder. I can now compile runnable apps that don't need node modules installed.

That looks pretty cool. I've been thinking about delving into the build scripts for @nrwl/node and seeing if I can't swap the webpack stuff for ncc. (since ncc is using webpack as well).

Your script is interesting, although are you installing ncc globally? It might be better a dep of the project, imported and run as a JS function?

[vordimous]

Your script is interesting, although are you installing ncc globally? It might be better a dep of the project, imported and run as a JS function?

I had about 10 apps and 4 libs. it took a long time and 6gb of js heap space to run as a JS dep. not sure if there is some better way to handle garbage collection, but running it as an executable runs faster and handles multiple apps on a small build server. Your mileage may very.

[alexkreidler]

Pinging some maintainers: @vsavkin @FrozenPandaz @jaysoo @bcabanes This is the 3rd most requested issue here.

Unfortunately, as a recent new user of Nx coming from Lerna, this issue was the single most confusing thing about the project.

I simply ask that you add a note to the getting started page that mentions that most users of Nx simply put all their external dependencies in the one package.json file. I'm personally fine with this, but I can see how it might be problematic for others, so figuring out how to address it seems key.

Thanks!

[AlonMiz]

Hi, we decided not to continue with nx mostly because of that issue.
But, we did implement two important libraries that helped us overcome the main downsides of not using one package.json and can help folks using nx as well.

1. One Version Rule for yarn workspaces - https://github.com/logzio/yarn-one-version-rule
2. Tool for installing a yarn workspace in a focus/isolate mode - https://github.com/Madvinking/yarn-isolate-workspace
   this tool will help anyone that uses yarn workspace installing a project that uses shared modules in an isolated way such that building the docker will be super-efficient, it will only install what you really need for that project.

[felipellrocha]

Does anyone have a good example how to make this work with yarn workspaces? What were some of the trade-offs made with that solution?

[dperetz1]

I'm considering to use something like https://github.com/KwakesProject/lerna-nx-demo - any suggestions/ drawbacks?

[aeremin]

@felipellrocha
I am currently working on mirating my pet project monorepo to the nx. In general it seems that there is nothing special about using nx together with yarn workspaces:

• You need to use nx layout with only packages folder (not apps + libs).
• You need to configure yarn workspaces as usual.
• Then you can just create package/foo folders (e.g. by using nx generate) and add a package/foo/package.json there. It will be only used to specifiy package-specific dependencies (as opposed to workspace dependencies, specified in the root package.json).
• I don't think you need lerna at all (at least if you don't need to publish). yarn will take care of dependency management and nx of everything else.

@dperetz1 This example uses lerna and doesn't use yarn workspaces. I am not sure that's a good idea TBH.

My example repo: https://github.com/sr-2020/nodejs-monorepo. master branch is non-nx, I am working on migration to nx in nx branch.

[dperetz1]

@felipellrocha
I am currently working on mirating my pet project monorepo to the nx. In general it seems that there is nothing special about using nx together with yarn workspaces:

• You need to use nx layout with only packages folder (not apps + libs).
• You need to configure yarn workspaces as usual.
• Then you can just create package/foo folders (e.g. by using nx generate) and add a package/foo/package.json there. It will be only used to specifiy package-specific dependencies (as opposed to workspace dependencies, specified in the root package.json).
• I don't think you need lerna at all (at least if you don't need to publish). yarn will take care of dependency management and nx of everything else.

@dperetz1 This example uses lerna and doesn't use yarn workspaces. I am not sure that's a good idea TBH.

My example repo: https://github.com/sr-2020/nodejs-monorepo. master branch is non-nx, I am working on migration to nx in nx branch.

seems interesting, thanks! actually my use case is multiple angular apps sharing the same libraries but I want the ability to create an independant app/lib (with breaking dependancies) or stop supporting one of the apps while changing depenancies of the shared libs - that's why I need multiple package.jsonfiles and ability to publish and manage it correctly without heavy builds of course. seems like my use case fits more to https://github.com/KwakesProject/lerna-nx-demo and your repo manages packages only, I wonder what approach would be better (lerna/yarn and how) - why do you think yarn is preferable? why you decised to integrate nrwl/nx into your repo and how does nrwl/nx schematicsworks on the workspace you created?

[Brian-McBride]

I think I just want a tool that walks the app built in the dist folder and looks for all the imports. I guess for lazy imports, it would be nice to have a file you could just enter those as well.
Then, it takes the root package.json file and creates a new package.json file in the dist directory alongside the app.

[aeremin]

@dperetz1 I am not sure what you mean by "your repo manages packages only". I have a bunch of apps and bunch of libs for them. But they are all in the same packages folder (no separate libs and apps folders). I think you can actually have those separate folders - you probably just need to have following in your package.json:

  "workspaces": [
    "apps/*",  "libs/*"
  ],

(instead of having just packages there, which is default setup for yarn).
I don't really publish my libraries to npm, so can't comment there.

I prefer yarn workspaces over lerna because:

• It seems to work better with TypeScript.
• It seems to confuse IDE/editors less (I am using WebStorm/VSCode, was struggling to make lerna-based repo always work properly in terms of error highlighting/autocompletion). Maybe it's just me. But yarn-based approach works out of the box.

nx seems to work good with my repo - I've migrated almost all packages to it without any major issues (but that's probably an offtopic here. I've enabled "Discussions" for my repo, feel free to ask questions specfic to my setup there.

@Brian-McBride
AFAIU apps in dist are webpack'ed and as such self-contained - they don't actually import anything at all, all dependencies are embedded.

[3Shain]

@dperetz1 I am not sure what you mean by "your repo manages packages only". I have a bunch of apps and bunch of libs for them. But they are all in the same packages folder (no separate libs and apps folders). I think you can actually have those separate folders - you probably just need to have following in your package.json:

  "workspaces": [
    "apps/*",  "libs/*"
  ],

(instead of having just packages there, which is default setup for yarn).
I don't really publish my libraries to npm, so can't comment there.

I prefer yarn workspaces over lerna because:

• It seems to work better with TypeScript.
• It seems to confuse IDE/editors less (I am using WebStorm/VSCode, was struggling to make lerna-based repo always work properly in terms of error highlighting/autocompletion). Maybe it's just me. But yarn-based approach works out of the box.

nx seems to work good with my repo - I've migrated almost all packages to it without any major issues (but that's probably an offtopic here. I've enabled "Discussions" for my repo, feel free to ask questions specfic to my setup there.

@Brian-McBride
AFAIU apps in dist are webpack'ed and as such self-contained - they don't actually import anything at all, all dependencies are embedded.

For node application，dependencies are not embedded.

[alejandrogkl]

NX11 have a new feature that could be what some people in the thread are looking for:

https://nx.dev/latest/angular/plugins/node/executors/build#generatepackagejson

[dperetz1]

NX11 have a new feature that could be what some people in the thread are looking for:

https://nx.dev/latest/angular/plugins/node/executors/build#generatepackagejson

seems like it maybe solves the building issues but it can't make a project indepandent of the global package.json as I understand, right?

[alejandrogkl]

what is does is:

• Generate a package.json inside the build app ( With only the project/app dependencies )
• If inside the app you already have a package.json with some other dependencies, it includes the ones that are not included

i assume the purpose is to install in the app server / container just what the app / library needs

Do you want to make a project independent of the global package to use a different version of a specific package or something else?

[dperetz1]

what is does is:

• Generate a package.json inside the build app ( With only the project/app dependencies )
• If inside the app you already have a package.json with some other dependencies, it includes the ones that are not included

i assume the purpose is to install in the app server / container just what the app / library needs

Do you want to make a project independent of the global package to use a different version of a specific package or something else?

exactly, I want the freedom to move forward with one of the workspace's packages without breaking dependant workspace's app by using specific version of the published package

[Gerrit-K]

@alejandrogkl unfortunately it seems that generatePackageJson does not respect transitive peer dependencies (similarly as generate-package-json-webpack-plugin doesn't either). E.g. if you have a shared module A with the peer dependency B and in some service you import A without importing B, the generated package.json for that service won't include B. Do you happen to know a workaround for that?

[alejandrogkl]

@Gerrit-K one workaround could be listing manually those that generatePachageJson can't include inside the package.json of the app.

For example we use sequelize and sequelize require tedious (mssql) so we list tedious manually in the package.json inside the app.

• If inside the app you already have a package.json with some other dependencies, it includes the ones that are not included and maintain the ones that you include manually

[alejandrogkl]

@dperetz1 you can do this in the global package.json:

"tedious": "^6.1.1",
"tedious_legacy": "npm:[email protected]"

in my case tedious_legacy is used by one legacy library still not migrated to the latest version

[dperetz1]

@dperetz1 you can do this in the global package.json:

"tedious": "^6.1.1",
"tedious_legacy": "npm:[email protected]"

in my case tedious_legacy is used by one legacy library still not migrated to the latest version

yes but it might be very messy once I stop supporting one project and move forward with others, it can potentially be tens of packages with multiple versions so I don't want to handle it with the global package. json but with file per project. on the other hand, I want to use other projects without the need of releasing and nx is convenient for this purpose (npm link isn't very good as I understand), of course I can manually tsconfig everything but it seems very not convenient and stable.
anyway, I got few huge projects that I want to migrate to monorepo but I want to do it the right way and it's hard to say if the proposed solutions will be easy enough to use (/is the overhead minimal?), do you have excperience with such a use case?

[pookdeveloper]

How I can execute one command for specify app ?
in my package.json I have:
"i18nextract" : "transloco-keys-manager extract",

I can't do: npm run i18nextract (not work)

If i go to the route of app
cd apps/demo-app
transloco-keys-manager extract"

Its works