Nx: Removing implicit dependencies and generating package.json for each project

[atifsyedali]

After using NX, I have a few suggestions for medium-large projects adopting NX. Collectively, they will solve the following issues: #1169, #1777, #602 and probably some others as well.

Affected command and implicit dependencies on workspace.json, nx.json, and package.json

From #1169:

If we introduce a new library every time that we add a new component, our implicit dependencies (angular.json, nx.json, and tsconfig.json) will be updated every time, and therefore all apps are considered affected.

This is a real problem that is affecting medium to large projects. One of the primary motivations to moving to a mono-repo (and hence move to using nx) is to figure out what we need to build and deploy.

I agree with having separate libs (as opposed to doing file-level dependency analysis to figure out what to build), but ultimately any changes to the more freqently changing files -- specifically the angular.json/workspace.json, nx.json, and package.json -- reduces the utility of the "affected" commands.

angular/workspace.json and nx.json

The primary issue here is adding new projects. But sometimes updating existing projects also leads to the same issue.

Proposed solution

1. Have a project.json in every app/lib folder, and this project.json would contain just the parts from angular.json/workspace.json + nx.json that are relevant to that project.

2. An explicit command called nx update-projects (which is also implicity run before any nx command), generate the root angular.json/workspace.json file + nx.json file from all the projects involved (recursive find of all project.json in the apps and libs folders **)

3. Have a top-level property called "generated" in angular.json/workspace.json and nx.json, which would indicate if those files are generated, and subsequently ignore them if they are during affected calculations.

---

** To know the names of all folders that contain apps and libs (even though today in some places in the code it is hardcoded), we can look at the root package.json for an "nx" property (e.g. nx: {appsFolder: "apps", libsFolder: "libs"}).

---

Root package.json

The primary issue here is adding a new external dependency to be used by one of the projects. We now have to build and deploy everything because affected command says so.

Proposed solution

The proposed solution below actually solves two problems: (1) affected command only shows those projects that are affected by the root package.json, (2) a package.json is generated in the build output of all projects, which helps further Dockerization / deployment (See #1777).

1. During the scan of source code to build a dependency graph for the affected command, nx only checks for imports/requires of mono-repo's npmScope. Instead, have it maintain a dependency graph of all imports, including external dependencies.

2. Maintain a subset of dependencies/devDependencies/peer/optional per project in the project's own nxdeps.json file (just like the cache in the output folder). This file would be updated after a call to nx update-projects. The purpose of this nxdeps.json file will be clear in (3b-ii) below.

3. Now, the affected command can be divided into two steps:

   3a) Like before, using the base and head (or affectedFiles), figure out which files were changed in which projects, and then check the dependency graph for all inter-project dependencies (i.e. any that starts with npmScope in the dependency graph).

   3b-i) If base and head were supplied, look at the two versions of package.json and check which dependencies in the dependencies/devDependencies/peer/optional have changed. Compare these with all non-project dependencies in the dependency graph.

   3b-ii) If affectedFiles was supplied instead, then lookup the current values of each non-project import from the dependency graph in the root package.json file, and compare these values (additions, removals, changes in version) with those in nxdeps.json for each project. *

4. For the build output, generate a package.json for each project using:

   4a-i) Any package.json in the project-root of each project as the base (if it exists -- some users already have it).

   4a-ii) If package.json does not exists in each project, then generate one using all the fields in the pacakge.json in the root of the monorepo ... but do not use the dependencies/devDep/peer/optional. Use the project name as the name field.

   4b) Now using the dependency graph, as well as the externalDependencies configuration option, figure out which dependencies are really external, look up their version value in the root package.json.

   4c) Use (4a) and (4b) above to create a package.json in the build output folder.

   4d) Also, copy over the yarn.lock and package-lock.json in the root to the output folder if they exist.

---

* If no nxdeps.json was supplied, but affectedFiles was used, give a warning to the user.

---

Update existing nx.json and angular/workspace.json

Finally, remove the following files from nx.json's implicit dependencies: package.json, workspace.json/angular.json, nx.json.

---

Importing publishable libraries into others

The generation of package.json for each project using the above techniques presents an opportunity to hoist individual projects and libraries to publish them to npm/registry. That is, imagine if you could also specify externalLibraries in addition to externalDependencies (or simply are able to detect which libraries are publishable). Then you exclude bundling these libraries during a build, and instead keep their @npmScope/library-name imports intact in the output (i.e. whitelist these imports in nodeExternals).

[FrozenPandaz]

The points and solutions that you outlined are great! Thank you for outlining your thoughts.

We are currently already working on being smarter about how we process changes to package.json. We will be drawing dependencies between projects to node_modules allowing you to update a dependency in package.json and only affect projects which import from that module.

This feature will also come with the infrastructure to be able to parse changes to JSON thus opening up more intelligent processing of other JSON files such as workspace.json, tslint.json, nx.json, .etc!

This is a lot of work and we will be working on improving this during the v10 timeline.

[FrozenPandaz]

Re: points and solutions

Separate project.json files

This would be a much welcomed way of configuring projects. angular.json and workspace.json can be quite large. Generating the workspace.json could work but it would be much better done where the root workspace.json can contain references to other project.json files. We can discuss this with the Angular CLI team to see if they have plans to support this.

The hope with the above changes is that we will be able to intelligently parse the changes one makes to the workspace.json thus not affecting all projects.

Generating package.json

Generating package.json files based off of the dependency graph is an interesting idea. I think the complexity is that even with the full lockfile copied over. Having a different set of dependencies can still result in different versions of installed projects because of scope hoisting. Ultimately it's not much better than writing your own package.json if you need to dockerize the runtime application. There's also a slight distinction between what is needed at runtime of a bundled application vs what the application depends on according the imports in its code. It's a complex problem and for now I think it's better to manually author the package.json files.

These are just my thoughts and I could very well be wrong :).

[studds]

I've written a little script called monorepo-package-tool that copies dependencies from a root package.json to "child" package.json based on imports. It's intended for publishing to npm, rather than dockerizing. (I webpack my code with dependencies for deployment).

It's fairly basic, but it's scratching my itch for the mo.

[atifsyedali]

Thanks @studds -- checking it out. I also have a couple of builders here https://github.com/Apployees/apployees-nx that will traverse the imports and output a package.json in the dist folder for that app/lib. If you have implicit dependencies you can create a partial package.json in each app/lib and it will take precedence over the versions defined in root package.json for specific dependencies of the app/lib.

[jorgekleeen]

I know is not directly related to this, but something I find difficult right now is to know after a while which dependencies in monorepo's package.json are being used in each specific project. What is the recommended way to do make this kind of analysis?

I find this information relevant when you want to remove an old legacy library, and then you want to know which dependencies were specific to that project, so you can get rid of those dependencies.

[FrozenPandaz]

Hi, sorry about this.

This was mislabeled as stale. We are testing ways to mark _not reproducible_ issues as stale so that we can focus on actionable items but our initial experiment was too broad and unintentionally labeled this issue as stale.