Dependabot-core: Monorepo ignore local dependency not working

[yss14]

I'm currently running dependabot on a lerne-managed monorepo. This week, I added a new package cassandra-schema-builder to the monorepo which is required by the projects/backend package. Since none of the packages are released to npm yet, I received a message from dependabot

Error whilst updating type-graphql in /projects/backend/package-lock.json:
404 Not Found - GET https://registry.npmjs.org/cassandra-schema-builder - Not found

Now, I'm trying to tell dependabot to ignore this package, since lerna is managing the linking and later the updating of this dependency. However, with the following config I still get the error message above.

version: 1

update_configs:
  - package_manager: "javascript"
    directory: "./"
    update_schedule: "live"

    ignored_updates:
      - match:
          dependency_name: "cassandra-schema-builder"

    automerged_updates:
      - match:
          dependency_type: "development"
          update_type: "all"
      - match:
          dependency_type: "production"
          update_type: "semver:minor"

The repo is public and can be found here.

[greysteil]

Thanks for the heads up. Dependabot should be able to understand Lerna monorepo interdependencies - let me take a look.

[greysteil]

Ooh, actually, I might have spoken to soon here. It looks like Dependabot would need to run lerna bootstrap to make the symlinking here work, which we don't currently do. I'm going to pass this one over to @feelepxyz who knows more.

[yss14]

@greysteil Thanks for the fast clarification.
Shouldn't dependabot ignore the cassandra-schema-builder due to the config file, even if it's an interdependency?

[greysteil]

Sadly ignores don’t work like that - Dependabot has to run npm to generate an updated lockfile, and if it does so without the correct symlinking it gets that error

[feelepxyz]

@yss14 👋 I've been looking into this and we should be able to support this by running lerna link before trying to do any updates. link symlinks all local/unpublished packages into the node_modules folder so we don't get any resolvability errors.

[feelepxyz]

@yss14 looks like this is actually a lot harder than I initially thought. TL;DR: you fix this by removing the package-lock.json from your backend project.

The issue is that the package-lock.json in your backend project doesn't include the unpublished package cassandra-schema-builder. This magically works when running lerna bootstrap because lerna removes the unbpublished packages from the package.json before running npm install.

It would be a lot of work on our end to do the same thing as lerna so I'm going to suggest either removing the lockfile or using yarn as the client for your lerna project (which will use a single root level lockfile and handles unpublished packages).

Sorry for not having a better fix. Do any of these options work for you?

[yss14]

@feelepxyz Thanks for the detailed problem description! I will evaluate the options let you know what works.

[feelepxyz]

@yss14 any success? Just found out that you can also solve this issue by running lerna link convert -this will convert all unpublished packages to use file: imports instead which work with Dependabot.

The command is mentioned here: https://github.com/lerna/lerna#common-devdependencies

[chyzwar]

I am facing a similar issue. My private packages are not resolved. Additionally, dependabot does not correctly update packake-lock.json. I think dependabot need to run lerna bootstrap for both correct resolutions of local packages and updating lockfile correctly in lerna monorepo.

I will try with yarn v2 (berry) maybe this would be a better long term solution.

[greysteil]

I'm pretty sure Dependabot won't work with Yarn v2 yet (we haven't tested it or done any work on it).

[existentialism]

I've been noticing that Dependabot used on a repo that has lerna with file: imports (via lerna link convert) seems to remove all file: items from the requires block:

"@my-org/a": {
  "version": "file:packages/a",
  "requires": {
-    "@my-org/b": "file:packages/b",
     "some-package": "^1.0.0"
  }
},
"@my-org/b": {
-  "version": "file:packages/b",
+  "version": "file:packages/b"
-  "requires": {
-    "@my-org/c": "file:packages/c",
-    "@my-org/d": "file:packages/d",
-  }
}

[sethfowler]

I'm seeing the same issue; file: items are being removed from requires blocks. You'd see this situation when your repo contains lerna-managed private packages that depend on each other. @greysteil, should we open a new issue for that?

[feelepxyz]

@chyzwar @existentialism @sethfowler just back from holidays so catching up on this. Could you send over an example or reproduction where Dependabot has messed up the lockfile? Would be super helpful to be able to properly debug this. 🙏

[chyzwar]

I created a reproduction case for simple Lerna project:
https://github.com/chyzwar/dependabot-lerna

see https://github.com/chyzwar/dependabot-lerna/pull/1

[existentialism]

@feelepxyz sorry just got back from some travels, will try and get a repro for file:/require block issue up ASAP. Apologies for adding noise to your issue @chyzwar, happy to move it to a new one.

EDIT:

https://github.com/existentialism/dependabot-lerna-bug
https://github.com/existentialism/dependabot-lerna-bug/pull/1/files

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within seven days. Thank you for your contributions.

[franky47]

I have a similar issue, but my monorepo does not use Lerna, only plain yarn workspaces. All of the workspace packages are private (not published), which causes an error when attempting to resolve them from NPM:

proxy | 2020/01/22 08:33:32 GET https://registry.yarnpkg.com:443/@chiffre%2fapi-types
proxy | 2020/01/22 08:33:32 GET https://registry.yarnpkg.com:443/@chiffre%2fclient
proxy | 2020/01/22 08:33:33 404 https://registry.yarnpkg.com:443/@chiffre%2fclient
proxy | 2020/01/22 08:33:33 404 https://registry.yarnpkg.com:443/@chiffre%2fapi-types
updater | INFO <job_21202482> Requirements to unlock own
updater | INFO <job_21202482> Requirements update strategy bump_versions
updater | INFO <job_21202482> Updating typescript from 3.8.0-beta to 3.8.0-dev.20200122
proxy | 2020/01/22 08:33:34 GET https://registry.yarnpkg.com:443/@chiffre%2fapi-types
proxy | 2020/01/22 08:33:34 GET https://registry.yarnpkg.com:443/@chiffre%2fclient
proxy | 2020/01/22 08:33:34 404 https://registry.yarnpkg.com:443/@chiffre%2fapi-types
proxy | 2020/01/22 08:33:34 404 https://registry.yarnpkg.com:443/@chiffre%2fclient
proxy | 2020/01/22 08:33:34 [014] WARN: Cannot write TLS response header from mitm'd client: write tcp 172.20.1.2:1080->172.20.1.3:52622: write: broken pipe
updater | INFO <job_21202482> Handled error whilst updating typescript: private_source_authentication_failure {:source=>"registry.npmjs.org"}
updater | INFO <job_21202482> Finished job processing

The repo and the issue raised by Dependabot are public:
https://github.com/chiffre-io/chiffre/issues/6

[yevheniiminin]

I have a similar issue, but my monorepo does not use Lerna, only plain yarn workspaces. All of the workspace packages are private (not published), which causes an error when attempting to resolve them from NPM:

proxy | 2020/01/22 08:33:32 GET https://registry.yarnpkg.com:443/@chiffre%2fapi-types
proxy | 2020/01/22 08:33:32 GET https://registry.yarnpkg.com:443/@chiffre%2fclient
proxy | 2020/01/22 08:33:33 404 https://registry.yarnpkg.com:443/@chiffre%2fclient
proxy | 2020/01/22 08:33:33 404 https://registry.yarnpkg.com:443/@chiffre%2fapi-types
updater | INFO <job_21202482> Requirements to unlock own
updater | INFO <job_21202482> Requirements update strategy bump_versions
updater | INFO <job_21202482> Updating typescript from 3.8.0-beta to 3.8.0-dev.20200122
proxy | 2020/01/22 08:33:34 GET https://registry.yarnpkg.com:443/@chiffre%2fapi-types
proxy | 2020/01/22 08:33:34 GET https://registry.yarnpkg.com:443/@chiffre%2fclient
proxy | 2020/01/22 08:33:34 404 https://registry.yarnpkg.com:443/@chiffre%2fapi-types
proxy | 2020/01/22 08:33:34 404 https://registry.yarnpkg.com:443/@chiffre%2fclient
proxy | 2020/01/22 08:33:34 [014] WARN: Cannot write TLS response header from mitm'd client: write tcp 172.20.1.2:1080->172.20.1.3:52622: write: broken pipe
updater | INFO <job_21202482> Handled error whilst updating typescript: private_source_authentication_failure {:source=>"registry.npmjs.org"}
updater | INFO <job_21202482> Finished job processing

The repo and the issue raised by Dependabot are public:
chiffre-io/chiffre#6

Same here, any suggestions?

[luisfmsouza]

Any news about this? We are facing the same problem here.

[TheRealAmazonKendra]

+1 on this issue.