Dependabot-core: Support Yarn v2

[eps1lon]

While yarn v2 is still in development alphas are already released and documented: https://yarnpkg.github.io/berry/

Since I'm a big fan of dependabot and yarn v2 I would like those two to get along better.

For the default configuration of yarn v2 dependabot already updates versions correctly. However, it does not update the PnP file of yarn. This means that in a fresh clone of a project that cannot enable zero-install you create a diff by simply running yarn because that will update the pnp.js. I don't know any dev history of dependabot with regards to yarn but it might make more sense for dependabot to "just" run yarn up which should cover package.json, yarn.lock and .pnp.js.

In addition to that a nice enhancement would be to run yarn cache clean for those who have the offline mirror checked into version control.

I solved both of these issues for me by letting a github action cleanup after dependabot by running yarn and yarn cache clean but that does mean dependabot can't make changes to the PR anymore.

A hardcoded yarn.lock filename might also be problematic in the future since the lockfile name is configurable in yarn v2.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within seven days. Thank you for your contributions.

[eps1lon]

Should I spam "Any updates" to get attention or how else can I assist?

[jonathangiamp]

Any updates ? 😄
Some projects start to migrate his yarn from v1 to v2, a good compatibility will be more and more requested.

[rebelagentm]

Hi! 👋 We're still pretty swamped integrating Dependabot into GitHub, so we haven't yet gotten to this.

[mormahr]

Now that yarn v2 is out, and pnp is enabled by default, maybe the prioritization of this issue should be revisited. We’re in the process of switching to yarn v2 / pnp, but will for the time being not include the pnp file in the repo, because it’ll cause problems with dependabot. We want to change this to use zero-installs, though.

[anthonator]

Agreed that prioritization on this should be revisited. Dependabot is pretty much broken for anyone using Yarn v2 / pnp.

[anthonator]

Is there anything the community can do to help push this along?

[eps1lon]

I push uncommited changes after yarn install on dependabot branches. This means that you have to recreate PRs instead of letting dependabot rebase them. But this isn't much of an issue for me personally.

Using azure pipelines:

trigger:
  - master

pool:
  name: 'Hosted Ubuntu 1604'
  vmImage: 'ubuntu-latest'

steps:
  - checkout: self
    clean: true
    persistCredentials: true

  - task: NodeTool@0
    inputs:
      versionSpec: '10.16.x'
    displayName: 'Install Node.js'

  - script: |
      yarn install
    displayName: 'Install packages'
  - script: |
      git config --global user.email "[email protected]"
      git config --global user.name "eps1lon[bot]"
      git add -A
      git status
      git diff-index --quiet HEAD || (git commit  --message 'yarn autofix' && git push -u origin HEAD:$(System.PullRequest.SourceBranch))
    # should test the actor but Build.RequestedFor does not point to dependabot but Microsoft.VisualStudio-something
    condition: and(succeeded(), startsWith(variables['System.PullRequest.SourceBranch'], 'dependabot/'))
    displayName: 'Autofix yarn for dependabot'

You could also include dependency deduplication or other autofixes in here.

[TheGrimSilence]

Same problem here. Working on Yarnberry Cookbook and dependabot breaks yarn.lock. Opening the generated PRs and running yarn throws YAMLException: end of the stream or a document separator is expected at ...:. So I'm guessing D'bot needs to know Yarn 2. As @eps1lon said,

A hardcoded yarn.lock filename might also be problematic in the future since the lockfile name is configurable in yarn v2.

My guess, maybe there should be a lockfile option or detect the configuration from .yarnrc.yml which might be smarter. Another issue is workspaces, it's also breaking my templates. I'm working on a similar approach as @eps1lon with Actions.

[koistya]

I see that Dependabot can resolve updates for Yarn v2 repo, but chokes on local dependencies such as "workspace:*".

See Node.js API Starter Kit (Yarn v2 based monorepo), kriasoft/nodejs-api-starter#215.

[AArnott]

Since Dependabot apparently doesn't work at all in our yarn v2 "Zero Install" repo, I wrote a GitHub Actions workflow that basically does what Dependabot did, but with just one PR for all updates:

name: Update dependencies

on:
  schedule:
  - cron: '0 2 * * *'
  workflow_dispatch:

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - run: yarn up '*'

    - name: Create pull request
      uses: peter-evans/[email protected]
      with:
        commit-message: Update all dependencies
        branch: automatic_dependency_updates
        title: Update all dependencies
        body: An updated update of all NPM dependencies.
        #labels: auto-merge
        #reviewers: # optional

[cometkim]

I'm using a similar as @AArnott does, with a simple plugin for using yarn up with more controls

https://github.com/cometkim/yarn-plugin-bump

[ylemkimon]

Similar to https://github.com/dependabot/dependabot-core/issues/1297#issuecomment-621458459, I'm using a GitHub Actions workflow to fix and update PRs created by Dependabot. This way we can still benefit from its version update logic and release notes.

name: Dependabot

on:
  push:
    branches: [ dependabot/npm_and_yarn/** ]

jobs:
  build:
    runs-on: ubuntu-latest
    if: github.actor == 'dependabot[bot]'

    steps:
    - uses: actions/checkout@v2
      with:
        fetch-depth: 2
        persist-credentials: false # minimize exposure
    - name: Use Node.js 12.x
      uses: actions/setup-node@v1
      with:
        node-version: '12'
    - name: Autofix lockfile
      run: |
        # change directory
        # assuming Angular commit style (build: bump XXX from AAA to BBB in YYY)
        # use $8 for default commit message style (Bump XXX from AAA to BBB in YYY)
        cd .`git log -1 --pretty=%s | awk '{ print $9 }'`

        # restore yarn.lock from the previous commit
        git checkout HEAD^ -- yarn.lock

        # install yarn-plugin-deduplicate
        yarn plugin import https://raw.githubusercontent.com/eps1lon/yarn-plugin-deduplicate/latest/bin/%40yarnpkg/plugin-deduplicate.js

        # if package.json was not updated, upgrade the dependency
        # assuming Angular commit style (build: bump XXX from ...)
        # use $2 for default commit message style (Bump XXX from ...)
        git diff --name-only HEAD^ HEAD | grep -q 'package.json' || yarn up `git log -1 --pretty=%s | awk '{ print $3 }'`

        # restore package.json from the last commit
        git checkout HEAD -- package.json

        yarn install

        # deduplicate lockfile
        yarn deduplicate
      env:
        YARN_ENABLE_SCRIPTS: 0 # disable postinstall scripts
    - name: Config Git
      run: |
        # use personal access token to allow triggering new workflow
        BASIC_AUTH=$(echo -n "x-access-token:${{ secrets.GH_TOKEN }}" | base64)
        echo "::add-mask::$BASIC_AUTH"
        git config --global user.name '${{ github.event.commits[0].author.name }}'
        git config --global user.email '${{ github.event.commits[0].author.email }}'
        git config --local http.$GITHUB_SERVER_URL/.extraheader "AUTHORIZATION: basic $BASIC_AUTH"
    - name: Commit changes
      run: |
        cd .`git log -1 --pretty=%s | awk '{ print $9 }'` # ditto
        git add yarn.lock .yarn/cache .pnp.* # only add yarn.lock if not using zero-installs
        git commit -m "Dependabot autofix"
        git push

EDIT (Aug. 13): Updated to support subdirectories. The test repo is available at https://github.com/ylemkimon/berry-dependabot-test.
EDIT (Aug. 19): Updated to use personal access token to allow triggering new workflow. Moved Git credentials setup after dependencies install to limit (not _remove_) exposure.

[paul-soporan]

Since Dependabot apparently doesn't work at all in our yarn v2 "Zero Install" repo, I wrote a GitHub Actions workflow that basically does what Dependabot did, but with just one PR for all updates:

name: Update dependencies

on:
  schedule:
  - cron: '0 2 * * *'
  workflow_dispatch:

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - run: yarn up '*'

    - name: Create pull request
      uses: peter-evans/[email protected]
      with:
        commit-message: Update all dependencies
        branch: automatic_dependency_updates
        title: Update all dependencies
        body: An updated update of all NPM dependencies.
        #labels: auto-merge
        #reviewers: # optional

FYI: I'd recommend not using this in production (at least for now). yarn up '*' doesn't really do what most people think it does. The way it currently works is that * matches all of the dependencies stored inside the project, and each one of them is then forwarded with range unknown to our suggestUtils.getSuggestedDescriptors function that resolves the unknown range into the latest npm version of that dependency. This means that yarn up '*' will turn all non-npm dependencies into npm dependencies, even when it shouldn't (just like yarn up utils will turn utils into an npm dependency, no matter if it is a git dependency / a portal dependency / whatever else).

Issue to track inside the Yarn 2 repo: https://github.com/yarnpkg/berry/issues/1492.

[AArnott]

Thanks, @paul-soporan. In my case, I only ever use npm dependencies (honestly, I didn't even know git dependencies existed). In one repo I also use a mono-repo where my package.json includes dependencies on other directories in the repo. I guess I should test how those behave in the yarn up '*' world.

[bartocc]

@rebelagentm Would it be possible to have an update on this issue?

[rebelagentm]

@bartocc 👋 I'm no longer working on Dependabot. @feelepxyz, is there an update to provide on this?

[feelepxyz]

@bartocc we started looking into supporting this recently but paused it as it turned out to be a significant change for how Dependabot works. We'll need to start cloning the entire repo instead of just fetching the main manifest files to support the new Plug'n'play/offline cache features. It's going to be a few months out at least as we're focusing on migrating Dependabot Preview features to the GitHub-native version.

[bartocc]

Ok, this is bad news 😢, but at least we know what to expect in the next coming month.

Thx @rebelagentm for forwarding to @feelepxyz 👍

@feelepxyz would you mind posting here some updates as the work on this advances 🚀 or stays idle 🥶 ?

[Milo123459]

I saw that it'd be a couple months at least before it came out, is this feature on a roadmap? Is there a roadmap?

[feelepxyz]

I saw that it'd be a couple months at least before it came out, is this feature on a roadmap? Is there a roadmap?

Yes! We're focusing the next few months on upgrading all ecosystems in dependabot to the latest version.

[juanpicado]

Does it also include #1736 ? (pnpm support)?

[Milo123459]

Wow! Can't wait! Also, is there a way for me to unignore a dependency, before I knew you could disable it I removed them all 😂

[jurre]

Does it also include #1736 ? (pnpm support)?

We've currently paused adding new ecosystems, so not in the foreseeable future at least. More context is here:
https://github.com/dependabot/dependabot-core/blob/main/CONTRIBUTING.md#contributing-new-ecosystems

[jurre]

Wow! Can't wait! Also, is there a way for me to unignore a dependency, before I knew you could disable it I removed them all 😂

I think this should cover it: https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/configuration-options-for-dependency-updates#ignore

[Milo123459]

Wow! Can't wait! Also, is there a way for me to unignore a dependency, before I knew you could disable it I removed them all 😂

I think this should cover it: https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/configuration-options-for-dependency-updates#ignore

Thank you, found out you had to reopen the PRs, maybe a way configure ignored dependencies in the dependabot section in insights?

[kiancross]

@feelepxyz Is there anything the community can do to help speed this up?