Renovate: Improve changelog detection for monorepos

[rarkins]

The original data model for changelog was to assume a 1:1 mapping between package and source repository.

For monorepos, there are multiple challenges:

• Nested changelog.md files (#2162)
• GitHub releases are sometimes done using raw version (e.g. v6.0.0) and sometimes with name+version, e.g. [email protected]

So our challenge is:

1. Find the right GitHub Release or changelog file for the package
2. De-duplicate them in PR bodies

Also related: what to do if a changelog.md file is updated (e.g. improved) in master branch well after publication of the package? Most of the time we'd want to update the PRs with that new info, but perhaps not always.

• If it is a github repo and then look for a "Release" named [email protected], [email protected], v1.0.0 and 1.0.0
• If there is a changelog file in the root of the repo, use it
• If there is a file like ........./package/changelog.md, then use it (i.e. changelog nested)

Then the challenge is how to de-duplicate it within the PR body. If each package has identical from/to versions and identical content, then we should just display it once. But what if it's a mix of shared and non-shared?

[rarkins]

In they the release notes for manager-package-release should always be the same. But the raw commit "compare" link can be different, namely whether the user is comparing it to the previous stable version or the previous unstable version.

Each "release" should have a "gitRef", whether that's a tag (e.g. v1.0.1) or a raw commit hash. Then the exact compare links can be generated per-PR.

[Flydiverny]

Having just found this issue after trying to get changelogs working for our monorepo packages, I take it there's no workaround at the moment? Using lerna with individual versioning and have one CHANGELOG.md in each package and also tried to update the release tags (@scope/[email protected]) with changelog entries with no success. Found it curious that changelogs wouldn't be picked up while compare links are using the correct tags and versions as well as showing all the links for each release.

[rarkins]

Can you replicate it in a public repo or find a public lerna repo that uses the same approach you do, that can be tested against?

[Flydiverny]

Not sure if this is something thats interesting or not but figured I'd share it here:
https://github.com/npm/rfcs/blob/latest/implemented/0010-monorepo-subdirectory-declaration.md

Suppose this could potentially help with detecting changelog files for monorepo packages? :)

[rarkins]

Thanks, I've seen that one and partially implemented it already. We already do a search for subdirectories containing the package name so that captures nearly all the use cases already anyway.

[felixfbecker]

Even if a changelog could not be detected, it would be nice if Renovate could add a "Compare source" link that links to comparison page of that subfolder so I can inspect the commits

[rarkins]

@felixfbecker I've never seen a subfolder compare link before. Do you have an example for GitHub?

[felixfbecker]

Hmm, doesn't seem to exit in the UI. Maybe a link to the history page of the folder then? https://github.com/renovatebot/renovate/commits/master/docs

[rarkins]

OK so if source directory is defined in package.json but no CHANGELOG file is found within that directory, then link to the directory's commit history?

[Flydiverny]

Thanks, I've seen that one and partially implemented it already. We already do a search for subdirectories containing the package name so that captures nearly all the use cases already anyway.

Does this mean that if I have a monorepo with a package name banana
and these files

- ..stuff in root..
- packages/banana/package.json
- packages/banana/CHANGELOG.md

It would detect the changelog and present it already?

I suppose this doesn't work for scoped packages where I would have eg @flydiverny/banana as package name but same file structure as above.

Is there anything I can set in package.json files to make it detect nested changelogs today?

[rarkins]

Can you describe your situation in full? Eg self hosted or GitHub.com? Directory structure, etc

[Flydiverny]

Its on github. So using this as an example:
monorepo with lerna + yarn

packages/renovate-bug-dep contains a package with name @flydiverny/renovate-bug-dep.
There I have a nested CHANGELOG.md.

For version 1.1.4 of the package there's also a github release tagged as @flydiverny/[email protected]

Here's a PR from renovate trying to update this dependency: https://github.com/Flydiverny/renovate-workspace/pull/2

Don't mind the repo and package names :)

In our project its similar with the main difference being private github repo and private packages on npm.

[rarkins]

@Flydiverny thanks for the example! That's exactly what I needed. Can I assume that the GitHub Releases entry and CHANGELOG.md entries should always be identical, i.e. we can use whichever one works best for our internal logic?

[Flydiverny]

@rarkins Yes they are supposed to match :) using lerna to generate both of them here as they recently added a --github-release flag.

The github release requires the end user to specify github token tho so its a bit more of a hassle to use. But I personally prefer github releases anyway.

[rarkins]

Alright thanks, I think we have enough info to fix it!

[rarkins]

Next steps: debug changelog retrieval using the reproduction repos from @Flydiverny here: https://github.com/renovatebot/renovate/issues/2926#issuecomment-471355772

Either fix/improve our existing release parsing (we should be able to locate the package/release combination) or locate the nested CHANGELOG.md files

[koba04]

Is there any update on this?
I have the same issue.

We use yarn workspaces + lerna for a monorepo and publish a new package using lerna publish.
As a result, the CHANGELOG.md is placed in each package and doesn't exist in the root directory on the repository.

Should I put CHANGELOG.md that aggregates all changelog for packages in the root directory?
Or should I put the changes in the GitHub releases?

The following is the monorepo.
https://github.com/kintone/js-sdk

The following is a PR to update the package version by Renovate.
https://github.com/koba04/gatsby-source-kintone/pull/13

Thanks!

[rarkins]

@koba04 No, you shouldn't need to change the way your do the individual CHANGELOG files, Renovate should be updated to accommodate/detect that.

[rarkins]

@koba04 actually, you do need to make some changes though. Currently your monorepo is not complying with npm's package.json recommendation:

Notably here: https://github.com/kintone/js-sdk/blob/edc5d0cd75e577b90d22c0234bbbcfaa5263a946/packages/rest-api-client/package.json#L37-L39

[koba04]

@rarkins Thank you for your help! I'll fixed it.

[viceice]

Always seeing duplicate version entries for traefik