Ckeditor5: Provide a single changelog that is easy to parse by a human being?

[wwalc]

An idea for far far future. Not sure if the current changelog says much about what has changed in the software considering that all packages are getting new versions, even if nothing has changed in them. Considering the number of packages CKEditor 5 has, I'm not sure if anyone is able to tell what has changed from version to version. In CKEditor 4 it is much easier to say what has changed: https://github.com/ckeditor/ckeditor4/blob/major/CHANGES.md

Not sure if anyone needs such changelogs though since nobody reported it yet.

If the current format of the CKEditor 5 changelog is a major issue for you, add 👍 to this ticket.

[Reinmar]

Yup, the unfortunate regression of having everything in one version is that the main changelog is completely pointless. We need to rethink it, I've got it on my todo list already, but we didn't properly prioritized it yet.

[Reinmar]

Intro

I did a bit of a research and the changelog of a similar size that spoke best to me was https://github.com/angular/angular/blob/master/CHANGELOG.md. What I also found is that:

• Even for Angular, there are multiple formats in use (e.g. there's also https://github.com/angular/angular-cli/releases). In general, every project does this differently.
• Our releases are quite big (taken the number of items to mention) compared to other projects. That's because of the size of our project as well as their bigger length. This makes some simpler forms inapplicable to our case.
• There's a spec which tries to standardise merge commits (https://www.conventionalcommits.org/en/v1.0.0-beta.2/) and many projects (including Angular) follows it. I wondered if we could use the tools from https://github.com/conventional-changelog this time (we used some of them in the past but AFAIR we had to drop them and replace them with our own implementation because we were missing features). However,  to me the form of these commits is quite messy and I'm quite sure we'll again hit some limitations (e.g. with our two types of breaking changes or processing only merge commits). I'd rather have a tool specialised for our case since from the past experience I can see that it allows to progress faster.
• While I don't think it makes sense to use the existing tools, we can learn a lot from them. E.g. from https://commitlint.js.org/#/ or https://github.com/z0al/commitlint-bot).
• If a single change can touch multiple packages, listing them all is pointless (see https://github.com/babel/babel/blob/master/CHANGELOG.md#v790-2020-03-20 :D).

Proposal

You can find my proposal in https://gist.github.com/Reinmar/eb12358e84caae87dda862eb438089bd. It's heavily inspired by conventional changelog and Angular's changelog, but with adjustments for our standards and requirements.

Links:

• Changelog format
• Commit message format
• Additional notes
• RFC:
  
  
  
  • Should we replace type:enhancement with type:improvement? I noticed that the word "enhancement" makes problems to non-natives and is harder to type.
  
  

[wwalc]

Look good to me :+1: I also find the Angular changelog easy to read & understand.

[mlewand]

@Reinmar Looks fine in general, it's actually quite similar to what I have proposed back in https://github.com/ckeditor/ckeditor5-dev/issues/606#issuecomment-605972964

Just to clarify:

Changes within one group (Bug fixes) should be sorted by the package name and date (asc). Non-scoped changes should be listed after scoped changes. Changes scoped to ckeditor5 should be listed first.

But this still means that the bug fixes can be reordered manually before the release, right? Sometimes we ship some critical bug fixes, important for the community, that should get proper exposure.

It would be great to wrap stuff in the "Released packages" section into a <details> & <summary> tags, so that it's folded by default and unfolded after clicking it. This will make it more human-oriented.

[Reinmar]

@Reinmar Looks fine in general, it's actually quite similar to what I have proposed back in ckeditor/ckeditor5-dev#606 (comment)

Haha :) Good to see that we came to the same conclusions.

But this still means that the bug fixes can be reordered manually before the release, right? Sometimes we ship some critical bug fixes, important for the community, that should get proper exposure.

I don't think it makes sense to reorder those items as this makes the order random at that point. If something's important, I'd add it to the release highlights.

However, your question made me think that perhaps we should order those changed by the date, not the scope. By ordering by the scope, I wanted to make it easier to check what changed in a particular package. But the truth is that with multi-package changes, you need to search for a specific name across all the items anyway. So, you will need to use the Ctrl+F to look e.g. for all things that changed in the engine. And that makes sorting by scope less reasonable. An alternative option would be to split a multi-package change to multiple items (one per scope name), but this will make the changelog longer.

So, I'd start with a changelog sorted by date (which is the simplest option) and we can think how to optimize this later. WDYT?

[Reinmar]

It would be great to wrap stuff in the "Released packages" section into a <details> & <summary> tags, so that it's folded by default and unfolded after clicking it. This will make it more human-oriented.

That's a great idea :+1:

[wwalc]

However, your question made me think that perhaps we should order those changed by the date, not the scope. By ordering by the scope, I wanted to make it easier to check what changed in a particular package. But the truth is that with multi-package changes, you need to search for a specific name across all the items anyway. So, you will need to use the Ctrl+F to look e.g. for all things that changed in the engine. And that makes sorting by scope less reasonable.

The problem with this approach is that unless you place dates in the changelog, for anyone looking at it from outside it will look messy and in a random order. Having it ordered by scope makes it easier to scroll all changes and stop on the packages that I'm interested in.

[mlewand]

By ordering by the scope, I wanted to make it easier to check what changed in a particular package.

I think it does it really well, and it is beneficial.

Group by change type vs package name 

If we group entries by package name, there would be benefit that you could easily inspect the package that interest you :+1: but there are some :-1: for that: we typically touch quite a few plugins during each release - so that would result with quite a few sections.

I think that grouping by change type has this benefit of placing features at the top. And I feel like this is what interests our community the most at this stage of the project. I personally also, before updating, check for braking changes first, then feature list. Rarely I check bug list, unless I really wait for something to be fixed (in which case I typically follow github issue for that).

To sum it up I think grouping by change type, sorted by package will be fine. It make sense to use highlights to expose most valuable bug fixes. :+1:

[Reinmar]

OK... so I lost 20 minutes of writing an answer here ;\<\<\< Gotta take another 20 minutes ;/

[Reinmar]

@mlewand, ordering by the type of the change is obviously a must have and wasn't under discussion.

What you may be missing is that a single change can touch multiple packages, in which case scanning just one fragment of the changelog for e.g. engine changes will not be enough as they may be spread across a couple locations. In fact, they will be spread across multiple places regardless of that, because we split changelog into new features, bug fixes and so on anyway.

Anyway, that's how the changelog will look if we go with what I originally proposed:

* core: Change in the core 1.
* core, engine: Change in the core 2.
* core, ui: Change in the core 3.
* engine, ui: Change in the engine 1.
* engine: Change in the engine 2.
* ui: Change in the ui 1.

To look for UI changes you need to scan the whole thing anyway. 

So, there are 4 options here:

• We accept this and hope that people will use Ctrl+F. They need to scan the fixes, features and other sections anyway, so it's just easier to search.
• We split a multi-pkg item into multiple sing-pkg items when generating a changelog. This theoretically solves the issue but makes the changelog much longer and we risk having changes like "ui: Change in the engine 1.". Unfortunately, when writing a changelog entry it's hard to write it with the fact that it will be split in your mind.
• We could also ban multi-package items and just force developers to make the split manually when making a PR (create items per package). This will ensure that they are reasonably written, so no "ui: Change in the engine." items will be created. However, it's more manual work for developers.
• Finally, we could allow multi-pkg changes and still sort them alphabetically, however, with one change – assuming that the first listed scope item is where majority of the change happen. A sing PR usually touches a single pkg and optionally make some adjustments in other packages so they are less important.

All these options have their pros and cons. 3rd will give best results but may sometimes require more work from a developer. However, since unnecessary PRs to multiple repositories are to be avoided, perhaps this sort of motivation will be beneficial. Also, we don't always have to mention a change so if we made change in the engine and adjusted 3 packages to that change, we don't have to perhaps mention these 3 packages anyway.

[mlewand]

Also, we don't always have to mention a change so if we made change in the engine and adjusted 3 packages to that change, we don't have to perhaps mention these 3 packages anyway.

Just to clarify, you mean here to limit it down to a single package? What about having an option to skip the package name for these very generic changes (I don't think it will often be the case)

For instance in this iteration we picked up quite a few minor performance improvements. So far most of them are in the engine package, but what if we add couple more minor PRs like that to e.g. ui and utils packages.

It would be nice to conclude it with a generic "Improved processing editor performance. Closes #xyz, #xyz, #xyz, #xyz." note skipped package name (and as a result it should be sorted as a first item).

In general both options third and fourth sounds good to me - so the third is :+1:

[Reinmar]

Closes #xyz, #xyz, #xyz, #xyz.

BTW, I'm not sure support GH supports such notation (whether it will close all these tickets). Right now we use Closes #123. Closes #3456.

[Reinmar]

What we could do is turn the split notation (multiple "Closes") into one "Closes" when generating the changelog.

[Reinmar]

Just to clarify, you mean here to limit it down to a single package? What about having an option to skip the package name for these very generic changes (I don't think it will often be the case)

Yes, a single item in a commit message should be limited to a single package.

However, a single commit message can have multiple items.

An item may be scoped (generic) or not.

It makes sense to have ~5 items in a single commit message maximum. Above that, one generic item will usually be better.

[tomalec]

Closes #xyz, #xyz, #xyz, #xyz.

BTW, I'm not sure support GH supports such notation (whether it will close all these tickets). Right now we use Closes #123. Closes #3456.

Just tested that it closes only the first one.
https://github.com/colliding-forks/test-issue-numbers-main/commit/6223f5db0963563eaab116fd082a10e0b5ab2158

[tomalec]

I'd like to put my tiny bit to the discussion.

Precisely the use of #xyz, org/repo#xyz vs https://github.com/org/repo/issues/xyz

As you can see in my example

• https://github.com/colliding-forks/test-issue-numbers-main/
• https://github.com/tomalec/test-issue-numbers-main/

Using just #xyz introduces a quite sneaky problem. If you marge from or to a fork it may close you some issues unintentionally. If you have separate issue boards in both repos, when you merge a commit with "Closes #123 (fix a typo in README)" from repo A to repo B, the issue #123 from repo B (the editor is crashing in Safari) will be closed as well.
In my previous experience, it was really a sneaky foot gun, as we were losing the issues out of our radar, without apparent reason.

That's why personally, I prefer to use verbose links (full URL, or at least org/repo#123) in commit messages. It has a few benefits:

1. Solves the problem mentioned above,
2. Full URL allows simple click to open from any git client (not only Github one),
3. Removes cognitive burden from the dev, like: "Was this issue reported in this repo, so I'd use #123, or from other repo, then repo#123..."
4. Github shortens and prettifies them anyway so in GH you will see short descriptions. (https://github.com/colliding-forks/test-issue-numbers-main/commit/6223f5db0963563eaab116fd082a10e0b5ab2158, https://github.com/colliding-forks/test-issue-numbers-main/commit/e00a164e86b78916a87f95806da2f8f13d7c4082)

The high-level rationale behind such the approach is:
Provide as complete data as possible, it's up to the git client to prettify and simplify the log.

[jodator]

1. Github shortens and prettifies them anyway so in GH you will see short descriptions.

I'm too lazy to clone. Does it work if a commit message has a full link and Closes?

[tomalec]

You don't need to clone, just check git logs and issue boards

• https://github.com/tomalec/test-issue-numbers-main/commit/e00a164e86b78916a87f95806da2f8f13d7c4082

  Add line that closes #3 and closes tomalec/test-issue-numbers-main#4
  

  Closed #3 in the local repo, and in any repo that it is merged to.
  Closed tomalec/test-issue-numbers-main#4

• https://github.com/tomalec/test-issue-numbers-main/commit/0ca9e6cc7346683478edb9ff879d751cbebff903

  Add line that closes https://github.com/tomalec/test-issue-numbers-main/issues/5
  

  Closed https://github.com/tomalec/test-issue-numbers-main/issues/5

They are all pretty-printed by GitHub to hide unnecessary stuff.

[Reinmar]

I've got a problem with this. We agreed to use full links when referring to tickets in code comments.

To be consistent, we should do the same in PRs.

However:

• In the changelog we should use the shortened forms anyway because otherwise there will be too much noise. So when generating changelogs we have to create shortened links like [#123](https://github.com/foo/bar/issues/123).
• Sometimes we have multiple "Closes" phrases in a PR and then using full URLs may make that harder to write and validate before merging.
• I'm using an editor for markdown which replaces full links with shortened ones so it will break suggested merge commit messages anyway.
• Personally, I check git logs outside GitHub really rarely. I often open tickets mentioned in the code, but I read code often. I don't read git logs often.
• Since we offer a PR template where we can have "Closes https://github.com/ckeditor/ckeditor5/issues/" so you only have to type the num, theoretically, there will be no cost fot the developer making A PR. However, then again, my markdown editor will break this anyway.

So, I can see benefits, but I also see some issues and some of the benefits are not particularly big (due to their infrequency – reading git logs, merging across forks).

WDYT, @tomalec?

[tomalec]

In the changelog...

Totally agree, I consider the changelog a presentation of the data from git log. Shortening does not look like a complicated thing to achieve.

Sometimes we have multiple "Closes" phrases in a PR and then using full URLs may make that harder to write and validate before merging.

Personally, due to my dyslexia and poor memory, I rarely type issue number by hand, I usually paste it. Coping issue URL is even easier than copying its number. Same goes for validating, I cannot validate by just looking at the number, I'd rather open the issue (click on full URL).

I'm using an editor for markdown ...

Valid point, that workflow should work well.

Personally, I check git logs outside GitHub really rarely...

Personally, I check git logs outside my local client really rarely... ;) But I totally agree that I check the code waaay more often.

---

In general, I don't see full URLs a crucial thing, just a thing to consider. If the team is not reading the logs using local client often, we do not care about our or community forks too much, we do not have separated issue trackers for individual packages, and move towards monorepo, maybe that's not needed at all.

[Reinmar]

Thanks @tomalec. I gave it some time to "sink" and I'd still lean towards simplified notation. But we can change it if we see some recurring problems.

I'd like to close this ticket as most of the work on designing the new format was already done. The work on the changelog generator is tracked in ckeditor/ckeditor5-dev#606 and we can polish the tool in separate tickets one this lands.

Thank you all for fruitful brainstorming! I'm sure the new format will be much better than what we have now :)