Pcl: [Release] One click release

[kunaltyagi]

Is your feature request related to a problem? Please describe.

Releasing PCL requires effort from multiple contributors to add the binaries for different platforms. That takes time, and is difficult to trust

Expected behavior

1. Release PR is merged to master
2. Maintainer runs a pipeline and it performs a release without requiring any input (except maybe the release number)

Describe the solution you'd like

We can use the CI to generate the binaries (and source code), and the SHA2 and MD5 checksums for each artifact for different platforms. The artifacts can then be added to the GitHub release page using the API (or a CI shortcut)

TODO list:

• [x] Create CI for compiling as Debian does for release (#3995)
• [x] Create CI for compiling perception_pcl, a large open-source downstream user of PCL (#3995)
• [x] Generate source code archives with SHA256 and SHA512 checksums (#4073)
• [x] Generate release on the release page (#4073)
• [ ] Sign binaries using PGP key (needs org PGP key)
• [ ] Sign MacOS binaries properly (tracked by #3714)
• [ ] Creating Windows installers
• [ ] Creating Linux flatpack
• [ ] Creating Mac package

Describe alternatives you've considered

Manual release is not feasible in the long term. Considered and discarded.

[kunaltyagi]

Should we be creating the independent binaries as well for Linux and Windows?

[SergioRAgostinho]

This would be amazing :heart: In personal order of priority:

1. Creating the Release on the release page
2. Windows Installer/Uninstaller
3. Linux flatpack
4. Mac package (aka installer/uninstaller)

[kunaltyagi]

Request for @PointCloudLibrary/maintainers

Can we get the steps used to create the releases as bash or powershell scripts?

[taketwo]

No Bash script this time, sorry. From my side the process always involved two PRs with a tag in between:

• 1st PR:
  
  
  
  • Bump version to x.x.x in CMakeLists.txt
  
  
  • Find replace all occurrences of x.x.x.99 CMake requirements in the tutorials and examples
  
  
  • Bump version off the release badge in README.md
  
  
  • Finalize changelog, fix the release date, commit
  
  
• Tag
• 2nd PR:
  
  
  
  • Bump version to x.x.x-dev in CMakeLists.txt
  
  

[SergioRAgostinho]

I'm not entirely sure if it's possible but let's go step-by-step here. This is our usual list of steps.

Task list

• [ ] Go through all PR for the milestone - make a check if there are open items for this milestone. python stuff
• [ ] Update Changelog - this is already done with every merge to master but it should be displayed once more to the maintainer before proceeding
• [ ] Start a release branch - I believe this actually just happens if we find that we broke something unintentionally. So in a sense, it only happens after the next step in this list.
• [ ] (If needed) Revert whatever unintentional API/ABI breakage that might have occurred in the release branch - run abi checker. It only needs to enforce that there are no issues between patch version updates. It should present to the maintainer the reports for all modules so that he/she could skim through and confirm everything is as expected.

Everything else from here onward can be automated

• [ ] Bump version to x.x.x in CMakeLists.txt
• [ ] Find replace all occurrences of x.x.x.99 CMake requirements in the tutorials and examples
• [ ] Try to build and test on all relevant platforms
• [ ] Bump version off the release badge in README.md
• [ ] Finalize changelog and fix the release date
• [ ] Tag
• [ ] Remove the release branch
• [ ] Add binaries to the GitHub release page
• [ ] Send a PR to vcpkg to update package to the new version
• [ ] Bump version to x.x.x-dev in CMakeLists.txt
• [ ] Send announcements to the mailing lists - push a new post to the website announcing new release.

That being said, PCL is becoming more and more a devops wet dream.

[kunaltyagi]

• API/ABI testing is useful for minor releases only. For the rest, it's not needed right now.
• #4058 removes tutorials as dependent on version number. The only places with 1.10 needs to be changed are CMakeLists.txt and README.md
  grep --exclude-dir=build '[^0-9a-zA-Z.]1\.10[^0-9]' * -nr
• Is the "Try to build and test on all platforms" needed since we have CI? It would complicate the release yaml file (Side-effects include: no release possible if Apple CI starts barfing errors on it's regular schedule)

[kunaltyagi]

Part 1 Update: To release

new_verion=<INPUT_FROM_SOMEWHERE>
sed -i "s,[0-9]\+\.[0-9]\+\.[0-9]\+,${new_version}," README.md
sed -i "s,VERSION [0-9.]*),VERSION ${new_version})," CMakeLists.txt

Part 2 Update: Post release

new_version=<INPUT_FROM_SOMEWHERE>.99
sed -i "s,VERSION [0-9.]*),VERSION ${new_version})," CMakeLists.txt

Questions:

• how to create these 2 commits using the release CI?
• does the CI wait to verify these 2 commits?

Once this is figured out, we'll have the achieved almost all milestones for a 1 click release

[kunaltyagi]

Will it be ok to create and merge the PR using GitHub API without waiting for CI result?

[UnaNancyOwen]

I'll share my notes that steps to generate PCL All-in-one Installer.
NOTE: It contains processes that have not yet been automated.
NOTE; It is written in Japanese. Please use auto translation tool like Google Translate or DeepL.
https://gist.github.com/UnaNancyOwen/f6025cf155acb8f444374c1c25530f1f

[kunaltyagi]

Is it possible to automate the setup? I noticed a lot of manual interruptions and restarts in the process you lined to.

[UnaNancyOwen]

Is it possible to automate the setup?

I've wanted do it for a long time, but I'm not sure if this possible to automated.

[kunaltyagi]

I'm not sure if this possible to automated

残念ね. What about the pdb zip files?

I saw that NSIS has a slack, it might help to ask around if there's some way to only perform a few steps. If not, maybe a future version will have the features.

[UnaNancyOwen]

What about the pdb zip files?

Currently, I'm manually collect the pdb files for compressing them into an archive.
However, I think it would be easy to automate.

[OgreTransporter]

I saw that NSIS has a slack, it might help to ask around if there's some way to only perform a few steps. If not, maybe a future version will have the features.

I'm not a fan of NSIS. The packages cannot be unpacked easily and it is complicated to perform a distributed installation e.g. via group policies. For the installation, admin rights are usually required. In managed domains not everyone has these rights. I'm more for the Windows Installer (MSI). The packages can be built with WiX, for example. If NSIS is unavoidable, it might be useful to create the packages as ZIP files in parallel.

[kunaltyagi]

Hi @OgreTransporter! Most of the terms sound quite foreign to a non-Windows user. Having said that, if you can create a process for using WiX to create MSI binary, we can work out how to integrate the binary generation and post-installation testing in the release pipeline.

IMO, the technology (NSIS or WiX) isn't as important as its integration in CI. If the end result is the same, I don't think even the users will have anything to complain.

[OgreTransporter]

Giving admin rights to all users in larger networks is quite dangerous, regardless of the operating system. From Windows on your home computer you are the only user who is used to have admin rights. But if you work in a company or university network, for example, not all users have these rights. Admin rights are required to install software, unless it is a self-extracting archive. If you then want to install PCL to a larger group of people, you can either do this manually on each computer or via a server in the system. To do this, the installation must either be an MSI package or support an "unattended installation". MSI is better, because you can install patches and updates more easily. The alternative are simple ZIP archives, which can be unpacked by any user - even without admin rights. As a second alternative the users can build the PCL themselves - which I do.

If you have NSIS integrated, this is OK. It's probably not that much effort to additionally pack the files as a ZIP file again, is it? But you can also save the effort for NSIS and Wix and just offer ZIP files, that shouldn't be a problem either.

[UnaNancyOwen]

The alternative are simple ZIP archives, which can be unpacked by any user - even without admin rights.

FYI, vcpkg can be exported package to a zip file by vcpkg export pcl --zip command.
https://devblogs.microsoft.com/cppblog/vcpkg-introducing-export-command/

[kunaltyagi]

If you have NSIS integrated, this is OK

We don't have NSIS integrated. The installers are made manually and tested manually by @UnaNancyOwen

[kunaltyagi]

Release pipeline works, tested for 1.11.1-rc1

Please contribute code to release the following:

• Docker image based on Debian (Difficulty: easy)
• Deb package (Difficulty: easy-medium)
• Windows installer/binaries (Difficulty: unknown)
• MacOS binaries, unsigned (Difficulty: easy)

[kunaltyagi]

Moving the target for rest of steps to 1.12

[OgreTransporter]

Windows installer/binaries (Difficulty: unknown)

Just compress the build artifacts in a zip archiv ;-)

FYI, vcpkg can be exported package to a zip file by vcpkg export pcl --zip command.
https://devblogs.microsoft.com/cppblog/vcpkg-introducing-export-command/

This is correct, but vcpkg means that PCL must be built with all dependencies first.

[kunaltyagi]

I'm not a Windows user, hence I wrote unknown. I've seen issues on Linux/Darwin when the artifacts aren't compiled as static.

I'd encourage you to submit a PR to the release script to do exactly that if possible since you'd be able to understand the error messages unlike me who'd have to ping internet search engines or other people for help.