Pmbootstrap: Binary repository problems with today's GCC and QT upgrade

[ollieparanoid]

GCC

TL;DR: run pmbootstrap update to fix pmbootstrap trying to compile gcc-armhf or gcc-aarch64.

GCC has been updated in upstream in Alpine with a bunch of new patches on top, so our cross compilers need to be updated as well (thanks @MartijnBraam for doing that in #1456).

Two problems:

• The official mirror of the sources of GCC is down, hence users can't compile the packages themselves right now (someone could probably find a mirror to the 6.4.0 patches tarball and use that instead by replacing the source in the APKBUILD)
• The package does not build in strict mode, it tries to install build-base-armhf when trying to build gcc-armhf. However with the normal pmbootstrap -y zap; pmbootstrap build gcc-armhf (where pmbootstrap does the dependency resolution, not apk), it appears to be compiling.

The binary package building PC has the sources in the cache, and I'm building the packages there right now to make them available for all postmarketOS users ASAP. As strict mode can't be used right now, I'm not using the regular repository build script, but the workaround described above.

QT

Alpine has also updated their QT version from 5.9 to 5.10. This means, we need to update our aports to use the new QT version as well, and then recompile them. Plasma Mobile can probably not be installed right now because of that. I'm on it.

Note: also some packages have been renamed in Alpine for consistency, e.g.

community/{qt5-websockets => qt5-qtwebsockets}

TODOs

• [x] compile and upload binary packages for gcc cross compilers
  > run pmbootstrap update to get the new packages if pmbootstrap tries to build gcc-armhf or gcc-aarch64
• [x] update our QT packages in the pmbootstrap repository to 5.10: https://github.com/postmarketOS/pmbootstrap/commit/6c6d0d3e28270afe699692fbf579fdcf798a314d
• [x] bump pkgrels of KDE packages of private API users: https://github.com/postmarketOS/pmbootstrap/commit/676d22fd8ad4fe3f9d738addae94e6f4ae9121e8
• [x] build binary packages (build is running right now!)
  
  
  
  • Almost all QT packages have been rebuilt already. qt5-qtwebengine is failing, which is needed for LuneOS UI (not for Plasma Mobile)
  
  
  • KDE packages still need to be rebuilt as of writing
  
  
  • See which packages have been built here: http://postmarketos.brixit.nl/
  
  
  • Run pmbootstrap update to make your copy of pmbootstrap aware of the latest packages that have been built (updates the APKINDEX)
  
  
• [ ] fix gcc-* not compiling in strict mode (#1468)

[ollieparanoid]

Update:

• qt5-qtwebengine and LuneOS UI are temporarily disabled (#1459)
• after a few roadbumps it seems like KDE is compiling now.
  
  
  
  • KDE: fix depend on renamed qt5-qtx11extras (6bb9600c4b10e2952f8c0657a667a543a1c81790)
  
  
  • KDE: bump pkgrels of pkgs using qt5-qtx11extras (f1d9745674d62ce8732bb6e7bb7ea23624d6ea47)
  
  
  • kirigami2: don't depend on plasma-framework-dev (eb3e409703a936330e31cd6510ba8c7f383b365c)
  
  
  • kdeclarative: bump pkgrel because of qt update (f5cf5400c75fc485953e5074762e9d40a36f7bc5)
  
  

I'm documenting which KDE packages need to be updated after a QT update here:
https://wiki.postmarketos.org/wiki/Repository_maintenance

I think Plasma Mobile should be working again by tomorrow this time. And yeah, there's a lot to optimize and streamline here, and I'd like to improve the workflow some time after the upcoming one year post.

[ollieparanoid]

Good news:

• all packages have compiled for x86_64 successfully
• Travis runs through again (if it does not with your PR, just rebase on master)

Bad news:

• Compiling plasma-framework for armhf is currently stuck (at 100% it seems to wait forever, maybe a deadlock or something? left it there for hours and it did not finish, although it did use 100% cpu). Does it work for anyone? $ pmbootstrap build --strict plasma-framework --arch=armhf
  
  
  
  • Seems to be similar to this issue (though that is from 2015)
  
  
• Compiling for aarch64 didn't even start because of this issue.

Summary:
Everything is working, except for pre-build plasma mobile packages for armhf and aarch64. And building plasma-workspace hangs at 100%, so it can't be used right now.

Still looking into this with highest priority.

[ollieparanoid]

Details about plasma-framework hanging:

• Last message in build log: [100%] Built target ksmserver
• This cmake command is running:

cmake -DGETTEXT_MSGFMT_EXECUTABLE=/usr/bin/msgfmt -DCOPY_TO=/home/pmos/build/src/plasma-workspace-5.12.3/locale -DPO_DIR=/home/pmos/build/src/plasma-workspace-5.12.3/po -P /usr/lib/cmake/KF5I18n/build-pofiles.cmake

• It does spawn a lot of msgfmt commands, that exit again

So it looks like it's just super slow because these run with QEMU emulation. I'll leave it there until tomorrow, let's see if it finishes.

[bhush9]

https://mail.kde.org/pipermail/plasma-devel/2018-May/084876.html you need patch in ki18n, fixed source package will be shortly available

[ollieparanoid]

Thanks @bhush9! I've committed the patch to master, but it turns out it still hangs there. I'm currently testing a patch that simply reverts the parallel building from https://github.com/KDE/ki18n/commit/5a70e96795f1e8529780f49ab30b19fcfab12a11, let's see if that helps.

EDIT: not really - with reverting that commit it acts the same as the patch you have linked. it does not hang at 100% anymore, but building just stops at [ 98%] Built target ksmserver with these processes running:

ps ax | grep make
 2180 pts/6    Sl+    0:00 /usr/bin/qemu-arm-static /usr/bin/make
 2193 pts/6    Sl+    0:01 /usr/bin/qemu-arm-static /usr/bin/make -f CMakeFiles/Makefile2 all
13361 pts/6    Sl+    0:00 /usr/bin/qemu-arm-static /usr/bin/make -f dataengines/places/CMakeFiles/plasma_engine_places_autogen.dir/build.make dataengines/places/CMakeFiles/plasma_engine_places_autogen.dir/build
13406 pts/6    Sl+    0:00 /usr/bin/qemu-arm-static /usr/bin/cmake -E cmake_autogen /home/pmos/build/src/plasma-workspace-5.12.3/dataengines/places/CMakeFiles/plasma_engine_places_autogen.dir/AutogenInfo.cmake Release
28071 pts/6    Sl+    0:00 /usr/bin/qemu-arm-static /usr/bin/make -f dataengines/powermanagement/CMakeFiles/plasma_engine_powermanagement_autogen.dir/build.make dataengines/powermanagement/CMakeFiles/plasma_engine_powermanagement_autogen.dir/build
28102 pts/6    Sl+    0:00 /usr/bin/qemu-arm-static /usr/bin/cmake -E cmake_autogen /home/pmos/build/src/plasma-workspace-5.12.3/dataengines/powermanagement/CMakeFiles/plasma_engine_powermanagement_autogen.dir/AutogenInfo.cmake Release

[ollieparanoid]

I have a workaround in place now: https://github.com/postmarketOS/pmbootstrap/commit/bb839551250accde28049e0dd87b54d82f60cc19

ki18n: downgrade to 5.44 and pretend to be 5.45

This worked when I did an initial test. I'll need to further debug which commit caused it to hang (git bisect or something), but at least this should finally unblock the binary package repository.

[bhush9]

Do you have instructions to test out this and reproduce bug? I have some thought on how to fix this

[ollieparanoid]

sure:

$ pmbotstrap build --strict --arch=armhf plasma-framework

It's the weirdest thing. I compiled it _on the same machine_ with https://github.com/postmarketOS/pmbootstrap/commit/bb839551250accde28049e0dd87b54d82f60cc19 before committing to master, and it ran through as expected. When I ran it again to build and sign for the binary repository on that machine, it was hanging again (with the 0% CPU usage) :man_facepalming:

It's like a race condition that sometimes works and sometimes doesn't (we had something similar when compiling qt5-qtwebengine).

So I will revert that commit again (with increased pkgrel though). What I tried afterwards in a testrun is compiling with the patch you've mentioned earlier and with using only one job (pmbootstrap -j 1 build --strict ...). That worked in the test run, I'll try that on the real repo as well, maybe it gets it unstuck for now.

I'm happy that you have a though on how to fix it!

[bhush9]

I assume there's typo in the command you mentioned and actual failing package is plasma-workspace ?

[bhush9]

And ofcourse I can not reproduce issue at all.. with both plasma-framework or plasma-workspace, because why not.. heisenbugs are fun.

[bhush9]

I've submitted a PR with workaround in plasma-workspace that would get binary repository working, in meantime I will try to reproduce bug locally and fix it upstream.

[ollieparanoid]

@bhush9: Yes, I keep confusing plasma-workspace and plasma-framework >_< I meant plasma-workspace all the time.

So yeah, it is a heisenbug. Sometimes the compilation works, sometimes not. I left it building with the -j1 approach described above, and while it worked in the test run, on second try (when building for the binary repo) it was stuck at [ 22%] Automatic MOC for target plasma_packagestructure_wallpaperimages.

In other words, not related to the ki18n change at all. That's why I've decided not to merge your patch with a workaround (I highly appreciate that you made one though). My conclusion is at this point, that it will get stuck sometimes, but not always, so I've simply kicked off the build in the binary repository again (at this point, we're having the stock ki18n package plus the patch you have linked, which really fixed the 100% CPU usage when cross compiling).

And this time it finally went through, so we have all binary packages for armhf again.

I am not sure where the bug is exactly, it could be in qemu (we don't have the latest version), in cmake, or possibly in a KDE build script (but I don't the latter is likely as the downgrade did not help it either).

What I can do to improve the situation:

• Modify the repo build scripts to kill the build when there is no output for 10 minutes like Travis does it (when the repo build script runs with --loop it will automatically retry then): #1487
  
  
  
  • As mentioned earlier, qt5-qtwebengine has a similar heisenbug where it gets stuck sometimes, so it would be helpful there as well
  
  
• Check how it behaves with an updated Qemu version
• Try to enable debugging output in cmake to figure out why it gets stuck

As the script went on to compile for aarch64, it said that qt5-qtx11extras is missing (it would be in Alpine's repos). Turns out build-edge-aarch64 from Alpine is currently stuck at llvm. I will temporarily add the package to the postmarketOS repositories to mitigate that issue as well.

[ollieparanoid]

All packages built for aarch64, armhf, x86_64 :tada:

@PureTryOut: would you like to make a PR for the KDE framework update?

[bhush9]

KF5 update is not public release yet.. it will be next week