Pmbootstrap: qemu bug: abuild-tar can't package symlinks when emulated in aarch64

[zhuowei]

The weston package seems to be missing a library file:

```pmbootstrap/chroot_rootfs_huawei-angler/usr/lib$ ls -l libweston-desktop-2.so.0
lrwxrwxrwx 1 root root 28 Sep 4 01:53 libweston-desktop-2.so.0 -> libweston-desktop-2.so.0.0.0

And listing the package confirms that the library is missing: 

```pmbootstrap/packages/aarch64$ tar tf libweston-desktop-9999-r2.apk
.SIGN.RSA.user-59ac85cf.rsa.pub
.PKGINFO
usr/
usr/lib/
tar: Ignoring unknown extended header keyword 'APK-TOOLS.checksum.SHA1'
usr/lib/libweston-desktop-2.so.0

I'm using the latest master (e421313632676cf2e332e62a0fe039a8499a9021) with a few Nexus 6P patches (no changes outside of aports/device/device-huawei-angler, aports/device/linux-huawei-angler)

[drebrez]

@zhuowei I'm getting the same issue, I think it might be due the recent update of libweston to the version 3.0.0

[ollieparanoid]

Yep, weston just got updated to 3.0.0 in Alpine upstream. Does this issue go away when building weston 3.0.0 from #495? It is in branch update-weston-3.0.0.

[drebrez]

I've found another solution, simply adding the libweston package to the weston dependencies solve it, creating the PR right now

[drebrez]

@ollieparanoid I'm not sure anymore about my solution, adding libweston dependency to our weston APKBUILD is kind of strange.
What I've done was, add the depends and run pmbootstrap build weston --arch=armhf, this fixed it.
Maybe the build was enough without adding the dependency.
@zhuowei can you try to rebuild the package to see if it works?

[zhuowei]

@drebrez I did a rebuild with pmbootstrap build weston --force -- arch=aarch64 before filing the bug report; it didn't help. Should I try again?

[zhuowei]

@ollieparanoid I'm getting a different error after cherry-picking the weston 3 branch:

localhost:/usr/lib$ cat /tmp/weston.log 
Error loading shared library libweston-3.so.0: No such file or directory (needed by /usr/bin/weston)
Error relocating /usr/bin/weston: weston_plugin_api_get: symbol not found
(a bunch more symbol not found)

and indeed it's not there:

pmbootstrap/packages/aarch64$ tar tf libweston-9999-r4.apk
.SIGN.RSA.user-59ac85cf.rsa.pub
.PKGINFO
usr/
usr/lib/
tar: Ignoring unknown extended header keyword 'APK-TOOLS.checksum.SHA1'
usr/lib/libweston-3.so.0

Is my build setup screwed up?

[zhuowei]

Ah. Found the issue: if a symlink is in a .tar file, abuild-tar would call write with a null data pointer and zero length, and QEMU incorrectly returns an error. I guess no-one else is running into this because this would work fine if the symlink is the last file that is added when tar runs, but on my system the symlink is added first.

[ollieparanoid]

Wow, thank you for figuring this out and reporting it upstream. It must have taken you a very long time to debug this, I appreciate it very much!

Note: pmbootstrap is able to compare package contents, in order to check if both contain the same binaries. This might be helpful in debugging such an issue in the future - you could compare your build with a build from another PC for example (pmbootstrap build --buildinfo weston and pmbootstrap challenge path/to/weston.apk).

Also I ran into another problem with weston on aarch64, just in case you will also have it (but it seems to happen to the development headers in weston-dev only): https://github.com/postmarketOS/binary-package-repo/issues/9

[zhuowei]

@ollieparanoid I'm thinking about creating a patch to abuild that sorts the files in the tar archive by name (like Debian's patch) so that this issue would consistently happen. Would that help track down issues?

[ollieparanoid]

As I understand, you have already found the cause of the missing libweston-desktop-2.so.0.0.0 file (it lies in qemu). But in general it would be helpful, if abuild produced stable tar output!

My development on the binary package repo with somewhat reproducible builds has slowed down due to other tasks as you probably have noted.

[zhuowei]

@ollieparanoid Thanks. Also, for the QEMU bug: QEMU only takes patches against the latest development version, not version 2.8 that we use. Do I also need to submit a patch to the maintainers of the Debian QEMU 2.8 package, or would they backport changes from the development version? (They only seem to backport CVE fixes as far as I know.)

[ollieparanoid]

If Debian only accepts CVE fixes, then I guess it's not worth trying.

Depending on how much time you want to invest in this (totally understandable if this is too boring for you):

• you check if this bug also appears in the latest development version
• if it does, create a patch and submit it upstream to Qemu
• package it for Alpine and apply the patch there
• then upstream the package to Alpine
• then we could adjust pmbootstrap to use that instead of the Debian one

[zhuowei]

@ollieparanoid I was confused about how we use qemu: I asked on IRC and it turns out we only use Debian qemu for running the qemu-* emulated devices, not for building, and Alpine does accept extra patches.

I'm starting to upstream the patch: I'm going to follow up on the feedback and hopefully get it into the next qemu release.

[ollieparanoid]

Cool that you're upstreaming the patch! We use qemu for the device-qemu-* devices (the native qemu installed on the host!) and for building and running armhf code from the chroot on x86_64 (the version repackaged from Debian):
https://wiki.postmarketos.org/wiki/Build_internals#Cross-compile_types (distcc)

[zhuowei]

@ollieparanoid So at least two other people ran into this when building Weston: should I patch abuild-tar to not make null zero-length writes instead? That's the workaround I'm using locally, and may be faster than waiting for the next QEMU release.

[ollieparanoid]

That sounds like a good short-term solution. Just keep in mind, that we can't put abuild in our aports-folder, because abuild is required to build packages (pmbootstrap would probably recurse infinitely then, and we only need this fix for one architecture anyway).

We already put a wrapper for gzip to /usr/local/bin/gzip, which changes the compression flag to -1 to speed up builds. So we could use a similar technique here:

• New package abuild-tar-qemu-workaround (with url set to this issue)
• Installs the patched abuild-tar as /usr/bin/abuild-tar-qemu-workaround
• Post-install script symlinks that to /usr/local/bin/abuild-tar

Finally, we would need to change the Python code to install abuild-tar-qemu-workaround if the target architecture is aarch64. That should be possible by adding the following lines at the end of pmb.build.init():

if suffix = "buildroot_aarch64":
    pmb.chroot.apk.install(args, "abuild-tar-qemu-workaround", suffix)

As always, this is only a recommendation. Alternatively, we could also apply the workaround to abuild-tar upstream, but I'm not sure if Alpine developers really want to carry around such a patch (and if we do it only in pmbootstrap, we get the workaround faster - after all, the goal of the workaround is to provide a solution that works now until we have the proper fix in Qemu).

[zhuowei]

@ollieparanoid The native cross build PR can work around this by completely avoiding qemu. So if that gets merged I don't think we would need another short term solution.

[ollieparanoid]

True! It will also take some time until that gets merged though, because its a deep architectural change and it really needs to be tested well.

[zhuowei]

@ollieparanoid I'm trying to get that PR ready faster by only whitelisting two packages for now. So hopefully it should not take too long to merge it?

[zhuowei]

@ollieparanoid You mentioned that you're adding a workaround for this: how does it work?

[ollieparanoid]

See for yourself :) But basically like described here: https://github.com/postmarketOS/pmbootstrap/issues/546#issuecomment-333664617

I'm not really happy with the workaround, but it's a good short-term solution. I'd still like to see this get patched in Qemu upstream (could you ask on the ML what the status is, or if we could help them?) and package qemu-user-static in Alpine instead of repackaging Debian's version.

I guess, that Alpine devs would only accept a PR for the latter after Alpine 3.7 is released though, because they are in something like feature freeze now and concentrating on the upcoming release.

[zhuowei]

@ollieparanoid I'm closing this since the workaround is merged, and you've tested that it works with Weston. Thank you so much!

[ollieparanoid]

You're welcome, thank you for finding out where to fix it in the first place 👍

But I'm reopening this until it is fixed properly in upstream Qemu and until we can remove the workaround again.

[ollieparanoid]

I've replied to the upstream ML discussion what is left to do for upstreaming @zhuowei's patch, and got this reply:

• update the patch to call write() with NULL and 0, something like:

--- a/linux-user/syscall.c
+++ b/linux-user/syscall.c
@@ -7912,6 +7912,10 @@ abi_long do_syscall(void *cpu_env, int num,
abi_long arg1,
         }
         break;
     case TARGET_NR_write:
+        if (arg2 == 0 && arg3 == 0) {
+            ret = get_errno(safe_write(arg1, NULL, 0));
+            break;
+        }
         if (!(p = lock_user(VERIFY_READ, arg2, arg3, 1)))
             goto efault;
         if (fd_trans_target_to_host_data(arg1)) {

• change TARGET_NR_read to do the same
• check if we need to do the same for pread64/pwrite64