Pmbootstrap: [titan] Cannot boot kernel compiled with pmOS

[PabloCastellano]

It can happen as well to other devices.

LineageOS boot image is 8MB and I can flash it but my pmOS boot image is 12MB, of which 8MB is the kernel.

We need to find some way to make the kernel smaller. I had to disable CONFIG_KERNEL_XZ otherwise the kernel would not boot. With CONFIG_KERNEL_LZMA it won't boot either.

Things to investigate:

• [x] Why just enabling XZ kernel compression doesn't work (this works for hammerhead)
• [x] How much we can achieve without the gzip wrapper (code)
• [x] CONFIG_CC_OPTIMIZE_FOR_SIZE=y
• [ ] Add a gzip layer around the already compressed vmlinuz (and repack boot.img) (*)
• [ ] Only compress initrd with XZ
• [ ] Load modules instead
• [ ] How can we know that is the size limit beforehand?
• [ ] Can we change this limit? Is it safe?
• [ ] Is this limit the same for all the devices?

(*) MartijnBraam said on IRC:

you can try compressing the xz kernel image with gzip so the bootloader unpacks only one layer
Your bootloader might be patched to always gunzip
Littlekernel should detect the gzip header and unpack before booting

This is the output of fastboot:

$ sudo fastboot flash boot boot.img 
target reported max download size of 536870912 bytes
sending 'boot' (11920 KB)...
OKAY [  0.461s]
writing 'boot'...
(bootloader) Image size exeeded partition limits
(bootloader) Preflash validation failed
FAILED (remote failure)
finished. total time: 0.501s

With LineageOS boot.img:

sudo fastboot flash boot boot.img 
target reported max download size of 536870912 bytes
sending 'boot' (8192 KB)...
OKAY [  0.352s]
writing 'boot'...
OKAY [  0.459s]
finished. total time: 0.811s

[ollieparanoid]

pmbootstrap uses a hack to make gzip always use the fastest compression method, instead of the best compressing one (trading speed for file size). That hack gets installed to /usr/local/bin/gzip and it impacts both the size of the initramfs, and the kernel (when the kernel gets compressed with gzip). These are side-effects, the original idea of the hack is to reduce the size of the apk packages, and therefore the build time.

What you could do now:

• Remove /usr/local/bin/gzip in all your chroots, and build the kernel image and initramfs again, it should be smaller. When this is all, that's needed to fix this issue for you, I think we should provide this as an option (or even better, upstream a compression parameter to apk, so we don't need the hack at all)
• Choose another compression algorithm in the kernel config (example)

[PabloCastellano]

Some numbers:

• Current initramfs is 3MB (being /lib/libcrypto.so.41 with 1.3mb is the biggest file)
• With GZIP compression, kernel is 8,5MB and boot.img is 12MB
• With GZIP compression and CONFIG_CC_OPTIMIZE_FOR_SIZE=y, kernel is 8,2MB
• With LZMA compression, kernel is 6,8MB and boot.img is 9,8MB
• With XZ compression, kernel is 6,5MB and boot.img is 9,5MB
• With XZ compression, and gzipped vmlinuz kernel is 4,8MB and boot.img is 7,8MB (*)
• I found no difference in the size with XZ compression and the GZIP wrapper removed

(*) But it doesn't boot either - it just reboots

[ollieparanoid]

Let me quote @drebrez, who made a very relevant comment in #127 (which is about including new programs in the initramfs to make automatic resizing of the system partition possible).

Another good solution would be to save all the additional tools (cryptsetup, parted, resize2fs, ...) that we need, inside the boot partition (or a dedicated one) of the pmos system image.
In this way we will reduce the initramfs size and it might help solving the #126.

@ollieparanoid @PabloCastellano what do you think?

Here some sizes based on what we include in mkinitfs.sh#L103:

BINARIES="/bin/busybox /bin/busybox-extras /sbin/cryptsetup /usr/sbin/telnetd /sbin/kpartx"

3'075 KiB

BINARIES="/bin/busybox /bin/busybox-extras /sbin/cryptsetup /usr/sbin/telnetd /sbin/kpartx /sbin/e2fsck /usr/sbin/resize2fs"

3'771 KiB

BINARIES="/bin/busybox /bin/busybox-extras /usr/sbin/telnetd /sbin/kpartx"

1'567 KiB

[ollieparanoid]

I think it's a good idea to put extra files in the boot partition - but I would put them in an extra file (e.g. initramfsbig-samsung-i9100?).

What we would need to keep in the real initramfs is:

• busybox
• kpartx (to detect the boot partition, if it is a subpartition of the system partition)

Dependencies:

/ # lddtree $(which kpartx)
kpartx => /sbin/kpartx (interpreter => /lib/ld-musl-armhf.so.1)
    libdevmapper.so.1.02 => /lib/libdevmapper.so.1.02
    libc.musl-armhf.so.1 => /lib/libc.musl-armhf.so.1
/ # lddtree $(which busybox)
busybox => /bin/busybox (interpreter => /lib/ld-musl-armhf.so.1)
    libc.musl-armhf.so.1 => /lib/libc.musl-armhf.so.1

Total size:

/ # du -h -c /sbin/kpartx /lib/libdevmapper.so.1.02 /lib/ld-musl-armhf.so.1 /bin/busybox
40.0K   /sbin/kpartx
288.0K  /lib/libdevmapper.so.1.02
536.0K  /lib/ld-musl-armhf.so.1
920.0K  /bin/busybox
1.7M    total

Remarks:

• that size is uncompressed
• we could make it even smaller by compiling only what we need from busybox to find/mount the boot partition and extract the big initramfs image. The busybox-extras binary (which contains telnetd among very few others) is 88kb in size for example (!) (we could have our own busybox-initfs package or something like that)
• we could make it smaller by compiling statically against musl. that way only the parts of musl, that are actually used, are kept.

[ollieparanoid]

I've created a new issue for the initramfs split solution: #211

[PabloCastellano]

After https://github.com/postmarketOS/pmbootstrap/pull/257 has been merged, I'm still unable to flash the boot partition. ~However the error message is different. I've not yet investigated:~

$ pmb flasher flash_kernel
[21:47:06] (rootfs_motorola-titan) mkinitfs motorola-titan
[21:47:18] (native) flash kernel motorola-titan
(bootloader) has-slot:boot: not found
target reported max download size of 536870912 bytes
sending 'boot' (10616 KB)...
OKAY [  0.408s]
writing 'boot'...
(bootloader) Image size exeeded partition limits
(bootloader) Preflash validation failed
FAILED (remote failure)
finished. total time: 0.456s
[21:52:12] ERROR: Command failed: (native) % fastboot flash boot /mnt/rootfs_motorola-titan/boot/boot.img-motorola-titan
[21:52:12] Run 'pmbootstrap log' for details.
[21:52:12] See also: <https://postmarketos.org/troubleshooting>

[PabloCastellano]

More news!

Enabling CONFIG_KERNEL_LZMA now the boot.img is 8MB and I can flash it \o/.
However I'm still having the same problem of not being able to boot with XZ compression (I forgot that it also happened with LZMA). But at least it is flashable :)

If you don't mind I'd like to keep the issue opened so that I don't forget about investigating and documenting some of the findings

[drebrez]

@PabloCastellano you can try some of these kernel options to reduce the size => http://elinux.org/Kernel_Size_Tuning_Guide#Kernel_Configuration_Options

[craftyguy]

@PabloCastellano

See if #599 helps reduce the initramfs size for you.

[walidham]

In arch/arm/mach-msm/Makefile.boot remove or commented thea dts files (if you want titan), like this:

MSM8226 Motorola Devices
ifeq ($(CONFIG_MMI_TITAN_DTB),y)
dtb-$(CONFIG_ARCH_MSM8226)  += msm8226-titan-4b.dtb
dtb-$(CONFIG_ARCH_MSM8226)  += msm8226-titan-4c.dtb
dtb-$(CONFIG_ARCH_MSM8226)  += msm8226-titan-4d.dtb
dtb-$(CONFIG_ARCH_MSM8226)  += msm8226-titan-4e.dtb
dtb-$(CONFIG_ARCH_MSM8226)  += msm8226-titan-4f.dtb
#dtb-$(CONFIG_ARCH_MSM8226) += msm8926-thea-p1a.dtb
#dtb-$(CONFIG_ARCH_MSM8226) += msm8926-thea-p1c.dtb
#dtb-$(CONFIG_ARCH_MSM8226) += msm8926-thea-p2.dtb
#dtb-$(CONFIG_ARCH_MSM8226) += msm8926-thea-p3.dtb

The size will be reduce, i have same issue in ubuntu touch, i got big kernel size and i can't flash it, but after this change, boot.img work well and ubuntu touch also

[PabloCastellano]

So finally I found the issue with XZ compression in my kernel :tada:

It turns out that the kernel makefile is broken because it uses a very weak method to detect if you are using the non-gnu version fo the stat command and it fails if you are using busybox's stat, which is the case in Alpine by default. The message in the log is cryptic and hard to spot because it doesn't fail hardly and the zImage probably remains corrupt somehow:

  LD      vmlinux.o
  MODPOST vmlinux.o
  GEN     .version
  CHK     include/generated/compile.h
  LD      .tmp_vmlinux1
  KSYM    .tmp_kallsyms1.S
  AS      .tmp_kallsyms1.o
  LD      .tmp_vmlinux2
  KSYM    .tmp_kallsyms2.S
  AS      .tmp_kallsyms2.o
  LD      vmlinux
  SYSMAP  System.map
  SYSMAP  .tmp_System.map
  OBJCOPY arch/arm/boot/Image
  Kernel: arch/arm/boot/Image is ready
  AS      arch/arm/boot/compressed/head.o
stat: can't read file system information for '%z': No such file or directory
expr: non-numeric argument
sh: invalid number ''
stat: can't read file system information for '%z': No such file or directory
expr: non-numeric argument
sh: invalid number ''
stat: can't read file system information for '%z': No such file or directory
expr: non-numeric argument
sh: invalid number ''
stat: can't read file system information for '%z': No such file or directory
expr: non-numeric argument
sh: invalid number ''
  XZKERN  arch/arm/boot/compressed/piggy.xzkern
  CC      arch/arm/boot/compressed/misc.o
  CC      arch/arm/boot/compressed/decompress.o
  CC      arch/arm/boot/compressed/string.o
  SHIPPED arch/arm/boot/compressed/lib1funcs.S
  SHIPPED arch/arm/boot/compressed/ashldi3.S
  AS      arch/arm/boot/compressed/lib1funcs.o
  AS      arch/arm/boot/compressed/ashldi3.o
  AS      arch/arm/boot/compressed/piggy.xzkern.o
  LD      arch/arm/boot/compressed/vmlinux
  OBJCOPY arch/arm/boot/zImage
  Kernel: arch/arm/boot/zImage is ready
  CAT     arch/arm/boot/zImage-dtb
  Kernel: arch/arm/boot/zImage-dtb is ready

I solved it by adding coreutils to makedepends= in the kernel and now I am also able to flash the kernel.

As a last note, I suspect that my device admits both appended dtb to zImage and to the boot image.

[ollieparanoid]

Awesome news! \o/

It might be enough to depend on the xz packages though, I guess coreutils pulls that in (altough I couldn't see it directly), because xz is only provided by xz (and busybox, but that isn't listed as it installs the links dynamically).