Pmbootstrap: Support building kernels from a local git repository

[ollieparanoid]

This would be very helpful for kernel development (mainlining devices is one of our goals after all), and also this would make it easier to bisect kernels in case something doesn't work anymore after a new kernel release.

[ollieparanoid]

CC: @ata2001, @bhush9, @drebrez, @filippz, @flacks, @fourkbomb, @opendata26, @pavelmachek, @z3ntu

I came up with the following concept for supporting the mainlining process with pmbootstrap. These are just ideas, I have not written code yet. Please verify from an users point of view, if that workflow makes sense to you and whether it would really simplify mainlining.

Overview

1. Set up the new configuration option for the Linux kernel tree in pmbootstrap init. The checked out source tree is a few GB in size, so my assumption is that people will only want it once on their machine, and the location won't change often.

$ pmbootstrap init
...
Build options: Parallel jobs: 3, ..., Linux tree: none
Change them? (y/n) [n]: y
...
Path to the checked out mainline Linux source tree (useful for mainlining your device):
Linux tree [none]: ~/code/linux
...

1. Install the debug shell initramfs hook as usually

$ pmbootstrap initfs hook_add debug-shell

1. Repeat the following until the kernel does what you want it to do.

(apply changes to local kernel sources)
$ pmbootstrap build linux-postmarketos-mainline --localsrc --force
$ pmbootstrap flasher boot --flavor postmarketos-mainline

Details

The important change is the --localsrc flag:

$ pmbootstrap build linux-postmarketos-mainline --localsrc --force

Here's how pmbootstrap would treat that flag:

• The local kernel folder gets bind-mounted to /mnt/local-kernel-src/ in the native chroot
• pmbootstrap sets a new environment variable PMB_LOCAL_SRC=/mnt/local-kernel-src/ during the build
• We modify the linux-postmarketos-mainline APKBUILD to check for this variable, and if it is set:
  
  
  
  • do not download any source from the web
  
  
  • do not apply any existing patches
  
  
  • set the kbuilddir variable to point to PMB_LOCAL_SRC
  
  
  • run make mrproper in prepare()
  
  
  • set the pkgdesc to the current git commit
  
  

Benefits

• Easy setup, no host-distro specific steps required (e.g. cross-compiler installation)
• Avoids creating a tarball of the sources and extracting them for every build again
• Ccache works out of the box, and you use the same cache as for regular pmbootstrap usage
• No need to export the cross-compiler and arch variables
• Kernel config used directly from the package, pmbootstrap menuconfig works as usually
• Proper integration with the initramfs generation
• Short test cycle from modifying the source to booting the change on the device

[z3ntu]

I don't want a full clean after each test 😉 Just running make -j9 is enough for me. Seems quite good for me otherwise.

For reference, these are the commands I use:
setup:

set -gx ARCH arm; set -gx CROSS_COMPILE arm-none-eabi- # fish shell equivalent of export ARCH=arm etc

build:

make -j9
cat arch/arm/boot/zImage arch/arm/boot/dts/qcom-msm8974-fairphone-fp2.dtb > ../zImage-dtb
cd ..
./make_bootimg.sh
cd out
sign_img mainline-boot.img
fastboot boot mainline-boot.img.signed

The make_bootimg.sh script contains

#!/bin/bash
rm out/*
mkbootimg --base 0 --pagesize 2048 --kernel_offset 0x00008000 --ramdisk_offset 0x01000000 --second_offset 0x00f00000 --tags_offset 0x00000100 --cmdline 'rdinit=/init cma=64m msm.vram=16m PMOS_NO_OUTPUT_REDIRECT' --kernel zImage-dtb --ramdisk ramdisk.cpio.gz -o out/mainline-boot.img

And about the sign_img script, see https://z3ntu.github.io/2017/06/16/Signing-boot-images.html
Quick TLDR about sign_img: the FP2 bootloader is 'technically' 'locked' so it doesn't accept images via fastboot boot that aren't signed, but the bootloader also doesn't care what the signature contains (as far as I can tell) so signing the image with your own key works fine. Then the 'signed' image can be normally launched with fastboot boot.

[drebrez]

@ollieparanoid Seems quite good also to me.
Possible problem with the last step, after you rebuild the linux-postmarketos-mainline, it doesn't get installed/updated in the chroot where the flasher will run, because in https://github.com/postmarketOS/pmbootstrap/blob/master/pmb/flasher/frontend.py#L36 will call https://github.com/postmarketOS/pmbootstrap/blob/master/pmb/chroot/initfs.py#L34 which it reinstall only the updated postmarketos-mkinitfs.
We should also reinstall the linux-postmarketos-mainline at some point, manually or always like for the initramfs, or with an additional flag in the flasher....