Pmbootstrap: Alternative flasher method (using netcat)

[drebrez]

Here the steps to flash the system image using netcat:
1) pmbootstrap initfs hook_add debug-shell
2) pmbootstrap flasher flash_kernel
3) boot the device and it will show "missing boot partition"
4) telnet 172.16.42.1
5) nc -v -l -p 60000 > /dev/mmcblk0p12 <= change with the partition you prefer (fdisk -l to list)
6) in another terminal: pmbootstrap chroot
7) apk add pv
8) pv /home/pmos/rootfs/samsung-maguro.img | nc 172.16.42.1 60000
9) when transfer ends, ctrl+c in both terminals and reboot device

We can put it somewhere in the wiki or we can consider adding it as an alternative flasher method 😄

[MartijnBraam]

We can probably make a initramfs hook that justs starts the nc process on the phone. You also probably don't have to run flash_kernel, flasher boot should be enough.

[drebrez]

@MartijnBraam the initfs hook that starts the nc process is a very good idea 👍

The only problem I see, is that right now we don't have any information about the partitions of the devices.
Is there a way to find the correct partition using the name (system)?

[MartijnBraam]

Hmm can't we just start NBD on the phone to export all the partitions on the phone to the host PC?

[ollieparanoid]

Nice! :D

@MartijnBraam: We could start the nc server in a hook. But then we can only directly start flashing the file, which is a bit inconvenient for proper integration into pmbootstrap's flasher abstraction. Also when we simply pass through the phone partitions (I think that is what NBD does?), we risk the host Linux mounting them which is not what we want during flashing. I would prefer to use the normal usb-shell hook and use it as outlined below.

@drebrez: If you're actually using this right now (for plasma mobile?) or we have another valid use case, implementing this in pmbootstrap makes a lot of sense to me.

Here is an idea for the implementation (Outdated, I've written a new version below!):

• Add the system (and userdata for Android) partitions to the deviceinfo
• Add the partitions to the variables, that can be used by the flashers
• Add a new package postmarketos-flasher-nc, that does the following:
  
  
  
  • Depend on "pv"
  
  
  • Contains two python/shell scripts:
  
  
  • flasher_nc_list_devices.sh
    
    
    
    
    
    • Check the connection to the telnet usb hook
    
    
    
    • On successful connect, list the device
    
    
    
    • bonus: you could output the device name from /etc/deviceinfo
    
    
    
  
  
  • flasher_nc_flash.sh $PARTITION $IMAGE
    
    
    
    
    
    • Wait until it reaches the device on the usb-shell hook telnet port
    
    
    
    • Test the telnet connection (with "echo hello" or something)
    
    
    
    • Start nc -v -l -p 60000 > $PARTITION
    
    
    
    • Transfer the file
    
    
    
    • Close the nc on the device again
    
    
    
    • Print a success message
    
    
    
  
  
• Add the flasher method "nc" to the config for list_devices, flash_system, flash_boot

As I see it, the best part is that this works with images of any size, and we do not need to care about simg etc. :+1:

[MartijnBraam]

with nbd they won't get mounted automatically, you can run a nbd client in the native chroot and use the block devices that creates to install it the same way the sdcard installer does.

[ollieparanoid]

@MartijnBraam cool, that would also be nice. then we wouldn't even need to generate an image file!

[MartijnBraam]

The process basically is: adding nbd-server to the initramfs and starting a server instance for every partition you want to export with:

$ nbd-server -C /dev/null 9000 /dev/mmcblk0p1 &
$ nbd-server -C /dev/null 9001 /dev/mmcblk0p2 &

Then on the hosts machine you create nbd devicenodes in the chroot for the amount of partitions you want to open and then start the nbd-clients

$ mknod /dev/nbd0 b 43 0
$ mknod /dev/nbd1 b 43 1
$ nbd-client 172.16.42.1 9000 /dev/nbd0 &
$ nbd-client 172.16.42.1 9001 /dev/nbd1 &
# Then do stuff on the phone's storage:
$ mkfs.ext4 /dev/nbd1

[drebrez]

@MartijnBraam seems good, did you already tried if it works into the initramfs? I will try it next time I have to flash an image

[MartijnBraam]

I haven't tried anything yet on actual devices, this is just the stuff I researched about nbd (which I have used before but a long time ago)

[drebrez]

@MartijnBraam I will test it on the actual device and if it works I will start working on integrating this functionality in pmbootstrap 👍

[drebrez]

@MartijnBraam with nbd-server we will need to add additional libs into the initramfs, or we can use the first method (netcat) to add them just right before starting the install procedure

[drebrez]

@ollieparanoid we don't have to depend on pv, since I didn't found a way to detect when the transfer is done, with pv I see a nice progress bar and I can safely close the connection when it reaches 100%.

[ollieparanoid]

Well, it doesn't seem like a big dependency. So if pv is packaged in Alpine and we get improved usability (progress bar!), then we don't need to re-invent the wheel in my opinion.

[ollieparanoid]

Reading through that thread again, I'd still prefer nc over nbd, since it has less dependencies and is easier to set up.

I think a streamlined nc flashing method would improve the flashing situation a lot. We wouldn't have to deal with strange/broken fastboot versions (ones that require sparse images or can't flash over a certain size #897) and especially it would work with Heimdall devices, where most of the time flashing anything other than the kernel + initramfs does not work at all.

My plan from above does not have a good way to start the flashing mode and I think having the user to manually install a hook just to enter that flashing mode isn't that good.

New implementation idea

target device related

• Always include a function in the initramfs' init_functions.sh: enter_pmos_flash_mode()
• That function bails out if the new kernel parameter PMOS_FLASH_MODE is not set, so we can execute it without building a new initramfs.
• It creates a telnet shell on a new port, so the python code can connect to it.
• It shows a splash screen when everything is set up (postmarketOS flash mode is active\n https://postmarketos.org/flashmode) or something like that

pmbootstrap related

• Completely replace flash_system and therefore get rid of all the quirks we have
• Here's what happens when running pmbootstrap flasher flash_system with python code:
  
  
  
  • Install the dependency pv in the native chroot to get a nice progress bar
  
  
  • It boots the kernel + initramfs we have built, and sets the PMOS_FLASH_MODE parameter. (For heimdall-isorec devices, we must actually flash the kernel when changing the kernel parameter.)
  
  
  • automatically connect to the telnet connection, find out the partitions that need to be flashed (deviceinfo? or can we automatically do it on Android devices?)
  
  
  • Set up nc on the device, use pv to flash the system partition (and later also the extra partitions for LVM #60)
  
  
  • Isorec: flash back the kernel with the normal command line
  
  
  • Reboot the device
  
  

If anyone wants to give it a try, I'll be happy to assist and otherwise I'll try to do it myself at some point in the future (after the 200 days of pmOS blog post though).

As always, feedback is welcome!

[z3ntu]

Please don't get rid of the normal fastboot system-flashing method. I am for adding an additional method but not for throwing out working code.

[ollieparanoid]

Maybe this came out wrong, I would only suggest to replace the method if we are sure, that we don't have any regressions. In general, I think it is good practice to throw out working code if we have something better, so we don't infinitely increase maintenance effort and have too many solutions to do the same task. But it should really cover all use-cases and not be a bad trade-off, that makes you feel like functionality was removed.

But for that specific case, we could look into how much work it would be to keep the current method (especially with the sparse images) around if there are people, that prefer to have both. Would that be acceptable for you?

[z3ntu]

Sounds good

[drebrez]

@ollieparanoid what you think about starting triggerhappy in the initramfs and detect a pressed button?
triggerhappy is something like 50kb and we can detect the volume keys (almost every device have them) to start a "pmos flash/recovery" mode

[ollieparanoid]

@drebrez: Sounds good to me! :+1:

@z3ntu: btw: we have pmbootstrap flasher --method= for some time now, which should make it easy to have this new flash method side by side with the old one. After thinking about it again, I don't think we would save much by removing fastboot/heimdall/... support (and it wouldn't really be possible anyway, because we need that to boot/flash the initial kernel which is then able to boot pmOS flash mode).

If we don't put the code that runs on the PC in python, but create an aport with a shell script inside, then we can treat it like fastboot or any other flasher, thus reducing extra code paths.

[ollieparanoid]

Regarding the custom flasher protocol, @MayeulC noted in #postmarketOS that it would be a good idea to add some sort of checksum, to which @MartijnBraam replied:

you still need to make sure the connection wasn't aborted, so you'll need to send a checksum or the total image size first