Magisk 19.1 breaks accessing the recovery on Sony Xperia devices

[okij1]

After installing Magisk 19.1, accessing the recovery doesn't work anymore.

This affects both

• invoking the recovery during boot process by pressing Vol-Down key when the pink LED light shows
• rebooting to the recovery via Power menu.

The LED light changes its colour as normally when the recovery is invoked, but instead of showing the TWRP splash screen the device reboots.

Bug reports and confirmations: See here, here, here and here on the Magisk General Support thread, as well as here, here, here and here on some of the Xperia custom ROM threads.

This bug isn't present in Magisk 19.0, so most likely it's caused by the new _recovery based Magisk_ feature, which is needed for the new Samsung phones.

In comparison to most other devices, the affected Xperia devices have their custom recovery in the FOTAKernel partition instead of the Recovery partition.

I'm not a programmer, but I suppose that adding a check if the FOTAKernel partition is present before invoking the _recovery based Magisk_ code (so that it behaves like Magisk 19.0 on the Xperia devices) could fix this bug.

[okij1]

Please also check official TWRP device support page for more infos about the FOTAKernel partition.

[russel5]

on my xz1 compact i'm don't have any problem with magisk. but i'm using canary builds.
one more, if you are using script to enter to recovery that is the problem in this script.
one more check this commit https://github.com/topjohnwu/Magisk/commit/71a91ac7a7f729fd845dcb920178d39611694175
and the last, at my old z3 compact with latest update i'm enter in recovery by turn off your phone - press vol down and power on - when phone vibrate release power button and hold volume down - welcome to recovery

[okij1]

Thanks for your reply. :-)

on my xz1 compact i'm don't have any problem with magisk. but i'm using canary builds.

Does the XZ1c also use FOTAKernel partition and are you able to reboot to recovery and to invoke recovery by pressing Volume-Down when the LED light shows during booting?
If both is the case, maybe this issue has already been fixed in the canary beta builds?

one more, if you are using script to enter to recovery that is the problem in this script.

No, IMHO the problem are the changes in Magisk 19.1 because this issue did't occur in Magisk 19.0 and all pevious versions.

one more check this commit 71a91ac

I'll leave this to the devs. :-)

and the last, at my old z3 compact with latest update i'm enter in recovery by turn off your phone - press vol down and power on - when phone vibrate release power button and hold volume down - welcome to recovery

Yes, this is the last possible way we have to enter recovery when using Magisk 19.1 on the Z3 compact. But - as I wrote in the issue report - two other ways are not working anymore: 1. Pressing Vol-Down when the LED shows and 2. rebooting to recovery from System. Because of this, it's also not possible anymore to install OTA updates, since this also needs rebooting to recovery.

[russel5]

yes it has.
once again, you do not have any issues and this thread is useless and may be closed.
all of your problem is just a SCRIPT that uses commands when phone is boot up to load the ramdisk of recovery from recovery partition. this SCRIPT was made because in older time we can't enter to recovery at all.
At NOW Sony give us this feature and please use it feature power+vol down
if you're lazy enough to push two buttons, that's your problem, not magisk.

[okij1]

Of course I'm not too lazy to push two buttons, but several users already have reported this issue on XDA and for sure there will be more of them once the custom ROM updateds with the May security patch level roll out and they try to install the ROM either as OTA update (which won't work anymore) or by entering TWRP by reboot to recovery from power menu (which is broken) or by trying to enter TWRP by pressing Vol-Down at the LED light.

Just the fact that there is _one_ way left to enter recovery by pressing Power + Vol-Down when the device is off doesn't justify the other functionality being broken.

So if this issue really has been fixed, of course it can be closed. As long as it isn't, it should stay open until it is fixed. And surely it can be fixed.

[toohazy]

I don't usually comment on github issues, but the hold vol down+power doesn't work for me, I'm running Oreo 8.1 carbon rom on a sony xperia z2, so it's not just a script issue.

[elem089]

@russel5 this has nothing to do with lazyness. At least on my Xperia Tablet Z / SGP311 / Pollux_Windy device there's no way to enter recovery if Magisk 19.1 is installed.
Tried
a) the boot script
b) key combination on boot
c) adb command
Reverting to 18.1 solves that for now.

Later on will try that on my Z1compact.
UPDATE:
same behaviour with my Xperia Z1 compact (D5503), i.e. no way to enter recovery mode with Magisk 19.1

[topjohnwu]

@russel5 I seriously need some Xperia experts for help :)
If you're willing to join my internal dev Slack group, leave me an email and I'll invite you in.

[okij1]

@topjohnwu: I had a look at the GitHub repositories of the devs who are working or worked on the Sony Xperia devices.

IMHO the most experienced devs to get in touch with in this matter are @jackfagner, @AndroPlus-org and @shoey63 who worked on the _Sony recovery boot script_, please see

• https://github.com/jackfagner/twrp-sony-recovery-boot-script
• https://github.com/AndroPlus-org/twrp-sony-recovery-boot-script
• https://github.com/AndroPlus-org/build_tools_xperia/tree/android-7.0/tools/twrp-sony-recovery-boot-script-XP
• https://github.com/shoey63/twrp-sony-recovery-boot-script

Maybe you can already figure something out by looking at the code, but hopefully one of the devs gets in touch with you.

Some other skilled devs who worked on recoveries and ROMs for different Xperia devices are @xNUTx, @Myself5, @rcstar6696, @Diewi, @nailyk-fr, @mcgi5sr2, @drakonizer, @derfelot and @tomascus.

I hope that at least one of those devs can give you a clue about how to fix this bug. :-)

[nailyk-fr]

I didn't looked at the issue but from the title I need to explain why happens with 'old' devices.
There is no recovery access from the bootloader.
So DeesTroy and Jerpela made a modification into init (extract_elf_ramdisk) to extract recovery ramdisk at early stage of boot ramdisk init process.
(Content of / got removed, then recovery ramdisk is extracted, then /init is called again)
https://github.com/omnirom/android_device_sony_common-init/tree/android-8.1/extract_ramdisk
It is easy to break (like changing compression method) so it could be related.

I can detail more if you think it is related. Also, I can help investigation if you need.

Again, I didn't not even read the issue, just answering to the email I got ;)

Hope this helps and good luck :)

Le 6 mai 2019 20:28:57 GMT+02:00, okij1 [e-mail] a écrit :

@topjohnwu: I had a look at the GitHub repositories of the devs who are
working or worked on the Sony Xperia devices.

IMHO the most experienced devs to get in touch with in this matter are
@jackfagner, @AndroPlus-org and @shoey63 who worked on the _Sony
recovery boot script_, please see

• https://github.com/jackfagner/twrp-sony-recovery-boot-script
• https://github.com/AndroPlus-org/twrp-sony-recovery-boot-script
  -
  https://github.com/AndroPlus-org/build_tools_xperia/tree/android-7.0/tools/twrp-sony-recovery-boot-script-XP
• https://github.com/shoey63/twrp-sony-recovery-boot-script

Maybe you can already figure something out by looking at the code, but
hopefully one of the devs gets in touch with you.

Some other skilled devs who worked on recoveries and ROMs for different
Xperia devices are @xNUTx, @Myself5, @rcstar6696, @Diewi, @nailyk-fr,
@mcgi5sr2, @drakonizer, @derfelot and @tomascus.

I hope that at least one of those devs can give you a clue about how to
fix this bug. :-)

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/topjohnwu/Magisk/issues/1418#issuecomment-489725052

--
Envoyé de mon appareil LineageOS avec K-9 Mail. Veuillez excuser ma brièveté.

[nailyk-fr]

After a quick reading of the issue it is definitively related. (z2 impacted which use that hack, xz1 not impacted and does not use that hack)
@ testers: to help debugging, can you please explain if the issue happens in the recovery image or into boot image ?
(like get a working device, flash the 'faulty' magisk, restore boot, check if you can still access recovery. Then same but with recovery.img restore. )
Notice: I won't bother to fix stock roms, will only fix ectract_elf_ramdisk if needed.
@topjohnwu : can you please explain briefly what do you change into boot.img & recovery.img and link to the related source code ?
FI, I would prefer implementing a fix into Magisk (if it is not too 'hackish' instead as into extract_elf_ramdisk as this needs to be reflected into every custom rom (and we could expect to forget some/not have it merged in some)

@russel5 the commit you link mostly did the same job as extract_elf_ramdisk . And about https://github.com/topjohnwu/Magisk/issues/1418#issuecomment-489338053 it is not totaly true. Only z3 & z3c (regular, no the dual) got that update. z2, z3dual and all previous does not have the new bootloader.
@topjohnwu I see a file called init.cpp, I hope you do not replace the init of the xperia boot image (or any other devices), as it will mostly break any bootimage. On xz1 (for example) we need a customized init to load some crappy properties from bootloader. Same on some Samsung or even LG.
One of the hackish way we used was to have 2 init. First one is called from kernel, then the 2nd one is renamed to /init and called again. Maybe it could be a workaround ?
Also, IMHO, you should not provide a way to boot into recovery. It is not to the 'root' app to do it, but to the rom maintainer.

Happy debugging :smiley:

[topjohnwu]

Magisk requires replacing init to work at all. I will not change this just to support Xperia devices. Someone have to find a way to make your "recovery" support function properly along side with Magisk.

[okij1]

:-(

@topjohnwu: Oh, come on.

You could at least answer @nailyk's request: _"can you please explain briefly what do you change into boot.img & recovery.img and link to the related source code ?"_, so that he can have a closer look and probably help you out with fixing this issue.

Or do you really want to alienate all Xperia users? :-/

[Myself5]

Magisk requires replacing init to work at all. I will not change this just to support Xperia devices. Someone have to find a way to make your "recovery" support function properly along side with Magisk.

When the new version of your product provably breaks something then this is definitely not an appropriate answer to a caring user.

That said, this is definitely not an "All Xperias will be dead now" case. Most legacy devices got bootloader updates at some point to support booting straight to the recovery. The only devices that do not have this feature would be the DualSIM variants as these never got the bootloader updates.

P.S. I am on the Magisk Slack. Hit me up if you are still intrested into fixing this.

[topjohnwu]

@okij1 @Myself5 It might seem that I am a little bit too harsh here, but I am serious about not being able to fix this issue. It is not me being lazy and ignoring your requests, but because of fundamental design decisions. So let me explain myself more clearly.

The reason why it previously worked is completely accidental. Magisk is never guaranteed to work with any other init hacks, this including MultiROM, and in this case booting to recovery on certain Xperia devices. The reason why it worked is because before v19.1, the /sbin tmpfs overlay is constructed after the original init took over and started the initial setup.

Pre-v19, the job of creating the /sbin is on the magiskd daemon, which involves remounting rootfs to rw and very dirtily messing with tons of stuff. In v19.0 the job is moved to magiskinit, but done by creating a new init daemon and wait for the early-init event trigger to do so. But this change broke tons of devices, so in v19.1, all the sbin overlay setup is done before the original init is invoked.

I assume the reason why v19.1 no longer boots to recovery in your case is because the "recovery booting" init is not aware of the /sbin overlay and cannot properly extract the recovery ramdisk to boot up.

Do you know how complicated magiskinit has become? These changes are important for the upcoming new approach to deal with system-as-root devices, which is a vital part for Android Q support. It is apparent to me that these legacy Xperia devices that does not have proper bootloader support to boot to recovery are in the minority. Do you think it is reasonable for me to revert such a big design decision, which works fine all the way back to Android 4.2 on normal devices, just to cater to a small group of people?

It is definitely much easier to fix the "recovery booting" init than changing how Magisk fundamentally works. I can tell you now that it mostly only requires an additional line of code to fix it: to unmount /sbin before doing what it supposed to do.

Maybe someone in the Xperia community can do this for you?

[okij1]

@topjohnwu: Thanks a lot for this detailed information! :-) Although this issue isn't solved yet, now at least we can understand your situation and our Xperia devs have something at hand to work on.

@Myself5 and @nailyk: You know that I'm not a programmer, so I only understand a fraction of the things that topjohnwu wrote of.

I personally have a Z3c, so I can still enter recovery by starting the turned-off device with Power + Vol(-), but for other devices it's really crucial that this bug gets fixed.

AFAIK not only the dual sim devices didn't get the updated bootloader from Sony, but also the Z and Z1 (and probably also Z2) line of devices.

Ideally there would be just a small addition to the Magisk source code which checks if the device is an affected Xperia device (maybe by checking if the FOTAKernel partition is present) and then allows the Xperia specific init stuff still to be executed correctly. I'm not sure if this is possible with manageable effort, but this way it wouldn't be necessary to change dozens of ROMs for different Xperia devices.

Otherwise the init code for _all_ affected current ROMs would have to be changed according to topjohnwu's hints, but IMO that would be a bigger effort.

[xNUTx]

Actually, I guess it's possible to pre-boot the device using the original init method to allow recovery access and have it start magisk's init be called when it isn't triggered. That way we still have full control over the init process and can hand over full control to magisk when not getting triggered.

It's basically keeping the Sony specific bootstrap in tact, and instead of loading the original init afterwards, load magisk's init.

Just my 2 cents 😉

[nailyk-fr]

Yeah but after seeing the motivation of magisk owner, I lost all my energy to work on it.
I agree he won't loose time or some design cuz by some OEM mess but 'I won't bother with two people' beats me off.
Open source is made to handle every cases, not only the most lucrative ones....
Everything is possible, especially in open source, when people want to.
When I got answers to my questions I could rethink it.
Meanwhile I will stick to older versions or to another root.
No offense, just being tired doing all the efforts when I got such answers.

Le 11 mai 2019 21:01:40 GMT+02:00, "[NUT]" [e-mail] a écrit :

Actually, I guess it's possible to pre-boot the device using the
original init method to allow recovery access and have it start
magisk's init be called when it isn't triggered. That way we still have
full control over the init process and can hand over full control to
magisk when not getting triggered.>
>
It's basically keeping the Sony specific bootstrap in tact, and instead
of loading the original init afterwards, load magisk's init.>
>
Just my 2 cents 😉>
>
-- >
You are receiving this because you were mentioned.>
Reply to this email directly or view it on GitHub:>
https://github.com/topjohnwu/Magisk/issues/1418#issuecomment-491536054

--
Envoyé de mon appareil LineageOS avec K-9 Mail. Veuillez excuser ma brièveté.

[okij1]

@topjohnwu: Please don't put the devs off who are trying to help, but provide them with the necessary information they ask for.

On May 7th, @nailyk-fr asked you:

can you please explain briefly what do you change into boot.img & recovery.img and link to the related source code ?

[okij1]

@nailyk-fr: Looking through the Magisk commits since version 19.0 IMO there are those commits which could have lead to this bug on Xperia devices:

• Support recovery based Magisk
  and maybe also
• Boot to recovery if volume up key is held
• Prevent accidental magiskinit execution

This and the comment of topjohnwu from May 11th (please see above) should hopefully answer the questions you have asked. I sincerely hope that you are not too pissed off and that you will have a further look at this issue. :-)

@Myself5 and @xNUTx: Also For your information. If you have some time, please also have a look at the commits I listed and if possible make proposals for a fix.

@topjohnwu: please correct me if I'm wrong and please link to other parts of the code which could have an impact on booting to recovery on Xperia devices.
BTW, I've seen quite a lot of code especially made for Samsung devices. Do you really think it's fair to support the big players like Samsung, but drop the support for phones with smaller market shares like Sony Xperia?
By now, my post on the Magisk General Support / Discussion thread where I reported this issue already got thanked by 12 users. And these are only the users who found my post on XDA and dared to click on the Thanks button, so you can be sure that a lot more Xperia users are affected by this bug.
That said, I'm kindly asking you to re-open this issue.

[topjohnwu]

I personally do not own any Xperia device, and I know nothing about it, so I can't fix the issue myself. I'm more than happy if anyone is willing to help and create a pull request to detect whether Xperia is booting to recovery.

I don't understand the mentality in people's mind that I should put effort on issues. In fact, I don't owe anyone anything, and maintaining Magisk is not my job. @nailyk-fr if you really want to help, do it by creating a PR to actually help this project in a good way, and not just saying "I'm tired doing all the efforts when I got such answers".

I AM tired of people constantly complaining about issues and VERY few actually contributes back to the project. This is an open source project, but nobody seems to understand what "open source" means. A lot of people helped fix issues in Magisk in many ways, and so can you.

@okij1 of course I will add specific Samsung support, because I own the freaking device. I don't think it is fair for you to ask me to support a device I don't have.

[okij1]

@topjohnwu: Of course I don't think that you have to support a device that you dont't have.

That's why I tagged the experienced Xperia devs like @nailyk-fr, @Myself5, @xNUTx and @drakonizer, in the hope that they have a look and help fixing this isue. I personally would like to help more if I could, but I'm not a dev.

On the other hand, just closing issues to have them "out of the way" is not the right way to treat people who have a severe problems with a new release. People who are using Magisk for a long time by now and really appreciate your great and ingenious work, as I certainly do.

I'm currently using Magisk 18.1, since both 19 and 19.1 gave me problems. But I don't want to get stuck with version 18.1 and miss future enhancements of Magisk. That's why I opened this issue and I am honestly disappointed that you just closed it and by doing so discouraged other devs to have a look and help fixing this issue.

@ the Xperia devs out there: Please help fixing this if you can. By now there should be enough hints in the comments of this issue to give some clues where to look at.

[nailyk-fr]

@topjohnwu I agree, and I come here to help.
And if take the time to click on my nick you will see I try to do my best to fix the issue I open ;)
I am also doing this on my spare time and know how time consuming it is.
In now way I was trying to offend you, I know how maintaining OSS can be a pain.

Let's try to start again peacuffly:
I do not have the knowledge's or the time to learn and find how Magisk is working.
I will be glad to help, but as the project owner, please point on the right direction.
Where should we look at?
Some interesting commits where pointed out. As it worked before, what about a build flag to disable the 'enter yo recovery' from Magisk. If it is too much, we will find another way.
Can you please explain (or link to the part) where bootimages are modified.
It looks like you are trying to produce the same behavior as 'extract_elf_ramdisk' do. Another way could be to get rid of it and go for magisk instead.

Also people which have the issue, to save time, please test and explain if the bootimage changes prevent the recovery booting or the recovery one. Just produce the issue, restore one, try, then the same with the other image (recovery/boot)

@okij1 as it seems the issue is really annoying you please do some more testings, logs reporting, etc... You can also try to use 0smosis mkbootimage tool to extract/repack the images to understand what is failing. Also a good thing could be to make some 'inventory' of impacted devices and ROMs (with maintainer list) to inform then about the patch when done (as it is most probably a failure into 'extract_elf_ramdisk'. ) Maybe someone fixed it already?

Le 12 mai 2019 20:04:11 GMT+02:00, okij1 [e-mail] a écrit :

@topjohnwu: Of course I don't think that you have to support a device
that you dont't have.

That's why I tagged the experienced Xperia devs like @nailyk-fr,
@Myself5, @xNUTx and @drakonizer, in the hope that they have a look and
help fixing this isue. I personally would like to help more if I could,
but I'm not a dev.

On the other hand, just closing issues to have them "out of the way" is
not the right way to treat people who have a severe problems with a new
release. People who are using Magisk for a long time by now and really
appreciate your great and ingenious work, as I certainly do.

I'm currently using Magisk 18.1, since both 19 and 19.1 gave me
problems. But I don't want to get stuck with version 18.1 and miss
future enhancements of Magisk. That's why I opened this issue and I am
honestly disappointed that you just closed it and by doing so
discouraged other devs to have a look and help fixing this issue.

@ the Xperia devs out there: Please help fixing this if you can. By now
there should be enough hints in the comments of this issue to give some
clues where to look at.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/topjohnwu/Magisk/issues/1418#issuecomment-491616267

--
Envoyé de mon appareil LineageOS avec K-9 Mail. Veuillez excuser ma brièveté.

[topjohnwu]

I already kept in touch with @Myself5. Let's see what he can fix in init before changing things in Magisk.

[okij1]

@topjohnwu: Thanks a lot for opening this issue again! :-) :-)

Since I'm actually using CarbonROM from @Myself5 on my phone, for me of course it would already be great if the issue gets fixed in the init code of the ROM.

However, a general fix in Magisk itself would still be better, because it would help on all affected Xperia devices, without the need of changing or updating the ROM. Something like "if this is an Xperia phone, then don't forget to execute its own init stuff too". Maybe this sounds naive, but maybe it's not too difficult to implement, without touching much of the Magisk code.

@nailyk-fr: Thanks a lot that you are participading again! :-) If you have some time, maybe you could look at the second approach.

[elem089]

Thanks all for your efforts. I'm fully aware that all this work is done in each dev's free time, and I really appreciate it. I can't do much here on a dev level but if some testing is needed I can participate; I own 2 rather old Xperia devices (Z1 compact and TabletZ) both currently running LineageOS 15.1. One of those is a "drawer" device so I can install anything on it without fearing to lose any data.

[jackfagner]

I've gave my Z3C to my sister, who lost it (this is standard procedure for all my old phones). I can't investigate this issue but I am following your progress 👍

[Myself5]

Fixed by the commit above. Thanks @topjohnwu for the hint.

[115ek]

I can confirm this fix to be working on Xperia Z1 compact - amami running Lineage 16.0.
Thanks @Myself5

[okij1]

@Myself5: Thanks a lot for examining and fixing this bug! :+1:
@topjohnwu: Thank you again for reopening this issue and for giving the hint to @Myself5!
@115ek: Thanks for confirmation that this fix is working!

I've one question left @topjohnwu: With the knowledge of how to fix this bug in android_device_sony_common/init/init_main.cpp, would it also be possible to implement a fix in Magisk itself? I'm asking because then also older ROMs would be compatible to Magisk > v19.1 again, which might be useful for some users. If it's possible, that would be awesome, if not, OFC you can now close this issue nonetheless. :-) (Edit while typing: I saw that you just closed it.)

@Myself5: Of course I'm awaiting your new CarbonROM builds which will include this fix. :-)

[Myself5]

With the knowledge of how to fix this bug in android_device_sony_common/init/init_main.cpp, would it also be possible to implement a fix in Magisk itself?

I don't think so. Magisk itself depends on the sbin it mounts, meaning it can't unmount it at any point. (Figured that out after flashing magisk 5 times and a lot of swearing until I moved the sbin unmount).

You can always still use an older Magisk version on outdated ROMs.

[jackfagner]

I don't think so. Magisk itself depends on the sbin it mounts, meaning it can't unmount it at any point. (Figured that out after flashing magisk 5 times and a lot of swearing until I moved the sbin unmount).

You can always still use an older Magisk version on outdated ROMs.

Did you try to unmount sbin (busybox umount /sbin) right before extract_elf_ramdisk in the recovery boot script (init.sh) ?

[Myself5]

The question was if Magisk could unmount it, and the answer is "No" :P The solution I pushed works fine, and to be perfectly honest I don't care about the old init solution. That has been obsolete for quite a few years now.

[nailyk-fr]

yes. And only an updated kernel image should fix.

Thanks for the fix  @Myself5 & @topjohnwu
I see it has already been merged into lineage. I guess RR too. Maybe you are still in touch with Max to get a fix into CarbonRom ?
Gonna push it into omni and trigger some rebuilds.

[okij1]

@Myself5: Thanks for your exlanation. :-)

@nailyk-fr: CarbonROM is maintained by Myself5 himself (no pun intended ;-))
About the Z2 support I'm not sure. After Max dropped it, @drakonizer picked it up and released a cr-7.0 (Pie) beta build for the Z2. I hope that it will also be an official CarbonROM build again (@Myself5 ?)

[Myself5]

It's not merged into Lineage yet, but it's merged in all Carbon devices. Z3, Z3C and Z2 will all get official Opal releases, only device that might get dropped is the Z2 Tablet. However, could we possibly refrain from this topic here? It's not related to the Magisk issue at all.

[okij1]

That's great and sorry for off-topic. :-)

[8lexliu]

@Myself5 Hi, it's kinda rude to ask, but do you think if issue1537(https://github.com/topjohnwu/Magisk/issues/1537) is related to this one?

[Myself5]

Highly doubt it.