Build: CPU throttling from orangepi-4.9 branch hangs CPU boot in OPi-Zero

[martinayotte]

As described in https://forum.armbian.com/index.php/topic/2808-orange-pi-zero-went-to-the-market/page-5#entry21298 and later posts, I didn't why OPiZero hangs during boot if CPU throttling is present.

Today, I've rebuilt both OPiOne and OPiLite and the issue doesn't occurred there, so why OPiZero have such behaviour ?

[halmartin]

@martinayotte what u-boot sources are you using? If I take u-boot v2016.11 from denx and add orangepi_zero_defconfig, dts from Linux,
and lower CONFIG_SYS_CLK_FREQ I don't have freeze on boot.
u-boot.patch.txt

Here is modified v2016.11 u-boot which boots on my Orange Pi Zero:
u-boot-v2016.11.zip

[martinayotte]

The v2016.11 is already the branch used by the build process, and the change you've done in boards/sunxi/board.c is identical to what I did.
But you DTS hasn't the CPU throttling defined, and the defconfig doesn't have the SYS_CLK defined too.
So, mix of those become the issue.

[megous]

What dts for Orange Pi Zero are you using?

[martinayotte]

In Armbian build process, the u-boot is still using the one from PiOne.
For the kernel, I've also tried the sun8i-h2-orangepi-zero.dts provided by halmartin in this PR
https://github.com/megous/linux/pull/7/files
For the kernel DTS, as soon there is the &cpu0 with the operating-points for throttling, it hangs during boot.

[megous]

I was getting these hangs because of incorrect setup of the gpio-regulator which created a small dip in voltage during initialization. But I haven't tested Opi One since 4.6, so perhaps there's another issue there in the newer kernels. I debugged it by adding a printk/stack print on the voltage change function. Perhaps that can help here too.

[martinayotte]

I've tested OPiOne and OPiLite, both don't have the disease, only OPiZero has it.
I'm currently rebuilding whole new image using halmartin patch.

[megous]

Did you try verifying the 1.1/1.3V voltages on your board?

[martinayotte]

Nope ! But I probably will if no other clues...
That make me think : has someone confirmed that PL6 is really the VCC_CPUX on PiZero ?
(because since the schematic is already wrong and not updated for led_opc on PA17 instead of PA15, maybe there are other discrepancies)

[megous]

The kernel does not boot reliably on Orange Pi One either. If you setup a boot loop There are random crashes in multiple places. Mostly somewhere in the clock code. Sometimes it boots. I suspect I'm getting more failures with rising temperature. I'll look into it.

[megous]

I would say that this is not specific to Opi Zero.

[halmartin]

@megous

I was getting these hangs because of incorrect setup of the gpio-regulator which created a small dip in voltage during initialization.

Where was this in the kernel? Can you point me to the file?

@martinayotte

In Armbian build process, the u-boot is still using the one from PiOne.

I merged the dts from the OPi Zero in kernel with the dts of OPi One in u-boot. Couldn't add &cpu0 section due to errors. I have no clue about dts files...

@megous

Did you try verifying the 1.1/1.3V voltages on your board?

I can confirm that the voltage is changing between 1.1/1.3V. I verified with a multimeter on 1V2C that the voltage is dropping to 1.1V at the time the system hangs during boot.

If you change the CONFIG_SYS_CLK_FREQ in u-boot to something lower, then the system will boot even if the voltage on 1C2V drops (and the Orange Pi Zero kernel dts is used defining the operating-points). But this is only with my u-boot modifications above and my kernel branch. I haven't tried from the Armbian build system yet because I want a working u-boot/kernel before I push to Armbian.

With my kernel tree with the above u-boot patches and my Zero is booting reliably. I haven't tried stress testing the system after boot though.

Even changing the kernel dts operating-points to all 1.3V, my Zero still hangs on boot if u-boot initialises the SoC to a high frequency.

@martinayotte

has someone confirmed that PL6 is really the VCC_CPUX on PiZero ?

No, and I brought this up in: https://forum.armbian.com/index.php/topic/2808-orange-pi-zero-went-to-the-market/?p=21341

The schematic of the OPi Zero says PL6 is NC. I checked the OPi One and it's the same: PL6 is NC. So I don't know how someone confirmed that PL6 was connected to FB on SY8113B in the OPi One. Surely this has been checked by someone?

I am not sure how to confirm that PL6 is connected to SY8113B. Measure the FB pin on the SY8113B?

because since the schematic is already wrong

The OPi One schematic block diagram says it has the SY8106A (later it is correctly shown as SY8113B), so I would say we cannot trust the schematic to be correct.

[martinayotte]

@halmartin

If you change the CONFIG_SYS_CLK_FREQ in u-boot to something lower

Using Armbian build process, I did the same you did and even more, I've printed the CONFIG_SYS_CLK_FREQ before setting it manually to 480000000, and the printf shows that it was already to that value, but unfortunately kernel was still hanging with my image.

The schematic of the OPi Zero says PL6 is NC

You probably don't have the same versions of schematics I have, because on my copies all PiOne/PiLite/PiZero have them connected as CPUX-VSET which is toggling a MOSFET on resistor voltage divider.
But as I mentioned earlier, can we trust those schematics since even the led-status of the PiZero is on the wrong GPIO ?

[halmartin]

@martinayotte

Using Armbian build process, I did the same you did and even more, I've printed the CONFIG_SYS_CLK_FREQ before setting it manually to 480000000, and the printf shows that it was already to that value, but unfortunately kernel was still hanging with my image.

Are you building the kernel from the PR, or the kernel in my fork? I haven't built the kernel in the PR because I can't figure out how to clone the other commits pulled into the PR.

From your feedback it feels like we're working with different kernels...

You probably don't have the same versions of schematics I have

I'm using the schematics available from sunxi wiki: http://linux-sunxi.org/File:Orange-Pi-Zero-Schanetics-v1_11.pdf

If there is a newer version available somewhere public I would love to know. I couldn't find anything else on the first few pages of Google results for "Orange Pi Zero schematic".

[martinayotte]

About the kernel, Armbian is also using megous's orangepi-4.9 branch. After trying 1 time with the PR, I've putting aside to figure out the issue, so plain Armbian for now, but still hangs.

For the schematic, it is the same version. Take a look at page 6 lower-right corner, PL6 is there connected to CPUX-VSET, which is used later on page 7 to drive the MOSFET.

[megous]

I have found several issues.

1) There's a longstanding PLL change issue, that was supposed to be worked around by the clk notifier patch that switches CPU CLK to 24MHz while the CPU PLL is being changed. But from my tests this did not fix it. If the PLL dividers change, I still get lockups. The only correct solution is to apply CPU PLL changes correctly in a staged manner, or to set NKMP clock to values that never change the dividers.

2) Incorrect setting for ramp-delay on the regulator. The system didn't wait for voltage to rise or fall before changing CPU frequency.

I fixed 1) by patching u-boot and linux kernel to not use dividers (m=1 always). Kernel uses p divider on very low frequencies, but the frequency spike on change should not cause instabilities with my OPP setup, even in theory.

Now my orange pi one seems to be perfectly stable so far.

[megous]

I've pushed regulator ramp-delay fix to the orange-pi-4.9 branch. CPU CLK fix needs a patch to u-boot. After this patch, you should be able to boot with any supported frequency, without lockups.

From 7f5071f906f79bdc99d6b4b0ccf0cb280abe740b Mon Sep 17 00:00:00 2001
From: Ondrej Jirman <[email protected]>
Date: Tue, 20 Dec 2016 11:25:12 +0100
Subject: [PATCH] sunxi: h3: Fix PLL1 setup to never use dividers

Kernel would lower the divider on first CLK change and cause the
lock up.
---
 arch/arm/mach-sunxi/clock_sun6i.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/arch/arm/mach-sunxi/clock_sun6i.c b/arch/arm/mach-sunxi/clock_sun6i.c
index 50fb302..91aa2a0 100644
--- a/arch/arm/mach-sunxi/clock_sun6i.c
+++ b/arch/arm/mach-sunxi/clock_sun6i.c
@@ -94,11 +94,10 @@ void clock_set_pll1(unsigned int clk)
    int k = 1;
    int m = 1;

-   if (clk > 1152000000) {
-       k = 2;
-   } else if (clk > 768000000) {
+   if (clk >= 1368000000) {
        k = 3;
-       m = 2;
+   } else if (clk >= 768000000) {
+       k = 2;
    }

    /* Switch to 24MHz clock while changing PLL1 */
-- 
2.10.2

[martinayotte]

Oh ! Thanks, Megous !
I will give it a try when I get chance !

[bz31]

Thank you very much Megous !
Using Buildroot, I could not boot my Nanopi NEO with cpu_freq enabled (CONFIG_CPUFREQ_DT = y). With this patch for uboot and your last kernel orange-pi-4.9 branch, cpu-freq works !

[Icenowy]

I think it's a CCU issue, which I faced also on A33.

http://lists.infradead.org/pipermail/linux-arm-kernel/2016-November/470260.html seems to be the fix

[martinayotte]

megous's orange-pi-4.9 branch already had this patch, and I added the above u-boot patch, and still facing hangs or oops with my PiZero

[martinayotte]

Ok ! I think I found my own issue : although it was working fine with one of my previous build image, those recent days, I was not able to even load this previous image. I changed USB Micro cable and suddenly it was working again. So, the other cable was a crappy one, was working, but in long run became defective ... Grrrr !

[zador-blood-stained]

I think the throttling issue should be solved.