build 🚀 - Move mvebu DEFAULT, NEXT and DEV branch to next kernel (LTS) and U-boot

@g-provost Does Default branch for Clearfog also need to be changed to 4.14 and U-Boot 2018?
I prefer to keep the default as it is now until someone from SolidRun change it.

aprayoga on 26 Jun 2019

@aprayoga We are already sharing the same kernel so there is no reason we can't the move for all mvebu. As for the U-Boot we can keep it segregated as it is already.

I'm trying to get someone from Solidrun to help... or anyone that has a clearfog board.

g-provost on 26 Jun 2019

Current default is 4.4.y which is IMO obsolete for some time and shell be removed, yes. Current (mainline + patches) next becomes default (we don't need legacy 4.14.y or?) ... while legacy/next u-boot is actually 2013 https://github.com/SolidRun/u-boot-armada38x I did few tests with a modern 2018/2019 but was long time ago. This means latest u-boot with both next and dev.

igorpecovnik on 26 Jun 2019

Yes we will do as the description says

Move Armbian Default to 4.14 Mainline + Uboot 2018
Move Armbian Next to 4.19 Mainline + Uboot 2019
Move Armbian Dev to 5.x Mainline + Uboot 2019

It's just that for clearfog we need someone to do the work for u-boot 2018 and 2019.

g-provost on 26 Jun 2019

Actually for 2018 it's already ready : https://github.com/SolidRun/u-boot/tree/v2018.01-solidrun-a38x

g-provost on 26 Jun 2019

@aprayoga Is your worry related to armbian boot script support by u-boot from solidrun ?

g-provost on 26 Jun 2019

Actually for 2018 it's already ready : https://github.com/SolidRun/u-boot/tree/v2018.01-solidrun-a38x

Aha, didn't notice. Zador made Clearfog implementation back then to have support for SD, eMMC, UART, M2 boot ... it would be nice to keep that. That's I am worrying about.

igorpecovnik on 26 Jun 2019

We will only upgrade U-boot if we manage to keep all the booting option supported so far. Same question apply to Helios4 with SPI boot since @aprayoga is already facing challenge with U-boot 2019. We shall see by end of the week.

g-provost on 27 Jun 2019

SPI support on U-boot 2019.04 broken since this commit https://github.com/u-boot/u-boot/commit/30c4383da3fa2652e551b0610e71360394aa51e1 (v2019.04-rc1)

aprayoga on 28 Jun 2019

SPI support on U-boot 2019.04 broken since this commit u-boot/u-boot@30c4383 (v2019.04-rc1)

Good to know. We have to keep at least one (2018?) capable of SPI boot ... perhaps this will be fixed in 2019.07?

igorpecovnik on 28 Jun 2019

Let's stay for now on stock Solidrun U-boot 2018.11. It seems alright ... Clearfog PRO was tested briefly, now fine-tuning, testing, fixing and implementing Helios4.

igorpecovnik on 30 Jun 2019

@igorpecovnik , are you going change helios4? please hold it. I'm working on helios4. I will adapt to clearfog configuration.

I have fixed the SPI issue on 2019.04 U-Boot.

aprayoga on 1 Jul 2019

No, I wait for your changes.

I have fixed the SPI issue on 2019.04 U-Boot.

Cool. Then attach NEXT and DEV to it and I will test on Clearfog. I hope we will be able to make a change this week, next I don't have access to hardware.

igorpecovnik on 1 Jul 2019

@g-provost

Move Armbian Default to 4.14 Mainline + Uboot 2018

Linux Kernel switched to 4.14 on commit https://github.com/armbian/build/commit/4f685a6fe0c05af4041b89091f818d5abdfa49a4 by @igorpecovnik

U-Boot switched to v2018.01 on commit https://github.com/armbian/build/commit/4f685a6fe0c05af4041b89091f818d5abdfa49a4 by @igorpecovnik for Clearfog (source pointed to SolidRun's repo)
U-Boot switched to v2018.11 on commit https://github.com/armbian/build/commit/e7c2942b1da7c714240bb82af524557eda646cd6 for Helios4 (source pointed to Helios4 repo)

aprayoga on 1 Jul 2019

@aprayoga ok so this is for DEFAULT branch.You are doing NEXT branch now ? If you can NEXT branch should use 2019.04 for both clearfog and Helios4.

g-provost on 1 Jul 2019

@igorpecovnik , I'm working on U-Boot 2019.04 under DEV branch.
I put the patch to re-enable SPI support on this commit https://github.com/armbian/build/commit/74b95ba16f16704f7b071114aad84a0c76d7d70f
I'm not sure if I need to put it in NEXT as it based on SolidRun's v2018.01 which doesn't seem to have this problem

aprayoga on 1 Jul 2019

@igorpecovnik Please test the @aprayoga U-boot 2019.04 patch on Clearfog. This way we can decide if we move NEXT branch to 2019.04 for both Helios4 and Clearfog.

Also @aprayoga rise a good point, upgrade path (through apt upgrade) should be tested.

g-provost on 1 Jul 2019

BTW another question, shouldn't we leave Debian 9 Stretch as CLI_TARGET for DEFAULT ? Debian 9 Stretch is LTS until 2022.

g-provost on 1 Jul 2019

I am out office atm and I am not sure my Clearfog has a SPI flash on it. Can't test that feature. Debian Buster is under testing and things just work: https://github.com/armbian/config#software-testings ... I guess those are minor problems from our perspective. I would vote to move on to Buster.

igorpecovnik on 1 Jul 2019

Ok then let's go for Buster for all branches then. We should take in consideration dist-upgrade path for user running Stretch.

@igorpecovnik So does your Clearfog has SPI ?

g-provost on 3 Jul 2019

dist-upgrade

... is users own risk until today and if will remain this way. Feel free to test for your board and note that. So far, it worked and I don't expect troubles.

No SPI chip.

igorpecovnik on 3 Jul 2019

initial testing buster-next on helios4.
~~- lm75 compiled as kernel module but not loaded during runtime.~~
- helios4 udev rules broken for cpu temp so the fancontrol failed to start. the device name changed from armada_thermal into f10e4078.thermal
Fixed on commit 77b9d71

~~- warning/info from systemd regarding fancontrol:~~

systemd[1]: /lib/systemd/system/fancontrol.service:9: PIDFile= references path below legacy directory /var/run/, updating /var/run/fancontrol.pid \xe2\x86\x92 /run/fancontrol.pid; please update the unit file accordingly.

Fixed on commit 9df9902

WoL working
ATA leds working
USB & Heartbeat leds working
~~- somehow i always got IP address outside my subnet~~

aprayoga on 5 Jul 2019

Testing upgrade/update scenario from current NEXT branch Armbian_5.75_Helios4_Debian_stretch_next_4.14.98.
Local network test repo created and added to the system and then run
sudo apt upgrade

Kernel updated to 4.19.56-mvebu and following issue also occurred

lm75 compiled as kernel module but not loaded during runtime.
helios4 udev rules broken for cpu temp so the fancontrol failed to start. the device name changed from armada_thermal into f10e4078.thermal

The fixes mentioned on previous comment does not applied to the system.

sudo apt dist-upgrade did not upgrade the system.

root@helios4:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@helios4:~#

aprayoga on 8 Jul 2019

did not upgrade the system.

You mean kernel package did upgrade while board bsp didn't?

igorpecovnik on 8 Jul 2019

on previous comment, i did
apt dist-upgrade
after
apt upgrade
and nothing happened.

Today i redo the test, from fresh install Armbian_5.75_Helios4_Debian_stretch_next_4.14.98 then do
apt dist-upgrade. _(After adding local test repo, created using aptly)_

The packages updated but still on Stretch.

 _   _      _ _           _  _
| | | | ___| (_) ___  ___| || |
| |_| |/ _ \ | |/ _ \/ __| || |_
|  _  |  __/ | | (_) \__ \__   _|
|_| |_|\___|_|_|\___/|___/  |_|

Welcome to Debian Stretch with Armbian Linux 4.19.56-mvebu
System load:   0.38 0.10 0.03   Up time:       0 min
Memory usage:  2 % of 2016MB    IP:            192.168.0.113
CPU temp:      42°C
Usage of /:    4% of 30G

New to Armbian? Check the documentation first: https://docs.armbian.com
Last login: Mon Jul  8 06:24:59 2019 from 192.168.0.181

root@helios4:~#

Note: I did not set/modified the repo source to point to buster repo as instructed on
https://www.debian.org/releases/stable/armhf/release-notes/ch-upgrading.en.html#upgrade-process

aprayoga on 8 Jul 2019

@aprayoga Unfortunately there is no trivial way like in Ubuntu to upgrade Debian version in one command. We will have to give instruction to user running Stretch on how they could upgrade to Buster (at their own risk). As you suggested we could point them to this : https://www.debian.org/releases/stable/armhf/release-notes/ch-upgrading.en.html#upgrade-process

g-provost on 8 Jul 2019

Debian repostory is tagged to release, not to status (stable)
https://github.com/armbian/build/blob/master/lib/general.sh#L138
That's why dist-upgrade doesn't work. Shell we change this?

igorpecovnik on 8 Jul 2019

Perhaps by adding:

if $RELEASE = bionic || buster then release="stable" else release=$RELEASE

igorpecovnik on 8 Jul 2019

I'll change the repo to stable and test it

aprayoga on 8 Jul 2019

@igorpecovnik I wouldn't recommend to use status name, it can be confusing. Plus anyway it's too late to do that now since existing install have their APT pointing to release name.

g-provost on 8 Jul 2019

I wouldn't recommend to use status name, it can be confusing.

True. Leave as is then.

igorpecovnik on 8 Jul 2019

@igorpecovnik Actually since Buster ship with LK 4.19, wouldn't it make more sense that for LK 4.14 (Armbian Default branch) we stick to Stretch ? This way it give the choice to user to still use Stretch since some 3rd party app might not yet be fully supported on Buster. What do you think ?

g-provost on 9 Jul 2019

We never were attached with a kernel version upstream and we don't need now. Armbian kernel is not near the same as the one that is shipped with Buster. Well, in case of this particular hardware is not that changed, but still - configuration is ours, there are small changes and there are additional drivers which are not present in Buster stock kernel. We ship user space with best kernel and different user space is here only to give users an option to stay behind for just any reason they have and loose nothing hardware/low level security wise.

If one for some reason wants older kernel he can change that freely: armbian-config -> alternative kernels

igorpecovnik on 9 Jul 2019

Ok understood. But then from user perspective and looking at mvebu situation, if both Default and Next image are based on Buster, what would be the reason for a user to choose the Default image since both 4.14 and 4.19 kernel provide full support of the hardware ?

I think for Helios4 we will provide Default / Stretch along Next / Buster. The reason is that some 3rd party app (provided through armbian-config) don't seem to install / work yet on Buster.

Also I'm a bit lost on the purpose of CLI_TARGET= variable in board config. For example in helios4.conf we have
CLI_TARGET="buster,bionic:next"
What does CLI_TARGET= does exactly ? Because I can still choose Stretch with Next kernel in the Armbian build tool. Thanks to shade some light here.

g-provost on 9 Jul 2019

purpose of CLI_TARGET

Tells build system which targets to build. It's an internal function so its not specially documented.

CLI_TARGET="buster,bionic:next"

... will build all 2 combinations of CLI images with Debian Buser and Ubuntu Bionic and a NEXT kernel target. Whatever that target is. Currently this is 4.14.y but when this is merged, we will have NEXT with 4.19.y

Stretch with NEXT which is already present for download will automatically upgraded to 4.19.y once we release an update.

igorpecovnik on 9 Jul 2019

Tells build system...

By build system you are talking about some nodes that are doing the nightly build ?

Stretch with NEXT which is already present for download will automatically upgraded to 4.19.y once
we release an update.

Not sure I understand. We don't want NEXT (4.19.y) with Stretch. I was just suggesting to have DEFAULT (4.14.y) with Stretch.

g-provost on 9 Jul 2019

By build system you are talking about some nodes that are doing the nightly build ?

Yes. It's when one use BUILD_ALL="yes" while for nightly CLI_BETA_TARGET variable is read.

We don't want NEXT (4.19.y) with Stretch. I was just suggesting to have DEFAULT (4.14.y) with Stretch.

That's upon what we will choose to build under current build logic and later expose (just some) for download, which is set elsewhere, manually in the website management. Stretch/Xenial/Jessie/Bionic/... with NEXT or DEFAULT or DEV can be build or user can upgrade or downgrade to it.

igorpecovnik on 9 Jul 2019

👍1

Hi there,

Testing for the clearfog pro board,
I made a fresh clone of the armbian build repo today, and using "./compile.sh EXPERT=yes LIB_TAG=mvebu" built a default, next and dev image. All using Debian Stretch. Left the kernel config untouched.

Afterwards I burned the images to a sdcard and tested them one by one.
Seems pretty good working so far. All versions boot up, and stuff like wan, lan ports (switched) are working. From 4.19.y onward the sfp port is also working out of the box. I noticed some problems in kernel 4.19.y, my sata card did not get detected. In 5.1.y everything worked. But that might be a problem attached to loading the wrong device tree. I will conduct further testing on that tomorrow.

Attached I put an excel sheet with my exact testing configuration, used hardware and armbianmonitor links.

As a next step I will clone my current production setup and try to upgrade it to newest kernel & uboot step-by-step. Will then report back, if and when it breaks.

Let me know if you want me to test anything specific. My board also does have the spi flash.

mvebu_testing.xlsx

Image Name | U Boot | Kernel | Debian | Armbian | Monitor | Board (by uboot) | Boot | UART | WAN | LAN | SFP | SPI Flash | mPCIe WiFi | mPCIe SATA | m.2 / sata
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
Armbian_5.90_Clearfogpro_Debian_stretch_default_4.14.132 | 2018.01 | 4.14.132 | stretch | 5.90 | http://ix.io/1O7l | SolidRun Clearfog Pro | ok | ok | ok | ok | not working | not found | ok | ok | ok
Armbian_5.90_Clearfogpro_Debian_stretch_next_4.19.57 | 2019.04 | 4.19.57 | stretch | 5.90 | http://ix.io/1O7t | SolidRun Clearfog | ok | ok | ok | ok | ok | ok | ok | not working | ok
Armbian_5.90_Clearfogpro_Debian_stretch_dev_5.1.16 | 2019.04 | 5.1.16 | stretch | 5.90 | http://ix.io/1O7x | SolidRun Clearfog | ok | ok | ok | ok | ok | ok | ok | ok | ok
Used Hardware: | | | | | | | | | | | | | | |
Clearfog Pro w/o eMMC | | | | | | | | | | | | | | |
mPCIe SATA Adapter with ASM1062 Chip | | | | | | | | | | | | | | |
mPCIe WLAN Adapter with QCA986X Chip (WLE600VX) | | | | | | | | | | | | | | |
SFP Cisco compatible GLC-T 1000M RJ45 Adapter | | | | | | | | | | | | | | |

heisath on 9 Jul 2019

@count-doku thanks.
regarding mPCIE SATA that failed, it's a PCIE to SATA adapter, right? not mSATA.

aprayoga on 10 Jul 2019

Correct, using the 4.19.57 kernel the ASM1062 mPCIe to SATA adapter is not recognized in linux. But this might just be loading the wrong device tree. I think the clearfog base has only 1 pcie. I will test again later.

mSATA I cannot test, as I have no mSATA drive.

heisath on 10 Jul 2019

So back from testing. I have some bad news unfortunately.

As everthing apart from mPCIe seemed to work ok I focused on testing the different mpcie sata devices I have (on the different kernel / uboot versions).

Tested devices:

mPCIe SATA Adapter with ASM1062 Chip
mPCIe SATA Adapter with Marvel 88SE9215 Chip

Things are not working as good as expected:
Uboot 2018 & Kernel 4.14 AND Uboot 2019 & Kernel 5.1:
After uboot, the "starting kernel" prompt hangs a long time (~2min), then the pcie devices show up in 'lspci' but the attached drives are not recognized.

lspci: 
00:02.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04)
00:03.0 PCI bridge: Marvell Technology Group Ltd. Device 6828 (rev 04)
01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
02:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)

dmesg:
[    7.389247] ata5.00: qc timeout (cmd 0xec)
[    7.389255] ata5.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[    7.865250] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   17.885244] ata5.00: qc timeout (cmd 0xec)
[   17.885251] ata5.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[   17.885254] ata5: limiting SATA link speed to 3.0 Gbps
[   18.361249] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[   50.141244] ata5.00: qc timeout (cmd 0xec)
[   50.141250] ata5.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[   50.617249] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[   51.097250] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   56.285245] ata6.00: qc timeout (cmd 0xec)
[   56.285252] ata6.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[   56.761249] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   66.781243] ata6.00: qc timeout (cmd 0xec)
[   66.781250] ata6.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[   66.781253] ata6: limiting SATA link speed to 3.0 Gbps
[   67.265249] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[   99.293243] ata6.00: qc timeout (cmd 0xec)
[   99.293249] ata6.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[   99.773248] ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 320)

Have you changed anything in the kernel / uboot settings for pcie or sata? With the current (official) next branch everything works flawless.

Interestingly on uboot 2019 but with kernel 4.19. the mpcie sata cards don't get recognized at all.

I also found another uboot 2019 problem / error:

UBOOT 2019.1

Found U-Boot script /boot/boot.scr
2948 bytes read in 207 ms (13.7 KiB/s)
## Executing script at 03000000
Boot script loaded from mmc
256 bytes read in 167 ms (1000 Bytes/s)
19383 bytes read in 428 ms (43.9 KiB/s)
4673244 bytes read in 1041 ms (4.3 MiB/s)
5672184 bytes read in 1222 ms (4.4 MiB/s)
## Loading init Ramdisk from Legacy Image at 02880000 ...
   Image Name:   uInitrd
   Created:      2019-07-09  13:08:39 UTC
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    4673180 Bytes = 4.5 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 02040000
   Booting using the fdt blob at 0x2040000
   Loading Ramdisk to 0fb8b000, end 0ffffe9c ... OK
ERROR: reserving fdt memory region failed (addr=2040000 size=6b000)
   Loading Device Tree to 0fb1d000, end 0fb8afff ... OK

UBOOT 2018 
No error, notice different address

Found U-Boot script /boot/boot.scr
2948 bytes read in 105 ms (27.3 KiB/s)
## Executing script at 00200000
Boot script loaded from mmc
172 bytes read in 84 ms (2 KiB/s)
20484 bytes read in 212 ms (93.8 KiB/s)
4683313 bytes read in 343 ms (13 MiB/s)
5455608 bytes read in 384 ms (13.5 MiB/s)
## Loading init Ramdisk from Legacy Image at 01800000 ...
   Image Name:   uInitrd
   Created:      2019-07-09   7:05:39 UTC
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    4683249 Bytes = 4.5 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 00100000
   Booting using the fdt blob at 0x100000
   Loading Ramdisk to 0fb88000, end 0ffff5f1 ... OK
   reserving fdt memory region: addr=100000 size=6b000
   Loading Device Tree to 0fb1a000, end 0fb87fff ... OK

Notice how with uboot 2019 there is some error when reserving fdt memory region. Wrong address maybe?

I attached a full boot up log with uboot, kernel, dmesg. This is loading uboot 2018 with 4.14.
There are also some kernel warnings in the dmesg output. The exact position and device differs from boot to boot. They appeared for sfp, pcie, phy...

boot.txt

heisath on 10 Jul 2019

Things are not working as good as expected:
Uboot 2018 & Kernel 4.14 AND Uboot 2019 & Kernel 5.1:
After uboot, the "starting kernel" prompt hangs a long time (~2min), then the pcie devices show up in 'lspci' but the attached drives are not recognized.

I wanted to confirm, both configuration/combination has this same issue, right?

Have you changed anything in the kernel / uboot settings for pcie or sata? With the current (official) next branch everything works flawless.

The current official next is LK 4.14 with U-Boot version 2013.01, is it?

Interestingly on uboot 2019 but with kernel 4.19. the mpcie sata cards don't get recognized at all.

I'll check whether the driver compiled or maybe compiled but somehow not loaded automatically.

I also found another uboot 2019 problem / error:
UBOOT 2019.1

## Flattened Device Tree blob at 02040000
   Booting using the fdt blob at 0x2040000
   Loading Ramdisk to 0fb8b000, end 0ffffe9c ... OK
ERROR: reserving fdt memory region failed (addr=2040000 size=6b000)
   Loading Device Tree to 0fb1d000, end 0fb8afff ... OK
Notice how with uboot 2019 there is some error when reserving fdt memory region. Wrong address maybe?

Yes this error appear after commit cd2429c. This changes needed to be able to run bootscript based on boot-marvell.cmd due to these lines:

setenv fdt_high 0xffffffff
setenv initrd_high 0xffffffff

If the load address is not updated, it would failed to boot.
This is more concerned for existing user. For new user /new sd card image, we don't need this changes as it already based on boot-mvebu-next.cmd. On Helios4, we had trouble during Armbian 5.68-5.70 where we upgraded the U-boot from 2013 to mainline 2018 because of this incompatibility, the system no longer boot.
I tested on Helios4 and nothing affected so i dismissed this more as warning, maybe I should have compare what in RAM and on the file.

aprayoga on 11 Jul 2019

I wanted to confirm, both configuration/combination has this same issue, right?

Yes the mpcie issue is the same.

The current official next is LK 4.14 with U-Boot version 2013.01, is it?

Yes, more specifically LK4.14.121.

Regarding uboot fdt memory region error, maybe this is a mere warning and has no influence on the other things noticed.

As a next step to further narrow down the mpcie problem would a be a viable step if I took my running LK4.14.121 / uboot 2013.1 install and updated only the uboot to 2018? Or just leave the uboot and update the kernel to yours (lk4.14.123 I think)?
That way we could confirm wether this issue is uboot or kernel related.

heisath on 11 Jul 2019

As a next step to further narrow down the mpcie problem would a be a viable step if I took my running LK4.14.121 / uboot 2013.1 install and updated only the uboot to 2018? Or just leave the uboot and update the kernel to yours (lk4.14.123 I think)?
That way we could confirm wether this issue is uboot or kernel related.

I'm more suspicious to u-boot since it's the one who configured the SerDes, so i think leave the kernel and update the u-boot to 2018

aprayoga on 11 Jul 2019

Didn't have much time left this morning, so I just cloned my production setup which is uboot 2013.1 with lk4.14.121 and upgraded uboot to 2018.01.

Doesn't boot at all. See log below:

U-Boot SPL 2018.01-armbian (Jul 09 2019 - 08:32:52)
High speed PHY - Version: 2.0
Detected Device ID 6828
board SerDes lanes topology details:
 | Lane #  | Speed |  Type       |
 --------------------------------
 |   0    |  3   |  SATA0       |
 |   1    |  0   |  SGMII1      |
 |   2    |  5   |  PCIe1       |
 |   3    |  5   |  USB3 HOST1  |
 |   4    |  5   |  PCIe2       |
 |   5    |  0   |  SGMII2      |
 --------------------------------
:** Link is Gen1, check the EP capability
PCIe, Idx 1: Link upgraded to Gen2 based on client capabilities
PCIe, Idx 2: detected no link
High speed PHY - Ended Successfully
DDR3 Training Sequence - Ver TIP-1.29.0
DDR3 Training Sequence - Switching XBAR Window to FastPath Window
DDR3 Training Sequence - Ended Successfully
Trying to boot from MMC1


U-Boot 2018.01-armbian (Jul 09 2019 - 08:32:52 +0200)

SoC:   MV88F6828-A0 at 1600 MHz
DRAM:  1 GiB (800 MHz, 32-bit, ECC not enabled)
MMC:   mv_sdh: 0
*** Warning - bad CRC, using default environment

Model: SolidRun Clearfog
Board: SolidRun ClearFog Pro
SCSI:  AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3 impl SATA mode
Net:
Warning: ethernet@70000 (eth1) using random MAC address - d2:63:b4:2d:9d:71
eth1: ethernet@70000
Warning: ethernet@30000 (eth2) using incremented MAC address - d2:63:b4:2d:9d:72
, eth2: ethernet@30000
Warning: ethernet@34000 (eth3) using incremented MAC address - d2:63:b4:2d:9d:73
, eth3: ethernet@34000
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot/boot.scr
1604 bytes read in 97 ms (15.6 KiB/s)
## Executing script at 00200000
Boot script loaded from mmc
load - load binary file from a filesystem

Usage:
load <interface> [<dev[:part]> [<addr> [<filename> [bytes [pos]]]]]
    - Load binary file 'filename' from partition 'part' on device
       type 'interface' instance 'dev' to address 'addr' in memory.
      'bytes' gives the size to load in bytes.
      If 'bytes' is 0 or omitted, the file is read until the end.
      'pos' gives the file byte position to start reading from.
      If 'pos' is 0 or omitted, the file is read from the start.
load - load binary file from a filesystem

Usage:
load <interface> [<dev[:part]> [<addr> [<filename> [bytes [pos]]]]]
    - Load binary file 'filename' from partition 'part' on device
       type 'interface' instance 'dev' to address 'addr' in memory.
      'bytes' gives the size to load in bytes.
      If 'bytes' is 0 or omitted, the file is read until the end.
      'pos' gives the file byte position to start reading from.
      If 'pos' is 0 or omitted, the file is read from the start.
4709951 bytes read in 341 ms (13.2 MiB/s)
5464240 bytes read in 382 ms (13.6 MiB/s)
## Loading init Ramdisk from Legacy Image at 01800000 ...
   Image Name:   uInitrd
   Created:      2019-07-07  16:49:18 UTC
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    4709887 Bytes = 4.5 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK

Starting kernel ...

Uncompressing Linux... done, booting the kernel.

Error: unrecognized/unsupported machine ID (r1 = 0x00000000).

Available machine support:

ID (hex)        NAME
ffffffff        Generic DT based system
ffffffff        Marvell Armada 39x (Device Tree)
ffffffff        Marvell Armada 380/385 (Device Tree)
ffffffff        Marvell Armada 375 (Device Tree)
ffffffff        Marvell Armada 370/XP (Device Tree)
ffffffff        Marvell Dove

Please check your kernel config and/or bootloader.

Might try again this afternoon with a fresh image from dl.armbian.com and upgrade from there.

heisath on 11 Jul 2019

This is the incompatibility issue I mentioned earlier.
could you try

setenv boot_a_script 'load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; setenv boot_interface ${devtype};source ${scriptaddr}'
setenv loadaddr 0x02000000
setenv fdt_addr ${fdt_addr_r}
saveenv

on U-Boot prompt. It supposed to fix it and the system can boot.

aprayoga on 11 Jul 2019

What should I run after 'saveenv'?

I tried 'run boot_a_script':
```=> run boot_a_script
* Bad device : 0x200000 *

Executing script at 00200000

Wrong image format for "source" command

And just the normal 'boot' and 'reset':
```=> boot
switch to partitions #0, OK
mmc0 is current device
Scanning mmc 0:1...
Found U-Boot script /boot/boot.scr
1604 bytes read in 97 ms (15.6 KiB/s)
## Executing script at 00200000
Boot script loaded from mmc
221 bytes read in 85 ms (2 KiB/s)
20596 bytes read in 223 ms (89.8 KiB/s)
4709951 bytes read in 339 ms (13.2 MiB/s)
5464240 bytes read in 382 ms (13.6 MiB/s)
## Loading init Ramdisk from Legacy Image at 01800000 ...
   Image Name:   uInitrd
   Created:      2019-07-07  16:49:18 UTC
   Image Type:   ARM Linux RAMDisk Image (gzip compressed)
   Data Size:    4709887 Bytes = 4.5 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 00100000
   Booting using the fdt blob at 0x100000
   Using Device Tree in place at 00100000, end 00108073

Starting kernel ...

Uncompressing Linux... done, booting the kernel.

Error: unrecognized/unsupported machine ID (r1 = 0x00000000).

Available machine support:

ID (hex)        NAME
ffffffff        Generic DT based system
ffffffff        Marvell Armada 39x (Device Tree)
ffffffff        Marvell Armada 380/385 (Device Tree)
ffffffff        Marvell Armada 375 (Device Tree)
ffffffff        Marvell Armada 370/XP (Device Tree)
ffffffff        Marvell Dove

Please check your kernel config and/or bootloader.

After not getting uboot 2018 to work with the old kernel I tried the other way round.

I reflashed uboot 2013 and updated the kernel to 4.14.133.
Noticed a few interesting things:

boots ok as expected
mpcie works with all devices (so the problem seems to be the uboot indeed)
the phy error appears in linux boot log, as posted before:

[    2.464149] ata4: SATA max UDMA/133 mmio [mem 0xf10e0000-0xf10e1fff] port 0x180 irq 45
[    2.472788] libphy: Fixed MDIO Bus: probed
[    2.477151] libphy: orion_mdio_bus: probed
[    2.484186] mvneta_bm f10c8000.bm: Buffer Manager for network controller enabled
[    2.493010] mvneta f1070000.ethernet eth0: Using device tree mac address 00:50:43:25:fb:84
[    2.504037] mvneta f1030000.ethernet eth1: Using device tree mac address 00:50:43:84:25:2f
[    2.512400] ------------[ cut here ]------------
[    2.517043] WARNING: CPU: 0 PID: 1 at drivers/net/phy/swphy.c:133 swphy_read_reg+0x27/0xf0
[    2.525330] Modules linked in:
[    2.528393] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.14.133-mvebu #3
[    2.535021] Hardware name: Marvell Armada 380/385 (Device Tree)
[    2.540964] [<c010cf49>] (unwind_backtrace) from [<c01090cb>] (show_stack+0xb/0xc)
[    2.548553] [<c01090cb>] (show_stack) from [<c06cc981>] (dump_stack+0x69/0x78)
[    2.555796] [<c06cc981>] (dump_stack) from [<c012031d>] (__warn+0xa5/0xb8)
[    2.562689] [<c012031d>] (__warn) from [<c01203d3>] (warn_slowpath_null+0x13/0x18)
[    2.570278] [<c01203d3>] (warn_slowpath_null) from [<c051a633>] (swphy_read_reg+0x27/0xf0)
[    2.578564] [<c051a633>] (swphy_read_reg) from [<c051ab8d>] (fixed_mdio_read+0x99/0xa0)
[    2.586588] [<c051ab8d>] (fixed_mdio_read) from [<c051a3f1>] (mdiobus_read+0x35/0x44)
[    2.594438] [<c051a3f1>] (mdiobus_read) from [<c0519d21>] (get_phy_device+0x35/0x13c)
[    2.602287] [<c0519d21>] (get_phy_device) from [<c051aca7>] (fixed_phy_register+0x5f/0x104)
[    2.610661] [<c051aca7>] (fixed_phy_register) from [<c05b2f53>] (of_phy_register_fixed_link+0xff/0x160)
[    2.620081] [<c05b2f53>] (of_phy_register_fixed_link) from [<c0523c5b>] (mvneta_probe+0x3d7/0x770)
[    2.629063] [<c0523c5b>] (mvneta_probe) from [<c04b8c3b>] (platform_drv_probe+0x33/0x70)
[    2.637174] [<c04b8c3b>] (platform_drv_probe) from [<c04b7c27>] (driver_probe_device+0x15f/0x1e8)
[    2.646067] [<c04b7c27>] (driver_probe_device) from [<c04b7d09>] (__driver_attach+0x59/0x5c)
[    2.654527] [<c04b7d09>] (__driver_attach) from [<c04b69d3>] (bus_for_each_dev+0x33/0x6c)
[    2.662723] [<c04b69d3>] (bus_for_each_dev) from [<c04b74a3>] (bus_add_driver+0xff/0x164)
[    2.670920] [<c04b74a3>] (bus_add_driver) from [<c04b8295>] (driver_register+0x31/0x84)
[    2.678946] [<c04b8295>] (driver_register) from [<c091b2d3>] (mvneta_driver_init+0x63/0x80)
[    2.687318] [<c091b2d3>] (mvneta_driver_init) from [<c0101991>] (do_one_initcall+0x3d/0x118)
[    2.695778] [<c0101991>] (do_one_initcall) from [<c0900bd3>] (kernel_init_freeable+0x177/0x1dc)
[    2.704498] [<c0900bd3>] (kernel_init_freeable) from [<c06d9aa7>] (kernel_init+0x7/0xd0)
[    2.712610] [<c06d9aa7>] (kernel_init) from [<c01062f9>] (ret_from_fork+0x11/0x38)
[    2.720213] ---[ end trace 08d953a54cff3b4d ]---
[    2.724844] ------------[ cut here ]------------
[    2.729478] WARNING: CPU: 0 PID: 1 at drivers/net/phy/swphy.c:133 swphy_read_reg+0x27/0xf0
[    2.737763] Modules linked in:
[    2.740824] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G        W       4.14.133-mvebu #3
[    2.748670] Hardware name: Marvell Armada 380/385 (Device Tree)
[    2.754605] [<c010cf49>] (unwind_backtrace) from [<c01090cb>] (show_stack+0xb/0xc)
[    2.762193] [<c01090cb>] (show_stack) from [<c06cc981>] (dump_stack+0x69/0x78)
[    2.769432] [<c06cc981>] (dump_stack) from [<c012031d>] (__warn+0xa5/0xb8)
[    2.776325] [<c012031d>] (__warn) from [<c01203d3>] (warn_slowpath_null+0x13/0x18)
[    2.783913] [<c01203d3>] (warn_slowpath_null) from [<c051a633>] (swphy_read_reg+0x27/0xf0)
[    2.792198] [<c051a633>] (swphy_read_reg) from [<c051ab8d>] (fixed_mdio_read+0x99/0xa0)
[    2.800222] [<c051ab8d>] (fixed_mdio_read) from [<c051a3f1>] (mdiobus_read+0x35/0x44)
[    2.808071] [<c051a3f1>] (mdiobus_read) from [<c0519d31>] (get_phy_device+0x45/0x13c)
[    2.815919] [<c0519d31>] (get_phy_device) from [<c051aca7>] (fixed_phy_register+0x5f/0x104)
[    2.824291] [<c051aca7>] (fixed_phy_register) from [<c05b2f53>] (of_phy_register_fixed_link+0xff/0x160)
[    2.833708] [<c05b2f53>] (of_phy_register_fixed_link) from [<c0523c5b>] (mvneta_probe+0x3d7/0x770)
[    2.842689] [<c0523c5b>] (mvneta_probe) from [<c04b8c3b>] (platform_drv_probe+0x33/0x70)
[    2.850798] [<c04b8c3b>] (platform_drv_probe) from [<c04b7c27>] (driver_probe_device+0x15f/0x1e8)
[    2.859691] [<c04b7c27>] (driver_probe_device) from [<c04b7d09>] (__driver_attach+0x59/0x5c)
[    2.868150] [<c04b7d09>] (__driver_attach) from [<c04b69d3>] (bus_for_each_dev+0x33/0x6c)
[    2.876347] [<c04b69d3>] (bus_for_each_dev) from [<c04b74a3>] (bus_add_driver+0xff/0x164)
[    2.884544] [<c04b74a3>] (bus_add_driver) from [<c04b8295>] (driver_register+0x31/0x84)
[    2.892567] [<c04b8295>] (driver_register) from [<c091b2d3>] (mvneta_driver_init+0x63/0x80)
[    2.900939] [<c091b2d3>] (mvneta_driver_init) from [<c0101991>] (do_one_initcall+0x3d/0x118)
[    2.909397] [<c0101991>] (do_one_initcall) from [<c0900bd3>] (kernel_init_freeable+0x177/0x1dc)
[    2.918116] [<c0900bd3>] (kernel_init_freeable) from [<c06d9aa7>] (kernel_init+0x7/0xd0)
[    2.926226] [<c06d9aa7>] (kernel_init) from [<c01062f9>] (ret_from_fork+0x11/0x38)
[    2.933824] ---[ end trace 08d953a54cff3b4e ]---

This looks like a warning only though.

I hope knowing that the above mentioned mpcie problems originate in uboot helps you. I will have to put my clearfog back in production setup tomorrow so probably no more testing, until the ones requested from solidrun arrive.

bootlog_201301_4_14_133.txt

heisath on 11 Jul 2019

Hey,

as I received the boards today, I powered up the clearfog pro and tried each kernel / uboot variant again. Biggest noted difference: Correct board detection, thanks @aprayoga. Saw your commit, works.

Apart from that, I used Debian Buster. Works like a charm, no (new) problems compared to Stretch!

SFP and SPI Flash are both working going forward from Kernel 4.19 as expected. M.2 SSD is working always. WAN and switched LAN ports are also fine.

Biggest issue is still the mPCIe recognition. I was unable to get the extender cards to really work. Same problems as before (obviously, as nothing was changed. With the wifi cards the driver doesn't get loaded correctly (iw list shows no devices) - SATA card gets detected on 4.14. and 5.xx but is not working with attached drives.

How are we going to procede here? I can leave a board powered up, and give you ssh access if you need to check out some logs or so. Or do you have an idea already what might be wrong?

I think we should concentrate on getting the boards to work with new images first. Afterwards think about the update route from old versions.

Attached logs of current boot, dmesg, etc. If you need anything specific just let me know.
Armbian_5.90_Clearfogpro_Debian_buster_default_4.14.133.txt
Armbian_5.90_Clearfogpro_Debian_buster_dev_5.1.18.txt
Armbian_5.90_Clearfogpro_Debian_buster_next_4.19.59.txt

I will try the clearfog base tomorrow, and maybe on the weekend put together some testbed with an relais to powercycle the boards and some additional sbc to access the serial ports via ssh.

heisath on 17 Jul 2019

Let's focus on Default and Next, and put aside Dev for now.

Can you provide a dedicated list of what works and doesn't work for each build (Default and Next).

Can you provide a picture of your setup ? How do you power supply the SSD/HDD connected to the mPCie SATA controller (Marvell 88SE9215) ?

g-provost on 18 Jul 2019

How are we going to procede here? I can leave a board powered up, and give you ssh access if you need to check out some logs or so. Or do you have an idea already what might be wrong?

Unfortunately I still have no idea what might be wrong. This morning I tried to see whether there is any configuration (in PCI subsystem) differences between u-boot 2013 and 2018. But on u-boot 2018 the pci command is not enabled and when i enabled it together with PCI support, got some compile error.

put together some testbed with an relais to powercycle the boards and some additional sbc to access the serial ports via ssh.

That would be nice

And regarding the phy error on LK 4.14, do you encounter the same issue with Armbian_5.75_Clearfogpro_Debian_stretch_next_4.14.98 ?

aprayoga on 18 Jul 2019

when i enabled it together with PCI support, got some compile error

Perhaps @rabeeh have some ideas what we are missing here?

igorpecovnik on 18 Jul 2019

Let's focus on Default and Next, and put aside Dev for now.

Ok. I'd go even further and say lets get default sorted out first. Because there might be a mess bigger than expected.

Can you provide a dedicated list of what works and doesn't work for each build (Default and Next).

Is listing this here ok, or did you mean some google docs sheet or so?

Default:

[x] Boot via sd card, with buster
[x] lan and wan port
[x] m.2
[x] usb
[x] cpu freq stepping (between 800MHz and 1.6 GHz)
[x] all the other things that worked before ;)
[ ] spi flash (does not appear in linux at all)
[ ] sfp (but I guess this cannot work here without backporting alot)
[ ] mpcie sata card (this gets detected fine, but attached drives don't)
[ ] wifi card (this gets detected fine, but driver is not loaded. )

[x] Boot via sd card, with buster
[x] lan and wan port
[x] m.2
[x] usb
[x] cpu freq stepping (between 800MHz and 1.6 GHz)
[x] sfp working very good. Including dhcp etc.
[x] spi flash (read / write tested)
[ ] mpcie sata card
[ ] wifi card

I focused on default now.

Regarding wifi:
Requires installation of 'firmware-atheros' (did so on older versions as well)
After that it works on LK4.14.98 but on LK4.14.113 dmesg gives:

[    7.824666] pci 0000:00:03.0: enabling device (0140 -> 0142)
[    7.824685] ath10k_pci 0000:02:00.0: enabling device (0140 -> 0142)
[    7.824929] ath10k_pci 0000:02:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[    7.849390] zram1: detected capacity change from 0 to 526585856
[    8.099943] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/pre-cal-pci-0000:02:00.0.bin failed with error -2
[    8.099948] ath10k_pci 0000:02:00.0: Falling back to user helper
[    8.389518] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/cal-pci-0000:02:00.0.bin failed with error -2
[    8.389523] ath10k_pci 0000:02:00.0: Falling back to user helper
[    8.394551] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/QCA988X/hw2.0/firmware-6.bin failed with error -2
[    8.394555] ath10k_pci 0000:02:00.0: Falling back to user helper
[    8.407939] ath10k_pci 0000:02:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043222ff sub 0000:0000
[    8.407947] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 0 tracing 0 dfs 0 testmode 0
[    8.408484] ath10k_pci 0000:02:00.0: firmware ver 10.2.4-1.0-00041 api 5 features no-p2p,raw-mode,mfp,allows-mesh-bcast crc32 f43fa422
[    8.454013] ath10k_pci 0000:02:00.0: Direct firmware load for ath10k/QCA988X/hw2.0/board-2.bin failed with error -2
[    8.454021] ath10k_pci 0000:02:00.0: Falling back to user helper
[    8.459085] ath10k_pci 0000:02:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[    8.909405] Adding 514240k swap on /dev/zram1.  Priority:5 extents:1 across:514240k SSFS
[    9.138638] zram0: detected capacity change from 0 to 52428800
[   10.230400] random: crng init done
[   10.230407] random: 7 urandom warning(s) missed due to ratelimiting
[   10.432337] EXT4-fs (zram0): mounted filesystem without journal. Opts: discard
[   11.132892] ath10k_pci 0000:02:00.0: failed to receive control response completion, polling..
[   12.156891] ath10k_pci 0000:02:00.0: Service connect timeout
[   12.156898] ath10k_pci 0000:02:00.0: failed to connect htt (-110)
[   12.239842] ath10k_pci 0000:02:00.0: could not init core (-110)
[   12.239877] ath10k_pci 0000:02:00.0: could not probe fw (-110)

Device does not appear in 'iw list' then.

Regarding 88se9215:
Works on LK 4.14.98; can not read drives on LK 4.14.113 / uboot2018

[    1.957959] ahci 0000:01:00.0: AHCI 0001.0000 32 slots 4 ports 6 Gbps 0xf impl SATA mode
[    1.957970] ahci 0000:01:00.0: flags: 64bit ncq sntf led only pmp fbs pio slum part sxs
[    1.959507] scsi host4: ahci
[    1.959735] scsi host5: ahci
[    1.959956] scsi host6: ahci
[    1.960189] scsi host7: ahci
[    1.960315] ata5: SATA max UDMA/133 abar m2048@0xe0040000 port 0xe0040100 irq 54
[    1.960321] ata6: SATA max UDMA/133 abar m2048@0xe0040000 port 0xe0040180 irq 54
[    1.960326] ata7: SATA max UDMA/133 abar m2048@0xe0040000 port 0xe0040200 irq 54
[    1.960332] ata8: SATA max UDMA/133 abar m2048@0xe0040000 port 0xe0040280 irq 54
[    2.272883] ata8: SATA link down (SStatus 0 SControl 300)
[    2.280871] ata6: SATA link down (SStatus 0 SControl 300)
[    2.280896] ata7: SATA link down (SStatus 0 SControl 300)
[    2.444484] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    7.652573] ata5.00: qc timeout (cmd 0xec)
[    7.652583] ata5.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[    8.131864] ata5: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[   18.147984] ata5.00: qc timeout (cmd 0xec)
[   18.147993] ata5.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[   18.147999] ata5: limiting SATA link speed to 3.0 Gbps
[   18.617905] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[   50.146595] ata5.00: qc timeout (cmd 0xec)
[   50.146604] ata5.00: failed to IDENTIFY (I/O error, err_mask=0x4)
[   50.616913] ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 320)

BUT and this is a big but.

The errors above are only there, if only on card is plugged in. So either sata or wifi. BUT if I plug in both cards bigger problems arise.

If I plug in wifi into the pcie closest to the cpu, only the wifi one gets detected. Not the sata adapter.
If I reverse the order, so sata is closest to cpu and wifi farthest. Nothing gets detected at all.
Above points are about linux detection, uboot always shows both devices.

But this is another problem that exists longer already, this is also true for LK 4.14.98. There even was some forum post about this a while back (here: https://forum.armbian.com/topic/9291-clearfogpro-lost-pcie-sata-card-after-update/) where we unfortunately told the guy that it has to be a problem on his end. It seems like it isn't... I will answer the forum post about that.

Can you provide a picture of your setup ? How do you power supply the SSD/HDD connected to the mPCie SATA controller (Marvell 88SE9215) ?

Using the same power supply as for the clearfog (12V 4A) and for the 5V rail I use some adjustable buck converter. Voltages are ok at all components (measured with oscilloscope, voltage and ripple is fine). The hdd is also running.
This is also working fine (hdd gets detected) with LK4.14.98 (Armbian 5.75)

IMG_20190718_174303

And regarding the phy error on LK 4.14, do you encounter the same issue with
Armbian_5.75_Clearfogpro_Debian_stretch_next_4.14.98 ?

No phy error here.

The test platform with ssh->serial access might be put together saturday (at the earliest). I have upcoming exams so not very much time.

Apart from that to conclude my very long post:

Board itself is working with new uboot & kernel.
pcie cards are not really working. But there are many different overlapping problems. Maybe we have to split this up.
Would it make sense to get everything working but with one card only attached? And then try to get multiple running?

Btw. the pcie device detection also fails if you insert some other cards. Just tried the wle600vx and some intel ac 3160. Inserted both -> nothing shows up with 'lspci'. Uboot detects both.

heisath on 18 Jul 2019

Tested some more. To minimize confusion and possible side effects I built a fresh image from armbian/mvebu default branch (uboot 2018.1, lk 4.14.133).

I booted the clearfogpro without any pci devices, installed the 'firmware-atheros' package via apt, then I tried all possible configurations and recorded uboot, lspci and dmesg output.

To clarify, PCIe Idx2 is the slot closest to the cpu.
IMG_20190718_163632

The issue got a bit better (why ever?). Now atleast the cards are not 'hiding' one another in linux.

Checkout the table and attached logs.

no_pcie.txt

wifi_in_idx1.txt
wifi_in_idx2.txt

sata_in_idx1.txt
sata_in_idx2.txt

wifi_in_idx2_and_sata_in_idx1.txt
sata_in_idx2_and_wifi_in_idx1.txt

heisath on 19 Jul 2019

Can you provide same test with Armbian_5.75_Clearfogpro_Debian_stretch_next_4.14.98 in order to compare.

Also will be nice to compare your findings / issue against Clearfog Base.

g-provost on 19 Jul 2019

Did that really quick, results are different from uboot 2018 and lk4.14.113...

no_pcie.txt
sata_in_idx1.txt
sata_in_idx2.txt
wifi_in_idx1.txt
wifi_in_idx2.txt
sata_in_idx2_and_wifi_in_idx1.txt
wifi_in_idx2_and_sata_in_idx1.txt

From the results I think, this is some driver problem in Linux. The uboot detections looks almost the same. I ordered another mPCIe to SATA card, (different chipset ASM1061) and tomorrow @aprayoga and I will try to debug this further together. I also noticed the atheros driver is different between stretch and buster so we might want to compare stretch uboot 2013 lk4.14.98 and stretch uboot 2018 lk4.14.113 first. This will eliminate another variable.

EDIT: Tested this with Debian Stretch @ Uboot 2018.1 and LK4.14.113, gives different behaviour on uboot level. But same behaviour in linux. This is confusing. But same outcome as with buster.

no_pcie.txt
sata_in_idx1.txt
sata_in_idx2.txt
wifi_in_idx1.txt
wifi_in_idx2.txt
sata_in_idx2_and_wifi_in_idx1.txt
wifi_in_idx2_and_sata_in_idx1.txt

Comparing against clearfog base will also follow on the weekend.

heisath on 19 Jul 2019

Hi,

Can you please double check that the pcie Force receiver detect patch is
applied?

https://github.com/SolidRun/lede-project/blob/sr-testing/package/boot/uboot-mvebu/patches/0014-mvebu-a38x-Force-receiver-detected-on-PCIe-lanes.patch

A similar patch was applied by Baruch to u-boot mainline few months ago.

On Fri, Jul 19, 2019, 10:36 AM count-doku notifications@github.com wrote:

Tested some more. To minimize confusion and possible side effects I built
a fresh image from armbian/mvebu default branch (uboot 2018.1, lk 4.14.133).

I booted the clearfogpro without any pci devices, installed the
'firmware-atheros' package via apt, then I tried all possible
configurations and recorded uboot, lspci and dmesg output.

To clarify, PCIe Idx2 is the slot closest to the cpu.
[image: IMG_20190718_163632]
https://user-images.githubusercontent.com/52237708/61517264-23957080-aa07-11e9-8933-5ca99672defe.jpg

The issue got a bit better (why ever?). Now atleast the cards are not
'hiding' one another in linux.

Checkout the table and attached logs.
Uboot lspci dmesg
no card detected no link, both slots just the pci bridges

Wifi in Idx1 detected no link, both slots found in idx1 driver loads,
but cannot connect
Wifi in Idx2 detects idx2 found in idx2 driver loads, but cannot
connect

Sata in Idx1 detected nothing in idx2 (correct); no message regarding
idx1 found in idx1 driver loads, attached device fails
Sata in Idx2 detected nothing in idx1 (correct); no message regarding
idx2 found in idx2 driver loads, attached device fails

Both with:
sata in 1; wifi in 2 detected nothing in idx2; no message regarding idx1 found
both correctly same as above, does not work
sata in 2; wifi in 1 detected nothing in idx1; no message regarding idx2 found
both correctly same as above, does not work

no_pcie.txt https://github.com/armbian/build/files/3409942/no_pcie.txt

wifi_in_idx1.txt
https://github.com/armbian/build/files/3409943/wifi_in_idx1.txt
wifi_in_idx2.txt
https://github.com/armbian/build/files/3409945/wifi_in_idx2.txt

sata_in_idx1.txt
https://github.com/armbian/build/files/3409944/sata_in_idx1.txt
sata_in_idx2.txt
https://github.com/armbian/build/files/3409946/sata_in_idx2.txt

wifi_in_idx2_and_sata_in_idx1.txt
https://github.com/armbian/build/files/3409947/wifi_in_idx2_and_sata_in_idx1.txt
sata_in_idx2_and_wifi_in_idx1.txt
https://github.com/armbian/build/files/3409941/sata_in_idx2_and_wifi_in_idx1.txt

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/armbian/build/issues/1426?email_source=notifications&email_token=AAJSS4HLSJ2KSCXA4MTCAF3QAFVGNA5CNFSM4H3OVOR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2K3CCI#issuecomment-513126665,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJSS4HWHJLT5UGJDFSG723QAFVGNANCNFSM4H3OVORQ
.

rabeeh on 19 Jul 2019

@aprayoga and I did further debugging today. He found the bug in uboot:
"The issue for SATA, actually simple. the u-boot 2018.01 is not compiled with PCI support.
I enabled it and recompiled"
Marvell 88SE9215 (sata expander) is now working, but as I understand it, there is more work to be done here.
"now my concern is,
on u-boot 2018 onward, the board configuration are using device tree.
under PCIe node there is properties of reset-gpio. On current driver i don't see the reset gpio being use.
in u-boot 2013, the gpio is executed under custom board code
I will try to modify the u-boot 2018 driver to make use of this gpio" (aprayoga)

The wifi card (wle600vx, qca988x) still has problems loading the driver, still trying to figure that out.
Intel wifi cards are already working (intel ac-3160 & intel n-2200 tested), so this does not seem to be a general problem.

heisath on 20 Jul 2019

Good news. After aprayoga pushed his uboot changes, I build new images for Clearfog Base & Pro, default and next options.

With default (Uboot 2018.1 and Kernel 4.14.133) about everything that can be expected is working.

Default:

[x] Boot via sd card, with buster
[x] lan and wan port
[x] m.2
[x] usb
[x] cpu freq stepping (between 800MHz and 1.6 GHz)
[x] all the other things that worked before ;)
[x] mpcie sata card (tested marvell 88se9215 and asmedia 1061)
[x] mpcie wifi card (tested wle600vx (qca988x) and intel ones)
[ ] spi flash (does not appear in linux at all)
[ ] sfp (but I guess this cannot work here without backporting alot)

I guess this is all that can be made working with reasonable amount of time. Everything above was tested in clearfogpro and base and the pcie cards in every possible combination.

-> This can be shipped.

[x] Boot via sd card, with buster
[x] lan and wan port
[x] m.2
[x] usb
[x] cpu freq stepping (between 800MHz and 1.6 GHz)
[x] sfp working very good. Including dhcp etc.
[x] spi flash (read / write tested)
[ ] mpcie sata card
[ ] wifi card

With next unfortunately the mcpie is not working (same problem left as before with default). I attached full log.

cfplog.txt

I did not test mSATA because I am unsure how to switch pcie to msata. The documentation has a dead link only. https://docs.armbian.com/Hardware_Marvell/

heisath on 22 Jul 2019

This can be shipped.

OK, I will make images once merged to master.

With next unfortunately the mcpie is not working

Since mpci is most important feature on this board ... Can you check how good is 5.x.y out of the box?

igorpecovnik on 23 Jul 2019

@igorpecovnik Please let @aprayoga gives green light when everything is ready to be built for mvebu.

Let's make it a rule that it's the board maintainer team lead who gives green light when updates can be built and published.

g-provost on 23 Jul 2019

@count-doku You said via PM that you have another mPCIe SATA controller (based on ASM1061), do you also see the same issue under Next build with the ASM1061 controller ?

Here I attach an email reply from Baruch which is linked to the issue. Might no be useful but for the sake of tracking better to attach it :

There is a known issue with the Marvell PCIe SATA controller on 32-bit
Armada 38x. The same controller works fine with Armada 8040 on Clearfog
GT-8K. I suspect that the driver has a 32-bit compatibility issue
somewhere.

An ASMedia PCIe SATA controller I tested works fine on Clearfog.

The Helios4 board uses no PCIe interface. All SATA disks are attached to
the native Armada 388 SATA controller.

I tested the same WLE900VX card with ASMedia SATA controller attached to
both mini-PCIe slots on Clearfog Pro. Both cards were detected. The SATA
controller seems to probe successfully. The WLE900VX card driver shows
the same error messages shown in the Armbian bug report.

[ 2.124633] ath10k_pci 0000:02:00.0: enabling device (0140 -> 0142)
[ 2.131140] ath10k_pci 0000:02:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[ 2.362597] ath10k_pci 0000:02:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043202ff sub 0000:0000
[ 2.371881] ath10k_pci 0000:02:00.0: kconfig debug 0 debugfs 0 tracing 0 dfs 0 testmode 0
[ 2.380861] ath10k_pci 0000:02:00.0: firmware ver 10.2.4-1.0-00041 api 5 features no-p2p,raw-mode,mfp,allows-mesh-bcast crc32 f43fa422
[ 2.425709] ath10k_pci 0000:02:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[ 4.566986] ath10k_pci 0000:02:00.0: failed to receive control response completion, polling..
[ 5.616981] ath10k_pci 0000:02:00.0: Service connect timeout
[ 5.622661] ath10k_pci 0000:02:00.0: failed to connect htt (-110)
[ 5.757716] ath10k_pci 0000:02:00.0: could not init core (-110)
[ 5.763677] ath10k_pci 0000:02:00.0: could not probe fw (-110)

g-provost on 23 Jul 2019

Let's make it a rule that it's the board maintainer team lead who gives green light when updates can be built and published.

Agree.

igorpecovnik on 23 Jul 2019

👍2

Hi,

do you also see the same issue under Next build with the ASM1061 controller ?

Yes. It does not matter on next what controller is in use. Same problem.

email reply from Baruch which is linked to the issue

Don't think that matters much because both the marvell controller and the asmedia and wle600vx are working fine under uboot 2018.1 lk4.14.133 now. Might be wrong though.

heisath on 23 Jul 2019

@count-doku , could you test on U-Boot 2018.01 and LK 4.19?
If there is no problem, maybe we should stay on 2018.01 for Clearfog NEXT

regarding SPI, there is known issue when SATA and SPI accessed concurrently. On Helios4 we use these block
https://github.com/armbian/build/blob/master/config/bootscripts/boot-mvebu-next.cmd#L66-L75
to enable the SPI flash while disabling internal SATA.
I suggest Clearfog also follow the same behaviour, disable SPI for normal use and when it needed, set spi_workaround=yes to enable SPI.

aprayoga on 24 Jul 2019

@igorpecovnik
We want to build and provide Stretch image for DEFAULT, because OMV is a key application for Helios4, and as of today OMV5 on Debian Buster is still in Beta and unstable. Installing OMV4 on Debian Buster is also not a viable solution either. Therefore we still need to offer a Stretch image to our user base.

We also decided to limit Helios4 Default image to Stretch only because there is no benefit for the user to choose between LK 4.14 + Buster vs LK 4.19 + Buster, user should go directly for LK 4.19 if they want to use Buster. Therefore to make the choice simple to user we will only offer 2x Armbian Debian Images variant for Helios4 which are easy to distinguish form each other :
• Armbian Debian DEFAULT : U-Boot 2018 + LK 4.14 + Stretch (OMV4 fully supported)
• Armbian Debian NEXT : U-Boot 2019 + LK 4.19 + Buster (OMV5 Beta available)

So this should also mean, there shouldn't be a linux-buster-root-helios4 in APT repo, only linux-stretch-root-helios4 and linux-buster-root-next-helios4

aprayoga on 24 Jul 2019

@igorpecovnik How to define CLI_TARGET to do as described above by @aprayoga ?

stretch : default
buster : next
bionic : next

g-provost on 24 Jul 2019

We want to build and provide Stretch image for DEFAULT

Of course. Stretch will remain relevant for some time in any case ...

Without RFC: CLI_TARGET="stretch,buster,bionic:default,next" which will create "some" overhead - all combinations.

On the download pages you select/choose manually what you want to expose/recommend so this is not a problem. Experienced users and wanna be Linux gurus will anyway try to upgrade to Buster or build on their own and there is little you can do about ... except make a note at the download pages/forum topic explaining why that is not a good idea.

Overhead can be manually removed or I build those targets manually.

igorpecovnik on 24 Jul 2019

could you test on U-Boot 2018.01 and LK 4.19?

Did that, by using the lk4.19 image and downgrading uboot to 2018.01. This helps somehow. Wifi works then, but neither of the sata cards does (asmedia & marvell). So this is no solution.

SPI switch sounds good to me.

heisath on 24 Jul 2019

Ok so let's conclude and close the chapter for Clearfog.

I actually remembered I had somewhere a Clearfog Pro board and also the required mPCIe devices to reproduce the same tests than @count-doku. With @aprayoga we spent the whole day compiling an exhaustive mPCIe test matrix with all the different U-Boot and Kernel version combination. Needless to say it's a tedious work, and we won't spend / waste any more time on this because beyond this point it's to Solidrun team (@rabeeh) to take action and fix the issues... because it's seem there were things that didn't even work in the first place.

Here the mPCIe test matrix on Clearfog
https://docs.google.com/spreadsheets/d/1VggzrfFibH0cmpSGW2FyoJW-d936Y2amwwKutUwX4-8

So the conclusion is that even if not perfect, the best combo is to use U-Boot 2018.01 for both Default and Next. This shouldn't introduce any regression on existing systems. For Dev we will leave U-Boot 2019.04.

So @aprayoga will update that in armbian clearfog config files, and then we should be ready to trigger the build.

As for the OS build choice for Clearfog, I would suggest the following image to be published

stretch : default
buster : default
buster : next
bionic : next

@igorpecovnik We will post again a message when all is green light to be built.

g-provost on 25 Jul 2019

👍2

You're test results fit mine. Seems there is some work to be done in U-Boot as well as Kernel for this to work perfectly.
But I agree let's not waste any more time, as some people are probably also waiting for SFP support etc. And at least the new default version will then be better as the current next.

One more question though, did you test anything related to upgrading from current versions? If not I can do some tests for this on the weekend. At least upgrading from current next, to new default should be possible without problems imho.
If we don't want to do this, I would suggest adding some note on the downloads page...

Apart from that, what is the best way to update the download site (https://www.armbian.com/clearfog/), @igorpecovnik ?
To include additional third party hardware tested and maybe add some warning header, that some hardware combinations don't work (yet). Maybe add a link to the gdocs table from @g-provost.
Armbian docs should also be updated when switching to new uboot & kernel.

heisath on 25 Jul 2019

To include additional third party hardware tested and maybe add some warning header

You got rights to edit both Clearfogs at www. You don't need me to add those warnings..

(technical instructions are written on the dashboard. change download images after they will be remade)

igorpecovnik on 25 Jul 2019

Great. I will edit the website after the new images are available. Will then also update docs.armbian.com (provide a PR there).

heisath on 25 Jul 2019

👍1

One more question though, did you test anything related to upgrading from current versions? If not I can do some tests for this on the weekend. At least upgrading from current next, to new default should be possible without problems imho.

Not extensively, just updating u-boot 2013.01 on Armbian_5.75_Clearfogpro_Debian_stretch_next_4.14.98 to u-boot 2018.01 to test whether it can boot/run the boot script. Please test the upgrade scenario.

btw, I saw
[ 7.967237] cfg80211: failed to load regulatory.db
on your Armbian_5.90_Clearfogpro_Debian_buster_next_4.19.59.txt log.
Does install _wireless-regdb_ and _crda_ solve this? We can put these package as dependencies if it's not installed yet.

aprayoga on 26 Jul 2019

failed to load regulatory.db
on your Armbian_5.90_Clearfogpro_Debian_buster_next_4.19.59.txt log.
Does install wireless-regdb and crda solve this? We can put these package as dependencies if it's not installed yet.

IMHO its safe to ignore that error ... but if wireless-regdg or crda solve that, we can add it. BTW. We are working on a minimal image (branch: mini https://github.com/armbian/build/compare/mini) and can be added there. Its already operational, but things has to be tested before we can merge up. Changes are how the images are debootstrapped. Not just for "mini" target.

igorpecovnik on 26 Jul 2019

Tried installing wireless-regdb and crda, both were already installed. The message stays. But wifi is working.

EDIT:
Got around to do some upgrade testing, works flawless.

I started with a fresh current next image (uboot 2013.1 lk 4.14.98), then updated to new next (uboot 2019.1 lk 4.19), as this would be the way users get the update via apt. Rebooting using uboot 2013.1 & lk 4.19 works. Wifi works also. Then I wrote uboot 2019.1 to the sd (as armbian-config would), rebooted again, still working. Wifi stops working though.
After that I downgraded to default (uboot 2018.1 lk 4.14.134), users could do this via armbian-config. Rebooting still using uboot2019.1 lk4.14.134 works, wifi and sata not working (as expected).
Last I wrote uboot 2018.1 to sd (as armbian-config would), rebooted again. Now with uboot2018.1 lk4.14.134. Boot is ok, wifi and sata working!

So the upgrade path seems ok.

Users currently on next will also get upgraded to the new next. When they need wifi & sata working they can just downgrade to default.
Users currently on default will get upgraded to new default, where everything works.

IMO the new versions are ok. There are probably be some people getting confused when after the upgrade something stops working, but with the notice on the download page & in docs we will probably get about 80% of them. If the rest posts in the forum we can still help them there.

heisath on 26 Jul 2019

Also tested above upgrade on uboot 2018.1 lk 4.19.x (the versions you proposed). Results are the same update works. mPCIe availability is according to your google sheet.

heisath on 30 Jul 2019

👍1

@igorpecovnik Just to confirm, now that you have merged mvebu branch to master, we are all good to trigger official build and publish of mvebu family images and packages.

CLI_TARGET are already properly configured to cover all the required dpkg amd images combo, but please for the images can you limit publication on the download page to only the following ones

For Helios 4

stretch : default
buster : next
bionic : next

For Clearfog Pro/Base

stretch : default
buster : default
buster : next
bionic : next

_Note : Even though for Helios4 we don't want anymore to publish Stretch Next image, we still need the debs to be built and published in the REPO. I'm saying that in case you are not using CLI_TARGET and building manually according to the list above._

g-provost on 31 Jul 2019

Hi,

I have added the warning notes regarding mPCIe on the download pages of clearfogpro and clearfogbase. I have also changed the docs to reflect new kernel versions and made a pull request there. Please check if everything is ok.

I could not figure out how to change the download option on the download page to access the new images - I guess this is something @igorpecovnik has to do.

heisath on 31 Jul 2019

on the download page to only the following ones

Images created, adjusted download links, removed unneeded.

I could not figure out how to change the download option

Download links point to new images after they are uploaded and after torrents and links are regenerated. Since I was in a manual mode, this was not done automatically.

Welcome to type some short summary what was done https://github.com/armbian/documentation/blob/master/docs/Release_Changelog.md

igorpecovnik on 31 Jul 2019

Repository also updated.

igorpecovnik on 31 Jul 2019

@igorpecovnik I posted a message a bit earlier here, either I'm starting to lose my mind, but cannot see it suddenly. Anyway I was saying thanks a lot for the builds.

But I just noticed that armbian-config is still 5.90 (from 7th July) in those built. Then all the little improvements I made in armbian-config are not in then :/

Regarding the dl.armbian.com could you please delete the following old links in Helios4 and Clearfog Pro/Base : Debian_stretch_next.*

BTW I think it would make actually sense to rebuild Clearfog_Bionic_default.

Also would be great if the armbian website (e.g https://www.armbian.com/helios4/) link to the latest builds.

Welcome to type some short summary what was done https://github.com/armbian/documentation/blob/master/docs/Release_Changelog.md

Done. I aslo updated the marvell page.

g-provost on 1 Aug 2019

But I just noticed that armbian-config is still 5.90 (from 7th July) in those built. Then all the little improvements I made in armbian-config are not in then :/

Actually it seems it's only softy file (aka debian-software) that it's not the last version. Rest of armbian-config looks up to date. Why ?

g-provost on 1 Aug 2019

But I just noticed that armbian-config is still 5.90 (from 7th July) in those built. Then all the little improvements I made in armbian-config are not in then :/

I forget about that. Will push that later on.

Regarding the dl.armbian.com could you please

This is auto generated. I have to delete actual images from archive.

Also would be great if the armbian website

But it should, unless there is a bug. Which link is not correct?

igorpecovnik on 1 Aug 2019

Debian Stretch link was wrong. Fixed.

igorpecovnik on 1 Aug 2019

Debian Stretch link was wrong. Fixed.

Thanks. However kernel version descriptions are not up to date. For Stretch should be _mainline kernel 4.14.y_, and for Buster and Bionic should be _mainline kernel 4.19.y_

g-provost on 1 Aug 2019

However kernel version descriptions are not up to date.

Fixed.

armbian-config was also added to repository ... rebuilding. Will be up in 10-15min.

igorpecovnik on 1 Aug 2019

👍1

Still only show 5.90 on https://apt.armbian.com/pool/main/a/armbian-config/

g-provost on 1 Aug 2019

Will be up in 10-15min.

You have to wait ;) Rebuilding hasn't finished yet.

igorpecovnik on 1 Aug 2019

😄1

Done. I was rebuilding two kernels in the back so it took little longer.

igorpecovnik on 1 Aug 2019

OK understood. Thanks.
Are you planning to rebuild the mvebu images also ?

g-provost on 1 Aug 2019

Are you planning to rebuild the mvebu images also ?

No, not that critical.

igorpecovnik on 1 Aug 2019

Oky. I guess I will rebuild on my side and publish the image with armbian-config 5.91 on our wiki. I like when fresh install are already using latest packages.

Ok then I guess we can finally close this issue.

g-provost on 1 Aug 2019

Build: Move mvebu DEFAULT, NEXT and DEV branch to next kernel (LTS) and U-boot

Most helpful comment

All 92 comments

Executing script at 00200000

Related issues