Linux: rpi-4.19-rt

[antonellocaroli]

there's going to be a 4.19 real time version?

[pelwell]

I hope so. We are currently relying on a volunteer - Tiejun Chen - to maintain the branch, and other commitments have meant that he hasn't had time recently.

[paul-1]

The RT patches apply cleanly to the 4.19.y branch. Been running pretty solid.

[vanfanel]

@paul-1 : I am trying to apply the RT patch to 4.19.y branch, too.
Looking at the contents of the patch TAR in , I see there is a list of patches available:

0009-kthread-convert-worker-lock-to-raw-spinlock.patch
0012-arm-Convert-arm-boot_lock-to-raw.patch
0023-EXP-rcu-Revert-expedited-GP-parallelization-cleverne.patch
0028-sched-migrate_disable-Add-export_symbol_gpl-for-__mi.patch
0032-signal-Revert-ptrace-preempt-magic.patch
0036-rt-Provide-PREEMPT_RT_BASE-config-switch.patch
0052-x86-Use-generic-rwsem_spinlocks-on-rt.patch
0059-preempt-Provide-preempt_-_-no-rt-variants.patch
0061-rt-Add-local-irq-locks.patch
0066-buffer_head-Replace-bh_uptodate_lock-for-rt.patch
0067-fs-jbd-jbd2-Make-state-lock-and-journal-head-lock-rt.patch
0070-genirq-Disable-irqpoll-on-rt.patch
0076-mm-page_alloc-rt-friendly-per-cpu-pages.patch
0077-mm-swap-Convert-to-percpu-locked.patch
0098-time-hrtimer-avoid-schedule_work-with-interrupts-dis.patch
0099-hrtimer-consolidate-hrtimer_init-hrtimer_init_sleepe.patch
0100-hrtimers-Prepare-full-preemption.patch
0101-hrtimer-by-timers-by-default-into-the-softirq-contex.patch
0102-sched-fair-Make-the-hrtimers-non-hard-again.patch
0103-hrtimer-Move-schedule_work-call-to-helper-thread.patch
0104-hrtimer-move-state-change-before-hrtimer_cancel-in-d.patch
0105-posix-timers-Thread-posix-cpu-timers-on-rt.patch
0115-rt-Increase-decrease-the-nr-of-migratory-tasks-when-.patch
0128-rtmutex-trylock-is-okay-on-RT.patch
0130-rtmutex-Handle-the-various-new-futex-race-conditions.patch
0135-locking-locktorture-Do-NOT-include-rwlock.h-directly.patch
0136-rtmutex-Add-rtmutex_lock_killable.patch
0137-rtmutex-Make-lock_killable-work.patch
0139-rtmutex-Avoid-include-hell.patch
0141-rtmutex-Provide-rt_mutex_slowlock_locked.patch
0142-rtmutex-export-lockdep-less-version-of-rt_mutex-s-lo.patch
0143-rtmutex-add-sleeping-lock-implementation.patch
0144-rtmutex-add-mutex-implementation-based-on-rtmutex.patch
0145-rtmutex-add-rwsem-implementation-based-on-rtmutex.patch
0146-rtmutex-add-rwlock-implementation-based-on-rtmutex.patch
0147-rtmutex-rwlock-preserve-state-like-a-sleeping-lock.patch
0148-rtmutex-wire-up-RT-s-locking.patch
0149-rtmutex-add-ww_mutex-addon-for-mutex-rt.patch
0151-locking-rt-mutex-fix-deadlock-in-device-mapper-block.patch
0152-locking-rt-mutex-Flush-block-plug-on-__down_read.patch
0153-locking-rtmutex-re-init-the-wait_lock-in-rt_mutex_in.patch
0155-rtmutex-annotate-sleeping-lock-context.patch
0168-rt-Improve-the-serial-console-PASS_LIMIT.patch
0183-rt-Introduce-cpu_chill.patch
0184-hrtimer-Don-t-lose-state-in-cpu_chill.patch
0185-hrtimer-cpu_chill-save-task-state-in-saved_state.patch
0196-seqlock-Prevent-rt-starvation.patch
0197-sunrpc-Make-svc_xprt_do_enqueue-use-get_cpu_light.patch
0207-printk-Make-rt-aware.patch
0214-kgdb-serial-Short-term-workaround.patch
0216-mm-rt-kmap_atomic-scheduling.patch
0219-arm-Enable-highmem-for-rt.patch
0227-x86-stackprotector-Avoid-random-pool-on-rt.patch
0228-random-Make-it-work-on-rt.patch
0240-sched-Add-support-for-lazy-preemption.patch
0242-x86-Support-for-lazy-preemption.patch
0245-arm-Add-support-for-lazy-preemption.patch
0246-powerpc-Add-support-for-lazy-preemption.patch
0247-arch-arm64-Add-lazy-preempt-support.patch
0249-drivers-block-zram-Replace-bit-spinlocks-with-rtmute.patch
0254-drm-radeon-i915-Use-preempt_disable-enable_rt-where-.patch
0259-cpuset-Convert-callback_lock-to-raw_spinlock_t.patch
0262-signals-Allow-rt-tasks-to-cache-one-sigqueue-struct.patch
0264-Linux-4.19.37-rt19-REBASE.patch

What set of patches do you apply to get a stable RT kernel on the Pi?

[antonellocaroli]

I applied this without any problems:

https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/4.19/patch-4.19.37-rt19.patch.gz

[vanfanel]

@antonellocaroli : Great! What other options, appart from GENERAL SETUP->PREEMPTION MODEL can you recommend me to change?
In KERNEL FEATURES->TIMER FREQUENCY, I already have 1000Hz (default in 4.14.y and 4.19.y branches anyway).

[antonellocaroli]

@vanfanel
you just need them...
but take a look at this to find out more:
https://rt.wiki.kernel.org/index.php/Frequently_Asked_Questions

[vanfanel]

@antonellocaroli : Thanks! I read that document time ago.
What I usually do to set realtime priority on a process is:
1) Set the rlimits system-wide accordingly so normal users can set realtime priority.
2) Execute the process with chrt -f 80 <executable name>
If you have any other observation regarding RT on GNU/Linux, please share it.

[guysoft]

Following, so I can get RealtimePi to point to it when ready

[NikiSchlifke]

+1

[guysoft]

Just realised no one actually mentioned @TiejunChina . So no wonder this has been hanging for 25 days.

[pelwell]

Tiejun is well aware of the situation. Pinging him (which I intentionally avoided) won't help.

[guysoft]

@pelwell Understood, it did not happen here and I could not guess at your private communication. Apologies.

I recommend notifying that next time, since I was starting to speculate on making my own PR branch.

[pelwell]

I was starting to speculate on making my own PR branch.

I wouldn't have a problem with that, if you can follow the same format as Tiejun did. The standard rpi- branches use merge commits to pull in upstream patches so they keep their commit IDs, and cherry-ick/rebase downstream commits to keep the commit history clear - you end up with two parallel streams of commits, "us" and "them".

As I wrote to Tiejun at the time:

"One of our internal rules is that, apart from back-porting patches to older kernels, we never change the upstream commits. This means that our development looks like this:

    L-L-L-L-L-L-L---L-L-L-L-L-L-L     = Upstream Linux
                 \       \       \
                  P-P-P-P-M-P-P---M-P = Downstream rpi- branch
                      ^   ^       ^
                      A   B       C

where L is an upstream Linux commit, P is a downstream Pi commit, and M is a merge commit. A represents the point where we stop rebasing our commits against the new kernel version, which corresponds to the point where it becomes our mainline branch. B and C are minor kernel version updates.

"Perhaps you could structure the -rt branch like this:

    L-L-L-L-L-L-L----L-L-L-L-L-L-L     = Upstream Linux
                \         \       \
                 R-R-R-R-M-M--M----M-M = rpi-rt branch
                        /    /     /
                 P-P-P-P--P-P-----P    = Downstream rpi- branch

"I hope you can understand my diagram. The important point is that the L and P commits in this diagram are identical to those in the previous one, with the same commit hash. The nice thing about this scheme is that by not rebasing your existing commits you only need to work on and think about (and fix) new commits."

If you can follow that pattern and prepare a branch in your repo then I'll be happy to bless it and call it rpi-4.19-rt. If, in doing so, you end up creating any tools to simplify the task then let me know - should it become easy enough then eventually the task may be brought in-house.

[paul-1]

Merging Upstream Linux commits into the RT branch is fun. I've never kept a copy of the rt-kernel git.

When maintaining for myself, when there is a new -rtxx version, I start from a clean rpi branch, and apply the rt patch set, then cherry pick any rpi-rt commits. I know that some don't like constant rebasing, but in my mind keeps the rt branch only a few commits different than the normal rpi branch.

[TheLongAndOnly]

@paul-1 As I understood you managed to get the RT patches to work for 4.19. I tried myself but ran into problems and hope you might have some hints.

As patching 4.19.y HEAD did not work, I pulled the matching commit for 4.19.37. These are the steps I took:

• git clone https://github.com/raspberrypi/linux.git --single-branch --branch rpi-4.19.y
• git checkout 19bb613acb9ad8e57593cad5118acaee117cc303
  (this is the commit changing to version 4.19.37)
• make bcm2709_defconfig (had to pull it from head, as it is not part of the commit)
  As the debos are not build, I updated to the correct commit of rpi-update
• sudo rpi-update 6cf2788e74b497c109865d052cd6fa68d094cf38

Unfortunately after installation of the new kernel I do not get past the rainbow screen. Changing kernel7.img back to the original one makes the system boot.

[guysoft]

I am not an expert, but looking over the tty serial output might give you
more information to what is failing at the rainbow screen.

[paul-1]

RPI is rebasing the 4.19.y tree, you cannot go back like that right now, unless you go and then pick all of the rpi downstream commits. did you try to apply the 4.19.37-rt19 against the current branch? Otherwise it's best to wait until the next rt patch comes out.

[TheLongAndOnly]

Ok, I didn't realize this, but it explains why the configs are missing. Applying 4.19.37-rt19 to the current branch does not work as there have been changes to mainline which makes the build fail.

[paul-1]

4.19.50-rt22 is out, it applies cleanly against the current rpi branch.

[pelwell]

Thanks for the tip.

[guysoft]

@paul-1 Isn't the dwc_otg fix patch also needed?
https://www.osadl.org/Single-View.111+M5c03315dc57.0.html
I have it in source control here:
https://github.com/guysoft/RealtimePi/blob/8b375fd1848276fa22c5c24a490d9696cd4a9e85/src/modules/realtimepi/filesystem/home/pi/patches/0001-usb-dwc_otg-fix-system-lockup-when-interrupts-are-th.patch

[paul-1]

I have not been applying that for a while, and definitely not on the 4.19 branch, I have not noticed a problem. But I've not done a ton of heavy USB testing.

Edit: You may be correct, I think with all the 4.19 changes, I made a mistake in my defconfig.

[pelwell]

Please take a look at https://github.com/pelwell/linux/tree/rpi-4.19.y.rt . It's a simple merge of the current rpi-4.19.y.rt with the RT commits on top. I've also copied across one of Tiejun's commits. Other than building it and booting it on a 3B+ I've not done any real testing, but if people think it's OK then I'm happy to push it as a semi-official, low-priority-supported branch off raspberrypi/linux.

[paul-1]

Should you change the defconfigs in that branch for
CONFIG_PREEMPT_RT_FULL=y

[pelwell]

Good point. I've updated the branch with updated defconfigs, but without the cherry-pick of Tiejun's patch (it was locking up for me with that included).

[pelwell]

I'm also using dtoverlay=dwc2.

[guysoft]

I'll try find the time and make a build or RealtimePi, so we can all be
testing the same thing.

[paul-1]

I'm using the standard dwc_otg driver (With fiq & fsm enabled).......I've never had good luck with the dwc2.

I've confirmed that the patch "don't thread shared irqs" from Tiejun is needed if SPI is enabled. I also applied the patch from osadl.org, but there has been changes to the dwc_otg driver that still hangs the system. I believe this commit adds a spot that needs to be locked with the fiq_fsm_spin_lock_irqsave 3ea9af8bc4eaac0cca72d603ed081973bce91bcf after patching that part, kernel has been stable.....but doing more testing now.

[pelwell]

Thanks for the feedback. You could probably save some time and effort if you could submit a Pull Request with the necessary changes, otherwise I'll have another go when I can.

[paul-1]

I will once I do some validation.....I don't want to make mistakes that I made with earlier RT builds.

[pelwell]

So that everyone is up-to-speed, two things happened early today:

1. @paul-1 submitted a PR (thanks!) to my candidate RT branch: https://github.com/pelwell/linux/pull/1
2. @TiejunChina (welcome back! (-8) created an "official" rpi-4.19.y-rt branch.

1) works for me (without the dwc2 overlay), but only if I revert the "no threaded irq" patch.
2) also needs reverting for me, along with an additional small patch to use fiq_fsm_spin_lock_irqsave in dwc_otg_read_common_intr (this is in paul-1's patch set).

One possible difference between my setup and yours is that I use the UART console on ttyS0 (the mini-UART with the shared IRQ) - running with dtoverlay=pi3-disable-bt switches the console to ttyAMA0/UART0 and prevents the lockup. A partial reversion - not threading the SPI interrupt but leaving the UART interrupt - is also effective, but I'm not using SPI; more testing is needed.

[guysoft]

Got a nightly build out with the new kernel:
Image:
* edit: nightly build here did not work, see newer build*

Build script sources for the rpi-4.19-rt branch:
https://github.com/guysoft/RealtimePi/tree/rpi-4.14.y-rt

[guysoft]

@pelwell Do you have the "no threaded irq" patch or link to commit, so I can revert it for the next nightly build?

[pelwell]

Of course - it's ca8809033b8f69f5999f3407f4f39667f07d7bee. To be clear, I'm not saying that we don't need some variant of that commit, but that it isn't correct as it stands.

[guysoft]

Ok, I am going to revert https://github.com/raspberrypi/linux/commit/ca8809033b8f69f5999f3407f4f39667f07d7bee and make another build.

BTW I just tested the nightly build without it. And it kernel panics on boot.
This is what I seem to be getting:

and with kgdboc=ttyS0,115200 in cmdline.txt:

[guysoft]

Ah wait, https://github.com/raspberrypi/linux/commit/ca8809033b8f69f5999f3407f4f39667f07d7bee is already in the rpi-4.19-rt, so it was shipped in the image.

Ok, so basically we are stuck because of the kernel panic above. I am not a kernel developer, so if anyone wants to help out and explain to me what is going wrong, and how to fix it, it could help make the branch usable.

[paul-1]

I'm not a kernel developer either, but I can follow code, and compare when things break.....

The branch 4.19.y-rt in the raspberrypi/linux git won't work as is if you have FIQ enabled. If that is the branch you are using,

If SPI is disabled, then you can revert. https://github.com/raspberrypi/linux/commit/ca8809033b8f69f5999f3407f4f39667f07d7bee]
If SPI is enabled, then you can not revert that commit, and can not use the serial console until things get figured out.

IF FIQ is enabled, then you also need to add https://github.com/pelwell/linux/pull/1/commits/02586ccd069c7f0e7be7be4610a1207295ca9bc5

[guysoft]

How would I disable FIQ?
Is that a good idea at all?
I am trying to have a distribution you can just flash and use this branch.
I think most people use it with Jack2, so that would be my focus. Would FIQ
effect USB devices?

[paul-1]

Add to command line dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0.

In the early single core days, the usb was nearly unusable without fiq....not sure how newer boards would run without it, let alone a RT kernel. But since it's still default for rpi to use fiq, I leave it enabled.

[guysoft]

@paul-1 Ok, adding dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 makes it hang in a different location now.
Looks like it hangs around the LAN driver. but the kernel panic is the same.

and with kgdboc=ttyS0,115200 in cmdline.txt:

[guysoft]

So this kinda went quiet because of the RaspberryPi 4 release - any opinters what is going wrong now?

[TheLongAndOnly]

A while ago I managed to get 4.19.y to work with the older RT patch. I will try to look up the config parameters tomorrow. Latency has not been great, but it worked stable. Unfortunately my build PC died, so I have to get it from the SD.

[TheLongAndOnly]

This is the command line I use to start:
dwc_otg.lpm_enable=1 console=serial0,115200 console=tty1 root=PARTUUID=46877fa5-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait splash plymouth.ignore-serial-consoles
likely only the first entry really has an effect.

I used the rt20 patch on kernel 4.19.46 and had to modify one file to make it compile. I'm pretty sure I had the usb-dwc_otg-fix patched ontop.

[guysoft]

Ok, using @TheLongAndOnly 's make the system boot! But with no USB working. Tested on a RaspberryPi A+.

I had to change:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=c1dc39e5-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh

To:

dwc_otg.lpm_enable=1 console=serial0,115200 console=tty1 root=PARTUUID=c1dc39e5-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait splash plymouth.ignore-serial-consoles

Uname gives over ssh:

Linux realtimepi 4.19.50-rt22-v7 #2 SMP PREEMPT RT Thu Jun 20 07:50:16 BST 2019 armv7l GNU/Linux

I will update cmdline.txt and make a new build.
However something seems unsable.

[guysoft]

Ok, I have a nightly that boots, using the branch. uname:

Linux realtimepi 4.19.50-rt22-v7 #2 SMP PREEMPT RT Sat Jun 29 10:17:23 BST 2019 armv7l GNU/Linux

What didnt' work:

• keyboard
• ethernet

what worked:

• boots
• wifi and ssh
• 3B+
• 3A+

Download at:
http://unofficialpi.org/Distros/RealtimePi/nightly/2019-06-29_2019-06-20-realtimepi-buster-lite-0.4.0.zip
md5:
http://unofficialpi.org/Distros/RealtimePi/nightly/2019-06-29_2019-06-20-realtimepi-buster-lite-0.4.0.zip.md5

Binary:
http://unofficialpi.org/Distros/RealtimePi/nightly/2019-06-29_realtimepi-kernel-4.19.50.tar.gz

Also upgraded to buster.

I think we need to add the usb-dwc_otg-fix for the keyboard to waork.

[pelwell]

We don't have a lot of time for diagnosis at the moment, but I'll happily merge a PR for any required changes.

[guysoft]

I am also kinda busy at the moment, but I can offer these builds of the
branch for testing since I have the distro in place.

Anyone here planning to play with it?

[paul-1]

It's a bit out of my understanding to play with interrupts. What I have works for our application (piCorePlayer) as most of our users are not interested in using the serial line in a RT environment. That appears to be the big problem maker.

Now if I could just get my hands on my pi4 shipments, I could get to work.

[TiejunChina]

Ok, I have a nightly that boots, using the branch. uname:

Linux realtimepi 4.19.50-rt22-v7 #2 SMP PREEMPT RT Sat Jun 29 10:17:23 BST 2019 armv7l GNU/Linux

What didnt' work:

• keyboard
• ethernet

what worked:

• boots
• wifi and ssh
• 3B+
• 3A+

Download at:
http://unofficialpi.org/Distros/RealtimePi/nightly/2019-06-29_2019-06-20-realtimepi-buster-lite-0.4.0.zip
md5:
http://unofficialpi.org/Distros/RealtimePi/nightly/2019-06-29_2019-06-20-realtimepi-buster-lite-0.4.0.zip.md5

Binary:
http://unofficialpi.org/Distros/RealtimePi/nightly/2019-06-29_realtimepi-kernel-4.19.50.tar.gz

Also upgraded to buster.

I think we need to add the usb-dwc_otg-fix for the keyboard to waork.

@guysoft I'm got stuck in my regular work so sorry for this inconvenience. Did you try our rpi-4.14.-rt? Even we decided not to upgrade that but I recall 'keyboard' and 'ethernet' both worked on my 3b board.

[guysoft]

@TiejunChina Hey, good to see you back.
Yes I have an RC for that:
https://github.com/guysoft/RealtimePi/issues/16

We will need though eveltually to move to rpi-4.19-rt.

[TiejunChina]

@guysoft my rpi-4.14-rt can work very well on my target.

pi@raspberrypi:/boot$ cat cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=7e31a93c-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles
pi@raspberrypi:/boot$ cat config.txt
.# For more options and information see
.# http://rpf.io/configtxt
.# Some settings may impact device functionality. See link above for details

.# uncomment if you get no picture on HDMI for a default "safe" mode
.#hdmi_safe=1

.# uncomment this if your display has a black border of unused pixels visible
.# and your display can output without overscan
.#disable_overscan=1

.# uncomment the following to adjust overscan. Use positive numbers if console
.# goes off screen, and negative if there is too much border
.#overscan_left=16
.#overscan_right=16
.#overscan_top=16
.#overscan_bottom=16

.# uncomment to force a console size. By default it will be display's size minus
.# overscan.
.#framebuffer_width=1280
.#framebuffer_height=720

.# uncomment if hdmi display is not detected and composite is being output
.#hdmi_force_hotplug=1

.# uncomment to force a specific HDMI mode (this will force VGA)
.#hdmi_group=1
.#hdmi_mode=1

.# uncomment to force a HDMI mode rather than DVI. This can make audio work in
.# DMT (computer monitor) modes
.#hdmi_drive=2

.# uncomment to increase signal to HDMI, if you have interference, blanking, or
.# no display
.#config_hdmi_boost=4

.# uncomment for composite PAL
.#sdtv_mode=2

.#uncomment to overclock the arm. 700 MHz is the default.
.#arm_freq=800

.# Uncomment some or all of these to enable the optional hardware interfaces
.#dtparam=i2c_arm=on
.#dtparam=i2s=on
.#dtparam=spi=on

.# Uncomment this to enable the lirc-rpi module
.#dtoverlay=lirc-rpi

.# Additional overlays and parameters are documented /boot/overlays/README

.# Enable audio (loads snd_bcm2835)
dtparam=audio=on
pi@raspberrypi:/boot$ uname -a
Linux raspberrypi 4.14.91-rt49-v7+ #32 SMP PREEMPT RT Mon Jan 7 11:29:31 CST 2019 armv7l GNU/Linux

[TiejunChina]

Let's first check if rpi-4.14.y-rt can work on your target.

[guysoft]

@TiejunChen It does, there is a release candidate out with rpi-4.14.y-rt .
https://github.com/guysoft/RealtimePi/issues/16

It boots and works.
Also got confirmation from several users.

The issue is with rpi-4.19-rt.

[guysoft]

Getting more reports - it seems like it works, but only if you set dwc_otg.lpm_enable=1, which I am guessing means we need usb-dwc_otg-fix to be patched.
See: https://github.com/guysoft/RealtimePi/issues/13#issuecomment-510181947

[guysoft]

Also, getting reports that this branch is not working for RaspberryPi 4 :https://github.com/guysoft/RealtimePi/issues/17

Have the build instructions changed?

[TiejunChina]

@guysoft I just ordered one pi4 last month but in China we will have to receive that next month. I'll check this once I get that.

[guysoft]

@TiejunChina Know well that feeling when you have bugs open in your codebase and no access to the hardware.
Ping us once you get it. Also feel free to ask people in my project to test stuff if that helps (issue https://github.com/guysoft/RealtimePi/issues/17 ).

Also there are interesting reports here of people testing stuff:
https://forum.linuxcnc.org/18-computer/36879-raspberry-pi-4?start=0#138874

[TiejunChina]

@guysoft @pelwell @paul-1 I just refreshed rpi-4.19.y-rt as
4.19.59 + v4.19.59-rt23 + rpi-4.19.y patches against 4.19.59 + "usb: dwc_otg: fix system lockup when interrupts are threaded" + "Don't need to use _nort" + "Enable CONFIG_PREEMPT_RT_FULL". I also enabled RT for bcm2711_defconfig as well.

[guysoft]

In the process of building a new image.
Will update when I have an image.
Should I also update the cmdline.txt to something else?

[guysoft]

Ok getting a strange behaviour.

It builds then stops in the middle:

+ make ARCH=arm bcmrpi_defconfig
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  YACC    scripts/kconfig/zconf.tab.c
  LEX     scripts/kconfig/zconf.lex.c
  HOSTCC  scripts/kconfig/zconf.tab.o
  HOSTLD  scripts/kconfig/conf
arch/arm/configs/bcmrpi_defconfig:32:warning: override: PREEMPT_RT_FULL changes choice state
#
# configuration written to .config
#
+ echo 'CONFIG_PREEMPT_RT_FULL=y
CONFIG_DEBUG_PREEMPT=n
CONFIG_PREEMPT_TRACER=n
CONFIG_RCU_CPU_STALL_TIMEOUT=21'
+ make -j4 zImage modules dtbs
scripts/kconfig/conf  --syncconfig Kconfig
.config:6391:warning: override: reassigning to symbol PREEMPT_RT_FULL
.config:6391:warning: override: PREEMPT_RT_FULL changes choice state
.config:6392:warning: override: reassigning to symbol DEBUG_PREEMPT
.config:6393:warning: override: reassigning to symbol PREEMPT_TRACER
.config:6394:warning: override: reassigning to symbol RCU_CPU_STALL_TIMEOUT
  SYSHDR  arch/arm/include/gen

[....]

  AR      drivers/char/ipmi/built-in.a
  AR      drivers/clk/actions/built-in.a
  CC [M]  net/ax25/ax25_uid.o
  CC      drivers/clk/bcm/clk-bcm2835.o
+ chmod 777 /mnt/sdc1/RealtimePi/src/build_dist /mnt/sdc1/RealtimePi/src/build.log /mnt/sdc1/RealtimePi/src/config /mnt/sdc1/RealtimePi/src/custompios_path /mnt/sdc1/RealtimePi/src/docker-compose.yml /mnt/sdc1/RealtimePi/src/image /mnt/sdc1/RealtimePi/src/image-variants /mnt/sdc1/RealtimePi/src/modules /mnt/sdc1/RealtimePi/src/vagrant /mnt/sdc1/RealtimePi/src/variants /mnt/sdc1/RealtimePi/src/workspace

Thats the last line, it just doesn't continue without any message.
That didn't happen before the update.

[pelwell]

What is mounted on /mnt/sdc1? Have you run an fsck on it recently?

[guysoft]

/dev/sdc1 is a harddrive where the image is being built.
I unmounted it and ran fsck (has been mounted 84 times without being checked, check forced. so i guess its a good thing to do anyway).

Mounting again and building, will see if there is a change, but doubt it because it failed at the same place twice.
Will update.

[pelwell]

Since it always fails in the same place, you could try running that chmod command on its own. It's also possible that it's the next command, the one it hasn't shown, that takes forever...

[guysoft]

@pelwell, ok, unmounting the docker environment and remounting fixed it. Strange but fixed :)

Image:
http://unofficialpi.org/Distros/RealtimePi/nightly/2019-08-07_2019-07-10-realtimepi-buster-lite-0.4.0.zip
md5:
http://unofficialpi.org/Distros/RealtimePi/nightly/2019-08-07_2019-07-10-realtimepi-buster-lite-0.4.0.zip.md5

kernel only that you can untar on top of an existing raspbian:
http://unofficialpi.org/Distros/RealtimePi/nightly/2019-08-07_realtimepi-kernel-4.19.59.tar.gz

Note: I didn't change the cmdline.txt mentioned here:
https://github.com/raspberrypi/linux/issues/2943#issuecomment-506561440
Likely now that the usd owc patch is in place I can remove that, rebuild, and hopefully release.

[guysoft]

Users are reporting at RealtimePi that usb-dwc_otg-fix does not help, is it merged in the tree?
If so, could there have been changes to the USB driver now that it also uses USB 3?

[TiejunChina]

Are you talking about this ?
https://github.com/raspberrypi/linux/commit/801df5aff559b438bbbc7c71180a85ec1b2f7124

[guysoft]

@TiejunChina Yes, it seems like the USB when enabled freezes the system on boot. There was a major change in the kernel in that aspect, so I would guess its there, but we really should find a way to debug what is going on there.

[sebastiannielsen]

And debugging becomes a little bit complicated since the whole system freezes. Is there some diff of the change, maybe the changes in the kernel could be analyzed and the "corresponding" changes in the fix code could be applied (kind of "blind fix")?

[guysoft]

It might be I already had. This is the output I got earlier on:
https://github.com/raspberrypi/linux/issues/2943#issuecomment-504059617
used kgdboc=ttyS0,115200 which gives more output on boot for the kernel.

Might be worth trying to add that on boot and see if we are still getting that error or if its something new.

[rlillback]

Is there any update on the progress? I have been attempting to patch this as well. I either get the Kernel panic's described above or the system boots without Ethernet and USB active.

[guysoft]

The current status is that this only works if you set dwc_otg.lpm_enable=1.
This means the current patch for the USB driver does not work with the new kernel and needs to be re-written. This means someone with knowledge of what is going on in the kernel needs to step in and write code. Which is none of us who are just using exiting patches.

The original patch was AFAIK was published here. By @oghorbel . But I have yet to reach him to provide input.

[guysoft]

@TiejunChina Any news on this? looks like there is a PR that will solve the USB/FIQ lockups

Thanks @schnitzeltony not trivial for me what you did there!

[guysoft]

@schnitzeltony I am building a nightly against your tree, so we have something to test

[guysoft]

Hey, the solution has been merged. And I built an have a release candidate based on the current branch. I think its worth for people here to test it. And also we can closet this issue.

Should work on all Pis including Rpi 4B (tested it and it does).
Download it here: https://github.com/guysoft/RealtimePi/issues/20

[pelwell]

Thank you for your efforts, Guy (fingers crossed for a successful outcome).

[gtrainavicius]

Should work on all Pis including Rpi 4B (tested it and it does).

Does it work on 2 and 3?

Linking related issue - https://github.com/raspberrypi/linux/issues/3172

[guysoft]

@gtrainavicius It should, CustomPiOS builds both for 2 3 and even 1 and zero kernels. I tested it on another nightly and it worked.

[guysoft]

@pelwell There is a Release Candidate out that I tested and works on Pi4, and should work also on any other Pi, the only thing missing is the 64bit kernel (kernel8.img). You are welcome to push people to test that and confirm. I think if I get no issues I shall release it in a week or so.

[antonellocaroli]

where do I find the kernel sources to compile?

[antonellocaroli]

for aarch64, I compiled the sources here...
https://github.com/raspberrypi/linux/tree/rpi-4.19.y-rt

and it works, the only thing that's weird about it is the use of the CPU.

uname -r 4.19.71-GentooPlayer-RT-MIN-rt24+

[guysoft]

This issue can be closed.
Also you can use this image if you like, released :)
https://github.com/guysoft/RealtimePi/releases/tag/0.4.0

[pelwell]

Thanks again, Guy.

[antonellocaroli]

Thanks

[gtrainavicius]

Do you get a warning and callstack printed out when running RT kernel on 3B+, as in the logs attached in this comment: https://github.com/raspberrypi/linux/issues/3172#issuecomment-569265745 ? RPi2 and 4 don't seem to produce it.