Linux: 5.x+ Kernel cannot recognize USB devices on external hub

[knro]

Is this the right place for my bug report?
This problem started appearing after upgrading from 4.19.xx to 5.4.xx Kernel. Downgrading the kernel to 4.19 restores the USB functionality.

Describe the bug

USB devices inaccessible if connected to an external powered USB hub.

To reproduce
Connect USB devices to an external powered USB hub. lsusb only shows the root hub:

Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Expected behaviour

All USB devices attached to the hub are listed and accessible. Running sudo lsusb shows all the devices correctly so it appears to be some permission issue.

Actual behaviour

No USB devices detected. /sys/devices/platform/scb/fd500000.pcie shows permission 660 in Kernel 5.4, while on Kernel 4.19, the permission is 755

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

• Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW

Raspberry Pi 4.

Hardware : BCM2711
Revision : c03112
Serial : 1000000089b8c084
Model : Raspberry Pi 4 Model B Rev 1.2

• Which OS and version (cat /etc/rpi-issue)?

Raspberry Pi reference 2019-09-26
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 80d486687ea77d31fc3fc13cf3a2f8b464e129be, stage4

• Which firmware version (vcgencmd version)?

Aug 6 2020 16:22:25
Copyright (c) 2012 Broadcom
version af3edc2de473197cdfe1ff5a8ff2d34095d5b336 (clean) (release) (start)

• Which kernel version (uname -a)?

Linux ikaruspi 5.4.51-v7l+ #1332 SMP Tue Aug 4 18:40:14 BST 2020 armv7l GNU/Linux

Logs
If applicable, add the relevant output from dmesg or similar.

boot.txt
cmdline.txt
config.txt
dmesg.txt
lsusb.txt

Additional context

This issue was reported by several users in INDI forum.

[rkaczorek]

I do confirm this issue

[6by9]

Link to Pi forum thread to avoid duplication of effort - https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=281454

[knro]

I can confirm that connecting any FTDI chipset USB-to-Serial cable causes this issue. As soon as the FTDI cable is removed, and after rebooting the system, USB is back to normal. FTDI chipsets are very common and I hope this narrows down the issue.

[macmpi]

What is first 5.4.x failing taking them from https://github.com/raspberrypi/firmware/commits/master
Would 5.4.49 be fine and issue start at 5.4.50 ?

[knro]

@macmpi How can I update the OS to a specific commit? using rpi-update?

[macmpi]

Yes, check options here

[knro]

Just tested with the earliest 5.x I found which was 5.4.35 and this exhibits the same issue. The commit before that which has Kernel 4.19.118 has no issues.

[k-ross]

Here is some of the relevant info from the forum thread, copied here for convenience.

It seems the offending device is a (FTDI) USB-to-Serial adapter. As soon as I plug it into the USB hub, the permissions in the /sys hierarchy get messed up. However, it does create a /dev/ttyUSB0 device, and it is perfectly usable!

So I plugged all my devices (WiFi dongle, two astronomy cameras that are powered via USB, an electronic filter wheel, also USB powered) into the USB3 powered hub, with power applied to the hub, and rebooted the Pi. Everything looks normal:

$ ls -ld /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb*
drwxr-xr-x 6 root root 0 Jul 31 22:04 /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1
drwxr-xr-x 6 root root 0 Jul 31 22:04 /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2
$ lsusb
Bus 002 Device 003: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 002 Device 002: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 006: ID 0bda:b812 Realtek Semiconductor Corp.
Bus 001 Device 010: ID 03c3:1f01
Bus 001 Device 009: ID 03c3:1604 
Bus 001 Device 008: ID 04b4:6572 Cypress Semiconductor Corp.
Bus 001 Device 007: ID 1618:0921 
Bus 001 Device 005: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 001 Device 003: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Then I plug the USB-to-Serial adapter into the USB3 hub, and bam! The permissions get messed up, and lsusb (and libusb) no longer work properly:

$ ls -ld /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb*
drw-rw---- 6 root root 0 Jul 31 22:12 /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1
drwxr-xr-x 6 root root 0 Jul 31 22:04 /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2
$ lsusb
Bus 002 Device 003: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 002 Device 002: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Here's the output from dmesg when I plugged in the adapter:

[  502.190760] usb 1-1.2.2.4: new full-speed USB device number 11 using xhci_hcd
[  502.357098] usb 1-1.2.2.4: New USB device found, idVendor=0403, idProduct=6001, bcdDevice= 6.00
[  502.357118] usb 1-1.2.2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  502.357134] usb 1-1.2.2.4: Product: FT232R USB UART
[  502.357149] usb 1-1.2.2.4: Manufacturer: FTDI
[  502.357164] usb 1-1.2.2.4: SerialNumber: A6021GB5
[  502.363824] ftdi_sio 1-1.2.2.4:1.0: FTDI USB Serial Device converter detected
[  502.363997] usb 1-1.2.2.4: Detected FT232RL
[  502.368594] usb 1-1.2.2.4: FTDI USB Serial Device converter now attached to ttyUSB0

Plugging the USB-to-Serial adapter into the built-in ports on the Pi works perfectly fine, and that's what I will be doing until this gets resolved.

[Avocette99]

I confirm I have a similar problem in my implementation. I can no longer use one USB device (Pegasus Astro Pocket Power Box) since I upgraded (unknowingly) to Kernel 5.4 from 4.19, and have learned that it contains an FTDI chip for USB to Serial interface. I have an RPi4 4GB v1.1

[knro]

Is there a way to trace this in the kernel somehow? i.e. trace the events that leads to this?

[Avocette99]

I am no expert, but I believe something has changed in the very latest Raspberry Pi OS updates that has made my system stable again with the FTDI serial adapter in the Pegasus Astro PPB in KStars/Ekos/Indi using /dev/ttyUSB1 and the Prolific adapter internal to the mount using /dev/ttyUSB0.
I have attached here the list of updatable files when carrying out the sudo apt update && sudo apt upgrade today.
Update list 17_8_2020.txt

[Andy1978]

Why has this been closed? It's still a problem in the latest stable release "Linux fmt978-x47282 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux"

[renekliment]

I also confirm this issue. (caused by kernel upgrade from 4.19.x to 5.4.y) It is still very present.
It is inconvenient, because I want to run a few power-demanding devices with FTDI USB interface from a powered USB hub and now I can't.

Running a RPi 4.

[knro]

This is not a kernel issue. Just remove the offending rpi-gpio.common package and it should be fine.

[ziesemer]

This is not a kernel issue. Just remove the offending rpi-gpio.common package and it should be fine.

How is GPIO messing with USB hub permissions? Is there another ticket related to this with more details then, to follow?

[knro]

I was able to trace it back to 60-rpi.gpio-common.rules

What was happening is that

/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1

was always reset to 660 whenever a USB device is plugged in. Sometimes even arbitrary changes to /sys/devices/platform/scb/fd500000.pcie

It took me a while (few hours) to figure out it's udev related and by process of elimination your script was the only one left. I was able to edit it by adding a slash before the dev_patch (%p) and that seems to make the system happy and operational. It looks now like this:

SUBSYSTEM=="gpio", KERNEL=="gpio*", ACTION=="add", PROGRAM="/bin/sh -c 'chown root:dialout /sys/%p/active_low /sys/%p/direction /sys/%p/edge /sys/%p/value ; chmod 660 /sys/%p/active_low /sys/%p/direction /sys/%p/edge /sys/%p/value'" was able to trace it back to 60-rpi.gpio-common.rules

What was happening is that

/sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1

was always reset to 660 whenever a USB device is plugged in. Sometimes even arbitrary changes to /sys/devices/platform/scb/fd500000.pcie

Any any rate, IMO the udev rules for that package are making quite dangerous changes. I emailed the maintainer about this and not sure if there was any action taken. So the safe bet to is remove the package.

[6by9]

@XECDesign are you aware of these funny udev rules? Can we patch?

At a guess it's trying to preserve the permissions on the ethernet port of SMSC9514, but I don't see why it's being triggered by the quoted udev rule.

[renekliment]

Thank you for the workaround (sudo apt remove rpi.gpio-common)! It indeed fixes the permission issue and my FTDI thingies and lsusb work.
Since I don't need the package for GPIO stuff, I am happy with it.

[XECDesign]

Sorry I missed the nudge. I've now modified the package with @knro's suggestion.

[vdavy]

Go the same strange stuff : type lsusb and got all usb devices listed. Plug an FTDI USB RS232 devices, and types lsusb and got no devices.
With the FTDI plugged, all devices need root access to work.
So I fixed it by adding this content to file : /etc/udev/rules.d/60-rpi.gpio-common.rules to override the one provided :

SUBSYSTEM=="bcm2835-gpiomem", KERNEL=="gpiomem", GROUP="dialout", MODE="0660"
SUBSYSTEM=="gpio", KERNEL=="gpiochip*", ACTION=="add", PROGRAM="/bin/sh -c 'chown root:dialout /sys/class/gpio/export /sys/class/gpio/unexport ; chmod 220 /sys/class/gpio/export /sys/class/gpio/unexport'"
SUBSYSTEM=="gpio", KERNEL=="gpio*", ACTION=="add", PROGRAM="/bin/sh -c 'chown root:dialout /sys%p/active_low /sys%p/direction /sys%p/edge /sys%p/value ; chmod 660 /sys%p/active_low /sys%p/direction /sys%p/edge /sys%p/value'"

REboot needed to have changes applied.

The bugged original one has the last line with added slash before each %p pattern : /sys/%p/active_low instead of the correct form /sys%p/active_low
The buggy apt package is rpi.gpio-common

[XECDesign]

There's some conflicting information in this thread now. @knro, any idea why this would cause issues for @vdavy ?

[henriknil]

Go the same strange stuff : type lsusb and got all usb devices listed. Plug an FTDI USB RS232 devices, and types lsusb and got no devices.
With the FTDI plugged, all devices need root access to work.
So I fixed it by adding this content to file : /etc/udev/rules.d/60-rpi.gpio-common.rules to override the one provided :

SUBSYSTEM=="bcm2835-gpiomem", KERNEL=="gpiomem", GROUP="dialout", MODE="0660"
SUBSYSTEM=="gpio", KERNEL=="gpiochip*", ACTION=="add", PROGRAM="/bin/sh -c 'chown root:dialout /sys/class/gpio/export /sys/class/gpio/unexport ; chmod 220 /sys/class/gpio/export /sys/class/gpio/unexport'"
SUBSYSTEM=="gpio", KERNEL=="gpio*", ACTION=="add", PROGRAM="/bin/sh -c 'chown root:dialout /sys%p/active_low /sys%p/direction /sys%p/edge /sys%p/value ; chmod 660 /sys%p/active_low /sys%p/direction /sys%p/edge /sys%p/value'"

REboot needed to have changes applied.

The bugged original one has the last line with added slash before each _%p_ pattern : _/sys/%p/active_low_ instead of the correct form _/sys%p/active_low_
The buggy apt package is rpi.gpio-common

This worked fine for me

[XECDesign]

Alright, I'll revert the previous "fix" until I can investigate it myself.

[XECDesign]

So it looks like what's actually happening is that it's hitting some character limit and cutting the command short. Instead of the full path, the chmod command gets cut short at usb1. I've split the chmod and chown into two rules, which seems to work as intended.

Updated package should show up later today.