Stf: usb xHCI hardware issues

[vovinio]

Hi,
First of all I wanna say thanks for the great job that you've done and for sharing this incredible knowledge.

I've built a test farm with 120 phones which are being controlled remotely.
The phones are connected with hubs (Plugable USB 2.0 7 Port Hub with 60W Power Adapter) mainly for charging and for re-connection (making restarts and tcpip 5555 when needed).

The problem I'm facing is that my host machine recognizes only a small portion of the devices (around 30 and not all of them).

When I run 'dmesg' in ubuntu, I get the following errors:

1. Max number devices this xHCI host support is 32.
2. couldn't allocate usb_device.
3. not enough host controller resources for new devices state.
4. can't set config #1, error -12.
5. device descriptor read/64, error -32.
6. device not accepting address 22, error -71.

As I understood, my motherboard does not support more than a certain amount of devices, therefore, I've bought the usb controller that you recommend (PEXUSB3S44V). When I've connected the devices to that usb controller, the problem remained the same and I still couldn't get more devices to be connected.
I've tried to disable the xHCI in the bios but that didn't help as well.

Currently, I'm running all 120 devices and mostly they work fine, but, when they do disconnect I need to connect them manually again which is a big pain in the ass :)

This are the specs of my host machine:
Linux 4.15.0.36-generic (x86_64), Ubuntu 18.04.1 LTS.
Intel core i7-8700K, 32 GB RAM, 256 NVME SSD
MB: Z370-A pro (ms-7b48)

Do you have any ideas or suggestions for what can I do fix the following issues?

[denis99999]

Hi @vovinio,

Perhaps you should migrate to a full deployment model of STF (cf. doc/DEPLOYMENT.md) in order to address the scalability issue you are meeting ?

A simple solution would be to have a set of low cost stf-provider physical machines (i.e. with a Plugable USB hub connected on each) remotely connected on a central server hosting all other STF units and located on a virtual machine (i.e. ~2Vcpu + 4 Go RAM).

Regarding the stf-provider machine, you could use an ARM one like Raspberry PI3 model B. The price is no more than 50$ for a RPI3, and you can connect up to 6 USB devices on the hub to keep a reliable system. Of course, you will have also to buy one or more 1Gb/s switches to interconnect the whole.

Regarding the operating system to use, I recommend you to choose Debian 9 lite on RPI3 and Debian 9 or CentOS 7 on the central server, and above all avoiding Ubuntu which is not enough a reliable operating system from my point of view.

Obviously, all that requires some extra cost and an integration work both at provider and server sides, but it is another problem :-)

I hope these some recommendations will help you.

[gounthar]

As for the Raspberry Pi3, I would go with HypriotOs, as docker and docker-compose are already there from the beginning.
You could even go more exotic with an OrangePi Zero and Armbian, which I tested successfully.

[sudhalucky]

Hi @vovinio
It seems the limitation is because of the way the xHCI drivers are implemented in the OS kernal, we also hit the same issue of provider not recognizing more than 9 devices irrespective of connecting them directly to USB or vis Hub (BTW our machine is only being used for provider service and all other STF app services are running on VMs in server grade hardware). We tried the below solution to force USB ports to use eHCI drivers instead of xHCI and everything is working fine now, our provider machine did not even have the option to turn off xHCI drivers in BIOS hence used the software kill switch.

https://askubuntu.com/questions/557058/disable-xhci-by-another-way-than-bios-usb-3-0

Let me know if it works for you

[sorccu]

Disabling USB 3 helps but it can be quite difficult like @sudhalucky mentioned. Overall introducing a dedicated USB PCIe card is likely to increase stability, so it's not a bad idea either way.

[vovinio]

Hi guys, thanks for the answers.
I'll elaborate from my last experience.

Intel's series 8 chipset only allows 96 usb endpoints. Connecting multiple dedicated USB PCIE cards is expansive while still not increasing much the number of endpoints you can get (most of PCIE cards won't give you more than 32 endpoints and might be pricey).

There are few other cheaper and more comfortable options:

1. Getting and AMD Ryzen based system. I've tried that and it was a great solution.
   "Each USB controller on the AM4 platform can support 127 USB endpoints. The chipset has one controller, Ryzen has another. That means we support a total of 254 endpoints. This can be done with up to five layers of hubs. Hubs count against the endpoint total."
   https://www.reddit.com/r/Amd/comments/689v85/ryzen_number_of_usb_endpoints/

1. Getting an old Intel system (older than the 8 series chipset). I've tried that with an old Intel pentium dual core (2008 model) and it handled it very well.

Disabling USB3 from within ubuntu didn't help.

Much thanks for your reply's ;)