Stlink: [compatibility] libusbx replaced by libusb-1.0 in most distros

[Nightwalker-87]

As of March 2020 on most distros libusbx has been replaced by the maintained libusb-1.0.
This reflects the re-merging of the libusbx-project-fork back into the libusb-project (see https://github.com/libusb/libusb/wiki/FAQ#libusborg_libusbxorg_and_libusbinfo) as referenced by @slyshykO in #782. As mentioned in PR #895, I did some small research on this topic to find out about the current state of support by various distributions currently maintained.

Here is the result from https://pkgs.org/search/?q=libusb (as of Mar 2020):

1.0.23

• Alpine Edge
• Alpine 3.11
• ALT Linux Sisyphus
• Arch Linux
• Debian Sid
• Fedora Rawhide _--> libusbx, but compatible, as libusb-codebase is used_
• KaOS
• OpenMandriva Cooker
• OpenMandriva Lx 4.1
• openSUSE Tumbleweed
• Slackware Current
• Solus
• Ubuntu 20.04 LTS
• Ubuntu 19.10

1.0.22

• Alpine 3.10
• Alpine 3.9
• CentOS 8 _--> libusbx, but compatible, as libusb-codebase is used_
• Debian 10 (Buster)
• Fedora 31 _--> libusbx, but compatible, as libusb-codebase is used_
• Fedora 30 _--> libusbx, but compatible, as libusb-codebase is used_
• Mageia Cauldron
• Mageia 7.1
• NetBSD 9.0
• NetBSD 8.1
• NetBSD 7.2

1.0.21

• CentOS 7 _--> libusbx, but compatible, as libusb-codebase is used_
• Debian 9 (Stretch)
• openSUSE Leap 15.2
• openSUSE Leap 15.1
• Ubuntu 18.04 LTS

1.0.20

• OpenMandriva Lx 3.0
• Slackware 14.2
• Ubuntu 16.04 LTS

... older libusb versions _--> would no longer be supported_

• Debian 8 (Jessie) - 1.0.19
• Ubuntu 14.04 LTS - 1.0.17
• CentOS 6 - 1.0.9
• Slackware 14.1 - 1.0.9

Special case
... on FreeBSD libusb is integrated into the system:

• FreeBSD 13 - linux_libusb-13.0r358841 _--> libusb-codebase 1.0.16 - 1.0.18 used_
• FreeBSD 12 - linux_libusb-11.0r261448_4 _--> libusb-codebase 1.0.16 - 1.0.18 used_
• FreeBSD 11 - linux_libusb-11.0r261448_4 _--> libusb-codebase 1.0.16 - 1.0.18 used_

Looking at this, we should set libusb 1.0.20 as the minimum required version (apart from FreeBSD) to ensure widespread compatibility.

[Nightwalker-87]

This initiated a ongoing discussion in PR #895 which shall be continued here, as the related PR should solely focus on (direct) changes in the code. The beginning of this discussion is now posted here:

Nightwalker-87:
@Vascom: You are on Fedora right? What do you think about this? Why does Fedora still use libusbx BTW? Is your library that outdated as well?

Reply by Vascom:

@Nightwalker-87 I don't know why it still not updated in rawhide. Need to ask maintainer.
But even if this update will be done it will be never updated for current releases of Fedora and RHEL/CentOS.

Reply by slyshykO:

Looks like starting from this commit https://src.fedoraproject.org/rpms/libusbx/commits/master Fedora switched from libusbx to libusb but don't change the package name.

Reply by Vascom:

No.
libusb - for compatibility with 0.1 API.
libusbx - main libusb as I know.

Reply by slyshykO:

@Vascom Yes. My point is they (Fedora) use sources from libusb-project for libusbx-package.

Reply by Vascom:

upstream libusbx has merged back with libusb and is now called libusb again, but we already have a libusb package for the old libusb-compat-0.1, renaming that to libusb-compat while at the same time giving this its name is a bit tricky, lets stick with the libusbx name for now

[Nightwalker-87]

@Vascom: There are three relevant things:
1) As @slyshykO mentioned before: the package name is _not_ the relevant thing, but if LIBUSBX_API_VERSION is used by the package which then remains from the libusbx-codebase
2) Nobody cares about libusb-compat, libusb-0.1 or whatever the older packages for libusb 0.1 compatibility versions are called. These are neither addressed by this issue, nor relevant.
¯\ _(ツ) _/¯
3) Please test the current development branch with your fedora environment as it comes without any reference to LIBUSBX_API_VERSION. If it works we are fine here, if not, the incompatibility with fedora is present for quite a long time, as previously LIBUSBX_API_VERSION could only be called on FreeBSD systems anyway (where it was not needed anymore, as we found out in #782 previously).

[Vascom]

1.0.23 pushed to Fedora rawhide now https://src.fedoraproject.org/rpms/libusbx/c/5684853d449b7fe3a81112cd59f9336fbc274319?branch=master

[Nightwalker-87]

That's good news for us.

[Nightwalker-87]

Similar result for Fedora 30 (oldest supported version):
Here it is libusb 1.0.22 despite the (false) package name: https://src.fedoraproject.org/rpms/libusbx/c/cbd5348ee14638264fe3bb79c3b27a5ed34271ed?branch=f30

One can conclude that this is done equally in Fedora 31, so we are fine here. That's the reason _why_ it works even without any LIBUSBX_API_VERSION remnants currently. :+1:

[Nightwalker-87]

@lhondareyte @DanielO: It appears you both use (or used) FreeBSD. Can you provide us some information regarding the libusb versions present on recent releases (11, 12, 13)? pkgs.org was not very helpful in this aspect.

[lhondareyte]

Hi,
For FreeBSD, libusb is in the base system, it's not a package. It's compatible with libusb1 & 2 . You can get more informations with "pkconf":
pkgconf --libs --cflags libusb-1.0
pkgconf --libs --cflags libusb-2.0
pkgconf is, as far I know, specific to FreeBSD.
Hope this help.
Regards

[Nightwalker-87]

@lhondareyte: Thx for the useful feedback. What FreeBSD-Version/Release are you on? Can you also checkout the develop branch and test-compile the stlink-tools? With this we could rule out possible regression when we merge the related PR. By now current FreeBSD Releases seem to be the only left "unknowns" (see list above).

[lhondareyte]

I am running the latest stable version (12.1p1) (almost). I tried to compile the latest snapshot and it actually failed:
/usr/home/luc/stlink/src/usb.c:34:22: error: use of undeclared identifier 'INT_MAX'
The compilation works with

// src/usb.c
#if defined (__FreeBSD__)
#include <sys/limits.h>
#endif

Which commit/branch do you want to try exactly?

About FreeBSD current (latest dev branch), I only use it on ARM platform. I'll give it a try this week-end: just try to compile only (USB support can be tricky)

[Nightwalker-87]

The latest develop would just be fine, as we still have to decide on, which codelines have to be added in order to make it compile in FreeBSD environments. Thx for helping out with this. :+1:

To me it looks like that nobody really bothered about FreeBSD too much before, apart from very few single contributions. Beginning with v1.6.1, I want to make sure which distributions (incl. versions) work and have that put into our documentation as well to reduce continuous "cannot compile on xyz-system" issues here. I'm fed up seeing these.

[lhondareyte]

I understand but don't be worried about FreeBSD: stlink is in the ports tree (well, not the latest release) with an active maintainer. And most FreeBSD users use ports ;)
And I regulary check the latest snapshot on FreeBSD and macOS (via rudix)
However, I'll give it a try on FreeBSD 13

[Nightwalker-87]

... well it does make sense to distribute recent fixes and features to this usergroup as well, doesn't it? Old packages are no help here, especially not for available maintainers on that platform.

[Nightwalker-87]

_Note:_ By setting the minimum version requirement for libusb to 1.0.20, we could make this equal for all supported systems (Windows, macOS and even most of the oldest supported linux/unix distributions), thus keeping it simple and memorable.

[DanielO]

Hi,
For FreeBSD, libusb is in the base system, it's not a package. It's compatible with libusb1 & 2 . You can get more informations with "pkconf":
pkgconf --libs --cflags libusb-1.0
pkgconf --libs --cflags libusb-2.0
pkgconf is, as far I know, specific to FreeBSD.

FreeBSD ships with pkg-config files for libusb-0.1, libusb-1.0 and libusb-2.0 (2.0 is a FreeBSD specific one).
eg..

[delamerelts 1:44] ~> pkg-config --cflags --libs libusb-0.1
-lusb
[delamerelts 1:44] ~> pkg-config --cflags --libs libusb-1.0
-lusb
[delamerelts 1:45] ~> pkg-config --cflags --libs libusb-2.0
-lusb

This works on Linux too (for me)

[ubuntu 12:17] ~ >pkg-config --cflags --libs libusb-1.0
-I/usr/include/libusb-1.0 -lusb-1.0

[slyshykO]

I think we should use the newest libusb that we can. On all platforms.

[Nightwalker-87]

_Note:_ There is a difference between _working_ versions and the _minimum required version_ .
This does not rule out that users should be encouraged (by a note in the documentation) to install the latest available version on their respective system. It only denotes that the tools would be _still_ be capable to compile and run with the oldest version defined. On Ubuntu 16.04 LTS for example 1.0.20 is the newest version available as you can see in the list (and there will be no newer version for this release anymore).

[Nightwalker-87]

@DanielO: What are the exact version strings (e.g. 1.0.22, etc.)? That would be the relevant point here...

[lhondareyte]

@DanielO : yes, you're right : pkg-config is a link to pkgconf
@Nightwalker-87 :
libusb_get_version() return 1.0.0.2016 with this code:

version = libusb_get_version();
printf("%d.%d.%d.%d\n", version->major, \
                        version->minor, \
                        version->micro, \ 
                        version->nano);

uname -r
12.1-RELEASE-p2

[Nightwalker-87]

@lhondareyte: o.O That is what I asked for, but it does not seem to help at all. I did not expect a different versioning scheme compared to the one the libusb-project uses...

[lhondareyte]

libusb define LIB_API_VERSION since v1.0.13 which describe features that are include. All OSs that you list have this definition. So why do you want to deal with lib version instead of API version?
Currently all supported FreeBSD versions (11,12) and current define this
#define LIBUSB_API_VERSION 0x01000102
Perhaps, I miss something, if so, let me know.
Best regards

[Nightwalker-87]

The LIB_API_VERSION is not the relevant point here as it is used in the stlink-tools codebase already. We have dealt with this in #782. Now we are solely on about how recent we require the package libusb to be _at least_. We can't help it that FreeBSD has libusb integrated and uses a different versioning scheme than the libusb-project, what makes it hard to compare both codebases with each other. So we can not say anything like libusb 1.0.0.2016 on FreeBSD derives from the e.g. libusb-1.0.20 package from the libusb-project or alike. We can do so for all other operating systems.

[lhondareyte]

In this issue, this is LIBUSBX_API_VERSION the problem. LIBUSBX_API_VERSION is almost marked as deprecated in libusb and does not exist on FreeBSD:

/* The following is kept for compatibility, but will be deprecated in the future */
#define LIBUSBX_API_VERSION LIBUSB_API_VERSION

It has nothing to do with FreeBSD. Versioning scheme for API_VERSION is the same than others OSes. And many ports use the native FreeBSD version without problems.

[Nightwalker-87]

So what would be a solution to this (apart from trusting on that it will run and without tracking a version on FreeBSD)? I feel uncomfortable with this as one would have to look at this special case every time a new version is published on FreeBSD and test it manually.

[lhondareyte]

I'll take a look. For short, any reference to LIBUSBX_API_VERSION should be remove from stlink code.
Do you know the minimal API_VERSION to works with stlink? In fact, wihch features do you need?
But it looks strange to me that you spend so much time for Ubuntu 12.04 that is no more supported since 2017

[Nightwalker-87]

I'll take a look. For short, any reference to LIBUSBX_API_VERSION should be remove from stlink code.

This is already the case in the develop branch, as I wanted to get rid of this deprecated stuff.

Do you know the minimal API_VERSION to works with stlink? In fact, wihch features do you need?

No, I don't, but that is what I am looking for. If we can find out the LIBUSB_API_VERSION of the 1.0.20 package, we can compare it to the one in FreeBSD. Maybe that is another approach to determine where the codebase derived from.

I would no longer like to see all these unknowns we have been dealing with up to now. As we have several tickets related to libusb issues, not having this defined clearly does not help. I feel like this is hindering maintenance in this project.

[lhondareyte]

I have made an inventory on latest officials releases:

• 1.0.12 and above dont define *API_VERSION
• 1.0.13 to 1.0.17 define LIBUSBX_API_VERSION only
• since 1.0.18 both LIBUSB_API_VERSION and LIBUSBX_API_VERSIONare defined as follow

define LIBUSB_API_VERSION 0x010001xx
define LIBUSBX_API_VERSION LIBUSB_API_VERSION

LIBUSBX_API_VERSION appeared in 1.0.13 and LIBUSB_API_VERSION in 1.0.18

A simple solution could be, for every OSes:

#include <libusb.h>

#if !defined ( LIBUSBX_API_VERSION )
 #if defined (__FreeBSD__)
   #define LIBUSBX_API_VVERSION LIBUSB_API_VERSION
 #elif !defined (LIBUSB_API_VERSION)
   #error unsupported libusb version
 #endif
#endif

#define MINIMAL_API_VERSION 0x01000102

#if ( LIBUSB_API_VERSION < MINIMAL_API_VERSION )
 #error unsupported libusb version
#endif

Ok on FB12 and macOS10.14

[lhondareyte]

About LIBUSBX_API_VERSION/LIBUSB_API_VERSION :

v1.0.13 : 0x01000100
v1.0.14 : 0x010000FF
v1.0.15 : 0x01000101
v1.0.16 : 0x01000102
v1.0.17 : 0x01000102
v1.0.18 : 0x01000102
v1.0.19 : 0x01000103
v1.0.20 : 0x01000104
v1.0.21 : 0x01000105
v1.0.22 : 0x01000106
v1.0.23 : 0x01000107

[Nightwalker-87]

@lhondareyte: Excellent, and what is the LIBUSB_API_VERSION string is present on each FreeBSD 11, 12, 13? Can one look that up somewhere on the web? By this, we are finally able to gain this information via the laborious route... I can then complete the listing above and a final decision can be made.

[DanielO]

They're all 0x01000102:
11.0.0 - https://svnweb.freebsd.org/base/release/11.0.0/lib/libusb/libusb.h?revision=354337&view=markup#l38
12.1.0 - https://svnweb.freebsd.org/base/release/12.1.0/lib/libusb/libusb.h?revision=354337&view=markup#l38
Head - https://svnweb.freebsd.org/base/head/lib/libusb/libusb.h?revision=326219&view=markup#l38

It hasn't changed since that #define was added in r301957
https://svnweb.freebsd.org/base/head/lib/libusb/libusb.h?revision=301957&view=markup

[Nightwalker-87]

@DanielO: Thx for checking! Please let us know if this changes someday.

ok, so we actually have to stay on libusb 1.0.16 ... 1.0.18 codebase then for FreeBSD for compatibility reasons. In terms of implementation we will likely check for the 0x01000102 in cmake here.

I'll do a new table for our documentation that is going to list all what we have found out by now.

@slyshykO: Can you extend you PR #895 with the discussed check for windows and maybe also update the cmake routine for FreeBSD?

[slyshykO]

I've updated the #895. Please, take a look.