Dlib: Simplify shared/static linking

[SoapGentoo]

Hi @davisking
one problem in distributions we face is the fact that CMake wasn't designed to concurrently handle building shared and static libraries without adding two targets etc. This leads to every CMake-based buildsystem having a different way of handling enabling/disabling static and/or shared libraries. I would like to recommend the following plan, which is what Kitware recommend:

Only define one library. The type of library can be switched by the native CMake variable BUILD_SHARED_LIBS. I recommend building static libraries by default (as most users who manually build dlib will prefer this). If users want shared libraries, they just pass -DBUILD_SHARED_LIBS=ON. This makes building dlib a lot easier for us, as for instance when building the test suite we don't build the static libs which we throw away in the end. Furthermore, because BUILD_SHARED_LIBS is a standard CMake library, it integrates well with how distributions can globally only build shared libraries. Furthermore, this would simplify the build system a lot for you too by removing lots of redundant definitions and conditional code.

Would you be willing to accept a PR for this?

[davisking]

What about when someone installs dlib on a unix system by?

mkdir build
cd build
cmake ..
make install

That's now only going to install a static library, or a shared library depending on the switches.

Also, what do you mean by "This makes building dlib a lot easier for us, as for instance when building the test suite we don't build the static libs which we throw away in the end". What is made easier? As an aside, I want the tests to build dlib in multiple modes to make sure the build works. They are tests after all.

[SoapGentoo]

@davisking correct. The proper CMake way is to configure it twice out of tree, and install that twice. In general for users this doesn't matter, as people who compile dlib don't usually want both (and in practice prefer static libs, as that doesn't require any RPATH or LD_LIBRARY_PATH mucking). The current solution (producing the shared out of the static library) has another disadvantage: it requires compiling the same source code twice (once with -fPIC and once without -fPIC), which given that as a distro we generally only install the shared library, is a waste of CPU cycles.

I don't see how the current system tests dlib in different modes? Currently, all tests link to the static library. In fact, with my suggestion, the shared library is also tested. To test both modes, you can easily set up a travis test matrix and test both shared and static libraries linked to the tests.

[davisking]

Yes, that's true. I mean it tests the build.

Anyway, if you want to submit a PR that makes this change I'm fine with it so long as it doesn't impact existing users.

[davisking]

What happens when you configure it two times, and get two different dlib/config.h files?

[xsacha]

@davisking Why is the config.h different for static/shared?
I see:

define DLIB_DISABLE_ASSERTS // asserts always disabled

define DLIB_NO_GUI_SUPPORT

define DLIB_USE_BLAS

define DLIB_USE_LAPACK

Anyway, if it was, it wouldn't matter anyway. It is trivial to separate them by library type if they depend on library type.

Every other CMake project I have does it like this (static unless BUILD_SHARED_LIBS is ON).
If you just do add_library(TARGET SOURCE), CMake automatically picks the type for you dependant on that variable.

[davisking]

I'm just paranoid about users doing stupid things. Having it separate gives them an opportunity to do something like recompile with different options and then run a make install which overwrites the config.h, and now the headers are wrong for one of the compiled libs.

This isn't hypothetical either. There are a huge number of dlib users who have no idea what a shared library even is. For instance, the simplest way to use dlib is to simply call add_subdirectory rather than do any kind of install. But I very regularly get people asking questions like "I built a .so, no what?". It's also very common for people to build part of dlib in one configuration and then mix in object files built with a different configuration, creating a bunch of ODR violations. Recently, I've honestly considered making CMake emit an error when users attempt to build dlib in any other way aside from calling add_subdirectory (e.g. like this https://github.com/davisking/dlib/blob/master/examples/CMakeLists.txt). For instance, it could emit a big message saying that using add_subdirectory is better and only people who really know what they are doing or are package maintainers should build it standalone.

I also have problems where package maintainers will compile dlib in a funny way, like without GUI support for instance. Users will then complain to _me_ about it. So I've considered making it an error to compile dlib standalone without a GUI.

All of the above seems insanely heavy handed so I haven't done it. But I get many emails every day about this kind of stuff, it's only so long until I lose my cool about it :(. How irritating do you think it would be if it was required to say this to compile a standalone dlib library cmake -DI_KNOW_WHAT_IM_DOING=1 dlib? :)

Anyway, I think the thing you are proposing is probably fine. I'm just worried I'm going to find out there are a bunch of people who habitually set BUILD_SHARED_LIBS to ON and will be surprised by this when they get runtime linker errors because the loader can't find their dlib shared library. Maybe that's not an issue. I'm sure I'll find out though if it is.

Here is another kind of example of the stuff I have to worry about: https://github.com/davisking/dlib/blob/master/dlib/dlib_basic_cpp_build_tutorial.txt. I used to be inundated with users bitching about how one thing or another doesn't work in dlib only to find out they were #including some of dlib's source files, and obviously getting linker errors. I have a bunch of preprocessor magic now that detects that and makes their compiler output that message in dlib_basic_cpp_build_tutorial. This problem is made even worse since there are blog posts on the internet telling people to #include .cpp files in dlib. But now that this error message is there it has significantly cut down on the number of emails I get about this.

It wasn't always so bad, but now that machine learning is super popular I get a very large number of users who have no idea what they are doing, many have never programmed before. So I try really hard to make things as simple and foolproof as possible. For instance, for many dlib users, simply installing boost is an insurmountable challenge. I know this because boost is required to use dlib's python API and I regularly get complaints like "I've been trying to install boost for 10 hours and I finally gave up!".

Anyway, sorry for the rant. I think your proposal is fine.

[xsacha]

Just a note that where I use dlib, in the hunter package manager, this condition is _already_ enforced for all packages.
So of course this was one of the first things to be changed.
https://github.com/hunter-packages/dlib/commit/dcaeb4274ceeeb60c74741e93378c4f70820a93c

Just makes sure you always know what will be built.

[SoapGentoo]

@davisking I was expecting this (and I'm totally on your side). Using a C++ library (even of such stellar quality as dlib truly is) requires intricate knowledge of build systems, flags, preprocessor macros, shared-vs-static libraries and what not. Yes, I feel your pain, and totally sympathize with it. This is why I initially suggested using static libraries, because for beginner-user-joe-sixpack, -ldlib works (given that you're also providing the private linking flags). Shared libraries are oftentimes more involved, and require LD_LIBRARY_PATH, which is generally beyond the reach of normal users.

I think it goes without saying that any breakage caused by this has to be picked up by me, given that I am the one trying to get this into dlib.

I have a different suggestion for you if you consider my proposal too invasive (and it's not without risk, given that anyone one configuration will never again produce both static and shared libraries simultaneously):

Add an explicit (non-CMake idiomatic) BUILD_STATIC_LIBS flag that is ON by default. This will lead to a situation where by default nothing really changes, both static and shared libs are produced. Yet, it allows us to disable building of static libs. This is not my favoured approach, because it goes against best practices and NIHes new variables, but it wouldn't change anything by default for users, unless they start mucking with those flags.

Which approach do you prefer?

[davisking]

Na, just do the idiomatic thing you proposed. The main source of user pain with regards to building is from users who want to install dlib and then use it, which is a different issue than this and not impacted by it. For instance, it's very common for people to build dlib into a static library and then copy the static lib file somewhere on their hard drive and use it with a different copy of dlib with a different config.h file. So what I really need to do, now that I'm thinking about it, is figure out how to create a linker error that happens in that case and includes a message telling them what they did.

Anyway, do what you originally proposed. If people complain to me about it later we can figure something out then. Just do the right thing now and how to educate new C++ users can be a problem for the future :)

[MyraBaba]

I am first class candidate of the "beginner-user-joe-sixpack" for C++ :) (java-scala guy)

How can I make a standalone working executable which include all required libs ?

I want to build in my book pro an executable and want to run another same config mac book pro ?

Is there any way like big-jar in our world ?

Save my life . pls :)

[davisking]

http://dlib.net/compile.html tells you exactly what to type to compile the example programs, which are standalone executables. Use the cmake instructions, especially if you are new to C++. Read the example's http://dlib.net/examples/CMakeLists.txt.html. It is a tutorial.

[MyraBaba]

Thanks,

I tried to make standalone workable file for "webcam_face_pose_ex" . like below but no succes.
Anychance to have an example for this file ? Could be useful for our smal world....:)

Best

[davisking]

Read the CMakeLists.txt in the examples. It's literally a tutorial that tells you how to do what you just asked.

[MyraBaba]

Dont shoot me we are in the same side :).

Thanks I will read again with quality time. Let you know the achievement for the other hidden rookies :)

Best

On 15 Dec 2017, at 15:58, Davis E. King <[email protected] [e-mail]> wrote:

Read the CMakeLists.txt in the examples. It's literally a tutorial that tells you how to do what you just asked.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/davisking/dlib/issues/923#issuecomment-351999661, or mute the thread https://github.com/notifications/unsubscribe-auth/AQscn_rmwGNQWuVIPFlW0n3_lvcurk6kks5tAmzagaJpZM4QTHp-.

[SoapGentoo]

This is the main problem of C++: you can only make it so much beginner-friendly. There are so many concepts involved that most Java/Python/R types have not heard of, that making it easy is generally not extremely easy. @davisking I get your point now :wink: (and will try to send in a PR over christmas).

[MyraBaba]

I am coding scala and java at least 5 years... I said myself how it could be difficult to play with c++ and getting familiar .. and here we are. :)

PS: Are already cursed by the IDEs . (using Clion). We used to breath with IDEs many years :((

[davisking]

If you use cmake it’s pretty easy.

[xsacha]

I really don't think this is anything to do with C++ (have you had to change any source?) but rather the buildsystem. As @davisking says, CMake is easy. For even more help you can use cmake-gui which shows you possible options and lets you configure with touch of a button.

Although I've only seen cmake-gui once or twice I am pretty sure I remember it handles the BUILD_SHARED_LIBS inbuilt variable for you too.

This is how every CMake project does it so if you get used to this, you'll know how to use all of them.

[davisking]

@xsacha yeah yeah, you guys can submit a PR that changes how BUILD_SHARED_LIBS works. I've come around to it and think it _probably_ won't increase the confused question rate :)

[hlef]

@davisking,

While I recognize the simplifying effect of these changes, defining only one dlib target in CMake has broken the Debian dlib package.

The Debian source package is organized as following:

• libdlibX-0: shared library
• libdlib-dev: (among others) static link files + development include files
• libdlib-data: models

We want to build everything at once. Before these changes, it was trivial; we only had to build one time and then split the resulting files in the different binary packages. Now it's a bit more tricky, AFAIK without changes to the CMake file it isn't possible to build everything at once.

What it your preferred solution to get the same behavior as before, that is, build both shared and static libraries in one run ? (Yes, I can still revert the changes made in this PR, I just ask in case you have alternative solutions)

Thanks for your work.

-- Hugo

[davisking]

Yes, it used to be that way and in many ways that was better. But it seems like the old way confused some people and hopefully the new way confuses less people. At the very least the new setup is more idiomatic. Just because something is more popular doesn't mean it's right though. But I could give you a list of examples of people who got confused and messed things up and how the new setup avoids their problems. But I'm sure there will be other people who are confused now as well.

Anyway, run cmake 2 times to generate the two configurations. One of the reasons I made this switch was because there were package maintainers that ran the build scripts once, cherry picked files out, and broke dlib's cmake scripts in their packages, causing users to get errors and complain to me. The new setup involves no cherry picking like that so hopefully package maintainers will mess up less.

[hlef]

But... generating two different configurations implies building two times, right ? This is not really something we want to do when building the Debian package, so I guess I will have to patch the build system to get the previous behavior back.

[davisking]

The old version built two times as well. Think about it. One with -fPIC and another time without.

[SoapGentoo]

@hlef Trying to emulate libtool's double-building mode with CMake is just a pain mostly. Building it with with a flag switched should be easy.

[hlef]

The point is, I see two solutions to this problem; the first one puts the build complexity into upstream's build system (CMake scripts generating both shared and static files "at once"), and the second puts it into the packaging system (builds two times with two different configurations).

I usually tend to avoid placing too much complexity into the packaging system because I think a package should be as straightforward as possible. However in this case, maintaining an alternative CMake config is going to be really painful, while writing the packaging system's rules so that it builds two times won't.

So, I guess the solution is pretty evident.

Anyways, thanks for the help.

[davisking]

What do other packages do? It is very common for libraries built with CMake to use BUILD_SHARED_LIBS exactly as dlib does now. It seems to be the most common setup. I'm sure many other debian packages face exactly the same issue. I wager mightily they simply run cmake two times.

[SoapGentoo]

Hi agree with @davisking here. While it might have broken a workflow, using a simpler CMake system means I don't have to jump through hoops to make it do the stuff I need it to do (linking tests only against the shared library?). Upstream build systems should be as vanilla and simple as possible. Integration stuff should happen downstream.

[hlef]

Yes, many packages face this issue, and a lot of them build two times with different configurations. This is a bit counter intuitive to me, but my core competence is not library packaging, so I might be wrong.

Meanwhile, I found a pretty elegant way to do it with our debhelper toolchain, so I'm going to opt for this option. Thanks !