Conan: Make 'shared', 'fPIC', and 'link_time_optimized' settings

[skizzay]

One issue that I run into quite often is adding shared, fPIC, and link_time_optimized for build options. However, these are fundamental settings to the behavior on the compiler/linker, just as indicating that a build is Debug or Release with respect to the flags passed to the components.

When I explain Conan to my coworkers, I usually use the analogy that options answer the question of "What should I build?" and settings answer the question of "How should I build it?" This is seen in the behavior of many of the existing projects listed in conan-center. For example, Boost essentially asks "What parts of Boost do you want to build?" with its options "without_". There are a few options that are listed as how to build Boost: "header_only", "shared", "fPIC". This is because conan doesn't have these configurations listed as settings.

Further, these _options_ are common enough that conan should have these readily available, making it easier for users to create recipes. Conan already acknowledges this in its current behavior. If you specify shared as an option, then the cmake generator/build tool helper will pass in the appropriate flag in order to create shared/static libraries. CMake already acknowledges that fPIC and link_time_optimized are common enough settings that it already supports these settings: CMAKE_POSITION_INDEPENDENT_CODE and CMAKE_INTERPROCEDURAL_OPTIMIZATION

Header-only libraries have special handling with respect to source build, yet seem to have pretty common output with respect to another header-only library. There is a How-To on how to deal with header-only libraries. By allowing the shared setting to have a header_only value, conan can do all of the work that would be otherwise manual for a user creating a header-only library. Conan already knows how to deal with these types of libraries, but the user must specify a lot configuration, which is consistent across these types of libraries. Instead, Conan should automatically update the configuration with the ability for the user to override the "header-only" behavior.

The reason I need to write a conanfile.py rather than conanfile.txt is typically to deal with handling the fPIC and shared options. If these were native settings, then I think it would make Conan much easier for end user because these settings would be handled already.

Bike shedding...
I really don't like the name shared as a setting. The reason that I find it a poor name is that many libraries are header only, in which case shared is inappropriate. In addition some libraries have requirements to support both a shared and static library build. Again, in these cases shared is inappropriate. Regardless of what the setting is named, I see four possible values (with None being allowed for SHA backwards compatibility): shared, static, both, header_only. Again, I'm not convinced that these are the best names for these values, but I think I get across what each value represents.

To help us debug your issue please explain:

• [X] I've read the CONTRIBUTING guide.
• [X] I've specified the Conan version, operating system version and any tool that can be relevant.
• [X] I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[skizzay]

This relates to #2464.

[lasote]

We see options in a different way: http://docs.conan.io/en/latest/creating_packages/getting_started.html#settings-vs-options

I agree that sometimes both concepts can mix , that is why in the PR managing the standard, the approach with options, is almost introducing a new kind of options , "the well known options" like shared or the standard. But that's the best and the worst of that approach. The "well known" options are activated by the user, can be defaulted and Conan knows his values to operate with the build helpers adjusting the flags, but they are hardcoded while setting are not (settings.yml).

[skizzay]

I think in the end, we are really saying the same thing when it comes to settings vs options. Settings will dictate "everything that is built on this site/project will be built this way". Options will dictate "these are the components being built, what features to enable/disable, etc". Any exceptions can be explicitly set in the IDE's/build tool's configuration (e.g. CMake can specifically set SHARED to a library even though we're conducting a static build).

Settings are project-wide configuration, something that typically affect to the whole project that is being built. For example the Operating System or the architecture would be naturally the same for all packages in a dependency graph, linking a Linux library for a Windows app, or mixing architectures is impossible.

By extension you cannot link an object with LTO enabled to one that does not. By that reasoning alone I feel that link_time_optimization should be considered a setting. Continuing along these lines, you cannot take a static built without fPIC and link it into a shared library. Further, we've already chosen that a Release vs Debug build is a setting value, when you can quite easily combine the two in a project (in some classified/DoD environments, this is actually a requirement).

That said, I completely acknowledge that one library will be static while another is shared. I'm actually running into this exact scenario today with OpenSSL. We are building our internal libraries using static, but keeping OpenSSL as a shared library so that security patches can be easily deployed by updating the shared objects.

[skizzay]

Alternative solutions:

1. Have conan new generate fPIC and shared options when passed certain flags. It would also generate a configure implementation that checks the compatibility of fPIC and shared:

   def configure(self):
      if self.options.shared and not self.options.fPIC:
         raise ConanException("All shared objects must be built with position independent code.")

1. Create something in tools that would handle the default setup.

   def config_options(self):
      # By passing in the ConanFile, it'll install a check like the one above that happens automatically before or after configure
      tools.add_fPIC_and_shared_options(self)

Either of the two approaches listed above still require that LTO being a setting. There really isn't a good way to do that reasonably.

[lasote]

if self.options.shared and not self.options.fPIC:

is this correct? the things that have to be built with fPIC are the dependencies, not the shared itself. Has fPIC any sense building a shared library?

[skizzay]

In order to build a shared library, your code must be built with fPIC. If you are linking a static library into your shared library, then the objects contained within that static library needs to be built with fPIC. Checkout this answer in SO.

Essentially, fPIC does not depend on shared/static. However, shared requires fPIC enabled. This is usually handled by CMake when you enable libraries to be built as shared. However, CMake also allows you to enable fPIC independently so that you can have fPIC enabled static builds. This is the case when you build individual static libraries that get linked into a shared library. Those static libraries must be built with fPIC.

This is why I say that even if you are mixing shared and static libraries, your entire project and its dependencies need to have consistency with respect to position independent code and link-time optimizations. It's basically you have your entire project and dependencies use them or don't use them. There really isn't a way to mix the two together cleanly.

[lasote]

Yes, but the "fpic=True" (seeing it as an option, as today is) is needed in the requirements of the shared library, not to build the shared library itself.
And I think it can be easily mixed in the same project:

My Project -> LibA, LibB
LibA (shared) -> LibC and LibD (both with fpic, static)
LibB (static without fpic)

So I still think it is an option.

[skizzay]

If "My Project" results strictly in an executable, then this may be theoretically possible. That said, the reality of the situation is that no one is going to do this. I have never come across a case where the engineering team has decided that they want to statically link one set of components without PIC and build these other components with PIC. The team typically commits to all static with very limited system components being shared (e.g. OpenSSL) or commits to some mixture of shared/static, but the static components must be built with PIC.

Given the example, the assumption is that LibB is reusable by other projects, just as LibC and LibD are. If that's not the case, then LibB would just be rolled into My Project. If it is indeed the case, then this engineering team is going to be running into a huge headache of tracking which static libraries can be combined into a shared library and which cannot. It just makes more sense to make a blanket decision to say that all static libraries are going to built with PIC in this case to prevent the sort of fragmentation described.

Regardless of PIC, your description for LTO fails. If you take your example and replace fpic with lto, then the dependency for "My Project" -> LibB fails.

[skizzay]

I'm also confused how Debug/Release satisfies the "criteria". The explanations provided so far suggest that build_type should be an option. Given the "My Project" example, any of those libraries can be Debug or Release. At the very least PIC holds a technical limitation on that a shared library cannot properly link a static library built without PIC. That doesn't hold true for a Debug build vs a Release build. I can mix those together at any point. In fact, you can have two different .o files built, Debug and Release, and still have them linked together without any issues.

Given that PIC is a stricter build requirement than build_type and LTO being even stricter than PIC, how does Debug/Release build types get to be a setting while the former configurations have to be an option?

[lasote]

Again about my fpic example, it happens specially when lgpl licenses are involved. You maintain the LGPL as shared and embed the rest. But well, let's think about it.

About the LTO, we will think about it too. If you never can mix it, yes, probably it is better a setting.
About the debug/release, it is also problematic to mix them, it is typically project wide, and universal. I prefer it as a setting, but anyway it is not something that we could change without breaking all, so it makes no sense to discuss this.

[lasote]

I labeled this issue to discuss with all the team. My status is: for me "shared" is still an option, but about fpic and lto I'm starting to wonder how should it work and, maybe, you are right. Unfortunately, there are big concerns about how to do it to not break the recipes, but that will be part of the discussion here to take a decision.

[db4]

If fPIC remains an option it should be required one for non-Windows libraries. I was quite surprised to see that e.g. conan zlib recipe does not have fPIC option, so I cannot link static zlib with a shared library.

[lasote]

You can. The fPIC is activated always in zlib.
About this issue, we are on it, I hope we take a decision soon.

[db4]

OK, for zlib fPIC is always enabled. But I still have a problem with a library which does not have fPIC option and where fPIC is not activated for static build: conan-jasper. I believe it's not the only one. Is it a fault of conan-jasper's author or conan should somehow enforce presence of fPIC option for libraries?

BTW, do you know how to check from a recipe if some option is present in a required package? I tried

def requirements(self):
    self.requires("jasper/2.0.14@conan/testing")
    if hasattr(self.options["jasper"], "fPIC"):
        ...

but hasattr() seems to (incorrectly?) return True here.

[skizzay]

@lasote Thanks for looking into this. It's certainly going to make my life easier :1st_place_medal:

[Adnn]

Regarding new settings, and in particular #2534, will the MACOS_DEPLOYMENT_TARGET become a package setting in upcoming versions of Conan?
It is a central aspect of binary compatibility on this platform (more prevalent that the compiler version as we found out in our projects), thus it seems like a prime candidate !

Thank you for reading and the awesome project

[3galki]

Because of fPIC option I can not use precompiled code from conan-center. It is like lottery will I manage to link with it or will I fail. May be you should remove all precompiled code from conan-center or add option (command line or better in registry.txt) to do not use precompiled sources from specified remotes.

[memsharded]

@3galki but even forcing the re-build from source (which you could do with --build), it doesn't solve the issue. If those recipes in conan-center, have hard-coded the fPIC value in their build system, which is not that unusual, then it doesn't matter if you rebuild, it will still get the same incompatibilities. This needs improvements in the existing package recipes. Please report to the package repos whenever you need them with an fPIC option.