Meson: Built-in Boost dependency takes precedence over explicitly provided pkg-config dependency

[agurtovoy]

I'm using meson together with conan/pkg-config (on Windows). One of the project dependencies is Boost, and conan dutifully generates the corresponding boost.pc file, but meson ends up trying to use the built in BoostDependency class before even checking whether the boost.pc file exists (https://github.com/mesonbuild/meson/blob/master/mesonbuild/dependencies/base.py#L2554):

def _build_external_dependency_list(name, env: Environment, kwargs: Dict[str, Any]) -> list:
    # First check if the method is valid
    if 'method' in kwargs and kwargs['method'] not in [e.value for e in DependencyMethods]:
        raise DependencyException('method {!r} is invalid'.format(kwargs['method']))

    # Is there a specific dependency detector for this dependency?
    lname = name.lower()
    if lname in packages:
        # Create the list of dependency object constructors using a factory
        # class method, if one exists, otherwise the list just consists of the
        # constructor
        if getattr(packages[lname], '_factory', None):
            dep = packages[lname]._factory(env, kwargs)
        else:
            dep = [functools.partial(packages[lname], env, kwargs)]
        return dep

    candidates = []

    # If it's explicitly requested, use the dub detection method (only)
    if 'dub' == kwargs.get('method', ''):
        candidates.append(functools.partial(DubDependency, name, env, kwargs))
        return candidates

    # If it's explicitly requested, use the pkgconfig detection method (only)
    if 'pkg-config' == kwargs.get('method', ''):
        candidates.append(functools.partial(PkgConfigDependency, name, env, kwargs))
        return candidates

Other pkg-config dependencies are picked up as expected. Here's my meson.build:

project('xxx', 'cpp',
    version: '2.2.1',
    meson_version: '>=0.50')

sources = files(
    'logger.cpp',
    'dll_main.cpp'
    )

inc = include_directories( '.', 'resources' )
shared_library('logger', sources, include_directories: inc, 
    dependencies: [
        dependency('pantheios'),
        dependency('boost'),
        dependency('stlsoft'),
        dependency('zlib')
    ])

Thoughts?

[dcbaker]

I don't have a .pc from my boost install on Linux, and looking at their trac, they don't have pkg-config support. Is conan generating the .pc file for you?

[agurtovoy]

@dcbaker Yep, together with .pc files for all my other dependencies. The boost.pc itself looks fine to me (this one is on Mac):

prefix=/Users/xxx/.conan/data/boost/1.70.0/conan/stable/package/2439fa14ae114198bd340332be9c9f3e28aa8c03
libdir=${prefix}/lib
includedir=${prefix}/include

Name: boost
Description: Boost provides free peer-reviewed portable C++ source libraries
Version: 1.70.0
Libs: -L${libdir} -lboost_wave  -lboost_container  -lboost_contract  -lboost_exception  -lboost_graph  -lboost_iostreams  -lboost_locale  -lboost_log  -lboost_program_options  -lboost_random  -lboost_regex  -lboost_serialization  -lboost_wserialization  -lboost_coroutine  -lboost_context  -lboost_timer  -lboost_thread  -lboost_chrono  -lboost_date_time  -lboost_atomic  -lboost_filesystem  -lboost_system  -lboost_type_erasure  -lboost_log_setup  -lboost_math_c99  -lboost_math_c99f  -lboost_math_c99l  -lboost_math_tr1  -lboost_math_tr1f  -lboost_math_tr1l  -lboost_stacktrace_addr2line  -lboost_stacktrace_basic  -lboost_stacktrace_noop  -lboost_unit_test_framework  -Wl,-rpath,"${libdir}"
Cflags: -I${includedir} -DBOOST_USE_STATIC_LIBS
Requires: zlib bzip2

...it's just that the code in _build_external_dependency_list I quoted above finds and returns the built-in BoostDependency class before checking if there is a .pc, and the BoostDependency class doesn't pick up the .pc configuration either.

[agurtovoy]

@dcbaker What if instead of unconditionally returning the built-in dependency detector (when present), _build_external_dependency_list returned a list of candidate detectors that included the built-in dependency?

[Spide85]

Have the same issue, showstopper, please fix this.

[jpakkane]

Does it also fail if you set the method explicitly:

boost_dep = dependency('boost', method: 'pkg-config')

[agurtovoy]

@jpakkane Yep:

src/meson.build:15:0: ERROR: Unsupported detection method: pkg-config, allowed methods are auto and auto

That was actually the first thing I tried; I should have mentioned it in the original report.

[nirbheek]

I think this is fairly easy to fix: meson supports a priority order of different dependency methods. We can always check for boost using pkg-config first.

[jpakkane]

Except that Boost upstream does not provide .pc files. They are provided by other people. Thus there is no guarantee that they work or that they even have the same names on different platforms (as they can be provided by different groups).

[agurtovoy]

@jpakkane If they exists, I'd assume that they work, otherwise how/why would they even be there?

Also, Conan is pretty big nowadays; its official Boost package has been downloaded 27,000 in the last 30 days. Granted, most people use it with CMake, but I've been using Conan+Meson for a couple of months now (on Mac and Windows), and I'm loving the ease of use, ease of comprehension, and hackability of both (it helps that both are written in Python).

I should note that, when using Boost with Conan, this issue only surfaces when you invoke Meson directly on the build dir; when you invoke Meson through conan build .., Conan also sets up BOOST_ROOT, thus masking this issue. Personally, I think the overall situation is super misleading, since all the other dependencies are picked up through their .pc files, and w/o knowing that dependency('boost') is special-cased in Meson to ignore the .pc, you can bang you head against the wall for a while trying to figure out why this sometimes works and sometimes doesn't.

[agurtovoy]

Re: the .pc file name, Conan dependency is called "boost" and generates boost.pc.

[nirbheek]

There are plenty of upstream projects that don't provide .pc files, but some distro patches it in for them. I would consider Conan to be another one of those for this case.

[kpeace]

Same issue. Would love a fix.

[jpakkane]

How do the various projects that provide Boost with .pc files do their naming? If you use, say, Boost Graph, what is its pkg-config name? boost-graph? Boost-Graph? boostgraph? Something else? Is it the same for all providers or is it named differently in different pc files?

As a workaround, if you _know_ that on your platform the dependency is called boost-something, you can get it to work by doing dependency('boost-something'). The special casing code only kicks in if the dependency name is exactly boost.

[agurtovoy]

@jpakkane WRT Conan, if you use granular Boost packages, e.g. boost_multi_index/1.69.0@bincrafters/stable, boost_range/1.69.0@bincrafters/stable, etc., the .pc files will be named after the package: boost_multi_index.pc, boost_range.pc, etc.

Personally, I wouldn't expect dependency('boost') to pick these up; I'd just mirror my Conan dependencies in meson.build the same way I do it for everything else: dependency('boost_multi_index'), dependency('boost_range'), etc.

The problem with (Conan) granular packages is that Boost's dependency graph isn't really that clean. You think you'd gain if you just pulled in the packages that you use, but in practice it ends up being a pessimization. AFAIK the majority of people use the "monolith" boost package.

[agurtovoy]

Personally, I wouldn't expect dependency('boost') to pick these up; I'd just mirror my Conan dependencies in meson.build the same way I do it for everything else: dependency('boost_multi_index'), dependency('boost_range'), etc.

Actually, scratch that; given the Boost dependency graph, doing the above manually is impractical. For example, pulling in just boost_multi_index, boost_range and boost_utility will also bring in all of the following:

boost_algorithm.pc      boost_function.pc       boost_proto.pc
boost_array.pc          boost_function_types.pc     boost_range.pc
boost_assert.pc         boost_fusion.pc         boost_ratio.pc
boost_atomic.pc         boost_integer.pc        boost_rational.pc
boost_bind.pc           boost_intrusive.pc      boost_regex.pc
boost_chrono.pc         boost_io.pc         boost_serialization.pc
boost_concept_check.pc      boost_iterator.pc       boost_smart_ptr.pc
boost_config.pc         boost_lambda.pc         boost_static_assert.pc
boost_container.pc      boost_lexical_cast.pc       boost_system.pc
boost_container_hash.pc     boost_locale.pc         boost_throw_exception.pc
boost_conversion.pc     boost_math.pc           boost_tokenizer.pc
boost_core.pc           boost_move.pc           boost_tti.pc
boost_cycle_group_a.pc      boost_mpl.pc            boost_tuple.pc
boost_cycle_group_b.pc      boost_multi_index.pc        boost_type_index.pc
boost_cycle_group_c.pc      boost_numeric_conversion.pc boost_type_traits.pc
boost_detail.pc         boost_optional.pc       boost_typeof.pc
boost_endian.pc         boost_phoenix.pc        boost_unordered.pc
boost_exception.pc      boost_pool.pc           boost_utility.pc
boost_filesystem.pc     boost_predef.pc         boost_variant.pc
boost_foreach.pc        boost_preprocessor.pc       boost_winapi.pc

Which is why people stick to the monolith package. IMHO it still doesn't mean that Meson's dependency('boost') needs to pick up these. The issue here is not so much about Boost dependencies as it's about transitive dependencies. For example, the above also pulls in bzip2.pc. Being able to automagically pass all the transitive dependencies from Conan to Meson would solve this and a bunch of other use cases, which is why I'm eyeing https://github.com/mesonbuild/meson/pull/5209.

[Spide85]

My current workaround:

export PKG_CONFIG_PATH=$(realpath ../build/)
export BOOST_ROOT=$(pkg-config --variable=prefix boost)
meson ../build

[jpakkane]

Being able to automagically pass all the transitive dependencies from Conan to Meson would solve this

This is what Pkg-config already does if you set it up correctly. If your pc files contain the correct transitive dependencies, everything will just work.

Or, in other words, if in this case dependency('boost_something') returns a dependency that does not work but requires manually adding dependency('bzip2'), then the .pc configuration is set up incorrectly.

[agurtovoy]

Or, in other words, if in this case dependency('boost_something') returns a dependency that does not work but requires manually adding dependency('bzip2'), then the .pc configuration is set up incorrectly.

Thanks for the clarification. Conan does generate .pc files for Boost correctly (see below); I'll double-check my specific use case that was giving me trouble.

boost_array.pc:

prefix=/Users/xxx/.conan/data/boost_array/1.69.0/bincrafters/stable/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
libdir=${prefix}/array/lib
includedir=${prefix}/array/include

Name: boost_array
Description: Shared python code used in other Conan recipes for the Boost libraries
Version: 1.69.0
Libs: -L${libdir} -Wl,-rpath,"${libdir}"
Cflags: -I${includedir} -DBOOST_ALL_NO_LIB=1
Requires: boost_assert boost_config boost_core boost_static_assert boost_throw_exception

[Randshot]

Is there an update for this?
I think that the best initial solution would be to just respect explicitly given method arguments, and error when unsuccessfull:

boost_dep = dependency('boost', method: 'pkg-config')
boost_dep = dependency('boost', method: 'cmake')

[agurtovoy]

@Randshot I agree that these two should definitely work, but IMHO it's not sufficient. dependency('anyotherlib') works as expected if anyotherlib.pc is present in the build dir for, well, any library other than Boost. I don't see a good reason why I'd have to spell this particular one as dependency('boost', method: 'pkg-config') to make it work. It's inconsistent and IMHO just weird:

deps = [
    dependency( 'fmt' ),
    dependency( 'gsl_microsoft' ),
    dependency( 'range-v3' ),
    dependency( 'boost', method: 'pkg-config' ) # ???
 ]

[Randshot]

That is true, but as there isn't a consensus on how to handle the module names yet, I thought that there should at least be an initial fix for this. This is really bugging me TBH.

[agurtovoy]

@Randshot Could you clarify where is your .pc for Boost is coming from and what does it look like?

[Randshot]

@agurtovoy I, like you, am using Conan for generating the pkg-config files. It's really the best option on Windows for getting the Boost libraries.

The generated file looks like this (in the case of the monolithic package):

prefix=C:/.conan/13e461/1
libdir=${prefix}/lib
includedir=${prefix}/include

Name: boost
Description: Boost provides free peer-reviewed portable C++ source libraries
Version: 1.69.0
Libs: -L${libdir} -llibboost_wave  -llibboost_container  -llibboost_contract  -llibboost_exception  -llibboost_graph  -llibboost_iostreams  -llibboost_locale  -llibboost_log  -llibboost_program_options  -llibboost_random  -llibboost_regex  -llibboost_serialization  -llibboost_wserialization  -llibboost_coroutine  -llibboost_fiber  -llibboost_context  -llibboost_timer  -llibboost_thread  -llibboost_chrono  -llibboost_date_time  -llibboost_atomic  -llibboost_filesystem  -llibboost_system  -llibboost_type_erasure  -llibboost_log_setup  -llibboost_math_c99  -llibboost_math_c99f  -llibboost_math_c99l  -llibboost_math_tr1  -llibboost_math_tr1f  -llibboost_math_tr1l  -llibboost_stacktrace_noop  -llibboost_stacktrace_windbg  -llibboost_stacktrace_windbg_cached  -llibboost_unit_test_framework  -lbcrypt 
Cflags: -I${includedir} -DBOOST_USE_STATIC_LIBS -DBOOST_ALL_NO_LIB
Requires: zlib bzip2

And manually setting the BOOST_ROOT env var also doesn't work for my setup, since it looks for the ABI specific library names:

LINK : fatal error LNK1104: cannot open file 'libboost_date_time-vc141-mt-gd-x64-1_69.lib'

[Randshot]

The separate module packages available from Bincrafters might be a solution, but the current lzma package requires an older Windows SDK than I can install for VS2019, and I don't want to install VS2017 again, so yeah.

[agurtovoy]

Yeah, I wouldn't recommend those anyway. They take much longer to install and the Boost dependency graph is too intermingled to make it worth the extra slowness and complexity.

I should mention that if you use Conan to build your project, it should still work as per https://github.com/mesonbuild/meson/issues/5438#issuecomment-516050754.

[Randshot]

It does work, if I use Conan to invoke meson, but that somewhat defeats the purpose of using meson in my opinion.

On the note of setting BOOST_ROOT, like I wrote above, that doesn't seem to work for my setup, as meson is looking for ABI specific library names:

LINK : fatal error LNK1104: cannot open file 'libboost_date_time-vc141-mt-gd-x64-1_69.lib'

[Randshot]

@agurtovoy Ok, I think I have a way of using Boost on Windows with Meson and Conan. It is not optimal, but it kinda works.
I am using both the pkg_config and virtualenv generators in the conan file.
The generated activate.bat/activate.ps1 will set the matching BOOST_ROOT env var.
I also had to make sure that I add the following build argument -DBOOST_AUTO_LINK_SYSTEM.
This will prevent the linking error in my post above.
Like I said, it is not an optimal solution, but it works for now.

[agurtovoy]

@Randshot Thanks for sharing! I plan on looking into a fix for this, but it's probably going to be another couple of weeks before I get to it. I'll be delighted if someone else beats me to it ;)

[sandervalcke]

I've also been running into this. In case it helps I created a sample project for reproducing the problem here.

[TheQwertiest]

This is also inconsistent with other custom meson dependencies: gtest dependency allows to specify a search method (i.e. works with pkg-config).

[mensinda]

In 0.54.0, the internal BoostDependency was completely rewritten, so that it automatically picks the correct libraries (with ABI tags, etc.) in a given BOOST_ROOT, so the linking problem from @Randshot should be resolved. If not please open an issue.

Adding *.pc support is definitely possible, but I would rather try fixing the internal dependency since supporting all the boost_* looks like a huge mess and the boost.pc adds everything in boost. In short the Conan *.pc files are far from ideal IMHO.

I have only skimmed through this issue, but from what I can tell it looks like this is more or less a Conan specific problem correct?

Does Conan have a predictable directory layout or are there some environment variables that would help meson automatically compute the BOOST_ROOT? Alternatively, where is the boost.pc file locate relative to the BOOST_ROOT? If this is a known location meson could use PKG_CONFIG_PATH (with the existence of the boost.pc) to calculate BOOST_ROOT.

I have never used Conan, so sorry if these are trivial questions.

[TheQwertiest]

In short the Conan *.pc files are far from ideal IMHO.

Yep. Currently conan does not have a way to specify intra-package dependencies (they are working on that though). So if a user wants to link only a single library from multi-library package, they'll have to extract that lib manually by using find_library(get_variable(pkgconfig: 'libdir') + '/' + libname) in meson.

Does Conan have a predictable directory layout or are there some environment variables that would help meson automatically compute the BOOST_ROOT?

Conan uses a standard directory layout:

• dep_root
  
  
  
  • include[/dep_name]
  
  
  • lib
  
  
  • ...
  
  

.pc files generated by conan contain the following variables (if the package has corresponding artifacts):

• prefix: points to dep_root
• libdir: points to dep_root/lib
• includedir: points to dep_root/include

Alternatively, where is the boost.pc file locate relative to the BOOST_ROOT? If this is a known location meson could use PKG_CONFIG_PATH (with the existence of the boost.pc) to calculate BOOST_ROOT.

.pc files are generated to a conan build directory (not documented and should not be relied upon). They are passed to meson via PKG_CONFIG_PATH.
Dependencies itself are stored in internal conan cache directory, and can be only accessed through .pc files in meson (since those paths are dynamic).

[TheQwertiest]

Note: I've not tried using boost conan package in meson on Windows, so I don't know if the names of libraries provided by conan package are consistent with the expected names in meson. But it just works(c) on *nix after specifying BOOST_ROOT.

[mensinda]

So, running pkg-config --variable=prefix boost would be the most reliable way for getting BOOST_ROOT?

[TheQwertiest]

So, running pkg-config --variable=prefix boost would be the most reliable way for getting BOOST_ROOT?

Yes, I thinks so (at least on *nix).

[TheQwertiest]

@mensinda , but IMO, it would still require the addition of method:pkg-config, since otherwise there is no way to ensure that the proper dependency is fetched (e.g. a pkgconfig dependency and not a system one)

[mensinda]

Why though? Just putting pkg-config --variable=prefix boost root as the first search path should be enough. It will only fall back to a system dependency if there is an issue with the prefix (ABI mismatch, etc.).

[TheQwertiest]

@mensinda , this is needed to minimize potential errors and unexpected behaviour.
A simple example:

1. Let's say we have a CI that has a system boost installed, but for our project we want to use our own boost from dependency manager (might have been built with special options and\or patched or maybe there is just a hard-requirement to use a specific binary).
2. A developer messes up and removes a boost dependency from dependency list or maybe the boost dependency was malformed for some reason. End result: missing boost.pc.
3. The project build starts, sees that there is no boost.pc in search paths and uses system boost instead.
4. Build completes successfully without any errors, but the final binary uses the wrong boost (which might lead to unexpected errors in dependent builds).

[mensinda]

Should be fixed (as in the meson system boost dependency should find a Conan boost installation) with #7185.

@TheQwertiest This problem arguably already exists anyway. A better solution would be to set the minimum boost version in the dependency() call, so that the correct boost dependency is found.

[mensinda]

@agurtovoy, @Randshot could you please check if the linked PR works for you? I have no experience with Conan (and not much with Windows), so checking this myself is a bit tricky.

[TheQwertiest]

A better solution would be to set the minimum boost version in the dependency() call, so that the correct boost dependency is found.

boost versioning is orthogonal with the problem I've described. They might have the same version (or might not). A user might not want to use just any boost (even if it meets the version requirements), he might need a specific boost, and this can be easily achieved by using a fixed method for searching dependencies. E.g. you can't get wrong boost using only wrap or only pkg-config, if they are configured properly.

[mensinda]

To be clear, I am not against adding a pure pkg-config method if it fixes a real problem. However, I don't think that just adding *.pc support will fix the issue you described. A simple example: What happens if you have installed multiple boost versions that provide *.pc files?

There is already a mechanism to specify a specific boost version without fallbacks and that is setting BOOST_ROOT. Meson won't find a system boost, if it can't find what it is looking for inside the prefix. Assuming that there only ever exists one boost version with *.pc files doesn't seem like a foolproof solution to me either (especially since BOOST_ROOTalready does what you want).

Also, you should be able to uninstall (rm -rf should also work in extreme cases) any system boost versions in your CI. If you want to be sure to use the correct version and don't to or can't set BOOST_ROOT this would be the best solution.

[TheQwertiest]

Regarding why env BOOST_ROOT is bad (or at least not really good) I think we've already discussed in in the other issue =)

What happens if you have installed multiple boost versions that provide *.pc files?

AFAIK, meson searches for .pc only in directories specified in PKG_CONFIG_PATH. So if one provides a curated list of these paths, then there won't be a problem.
But (again AFAIK) there is no way to forbid meson from searching system paths.

Also, you should be able to uninstall (rm -rf should also work in extreme cases) any system boost versions in your CI.

It's not always possible: some system packages (system as in installed via apt and friends) might depend on it.

TBH, I don't really see why it's not possible (or undesirable) to add a method:pkgconfig - boost meson dependency already uses it internally, this kwarg would just restrict the available search paths in the internal logic, without changing anything else.

[mensinda]

TBH, I don't really see why it's not possible (or undesirable) to add a method:pkgconfig - boost meson dependency already uses it internally, this kwarg would just restrict the available search paths in the internal logic, without changing anything else.

The internal dependency will use it once #7185 is merged. However, method: 'pkg-config' will just use the boost.pc file without the internal logic of BoostDependency (so basically just return a PkgConfigDependency('boost') instead of a BoostDependency). This is a significant difference with all the disadvantages associated with the monolithic boost.pc (most importantly no modules support).

Also, for a CI environment, wouldn't export BOOST_ROOT=$(pkg-config --variable=prefix boost) be acceptable in your .travis.yml or GitHub workflow? Not ideal but if you install boost via Conan and want to be really sure that only that version is used then this would be IMHO the best option. This way you can also sanity check if BOOST_ROOT is correct before calling meson.

[TheQwertiest]

However, method: 'pkg-config' will just use the boost.pc file without the internal logic of BoostDependency (so basically just return a PkgConfigDependency('boost') instead of a BoostDependency)

Thanks, I didn't know that, I thought method only influenced search methods.

Also, for a CI environment, wouldn't export BOOST_ROOT=$(pkg-config --variable=prefix boost) be acceptable in your .travis.yml or GitHub workflow?

It doesn't work exactly like that (because .pc file locations are not really known outside of conan, it's known only to conan recipe), but it will still require to special-case boost dependency. So, yeah, not ideal =)

Currently I have to use the following code in conan recipes to handle this problem (cleaned up for brevity):

if has_dep('boost'):
    # Gets boost root path and sets it to BOOST_ROOT
    cm = with_boost_dep_env(get_dep_info('boost'))
else:
    cm = nullcontext()
with cm:
    meson.configure(...)

I want to emphasize that this is the only dependency that needs such special handling.

[mensinda]

Well, you only need to set BOOST_ROOT because you want to ensure that your CI doesn't accidentally break. If you don't need that requirement then it just works (TM) with #7185.

So if this additional insurance (required for Conan only as far as I can tell) is the only thing that is missing, I would say that it isn't worth adding support for the plain boost.pc file. Especially since it is possible to support this (arguable niche) problem in Conan, even if it is a bit clunky.