Conan-center-index: [package] Many packages missing pdbs...strategy?

[rconde01]

Package and Environment Details (include every applicable attribute)

• Package Name/Version: e.g. openssl/1.1.1g, gdal/3.1.0, cspice/0066
• Operating System+version: Windows 10 x64
• Compiler+version: Visual Studio 2019
• Conan version: conan 1.26.0
• Python version: Python 3.7.2

Steps to reproduce (Include if Applicable)

Doing a debug build on windows I'm getting warnings for many missing pdbs (would also apply to RelWithDebInfo). I can create issues and potentially PRs for each one, but since it appears systemic I thought it justified further discussion. I'm also not sure if the additional size is a problem, and thus maybe there should be an option 'with-pdbs' so that cci doesn't need to manage them. Although, in that case better not produce them at all.

[sourcedelica]

I am in favor of having PDBs in the debug Windows packages. That would put them on par with the Linux packages. Our developers expect debug packages to have symbols.

[Croydon]

See https://github.com/conan-io/conan-center-index/wiki/Error-Knowledge-Base#kb-h017-pdb-files-not-allowed

[rconde01]

Ouch - that seems fairly problematic. I would really like to see then a move towards using a 'with-pdbs' option which defaults to false as standard practice so you can build them locally with the same recipe.

[madebr]

Maybe consider adding the /Z7 option:
https://docs.microsoft.com/en-us/cpp/build/reference/z7-zi-zi-debug-information-format?view=vs-2019#z7

[rconde01]

At way level are you suggesting to use Z7? For the packages it would be ok for debug builds, but no good for relwithdebinfo. At that point maybe it’s better to be consistent.

[madebr]

What /Z7,'/Zi and /ZI does is adding information for the debugger.
So it's perfectly fine to add it for all build types except Release, which one expects to not contain any debug overhead, and MinSizeRel.
The difference in these flags is where/how the compiler stores this information.

The/a difference between RelWithDebInfo and Debug, is that the former defines the NDEBUGmacro during compilation and the latter DEBUG.
And maybe different runtime (compiler.runtime)

[rconde01]

Well I think you want debug information for your delivered binaries, so when your customers give you a crash dump you can get symbols. So this is RelWithDebInfo, but you want this as pdbs because you don’t want the bigger binary sizes (nor deliver your symbols to the customer generally).

[madebr]

Indeed.
Unless it's GPL'ed code :smile:

[sourcedelica]

If there's a need for some to not deliver symbols to an external customer (ours are internal) then it seems like an option to generate PDBs (or /Z7) is the best choice. We really don't want to fork every third party recipe that we use from Conan Center just to get symbols for Windows.

[rconde01]

Did you mean Zi? I’m not sure the exact point you’re making. My current idea is not to fork, but to have a “with-pdb” option on windows that you could specify but which would default to false so that the cci rules are still followed. Although I think it might be desirable to use Z7 for static libraries...I’ve had issues with pdbs being located when using Zi with static libraries...but IIRC I had trouble reproducing it in a simple config so :shrug:.

[rconde01]

An additional problem here is that even apart from the pdb source location not being valid, generally a package will not include the source. This should be equally true on all platforms. OTTOMH I’m not sure if there’s an easy way to ensure you have the sources for all requirements...if not that’s something to consider as well.

[madebr]

It would be nice to use the cci recipes to build your program in RelWithDebInfo/Debug and have pdb's for all dependencies available.
As written by @rconde01 , this would make debugging releases super convenient.

I feel this issue touches a similar problem as https://github.com/conan-io/conan-center-index/pull/1783 .

We should separate two things here: what we want the recipe to provide and what we want to/can provide on bintray.

The recipe should be as general as possible. This means providing pdb's for all libraries/executables.
The disk space constraint, imposed by bintray, can be solved by adding a with_pdb for each recipe buildable on MSVC. As proposed up here.

This would require (at least) the following actions:

• For a particular recipe:
  
  
  
  • Agree on how we should name this new option. pdb? with_pdb? with_pdbs?
  
  
  • Add a with_pdb option to msvc capable recipes, with default value False. (assuming that name is chosen)
  
  
  • The with_pdb options should be kept when shared=False. So the final binary has these symbols available.
  
  
  • Change whatever needed to build and package to add pdb support.
  
  
  • Add the following code to config_options:
  
  

if self.settings.compiler != "Visual Studio":
    del self.options.with_pdb
    # question: Should this options also be available when self.settings.build_type == "Release"?

• For conan hooks:
  
  
  
  • The hooks should check, when options.get_safe("with_pdb", False) is True'ish, whether pdb's are available in the package. This would override KB-017.
    
    Also pass when self.options.get_safe("with_pdb", False) and options.get_safe("shared", None) == False and len(pdb_files) == 0
  
  
• Conan: I'm not sure about this
  
  
  
  • The /Zi option should be added to the C/CXXFLAGS when with_pdb == True
    
    I'm not sure whether this should be handled by the build helpers or by the recipe.
    
    It would be nice if conan would support this as it does for fPIC.
    
    See this piece of code how it is done right now.
  
  

Opinions?

[sourcedelica]

I'm assuming the developer would get the source from Github or wherever it came from - though it sounds like there may be an issue with the VS debugger mapping the source references in the PDB to the downloaded source (easy to do with gcc)? Actually for our developers just having the symbols + file names + line numbers would be enough to let them follow the source in the Github web UI.

[rconde01]

@sourcedelica IMO that’s way too manual and giving up a lot. Normally you can move around the callstack and jump to the corresponding source locations, inspect variables etc. You wouldn’t want to give that up.

[rconde01]

@madebr i need to think about it more thoroughly but I think you have hit most of the main points. A few thoughts:

• As I said previously I think you may actually want Z7 for static libs. In this case mentioning “pdb” seems inappropriate...so maybe something more generic like “keep_symbols”.
• Note that many recipes already generate the pdbs, and then actively delete them.

[madebr]

@rconde01

* As I said previously I think you may actually want Z7 for static libs. In this case mentioning “pdb” seems inappropriate...so maybe something more generic like “keep_symbols”.

This should be investigated. I think the /Zi option should be kept. So the linker will create a pdb when linking. No pdb will be created in that particular package, but the consuming packages can make use of its pdb capability.

* Note that many recipes already generate the pdbs, and then actively delete them.

This is fine, but only when with_pdb is Falseish.

[sourcedelica]

@rconde Definitely but that's what they are doing right now :). The other options are mapping source from the PDB to a cloned Git repo or building the recipe from source.

[jeremy-coulon]

What is the point of having VS debug binaries on Bintray without debug symbols? I would prefer that recipes give me debug symbols by default when I build it locally.

[rconde01]

What is the point of having VS debug binaries on Bintray without debug symbols? I would prefer that recipes give me debug symbols by default when I build it locally.

It's a good question. I think at the least it will be linked to a consistent runtime. Of course, I think we would all prefer symbols by default, but we also understand there are space constraints on conan-center. I guess an alternative is with-pdb = true by default, and disable it for all cci packages.

[rconde01]

@memsharded I would be good to get some thoughts from the conan team on this. Any ideas on how this could move forward?

[uilianries]

@rconde01 Conan team will discuss it during this week, please, be patient.

[rconde01]

I apologize, I just didn't know if it "scrolled off the feed" as the discussion happened over the weekend.

[uilianries]

np, PDB has been requested many times, but as described in the wiki, Conan has a reason for not adopting it. Indeed for remote debug is very useful.

[rconde01]

bump - it's been a few weeks, any thoughts?

[uilianries]

Yes! Conan team just discussed more than once about this topics, the last meetings was few hours ago. Sorry the delay!

There were some options, pros and cons ... anyway, there will be a way to provide PDB files through an extra package folder, which will be optional. This feature will cover more cases, like when users want to package benchmarks or sanitizer results. It need to be studied, but can sure there will be an option for PDB distribution. Thanks for pushing it, this kind of discussion and community engagement it's really important to find new solutions!

[rconde01]

intriguing...looking forward to further details.

[SSE4]

just in case someone interested, here the summary of approaches discussed with their pros and cons:

1. do nothing

2. cons: users need to build locally (from sources) and modify exported conanfile or fork it, not that simple

3. add new configurations (build types), such as DebugWithDebugInfo

4. cons: configuration names are really weird, obscure and confusing (most people expect debug to contain debug info by default)
5. cons: increases matrix of supported configurations

6. cons: tooling problem, need to match them with IDE and build system configurations

7. add new options like with_pdb or without_pdb

8. cons: doesn't scale, options have to be added into each recipe, plus code handling them
9. cons: PDB is not only debug information format, the same story actually applies for .dwarf files on macOS, for instance

10. cons: option actually has no effect on ABI or package ID, thus it doesn't seem it should be an option at all

11. remove .pdb files in conan center hook as post-process (pre-upload, post-package)

12. pros: single place, elegant implementation, keeps recipes clean and simple

13. cons: locally generated packages aren't the same as packages provided on conan center, might be source of confusion for users (e.g. why am I getting 2 Gb build locally but 100 Mb build from bintray?), no longer truly reproducible

14. add auxiliary archive(s) in addition to conan_package.tgz

15. pros: aligns with other package managers (e.g. debian allows to have optional openssl-dev, openssl-doc, openssl-src and openssl-dbg in addition to the openssl main package)
16. pros: allows to handle supplemental files for various use cases besides debug information (e.g. to package benchmark results)
17. pros: doesn't affect on main conan package and leaves it untouched (same revision, same hash)
18. cons: requires significant development effort on conan side, probably also on server side (bintray)

some summary:

• right now we're not going to store debug information on conan center, neither as PDBs, nor as binaries compiled with /Z7 - they contain lots of space and will slow down downloads and consume additional disk space for users, although very few users will actually try to debug 3rd-party libraries
• we're not likely to going with approaches 1-3 as they introduce more headache than benefit

[rconde01]

Still digesting this, but i wanted to capture that if the packages won't have pdbs then to not generate them at all rather than delete them. This should, I think, fix warnings that the pdb is missing.

[rconde01]

btw if the same issue applies to MacOS I think you could go with "symbols" rather than "pdb"...and technically you can strip symbols on linux and do the same.

[rconde01]

One issue I see with the archive solution is that the compiler needs to find the pdbs which IIRC generally means they need to be located next to the dlls (at least with msvc). This requires a capability to overlay an archive on top of the normal package, rather than in a separate folder.

[SSE4]

@rconde01 not really, PDBs could be loaded from anywhere. you may locate them manually with load symbols dialog, or add to the symbols file path. you may even upload em to the symbols server and use from there.

[rconde01]

@SSE4 That's on the debugging side...I'm talking about at compilation time e.g. for static libraries.

[rconde01]

And now that i think about it, maybe i'm just dumb but i've found pdbs with static libraries to be problematic (although it's possible that was just from a package?)...but in any case, the archive approach doesn't play nice with Z7 I think.

[rconde01]

Not that i'm married to it, but thoughts on 3.

3. add new options like with_pdb or without_pdb

* cons: doesn't scale, options have to be added into each recipe, plus code handling them

yes - you need to add the option, and propagate it down to dependencies. But you'll need code the handle the archive too right? And regardless you're gonna need the code to ensure they're generated for debug/releasewithdebinfo

* cons: PDB is not only debug information format, the same story actually applies for .dwarf files on macOS, for instance

Yeah - as mentioned in my other comment..."with-symbols" or "with-debug-info" would work.

* cons: option actually has no effect on ABI or package ID, thus it doesn't seem it should be an option at all

Maybe not ABI, but it potentially has an effect on the binary. The pdb name is embedded in the dll. I don't think symbols are the same as docs like some "independent extra thing".

So - i don't find those cons particular strong. Also you didn't list any pros:

• Relatively simple
• Doesn't require any additional conan features
• Works for static libraries
• Can put it in your conan profile (I think)

[uilianries]

It's a not good idea, today is PDB, tomorrow will be with_benchmark, after, with_sanitizer ... It's simple, but also is a good excuse for adding new options in the future. I also considered option 3 at beginning, but in long term it will be a big problem, I don't want to maintain one more option, juse like fPIC, where we need manage when to remove or not.

Following option 5, you can add anything there, which will work not only for PDB, but also for any other file that users have requested for a long time, benchmark, for example. So we can hit two birds with a single rock.

[rconde01]

Ok - forget about option 3...what about the problems I mentioned with option 5?

[rconde01]

Any progress on this issue?

[boussaffawalid]

We are also very interested in having pdb files. Option 5 sounds like the most reasonable approach. Is there any decision so far ?