Bazel: CPP gdb lldb no debugging symbols found

[llvim]

Please provide the following information. The more we know about your system and use case, the more easily and likely we can help.

Description of the problem / feature request / question:

i build CPP with bazel on my mac, when i debug it

bazel build -c dbg :main
gdb ./bazel-bin/main

it came out:

/main.pic.o': can't open to read symbols: No such file or directory.
(no debugging symbols found)...done.

If possible, provide a minimal example to reproduce the problem:

#WORKSPACE empty
#BUILD 
cc_binary(
 name='main',
srcs=['src/main.cc']
)
#src/main.cc
#include<iostream>
int main(){
  std::cout<<"test";
}
bazel build -c dbg :main
gdb ./bazel-bin/main

Environment info

• Operating System:
  mac 10.12.3

• Bazel version (output of bazel info release):
  release 0.4.0-homebrew

• If bazel info release returns "development version" or "(@non-git)", please tell us what source tree you compiled Bazel from; git commit hash is appreciated (git rev-parse HEAD):

Have you found anything relevant by searching the web? (e.g. GitHub issues, email threads in the [email protected] archive)

no

Anything else, information or logs or outputs that would be helpful?

(If they are large, please upload as attachment or provide link).

FYI:
on ubuntu, gdb work is fine, but lldb still can't find debugging symbols

[llvim]

dSYM is generated,but is is empty, even when add --strip never

/Users/liyanan/dev/clion/bazel-bin/main empty dSYM file detected, dSYM was created with an executable with no debug info

[david-german-tri]

RobotLocomotion/drake developers who use OS X are experiencing this also. We are actively investigating workarounds. If we find something, I'll update this thread, but any guidance from the Bazel team would be hugely appreciated!

[david-german-tri]

Related: https://github.com/bazelbuild/bazel/issues/327.

I'm having some preliminary success running dsymutil from osx_cc_wrapper.sh. With that technique, I can at least produce a usable .dSYM somewhere deep in the sandbox, then manually symbolicate.

[david-german-tri]

Here's an outline of a workaround we've prototyped locally:

1. For every cc_{library,binary,test}, add a genrule like the following:

genrule(
    name = "foo_dsym",
    srcs = [":foo"],
    outs = ["foo.dSYM"],
    output_to_bindir = True,
    cmd = "dsymutil $(location :foo) -o $@"
)

We have binary/library/test wrapper macros that simplify insertion of this genrule across the board.

1. Use --spawn_strategy=standalone to disable sandboxing, so that the DWARF-laden .o files will be visible to dsymutil. (Also, of course, use -c dbg, which in turn sets the compiler flag -g.)

2. If you need to support platforms other than OS X, use a select statement on the cmd, and simply touch $@ on non-Apple platforms.

This has a bunch of deficiencies. It disables sandboxing, it declares a directory output from a genrule (which is unsound), it clutters the build with extra targets, and it doesn't actually build the debug symbols on bazel build :foo. But it at least provides a way to get debug symbols on OS X. If someone else has a better way, I'd be very happy!

[Whiteseeker]

Is the workaround above still the one we should use ?

[hlopko]

CC @c-parsons @calpeyser

[yageek]

@david-german-tri Would it be possible to give some more explanation on how to use the workaround ? I added a genrule after a cc_library library statement but I still get an empty .dSYM error. Should we also add --strip=never to the build/test command ?

[dattanchu]

I use the workaround and can generate the dSYM required to step through the executable with lldb. When we run in sandbox mode, is it possible to access the object files required by dsymutil?

[baskus]

What is the status of this? When will we get debug symbols without hacks?

Thanks! :)

[jmarantz]

+1 -- I had the misfortune of introducing a problem that could be repro'd only on Mac and the question was whether to deal with this hack, or just use printfs. I used printfs, but I next time I can just debug on mac.

[steve-the-bayesian]

+1. This is a major pain point for me. Thanks to everyone who's working to fix it.

[tony-shi]

+1. BTW, will this issue be fixed/upgrade officially or we have to take a hack described above to enable generate debugging symbols?
Thanks!

[hlopko]

I consider it as P2, and I interpret P2 as likely being fixed in the mid term. I just didn't have time to look into it yet (in the last year apparently :).

[kkiningh]

+1. This is also a major pain point for me.

[barbu110]

Anything on this yet?

[hlopko]

No updates. We plan to have this fixed before Bazel 1.0 is released. When is that happening should be announced soon.

[haberman]

@mhlopko Is there a specific plan/design for the solution yet?

[hlopko]

Nope, I didn't investigate this issue at all yet :/

[haberman]

Maybe if we had a plan of record it would be easier to get this implemented in the short term.

Here is my understanding of the issue:

The macOS linker doesn't put debug info in the final binary like Linux does. Instead it leaves the debug info in .o files and other tools read this debuginfo directly out of the .o files.

The linker will store full paths to the .o files in the final binary. So here is a comparison of the same binary built with both Bazel and CMake:

$ nm -a bazel-bin/test_handlers | grep '\.o$'
                 U /private/var/tmp/_bazel_haberman/0c1a0f40a94930ea7d807ae60a66a4ea/sandbox/darwin-sandbox/346/execroot/upb/bazel-out/darwin-dbg/bin/_objs/test_handlers/test_handlers.o
$ nm -a ninja/test_handlers | grep '\.o$'
                 U /Users/haberman/code/upb/ninja/CMakeFiles/test_handlers.dir/tests/test_handlers.c.o

In the Bazel case, the full path to the .o file lives in a temporary, sandboxing directory. It will not be available at this path after the linking step. This is what is causing debugging to fail.

There are also foo.dSYM bundles (directories) that can be created with dsymutil(1). These contain the debugging information fully linked/relocated, and debuggers will look for this file first. So if you are debugging a program foo and a foo.dSYM bundle is available in the same directory, it won't even bother looking for the .o files.

I see two main options for solving this problem:

1. somehow run the linking step such that the absolute paths of the input filenames will stay accessible after the link. This would probably require some kind of tweaking to the sandbox. The linker appears to resolve all symlinks before writing the .o filename into the binary. This option might be hard to make work robustly given all the symlinking that Bazel does.

2. explicitly invoke dsymutil immediately after the link, when all the .o files are still accessible at their original paths inside the sandbox. Then foo.dSYM would be an extra output artifact of cc_binary(foo)/cc_test(foo) rules on macOS. It should go in bazel-bin like the binary itself. Then the debugger and other tools will be able to find it.

Overall (2) seems simpler and easier. It's what @david-german-tri's solution above is simulating.

Does that make sense?

[hlopko]

Thanks Josh this was indeed helpful! Because it made me remember that we do call dsymutil to generate the bundle, but only for apple rules. The next step is to look into apple rules and see how to generalize the implementation.

[rmstar]

Any updates on this?

[hlopko]

Sorry @rmstar, no updates. I don't think I will have time to look into it in the second quarter.

@trybka, @jyknight - do we have this problem internally?

[trybka]

I don't think a ton of folks debug cc_binary targets internally on a mac.

That being said, I'd believe that we need to add dsym generation for cc generally.

[rmstar]

FWIW I'm an internal user working on gRPC. It would be great if we could get a fix for this issue :smile:

[hlopko]

We have fission (.dwo files), so maybe extending the support to .dsym won't be that hard? I'm saying this without actually investigating the details, beware :)

[kkiningh]

Is this still planned on being fixed for the 1.0 release?

[hlopko]

Hi all,

so I finally tried to look into this. Do you all agree that you can debug binaries on a mac just fine when you build them using bazel build -c dbg --spawn_strategy=local? Because that's what I'm seeing. I'm not trying to suggest disabling the sandbox is the fix for this issue, just that this workaround is not as complicated anymore as what was proposed in https://github.com/bazelbuild/bazel/issues/2537#issuecomment-281497449.

Thank you!

[DavidGoldman]

rules_apple supports building dSYMs inside the linking action itself (like @haberman mentioned), although this means slowing down your critical path. If you ever build remotely you'll need dSYMs in order to debug unless your machine's paths matches the builders' paths exactly.

If you don't use dSYMs you'll need to disable sandboxing for the linking step, e.g. setting --strategy=ObjcLink=standalone and --strategy=CppLink=standalone. Like @haberman this wouldn't be a problem if the absolute paths to the .o files were still valid after the linking action finishes.

[wesleyw72]

Hi @hlopko , even with setting --spawn_strategy=local and -c dbg I still can’t get source level debugging. For example typing l <source file path> doesn’t work.

[hlopko]

@wesleyw72 I'm afraid 'll need a repro case for that, it should work AFAIK. Pls include Bazel version and all Bazel options you pass (don't forget the .bazelrc).

[wesleyw72]

@hlopko I think the issue I was hitting is that the cache was being hit, so changing the spawn_strategy wasn't making a difference as no actions were being spawned and thus the object files with the debugging symbols were missing (in a previously cleaned up sandbox). If I do a clean and then build with spawn_strategy=local then it does seem to be working. Thanks!

[nikhilm]

Trying to understand exactly what the deficiency here is, because it seems the macos_* rules from rules_apple can build dsyms.
Specifically, if one wants a dylib on macOS

cc_binary(
  name = "libfoo.dylib",
)

will not produce a dsym (nor will dwarf debug info be preserved in the dylib itself)

but

cc_library(
  name = "foo",
)

macos_dylib(
  name = "libfoo",
  deps = [":foo"],
  ...
)

when built with --apple_generate_dsym will generate valid dSYMs. I also see that wrapped_clang has a notion of DSYM_HINT_DSYM_PATH that is only set for objc targets in https://github.com/bazelbuild/bazel/blob/fa1740d699574d1f18446c1643470f9d6d8db10c/tools/osx/crosstool/cc_toolchain_config.bzl#L5710. Would extending that to also work for C/C++ executable actions be sufficient?

[haberman]

@nikhilm I am concern about the statically-linked binary case, not dylib.

In my view, this bug is fixed when I can:

1. build a cc_binary() in Bazel using -c opt or --copt=-g
2. run the binary in gdb/lldb/Instruments/etc
3. have full debugging symbols available.

When I do this today, I get no debugging symbols available.

[keith]

dSYMs are the solution for both of those cases. A workaround on macOS could actually even be to build a macos_command_line_application just for the dSYM support.