Carthage: Correct framework DWARFs and symbols to workaround broken debugging 🔥

[NachoSoto]

_This is a follow up to #832 with details on a possible solution._

The story:

We've all been aware since the very beginning that debugging projects that include pre-compiled (Carthage) frameworks has been very broken. As Apple _promised_ that this got better in Xcode 7.2, a few people on #832 did the test (I still can't test it myself because reasons), and realized that it's still just as broken.

Upon further investigation, looks like DWARF files use absolute paths everywhere, based on when they were compiled.

Using dwarfdump:

As you can see, they're full of paths to the derived data directory of the user who built the framework.
Needless to say, it's not just a problem because we can't share these frameworks with other devs, but also for personal use, given how extremely frequent one must clear derived data to work around Xcode.

This is why I finally decided to file a radar, because after two years this is still utterly broken: http://www.openradar.me/23551273.

Of course, I don't expect this to be fixed (ever?), so we need a solution.

Proposed solutions

Abandon Carthage:

We can go back to working like animals, adding all dependencies as submodules. However, this just doesn't cut it, Apple, when will you see this?

• Swift compilation times are already insanely long. _Khan Academy_ was taking 15 minutes for a clean build before I changed Swift dependencies to pre-compiled frameworks.
• Nested dependencies: this solution would not be feasible if you have multiple dependencies that depend on a common one. This is one of the reasons why CocoaPods was born:

CocoaPods

_(Yeah, I went there, but I wanted to analyze this too)_

• Obviously deals with the dependency graph problem.
• It's _invasive_ and very complex, which is why Carthage was born.
• It's still not what we want for development. It's the year 2015, we want to be able to distribute built dependencies and compile our applications in the order of seconds, not minutes.

Work around it in Carthage

I have an idea that might work. Distributing pre-compiled frameworks would still be broken (so --no-use-binaries would have to be the only option), but we'd be able to compile once and forget:

xcodebuild -sdk $SDK -project $PROJECT -configuration Release -scheme $SCHEME ONLY_ACTIVE_ARCH=NO SYMROOT=$DIR/Carthage/Build/$PLATFORM/$FRAMEWORK clean build

Notice the SYMROOT parameter. In my tests compiling with this option correctly sets _the absolute paths_ in the DWARF to that path, instead of derived data! Which means that debugging should hopefully work.

With this, the recommended approach would be to .gitignore Carthage/Build, and using carthage update to build the frameworks.

Once we do that, one can freely clear derived data, and that won't affect the compiled frameworks.

It's still not clear that this is enough, though. It's possible that we'll also need OBJROOT, and putting everything in Carthage/Build:

---

_Prolonged sigh_.

[JaviSoto]

xcodebuild -sdk $SDK -project $PROJECT -configuration Release -scheme $SCHEME ONLY_ACTIVE_ARCH=NO SYMROOT=$DIR/Carthage/Build/$PLATFORM/$FRAMEWORK clean build

This seems like a very sensible approach :+1:

[alanjrogers]

Is it not possible to get DWARF files to use relative paths or re-write them in the DWARF file after the fact?

[alanjrogers]

DBGSourcePath from http://lldb.llvm.org/symbols.html looks promising.

[NachoSoto]

Interesting! That could be another option.

Note though that it's possible that in order for debugging to work, we might also need the rest of files in that last screenshot.

[mdiep]

Ugh. Thanks for debugging this.

IIUC setting the SYMROOT root would be similar to using a custom derived data folder (#459). It's not really clear to me if/how those two things would be different. But really the intent is to not tie the derived data of the Carthage-built frameworks to the derived data of the consumer, right?

Is it not possible to get DWARF files to use relative paths or re-write them in the DWARF file after the fact?

This is a pretty interesting idea. Unfortunately I'm not finding any promising information about it.

[NachoSoto]

But really the intent is to not tie the derived data of the Carthage-built frameworks to the derived data of the consumer, right

Right, for 2 reasons:

• So that projects can keep other relevant files that are necessary for debugging.
• So that the DWARF and other files point to absolute paths that aren't transient.

[chbeer]

Any news on this one? Is there any solution/feasible workaround for this? We had to crawl back to Cocoapods in our project because of this 😩

[mdiep]

I believe you can work around this by using --no-use-binaries.

[chbeer]

You can. But that only works as long as you don't delete the DerivedData
directory.

On Wed, Dec 2, 2015 at 4:02 PM, Matt Diephouse [email protected]
wrote:

I believe you can work around this by using --no-use-binaries.

—
Reply to this email directly or view it on GitHub
https://github.com/Carthage/Carthage/issues/924#issuecomment-161325121.

[ratkins]

I'm pretty sure this is the same issue as https://github.com/Carthage/Carthage/issues/785?

[NachoSoto]

Yes

[kcharwood]

Just to clarify...

This only affect Swift frameworks? Or does this also affect Objective-C frameworks?

[ratkins]

Apparently only Swift frameworks. From the Xcode 7.2 release notes: "Frameworks written in Swift should be compiled from source as part of the same project that depends on them to guarantee a single, consistent compilation environment. (22492040)"

This really needs to be noted in the Carthage readme. Why is downloading prebuilt Swift binaries even an option if it demonstrably doesn't work?

[JaviSoto]

This really needs to be noted in the Carthage readme. Why is downloading prebuilt Swift binaries even an option if it demonstrably doesn't work?

@ratkins when Carthage was conceived, and when that feature was implemented, Apple hadn't documented anywhere that this was broken. Building all of your dependencies from source every single time is a terrible idea, especially considering the long Swift build times. Being able to download pre-built frameworks from Carthage is still desirable, and I hope it will work one day.
That been said, even if you don't download pre-built frameworks, you're going to run into this issue as soon as you clear the derived data directory...

[ratkins]

I get that ultimately it's Apple's bug, but I haven't had a working debugger and haven't been able to commission a new build server for six months and I had no idea why. The first sentence of this issue ("We've all been aware since the very beginning that debugging projects that include pre-compiled (Carthage) frameworks has been very broken") makes me punchy because it's demonstrably not true—I had no idea this was a known issue and it's not mentioned in the readme, which however _does_ describe a default-on feature that doesn't work for Swift projects.

Wishing that a desirable feature works doesn't make it work. Not mentioning it doesn't work when it's known to be the case that it doesn't work wastes a lot of people's time and effort.

[NachoSoto]

@ratkins I don't know if you realize that every single one of us working on Carthage and other open source projects do it for free. The tone of blame with which you're coming across is not well received.

When I said "we've all been aware..." I wasn't implying that we knew from the beginning that it was the fact that they were pre-compiled what was causing the issues. Swift 1.0 was no where near 1.0, so it was hard to know what was going on. The time I put to elaborate my findings in this issue was precisely to alert everyone. You seem to imply that just because nobody (and when I say nobody, I don't mean just contributors) thought to include this in the README is for some other reason than the fact our time is very limited.

The people who have created and worked on Carthage have done that to serve the community. They don't owe anything to anyone. Even less to users who can't appreciate how much work has been put here and in other projects like CocoaPods, precisely to make our jobs easier.

I'm very sorry you didn't find Carthage to your liking. You can help make it better by clicking on the "fork" button. Otherwise, sending messages like that is not helping anyone.

[ratkins]

You're right, I apologise for the blame-y tone—I do realise nobody owes anyone anything, and I did interpret it as the problem having been known about well before the date on this issue. Penance: https://github.com/Carthage/Carthage/pull/989 (rdar://23551273 duped also.)

[ollieatkinson]

What I have done to work around this is:

1. Do not commit Carthage/Build
2. Build using carthage build --platform iOS --no-use-binaries

... once built I
Copy Cartfile.resolved to Carthage/Build/Cartfile.resolved

... When checking out on another computer I
Check to see if there is a Carthage/Build directory as well as check the difference between Cartfile.resolved and Carthage/Build/Cartfile.resolved. If either of these fail I re-run carthage build --platform iOS --no-use-binaries and then copy the resolved file.

I tied this up into a rake task and it seems to work when deploying to different machines.

If you wanted (like me) I execute the script in the post-checkout git hook to automatically build. For reference here is the rake task:

require 'digest'

namespace :carthage do 

  desc "Build frameworks using carthage"
  task build: do

    directory_exists   = File.directory? './Carthage/Build'
    same_resolved_file = md5('./Carthage/Build/Cartfile.resolved').eql? md5('./Cartfile.resolved')

    unless directory_exists && same_resolved_file

      sh "carthage build --platform iOS --no-use-binaries"
      cp './Cartfile.resolved', './Carthage/Build/Cartfile.resolved'

    end

  end

end

def md5 filename 
  Digest::MD5.file(filename).hexdigest if File.exist? filename
end

To cache the files on travis you can add this to your .travis.yml so that build times are not increased on CI. Using the above script will force it to build the new frameworks only if there is a change.

.travis.yml

cache:
  directories:
    - Carthage/Build

[NachoSoto]

I won't believe this until I see it:
https://twitter.com/NachoSoto/status/686619898944958464
https://twitter.com/NachoSoto/status/686620062019538944

[NachoSoto]

As I imagined, this isn't actually fixed: https://twitter.com/jckarter/status/686620845926526976

[kcharwood]

I was so excited reading your tweets... 😭

[mdiep]

Not crashing is still a huge step forward. :smile:

[NachoSoto]

Not crashing is still a huge step forward. :smile:

Very much welcome after 2 years :D

[schickling]

:angel:

[drewcrawford]

Rumor has it this is fixed in b2. I sent a sample project to a coworker to confirm, but it works for me when deleting my derived data folder.

[NachoSoto]

What do you mean by fixed? There's no mention about this in the release notes. Do you mean that debugging actually works, or that it doesn't crash?
The crash _was_ fixed in beta 1.

[drewcrawford]

I'm able to use the debugger in b2, and I _think_ I deleted all the build products.

I'm still waiting for confirmation from a coworker, as there may be a build product on my computer somewhere that I don't know how to delete.

[NachoSoto]

Can you reference symbols from pre-built modules, as well as calling methods?

[drewcrawford]

I'm able to call the constructor on a class defined inside a pre-built Swift module at the lldb prompt. I also get autocomplete now in lldb, so that's fun.

Again, it's possible there's still some build product squirreled away on my system, I've made a test case for a coworker and waiting to hear back before I declare the problem solved.

[NachoSoto]

That's exciting! Looking forward to hearing about your coworker.

[kcharwood]

Waiting with anticipation...

[mathiasnagler]

Hopefully...

[drewcrawford]

Coworker can debug my sample project too.

He doesn't have the sourcecode for the library, so he has never built it on his system.

Sent from my iPhone

On Jan 28, 2016, at 2:40 PM, Mathias Nagler [email protected] wrote:

Hopefully...

—
Reply to this email directly or view it on GitHub.

[mdarnall]

Xcode 7.2.1 Released today.

Resolved a debugger crash that could occur in code depending on a binary Swift library or framework

ಠ_ಠ

[NachoSoto]

They probably backported the crash fix to 7.2.x.

[NachoSoto]

I can confirm: LLDB works MUCH better now thanks to this new "not crashing" feature they've added. Black magic!

I think we should leave this open until we can confirm that all symbols and everything is available for debugging, but I think this is a small victory already :) I hadn't been able to print things in the debugger or step through frames in the stacktrace in over a year...

[kcharwood]

@NachoSoto Does this mean I can have another team member build the swift binary, and I have debug access? Have you validated the paths look good in the DWARFs?

[NachoSoto]

I haven't looked at the DWARFs now, but I doubt anything has changed because .frameworks don't contain anything new. I meant that we can now debug symbols that we have information about without it crashing completely due to other modules being missing. When stepping through a RAC stack trace for example all I could see was assembly, which is 99% times than immediately crashing...

[carlossless]

Resolved a debugger crash that could occur in code depending on a binary Swift library or framework

:confetti_ball: 🙃 :confetti_ball:

When stepping through a RAC stack trace for example all I could see was assembly, which is 99% times than immediately crashing...

@NachoSoto I don't quite get this, isn't it the normal behaviour given your using framework binaries?

[NachoSoto]

@NachoSoto I don't quite get this, isn't it the normal behaviour given your using framework binaries?

But frameworks can contain debug symbols (DSYMs, etc) so it doesn't have to be that way. In practice, however, this debug information is only stored in derived data so it's completely volatile.

[ColdLogical]

So does release of 7.2.1 mean we can distribute the .framework binaries to clients without fear of instantly crashing the application when they try to debug?

[MaxDesiatov]

was anyone able to reproduce this with Xcode 7.3?

[icecrystal23]

Apple says 22492040 was fixed in Xcode 7.3: https://developer.apple.com/library/ios/releasenotes/DeveloperTools/RN-Xcode/Chapters/xc7_release_notes.html#//apple_ref/doc/uid/TP40001051-CH5-DontLinkElementID_41

[ikesyo]

This should have been fixed in Xcode 7.3, let's close this.

[AndrewSB]

If I understand this correctly, it looks like crashing on pre-built binaries is no longer an issue. If so, can it be removed from the README https://github.com/Carthage/Carthage#known-issues?

[NachoSoto]

:+1:

[werner77]

Mind that I fixed this issue in our fork of Carthage: https://github.com/nsoperations/Carthage

The approach is based on this feature of llvm (plists for mapping build source location to actual source location): https://lldb.llvm.org/use/symbols.html

About to be released in version 0.35.0+nsoperations.

[thedavidharris]

Mind that I fixed this issue in our fork of Carthage: https://github.com/nsoperations/Carthage

The approach is based on this feature of llvm (plists for mapping build source location to actual source location): https://lldb.llvm.org/use/symbols.html

About to be released in version 0.35.0+nsoperations.

Would this fix the current issue where you get Couldn't IRGen expression?

[ollieatkinson]

@thedavidharris are you able to use the swift tips on debugging LLDB failures:
https://github.com/apple/swift/blob/master/docs/DebuggingTheCompiler.rst#id23