Carthage: Static *framework* support

[davbeck]

I know that Carthage only support dynamic frameworks at the moment. I realize it's significantly more difficult to support plain static libraries because you need to keep track of both the binary library and the headers, but has there been much thought about support static _frameworks_?

Xcode doesn't provide a template for this out of the box, but if you create a dynamic framework target and then change the Mach-O type to static library, the compiled product is then a .framework, with proper modulemap and headers. I've been playing around with this today and it seems to work pretty well when just dropping the framework into a project. Here is a test framework that I made and used lipo to create a universal iOS binary: http://cl.ly/3s332V3g0S08. You can drop it into any project (both framework and app) and import like any other framework. Even in a Swift project. (AThing.coolThing() if you want to try and use the code in the framework.)

Note that there is one big downside to static frameworks in that they can not contain assets like images, but I would bet a large majority of libraries don't use assets.

I've been thinking about this quit a bit after attending WWDC 2016 Session 406 "Optimizing App Startup Time". In that talk they discuss how dynamic frameworks can slow down the launch of your app because of the work needed to link the libraries. According to the session, static libraries don't have this issue because the work is done at compile time.

[mdiep]

We haven't supported static frameworks because AFAIK static frameworks aren't explicitly supported by Apple. But if they are, I'd certainly be willing to look at a PR that added support for them. I can't make any guarantees that I'll accept it though—it depends on the details and I'd need to play around with them.

[NachoSoto]

Xcode doesn't provide a template for this out of the box, but if you create a dynamic framework target and then change the Mach-O type to static library, the compiled product is then a .framework, with proper modulemap and headers.

But the framework doesn't get linked statically, does it?

[davbeck]

It does get linked statically as far as I can tell. file reports the framework is static and I can link to it in an app but not embed it and it will run on a device. Examining the ipa, the framework isn't included in the Frameworks directory.

Also, it looks like Fabric/Crashltyics is using a static framework, although I'm not sure what their build process looks like.

$ file Crashlytics
Crashlytics: Mach-O universal binary with 5 architectures
Crashlytics (for architecture armv7):   current ar archive random library
Crashlytics (for architecture armv7s):  current ar archive random library
Crashlytics (for architecture i386):    current ar archive random library
Crashlytics (for architecture x86_64):  current ar archive random library
Crashlytics (for architecture arm64):   current ar archive random library

[NachoSoto]

It does get linked statically as far as I can tell. file reports the framework is static and I can link to it in an app but not embed it and it will run on a device. Examining the ipa, the framework isn't included in the Frameworks directory.

Whoa... and this works, with Swift!?

Also, it looks like Fabric/Crashltyics is using a static framework, although I'm not sure what their build process looks like.

Yeah, they're using what you describe, traditionally called "fake frameworks", but that's 100% Objective-C.

[davbeck]

Oh yeah, to be clear Swift frameworks cannot be static, at least on iOS. Oddly Swift Package Manager _only_ supports static libraries. However, a Swift framework or app can link against static ObjC frameworks as long as they have a modulemap.

[NachoSoto]

Oooooh okay, then yeah, this is nothing new :D I got too excited for a second.
I don't think Carthage would _not_ support "fake frameworks", since they get built by Xcode in the same manner, no?

[jarrydfillmore]

+1!

[brianhei]

After watching WWDC 2016 Session 406 "Optimizing App Startup Time", Apple said we can merge dylib to improve the app start up time. Does it also apply to static framework? Let say my app contains 60-70 static frameworks. If I merge them into one, does it help to improve the app startup time?

[davbeck]

Static frameworks and libraries don't have the same performance penalty that dynamic ones do. They get linked and merged in at compile time.

[brianhei]

Thanks @davbeck. So merging static library does not help improve app start up time.

I did some experiment today. My app cold start up time normally takes 3.1-3.2 secs, after merging 20 static frameworks the app start up time reduce to 2.8-2.9 secs. I am not sure whether it is related to reduce the number of framework using in the app...

[brianhei]

@davbeck Have you tired merging several dynamic libraries into one? I tried using libtool but it seems only work for static library

[foobra]

After watching wwdc 406, I'm wondering if too many dynamic frameworks will lag the app launch?

[brianhei]

@foobra if you use too many dynamic frameworks it will increase the app launch time

[foobra]

@brianhei Can I use Carthage or something else to merge all dynamic frameworks into one? Just like Cocoapods do with static libraries "libPods.a"?

[mdiep]

Can I use Carthage or something else to merge all dynamic frameworks into one? Just like Cocoapods do with static libraries "libPods.a"?

Nope. CocoaPods doesn't merge dynamic frameworks (to my knowledge); it directly compiles them into a static library.

I'm not aware of a way to merge dynamic frameworks.

[fabb]

Oh yeah, to be clear Swift frameworks cannot be static, at least on iOS.

I never understood why not. Can someone explain that to me?

Swift Package Manager only supports static libraries.

Does this mean these static libraries are useless for iOS?

[foobra]

@brianhei do you find a way to merge dynamic frameworks?

[steeve]

To be clear, you can make static swift frameworks and link them in a swift app.
You cannot however convert a dynamic one into a static one since you need the objects file for that.

However they are all in the derived data folder so it's doable.

I've successfully converted all my Carthage deps into static ones without issues.

[foobra]

@steeve

I've successfully converted all my Carthage deps into static ones without issues.

Can swift codes compile as static ones?
The only way I found to avoid dynamic swift deps is drag and drop deps' codes into the app target

[steeve]

You can link the .o files from the intermediate folder into static archives with libtool, following this method: https://github.com/aliceatlas/buildstatic

[steeve]

Btw, here are my make rules:

PLATFORM = iOS
CARTHAGE = Carthage
CARTHAGE_DERIVED = $(CARTHAGE).derived
CARTHAGE_BUILD = $(CARTHAGE)/Build/$(PLATFORM)

.PHONY: carthage_update

carthage_update:
    carthage update --platform $(PLATFORM) --derived-data $(CARTHAGE_DERIVED)
    for pkg in `find Carthage.derived/Build/Intermediates -type f -name '*.o' | sed -E 's/.*\/Release-[a-z]+\/([A-Za-z]+).*/\1/' | sort | uniq`; do \
        libtool -static -L. `find Carthage.derived/Build/Intermediates -type f -name '*.o' | grep $$pkg[-_]$(PLATFORM)` -o $(CARTHAGE_BUILD)/$$pkg.framework/$$pkg; \
    done

And I just do instead:

$ make carthage_update

[fabb]

Will this work with whole module optimazation?

[steeve]

Not sure though

[neilco]

Is there a reason why Carthage doesn't support _pre-built_ static frameworks?

I have a GitHub release with a zipped binary-only static framework attached. carthage update downloads it, unzips it, and moves it into Carthage/Build/iOS, but then throws a Invalid architecture UUIDs: Could not parse UUIDs using dwarfdump error. I can drag the framework in Xcode and it statically links with my project just fine.

[mdiep]

Pre-built static frameworks aren't different than non-pre-built ones from Carthage's perspective in terms of work, risk, etc.

[neilco]

Why? There's nothing for Carthage to compile since it's all been done.

[mdiep]

Because we still need to provide automation to let people include these in their projects.

But I'm against pre-built-only options since (1) everything should have a compile option for those who want it and (2) pre-built frameworks present issues for Swift code.

[neilco]

we still need to provide automation to let people include these in their projects.

Carthage doesn't do this for pre-built dynamic frameworks—it simply fetches them. Also I prefer Carthage over Cocoapods _precisely_ because it _does not_ change my project settings through automation.

(1) everything should have a compile option for those who want it

Not all Carthage-compatible projects have an option to compile from source and not all Carthage-compatible projects contains targets for all supported platforms. This is an idealism and the reality is far from it.

(2) pre-built frameworks present issues for Swift code.

I'm curious as to why Carthage supports fetching pre-built dynamic frameworks, if this is so.

[bsarrazin]

@mdiep (cc: @ikesyo)
Has there been an update on this? After troubleshooting for #1657 it seems like supporting static frameworks could be done. This would allow people to use Carthage for pre-built frameworks by comapnies who do not provide CocoaPods/Carthage support, namely Fabric/Crashlytics, and UserTesting.

[mdiep]

Since this isn't really supported by Apple, we don't really want to get into it.

But I would be willing to review a PR that added this functionality. I can't promise that it'd be accepted—I think it'd likely depend on how much complexity was required. A good first step might be to investigate and report the things that would be required to support it.

[aschuch]

I just stumbled upon this project by @keith. Looks promising.

[erichoracek]

After digging into this with @keith's project mentioned above, I've been able to get Carthage to build static .frameworks from projects (Swift or ObjC, without modification) that currently produce dynamic .frameworks.

It works as follows (with some modifications to Carthage):

• For each of Carthage's xcodebuild invocations, provide an additional -xcconfig flag with an xcconfig file with the following content:

LD = /Path/to/ld.py
LIPO = /Path/to/lipo.py
DEBUG_INFORMATION_FORMAT = dwarf

• This is directly lifted from @keith's project, but effectively overrides the LD and LIPO to commands and remaps their inputs to produce a static framework rather than a dynamic framework.
• It additionally ensures that xcodebuild does not attempt to produce a dSYM. Since Xcode is no longer producing a dynamic framework (even though it thinks it is), this step would error otherwise.

Some additional changes were required to prevent Carthage from erroring when it encountered a binary within a framework that had no dSYM UUIDs (see above, and #1930). I'll go ahead and put up a PR for this, since they're relatively small changes.

This is really exciting for us. It means that with small modifications to our current build tools (and without forking any 3rd-party libraries), we could drastically reduce our launch times by combining all of our dynamic frameworks into a single dynamic framework.

We'd love to be able to use static frameworks built by Carthage in our project. To do so, we'd need to make one of the following changes to Carthage:

Add a new flag to Carthage, e.g. --build-static

This would transparently perform the logic discussed above (remap LD, LIPO, and DEBUG_INFORMATION_FORMAT by supplying an xcconfig to all invocations of xcodebuild under the hood).

It would likely require Carthage to expose some hidden commands that would perform the same logic as ld.py and lipo.py in @keith's project, which admittedly would be somewhat fragile. However, this would be the easiest in terms of complexity exposed to consumers for the project.

Allow Carthage to take an --xcconfig flag, which is passed through to xcodebuild

This would require consumers that want Carthage to build static frameworks to provide their own xcconfig file that has the behavior discussed above. While simpler to implement, it would push much of the complexity onto consumers. Since building static frameworks from projects that produce dynamic frameworks is a bit of a hack, this approach seems to make more sense to me, especially as a first step.

Does anyone have any thoughts about which approach they'd prefer (or whether Carthage should allow this at all)? Thanks for reading though this!

[keith]

FWIW this is actually even easier now, you can toss the LIPO override all together, only the LD is needed now.

[mdiep]

I'm wary of adding built-in support. But if you put together a PR that bakes in support, I'll take it to a WWDC lab next month and try to discuss it with someone knowledgable there. I doubt they'd give explicit approval, but the strength of their reaction should be telling. ☺️

Otherwise I think an xcconfig flag makes sense.

[ikesyo]

FYI even now you can pass a xcconfig file with https://github.com/Carthage/Carthage/issues/1622#issuecomment-267919074.

[erichoracek]

@ikesyo That worked—thanks! #1932 should be the only change necessary then.

[iby]

Can those who played with this enough, ping @erichoracek and @keith, comment on any downsides of this approach? Will this in any way affect crash symbolication since we don't get dSYM from the framework or will the final target be able to extract dSYM from the linked library? Any issues at all?

[erichoracek]

@ianbytchek Crash symbolication seems to work fine with this approach, the symbols of the static framework just end up in the dSYM of the binary that they're linked into. The only bit of weirdness is that they'll be grouped under the name of whatever binary they're linked into in the crash report, e.g. ReactiveCocoa would show up within <AppName> in the stacktrace if ReactiveCocoa was statically linked into a binary named <AppName>.

[erichoracek]

On a separate note, with the latest master of Carthage and our 34 embedded dynamic libraries reduced to 1 mega-dylib and 2 normal dynamic libraries (that were non-trivial to build statically), we're seeing app launch times that are approximately 2x faster than before as measured with DYLD_PRINT_STATISTICS. 🎉

If you think it would be useful, I can write up a static lib-specific readme for the next release.

[RomainBoulay]

@erichoracek I think that'd super useful!! Looking forward to reading this :)

(cc @pepibumur)

[iby]

@erichoracek jeez, I feel like what my grand-grand-father would have felt if he saw an iPhone… Can't believe this is here and working! Any information, know-hows and examples would be awesome!

[erichoracek]

Awesome! All thanks should of course go to @keith for figuring out how to trick Xcode by overriding ld with libtool!

[keith]

@ianbytchek on the libtool side the configuration is dead simple, you ca see the entire 30 line implementation here: https://github.com/keith/swift-staticlibs/blob/master/ld.py

[iby]

@keith apparently the solution has been out there from the beginning – initial commit in aliceatlas/buildstatic is two years old. I'm more surprised by the fact that it's not as widely discussed / publicised as I expected. LD variable discovery is pretty awesome though. In any case, with all recent pains Swift was giving me this discovery is beyond awesome! ❤️💯

[erichoracek]

I've taken a shot at documenting a static framework workflow in https://github.com/Carthage/Carthage/pull/1939. Please read over it, try it out with the latest master of Carthage, and let me know if it helps your launch times or if you have any other feedback. Thanks!

[drkibitz]

@keith @erichoracek I would like to ask if someone can explain the full approach? Would be much appreciated.

Meaning, what does this look like in a large application project setup?

• Lets keep it simple and say we have like 20 dynamic frameworks, and all are objective-c.
• Right now the build process is a mix of embedding precompiled frameworks, and building then embedding all the rest.
• What I would like to have is an intermediary target and preferably and totally separate project for this target.
• The separate project does everything the app project does now, except it's product is a single dynamic framework
• The app project now simply embeds the merged framework product of the intermediary project/target

Does this all sound doable? Does it sound like it would have reasonable build times in a large project?

[erichoracek]

@drkibitz We are shipping the approach you're discussing to the app store without any major issues (except with ~15 more static frameworks merged together—around 35 total!).

To elaborate on our approach and hopefully answer some of your questions:

• We have an <AppName>Core.framework dynamic framework target in our app that has many static frameworks (built by Carthage using the approach in #1939) linked into it
• We have three app extensions in addition to our app that all depend with our merged "core" framework, ensuring that there's nothing duplicated between executables
• We chose to have the framework target in the app's and extension's project for simplicity since it is not built with Carthage and it does not make sense (for us) to extract out
  
  
  
  • I don't see any reason why you wouldn't be able to have it in a separate project though
  
  
• We use rome as a build cache (works with static frameworks too!) to keep build times down.

Let me know if you have any other questions!

[mitisBlack]

i haven't tested, yet, but XC9 says it now knows how to do static swift libs: https://developer.apple.com/library/content/releasenotes/DeveloperTools/RN-Xcode/Chapters/Introduction.html#//apple_ref/doc/uid/TP40001051-CH1-SW911

[iby]

@mitisBlack confirming, I've been successfully using native static Swift libs offered by Xcode for a month now without any external tools.

[CocoaPriest]

This definitely should be a built-in option for Carthage

[hamdshah]

@ianbytchek Do you use Carthage for those swift static libraries generated with Xcode-9 ?

[muukii]

@erichoracek

Hi, Thank you for your information.
I have a question for building Core.framework and linking.
How do we use the symbol that is included in Core.framework?
Can you tell me your project settings?

If Core.framework is made below.

• Core.framework
  
  
  
  • A.framework
  
  
  • B.framework
  
  
  • C.framework
  
  

In executable application code.

Can we write following code?
I've tried this, but I've found error Use of unresolved identifier ~

import Core

A.MyClass.init()

Next, I've written import A.
But, I've found error ld: framework not found A

import Core
import A

A.MyClass.init()

[akhilcb]

@muukii Did you find any solution? I am having the same issue( framework not found FrameworkA). How do we statically link a Static framework with a dynamic umbrella framework? Is there any sample projects available? Could you please help @erichoracek ?

[muukii]

@akhilcb No not yet.
I’m also finding that.