Firebase-ios-sdk: Support for Swift Package Manager binaryTarget by providing xcframework in a release

[ollieatkinson]

Feature proposal

• Firebase Component: All

Swift Package Manager support is great and has enabled a much easier integration with many projects which use Firebase, however this comes at a cost of having each developer to compile all of the sources. Swift Package Manager added support for binaryTarget dependencies and from looking at https://github.com/firebase/firebase-ios-sdk/tree/master/ZipBuilder you've already made a start on this but is still lacking things like x86_64 support.

I am raising this as it will likely save hundreds if not thousands of hours of developers time in due course.

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[paulb777]

@ollieatkinson Thanks for the request. We've thought about a complete binary distribution and will continue to evaluate. However, we have no plans beyond Analytics in the near-term for the following reasons:

• We're focused on completing a successful beta of the primarily source SPM distribution
• The SPM binary implementation is still immature with major bugs and usability issues. Browse the issues here for examples
• Providing platform and version coverage with a binary distribution is much more challenging than a source distribution
• We're considering the alternative of improving this ZipBuilder tool so that it's easy for developers to create the binaries that they need

[ollieatkinson]

We're considering the alternative of improving this ZipBuilder tool so that it's easy for developers to create the binaries that they need

This would be hugely enabling! Please let me know if you are willing to accept community support and provide some instruction to do so.

I think another useful outcome of this would be that ZipBuilder will support building x86_64 macOS (for the set of frameworks which you already support)

[paulb777]

We're definitely willing to accept community support. In fact, a large part of the Catalyst support came from @steipete in the community.

Instructions for getting started are at https://github.com/firebase/firebase-ios-sdk/tree/master/ZipBuilder.

We'll be happy to answer any questions and provide additional feedback and guidance on issues and PRs here.

[ollieatkinson]

Thanks!

From my initial investigation it seems we need to rephrase FrameworkBuilder and how it is managing it's architectures since x86_64 can mean macOS platform as well as iOS simulator platform.

When I've generated xcframeworks in the past I have specified the destination parameter to xcodebuild , do you think this is something that ZipBuilder should use? Currently you specify -arch instead.

xcodebuild archive \
..
-destination destination="generic/platform=iOS"
..

struct Destination {

    struct Platform: RawRepresentable {
        let rawValue: String
        static let macOS = Platform(rawValue: "platform=macOS")
        static let iOS = Platform(rawValue: "generic/platform=iOS")
        static let iOS_Simulator = Platform(rawValue: "generic/platform=iOS Simulator")
        static let watchOS = Platform(rawValue: "generic/platform=watchOS")
        static let watchOS_Simulator = Platform(rawValue: "generic/platform=watchOS Simulator")
        static let tvOS = Platform(rawValue: "generic/platform=tvOS")
        static let tvOS_Simulator = Platform(rawValue: "generic/platform=tvOS Simulator")
    }

    struct Variant: RawRepresentable {
        let rawValue: String
        static let catalyst = Variant(rawValue: "variant=Mac Catalyst")
    }

    let isMacCayalyst: Bool
    let string: String

    init(platform: Platform) {
        isMacCayalyst = false
        string = platform.rawValue
    }

    init(platform: Platform, variant: Variant) {
        isMacCayalyst = variant == .catalyst
        string = [platform.rawValue, variant.rawValue].joined(separator: ",")
    }

}

[paulb777]

It looks like we're currently using -sdk combined with ARCHS. -destination could be a cleaner implementation.

[mikehardy]

I'm looking at this while investigating a response to a request in our "pre-compiled Firestore framework" repo: https://github.com/invertase/firestore-ios-sdk-frameworks/issues/19

It looks like we could maybe provide an SPM version of the compiled Firestore framework as well, based on this issue, by using ZipBuilder? Perhaps with SSL collision errors though (#6869)

It appears ZipBuilder has moved to here (a couple directories deeper): https://github.com/firebase/firebase-ios-sdk/tree/master/ReleaseTooling/Sources/ZipBuilder if I am looking at the correct thing?

Related (and more about how ZipBuilder works) I am working on re-packaging the binary release of an iOS SDK from previous .framework with one fat library (fails on Apple Silicon simulators with arm64 slice collision as you all clearly know) to a modern .xcframework and adding in all the flavors (catalyst, tvos etc) while I'm there. It's all working now after quite a few different tests to get there.

During the work I consulted your ZipBuilder and to @ollieatkinson's point I think destination is more precise than combination of sdk and ARCH. Most specifically, I tried the sdk macosx used in your binary catalyst builds from ZipBuilder in my build for catalyst and it actually brings in other symbols (e.g. AppKit vs UIKit) vs the destination that Apple appears to bless (and which worked for me compile+runtime demo, since we use UIKit symbols): -destination='platform=macOS,arch=x86_64,variant=Mac Catalyst'

So I'd consider moving to destination unless I'm missing something. Allows you to remove ARCHes entirely in my experience (I end up with the same ARCH skew in my .xcframework that ZipBuilder specifies explicitly, but I get it with nothing specified - xcodebuild automatically generated the fat multi-arch-per-flavor frameworks)

Not sure what I'll do with regard to SPM support for the pre-compiled framework given status here - probably have to let that sit pending a community contribution as it appears pretty bleeding edge w.r.t. what's possible

Cheers

[paulb777]

@mikehardy It's definitely possible to do build xcframeworks with the ZipBuilder and that's how we build Analytics for Swift Package Manager. However, as I noted above, SPM is still hard to use with binaries. See especially #6472. The combination of Firestore and its dependencies would exacerbate the difficulty of working around the usability issues.

Despite these difficulties, it may all be solvable with additional tooling and we'll be happy to help you continue down the path.

Also we're soon planning to merge a refactor of the ZipBuilder that simplifies the options and adds macOS/tvOS support. See #7221

[mikehardy]

Cool re: support for other platforms, it appears that despite it being fairly bleeding on all edges react-native (via firebase at SDK level, then react-native / react-native-firebase) is going to have a much larger platform reach with modern app dev facilities mainstream shortly. I suppose FlutterFire will be quite excited as well

For the SPM firestore binary distribution idea I said effectively the same thing as I carried it back to the issue in our repo: I highlighted there were some hurdles to clear but that the team here is quite collaborative and would likely welcome any contributed effort to get it done. It's not a priority for me/us as maintainers but similarly we'd happily merge anything needed in the pre-compiled framework repo

[mikehardy]

Just a note - I think I need to retract a statement here:

During the work I consulted your ZipBuilder and to @ollieatkinson's point I think destination is more precise than combination of sdk and ARCH. Most specifically, I tried the sdk macosx used in your binary catalyst builds from ZipBuilder in my build for catalyst and it actually brings in other symbols (e.g. AppKit vs UIKit) vs the destination that Apple appears to bless (and which worked for me compile+runtime demo, since we use UIKit symbols): -destination='platform=macOS,arch=x86_64,variant=Mac Catalyst'

I have continued testing different combinations of destination and sdk arguments for xcodebuild command line and @ollieatkinson I am not able to make 'destination' work correctly on Xcode12.3 with destination alone for most targets. For catalyst build it appears the special destination argument we've been using works without sdk specified but for other targets (at minimum ios, ios simulator) the sdk by itself is sufficient, sdk + destination works, but destination alone does not.

So in my testing there is no way to rely on destination alone. I'm still learning xcodebuild so if I'm missing something I'd love pointers to examples that work, but I didn't want to let that comment stand in case someone else based work on it. Cheers

[ollieatkinson]

Do you have some examples for what's not working?
Currently, I am able to build a vanilla project using the destination specifier:

xcodebuild archive -project example.xcodeproj -scheme example -destination "generic/platform=iOS" -archivePath archive/iOS BUILD_LIBRARY_FOR_DISTRIBUTION=YES SKIP_INSTALL=NO
xcodebuild archive -project example.xcodeproj -scheme example -destination "generic/platform=iOS Simulator" -archivePath archive/iOS-Simulator BUILD_LIBRARY_FOR_DISTRIBUTION=YES SKIP_INSTALL=NO
xcodebuild archive -project example.xcodeproj -scheme example -destination "generic/platform=tvOS" -archivePath archive/tvOS BUILD_LIBRARY_FOR_DISTRIBUTION=YES SKIP_INSTALL=NO
xcodebuild archive -project example.xcodeproj -scheme example -destination "generic/platform=tvOS Simulator" -archivePath archive/tvOS-Simulator BUILD_LIBRARY_FOR_DISTRIBUTION=YES SKIP_INSTALL=NO
xcodebuild archive -project example.xcodeproj -scheme example -destination "generic/platform=watchOS" -archivePath archive/watchOS BUILD_LIBRARY_FOR_DISTRIBUTION=YES SKIP_INSTALL=NO
xcodebuild archive -project example.xcodeproj -scheme example -destination "generic/platform=watchOS Simulator" -archivePath archive/watchOS-Simulator BUILD_LIBRARY_FOR_DISTRIBUTION=YES SKIP_INSTALL=NO
xcodebuild archive -project example.xcodeproj -scheme example -destination "generic/platform=macOS,name=Any Mac" -archivePath archive/macOS BUILD_LIBRARY_FOR_DISTRIBUTION=YES SKIP_INSTALL=NO
xcodebuild archive -project example.xcodeproj -scheme example -destination "generic/platform=macOS,variant=Mac Catalyst" -archivePath archive/macOS-Catalyst BUILD_LIBRARY_FOR_DISTRIBUTION=YES SKIP_INSTALL=NO

xcodebuild -create-xcframework \
    -framework archive/iOS.xcarchive/Products/Library/Frameworks/example.framework \
    -framework archive/iOS-Simulator.xcarchive/Products/Library/Frameworks/example.framework \
    -framework archive/tvOS.xcarchive/Products/Library/Frameworks/example.framework \
    -framework archive/tvOS-Simulator.xcarchive/Products/Library/Frameworks/example.framework \
    -framework archive/watchOS.xcarchive/Products/Library/Frameworks/example.framework \
    -framework archive/watchOS-Simulator.xcarchive/Products/Library/Frameworks/example.framework \
    -framework archive/macOS.xcarchive/Products/Library/Frameworks/example.framework \
    -framework archive/macOS-Catalyst.xcarchive/Products/Library/Frameworks/example.framework \
    -output example.xcframework

xcframework successfully written out to: example.xcframework

ll example.xcframework
drwxr-xr-x oliver wheel 352 B  Fri Jan  8 10:15:17 2021 .
drwxr-xr-x oliver wheel 224 B  Fri Jan  8 10:15:17 2021 ..
.rw-r--r-- oliver wheel 3.1 KB Fri Jan  8 10:15:17 2021 Info.plist
drwxr-xr-x oliver wheel  96 B  Fri Jan  8 10:15:17 2021 ios-arm64
drwxr-xr-x oliver wheel  96 B  Fri Jan  8 10:15:17 2021 ios-arm64_x86_64-maccatalyst
drwxr-xr-x oliver wheel  96 B  Fri Jan  8 10:15:17 2021 ios-arm64_x86_64-simulator
drwxr-xr-x oliver wheel  96 B  Fri Jan  8 10:15:17 2021 macos-arm64_x86_64
drwxr-xr-x oliver wheel  96 B  Fri Jan  8 10:15:17 2021 tvos-arm64
drwxr-xr-x oliver wheel  96 B  Fri Jan  8 10:15:17 2021 tvos-arm64_x86_64-simulator
drwxr-xr-x oliver wheel  96 B  Fri Jan  8 10:15:17 2021 watchos-arm64_32_armv7k
drwxr-xr-x oliver wheel  96 B  Fri Jan  8 10:15:17 2021 watchos-arm64_x86_64-simulator

[mikehardy]

Not sure why, but when I did that without specifying sdk for the iphone builds, it was attempting to compile it catalyst style (I was receiving API warnings about things not available for catalyst or deprecated in catalyst whereas I would not receive those during an -sdk iphoneos build) using the exact destination you propose. At the same time / conversely, if I specify sdk during the catalyst destination builds catalyst compiles incorrectly (it generates binaries lipo will tell you are for iphone and -create-xcframework complains of duplicatation). Perhaps your SDK API usage does not have any availability differences between macOS Catalyst API set and iphoneos API set?

I had a separate issue bringing the macos target in where it built but left the thing in DerivedData/.../UninstalledProducts and did not actually drop it in the specified archivePath, which was quite odd given #? was 0, and especially as -create-xcframework after would work if I copied contents into the expected structure in the specified archivePath.

In the end I have a real library with real functionality working with a directory result (and verified correct xcframework/Info.plist) as you demonstrate but it wasn't nearly so clean producing it :shrug:

This is still an active area of research for me, I'll see if I can isolate something, I appreciate the extra info