Carthage: Verify that the Carthage build folder is in sync with the Cartfile.resolved

[erichoracek]

In a project that is frequently undergoing updates, a common pain point of development is accidentally building with a Carthage/Build folder that does not match the contents of your Cartfile.resolved. For example, if a bug was fixed in a dependency and a carthage update was performed by developer A, developer B must first make sure to run carthage bootstrap before they build and run the project or else they will encounter the same bug in their local environment.

As of the addition of --cache-builds command, .Project.version files are now written to the Carthage/Build directory. As such, I propose that these .Project.version files could be used in a new command to validate that the contents of the Carthage/Build folder is in sync with the Cartfile.resolved. This command could be added to a run script build phase with a bash -e shell to fail the build with a non-zero exit status and explanation of the dependencies that are out of date.

As far as I can tell, the majority of the work in implementing this command would be:

• Archiving .Project.version files into the archive when used with the carthage archive command
• Unpacking .Project.version files into the Carthage/Build from prebuilt binaries directory on checkout/bootstrap
• Iterating through all .Project.version files when this new command is invoked to verify that the commitish field matches the same field in the Cartfile.resolved

For this command, I propose the following syntax:

carthage verify

Invoking carthage verify with a build folder that is in sync with the Cartfile.resolved would produce the following output with a 0 exit status:

All builds are in sync with their resolved version.

Invoking carthage verify with a build folder that is _not_ in sync with the Cartfile.resolved would produce the following output with a non-zero exit status:

The following builds are out of sync with their resolved version:
*** ReactiveCocoa at "4.0.0", expected "5.0.0"
You must bootstrap to ensure they are in sync.

Finally, since many prebuilt binaries will not be compatible at first, I propose writing an abbreviated .Project.version file on checkout of a prebuilt binary (only if not included in the archive itself). This file would only include the commitish field from the release that the prebuilt binary was attached to, e.g.:

{
  "commitish" : "5.0.0",
}

Please let me know if I've missed an exiting proposal for this. If not, let me know if this seems like something that would be a welcome addition to Carthage, and if so, whether it would be something that I could take a shot at implementing. Thanks!

[mdiep]

Is there a reason to do this instead of running carthage bootstrap --use-cache as part of your Xcode build?

[erichoracek]

@mdiep I'd say the primary reason is that carthage bootstrap takes about 30s on our machines with a warmed build cache. This would increase iterative build time for our project by ~3x:

time xcodebuild -scheme <app> -destination <sim> ONLY_ACTIVE_ARCH=YES > /dev/null 2>&1
... 9.131 total

time carthage bootstrap --cache-builds --platform ios > /dev/null 2>&1
... 26.529 total

For users that do not yet have the build cache warmed, this command could take nearly ~15 minutes.

Additionally, as a matter of opinion, I personally prefer having the "set up dependencies" step separate from the "build and run" step.

[mdiep]

I'd rather see if we can speed up carthage bootstrap. It shouldn't need to take 30s.

For users that do not yet have the build cache warmed, this command could take nearly ~15 minutes

But those users won't be able to build anyway? They'd need to go to the command line, build with Carthage, and then come back and build in Xcode. If anything, that's slower.

[erichoracek]

If carthage bootstrap was nearly instant with a warmed build cache it could work for us. Even if it's 10 seconds and a developer builds 75 times a day we'd still be wasting over an hour per developer per week waiting on carthage bootstrap.

Regardless, I do think that for those of us that would like to script this functionality on top of Carthage, it would be nice to be able to have a .Project.version available for all frameworks, either cached or downloaded.

[mdiep]

Regardless, I do think that for those of us that would like to script this functionality on top of Carthage, it would be nice to be able to have a .Project.version available for all frameworks, either cached or downloaded.

That makes sense to me. 👍

That would also let you build a carthage verify script yourself (since carthage will execute a carthage-{command} executable if one exists).

[erichoracek]

Cool! Will take a shot at always writing .Project.version files when I get a chance.

[tmspzz]

From the discussion I understand that this won't be implemented in carthage and it will be left to others. Correct?

[erichoracek]

My interpretation is that Carthage should make a best effort to have .Project.version files available for all artifacts in the Carthage/Build folder, but it would be up to another tool or individual developers to verify that these match up with the Cartfile.resolved as part of a single command.

[mdiep]

At this point in time, I don't think this makes sense to include in Carthage. But if someone built a version using CarthageKit and we're unable to make carthage bootstrap --use-cache run at a similar speed, then we might consider including it in the future.

[erichoracek]

As a heads up, I've added support for this command in a PR to https://github.com/Carthage/workflows/ in https://github.com/Carthage/workflows/pull/1

[mdiep]

I wonder if compiling Carthage with SwiftPM (#1559) would help with this by eliminating the dylibs that we use. 🤔

[erichoracek]

Is that in reference to the amount of time it takes to run bootstrap with a warmed build cache? Do dylbs incur a performance cost beyond increases in launch times due to loading and fix-ups? That cost seems relatively negligible as far as I can tell.

To break out the costs of bootstrap for us with a warmed build cache, it seems that 4/5ths of the total time spent on checkout, while 1/5th is spent on building:

$ time carthage checkout
...
carthage checkout  10.99s user 5.68s system 72% cpu 22.877 total

$ time carthage build --cache-builds --platform ios
...
carthage build --cache-builds --platform ios  6.28s user 1.87s system 131% cpu 6.206 total

With respect to the dyld time, with DYLD_PRINT_STATISTICS set to 1, it appears to add little overhead:

Total pre-main time:  53.37 milliseconds (100.0%)
         dylib loading time:  28.47 milliseconds (53.3%)
         ...

28ms is less than 1% of either of the above durations.

Are checkouts performed concurrently? If not, that seems like it may be a first place to target.