Carthage: xcodebuild timed out while trying to read Braintree.xcworkspace on Travis

[dcaunt]

Carthage's xcodebuildTask consistently errors due to XcodebuildTimeout when trying to build Braintree frameworks in our private project. Both xcodebuild -list and xcodebuild -showBuildSettings complete within about a second on my Macbook Pro, so I'm surprised that even the 15s timeout is being hit on Travis.

I've created a repro project here which attempts to carthage bootstrap on Travis CI and build the Braintree frameworks. You can see an example failed build here.

[mdiep]

GitHub API request failed: Maximum number of login attempts exceeded. Please try again later

You might want to figure that out—then you could use binary frameworks downloaded from GitHub releases. That would avoid the timeout.

---

Could you try running time xcodebuild -showBuildSettings with the appropriate arguments on Travis to see how long it takes to complete? (You can get those from the xcodebuild output or use the --verbose flag.)

We can up the timeout if that's the issue, but there's a bug where xcodebuild hangs forever (which is why we added the timeout), so it'd be nice to know (a) if it completes and (b) what a more reasonable timeout value would be.

[dcaunt]

Sorry, the GitHub API request failure is a bit of a red herring, Braintree don't provide binary frameworks – and our private repo has the token set but this example project doesn't have a token for Travis.

I've added some time commands to Travis and I'll report back when I have some output.

Thanks for looking at this.

[dcaunt]

Here is a Travis build output running a debug build of Carthage, which simply prints the xcodebuild -showBuildSettings task and arguments.

It appears to halt at /usr/bin/xcrun xcodebuild -workspace Carthage/Checkouts/braintree_ios/Braintree.xcworkspace -scheme UnitTests-StaticLibrary -configuration Release CODE_SIGNING_REQUIRED=NO CODE_SIGN_IDENTITY= CARTHAGE=YES -showBuildSettings every time.

In isolation, as shown here, the command completes as expected.

Any ideas? Let me know if there's anything else I can run to get more useful information.

[mdiep]

I'm really unsure what to do here, unfortunately. :confused: I don't know why xcodebuild just hangs, and I don't think there's really a way around it. We're at xcodebuilds mercy.

You could try upping this retry.

[dcaunt]

So I tried upping the retry to 50, and still no luck. I also tried unsharing the UnitTests-StaticLibrary scheme, so as to skip the blocking scheme, and found that the one before then becomes the hanging point. Unsharing all of the non-framework schemes (Demo app, Tests, Static Libraries) allows Carthage to progress to building the frameworks successfully, and there is no xcodebuild timeout.

It's frustrating, but I think the easy solution for now is to use a fork with prebuilt binaries.

[nlutsenko]

Hey guys, we also started encountering it, for example:
https://travis-ci.org/ParsePlatform/Parse-SDK-iOS-OSX/jobs/114691814

Since we are using Xcode 7.2 for quite a long time in Travis and didn't change anything in regards to how our workspace or project is setup.

All this translates to the fact that 0.15 or 0.15.1 breaks us in a major way :cry:

The previous build that ran on 0.14 and succeeded no problem:
https://travis-ci.org/ParsePlatform/Parse-SDK-iOS-OSX/jobs/111331930

Probably, you are right - looks related to the 15s time limit here:
https://github.com/Carthage/Carthage/blob/master/Source/CarthageKit/Xcode.swift#L493

I can try to isolate the xcodebuild piece there, but still think we probably are better of changing the limit not to 15s but to 15 minutes, since on a slow piece of hardware this breaks...
Let me know how I can help debug this and drive the change here.

[mdiep]

I can try to isolate the xcodebuild piece there, but still think we probably are better of changing the limit not to 15s but to 15 minutes, since on a slow piece of hardware this breaks...

If you time how long it takes xcodebuild -showBuildSettings (with the appropriate arguments) to run on Travis, I'll happily increase the timeout. Unfortunately, this is working around a bug in xcodebuild where it sometimes hangs _forever_. Making the victims of that bug wait 15 minutes on every build seems cruel. We need to measure before changing it.

[nlutsenko]

@mdiep, sounds good.
Will test it and get back to you here, since I believe this is somewhat a pressing issue for us here.

In regards to increasing the limit in the nutshell:
I understand that xcodebuild has a bug, and yes, it's quite cruel to make people wait 15 minutes on every build, so I am sure we can work together on this issue to find the time limit that works great for us all.

Here is a tad more details on where I am basing my current theory on:
If you go to this page and find the last build in the list for PR #834 (last in the list, not in time).
https://travis-ci.org/ParsePlatform/Parse-SDK-iOS-OSX/pull_requests

You can see that the same code, minus style changes doesn't work on 0.15, where it worked great on 0.14. Specifically: This build worked, this one didn't. The only thing that changed (except of style changes) - version of Carthage.

[nlutsenko]

Few test results so far:
https://travis-ci.org/ParsePlatform/Parse-SDK-iOS-OSX/jobs/115005316

If you grep (search) for real 0m - you'll be able to find that few of the calls take around 11-12 seconds, meaning that under stress of taking into account that we are running this through libDispatch and not directly against bash - probably take more than 15 seconds?
This also is based on my assumption of how ReactiveCocoa calculates timeout time for a signals, which might be different from what time gives us back in real wall-clock time.

The schemes that this takes such a long time to show build settings are for a iOS/watchOS sample app scheme. I bet it takes way more time for xcodebuild to come up with what is going on in the space of iOS/watchOS apps.

Frankly, I am not sure on the action at this point. Should we up this to 30s and see how it goes?
Another idea: What if we change the error to a warning? This probably has about the same level of verbosity and warning to people, while doesn't necessarily block long execution of xcodebuild on Travis-CI. Or maybe even do warning after 15s, then fail after 60s?

Save us @mdiep -Wan Kenobi, you are our only hope!

[dcaunt]

Thanks for adding your experience here @nlutsenko – I hadn't tried Carthage 0.14.

I have built two branches of my test project, carthage-0.14 and carthage-0.15, where the only difference is the version of Carthage used (via pkg installer). The branch using Carthage 0.14 builds successfully, where the Carthage 0.15 branch fails.

This suggests to me that we are not seeing xcodebuild timeouts, but a bug introduced in the changes between Carthage 0.14 and Carthage 0.15. This goes some way to explaining why it _appears_ to hang on the last scheme, and removing that scheme causes it to hang on the previous one.

[mdiep]

The timeout was added in Carthage 0.15, to work around the hanging I mentioned above (#1120), which is why you're not seeing this problem with 0.14.

I'll look at either doubling the timeout or doing an exponential back off. I probably won't get to it for a few days. But if you'd like to open a PR, that would probably speedup the process.

[dcaunt]

Sorry, perhaps I'm misunderstanding what you're saying, but on my Carthage 0.14 build, there is no hang at all – the bootstrap completes as expected and Carthage builds all frameworks in the workspace. Are you saying that before the timeout was added in 0.15, Carthage would have skipped the problem altogether somehow? As I understand the timeout, it just changes the behaviour from an indefinite hang to a hang of 15 seconds.

[mdiep]

The xcodebuild hang is nondeterministic. Some projects seem more prone to hit it than others, but even those wouldn't hit it every time.

[izqui]

I'm experiencing the same in Travis, xcodebuild hangs when reading the Realm project.

[nlutsenko]

My suspicion that the xcodebuild -showBuildSettings linearly depends on the project size, so it fails with a 15s timeout on big projects like Realm or Parse...
@mdiep, any chance you are about to fix this? I wish I had time to contribute to it, but unfortunately won't be able to get to it before weekend.

[dcaunt]

@lutsenko The part that still confuses me is that Carthage 0.14 was fine, and Carthage 0.15 is not. My hangs are consistent with the Carthage version. Why doesn't xcodebuild exhibit the same bug?

[nlutsenko]

Because that bug is transient and non-deterministic, meaning that we don't hit it very much often, but Carthage 0.15 added 15s timeout for xcodebuild -showBuildSettings operation.

[mdiep]

any chance you are about to fix this? I wish I had time to contribute to it, but unfortunately won't be able to get to it before weekend.

I _might_ be able to get to it tonight.

[izqui]

I downgraded to Carthage 0.14 and what I'm getting now is:

xcodebuild: error: Scheme Realm is not currently configured for the build action.

[mdiep]

@izqui Please open a new issue. This issue is tracking the timeout issue specifically.

[mdiep]

I opened #1192, which just doubles the timeout. If anyone here wants to try building a copy of carthage from that PR and trying it on CI to see if it fixes your issue, that would be much appreciated!

[mdiep]

I just released 0.15.2 with the increased timeout and submitted a PR to Homebrew.

[dcaunt]

Many thanks @mdiep

[menteb]

UPDATE: Never mind. Was a problem with the SwiftMongoDB version I used.
Needed the "master" version. Version 0.4 isn't ready yet for Carthage.

Hey! 0.15.2 still times out for me...
xcodebuild timed out while trying to read SwiftMongoDB.xcodeproj 😭

[mdiep]

@menteb Can you please test how long it takes to run the relevant xcodeubild command in whatever environment it's timing out? You can look in the xcodebuild log from Carthage for the command to use or run carthage with the --verbose flag.

[menteb]

@mdiep I tried to run with the flag (the log is always empty), but I get "Unrecognized command: '--verbose'. See carthage help." The SwiftMongoDB developer told me it's a 0.4 problem. The master version seems to compile fine. I don't think it's a Carthage problem.