Carthage: Dependency's Cartfile cannot have local paths to binaries inside repo

[sven-m]

TLDR: I cannot use a dependency that itself has a binary dependency with a local path (e.g. deps/lib.json). Carthage assumes the path is to be used from the current working directory instead of the dependency's repository directory.

• carthage install method: [ ] .pkg, [x] homebrew, [ ] source
• which carthage: /usr/local/bin/carthage
• carthage version: 0.29.0
• xcodebuild -version: Xcode 9.4\nBuild version 9F1027a
• Are you using --no-build? No
• Are you using --no-use-binaries? No
• Are you using --use-submodules? No
• Are you using --cache-builds? No
• Are you using --new-resolver? No

Client app Cartfile

git "git.example.com:example/private-sdk.git" "make-compatible-with-carthage"

Private SDK's Cartfile

# the path dependencies/carthage-json is a local path within the private sdk repository
binary "dependencies/carthage-json/fabric.json"
binary "dependencies/carthage-json/crashlytics.json" 

Carthage Output

$ pwd
/<redacted>/client-app
$ carthage bootstrap --platform iOS
*** No Cartfile.resolved found, updating dependencies
*** Fetching private-sdk
*** Downloading binary-only framework fabric at "file:///<redacted>/client-app/dependencies/carthage-json/fabric.json"
*** Downloading binary-only framework crashlytics at "file:///<redacted>/client-app/dependencies/carthage-json/crashlytics.json"
Failed to read file or folder at /<redacted>/client-app/dependencies/carthage-json/fabric.json: Error Domain=Result.AnyError Code=1 "The requested URL was not found on this server."

Actual outcome
Carthage tries to resolve the local file URL with the current working directory as the base path

Expected outcome
Carthage should somehow resolve the local file URL with the checked out private-sdk repository's directory as base path, e.g. file:///<redacted>/client-app/Carthage/Checkouts/private-sdk/dependencies/carthage-jsons/fabric.json. The obvious problem here is that this would not work with --no-checkout.

Notes
At first I intended to submit a PR, but I'm stuck at the part where the Cartfile.resolved in its current form cannot represent the "base URL" of a certain resolved binary dependency. The Cartfile from which the dependency originates still has this information by virtue of its own location, but the Cartfile.resolved at the top level does not.

[tmspzz]

Seems like a good feature to add. Would you like to take a shot at it? @sven-m

[sven-m]

@blender, good to hear that :). I actually have been trying to implement this myself and have had some progress in doing so, but right now I am stuck, so I decided to open this issue in order to get some help.

I have been able to add "repository" information to the BinaryURL, but I do not know how to encode this information into the Cartfile.resolved. In the current implmentation this information is lost upon generating the Cartfile.resolved.

It would be nice to have a way to determine the repository based on the resolvedDescription (currently the local path), but I do not know if that is possible. Also, in case there are multiple dependencies which happen to have a dependency using a local binary json using the exact same path, we cannot know which repository to use, unless we somehow encode that information into the Cartfile.resolved in a backwards compatible way.

[tmspzz]

You should not need to encode path info in the Cartfile.resolved

Can't you use the location of the current dependency when reading the Cartfile.resolved to resolve binary relative paths, local to that dependency?

[sven-m]

I'm not sure if I understand you correctly.

Can't you use the location of the current dependency

"current dependency" here means "private-sdk", right?

when reading the Cartfile.resolved to resolve binary relative paths, local to that dependency?

well, the Cartfile.resolved of the "client-app" only contains the following:

binary "dependencies/carthage-json/fabric.json" "1.7.7"
git "git.example.com:example/private-sdk.git" "make-compatible-with-carthage"

My current problem is that I do not know how to find the dependency that introduced binary "dependencies/carthage-json/fabric.json" "1.7.7", when everything has been flattened into a single Cartfile.resolved.

[tmspzz]

How about

__Cartfile__

git "git.example.com:example/private-sdk.git" "make-compatible-with-carthage"

Should produce the following __Cartfile.resolved__

binary "Carthage/Checkouts/private-sdk/<path>/fabric.json"
binary "Carthage/Checkouts/private-sdk/<path>/crashlytics.json"

Input here from @dmiluski can help

[dmiluski]

Correct me if my interpretation is wrong:

Given:

1. You have a main project, with a dependency installed via Carthage.
2. And that dependency has another binary only dependency specified in its Cartfile

When you run bootstrap or update:
You want it to reference:
file:////client-app/Carthage/Checkouts/private-sdk/dependencies/carthage-jsons/fabric.json

But instead it's attempting to reference: (Where a file doesn't exist)
file:////client-app/carthage-jsons/fabric.json

Yeah, that's a use case I overlooked.

I'm not super positive of the full file structure, but wondering what is available where, and if we could update:
https://github.com/Carthage/Carthage/blob/master/Source/CarthageKit/Dependency.swift#L151

To reference path relative to the dependency?

[sven-m]

First of all, @blender & @dmiluski, thanks for getting involved, I am grateful for the advice and help.

@blender, yes that could work, except for the fact that Carthage normally does not checkout out the dependencies until late in the process.

@dmiluski, I have tried getting the json from the dependency's cached repository in the ~/Library, basically using the same mechanism as is used to get the Cartfiles of all the dependencies (Project.contentsOfFileInRepository(_:_:revision:)). I do so by adapting BinaryURL and Dependency.from(_:) to house a URL to the cached repository for binary local dependencies.

It break down, however, at the moment when the Cartfile.resolved is written and is read to actually checkout the dependencies. I should either make sure the dependencies are checked out sooner, so I can depend on the Checkouts/private-sdk/ directory to exist or somehow get the repository information over to the second stage where the dependencies are checked out using Cartfile.resolved.

[sven-m]

By the way, in order to make it more clear, I pushed a branch to my fork to demonstrate my attempt: https://github.com/Carthage/Carthage/compare/master...sven-m:transitive-binary-local-json-url?expand=1

[tmspzz]

I think you have a mistake here: https://github.com/Carthage/Carthage/compare/master...sven-m:transitive-binary-local-json-url?diff=split&name=transitive-binary-local-json-url#diff-d36e2d0823f4b902aeb9274469b117c0R159 url from decomposing the enum hides the local url variable. The line does not make sense to me as it stands.

[sven-m]

oops, you're right, that's very confusing and wrong. Thanks, I'll fix it right away. But of course that implementation is still missing a big piece: how to handle the Cartfile.resolved.

[tmspzz]

@sven-m

If you're able to encode in the resolved:

binary "Carthage/Checkouts/private-sdk/<path>/fabric.json"
binary "Carthage/Checkouts/private-sdk/<path>/crashlytics.json"

Then by looking at the path you know that you have to check out private-sdkfirst

[tmspzz]

Each .resolved has to encode the relative paths as seen by the project it lives in.

Example:

A -> B
B -> C
C -> D

where D binary expressed relative to C

D appears in:

A.resolved -> Carthage/Checkouts/C/D.json
B.resolved -> Carthage/Checkouts/C/D.json
C.resolved -> D.json

In case also A->D

Then

A.resolved -> D.json
B.resolved -> Carthage/Checkouts/C/D.json
C.resolved -> D.json

[sven-m]

Sounds logical! It seems that apart from populating the BinaryURL.resolvedDescription with "Checkouts/<dep>/relative/path/to.json" I need to make sure that "Checkouts/*" paths are recognized when processing Cartfile.resolved so that they get checked out! Will get on it tonight 😄

Probably will be challenging though, so expect more questions on that. Thanks for the help so far :)

[tmspzz]

yes, you have to ignore Carthage/Checkouts/ prefix

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shoerob]

Has this been solved? I've began running into this problem as of lately.

[chirag-farfetch]

same i am getting the same error as well.