Shards: "Failed to match versions for" errors on 0.9.0.rc2

[straight-shoota]

This shard file reproduces an error with the current RC2 of 0.9.0:

# shard.yml
name: test
version: 0.1.0

dependencies:
  shards:
    github: crystal-lang/shards
    branch: master
  raven:
    github: Sija/raven.cr

$ shards-0.9.0.rc2 install
Resolving dependencies
Fetching https://github.com/crystal-lang/shards.git
Fetching https://github.com/Sija/raven.cr.git
Fetching https://github.com/sija/any_hash.cr.git
Failed to match versions for #<Shards::Dependency {any_hash => {"github" => "sija/any_hash.cr", "version" => "0.2.1"}}>
Conflict can't install 'raven' versions 1.2.0 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 1.1.0 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 1.0.0 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 1.0.0-rc3 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 1.0.0-rc2 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 1.0.0-rc1 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 0.9.0 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 0.8.0 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 0.7.0 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 0.6.0 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 0.5.0 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 0.4.0 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 0.3.0 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 1.3.0 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 1.3.0 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 1.3.0 requires any_hash ~> 0.2.2
Conflict can't install 'raven' versions 1.1.0 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 1.0.0 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 1.0.0-rc3 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 1.0.0-rc2 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 1.0.0-rc1 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 0.9.0 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 0.8.0 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 0.7.0 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 0.6.0 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 0.5.0 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 0.4.0 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 0.3.0 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 1.2.0 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 1.2.0 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 1.2.0 requires any_hash ~> 0.2.2
Conflict can't install 'raven' versions 1.0.0 and 1.1.0 at the same time.
Conflict can't install 'raven' versions 1.0.0-rc3 and 1.1.0 at the same time.
Conflict can't install 'raven' versions 1.0.0-rc2 and 1.1.0 at the same time.
Conflict can't install 'raven' versions 1.0.0-rc1 and 1.1.0 at the same time.
Conflict can't install 'raven' versions 0.9.0 and 1.1.0 at the same time.
Conflict can't install 'raven' versions 0.8.0 and 1.1.0 at the same time.
Conflict can't install 'raven' versions 0.7.0 and 1.1.0 at the same time.
Conflict can't install 'raven' versions 0.6.0 and 1.1.0 at the same time.
Conflict can't install 'raven' versions 0.5.0 and 1.1.0 at the same time.
Conflict can't install 'raven' versions 0.4.0 and 1.1.0 at the same time.
Conflict can't install 'raven' versions 0.3.0 and 1.1.0 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 1.1.0 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 1.1.0 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 1.1.0 requires any_hash 0.2.2
Conflict can't install 'raven' versions 1.0.0 and 1.0.0-rc3 at the same time.
Conflict can't install 'raven' versions 1.0.0 and 1.0.0-rc2 at the same time.
Conflict can't install 'raven' versions 1.0.0 and 1.0.0-rc1 at the same time.
Conflict can't install 'raven' versions 0.9.0 and 1.0.0 at the same time.
Conflict can't install 'raven' versions 0.8.0 and 1.0.0 at the same time.
Conflict can't install 'raven' versions 0.7.0 and 1.0.0 at the same time.
Conflict can't install 'raven' versions 0.6.0 and 1.0.0 at the same time.
Conflict can't install 'raven' versions 0.5.0 and 1.0.0 at the same time.
Conflict can't install 'raven' versions 0.4.0 and 1.0.0 at the same time.
Conflict can't install 'raven' versions 0.3.0 and 1.0.0 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 1.0.0 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 1.0.0 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 1.0.0 requires any_hash 0.2.2
Conflict can't install 'raven' versions 1.0.0-rc2 and 1.0.0-rc3 at the same time.
Conflict can't install 'raven' versions 1.0.0-rc1 and 1.0.0-rc3 at the same time.
Conflict can't install 'raven' versions 0.9.0 and 1.0.0-rc3 at the same time.
Conflict can't install 'raven' versions 0.8.0 and 1.0.0-rc3 at the same time.
Conflict can't install 'raven' versions 0.7.0 and 1.0.0-rc3 at the same time.
Conflict can't install 'raven' versions 0.6.0 and 1.0.0-rc3 at the same time.
Conflict can't install 'raven' versions 0.5.0 and 1.0.0-rc3 at the same time.
Conflict can't install 'raven' versions 0.4.0 and 1.0.0-rc3 at the same time.
Conflict can't install 'raven' versions 0.3.0 and 1.0.0-rc3 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 1.0.0-rc3 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 1.0.0-rc3 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict can't install 'raven' versions 1.0.0-rc1 and 1.0.0-rc2 at the same time.
Conflict can't install 'raven' versions 0.9.0 and 1.0.0-rc2 at the same time.
Conflict can't install 'raven' versions 0.8.0 and 1.0.0-rc2 at the same time.
Conflict can't install 'raven' versions 0.7.0 and 1.0.0-rc2 at the same time.
Conflict can't install 'raven' versions 0.6.0 and 1.0.0-rc2 at the same time.
Conflict can't install 'raven' versions 0.5.0 and 1.0.0-rc2 at the same time.
Conflict can't install 'raven' versions 0.4.0 and 1.0.0-rc2 at the same time.
Conflict can't install 'raven' versions 0.3.0 and 1.0.0-rc2 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 1.0.0-rc2 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 1.0.0-rc2 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 1.0.0-rc2 requires any_hash *
Conflict can't install 'raven' versions 0.9.0 and 1.0.0-rc1 at the same time.
Conflict can't install 'raven' versions 0.8.0 and 1.0.0-rc1 at the same time.
Conflict can't install 'raven' versions 0.7.0 and 1.0.0-rc1 at the same time.
Conflict can't install 'raven' versions 0.6.0 and 1.0.0-rc1 at the same time.
Conflict can't install 'raven' versions 0.5.0 and 1.0.0-rc1 at the same time.
Conflict can't install 'raven' versions 0.4.0 and 1.0.0-rc1 at the same time.
Conflict can't install 'raven' versions 0.3.0 and 1.0.0-rc1 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 1.0.0-rc1 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 1.0.0-rc1 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 1.0.0-rc1 requires any_hash *
Conflict can't install 'raven' versions 0.8.0 and 0.9.0 at the same time.
Conflict can't install 'raven' versions 0.7.0 and 0.9.0 at the same time.
Conflict can't install 'raven' versions 0.6.0 and 0.9.0 at the same time.
Conflict can't install 'raven' versions 0.5.0 and 0.9.0 at the same time.
Conflict can't install 'raven' versions 0.4.0 and 0.9.0 at the same time.
Conflict can't install 'raven' versions 0.3.0 and 0.9.0 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 0.9.0 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 0.9.0 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 0.9.0 requires any_hash *
Conflict can't install 'raven' versions 0.7.0 and 0.8.0 at the same time.
Conflict can't install 'raven' versions 0.6.0 and 0.8.0 at the same time.
Conflict can't install 'raven' versions 0.5.0 and 0.8.0 at the same time.
Conflict can't install 'raven' versions 0.4.0 and 0.8.0 at the same time.
Conflict can't install 'raven' versions 0.3.0 and 0.8.0 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 0.8.0 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 0.8.0 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 0.8.0 requires any_hash *
Conflict can't install 'raven' versions 0.6.0 and 0.7.0 at the same time.
Conflict can't install 'raven' versions 0.5.0 and 0.7.0 at the same time.
Conflict can't install 'raven' versions 0.4.0 and 0.7.0 at the same time.
Conflict can't install 'raven' versions 0.3.0 and 0.7.0 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 0.7.0 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 0.7.0 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 0.7.0 requires any_hash *
Conflict can't install 'raven' versions 0.5.0 and 0.6.0 at the same time.
Conflict can't install 'raven' versions 0.4.0 and 0.6.0 at the same time.
Conflict can't install 'raven' versions 0.3.0 and 0.6.0 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 0.6.0 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 0.6.0 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 0.6.0 requires any_hash branch master
Conflict can't install 'raven' versions 0.4.0 and 0.5.0 at the same time.
Conflict can't install 'raven' versions 0.3.0 and 0.5.0 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 0.5.0 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 0.5.0 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 0.5.0 requires any_hash branch master
Conflict can't install 'raven' versions 0.3.0 and 0.4.0 at the same time.
Conflict can't install 'raven' versions 0.2.0 and 0.4.0 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 0.4.0 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 0.4.0 requires any_hash branch master
Conflict can't install 'raven' versions 0.2.0 and 0.3.0 at the same time.
Conflict can't install 'raven' versions 0.1.0 and 0.3.0 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 0.3.0 requires any_hash branch master
Conflict can't install 'raven' versions 0.1.0 and 0.2.0 at the same time.
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 0.2.0 requires any_hash branch master
Conflict can't install 'any_hash' versions 0.2.2+git.commit.b7867dd105b0af7ab38bd0a63162494416cdb27f and 0.2.2 at the same time.
Conflict raven 0.1.0 requires any_hash branch master
Conflict test requires raven *
Failed to resolve dependencies

Using shards 0.8.1 it installs correctly:

$ shards-0.8.1 install
Fetching https://github.com/crystal-lang/shards.git
Fetching https://github.com/Sija/raven.cr.git
Fetching https://github.com/Sija/any_hash.cr.git
Installing shards (0.9.0.rc2 at master)
Using raven (1.3.0)
Using any_hash (0.2.2)

The dependency on shards seems to be important for the error. Originally, there were other dependencies as well, but I could remove all of them and the error only reproduces with shards.
When there are more dependencies, output hangs after the line Failed to match versions for #<Shards::Dependency {any_hash => {"github" => "sija/any_hash.cr", "version" => "0.2.1"}}>, probably because of exponential combinations.

[ysbaddaden]

I found a problem —I'm not sure this is the actual issue—: there are two any_hash repository definitions in Raven's shard.yml:

1. sija/any_hash.cr;
2. Sija/any_hash.cr;

Shards considers both shards to be the same (same name) yet to not be the same (distinct repositories). Git doesn't fail when cloning repositories because GitHub considers user/repo to be case-insensitive...

[ysbaddaden]

This at least explains the Failed to match versions warning message.

No, it doesn't. I force all Dependency to downcase all github: repositories and it still fails.

[Sija]

I found a problem —I'm not sure this is the actual issue—: there are _two_ any_hash repository definitions in Raven's shard.yml:

@ysbaddaden I can't verify it, where exactly you see them?

[ysbaddaden]

You changed the definition at some from sija/any_hash.cr to Sija/any_hash.cr (around raven 0.7 or 0.8 I think). Since Shards 0.9 builds a graph of all dependencies (before it optimistically looked at latest versions only), so it could lead to an issue, thought it doesn't seem to be the _actual_ issue (sadly, it could have been easy to fix).

[ysbaddaden]

I found another lead.

To speed up the dependency graph generation (extracting, parsing and following shard.yml definitions involves a lot of external calls to git which is super slow) and to speed up the SAT resolution time (less variables and clauses == less noise == faster resolution time) I skip inaccessible versions when building the graph, which avoids to fetch useless specs and follow useless nested dependencies.

_NOTE: The speedup isn't negligible, it's easily 10× or 20× faster, dropping from 10s to 0.5s_ in simple cases.

But it seems to be completely wrong for this spec :disappointed:

For example raven:0.1.0 depends on any_hash:master but clauses actually make it depend on any_hash:0.2.2+git.commit... (correct) or any_hash:0.2.2 (I think it's wrong). Further versions have an open * version, so raven:0.7.0 should depend on all _versions_ of any_hash but it only matches master and 0.2.2, and so on with other versions.

I think the first time a dependency is considered it will force its versions to be absolute and will then never consider any other version. This is fine for the project's spec (it dictates thresholds) but that's plain wrong for nested dependencies... or rather, it should be temporary, only applying to the current scope —with the project's spec always being in scope.

This explains the Failed to match versions warning: any_hash:0.2.1 is never considered when it should be. Shards notices it and complains.

Since the dependency graph is invalid, it's possible for the solver to fail to find a satisfying solution, which may explain this issue (hopefully).

[ysbaddaden]

:bug: found: GitResolver is initialized with the first Dependency definition found (i.e. raven:0.1.0 depends on any_hash:master), since there is a git refs, #available_versions will only return the version for that refs commit (i.e. any_hash:0.2.2) and skip all other versions :hankey:

This is a legacy relic from Shards 0.8 (and below) that must disappear for Shards 0.9 to correctly generate the dependency graph.

[ysbaddaden]

It fixes the Failed to match versions issue, but still doesn't fix the Failed to resolve dependencies issue :sob:

[straight-shoota]

Do you have any clue why this error only appears when shards is also a dependency, bot not with other dependencies?

[ysbaddaden]

Yes: the culprit is shards, not raven (it exposed unrelated issues explained above). The following spec will fail:

dependencies:
  shards:
    github: crystal-lang/shards
    branch: master

Using any refs will fail to resolve dependencies.

[ysbaddaden]

Allright, :bug: found (for real): shards' version at branch:master is a _pre-release_ (0.9.2.rc2), but pre-releases are opt-in, and if a proposed solution selected one, then it must have been requested somewhere in the dependency graph, otherwise the solution is skipped, but Git refs don't count as requesting a pre-release version...