Shards: Clash repository names under global cache

[luislavena]

Hello!

While shards nature encourages decentralized repositories (ie. github: user/repo), found that the dependency name provided will trigger clashes when dealing with forks or other projects sharing the same name.

As example, imagine two projects in your computer (project1 and project2) which both reference with the same name a different dependency:

# project1/shard.yml

dependencies:
  db:
    github: crystal-lang/crystal-db
    branch: master

# project2/shard.yml

dependencies:
  db:
    git: [email protected]:user/db.git
    branch: stable

The dependency is named db, but the source for it is not the same. This causes some issues when working on different projects, as the specific directory from the cache needs to be removed.

I've work around this by appending computed SHA1 of the repository url to the name, so instead of using .cache/shards/db results in .cache/shards/db-ea075e132540dcb680b364c5694bd78afbfeb5b5 (for https://github.com/crystal-lang/crystal-db.git).

Before I submit a PR, will be great to hear thoughts about this issue. Please note that db in this is just an example, it could be any other sort of repository.

Thank you in advance.

Cheers.

[straight-shoota]

How is this only an issue for the cache? Wouldn't the name clash happen in the lib directory as well?

The cache should probably use a key based on the remote, not the internal name. In this case for example .cache/shards/github-crystal-lang-crystal-db and .cache/shards/[email protected]. Because if you change the remote, the cache needs to change as well.

[luislavena]

How is this only an issue for the cache? Wouldn't the name clash happen in the lib directory as well?

It will not, the example are two different shard.yml (one for each project). AFAIK you cannot define two dependencies with the same name, so there shouldn't be a problem.

The cache should probably use a key based on the remote

Perhaps that is a better idea than using the dependency name + SHA1 of the URL.

Cheers.

[straight-shoota]

Yeah but what if there was a shard that had both project1 and project2 as a depenency? They use the same key for different dependencies. But all dependencies (including sub dependencies) would be put in ./lib/.

[straight-shoota]

I justed tested this and shards just installs one of the homonymous dependencies without stating any issues. I guess this problem can't really be solved, but shards install should fail to install two dependencies of the same name.

[luislavena]

Not sure that I follow, but the scenario I presented is more common when you switch projects than having these nested dependency issues you describe.

I agree that might be an issue when dealing with dependencies of dependencies and lib involvement, but that is not what is affecting me or the reason I raised this issue.

[ysbaddaden]

Name clashing is out-of-scope of Shards: it's decentralized, shard names may clash; without a central registry (or switching to java-like namespaces) there is little we can do to prevent this.

Yet, a shard may come from different remote sources (e.g. fork, server mirror). For example using a remote for a project shouldn't clash with another project that uses the upstream version. A local cache avoided the issue completely, a global cache... must deal with it.

Teaching shards to deal with different remotes is error-prone: we may have project A using a more up-to-date fork of dependency X, but project B still points to an older repository, suddenly we have non-reproducible installations because A influenced B on one computer versus another one —some refs may not be found, for example a newer version tag.

I suppose that we should have a cache/host/path scheme. For example:

~/.cache/shards/github.com/crystal-lang/crystal-db
~/.cache/shards/gitlab.com/user/db

[straight-shoota]

I'd suggest the naming scheme to consist of the name of the resolver as first component and a unique identifier depending on the resolver. There should be an abstract method for this.
For a github, gitlab and bitbucket resolver this should the value of dependency[self.class.key] (like it is used in the git_url method). Git resolver will need the complete URL.

For the naming clash in the lib directory I'll open another issue.

[luislavena]

I'd suggest the naming scheme to consist of the name of the resolver as first component and a unique identifier depending on the resolver.

That might result in duplicated repositories if git resolver is used directly instead of github, gitlab or bitbucket:

dependencies:
  db:
    git: https://github.com/crystal-lang/crystal-db.git
    branch: master

If a project uses that instead of github resolver, it will result in two separate clones of the same repository.

I think @ysbaddaden suggestion of using host + path might reduce these occurrences. I still need to play a bit more with other non URI based repositories.

Cheers.

[RX14]

The github and gitlab resolvers would ideally be exact shortcuts for the git resolver, and they should behave the same. Future possible use of the github API instead of git directly would be done by special-casing the hostname instead of checking that the dependency was github:.

[ysbaddaden]

Don't raise concerns out-of-the-blue. Stop, think and explain why .cache/shards/:host/:path is not enough.

1. There are only 2 resolvers: git and path. The others, namely github, gitlab and bitbucket are virtual resolvers on top of the git resolver —they merely rewrite the user/repo as git://host/user/repo.git. Out of these 2 resolvers, we only need to cache the git resolver. No problem.

2. Let's say we add support for Mercurial (hg) someday, we'd now have 2 real resolvers that should be cached. GitHub and GitLabl only support Git. Bitbucket supports both Git and Mercurial but a bitbucket.org/user/repo may be either Git or Mercurial and can't be both. No problem.

3. Let's say we add support for Subversion (svn) and a provider supports both a Git and a Subversion view for a same repository, and 2 different shards refer to either. In that purely hypothetical situation, then yes, maybe there is a problem.

Maybe .cache/shards/:host/:path is imperfect and future-proof to a purely theoretical issue. I don't think it's worth it, but maybe we can add a .git suffix (or .hg, .svn, ...). It's useless (all cache directories will have a .git suffix) but if you insist:

~/.cache/shards/github.com/crystal-lang/crystal-db.git