Shards: Global cache issues

[paulcsmith]

I just updated to 0.25 of Crystal which uses shards 0.8 with the new global cache.

The shard cache failed to update under these circumstances:

• Defined a shard (lucky_cli) in shard.yml using version
• Then changed the definition to use branch: master
• Ran shards update
• Then made some changes to the lucky_cli
• Ran shards update
• Cache still used the old version of branch: master instead of updating

Temporary fix

rm -rf ~/.cache/shards and re-run `shards update.

Possible fix

• When running shards update always update cache if using branch or tag?

[kostya]

oh, i just try to find why my shard not updated, rm -rf ~/.cache/shards helps. it not update even for version not for master only. (after update crystal and shard to 0.8.0)

[wontruefree]

It seems like if you are using a branch for the version it should always have a cache miss.
It seems like something would have to be added here https://github.com/crystal-lang/shards/blob/master/src/commands/check.cr#L41 to always return false for installed.

[kostya]

dependencies:
  myhtml:
    github: kostya/myhtml
    version: ">= 0.29"

shards update
Fetching https://github.com/kostya/myhtml.git
Error resolving myhtml (>= 0.29)

but 0.29 there is, i just push it.
i think this is quite critical issue.

rm -rf ~/.cache/shards
shards update
Fetching https://github.com/kostya/myhtml.git
Installing myhtml (0.29)
Postinstall cd src/ext && make package

[ysbaddaden]

Sigh. Can someone reproduce and run shards commands with --verbose, look at the global cache structure, and start investigating?

I won't be able to look at this anytime soon.

[ysbaddaden]

It's probably not the global cache, but not mirroring the Git repository anymore. A simple --bare and fetch isn't enough to pull Git tags. Without them, Shards is incapable to resolve versions.

[ysbaddaden]

If you type git commands in the cached git folder, you probably won't see the new tags, even after a shards update.

[kostya]

shards update --verbose
git ls-remote --get-url origin
Fetching https://github.com/kostya/stuffs.git
git fetch --all --quiet
git tag --list --column=never
versions: 0.1
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml
Installing stuffs (0.1)
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git archive --format=tar --prefix= v0.1 | tar -x -f - -C '/Users/kostya/projects/blah5/lib/stuffs'
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml

after pushing v0.2

shards update --verbose
git ls-remote --get-url origin
Fetching https://github.com/kostya/stuffs.git
git fetch --all --quiet
git tag --list --column=never
versions: 0.1
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml
Using stuffs (0.1)
git ls-tree -r --full-tree --name-only v0.1 -- shard.yml
git show v0.1:shard.yml

after rm -rf ~/.cache/shards/

shards update --verbose
Fetching https://github.com/kostya/stuffs.git
git clone --bare --quiet -- 'https://github.com/kostya/stuffs.git' /Users/kostya/.cache/shards/github.com/kostya/stuffs.git
git tag --list --column=never
versions: 0.2, 0.1
git ls-remote --get-url origin
git ls-tree -r --full-tree --name-only v0.2 -- shard.yml
git show v0.2:shard.yml
git ls-tree -r --full-tree --name-only v0.2 -- shard.yml
git show v0.2:shard.yml
git ls-tree -r --full-tree --name-only v0.2 -- shard.yml
git show v0.2:shard.yml
Installing stuffs (0.2)
rm -rf '/Users/kostya/projects/blah5/lib/stuffs'
git ls-tree -r --full-tree --name-only v0.2 -- shard.yml
git archive --format=tar --prefix= v0.2 | tar -x -f - -C '/Users/kostya/projects/blah5/lib/stuffs'
git ls-tree -r --full-tree --name-only v0.2 -- shard.yml
git show v0.2:shard.yml

[kostya]

i just test it, it works if replace --bare with --mirror (from https://stackoverflow.com/questions/3382679/how-do-i-update-my-bare-repo).
Git fetch not update in bare repo, but update in mirror repo.