Shards: Local overrides for shards development

[bcardiff]

When I am developing multiple shards and they have dependencies between them, like crystal-db and crystal-sqlite I can't specify that, during development, in my local machine the dependencies should be found in another way than the usual.

I would like to use a path: in crystal-sqlite but leave with the github: dependency in shard.yml for public usage.

I would like to be able to override where the dependencies should be looked in my environment. This is similar to http://bundler.io/v1.5/git.html allows me to do.

In order to allow this behavior per project basis I propose allowing shards to look up for shard.yml overrides in shard.local.yml that should not be checked in.

# crystal-sqlite/shard.yml
dependencies:
  db:
    github: bcardiff/crystal-db

# crystal-sqlite/shard.local.yml
dependencies:
  db:
    path: path/to/working-copy-of/crystal-db

Additionally we could expand shard.local.yml when creating a lib and adding it to the default .gitignore file.

[ysbaddaden]

Let's say I create shard.local.yml then run shards update: what happens to shard.lock? The resolver changed, the lock should change too, but then shard.lock won't represent shard.yml anymore... shall we stop updating shard.lock when shard.local.yml is present? But what if I update shard.yml and need shard.lock to be updated too? Shall I remove shard.local.yml first or introduce new options to deal with that?

I'm not sure this is better than changing the resolver in shard.yml or manually creating the link. We can also tell crystal to look the dependency somewhere else. First by creating an alternative libs directory:

$ mkdir local
$ cd local
$ ln -sf /path/to/crystal-db/src db

then telling Crystal to look into this directory first:

$ export $(crystal env)
$ export CRYSTAL_PATH="local:$CRYSTAL_PATH"

[ysbaddaden]

I didn't know the "local git repos" bundler feature. It's a well-thought one! It merely uses a locally cloned repository instead of the remote one, and still applies the Git resolver by using and locking commits. It doesn't change the resolver for another one on the local machine, and doesn't alter the behavior.

No need to clone the repository, local commits & branches are immediately available without pushing, and shard.lock can be updated to target individual commits. I can foresee a .shards/config or even a $HOME/.shards/config with an entry like:

local:
  db: /path/to/crystal-db

[bcardiff]

I see there are some issues around what will happen with the shard.lock but that applies only to applications, not to libraries.

Even so, for applications shouldn't be sound to either

1. exclude the overridden lib from the lock (imo this is prefered)
2. point the lock to the current commit of the local git repository

If I want to work with a local copy of the library is probably because I am trying to fix something so the app is not to be used in production or even shared in the middle of the fix.

All this is assuming the override is always for local path which actually is just one of the supported repositories. So the shard.local.yml could be restricted to path entries and keeping the same syntax.

The alternative of changing the crystal_path is not bad at all but it happened after a couple of projects that global overrides are a pain. Specially if you switch from one to another. I forget the global overrides and that affect me constantly.

[MaxBerendsCG]

What about being able to overwrite it as an development_dependancy like so:

dependencies:
  db:
    github: bcardiff/crystal-db

development_dependencies:
  db:
    path: path/to/working-copy-of/crystal-db

This way when you release it it should use the correct repo and when developing use the local one

[Sija]

@MaxBerendsCG Such solution still leaves dev with "tainted" shard.yml which cannot be commited into the git repository.

[lee-b]

That also requires all developers to use the same paths. Which aside from being jarring, raises cross-platform development questions.

A shards.local.yml which can override keypairs in shards.yml seems to make sense, as this can be excluded from version control.

I'm not sure what would be best regarding locks vs. shards.local.yml. Pointing shards to a path implies "Use whatever version is at this path", so that much seems fine, but it would be nice to ensure that once everything is tested, you can just push, and the locks will be correct. I tend to think a final test should be done locally after everything is ready and rebased etc., before anything is pushed, and so the locks should be updated from the final version at the path.

[RX14]

Adding local overrides creates the possibility of accidentally committing code which only compiles with the local override enabled. Having to modify shard.yml to use uncommitted dependencies is a reminder to yourself that you have to commit your work on your dependencies and publish them to git before committing. Having uncommitted changes in shard.yml is a feature not a bug.

The only good solution I can think to this problem is to have a quick command to edit (and revert) the shard.yml automatically using a shards.local.yml or similar. Uncommitted changes to shards.lock isn't enough since developers are trained to ignore those. Of course this runs into the same problem as shards add.

[bcardiff]

I have a bit of use case to share.

---

Scenario 1

I am building an app, with a framework that has a lib dependency X. While trying to upgrade crystal or the framework I notice X has issues. But X is not a direct dependency of my app.

As app developer I am ok to checkout the code and start digging and fixing things. But in order to do that as easy as possible I need to have a working copy of X and maybe of framework. And do not worry about the shard.yml in framework.

Scenario 2

The framework test suite uses crystal init or some sort of scaffolding adding dependencies. While trying to fix X, the specs/shard wont pick up those changes unless I change the source expanded in the scaffolding and maybe a fork is published.

---

The task I am trying to achieve is to fix X and submit that changes. After that is merged then the changes in the framework would come, there is no gain to do the framework changes it in a dirty way in my machine.

• I would value the possibility to have a global override. I don't care that much of per project override.
• The shard.lock could reflect the path as dependency and discard the original source. It would be inconsistent with the shard.yml. But would mean the override hooks in the source/version resolution of shards.
• If the shard.lock is generated with a override from a global shard.yml a warning in comment could be included as header. Helping to user not to commit the file.

The user has always the power to use shards wrong. For example checking in shard.lock in a lib shard. We try to minimize that by adding it to the .gitignore but there is no guarantees. Adding a banner could help minimize the possibility of checking a dirty state or at least clearly identifying one.

On every crystal release with enough breaking changes I might need to test crystal-db and drivers in the different frameworks to check everything runs. And since we have no configurable override is almost impossible to check the suite entirely before the proper release. The thing that I am able to do today is checkout a branch and replace the directory in lib, but that those not cover the 2nd scenario.

I have also used this technique in bundler to either diagnose or fix something that was happening in the app due to a gem. I found it really valuable if you are dealing with multiple and nested dependency that need to move forward.

[RX14]

test-ecosystem is a very fringe usecase, and one that will disappear after 1.0 (since any compilation failure would break backwards compatibility, there's no point in attempting to fix the code because the bug is in crystal itself). I'd really advise not optimizing for it, and instead overriding PATH and replacing it with a stubbed git command or some similar hackery. This pushes the complexity into test-ecosystem, not shards itself.

scenario 1 is more interesting, and I think it'd perhaps be better served by a way to override transitive dependencies' versions inside shards.yml. This way any modifications required to transitive dependencies are recorded inside shards.yml as they should be.

I don't think that we need to sacrifice usability to avoid global state in shards or to avoid making it easy to shoot yourself in the foot with a shards.local.yml. What we do need, regardless of the path forwards for this issue, is a robust way of modifying shards.yml, and if we can't get that with YAML we should consider other formats, or parsing only a very strict yaml subset

[bcardiff]

@RX14

test-ecosystem is a very fringe usecase

test-ecosystem is not my usecase. Fixing dependencies and nested dependencies is.

, and one that will disappear after 1.0

Although I look for fixes because of crystal/std lib the same use case will come after 1.0 with other group of shards were one of those can be changed.

... overriding PATH ..

Changing PATH is also error prone. In that scenario the shard.lock will be directly useless and will have misguided information. How are you sure that the PATH is not making things work in your local environment? How is that different of checking a warning banner in the shard.lock. With my last proposal that is more straight.

What we _do_ need, ...

Mentioning other things that we need does not add value to the discussion. It could change priority, but it does not help with argument specific to the topic. I don't find the issue regarding that feature, either link or create one to have that discussion there.

[RX14]

test-ecosystem is not my usecase

sorry, I misunderstood your usecase. I meant overriding PATH to point at a git-wrapper would be a limited solution only for test-ecosystem. Yes, it would be error prone and the shard.lock would end up with the incorrect repository names in, but that's vastly preferential to adding global state to shards.

My reference to programatically editing shards.yml was a reference to my previous solution to making it easy to override shards. The workflow would simply be:

shards override foo ../path_to_checkout

shards would then edit shard.yml from this:

dependencies:
  foo:
    github: user/foo
    version: 1.2.3

to this:

dependencies:
  foo:
    # github: user/foo
    # version: 1.2.3
    path: ../path_to_checkout

and then after you've submitted your changes upstream, you can simply use

shards override -d foo

and return back to normal.

I can't see there being any common usecase for a global override. And local overrides should be done by editing the shard.yml. I very strongly believe that shard.yml should be the sole source of truth for shards update and shard.lock should be the only source of truth for shards install. I believe the only thing to be done is to make it not feel like a pain to override.

Putting a large warning in the shard.lock won't help because unfortunately most devs don't review their diffs before comitting. Especially so for files like shard.lock which they are trained to ignore.

[ysbaddaden]

The user has always the power to use shards wrong. For example checking in shard.lock in a lib shard.

There is nothing wrong with that. Shards ignores it when installing the dependency, but uses it when developing the library.

programatically editing shards.yml

That doesn't solve anything: shard.yml and shard.lock are still altered, and the CLI command is just as complex as editing the file.

[RX14]

@ysbaddaden Then there's nothing solvable here.

Yes, you can use shards wrong but we should try to prevent it as much as possible at the design level. If that conflicts with usability, then designing robust software wins every day. You'll be thanking me when you push a broken shards.yml to deployment that "worked on your machine" because of local config.