Shards: name clash for root directory 'lib'

[patleb]

I know it has already been discussed here, but I wanted to know if you would consider maybe changing the directory used for shards as something less problematic when crystal is used in a mono-repo with several languages?

Something like crystal_shards (similar to node_modules) might be a good candidate, what do you think?

I'm asking because lib is often used in projects I work on: mostly in Ruby, but also in NodeJS and PHP (and I suspect in some other languages that I don't use). I know it's a bit late to change something like that, but Crystal has not reached 1.0 yet and I thought it could be worth it to ask before that time.

[patleb]

Also, I would not be asking if lib wasn't something we need to gitignore in crystal, otherwise, it would be manageable.

[j8r]

How is it problematic? Adding /lib to .gitignore does the job. This directory name is defined in the Crystal compiler, used for requires, not in shards. Thus, its name needs to be standard, and not mention a specific package manager.

[patleb]

I don't think you understand what I meant. Having actual application code in lib and shards in it is problematic. It's pretty much the same as having lib and src merged together.

[patleb]

The problem is when you want to use crystal with other project structures which use lib directory for other means than external libraries (which happens pretty much for almost all of the projects I've worked on).

[patleb]

The important point here is: using crystal with other languages.

[j8r]

I see. There is already an answer from @ysbaddaden on this topic https://github.com/crystal-lang/shards/issues/198#issuecomment-378235009 . I don't anything will change.

[patleb]

I already read the discussion, but I thought it was important enough to mention the problematic from another angle since I think it might have unforeseen consequences down the road: one of which is gradual integration of crystal into existing projects.

Let's put this another way: if NodeJS would have decided to name lib their node_modules directory, how well do you think it would have played with existing code bases?

[ysbaddaden]

A monorepo doesn't mean that everything must be flattened and mix languages at its root. Both Node.js and Ruby put their sources into lib and both Rust and Crystal put them into src and that's already conflicting, without looking at where dependencies are installed.

Do you have actual use cases where the lib folder is an actual issue that can't be fixed?

[patleb]

Yes, it could be fixed, but implies playing with the gitignore. As for actual use cases, I have several:

1- Rails app with rake|capistrano tasks in lib directory, overrides in lib directory;
2- Express.js/Symfony(php) app with generic code in lib directory;
3- Ruby gem/engine with actual code in lib directory;

These are examples of repo without shared code and as for examples which I would like to share code:

1- Rails app besides an Amber app with shared settings, migrations, model configurations, etc.;
2- Ruby gem besides a Crystal Shard sharing C or C++ extension with some part of the build process;
3- Ruby gem besides a Crystal Shard sharing some settings, migrations, model configurations, etc. (same use case as for an application);

I know theses scenarios could all be covered by extracting everything into different repos, but I don't think this is the simplest solution.

But again, I think I might not have been clear enough: lib is something that normally you would want in your repo but, in crystal context, it's used for external libraries, so this is not something that you want in your repo. Thus, the lib directory has a conflicting role if you use crystal with other languages. And again, for comparison sake, imagine that NodeJS would have named node_modules lib or src, then I'm pretty sure that you could come with countless scenarios where there would be conflicting roles for that directory.

[patleb]

Also, I don't think that Ruby and Rust are good examples for comparison since their dependencies are managed outside of the code repository. The closest language that I can think of that does something similar to Crystal for dependencies is NodeJS.

[RX14]

What @ysbaddaden was suggesting was to put different projects in different languages in different directories so there's no clash. Are there any usecases where the crystal and ruby/nodejs projects must live in the same folder?

I think it's unlikely we'll rename lib.

[ysbaddaden]

I'm not sure whether this is an actual problem, but I understand the concern.

Thought unlikely, we could change the folder —I'm not against using vendor/crystal to mimic Bundler for example— but this ain't for Shards alone to decide: the Crystal compiler expects dependencies to be installed under lib.

ping @asterite @bcardiff

[asterite]

I wouldn't mind changing the name of the directory, for example crystal_shards or similar

[Sija]

I'd go for vendor,vendor/shards to be exact, since it would reuse commonly accepted dir structure for vendored libs, adding shards as a package manager name - similarly to vendor/bundle used by Bundler.

[RX14]

I don't want to change the name, and if we did using vendor would be a huge mistake before there was a well thought out complete, implemented vendoring story in shards. I don't think that just committing lib will turn out to work well for vendoring. I might be wrong but locking ourselves into that would be a mistake.

[j8r]

lib is good. It's a common name, used in languages like Ruby and in the system with /lib, /usr/lib. In anyway the name must be independent of the package manager.
Maybe crystal_modules or modules, because usually a library is in fact a Module (or another namespaced object)

[ysbaddaden]

There is no rush to change. I'm just open to the suggestion.

@RX14 can you detail/open an issue about vendoring as you intend it? Shards currently installs a snapshot of each dependency; what would be different with vendoring? Maybe the path resolver (copy instead of link)?

[asterite]

We could aso have a .crystal file that specifies some configuration, for example where to find dependencies. It could be lib by default but easily changeable to something else. Just an idea.

[bcardiff]

I don't mind changing lib to crystal_shards, crystal_deps or something like that to stay agnostic of how the dependencies got installed.

If there is to be a configuration file, I would like that to be discussed independently of this feature, so we can handle per-project / per-user settings. Things like default spec formatter, verbosity, flags are things that might get into general configuration.

[ysbaddaden]

Maybe a .crystal directory? With dependencies installed under .crystal/shards.

That would allow an eventual .crystal/config and other crystal specific things. It wouldn't take the space for a vendor/crystal for vendoring, and there should be no need for configuring where dependencies are installed (no clashes).

[RX14]

I think even discussing what it would be changed to is getting ahead of ourselves. Finding a better name by itself is not a good reason to change. These kind of discussions tend to veer off into the interesting discussion of naming, forgetting the reason - if any - there were for making the change in the first place.

So please, let's discuss why change and usecases to change before anything else.

[patleb]

@RX14, thanks for clarifying what @ysbaddaden was saying. Actually, this is what I was doing before I changed to just un-gitignoring each directory I don't want to gitignore. The thing is that it doesn't make it less of a hassle when referencing files from one language to the other: you would have to chose which language is the main one and, then, use require accordingly. Otherwise, if you extract common code/settings outside both languages, you're losing path conventions.

must is a strong word, I would say having both projects at the same level makes it easier to integrate both and share code. The things is that having lib for shards just prevent you to architecture your root directories in a way that I think would have made my use cases easier to work with.

Concretly, let's say you have a Rails application which already manage migrations, assets, settings like secret_key_base and deployment. Then, you decide to use Amber for the API part of your application which, from my point of view, is a perfect use case for Crystal for the task at hand. The thing is that you will want to share migrations, settings, deployment and maybe some other stuff, so having both projects in the same folder would be beneficial, but lib directory makes it less straightforward than it should be.

[RX14]

@patleb are you requiring the same files from ruby and crystal? That definitely won't work out well. Eventually you'll need to use some syntax which doesn't compile in either one of the languages (i.e. type restrictions) and you'll be stuck. You'll be happier in the long term if you give up on ruby <> crystal code sharing now.

Even if you're not, using the same folder for a rails and amber project really doesn't seem like it'll end well. Amber of rails could easily introduce other naming conflicts than lib. I strongly suggest separating the two projects cleanly, through at least an agreed DB schema or ideally a real API. I think you'll have a lot more problems then lib doing what you're trying to do.

[j8r]

Furthermore, having a modular monorepo (composed of repos) ease, a lot, the usability of your projects. It'll be easier to search in the commit history and others to use & contribute to your projects.

Even for you, to know which application depends on which library, each with its own version. This allows you to make changes more serenely without risking to impact all projects.

[patleb]

YAML files are shared. C/C++ extensions are shared. DB/Tasks/Deployment/Assets are managed Ruby side. Basically, bin, config, db, lib, public and tmp are shared directories (with lib causing problems). Ruby applicative code is in app (with some in lib) and Crystal applicative code is in src.

I don't think that you understand the concern I'm pointing here: lib is more often than not something unrelated to dependency management, so it conflicts with the usual roles for this directory and one consequence is that you need to put it in your gitignore because it's system related, not application related.

Regardless of what you think about how I should structure my applications, lib causes and will cause conflicts role-wise whenever you need to use the directory for applicative code. So, basically, it prevents you to use easily the directory if it's needed.

[patleb]

I'm not advocating to cram everything in the same repo, but I'm pointing out that if you want to use crystal within a project/library that uses lib, then crystal has an opinion on this specific directory role and you are discouraged by default to use this directory for applicative code.

[ysbaddaden]

The more I think about it, the more I agree that using a folder as generic as lib for artifacts to a project is maybe not a good idea. Using a well named, hidden, folder (such as .crystal/shards) sounds like a better alternative to me. This would free the lib folder for whatever usages an app may have (e.g. rake/capistrano tasks, C libraries, ...).

[RX14]

Changing away from lib is an annoying breaking change, the assumption is baked in in a lot of places. So far there's been one complaint, so I'd be cautious changing it.

[asterite]

@RX14 I don't think it's that terrible. We already changed it from libs to lib in the past. We did this by having both paths be used at one point, and letting shards start using lib instead of libs after that. I think shards would even rename that directory or ask you if it's OK to do that (not sure), but we could do that too. lib is mostly managed by shards.

[j8r]

Anyway, not matter the directory name, this isn't a shards issue at the end - but a Crystal compiler one. I've searched a bit in both on shards and crystal (I've found a Require class), and it haven't seen a shared variable that refers to the library path.

What I mean is:

• A variable, like Crystal.library_path, is exposed by the compiler. This is the starting point.
• Then, shards or any package manager / vendoring tool / whatever use this common variable to put Crystal libraries to the directory.
• At compile time, when require is used, the compiler searches the library in Crystal.library_path.

This makes the change of the directory name much easier - that's only a String in one place to modify.

Therefore, could this discussion be continued in https://github.com/crystal-lang/crystal ?

[ysbaddaden]

@RX14 When I opened the issue I shrugged and thought "meh, who cares, that won't change", but then... I started wondering: why install external dependencies in such a prominent path? they're external to Crystal projects after all; we can bury them in a hidden directory, and nobody would care.

@j8r The compiler doesn't care about the library path. In fact there can be many of them (which allowed the change from libs to lib to be smooth). The wrapper shell script to the compiler presets the CRYSTAL_PATH environment variable to lib:/path/to/crystal/src and that's it.

[RX14]

I don't want to have the folder hidden, since that creates an awareness problem of how crystal resolves dependencies. Newcomers see lib and they know they have to execute their command in the same directory as lib. It's easier for them to see that lib is not there -- they've forgotten to run shards. So at a very minimum the folder needs to be visible, like lib and node_modules.

If you want to change it to crystal_lib or crystal_shards to avoid conflicts, then that just carries the breaking change cost. I think hiding the folder, hiding how CRYSTAL_PATH works, and making it "magic" will invoke a lot more costs as the community grows and mroe newcomers appear.

[ysbaddaden]

The error message when a require fails seems helpful enough:

Error in test/test_helper.cr:1: while requiring "minitest/autorun": can't find file 'minitest/autorun'

If you're trying to require a shard:

• Did you remember to run shards install?
• Did you make sure you're running the compiler in the same directory as your shard.yml?

I don't think the presence, absence or visibility of a lib or ugly node_modules-like folder has any impact.

[j8r]

@ysbaddaden ok that's not the compiler, I think it's worse to be in the crystal wrapper script. How this path can be used in other crystal programs? We have to look manually to the wrapper script an see what is it.

I don't say to remove CRYSTAL_PATH, just saying that there should be at least some standard paths by default if the variable isn't set, and being able to retrieve this paths through an API.

[patleb]

Not so sure either about hiding the shards folder, but I don't think I have much weight into this matter. Anyways, thanks for taking the time to re-evaluate the relevance of the shards folder name.

[patleb]

FYI, I came accross another side effect of sharing the lib folder: shards prune will clear anything that doesn't belong to the shard.yml.

[ysbaddaden]

Yet another reason to change the folder to be more specific.

[luislavena]

I've faced the same issue from OP in some Ruby/Rails projects. Ended placing my Crystal project within another directory (ext/my_project) so shards install generates it's own lib directory and then using symlinks to deal with shared files that the Crystal project must use 😕.

As said by some, I would suggest a non-hidden, easily identifiable location to exclude (crystal_shards, vendor/crystal).

Cheers.

[dmitryrck]

My use cases are two different ones:

1. I keep things out of my app directory
2. I recently migrated an app from ruby to crystal

In use case (1) I do that because I don't want to backup third-party code (shards.lock is there for me to know what version I was using). My backup program only backs up my code.

I achieve that with _Rust lang and cargo_ with a file .cargo/config:

[build]
target-dir = "/dev/shm/rust-cargo/hello_world"

With _Ruby lang and bundle_ installing things with: bundle install --path /dev/shm/bundle-hello-world, and that command creates a file .bundle/config:

BUNDLE_PATH: "/dev/shm/bundle-hello-world"

I recently managed that with crystal using these commands:

$ SHARDS_INSTALL_PATH=/dev/shm/hello-world-crystal shards install
$ export CRYSTAL_PATH="/home/dmitry/.local/crystal/bin/../share/crystal/src:lib:/dev/shm/hello-world-crystal"
$ SHARDS_INSTALL_PATH=/dev/shm/hello-world-crystal shards build

But these changes are per project; I created a Makefile in order to avoid repeating that long commands.

My use case (2) is that I recently migrated a project from Ruby to Crystal.

My steps:

1. Rename lib to lib_ruby. But it was pretty hard to use git log, I always had to come to github to see my changes before that commit)
2. Start a Crystal project and let it use lib
3. Remove the ruby code

As you can see this is not a multi-language software.

My next problem is that I will migrate a Rails project (a lot of files in lib). I will probably use my ENV var approach this time.

I would suggest to use lib by default. This way it will be retro-compatible. (To avoid things like this: https://github.com/crystal-lang/shards/issues/258#issuecomment-455773696 )

But if the user creates a .crystal/config (suggested here https://github.com/crystal-lang/shards/issues/258#issuecomment-455270479 ) like this:

SHARDS_INSTALL_PATH: "/dev/shm/hello-world-crystal"

shards install and shards build use that path. It will remove the problem with lib.

[luislavena]

Hello @bcardiff and @asterite, apologies for at you, but perhaps will be nice to evaluate this issue on the work towards 1.0?

Thank you.

[bcardiff]

I'm on board on changing lib to crystal_shards.

As a migration I would propose to

• On the next release the compiler we can ensure that crystal_shards and lib are both in CRYSTAL_PATH. Priority to crystal_shards.

• The next version of shards could show a warning if lib exists but not the crystal_shards, so the user can be asked to move it or delete it. If both exists, then everythin is fine.

• On the next+1 release of the compile we can remove lib from CRYSTAL_PATH

• On the next+1 of shards we can remove the warning.

[straight-shoota]

I'd also support moving away from lib. The battle plan looks good so far, but might need to be spaced apart a little bit more (make the last steps +2 or +3). And maybe even delay step 2 so that next shards is still compatible with current compiler.

And please let's not commit to a new name right now. We can use crystal_shards as working title, but whether it's going to be that name specificially or a different one should get some more detailled consideration. Until now we have just been talking about rough ideas and several suggestions for the naming have been made. There are probably also other not yet mentioned options. This needs to be

We need to agree on moving forward with this, then we can talk about details like what's the new install path going to be.

[luislavena]

So far the options described have been:

• Hidden approach: .crystal/shards, option for .crystal/config in the future and leaving vendor/crystal for possible vendoring at some point
• Visible option: crystal_shards

[j8r]

Having the name crystal_shards for a generic directory to place Crystal modules sounds weird. Sure this directory will be mainly used by shards, but not only.

This is a lookup directory where the compiler looks for files, where we can also copy-paste to vendor libraries, it is not stuck to shards specifically.
It just appears to be a user of this directory, like us or other tools.

Real example: in this very shards project, make lib downloads molinillo to lib, without using shards at all.

One another consideration: should the directory be hidden or not (starts with a dot)?

I have no strong opinions, why not simply .crystal, or crystal_lib.

[waj]

Shards is how it was decided to be called the Crystal's modules or libraries. It's just sources plus some conventions. I didn't see anyone making reusable Crystal code with a different shape.

Also, crystal_shards is the directory managed by shards and it shouldn't be used for anything else. If you want to add sources for something that is not a shard you could place it anywhere else.

Molinillo is either downloaded with shards or curl as a fallback, just to bootstrap this project. But it's also a shard.

[straight-shoota]

Having the name crystal_shards for a generic directory to place Crystal modules sounds weird. Sure this directory will be mainly used by shards, but not only.

That's the thing: We're trying not to use a generic directory, but a directory specific for shards dependency manager. Shards expects to be in full control of the directory it manages, so it should have a dedicated location. For example shards prune deletes everything that's not currently listed as a dependency in shard.yml. So currently you effectively can't use the generic lib directory generically because it's occupied by shards.
We just want one directory that's specific to shards and only used for shard dependencies. Thus having shards in the name seems like a good idea.

You obviously can still use other directories (like lib) for making code available for require. CRYSTAL_PATH allows to use different locations.

[j8r]

Fair point @straight-shoota .

Looking up to directories inside .crystal sounds good. For example, inside .crystal/shards, .crystal/vendor or .crystal/whatever_i_want.
This is both safe and flexible.

[Sija]

Care to vote? Names gathered so far:

• 👍 crystal_shards
• 😄 .crystal/shards
• 🎉 vendor/crystal
• 🚀 vendor/shards

[waj]

vendor sounds like the third party code is embedded. Even Rails uses vendor/assets for example and its content is normally committed in the repository. So I actually think that Bundler did a poor decision choosing vendor/bundle as an option. It would lead to unintentional commits with shards sources.

[jhass]

crystal init already puts /lib into .gitignore, why wouldn't it do the same for whatever we choose, including /vendor/shards? Why are people not committing lib already, all Ruby gems commit it!

Not that I really care what it's named.

[waj]

Actually, is not that Bundler did a bad decision. The goal of vendor/bundle is pack all the sources required to run a Rails app in a single place. But we don't need that for Crystal because it's a compiled language.

[jhass]

Not quite true, that's what bundle package is for and that uses vendor/cache by default. Of course neither directory is supposed to be committed.

[Blacksmoke16]

FWIW composer uses vendor. However, has there been any thought on grouping the shards by owner? Currently if two people define a shard with the same name they would conflict as well.

[straight-shoota]

However, has there been any thought on grouping the shards by owner? Currently if two people define a shard with the same name they would conflict as well.

That conflict would also affect the namespaces in Crystal code. So I don't think avoiding name conflicts on the shard folder level would help anything. Besides there is currently no concept of owner for a shard. There's only a name (and maybe authors could be considered as a related property to owners).
That's definitely a different issue, though.

[patleb]

Please consider leaving already known directories out of the potential candidates: it would defeat the purpose of this issue. bin, lib, tmp, vendor, etc. are all common names used in a wide variety of languages/frameworks. I've suggested crystal_shards because the way shards works (well, how I saw it work...) is more similar to npm or yarn than bundler or cargo and node_modules would translate to crystal_shards in a crystal context.

[RX14]

I'd prefer not to bikeshed. crystal_shards is fine, it solves the purpose of making it not conflict.

[j8r]

Some company/individuals prefer to vendor dependencies, and/or using other means like git subree/submodules. This is something we have better to think before doing any change.
If we go with crystal_shards, what would be the vendoring directory? Another one, like crystal_vendor?
That's why I think having crystal/shards (and thus possibly crystal/vendor) instead of crystal_shards may be a good idea.

[asterite]

@j8r They can change CRYSTAL_PATH and use whatever directory they want.

Let's make things work for 99.99% of the cases. We don't need to cover every possible case.

Or, if a vendor directory is needed later, it can be added to the default CRYSTAL_PATH. It's unrelated to where shards are stored.

[j8r]

@asterite but the current status-quo already works for 99.99% use cases, what you said with CRYSTAL_PATH can already be applied.
Do you mean it is a non-issue then?

[asterite]

The problem with lib is that shards can remove its contents. With a vendored directory it's not a problem.

Sorry, I don't feel like discussing this anymore. This is a minor and trivial change. It's just the name of a directory. It becomes a name that's less likely to clash with anything else. I really don't see why so much discussion needs to happen around such a tiny thing.

[j8r]

This tiny thing will be a major breaking change - all projects will have to change their .gitignore(!).
Better thinking twice before doing any change.

[asterite]

I mentioned in the core team meeting that when running shards and a lib directory is found but not a crystal_shards, it will prompt the user and ask if they want the .gitignore to be fixed (remove lib, add crystal_shards, or just add crystal_shards).

It's really not a big deal.

[patleb]

@j8r, this discussion has come full circle. Please let me re-iterate on the goal of this issue: preventing the usage of a pre-existing or potential future reserved directory by shards. Vendoring isn't related to this issue... unless shards uses vendor to keep shards which could be a problem that this ticket is trying to solve/prevent.

@asterite, I'm with you on this and a bit confused: the problematic has already been discussed at length and, except for @Blacksmoke16 comment, nothing new or of substance has been added to the discussion since.

[rafaelfesi]

I'd prefer not to bikeshed. crystal_shards is fine, it solves the purpose of making it not conflict.

I agree with @RX14

Better to stay with crystal_shards and ends this discussion. Then, the initial problem of name clash is solved.

[dmitryrck]

Why not leave it ./lib/ and whomever is in that 1% can set CRYSTAL_LIB_PATH or (even better) configure .crystal/config?

It won't break anything working nowadays and it will solve the issue for those 1% cases.

[waj]

@dmitryrck not a bad option actually

[patleb]

@dmitryrck, if I may ask, where does this 1% comes from and why should this make the usage of lib a good choice by default? Also, how does this change affects you? What are the breaking changes that you are talking about? I don't get why there is so much push back against something that prevents future WTF (by name clashes), is quite simple and has already a pull request for it. I too have workarounds for this issue, but I'm still convinced that it improves shards. How does this do not improve shards?

[dmitryrck]

@dmitryrck, if I may ask, where does this 1% comes from and why should this make the usage of lib a good choice by default? Also, how does this change affects you? What are the breaking changes that you are talking about? I don't get why there is so much push back against something that prevents future WTF (by name clashes), is quite simple and has already a pull request for it. I too have workarounds for this issue, but I'm still convinced that it improves shards. How does this do not improve shards?

It is not my data/study/research, I am quoting here: https://github.com/crystal-lang/shards/issues/258#issuecomment-621410426 and https://github.com/crystal-lang/shards/issues/258#issuecomment-621412350 (in this issue)

I wrote a long post before in this very same issue: https://github.com/crystal-lang/shards/issues/258#issuecomment-525615778

Breaking change is if suddenly projects that rely on ./lib/ have to move everything to ./crystal_path/ (or any other name). There is a real world example in this comment (also in this issue): https://github.com/crystal-lang/shards/issues/258#issuecomment-620921257

[patleb]

Ok, yes, I read the thing about 99.9% (which I'm pretty sure is made up), but it doesn't have anything to do with this matter: you're usually not naming class or file names by commity or based on how many thinks it should be named differently, it's an architectural choice based on practicality and meanings.

Also, I read your post and, as I said, I do use a workaround for this issue as well, it works, but this is not ideal: good defaults makes enjoyable tools.

require will look into the external dependency path which, at the moment, is lib by default, but it's not something that you add to your code when requiring files: meaning require "lib/some_shard/class_name", instead of require "some_shard/class_name", so unless there is something I missed, the breaking changes would be very minimal if any.

About the real world example, please read the answer just after that, it'll explain why the example wasn't relevant. Although, the examples that I've wrote about earlier in this issue are real world examples and considering that Crystal is heavily inspired by Ruby and attracts a part from this crowd (including Rails), I don't see why it's beneficial to dismiss the incompatibility between Crystal and any other tool/framework that use the lib directory.

lib is reserverd/used by shards, which isn't a good choice and it's not a made up issue, the name clash is real. It has been discussed at length here. There is no project that could possibly rely on lib that would work in the long run: simply because shards uses lib, this is why I opened this issue. I must say though that it's possible that some tools might rely on Crystal using lib (I can think of maybe a plugin for a text editor or something like that), but it's generally a bad idea to rely on conventions for this kind of thing (considering also that CRYSTAL_LIB_PATH could be used). This is also why I think the associated pull request is good: now we would have access to a defined constant Shards::INSTALL_DIR which we didn't have before and these tools could rely on that.

So, I'll ask again, how does this affect you? and how does this do not improve shards?

[dmitryrck]

I will quote myself from here: https://github.com/crystal-lang/shards/issues/258#issuecomment-525615778

[…]
My use case (2) is that I recently migrated a project from Ruby to Crystal.

My steps:

1. Rename lib to lib_ruby. But it was pretty hard to use git log, I always had to come to github to see my changes before that commit)
2. Start a Crystal project and let it use lib
   […]

My other use case is: If I start a crystal project from scratch I don't care if it uses the ./lib/ or not, because I know crystal uses ./lib/ so I won't use it myself.

(Like I said before) It improves shards because it will stay compatible with all projects by today, it won't force people to "re download shards", while it will let devs change from ./lib/ to ./any-other-thing/ if they need.

[patleb]

For your quote, not sure what you're trying to convey from that: having Crystal not using lib would have been a benefit to you.

For the answer to the first question, you stated that it actually doesn't affect you if it's resolved, because you made the decision to not use lib... so why do you care?

For the answer to the second question: it's actually still compatible, it's not a breaking change, it's an inconvenience that you mentionned:

it won't force people to "re download shards"

You actually told what to do if you don't want to re-download (rename your directory lib to any-other-thing)... actually, it could be a good idea to automate this step in the associated pull request so after adding a line to the .gitignore, it would also rename the lib directory if you want. Maybe @waj could weight on this? Although, I'm not convinced that it would be worth it.

[patleb]

@dmitryrck, very mature, thanks for the thumbs down. I was making an effort to understand your point of view and pointing out were it wasn't making sense in the context of this issue, so it might help you understand why this issue is open and why I think it's important to get this fixed. I'm still insterested to know which real problem the associated pull request could cause.

[dmitryrck]

I am removing myself from this talk the guy clearly does not want to talk and understand someone else's point of view.

My final cents: cargo (rust) and bundle (ruby) let you configure where to download the dependencies.

Thanks everyone.

[patleb]

@dmitryrck, I think that you're confusing "understanding" with "agreeing", I understood very well your point of view, but I didn't agree with your conclusions/reasoning. Sorry if I you felt offended, but this was an argumentation which, by the look of it, got under your skin.

For Cargo and Bundler, this is not the same thing, they behave differently and, by default, the dependencies are outside the root path (which prevents this very issue to happen by default).

This issue is about the default path of shards being within the projet root path and having a conflicting name.

[straight-shoota]

@dmitryrck Your suggestion to change CRYSTAL_PATH if it clashes with other usage of lib is not a solution to the problems with shards exercising control of lib directory. Shards would still use that directory.
To change that it would require to make the shards install path configurable. But I don't think this would be a good idea because it can cause even more friction than any alternative.

[waj]

I just had a meeting with the core team and we decided that it's actually a good idea to have a directory (_crystal) where we could put stuff that normally doesn't go committed to the repository. Initially this will be used for shards (_crystal/shards) but we envision other uses in the future, like local configuration overrides, intermediate object or output files, etc.

I'll update #367 so we can proceed with the migration.

Why not .crystal

I think the build process must be as transparent as possible. Hiding the directory doesn't help. Also making it visible makes it easier to navigate shard source files.

[RX14]

I'd rather have kept the original "decision" of crystal_shards. This is a lot uglier, I'd prefer a snake_case or single-word name.

[patleb]

Not sure about the underscore as a prefix. It's ambiguous: it looks like you want the directory to be hidden, but it's not... or you want it to be private, but crystal in the directory name already suggests that this directory is used by crystal. I think that @j8r suggestion of crystal/shards might be a better option.

[asterite]

For example Elixir uses _build. I don't think that was a good name, but it's what they have and it seems to work well. And you don't want it hidden because you can see the code inside it, the repos, etc.

Is there any non-subjective opinion here regarding the name? What I mean is, all arguments seem to be "the name is ugly", "the directory looks like hidden but it's not". I'm trying to find an argument that's like "well, if you use that name, that will definitely clash with other directories because in language X and Y they use the name _crystal too". Or "we shouldn't use that name because it exceeds the filesystem limit".

Otherwise, you can do this exercise: imagine shards didn't exist and in 2 days it will be released. It's released, and it uses "_crystal/shards" as the name. The options of "lib" or "libs" or other options were never put in the table because this is the first release. Would you be upset? Is that directory name such a huge decision? Shards before molinillo would hang installing dependencies, but the discussion around that is shorter than this name change. So could we please try to accept one decision, even if it's not what everyone likes... because it's impossible that we can decide a name that will please everyone?

[patleb]

I don't think that your Elixir example is right though: _build is for compiled files.
The directory which is the most similar to what shards uses lib for would be deps.

Ambiguity isn't really a sujective opinion: I gave 2 meanings (hidden and private) that a prefixed underscore suggests and if it's private, then the underscore is redundant because of the directory name.

Regardless, you're right, _cryastal/shards isn't really a big deal, I'd much rather have this option than lib.

[j8r]

Thanks for the detailed explanation @waj , that what we need to move forward to keep, or put away, options.
I'm wondering, why the leading underscore before, _crystal/, instead of crystal/?

@asterite It may be inconvenient to have such a lenghty discussion, but this does not really hurt much since it is not a critical issue – more time can be taken to make a final choice.

[asterite]

@j8r Well, if we want to settle the discussion and release 1.0, I think we'll never release 1.0

[Sija]

My 2c about the _crystal name:

• _ prefix is confusing, reminds the auto-generated dirs for artefacts like _build, but then it's not
• . is a commonly used prefix for _system related_ files, see .github as an example (with template and other files, which are meant to be readable). Also, it's not like you're going to checkout this dir often, in many cases lib directory is explicitly ignored already (think docs, formatter and others) so there's already a precedence of making it _less_ visible
• It's uncommon/weird/ugly (subjectively offcorz)

[straight-shoota]

I'm wondering, why the leading underscore before, _crystal/, instead of crystal/?

I think it doesn't really makes much of a difference either way.

_ prefix is confusing, reminds the auto-generated dirs for artefacts like _build, but then it's not

I'd argue shard's usage is pretty similar to that.

[Sija]

I'd argue shard's usage is pretty similar to that.

@straight-shoota https://github.com/crystal-lang/shards/issues/258#issuecomment-625901995 👇

[...] but we envision other uses in the future, like local configuration overrides [...]

[straight-shoota]

We're looking at the current use case. There are some ideas what the _crystal directory might be useful for later. But that's totally unspecific and not relevant right now. If it's something that doesn't fit into that directory, it won't go there.

[RX14]

I'm sorry but naming a directory beginning with _ gives me quite a visceral reaction of disgust. I don't like bikeshedding but imagining seeing this directory all the time gives me quite an illogically angry reaction. I've been sitting at my computer trying to let it go for about 10 minutes and I can't. Either name it something nice - because it should be visible - or hide it with a ..

[asterite]

@RX14 What's your coding workflow?

When I code I use Sublime Text. I hide the navigation bar and jump to files by name. Sometimes I would display the navigation bar to create some files, or maybe I'll do it from the command line with cd, mkdir and/or touch.

I then do git status or tig status, add the files, commit them, etc.

So when are the times where you are going to see "_crystal" in the coding workflow?

[j8r]

What is the reason to make it more visible, by using _ instead of .?

• This directory is not commonly manipulated, we usually don't care much about it (except to add it first to .gitignore on new projects)
• A hidden directory can still be accessed, like .travis.yml, .gitignore.

I think it is perfect to hide a directory which we don't have to care, most of the time.

[RX14]

I spend a lot of time in the terminal. I'll see _crystal when i use ls or tab-completion.

I disliked a few of the suggestions before, but there were none which I couldn't accept. I'm sorry that I couldn't make the core team meeting yesterday, but I find it really hard to accept anything beginning with _.

[asterite]

What is the reason to make it more visible

I often go to the current "lib" directory to browse code. Even from the text editor. If it were hidden, it would be a bit more difficult to reach those files, or know that they exist. By making it visible, it's clear what's going on.

[j8r]

Ok, fair use case @asterite .
Some does not like _crystal/shards. Anyone have arguments against crystal/shards then?

[Blacksmoke16]

FWIW isn't that more of an editor thing? I'm sure pretty much every editor allows some way to filter what files/folders are displayed in the tree view.

Most of which I would assume show hidden by default.

[RX14]

I'd be fine with using .crystal, now I thought about it more.

[patleb]

I know that it might not be the usual use case, but when your app has multiple shared libraries, it's useful/clearer to have the root path visible (I'm talking about using path: /path/to/shared_shard in shard.yml when developing locally which would become a symlink in the shards folder).

For this specific use case, the shards folder isn't just a folder that you ignore in your gitignore and forget about it, but it's pretty much a living part of your project (even though it's only local and temporary).

I must say though that I still prefer having a hidden folder than having lib, so either way, I can work with your decision.

[asterite]

After checking a couple of editors (Sublime Text and VSCode), .crystal shows in both of them. So now I'm fine with using .crystal/shards for this.

[waj]

Ok, let's use .crystal/shards then. I still prefer a non-hidden directory, but it's true that editors will still show them and on the other hand it's easier to type.

[oprypin]

What's the significance of the shards part there? Whatever the colloquial name of Crystal libraries may be, this is still just the name of a specific package manager. Will be confusing when a different package manager comes out.

[oprypin]

I don't like the directory being hidden either. Source code that will ultimately go directly into your program should not be implicit in any way.
But it's not a big deal. I suppose dot-files don't mean anything anymore, I'll just try to find some way to always show them everywhere except my home directory.

[jhass]

I think we had a discussion about how to name crystal libraries and the result was "shard", shards then was actually named after that, not the other way around :)

One of the earlier comments I could find about that: https://github.com/crystal-lang/crystal/issues/220#issuecomment-72645807

[straight-shoota]

Will be confusing when a different package manager comes out.

Shards assumes full ownership of the directory where it installs dependencies - that's one of the reasons why we decided to move away from lib - so it won't play nice when another application was using the same generic path (like .crystal/libs). Even if you intended to use one dependency manger exclusively, it would be easy to mess things up unintentionally and then it would be really confusing.

A different package manger should use a different location. It can simply be added to CRYSTAL_PATH either to accompany .crystal/shards or replace it.

[j8r]

@oprypin the name is shards because it is handled by it.
We can imagine someone using .crystal/vendor for vendored dependencies, and why not .crystal/cache (even though don't know the use case of placing the cache there).

Hidden files/folder are generally for special meanings (.gitignore), or when it is not handled by the user directly (~/.cache), or sometimes both (~/.config).

Here, it just tells it is not a regular directory in the project.

[oprypin]

No I don't buy that at all. You say the path is "handled by it" as if it's done in a format specific to Shards, but in reality the format has to be specific to what Crystal knows. When someone develops another package manager, will they willingly choose to require users to override CRYSTAL_PATH whenever using it? Probably not. Instead they can use the same destination and easily populate it in the exact same way, because thankfully this format is not complicated at all, and is about the source code, not any config. Or will any new package manager have to be approved by the core team and proliferate the hardcoded paths?
I sometimes like to not rely on Shards as it's not very portable or extensible. And again, thankfully I can do that very easily (with fewer lines than it takes to configure Shards!) - really just clone any repo into lib/ (or now, sadly, .crystal/shards)

[straight-shoota]

but in reality the format has to be specific to what Crystal knows

And indeed it is. So you can definitely use .crystal/shards with a different dependency manager. I'm just arguing that it might not be a great idea when it could easily be mixed with shards being run on the same project. It might as well work pretty seamlessly, I don't know because there's no practical example.

Or will any new package manager have to be approved by the core team and proliferate the hardcoded paths?

Definitely not. CRYSTAL_PATH can be set as appropriate for every project.

Shards as it's not very portable or extensible.

Different topic, but what do you mean by that? Shards should be able to run everywhere the compiler runs. What extensibility do you miss?

[j8r]

@oprypin .crystal/shards should never be modified intentionally by any other means than shards (unless you want to test someting temporarily).

Perhaps .crystal/vendor could be added by default to CRYSTAL_PATH? It may be too premature.

This way, each directory owner is very clearly defined. No more mashups:

• shards is managed by shards, and only
• vendor is managed by third party

[patleb]

thx @bcardiff

[pseudonym]

I am new to this language, but have more experience than the average bear when it comes to build logic. I see a lot of bikeshedding in this issue on the name of the directory, but nobody seems to be talking about the semantics of CRYSTAL_PATH and the unnecessary tight coupling between shards and the compiler.

The typical interpreter/compiler toolchain gets include paths from 3 places: command-line arguments, a PATH-like environment variable, and system-wide configuration (which may be somewhere in /etc or just hard-coded into the binary). Lookups are done left-to-right within these categories, and in the specified order between categories. Ruby and Perl work exactly like this, and GCC is very similar.

In contrast, Crystal's default search path includes ./lib, and CRYSTAL_PATH overrides the built-in search path instead of prepending to it. The compiler shouldn't have to care about where shards is putting things, because the path should be passed to it instead of there being an implicit dependency between the two pieces of software.

[oprypin]

Not a sound argument. Any software (or not even software) can put something into lib/. What is the dependency that you speak of when I can create the lib directory myself, clone a repo into it and be done?

[oprypin]

but nobody seems to be talking about

I'm pretty sure most, or all, of your points had been brought up. Seems like the point about additive CRYSTAL_PATH is not in this thread, but there was another one. https://github.com/crystal-lang/crystal/pull/9280#issuecomment-626965657
We should indeed revive that particular point.

[pseudonym]

The compiler thinks ./lib is special. Shards thinks ./lib is special. There's a constant that needs to be kept in sync between the codebases. That's a dependency.

That comment provides a way to fix the problem of finding where the stdlib is if you want to use CRYSTAL_PATH, but is still stuck on having this tight coupling. Having behavior that differs from every other language runtime (by having a "blessed" local path to search for requires) means you're operating on the Principle of Maximum Astonishment. That's a bold strategy, Cotton.

[straight-shoota]

@pseudonym I think this discussion here about the name of the directory where shards puts libraries should be considered finished.

I don't fully understand what you're trying to argue for, i.e. what you would see as desired behaviour. I don't even know if there's a practical problem when shards and crystal agree on a path to lookup libraries.
If you want to continue this discussion, I'd suggest to elaborate on that in a new issue. Thanks.

[bcardiff]

To avoid the dependency/hack between shards and crystal a draft idea discussed with @waj was to have a project configuration file where that information could exist. The tools will need to share the schema or the way to access the information. This project could also help to deal with overrides of link annotations.