Shards: Turn shards into a project manager

[sanks64]

Currently Shards is a dependency manager rather than a project manager and I see several disadvantages in this. Here are a few of my thoughts:

• If we want to have things like #298 and #202 then it's just not dependency management anymore.
• crystal docs and crystal spec commonly used with projects rather with single files and that would make sense to combine them into one utility (if it's hard to untie them from the compiler then aliases is a temporary solution that would work good).
• This would be friendlier to beginners. We have crystal spec, crystal docs, but at the same time we have shards update, possible shards run and it's a bit confusing.
• In case of alternative Crystal implementations (JCrystal, Crystal for microcontrollers and everything that the community will make) it would be possible to use Shards together with them.

And a list of commands that would look good with this change:

• shards release - compiles code in the release mode (an alias for crystal build ./src/filename.cr --release)
• shards init - replacement for crystal init (as suggested in https://github.com/crystal-lang/crystal/issues/8844)
• #298 and #202 as mentioned earlier
• Maybe something else

[straight-shoota]

I don't think shards should evolve like that.

The way I see it, shards should focus on a single feature in good old unix tradition. That is: managing dependencies. #202 is definitely in scope for that. It's just allows custom extensions.
Shards serves as the major backbone carrying the Crystal ecosystem, but it's not strictly tied to Crystal as it doesn't care what language the dependencies are implemented in. So it's actually a generic language-agnostic tool, which is really awesome!

Arguably, shards run (#157) is already a bit more than just dependency management. So it might be outside the core feature, but not by far. The same goes for shards build which is the only reference to crystal program in shards. Maybe these commands shouldn't exist in an ideal implementation and we might have to revisit them to focus on separation of concerns. I suppose they're mostly there for convenience and essentially just more expressive shortcuts for tasks you could otherwise solve with a simple shell command. But then you could already just use that shell command...

crystal docs and crystal spec are IMO perfectly placed commands. Whether they should actually be included in the compiler or just an interface for external programs, is a different topic.
They provide features that are specific to crystal language, so they should be run using the crystal tool.

However, there is something to avoiding the use of different tools for different aspects of managing a (Crystal) project. I just don't think shards can or should be the right tool for that. It rather provides the plumbing, you just need to install the porcelain. There are many tools available for that. I'm really happy with using make. It allows defining specific and easily customizable recipies for the tasks that are important to each project. Maybe my Makefile template can be an inspiration for that.

[sanks64]

The way I see it, shards should focus on a single feature in good old unix tradition.

Do you think it would break the Unix Philosophy if we'd had #202? IMO, this fits very well. Shards still would be a pretty simple tool, just a little bigger one.

However, there is something to avoiding the use of different tools for different aspects of managing a (Crystal) project. I just don't think shards can or should be the right tool for that.

I can agree that maybe Shards isn't the right thing for such functionality, but on other hand: we don't have any more built-in tools in Crystal that would this. Not like it's a good idea to make a new one too...

[straight-shoota]

Yes, I don't think #202 would break single responsibility, I already mentioned that:

#202 is definitely in scope for that.

---

we don't have any more built-in tools in Crystal that would this. Not like it's a good idea to make a new one too...

So it's a good thing, there are tons of existing tools out there =)

[sanks64]

So it's a good thing, there are tons of existing tools out there =)

Yes, but wouldn't it be also good to have one that ships with Crystal?

In Elixir, for example, Mix is an indispensable (in terms of it's quality) tool used by the whole community.

For example:

• mix archive.install hex phx_new 1.4.16 - to install a Phoenix archive (extension).
• mix phx.new hello - to create a template for Phoenix application. It can be any command depending on the extension.

In the future, this would help to avoid a scenario where everyone uses different tools to manage their projects (although it's not completely avoidable).

Edit: I know that you already said there is should be something to avoid different tools. Just wanted to be sure and show how I see it.

[RX14]

I'd like to hear more about specific usecases and specific features you think would address them. shards release vs shards build --release is really not worth the extra effort. A lot of these are sane suggestions, but it'd be nice to know what struggles you have with the toolchain as it exists right now.

"Generic" issues like this tend to not fare so well, and are perhaps better as forum threads. Specific suggestions and usecases are a lot easier to discuss on an issue tracker.

[sanks64]

What struggles me is that we have tools that are used together, but located in different utilities. It's just not convenient, but let's imagine a beginner that comes to Crystal. He/she runs shards build or shards run and everything works, but then he/she decides to use specs and... specs just not there. It's also easier to navigate in the terminal using tab with one utility than with two.

Also if everything remains as it is now, but we will need to add new tools then there are always the same questions: where to put them? In the compiler? Or maybe in Shards? It's gonna go crazy as soon as there are too many tools.

Commands I see fit (except shards release):

• shards extension - (as an addition to #202) list installed extensions and next commands
  
  
  
  • shards extension install [target] - add shards-[target] as a part of shards.
  
  
  • shards extension uninstall [target] - remove shards-[target] from shards.
  
  
• shards format - formatter is just a tool that forces code style and it shouldn't be a part of the compiler. Someone can make a better one (it may even not follow official code style).
• shards init - maybe it's a good thing that Shards isn't strictly tied to Crystal, but every language has it's own ecosystem and I don't think that we need to pay attention to this first.

[RX14]

crystal has anything related to source code, or parsing (formatter, etc)
shards has anything using data from shard.yml (build targets, etc)

I don't think these will change.

shards init vs crystal init is the one place where I wouldn't be opposed to changing things around. Removing crystal init and making it part of shards with templates and stuff wouldn't be too bad. The problem is that it doesnt seem quite in scope for shards or crystal.

Making shards manage global state with shards extension install/uninstall doesn't appeal to me either, though I guess it's better than installing to PATH.

[z64]

I don't feel strongly about any of the others, but what has always felt awkward to me is the placement of today's crystal init's features in the compiler. It feels odd that shards cannot help more with creating a shard beyond creating a default shard.yml, and I must instead invoke the compiler for this 99% of the time.(*)

A justification of crystal init being convenient for newcomers being where it is isn't particularly compelling. The overwhelming majority are going to use shards eventually, if not immediately, and I don't think the kind of person who is going to forgo shards would also still use crystal init.

However, I guess crystal spec would suffer from this, as it is going to check for spec/ by default.. so I don't know how that fits into this for those that forgo shards/init. Especially since require "spec"/"minitest"/etc is usually what enables the testing harness. crystal spec is effectively just a crystal run spec/... shortcut, you would have to mkdir spec yourself if you wanted to make use of it..

So somehow, I would love to see this feature more cleanly allocated.

---

* - I realize shards is, at its sum, a generic dependency tool. shard.yml is all that is strictly needed to create a shard. It still feels odd to not provide more optional things that specifically benefit Crystal shards, when on the tin it says "Dependency manager for the Crystal language", and the shard.yml spec has a crystal key filled by default, and has become at least culturally integral to the Crystal ecosystem.

[straight-shoota]

I don't see why crystal init wouldn't fit in the compiler. It's the tool that is recommended for creating a new crystal project. And that includes among other things the initialization of a shard.yml file because we see shards as the dependency manager for Crystal projects.

shards format - formatter is just a tool that forces code style and it shouldn't be a part of the compiler.

Strongly disagree on that one, too. Formatting code has 0% relation to managing dependencies. Having a small common denominator on code formatting is important so it is considered an integral component for developing in Crystal. It's also tightly coupled with the compiler implementation. For both reasons it seems quite natural to have it bundled.

Someone can make a better one (it may even not follow official code style).

Sure, you can. crystal format sets a baseline, yet leaves much open for particular styles. So you can even implement a formatter for your custom style while being 100% compatible with crystal format.

[z64]

I don't see why crystal init wouldn't fit in the compiler. It's the tool that is recommended for creating a new crystal project.

Yeah, this is backwards logic to me. A compiler should compile code, defining the language, etc. - why would it need to do anything else? If we're pulling unix philosophy here..

The language does not care at all how you structure your project - itself having no concept of "projects" to begin with. So I don't think it belongs anywhere in the compiler - it's an orthogonal concern entirely in the user's lap. In contrast to the formatting and testing of the code, which at least has an intersection, directly benefiting from living inside the compiler/stdlib.

shards deals with packaging code.. while it also does not care how you structure your project, you at least inform it of your project's structure; targets, post-install scripts, etc. Therein, templates for shards would compliment this, without interfering with uses for non-crystal distributions.

[z64]

@RX14 Oh, missed your comment.. Yeah I agree shard templates aren't a perfect place inside shards either, the line is blurred quite a bit. A lot of other languages also entirely leave this up to community tools. Python I think? Go?

But, there's still a lot of popular dep managers that do offer templates; out of ones I've used, gem new / bundle gem, mix, cargo new, nimble init.. probably some more I can't think of. I am inclined to think this (albeit shallow) familiarity is on average better for newcomers than crystal init.

While it was nice for a short time when I first started, after 4 years the only impression I'm left with is wondering why I'm using the compiler to do this..

[j8r]

Agree with @z64. crystal init is creating a shards project with a shard.yml. It makes sense to move it to shards then.

What is a Crystal project? The common denominator is a .cr file, everything else is optional, conventions or opinions.

The compiler should only care about actual code, not the organization around.

[asterite]

Just my 2 cents: originally crystal came with a dependency manager. You would call crystal deps instead of shards. But our dependency manager was not good so we eventually used shards and delegated crystal deps to it. But that was confusing because you would get errors mentioning "shards"... what was that? So we eventually decided to use shards as a separate tool.

shards is not just some tool that you can use Crystal with. It's a fundamental building stone to create applications. That's why shards.yml is generated by crystal init: because it's an essential part of Crystal.

I think the current situation is fine. In Ruby there's ruby, rspec, rdoc, etc. In Go it's more uniform, but I'm sure there are other tools outside of the Go compiler. No problem with that.

[j8r]

I disagree @asterite , I don't see what shards brings for a proprietary applications with no dependencies, or vendored ones.
Sure there are some minor shortcuts like shards build - can be done with make.

Therefore, shards is not essential for writing Crysal apps, like crystal spec for running tests (minitest).

[straight-shoota]

@j8r A .cr file doesn't make Crystal project. If you have just a single crystal file in a project with thousands of XYZ files, I'd call that an XYZ project. In the other direction, I wouldn't necessarily say even a single .cr file is needed for being a Crystal project. For example when you would have a meta-shard which just pulls in other shards as dependencies, I'd still call that a Crystal project.

Obviously these examples are extreme and I'm not saying they're typical or even likely cases. But it demonstrates the point that the definition on what is considered a Crystal project might be quite wide and people have different opinions on this.

shards may not be essential for writing Crystal code. But neither is spec nor most other compiler tools; even the compiler itself is only de-facto essential because there happens to be no other Crystal implementation around.

It is common understanding that shards, spec and other features are important components of using Crystal.

[RX14]

I think the current situation is fine, for what it's worth. I'm just saying I wouldn't directly oppose moving crystal init if someone else pushed it and could convince the rest of the core team.

[asterite]

initialize a crystal project

From a beginner perspective, or for anything actually (I use crystal init all the time), I see nothing wrong with crystal init. shards init just initializes shard.yml. I wouldn't change that...

[z64]

shards init just initializes shard.yml. I wouldn't change that...

I wouldn't change that either - that's a good default for what shards is. Adding a specific template type argument to shards init would have been my suggestion, which would be visually representative of adding more files than just shard.yml, and "backwards compatible".

[sanks64]

Adding a specific template type argument to shards init would have been my suggestion

Looks confusing. What types would you want to add?

[z64]

It would be

• shards init: creates shard.yml
• shards init app [name]: same as crystal init app [name]
• shards init lib [name]: same as crystal init lib [name]

[sanks64]

I think I'm more in favor of keeping crystal init in the compiler unless we plan to also add at least shards extension.

On the other hand, if we only add shards extension and not everything else, it will be even more confusing.

[j8r]

Is there any difference between a shards project and crystal project @asterite ? What crystal init does for me is closer of the first one than the later one.
Having both commands is definitely confusing.

[asterite]

Is there any difference between a shards project and crystal project

What's a shards project?

[j8r]

What's a shards project?

A project using a shards.yml.