Shards: Usecase: installing a binary into path

[waterlink]

Imagine, there is some tool called my_super_tool and it is a Crystal shard. I add it as a dependency to the shard.yml and run shards install. my_super_tool is actually a library, that is designed to be called through a compiled binary as a CLI application.

Possible options I see:

• Conventional ./bin/ path in the current project directory. Probably can be already done through a custom postinstall script.
• New command shards exec BINARY [ARGS...] and an option binary: my_super_tool and binary_source: ./src/binary.cr for the shard.yml of the library in question. Ultimately, shards activate command, that modifies current PATH to include all binaries for the current project in the current shell, so that shards exec part can be dropped and binaries can be used directly.

WDYT?

/cc @ysbaddaden @asterite

[ysbaddaden]

I intentionally left executables aside for Shards, because we're dealing with library dependencies in order to build software, but not distributing software.

What is a real use case you have? I need concrete proof that we should have a standard way of handling binaries.

[waterlink]

My usecase is a development dependency, that has primary interface as a CLI application. Concretely, this: https://github.com/waterlink/expand.cr

There are more usecases I can think of:

• CLI tool for generation of snippets of code
• A tool, that watches file changes and does something (for example runs tests)
• A refactoring tool, that given a couple of arguments through the CLI, can do simple refactorings (rename, extract, etc.)

Notice, how my particular case at hand, and 1 or 2 cases from examples, probably will use compiler intrinsics, which would mean, creating a library, to be used directly by users, and compiling it each time they use (with crystal src/the_tool.cr -- args go here) will cause 1-3 minutes of compilation each time (since it embeds full or partial compiler).

[ysbaddaden]

I thought about some use cases, for example pre-compiling binaries to be used in macros, or just when installing crystal_lib. I'm not sure they should be installed in a centralized bin repository thought, but maybe they could?

Maybe having a CRYSTAL_PROJECT_BINDIR environment variable be useful? And maybe other variables, too? This way the postinstall hook could access them.

Maybe having an executables entry in shard.yml would be nice, so we can know about the executables just by reading it?

[asterite]

Another example is micrate, where one would like to install this tool for a project to easily run migrations. Right now one has to install it globally, but maybe installing it inside the project would be nice.

[ysbaddaden]

Already promoted to feature! I propose to implement it like this:

1. have an executables entry in shard.yml (array of filenames);
2. the install and update command would create a shell binstub (with useful environment variables) copy each executable from libs/foo/bin/<executable> as bin/<executable> (setting the executable bit, if needed) _after_ the postinstall script;
3. the prune command would remove links/binstubs from bin for pruned shards;
4. a new binstubs command could recreate all the binstubs.

[jhass]

Not sure what the binstub would do here? Why not build/copy the binary directly there?

[jwaldrip]

@ysbaddaden if you were to have executables I would say it should be a dictionary instead where:

executables:
  foo: ./commands/foo.cr
  bar: ./commands/bar.cr

In addition, I think there should be a shards bin [command] that would execute a binary in the libs/foo/bin directory.

[ysbaddaden]

No. An executable is merely an executable (e.g. a shell script). If they must be compiled, this it out of scope. It can be achieved with a postinstall script, or be executed automatically if there are build targets (#110).

Though this increases the complexity, since to build the targets of a dependdency, then its dependencies must have been installed already (we don't have any order for now), and be made accessible to the dependency, which is tricky, because the dependency becomes an actual project...

[ysbaddaden]

Isn't bin/tool simpler than shards bin tool? Let's avoid useless commands.

[RX14]

So a shard maintainer who would like to provide a tool would update/add a postinstall script to create a bin/foo executable, and then just add

executables:
- foo

to their shard's shard.yml and anyone who depends on that lib would get bin/foo? Sounds good. What about transitive dependencies?

[jwaldrip]

@ysbaddaden I did not see the stub reference. Got it.

[ysbaddaden]

@jhass you're right, the binstubs make little sense. I dropped that.

[bcardiff]

I have one scenario I am not sure is covered. Basically is when a shard wants to allow some template expansion in the host app. Let's say like rails g controller. I think crystal could be used as a scripting language for this scenarios.

Maybe there is a simplest way to do this and I am all ears/eyes.

But this is what I would try to achieve now.
In a shard with a template expansion feature I would expect something like this layout:

myshard $ tree .
.
├── bin
│   └── generate
├── shard.yml
└── src
    ├── generator.cr
    └── template.ecr

The bin/generate could be a #!/bin/sh that compiles & execute src/generator.cr passing the arguments. The src/generator.cr would use src/template.ecr.

Once the shard is installed on the hosted app I would expect that the app bin/generate (or bin/myshard/generate) would be a symlink or something that will invoke the shardbin/generate. But it would be great if there is a way for the command to know about the app context. So for example it could read configuration files or even allow overriding the template.ecr thanks to some conventions.

The app structure could look something like this:

myapp $ tree .
.
├── bin
│   └── generate
├── lib
│   └── myshard
│       ├── bin
│       │   └── generate
│       ├── shard.yml
│       └── src
│           ├── generator.cr
│           └── template.ecr
├── shard.yml
└── src
    └── template.ecr

I don't have a strong opinion on how to achieve this, but I think is a scenario that will appear on multiple occasions.

So,
Would you agree that this scenario is useful?
If so, could we find a way to supported with post_install or the next future features of shards?

[ysbaddaden]

I'll experiment a bit (or you can in the branch, putting binaries in src/bin for now).

Some ideas out if my mind:

The executable can be a crystal source code with a shebang:

#! /usr/bin/env crystal
require "foo/generator"
# ...

The app context is the parent of the current executable path. We need to add support for that in Crystal (core or std?). There is already a PR that must be expanded to support FreeBSD.

The lib context can be assumed to be ../lib/foo relative to the executable path. I think executables should embed as much as possible, but it's OK for pure development executables that aren't expected to be run.

It would be nice to have non compiled templates, in addition to ERB. We could have custom templates being rendered at runtime. Maybe I'll port Slim Liquid templates someday...

[RX14]

@ysbaddaden A compile-time slim exists: https://github.com/jeromegn/slang. I'd much prefer something jade-like with extending, inheritance, includes etc. built in.

[ysbaddaden]

I prefer the Slim Liquid syntax, and interpreted at runtime would be a great for speed and customisation. Having worked with ECR to build an app, it was _very_ frustrating to wait several seconds for the app to recompile to see a template change...

[RX14]

@ysbaddaden jade is 90% slim syntax, and in my opinion the changes actually seem rather nice and useful. I also think it's possible to generate templates fast using precompiled binary generators. A source-to-source compiler is not more complex than an interpreter, so I believe that they will have the same performance. There's probably quite a bit of difference due to the extra code in the compiler itself, but I haven't benchmarked anything really optimised.

[ysbaddaden]

Sorry, I meant liquid templates, not Slim ...

Anyway, we're drifting away from the topic.

[bcardiff]

I spent a couple of hours today making a development tool in crystal I am eager to use in shards. It is a CLI tool. I manage to use shards build and other bash commands in postinstall to shameless create a link of the compiled CLI program in the host shard bin directory.

The tool is https://github.com/bcardiff/ghshard and the sample shard is https://github.com/bcardiff/crystal-lorem

The relevant parts is:

targets:
  ghshard:
    main: src/cli.cr

scripts:
  postinstall: |
    shards build
    mkdir -p ../../bin
    rm -f $(pwd)/../../bin/ghshard
    ln -s $(pwd)/bin/ghshard $(pwd)/../../bin/ghshard

It would be great if something like the following will do the work IMO:

targets:
  ghshard:
    main: src/cli.cr

scripts:
  postinstall: shards build

executables:
  - ghshard

I guess this is possible since ghshard has no dependency. Otherwise the CRYSTAL_LIB would need to be hacked a little bit to include the parent lib directory where the postinstall is performed.

[RX14]

I should think that that usecase would be covered by this
https://github.com/crystal-lang/shards/pull/126 PR once merged. I’m
sure that |CRYSTAL_PATH| can be properly set when running postinstall
scripts too.

[ysbaddaden]

@bcardiff what you want is both #126 (executables) then #124 (dependencies)

[bcardiff]

Yes, definitely. My bad I didn't track the issues/pr.
I wanted to share the little yet success here and check if we are aiming to this.

I should start contributing in shards...

[chyzwar]

I am excited about his feature. Combined with direnv this will be really nice.

[kazzkiq]

Any news about this feature?

It would be effortless to install CLI tools and propagate Crystal projects,
pretty much like npm and go get does.

[kevinelliott]

I'm curious about this too. Any thoughts on ETA?

[RX14]

There's no ETA, and nobody working on this. If you want to work on it, submit a PR :)

[jwaldrip]

@rx14 you just gave me something to do :-)
On Tue, Feb 6, 2018 at 3:39 PM Chris Hobbs [e-mail] wrote:

There's no ETA, and nobody working on this. If you want to work on it,
submit a PR :)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/crystal-lang/shards/issues/95#issuecomment-363589767,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAConAK4QXCB2YjIal3Np8UMAGKnZNzLks5tSNSqgaJpZM4H0nmJ
.

[ysbaddaden]

Pending. #126

[RX14]

Oh, my bad. I forgot that there was a PR pending. It's probably better to iterate on that.