Amber: Allow support for project-specific Amber binaries

[robacarp]

Currently a fair amount of the Amber workflow is launched from the system-wide installed amber command, which makes it difficult to have more than one Amber binary installed on a given system.

As an Amber implementer, this makes it difficult to maintain more than one Amber site. The Amber project is releasing quite frequently, so it's easy for projects to get stale.

As Amber maintainers, it means we have a bigger overhead for releasing. Each release must be accompanied by a release to any of N package managers Amber decides to publish to. Currently that's just brew and aur, but publishing an apt or yum package is probably not too far off.

Shards is fundamentally different from RubyGems in that it installs dependencies project-local instead of system wide. The Amber workflow as it stands does not support project-local specification of an Amber version, which breaks the paradigm that Shards is presenting.

---

I'd like to propose a solution to these problems: generate a project with a bin/ folder which has scripts for finding and calling the correct Amber binary, and set a precedent to execute all repository commands from that bin folder: bin/amber migrate up or even simpler: bin/migrate up.

Consider this simple bash script, which will find and compile the appropriate version of amber in the lib directory, and use that binary to run migrations:

# file: u+x bin/amber

# Check to make sure the lib amber is compiled
# This should only have to run once per project per update of Amber
if [[ ! -x ../lib/amber/bin/amber ]]; then
  # build the amber binary once
  pushd ../lib/amber/bin/amber
  shards build amber
  popd
fi

# execute whatever command against that binary
../lib/amber/bin/amber "$@"

Now, from a project directory, it's easy to run migrations using the locally specified Amber version: bin/amber migrate up. Helper scripts to make that command shorter would be basically one line and might make the workflow even shorter bin/migrate up.

---

About the Amber CLI
Currently Amber ships a compiled version of the entire binary to brew and aur, and it must re-ship that binary every time Amber releases a new version. But if generate command for a new project changed a little, that would almost never need to happen. Consider replacing the amber new command with this workflow:

• amber new fancy_blog command is run by user
• system binary: query github or amberframework.org for current release version of Amber
• system binary: create a directory, add basic shard.yml and run shards install
• system binary: compile local amber: cd fancy_blog/lib/amber && shards build amber
• system binary: pass off app generation to versioned amber: ./fancy_blog/lib/amber/bin/amber --generate-app
• locally compiled amber binary: populate the rest of a generated app with the latest and greatest Amber upgrades.

Because the Amber CLI is simply a shortcut for bootstrapping an Amber project, it doesn't need to be updated frequently. Releases are easier No more brew issues targeting the wrong sha, etc etc.

---

Benefits

• Amber developers and implementers can easily have more than one version of Amber installed, and each Amber project will use the right version of Amber. This is difficult for my personal workflow, and was brought up in Gitter this morning. Hat tip to @zapnap and @damianham.
• Amber CLI commands can be shorter: bin/migrate up, bin/watch etc.
• CLI plugins are basically free. If you build a plugin to generate a blog, it can be triggered by simply: bin/fancy_blog generate. Installing the scirpt into bin/ might be accomplished with a shard post_install command. No maintenance overhead for a CLI plugin api.
• Amber project maintainers can easily override and extend these commands when needed without deviating from the canonical interface. For example, instructing an haproxy load balancer to put up a splash page while migrations are run.
• Opens the possibility for automated or assisted upgrading of older Amber applications like that which is proposed in #489

Drawbacks

• Not able to run Amber commands from anywhere within an Amber project. I don't think this is much of an issue, but Rails has this convenience and it is developer friendly.
• I'm sure there are others.

As always, comments welcome 🍐

cc #573

[robacarp]

This bash script works:

#!/usr/bin/env bash

set -euo pipefail
IFS=$'\n\t'

amber_shard="./lib/amber"
amber_bin="$amber_shard/bin/amber"

# Check to make sure the amber installed by shards is compiled.
# This should only have to run once per project per update of Amber.
if [[ ! -x "$amber_bin" ]]; then
  echo "Compiling Amber. This should only happen once..."
  pushd "$amber_shard" > /dev/null
  shards build amber
  if [[ $? -ne 0 ]]; then
    echo "Building local copy of Amber failed!"
    echo
    exit 1
  fi
  popd > /dev/null
fi

# execute whatever command against that binary
"$amber_bin" "$@"

[faustinoaq]

@robacarp nice 👍 I tested it and worked as expected:

➜  forum nano bin/amber
➜  forum chmod u+x bin/amber 
➜  forum ./bin/amber -v
Compiling Amber. This should only happen once...
Fetching https://github.com/luislavena/radix.git
Fetching https://github.com/jeromegn/kilt.git
Fetching https://github.com/jeromegn/slang.git
Fetching https://github.com/stefanwille/crystal-redis.git
Fetching https://github.com/amberframework/cli.git
Fetching https://github.com/mosop/optarg.git
Fetching https://github.com/mosop/callback.git
Fetching https://github.com/mosop/string_inflection.git
Fetching https://github.com/amberframework/teeplate.git
Fetching https://github.com/juanedi/micrate.git
Fetching https://github.com/crystal-lang/crystal-db.git
Fetching https://github.com/jwaldrip/shell-table.cr.git
Fetching https://github.com/askn/spinner.git
Fetching https://github.com/will/crystal-pg.git
Fetching https://github.com/crystal-lang/crystal-mysql.git
Fetching https://github.com/crystal-lang/crystal-sqlite3.git
Fetching https://github.com/veelenga/ameba.git
Installing radix (0.3.8)
Installing kilt (0.4.0)
Installing slang (1.7.1 at master)
Installing redis (1.9.0)
Installing cli (0.7.0)
Installing optarg (0.5.8)
Installing callback (0.6.3)
Installing string_inflection (0.2.1)
Installing teeplate (0.5.0)
Installing micrate (0.3.0)
Installing db (0.5.0)
Installing shell-table (0.9.2)
Installing spinner (0.1.1)
Installing pg (0.14.1)
Installing mysql (0.4.0)
Installing sqlite3 (0.9.0)
Installing ameba (0.4.2)
Postinstall make bin
Building: amber
Amber CLI (amberframework.org) - v0.6.4
➜  forum ./bin/amber -v
Amber CLI (amberframework.org) - v0.6.4

BTW, we should use shards build amber --production instead to avoid ameba dependency.

[drujensen]

@robacarp I really like this idea the more I think about it. I wonder if we can do something like this?:

crystal init app blog
# add amber shard
crystal deps
bin/amber new -d mysql -t ecr -m crectco --deps
bin/amber watch

[faustinoaq]

@drujensen That workflow would be awesome ✨

[faustinoaq]

@drujensen I tried it and kinda works, we need to tweak some things though

See my gist 👉 Trying to create an amber project without a preinstalled CLI

[eliasjpr]

@faustinoaq I think it would be better to create gists for these long outputs instead of pasting inline since it distracts a lot from the conversation happening in this thread

[robacarp]

I've been using these scripts for several days now and I think I love it.

> ls bin
drwxr-xr-x  11 robert  staff      352 Jan 29 20:46 .
drwxr-xr-x  31 robert  staff      992 Jan 29 20:46 ..
-rwxr--r--   1 robert  staff      581 Jan 25 16:58 amber
-rwxr--r--   1 robert  staff       81 Jan 29 20:46 generate
-rwxr--r--   1 robert  staff       80 Jan 29 20:46 database

> cat bin/generate
#!/usr/bin/env bash

set -euo pipefail
IFS=$'\n\t'

exec bin/amber generate "$@"

> cat bin/amber
#!/usr/bin/env bash

set -euo pipefail
IFS=$'\n\t'

amber_shard="./lib/amber"
amber_bin="$amber_shard/bin/amber"

# Check to make sure the amber installed by shards is compiled.
# This should only have to run once per project per update of Amber.
if [[ ! -x "$amber_bin" ]]; then
  echo "Compiling Amber. This should only happen once..."
  pushd "$amber_shard" > /dev/null
  shards build amber
  if [[ $? -ne 0 ]]; then
    echo "Building local copy of Amber failed!"
    echo
    exit 1
  fi
  popd > /dev/null
fi

# execute whatever command against that binary
"$amber_bin" "$@"

[faustinoaq]

@robacarp I like your script but would we still need to install amber cli globally?

@amberframework/contributors WDYT about https://github.com/amberframework/amber/issues/582#issuecomment-360672932 and https://github.com/amberframework/amber/issues/582#issuecomment-360683842 ?

We can require only crystal and shards commands using local compiled amber via your script and saving time and resources maintaining AUR and brew, (possibly launchpad in the future and other package managers as you said), we can still use them though

[faustinoaq]

I would suggest to implement a plugin script as well 😅

[docelic]

@robacarp I like the idea, but why do we assume that these must be bash scripts? It is suboptimal that this great workflow (which we could imagine would only grow in capabilities/efficiency over time) would be based on poor shell scripts which execute outside of Amber and have no app context.

Wouldn't it make sense for the 'amber' command itself (a Crystal, compiled binary) to perform these things? (Same workflow and steps as you described, but done by Amber CLI.)

There would be a command line switch --local added to amber cli (defaulting to true, and also controllable via environment config?) which controls whether one wants to "delegate" all work to local Amber instance (default behavior) or use amber's built-in behavior (must be activated with --system, or --no-local, or --self, or something similar).

This way we'd get:

1. Real value would be added to Amber binary, rather than dissipating pieces of relevant functionality in irrelevant scripts
2. Invocation would always be identical ("amber ...something...")
3. One wouldn't have to start writing "bin/" on all commands
4. It would be possible to discover the base dir of an Amber app, so one wouldn't be limited to running commands in the top-level dir only

Such as:

crystal init app blog
cd blog
amber init   # Produces lib/amber/bin/amber
amber new    # Invokes "lib/amber/bin/amber --no-local new [options]"
cd some/sub/dir
amber watch  # Runs fine regardless of cwd (no need for ../../bin/amber ...)
amber -v [--system] [--local]  # Reports local/system version

It looks like this approach would fit so nicely with future planned improvements (plugins, configurability, upgrades).

[faustinoaq]

amber init

+1 for this one

Also I think --local and --system flags could work, Good idea @docelic 👍

cd some/sub/dir

@docelic I don't know why we need to use cd some/sub/dir after amber new, following your example we would be already inside blog directory

[docelic]

(Yes, yes, the "cd some/sub/dir" is just showing an example of how the user is not limited to running amber commands from the top level directory only, because amber would find it.)

[damianham]

@docelic I don't see bash scripts as sub-optimal, rather they are an extremely powerful and flexible tool with an easy install procedure and easy to modify to suite your needs. Remember that if the amber executable in the path is a script we remove the need to build a binary for any release. You don't have to start writing 'bin/' on all commands - see @robacarp script - it delegates to the amber binary in the shard.

+1 for amber init but we could go even further and boot strap the amber web app in one go

crystall init app blog
cd blog
amber init -d mysql -t ecr -m crectco --deps
amber watch

[drujensen]

@damianham just to clarify. The amber init would use the system version and the amber watch would use the local version?

[docelic]

I think there's a confusion between init and new that I suggested.
The way I suggested it, 'amber init' would build local amber in lib/amber. The procedure that does this build would be in system's amber, yes.
After that, all other commands (new, watch, and everything else) would also be accessed using 'amber', but it would default to passing all those commands to the local amber in lib/amber.

[drujensen]

It sounds like we are all in agreement on how it should work. amber init will be a system version and amber watch will be a local version.

The question at hand is if we should have a script to determine which version to use or should we have the system version check for a local version and use it instead, correct?

If that is the case, I think I would vote for a bash script.

[docelic]

@damianham If Amber as a project changes/improves the contents of bin scripts, then those scripts need to be upgraded in the same way that the amber binary would have to be upgraded.

Regarding custom modifications, I don't see why would one want to modify those scripts. (And if you do have a legitimate reason/purpose, nothing is stopping you from creating and keeping around your own 5-liner in bash that calls system's or local amber with the arguments and options you want.)

Also, I don't understand how do you mean a person doesn't need to call bin/amber. You need to call at least bin/amber (or bin/generate, and/or other commands) to have that command run and delegate real work to lib/amber/bin/amber.

But all those are minor remarks from my side. My main point is that I don't see how can the group favor bash scripts over adding functionality to the one and authoritative 'amber' binary.

[drujensen]

My take is we would compile a local version in bin/amber when the shard is installed. I see this as a binary version, not a script. This is fairly straight forward. This eliminates the need for a system wide amber install if one so desires.

So you can use /usr/local/bin/amber for system or you can use ./bin/amber for local.

The only script that we might want is the /usr/local/bin/amber could be a bash script that determines which one to call. It should simply check for a local amber version and if that is not found, it will use the system one.

[drujensen]

I think amber init should do the same thing as amber new but it would apply it to an existing project (current directory). WDYT?

[docelic]

@drujensen what you suggest seems like the simplest possible way. Two things would be a consequence of that, though:

1) There should be a way to rebuild bin/amber. Since your approach does not involve making any code changes to 'amber' itself, how would you trigger a rebuild? By re-installing a shard (if that's possible) or by shipping default amber project with a Makefile which has a target for rebuilding? (Both seem non-ideal)

2) If we go creating any scripts (especially if they are so simple as to check if bin/amber exists and call that or system's amber) then we'll need to ship both the amber binary (under some slightly different name) and 'amber' itself which will be an executable shell script and have 10 lines.

It would make sense to add those 10 lines of code to amber cli, and just eventually add a command line switch which controls whether existence of local amber would be checked or not.

That code change would be simple and self-contained. It would just decide whether to call local amber with the same ARGV and exit afterwards, or go into normal execution.

[robacarp]

There are several things being discussed here, some of which are big choices.

1) how an app should be bootstrapped
2) whether to use a local or global paradigm for executing amber commands ./bin/amber vs /usr/local/bin/amber which amounts to: bin/amber vs amber
3) the closely related question, subcommamds: bin/generate vs amber generate
4) how much logic should be installed and updated via package manager
5) how to write this software

I don’t have any opinion in bash vs whatever. If the software works and provides a good user interface, I’m happy.

My proposal here is simple and straightforward. Amber needs a way to ensure that commands such as migrate are run using the correct version of amber for a given app.

Using the paradigm bin/amber makes this simple, and more importantly _obvious_ to the user. Using a global install and magically switching to the local binary is misleading and breaks the shell paradigm. which amber becomes a kind of lie.

Along with that, as @doclic said, all amber commands would be prefixed with /bin/amber: bin/amber generate etc. However, the reason for those command to be nested in the first place is to make them _easily accessible_ and more importantly _discoverable_ to the user. If the commands are all project local, accessibility and discoverability are no longer an issue. So, ditch the amber: bin/generate

None of this requires any change to the actual amber binary. This way of organizing and presenting the user interface of amber is decoupling the interface from the implementation. That is why I elected to use a simple bash script to accomplish what I’ve done here.

As for the question of bootstrapping and what code should live in the binary managed by system package managers, that can wait until the dust settles here.

---

As a side note, bash is not poor. It’s a technology. There is no need to resort to subtle jabs to convey preferences. Technology exists and we all prefer one thing over another.

edit: formatting

[drujensen]

@docelic Regarding 1, I think the shards update will call the post install script that can check the bin/amber version and rebuild it if its not the latest.

[docelic]

@robacarp I am of opinion that introducing bash scripts into "critical path" of regular user workflow is not a good idea for a project whose specific strong point is that it has a single, compiled binary. That was the reason for my comment.

I also find it not good to populate bin/ with amber's subcommands (generate, watch, etc.). First because it makes the user believe there are multiple commands when in fact there is only 'amber', and second because you enter a dilemma which of all amber's subcommands will have bin/[subcommand] and which won't.

(And also I would prefer to keep the number of files placed by Amber in the default project tree to a minimum.)

But, those are my suggestions. I don't have anything else to say on the matter. It's probably gonna be useful and an improvement whichever approach is taken.

[elorest]

Wow this is really long. I'd like to point out shards build pulls the correct shards and builds the amber binary from the locked amber version in ./bin exactly how rails does.

[drujensen]

Ok, I've added back the target to the shard.yml in PR #603

To create a local binary, run shards build amber.

[damianham]

@drujensen yes my thought was that amber init would use the system amber in your path and amber {everything else} would use the built binary from the shard and amber init would do the same thing as amber new only apply it to the current directory.
@docelic when I said you don't need to call ./bin/amber it is because in the bash script given by @robacarp above, the amber binary is built into ./lib/amber/bin and the bash script delegates to it. Bash scripts are easy to maintain, can have a single responsibility and work on all versions of unix out of the box. In this scenario we would never need to ship an amber binary and I think that is the biggest advantage - the install process is simple.

[elorest]

With the amber binary built into ./bin on shard install what advantage is there to having a shell script.

One can already run ./bin/amber db migrate as of @drujensen PR which adds that functionality back.

[robacarp]

This is great progress. I love the idea of having bin/amber available and easy to build, thanks @drujensen. Lets let this sit for a while, and revisit later if needed.