Diesel: Precompiled diesel_cli binaries for major operating systems

[lpil]

Hello! Thank you for this excellent project!

I wish to make use of diesel_cli on machines that are both resource constrained and do not have Rust installed.

Would it be possible to make precompiled binaries availible so they can be downloaded and run without needing to install Rust and compile them ourselves?

Thank you,
Louis

[weiznich]

In theory this it should be possible to build binaries for all major operation systems. As this should be as easy as possible for us maintains I would like to have this as part of our CI workflow. https://github.com/rust-analyzer/rust-analyzer/blob/master/.github/workflows/release.yaml As we are planing to migrate our CI to github actions anyway sometime soon this should probably be integrated into #2269. rust-anaylzer is already doing something like this for their release process, so maybe some parts from there could be reused? At least I personally do not have currently the bandwidth to work on that, so if someone wants to do that feel free to submit a corresponding PR integrating that with our current CI.

Open questions that needs to be answered:

• [ ] How to handle dependencies like libpq, libmysqlclient and libsqlite3? If we link them dynamically we need to assume something about their location and name on the target system, right? Statically linking them will require a lot of fiddling with the build system.

[benedictjohannes]

The way Diesel define migration in pure SQL statements makes source control of DB schema extremely easy to deploy. One just need to check out SQL files, have the Diesel CLI binary, execute migrations.

However, having to compile Diesel CLI for every system in which it needs to run is a dealbreaker.

(forgive my ignorance) Being a nodeJS/web developer I'm checking whether I can use Diesel CLI to handle migrations. I usually handle migrations with knexJS, but it requires node and package installation to be able to do its job. Both Knex and Golang's implementation require code in the framework's language (.js or .go).

However, Diesel CLI's approach of migrations makes it (almost) language agnostic: Any DBA and database-connected app developer can write pure SQL (I'm not aware of other small binary that does the job). Diesel (at least its CLI) can then potentially reach beyond Rust developers.

Is it possible to static link all DB client libraries (I mean, bake it in the compiled binary)? I had no problem connecting to PG11.x server using psql v12. Arch Linux has (precompiled) binaries as well (requiring dependencies of client libs, though).

[weiznich]

It is certainly possible to link all DB client libraries statically. For sqlite this can be done via the sqlite-bundled feature. For libpq and libmysqclient some configuration of the corresponding sys crates (libpq-sys, libmysqlclient-sys) via environment variables is required. Checkout their README's and build.rs scripts for details.

As mentioned above: I won't work by myself on this issue, but I will accept a well written PR here.

[lpil]

However, Diesel CLI's approach of migrations makes it (almost) language agnostic: Any DBA and database-connected app developer can write pure SQL (I'm not aware of other small binary that does the job). Diesel (at least its CLI) can then potentially reach beyond Rust developers.

To clarify this is my use-case, I'm working with projects written in Erlang rather than Rust and find Diesel an attractive alternative to Flyway as it requires us to install and run the JVM.

[benedictjohannes]

@lpil Probably the only part giving away Diesel CLI's power as not a migration toolkit is that it requires Cargo.toml to operate. I bet this feature can gain traction and will probably benefit both Rust and Diesel in terms of developer recognition.

I tried do dive into Rust toolchain the best that I could, but with setting target to linux musl. Diesel CLI freshly cloned from master branch compiled successfully, but the binary is still not portable (musl target only has glibc statically linked - build log)
I even played around for a few hours on fresh GCP VM with Docker Rust Musl builder image but I still failed to build Diesel CLI for musl. It must be me, as the container image passed tests building Diesel. I believe providing such binaries is a small challenge for true Rustaceans.

@weiznich I understand that this might not be a priority, but I believe that this would be a great feature.

[weiznich]

Probably the only part giving away Diesel CLI's power as not a migration toolkit is that it requires Cargo.toml to operate.

With https://github.com/diesel-rs/diesel/pull/2069 it does not require a Cargo.toml anymore. Using a diesel.toml that optionally configures the migration directory also works now.

I tried do dive into Rust toolchain the best that I could, but with setting target to linux musl. Diesel CLI freshly cloned from master branch compiled successfully, but the binary is still not portable (musl target only has glibc statically linked - build log)
I even played around for a few hours on fresh GCP VM with Docker Rust Musl builder image but I still failed to build Diesel CLI for musl. It must be me, as the container image passed tests building Diesel. I believe providing such binaries is a small challenge for true Rustaceans.

https://github.com/emk/rust-musl-builder might be helpful as they have explicit examples how to build diesel applications using musl and diesel-cli is in the end just a diesel application.

[weiznich]

@GopherJ I do not see what this adds to the discussion, that's already written in my comment above. If you've nothing to say, please stop adding comments here.

[GopherJ]

ok @weiznich I'll delete my comment above, I just want to say it's important while seeing so many .so

However, pls don't use github as dictatorship platform, I know you are a good contributor of diesel, but this shouldn't stop others speaking...

being a good contributor doesn't mean:

You are superior to others

[weiznich]

@GopherJ It's not about stopping someone from speaking, it's just about increasing the signal to noise ratio for potential contributors. I'm happy to hear out comments that add something to the discussion, but I dislike going though my notifications and spend my time with reading basically "me too messages" coming up in my post box.

[GopherJ]

After applying the following changes:

I'm able to use rust-musl-builder to compile a static linked diesel_cli

alias rust-musl-builder='docker run --rm -it -v "$(pwd)":/home/rust/src ekidd/rust-musl-builder'

rust-musl-builder cargo build --release --manifest-path=diesel_cli/Cargo.toml --no-default-features --features postgres

[benedictjohannes]

@GopherJ Your last comment looks helpful. Can you share in more detail, maybe on a separate branch on your fork or maybe using github gist? It's quite hard for me to understand what you've done by looking at a screenshot.

[weiznich]

I would rather not like to have a direct dependency on openssl-sys. That's an internal libpq dependency, so I think that should be handled there. Additionally it seems that libpq + openssl + musl is causing some issues in general. See for example #813 or #700 for example.

[GopherJ]

@benedictjohannes Hi, basically what I did comes from: https://github.com/emk/rust-musl-builder#making-diesel-work and https://github.com/emk/rust-musl-builder#making-openssl-work

without these I ran into: https://github.com/emk/rust-musl-builder/issues/69

fork:https://github.com/GopherJ/diesel (not tested, just added patch for you)