Migrate: Proposal: Move DB and source drivers to a separate module

[dhui]

This would allow:

1. Backwards incompatible DB and source driver changes can be released independently of the core migrate library and CLI while still "adhering" to SemVer
2. Reduce number of dependencies (See: https://github.com/golang-migrate/migrate/issues/174)
   
   
   
   • ~Not sure is this is an actual pain point since unused "dependencies" won't affect your builds~ This is a pain point because:
     
     
     
     
     
     1. More dependencies means that builds are more likely to fail/break b/c the dependency host is unavailable or down
     
     
     
     2. go mod download (used for build caching) will create larger docker images
     
     
     
   
   

~The release notes would mention any driver changes.~ (Shouldn't matter since each driver will be versioned independently and have it's own release notes)

Issues:

1. Makes using the pre-built CLI in conjunction with the library harder, if not impossible.
2. Can the CLI help text show the driver version?
3. May have to move drivers to another repo since multiple modules in a single repo can be more work

[1995parham]

@dhui Is there any progress on this? Or can we create a pull request for it?

[dhui]

I haven't worked on this issue yet.

Before opening up a PR, we should resolve the issues.
What are you thoughts on the issues?

[1995parham]

I think creating a sperate module for each database driver would be good and reduce the number of dependencies. In the second step, we can move the drivers into a seprate repository.

[dhui]

@1995parham What about releases and versioning?

[1995parham]

@dhui I think we can scrap the go.mod file of each driver to find out its version but I have no idea about CLI.

[dhui]

@1995parham I was referring more to the semantic versioning aspect.
e.g. How migrate CLI, migrate core (the Go library), and driver (DB and source) versions interact w/ each other. This mainly affects the CLI which may not change major versions, but may change the major versions of the DB/source drivers it depends on.

Also, I don't think the pain of multi-module single repo is worth the organizational benefits.

To be clear, reducing the number of dependencies is not a primary goal for this issue, it's just a nice to have since the dependencies aren't "strict". The extraneous dependencies are just noise.

[amadsen]

it's just a nice to have since the dependencies aren't "strict". The extraneous dependencies are just noise.

While I realize there are real issues to address to pull this off, the extra dependencies are not "just noise". They can cause builds to fail for 2 main reasons:

1. the dependency is unavailable (this is mitigated by using a GOPROXY)
2. the downloaded artifacts from the unused dependency end up in an unnecessarily large build artifact (particularly a docker image), which can cause other build steps to fail

The noise, in and of itself, is also an issue for developers who attempt to audit their code.

[L11R]

@amadsen +1

I can also add that here at my work we have slow internet provided by MITM corporate proxy. All those dependencies loads around 10 minutes for me. Yes, it could be solved by deploying own corporate GOPROXY, but anyway it's real problem, not just "noise" as @dhui described.

[shuz]

I like this idea. I'm using it as a library, and somehow it caused the main go project to upgrade to protobuf 1.4.2, which our code base is not ready for.

My work around is that I made a fork and kept only the necessary drivers and databases and run go mod tidy. Then many of the dependencies are gone.

It would be perfect if it natively separates the drivers and databases to different go modules.

[bshimc]

@dhui Regarding the two objectives (goal#1) decouple source/driver releases from core and (goal#2) reduce dependencies: it seems like, at least by the activity/discussion I've been able to track down in the issues tracker, (2) is viewed by users as more pressing [disclosure: I fall into this category of potential user].

Regarding the issues raised in original post:

1. Perhaps I'm misunderstanding the use case but it doesn't seem like an issue for folks using go modules to compose their dependencies (https://marcofranssen.nl/manage-go-tools-via-go-modules/).
2. For folks using go modules to manage their tools this is probably a non-issue since their tools are versioned and managed in revision control and these users probably have binary stamping capabilities built into their CI process if they care. For folks that simply want to use an official released binary, this seems like some additional effort if you want to achieve goal#1. For these users, goal#1 doesn't really help (or hurt) them in any way so it's unclear to me the complexity trade-off is worth it.

If goal#1 is indeed non-negotiable then multiple modules in a single repo will likely be a non-starter. However, the benefit to using submodules is a middle ground where you can release all modules in lockstep (the go modules documentation is less than clear about this so I may be making wrong assumptions) but also address goal#2 without having to make invasive/large investments in project infrastructure (CI, etc).

As a test, I converted the relevant drivers into submodules:

#!/bin/bash
pushd "$( git rev-parse --show-toplevel )"

# These packages should be re-organized into one directory (cmd perhaps) to
# avoid the more painful corner cases in using the "replace" directive.
for cut in cli/go.mod cmd/go.mod internal/cli/go.mod; do
  cp /dev/null "$cut"
done

for pkg in neo4j clickhouse snowflake mongodb redshift postgres sqlcipher cassandra sqlite3 mysql sqlserver spanner cockroachdb ql firebird; do
  cat << EOF > "database/$pkg/go.mod"
module github.com/golang-migrate/migrate/v4/database/$pkg

go 1.13

replace github.com/golang-migrate/migrate/v4 => ../../
EOF
  pushd "database/$pkg"
  go mod tidy
  go build ./...
  popd
done

And this seems like a fairly non-invasive change that would be hugely beneficial for your users that want to embed this package as a library.

$ go mod tidy && git diff --stat go.mod
 go.mod | 42 +++---------------------------------------
 1 file changed, 3 insertions(+), 39 deletions(-)