Migrate: Too many dependencies

[dumindu]

Current dependencies:

require (
    dmitri.shuralyov.com/app/changes v0.0.0-20181114035150-5af16e21babb // indirect
    dmitri.shuralyov.com/service/change v0.0.0-20190203163610-217368fe4577 // indirect
    git.apache.org/thrift.git v0.12.0 // indirect
    github.com/Shopify/sarama v1.20.1 // indirect
    github.com/aws/aws-sdk-go v1.16.36 // indirect
    github.com/coreos/go-systemd v0.0.0-20190212144455-93d5ec2c7f76 // indirect
    github.com/cznic/b v0.0.0-20181122101859-a26611c4d92d // indirect
    github.com/cznic/fileutil v0.0.0-20181122101858-4d67cfea8c87 // indirect
    github.com/cznic/golex v0.0.0-20181122101858-9c343928389c // indirect
    github.com/cznic/internal v0.0.0-20181122101858-3279554c546e // indirect
    github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 // indirect
    github.com/cznic/sortutil v0.0.0-20181122101858-f5f958428db8 // indirect
    github.com/cznic/strutil v0.0.0-20181122101858-275e90344537 // indirect
    github.com/cznic/zappy v0.0.0-20181122101859-ca47d358d4b1 // indirect
    github.com/docker/distribution v2.7.1+incompatible // indirect
    github.com/edsrzf/mmap-go v1.0.0 // indirect
    github.com/fsouza/fake-gcs-server v1.5.0 // indirect
    github.com/go-ini/ini v1.41.0 // indirect
    github.com/go-logfmt/logfmt v0.4.0 // indirect
    github.com/gocql/gocql v0.0.0-20190208221138-c53c3654dc8a // indirect
    github.com/golang-migrate/migrate/v4 v4.2.4 // indirect
    github.com/golang/lint v0.0.0-20181217174547-8f45f776aaf1 // indirect
    github.com/google/pprof v0.0.0-20190208070709-b421f19a5c07 // indirect
    github.com/googleapis/gax-go v2.0.2+incompatible // indirect
    github.com/gopherjs/gopherjs v0.0.0-20181103185306-d547d1d9531e // indirect
    github.com/gregjones/httpcache v0.0.0-20190212212710-3befbb6ad0cc // indirect
    github.com/grpc-ecosystem/grpc-gateway v1.7.0 // indirect
    github.com/jackc/pgx v3.3.0+incompatible // indirect
    github.com/kshvakov/clickhouse v1.3.5 // indirect
    github.com/mattn/go-sqlite3 v1.10.0 // indirect
    github.com/microcosm-cc/bluemonday v1.0.2 // indirect
    github.com/mongodb/mongo-go-driver v0.3.0 // indirect
    github.com/openzipkin/zipkin-go v0.1.5 // indirect
    github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90 // indirect
    github.com/prometheus/common v0.2.0 // indirect
    github.com/prometheus/procfs v0.0.0-20190209105433-f8d8b3f739bd // indirect
    github.com/russross/blackfriday v2.0.0+incompatible // indirect
    github.com/shurcooL/go v0.0.0-20190121191506-3fef8c783dec // indirect
    github.com/shurcooL/gofontwoff v0.0.0-20181114050219-180f79e6909d // indirect
    github.com/shurcooL/highlight_diff v0.0.0-20181222201841-111da2e7d480 // indirect
    github.com/shurcooL/highlight_go v0.0.0-20181215221002-9d8641ddf2e1 // indirect
    github.com/shurcooL/home v0.0.0-20190204141146-5c8ae21d4240 // indirect
    github.com/shurcooL/htmlg v0.0.0-20190120222857-1e8a37b806f3 // indirect
    github.com/shurcooL/httpfs v0.0.0-20181222201310-74dc9339e414 // indirect
    github.com/shurcooL/issues v0.0.0-20190120000219-08d8dadf8acb // indirect
    github.com/shurcooL/issuesapp v0.0.0-20181229001453-b8198a402c58 // indirect
    github.com/shurcooL/notifications v0.0.0-20181111060504-bcc2b3082a7a // indirect
    github.com/shurcooL/octicon v0.0.0-20181222203144-9ff1a4cf27f4 // indirect
    github.com/shurcooL/reactions v0.0.0-20181222204718-145cd5e7f3d1 // indirect
    github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect
    github.com/shurcooL/webdavfs v0.0.0-20181215192745-5988b2d638f6 // indirect
    github.com/smartystreets/assertions v0.0.0-20190215210624-980c5ac6f3ac // indirect
    github.com/smartystreets/goconvey v0.0.0-20181108003508-044398e4856c // indirect
    go.opencensus.io v0.19.0 // indirect
    go4.org v0.0.0-20181109185143-00e24f1b2599 // indirect
    golang.org/x/build v0.0.0-20190215225244-0261b66eb045 // indirect
    golang.org/x/exp v0.0.0-20190212162250-21964bba6549 // indirect
    golang.org/x/oauth2 v0.0.0-20190212230446-3e8b2be13635 // indirect
    golang.org/x/perf v0.0.0-20190124201629-844a5f5b46f4 // indirect
    golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a // indirect
    golang.org/x/tools v0.0.0-20190214204934-8dcb7bc8c7fe // indirect
    google.golang.org/genproto v0.0.0-20190215211957-bd968387e4aa // indirect
    google.golang.org/grpc v1.18.0 // indirect
    gopkg.in/ini.v1 v1.41.0 // indirect
    honnef.co/go/tools v0.0.0-20190215041234-466a0476246c // indirect
    sourcegraph.com/sqs/pbtypes v1.0.0 // indirect
)

Don't you think that this is too much for a db migration package?

Compared to other alternatives, this is too much

• ex. pressly/goose; only relative db drivers, even those can be reduced from binary via build -tags=

require (
    github.com/go-sql-driver/mysql v1.4.1 // indirect
    github.com/lib/pq v1.0.0 // indirect
    github.com/mattn/go-sqlite3 v1.10.0 // indirect
    github.com/pressly/goose v2.4.5+incompatible // indirect
    github.com/ziutek/mymysql v1.5.4 // indirect
)

[dhui]

This is probably due to migrate supporting more DBs than pressly/goose.

go mod tidy will add all dependencies regardless of build tag, etc
Not all of these dependencies will be used in your final build.

[dhui]

Related issue: https://github.com/golang-migrate/migrate/issues/60

[dumindu]

But why we need openzipkin, prometheus like packages? Also github.com/shurcooL/home is the repo for Dmitri Shuralyov's personal website. Looks like something is very wrong.

[dhui]

go mod why is your friend

You may see some packages in go.sum, this explains why

Also, the go.mod snippet you originally provided doesn't look like the one for migrate...

[dumindu]

what I did/ steps to reproduce

• mkdir testpack && cd testpack
• go mod init testpack
• go get -u github.com/golang-migrate/migrate/v4

[dhui]

Looks like that's your own go.mod file...
Check the module directive at the top of the file...

[dumindu]

@dhui The list I mentioned above is dependencies added for github.com/golang-migrate/migrate/v4 for an empty project. You also can reproduce this via above steps :)

[dhui]

Unfortunately, we can't reduce the number of dependencies in go.mod since they're used by the various supported source and db drivers.

Run go mod tidy after your steps in your test package to prune unused packages.

[sagikazarmark]

While I don't think there is such thing as "too much dependencies" (if they are needed, they are needed), the number of dependencies can make builds huge on the other hand.

Given golang-migrate has it's own organization, I would consider cutting the library into smaller pieces (modules) (eg. one for each driver and a separate one for the cmd).

It could ease the pain of users, but it would probably make development slightly harder.

Currently that's the recommendation you will get from go modules experts as a solution to this problem.

[dhui]

I would consider cutting the library into smaller pieces (modules) (eg. one for each driver and a separate one for the cmd).
It could ease the pain of users, but it would probably make development slightly harder.

The base library, CLI, and drivers (source and database) are already separate packages. What's to be gained by making them separate modules? Could you be more specific about the pain points splitting into separate modules would alleviate?

[sagikazarmark]

Currently regardless of which package you import in a project, go modules will download all transitive dependencies, whether they are going to be compiled into the final binary or not, so separate packages make the work easier for the linker, but the amount of downloaded dependencies will still be the same. By making separate modules, you can limit these dependencies to the absolute minimum.

IMHO the problem of excessive amount of dependencies is obvious: slower CI, more downloaded stuff then necessary, etc.

I'm not saying you should do this, all I'm saying is if this is a problem and you want to solve it, splitting into separate modules can be a solution.

[dumindu]

While I don't think there is such thing as "too much dependencies" (if they are needed, they are needed)

@sagikazarmark According to the dependency list(on go.mod) which is generated via go get -u github.com/golang-migrate/migrate/v4, this package requires even github.com/shurcooL/home (Dmitri Shuralyov's personal website). This is only one dependency which might not needed or something is wrong if any dependency of this depends on someones' personal website.

[sagikazarmark]

I guess it's a mistake in a transitive dependency, not in golang-migrate itself. Go mod installs all transitive dependencies and test dependencies of all transitive dependencies as well.

[dhui]

IMHO the problem of excessive amount of dependencies is obvious: slower CI, more downloaded stuff then necessary, etc.
I'm not saying you should do this, all I'm saying is if this is a problem and you want to solve it, splitting into separate modules can be a solution.

@sagikazarmark Thanks for the stating an advantage of using separate modules. At this time, extraneous transitive dependency downloads is not a concern for this project. Maybe the go tool chain will improve this aspect of dependency management later...

[sagikazarmark]

I believe it will, although not the go tool per se: https://blog.golang.org/modules2019#TOC_6.

Nevertheless, I'm going to link this issue to another one where I try to keep track of the problem (a few libraries of mine suffer from the same issue). I'm also a regular complainer on the Gophers slack #modules channel, so I will also gather information from there and record in the issue.

[thepudds]

Also github.com/shurcooL/home is the repo for Dmitri Shuralyov's personal website. Looks like something is very wrong.

A healthy chunk of of the dependencies cited above seem to be coming in via golang.org/x/..., such as via golang/x/build:

golang.org/x/[email protected] 
github.com/shurcooL/[email protected]

The various golang.org/x repos have been updated recently to (a) have go.mod files and (b) to cut down on the number of extra dependencies, especially for golang.org/x/build.

Updating to the latest golang.org/x repos would likely help. That can be down manually, or via something like the following:

$ for x in $(go list  -f '{{.Path}}' -m golang.org/x/...); do go get $x@latest; done

And then run go mod tidy to eliminate from go.mod and go.sum any dependencies that are no longer needed.

[thepudds]

@dumindu

According to the dependency list(on go.mod) which is generated via go get -u github.com/golang-migrate/migrate/v4, ...

Note that you should consider whether or not you wanted to include a -u there.

From the "How to upgrade and downgrade dependencies" section of the "Modules" wiki:

go get foo updates to the latest version with a semver tag, or the the latest known commit if there are no semver tags. go get foo is equivalent to go get foo@latest — in other words, @latest is the default if no @ version is specified.

Note that a -u in go get -u foo and go get -u foo@latest also upgrades all the direct and indirect dependencies of foo. A common starting point is instead go get foo or go get foo@latest without a -u (and after things are working, consider go get -u=patch foo, go get -u foo, go get -u, or go get -u=patch).

[hbobenicio]

Currently regardless of which package you import in a project, go modules will download all transitive dependencies, whether they are going to be compiled into the final binary or not, so separate packages make the work easier for the linker, but the amount of downloaded dependencies will still be the same. By making separate modules, you can limit these dependencies to the absolute minimum.

IMHO the problem of excessive amount of dependencies is obvious: slower CI, more downloaded stuff then necessary, etc.

I'm not saying you should do this, all I'm saying is if this is a problem and you want to solve it, splitting into separate modules can be a solution.

I agree with the suggestion of @sagikazarmark about the modularization.
If I need to use migrations for my PG database, it feels just awkward to have dependencies (either transitive or not) for things like github.com/aws/aws-sdk-go or github.com/denisenkom/go-mssqldb that are just relevant to the respective datasources, which does not match the actual user datasources.

Maybe the driver pattern could come in handy here in order to separate interfaces from datasource specific implementations, allowing modularization. This is the approach that others like (db-migrate)[https://github.com/db-migrate/node-db-migrate] (which would require db-migrate and db-migrate-pg dependencies for the user).

[dhui]

@hbobenicio AFAIK, the "extraneous" (e.g. unused) dependencies won't affect your builds.

That being said, we may split drivers into their own modules so we can better adhere to SemVer and have different release cycles/cadences.

[majewsky]

the "extraneous" (e.g. unused) dependencies won't affect your builds.

It does affect the build time because it spends precious minutes finding and downloading all these junk dependencies. That may not be a big problem when your pipeline takes hours anyway, but I use Go precisely because everything is fast. An additional minute of slowdown is really painful when your entire pipeline (from git commit to prod) is 5 minutes.

I will definitely hold off upgrading to migrate v4 until this issue is sorted out.

[thepudds]

It does affect the build time because it spends precious minutes finding and downloading all these junk dependencies.

@majewsky If you haven't already, you should try with Go 1.13.

Downloading dependencies is also often 5x - 10x faster in Go 1.13 due to visiting fewer dependencies, and due to defaulting to a GOPROXY (which is more efficient than using VCS as a coarse-grained transport).

Note: If you have private code, you'll likely want to configure the GOPRIVATE setting (such as go env -w GOPRIVATE=*.corp.com) as described in the 1.13 release notes and elsewhere.

[majewsky]

The 1.13 release is what prompted me to bump my deps in the first place. And I don't have private repos.

[sagikazarmark]

Chances are go mod download will only be slightly faster, as it still downloads everything. So locally and in CI do not use go mod download. In docker you don't really have a choice though. 😕

I ended up splitting up my adapter type packages into smaller packages. Not sure if it's better (yet), but solved my dependency issues.

[amadsen]

We ran in to this issue this week because, as @sagikazarmark points out, go mod download downloads _everything_. For our docker based build this results in a layer in our docker image that is over 500MB. The build would technically work fine, but we then have issues pushing this large layer to a private docker registry because it is too large.

We've worked around this by doing a multi-stage docker build so that the deployed image does not contain the go mod download layer, but it would be much appreciated if my project didn't indirectly depend on every library needed for every supported database and source for migrations.

[sagikazarmark]

Unfortunately, this is a Go modules issue. The alternative would be splitting the drivers into separate go modules. Hashicorp vault project is a good example.