module github.com/spf13/viper
require (
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6 // indirect
github.com/coreos/bbolt v1.3.2 // indirect
github.com/coreos/etcd v3.3.10+incompatible // indirect
github.com/coreos/go-semver v0.2.0 // indirect
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e // indirect
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f // indirect
github.com/dgrijalva/jwt-go v3.2.0+incompatible // indirect
github.com/fsnotify/fsnotify v1.4.7
github.com/gogo/protobuf v1.2.1 // indirect
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef // indirect
github.com/google/btree v1.0.0 // indirect
github.com/gorilla/websocket v1.4.0 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0 // indirect
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
github.com/grpc-ecosystem/grpc-gateway v1.9.0 // indirect
github.com/hashicorp/hcl v1.0.0
github.com/jonboulle/clockwork v0.1.0 // indirect
github.com/magiconair/properties v1.8.0
github.com/mitchellh/mapstructure v1.1.2
github.com/pelletier/go-toml v1.2.0
github.com/prometheus/client_golang v0.9.3 // indirect
github.com/soheilhy/cmux v0.1.4 // indirect
github.com/spf13/afero v1.1.2
github.com/spf13/cast v1.3.0
github.com/spf13/jwalterweatherman v1.0.0
github.com/spf13/pflag v1.0.3
github.com/stretchr/testify v1.2.2
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5 // indirect
github.com/ugorji/go v1.1.4 // indirect
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 // indirect
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77
go.etcd.io/bbolt v1.3.2 // indirect
go.uber.org/atomic v1.4.0 // indirect
go.uber.org/multierr v1.1.0 // indirect
go.uber.org/zap v1.10.0 // indirect
golang.org/x/net v0.0.0-20190522155817-f3200d17e092 // indirect
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4 // indirect
google.golang.org/grpc v1.21.0 // indirect
gopkg.in/yaml.v2 v2.2.2
)
Why do I need GRPC for configuration?
The commit that did this, https://github.com/spf13/viper/commit/b5bf975e5823809fb22c7644d008757f78a4259e, added 24 dependencies, while only removing 5 to the go.mod and a rather large 148 additions to 5 deletions in the go.sum
I believe it is the "remote" feature that is causing all of these imports, but I'm not sure. Perhaps that feature should be behind a build tag? Or perhaps it's introduction should be considered a breaking change?
cc @spf13
markbates
All 17 comments
Hi @markbates, I wondered about the same question too. The added dependencies turns out to be a direct result of merely running go mod tidy.
Go Wiki provides an answer to "Why does 'go mod tidy' record indirect and test dependencies in my 'go.mod'?" at https://github.com/golang/go/wiki/Modules#why-does-go-mod-tidy-record-indirect-and-test-dependencies-in-my-gomod
And here is an example command showing why GRPC is added:
$ go mod why -m google.golang.org/grpc
# google.golang.org/grpc
github.com/spf13/viper/remote
github.com/xordataexchange/crypt/config
github.com/xordataexchange/crypt/backend/etcd
github.com/coreos/etcd/client
github.com/coreos/etcd/client.test
github.com/coreos/etcd/integration
google.golang.org/grpc
Granted, being a relatively Go perpetual newbie myself, I am still trying to wrap my head around it, but looks like it is what the Go developers would call #WAI, working as intended.
anthonyfok
on 26 May 2019
I'm looking into this. I'm also not quite clear why this is a problem. Viper + remote does depend on all of these but I believe that if you are using a mirror, it won't download or build them unless you are using the remote functionality.
spf13
on 7 Jun 2019
It definitely needs all these new deps, but my question is more abstract in that I question whether this featured which brings all these deps is one that the majority of people will use or not. I know I don’t need the feature and now I have a bunch of crazy deps. :(
I’m considering moving all my projects off of Cobra to cut out all of these extra deps.
markbates
on 7 Jun 2019
I'm confused about what you mean by "deps" or really, why you are concerned. Right now they are just lines in a config file. They aren't dependencies until you import them. Viper doesn't import them by default, it requires you to add an additional import. What additional cost are these lines having on you? (I'm not being sarcastic, I'm genuinely trying to understand better).
spf13
on 7 Jun 2019
I’m not using that feature but all of these new dependencies are part of my app now.
For example none of my apps use grpc, but do use cobra and now I have all these dependencies in all of my applications.
Go modules downloads all of these deps even though I’m not using them. That’s my concern.
markbates
on 7 Jun 2019
@spf13 to give you an idea:
go get -u github.com/spf13/cobra
go: finding golang.org/x/crypto latest
go: finding github.com/tmc/grpc-websocket-proxy latest
go: finding golang.org/x/time latest
go: finding golang.org/x/net latest
go: finding golang.org/x/sys latest
go: finding github.com/xiang90/probing latest
go: finding golang.org/x/tools latest
go: finding gopkg.in/check.v1 latest
go: finding github.com/golang/groupcache latest
go: finding github.com/golang/glog latest
go: finding github.com/armon/consul-api latest
go: finding github.com/coreos/go-systemd latest
go: finding github.com/coreos/pkg latest
go: finding github.com/prometheus/client_model latest
go: finding github.com/prometheus/client_golang v0.9.4
go: finding github.com/coreos/etcd v3.3.13+incompatible
go: finding google.golang.org/genproto latest
go: finding github.com/prometheus/common v0.4.1
go: finding golang.org/x/sync latest
go: finding github.com/mwitkow/go-conntrack latest
go: finding github.com/alecthomas/units latest
go: finding google.golang.org/appengine v1.6.1
go: finding google.golang.org/grpc v1.21.1
go: finding github.com/prometheus/tsdb v0.8.0
go: finding github.com/kr/logfmt latest
go: finding golang.org/x/lint latest
go: finding github.com/dgryski/go-sip13 latest
go: finding golang.org/x/sys v0.0.0-20190606165138-5da285871e9c
go: finding golang.org/x/oauth2 latest
go: finding honnef.co/go/tools latest
go: finding golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b
go: finding github.com/prometheus/procfs v0.0.2
go: finding github.com/spaolacci/murmur3 v1.1.0
go: finding golang.org/x/net v0.0.0-20190603091049-60506f45cf65
go: finding github.com/alecthomas/template latest
go: finding github.com/json-iterator/go v1.1.6
go: finding github.com/OneOfOne/xxhash v1.2.5
go: finding github.com/modern-go/concurrent latest
go: finding github.com/google/renameio v0.1.0
go: finding golang.org/x/exp latest
go: finding github.com/google/pprof latest
go: finding golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e
go: finding google.golang.org/api v0.6.0
go: finding github.com/jstemmer/go-junit-report latest
go: finding go.opencensus.io v0.22.0
go: finding golang.org/x/tools v0.0.0-20190530171427-2b03ca6e44eb
go: finding golang.org/x/image latest
go: finding golang.org/x/mod v0.1.0
go: finding github.com/BurntSushi/xgb latest
go: finding golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529
go: finding golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd
go: finding golang.org/x/mobile latest
go: finding google.golang.org/grpc v1.20.1
go: finding google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb
go: finding golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c
go: finding golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b
go: finding google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873
go: finding cloud.google.com/go v0.38.0
go: finding golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c
go: finding google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7
go: finding google.golang.org/api v0.4.0
# I gave up because I'm on hotel wifi
Notice things like this:
go: finding github.com/armon/consul-api latest
go: finding github.com/coreos/go-systemd latest
go: finding github.com/coreos/pkg latest
go: finding github.com/prometheus/client_model latest
go: finding github.com/prometheus/client_golang v0.9.4
go: finding github.com/coreos/etcd v3.3.13+incompatible
go: finding google.golang.org/genproto latest
go: finding github.com/prometheus/common v0.4.1
It's pulling down all of those packages to my machine that I'm not actually using. On my hotel wifi it means I'm basically stuck from working because of the downloading modules needs to do.
Since nothing in my app uses consul, prometheus, grpc, coreos, etc... they're all still being downloaded. :(
markbates
on 7 Jun 2019
Ok. I understand now. I wouldn't say they are part of your app (as you aren't really importing them), but they are definitely part of your download right now and that's obviously a problem.
However, once the Go module mirror is in place, which should be soon, it won't be. The mirror will permit you to only download the modules you use. Without the mirror there's no mechanism to get just the go.mod files from the dependencies so it downloads the entire repo.
For what it's worth, these dependencies were added several years ago, but moved into another package to require an additional import to avoid exactly this problem. Unfortunately we're in this odd transition time between modules and the mirror being live so this problem appeared again.
I'm not sure what the right short term solution is TBH. I'm open to suggestions though.
spf13
on 7 Jun 2019
Right, the dependencies existed already. The go mod tidy commit just made them more visible. Because dependencies were invisible pre-modules, many popular packages in the Go ecosystem carry far more dependencies than you might expect. Viper is one example; there are more. The right long-term fix is to think more carefully about which dependencies make sense and which don't.
We do have some fixes on the go command side on the way too, though.
The go get -u github.com/spf13/cobra in @markbates's transcript is resolving not just the latest cobra but the latest of every module listed in cobra's go.mod, and their go.mod files, recursively. For Go 1.13, we have cut this back to be more like the old GOPATH mode: instead of working a module granularity it will work at package granularity, so that go get -u github.com/spf13/cobra will get the latest cobra, and then the latest modules for the packages github.com/spf13/cobra actually imports, and so on, recursively. If cobra/remote is what pulls in tons of deps and your program does not import cobra/remote even indirectly, then go get -u will not go looking for the latest copies of those anymore (again, in Go 1.13). Go 1.14 may improve things further.
(I am not sure about the difference between viper and cobra here, but cobra is what the transcript said even though we are in the viper issue tracker.)
rsc
on 11 Jun 2019
This is very helpful to know. We should share this more broadly :)
Viper is a config management framework (flags, env vars, config files, etc)
that optionally supports remote config systems like etcd.
Cobra is a CLI framework (commands, subcommands, flags).
They work well together and also work well independently.
Cobra includes an application generator (github.com/spf13/cobra/cobra)
which optionally can add viper support to the CLI.
So it seems this new behavior is hitting Cobra users doubly hard importing
not only Viper but also all the remote functionality even if you are only
using Cobra.
On Tue, Jun 11, 2019 at 5:07 PM Russ Cox notifications@github.com wrote:
Right, the dependencies existed already. The go mod tidy commit just made
them more visible. Because dependencies were invisible pre-modules, many
popular packages in the Go ecosystem carry far more dependencies than you
might expect. Viper is one example; there are more. The right long-term fix
is to think more carefully about which dependencies make sense and which
don't. We do have some fixes on the go command side on the way too, though.The go get -u github.com/spf13/cobra in @markbates
https://github.com/markbates's transcript is resolving not just the
latest cobra but the latest of every module listed in cobra's go.mod, and
their go.mod files, recursively. For Go 1.13, we have cut this back to be
more like the old GOPATH mode: instead of working a module granularity it
will work at package granularity, so that go get -u github.com/spf13/cobra
will get the latest cobra, and then the latest modules for the packages
github.com/spf13/cobra actually imports, and so on, recursively. If
cobra/remote is what pulls in tons of deps and your program does not import
cobra/remote even indirectly, then go get -u will not go looking for the
latest copies of those anymore (again, in Go 1.13). Go 1.14 may improve
things further.(I am not sure about the difference between viper and cobra here, but
cobra is what the transcript said even though we are in the viper issue
tracker.)—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spf13/viper/issues/707?email_source=notifications&email_token=AABKKZHAE3X6MK6YV5MIRFTP2AHXPA5CNFSM4HPVMEF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXOQDTY#issuecomment-501023183,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABKKZGOAEXCRSMXRLNNH2LP2AHXPANCNFSM4HPVMEFQ
.
spf13
on 11 Jun 2019
An almost identical question on viper happen to come up today in another forum, so adding some quick comments here as well.
One thing that might help is to at least initially not use the -u for something like go get -u github.com/spf13/cobra and instead do something like go get github.com/spf13/cobra@latest. That still gets the latest version of cobra, but no -u means it does not have to also hunt to find the most recent versions of all of the large number of direct and indirect dependencies of cobra. That can help the go command do less work, especially if on something like a development laptop that has built up a module cache in GOPATH/pkg/mod. If desired, the -u can then be used later at a more convenient time if there is a desire to upgrade to the most recent versions of all the direct and indirect dependencies of cobra.
Also, a quick test with a simple viper client does show things like etcd and prometheus in the module-level graph, but things like etcd and prometheus do not appear to be package-level dependencies in the actual build. One way to see that (from the modules wiki):
go listcan show the exact versions used in your build excluding test-only dependencies:
go list -deps -f '{{with .Module}}{{.Path}} {{.Version}}{{end}}' ./... | sort -u
Finally, in advance of Go 1.13, there is a healthy chance export GOPROXY=https://proxy.golang.org would speed things up for this example in Go 1.12. (proxy.golang.org is beta status. Alternatively, one can try export GOPROXY=https://gocenter.io, which has been around longer). One approach is you can turn GOPROXY on temporarily when getting the public modules you depend on that have the largest module graphs. It is not uncommon for there to be something like a 5–10x improvement in go get download time and total bytes downloaded if you have a project with a large module graph (including due to the ability to get remote go.mod files via a small HTTPS GET rather than a repo fetch).
thepudds
on 12 Jun 2019
That's something I would love for the long term, but that's a breaking change for the existing code.
Update: Looks like the comment I was responding to had been deleted. I meant separate packages would be a nice solution for remote implementations, but would also be a breaking change.
sagikazarmark
on 12 Jun 2019
Something to consider might be making github.com/spf13/viper/remote a separate module (with its own go.mod) -- but still in the same repo. This would also require an additional git tag (e.g. remote/v1.4.1) so it's sort of a PITA to maintain, but it would remove all of the _etcd_ and/or _consul_ dependencies from viper proper and only pull them in if the user is actually leveraging remote configs.
I've done something similar with toolman.org/net/peercred and its subordinate package toolman.org/net/peercred/grpcpeer so that gRPC's giant dependency tree is only included when it's actually being used.
Whether this is a recommended or idiomatic solution I can't say, but at least it works.
tep
on 15 Jun 2019
Although tightly coupled dependencies were probably always the case, only recently did we start digging deep into viper after https://github.com/spf13/viper/pull/635. These issues have unfortunately pushed us to remove viper from a couple dozen of our projects and reinvent the wheel (github.com/knadh/koanf).
The following example, although not a 1:1 comparison, is illustrative of the effects of hardcoded dependencies (go1.12 linux/amd64).
This produces a 2.4 MB binary.
package main
import (
"fmt"
"encoding/json"
)
func main() {
// Dummy, just to bring in a parser.
var o map[string]interface{}
json.Unmarshal([]byte(`{"hello": "world}`), &o)
fmt.Println(o)
}
Here, simply including viper, irrespective of the features used, produces a 12 MB binary.
package main
import (
"fmt"
"github.com/spf13/viper"
)
func main() {
fmt.Println(viper.GetString("hello world"))
}
I agree with @sagikazarmark . The only probable solution is to split dependencies into separate packages, but like he said, it would be a breaking change.
knadh
on 20 Jun 2019
I'm ok with making a breaking change provided we clearly tag the release
prior to and after the change and communicate it clearly. It's not 1.0 yet
allowing for such changes.
If I understand it all correctly, the fix for the break is likely just
adding some additional import statements if you happen to be using a
feature in the separated package. Pretty trivial.
On Thu, Jun 20, 2019 at 2:05 AM Kailash Nadh notifications@github.com
wrote:
Although tightly coupled dependencies were probably always the case, only
recently did we start digging deep into viper after #635
https://github.com/spf13/viper/pull/635. These issues have
unfortunately pushed us to remove viper from a couple dozen of our projects
and reinvent the wheel (github.com/knadh/koanf).The following example, although not a 1:1 comparison, is illustrative of
the effects of hardcoded dependencies (go1.12 linux/amd64).This produces a 2.4 MB binary.
package main
import (
"fmt"
"encoding/json"
)
func main() {
// Dummy, just to bring in a parser.
var o map[string]interface{}
json.Unmarshal([]byte({"hello": "world}), &o)
fmt.Println(o)
}Here, simply including viper, irrespective of the features used, produces
a 12 MB binary.package main
import (
"fmt""github.com/spf13/viper"
)
func main() {
fmt.Println(viper.GetString("hello world"))
}I agree with @sagikazarmark https://github.com/sagikazarmark . The only
probable solution is to split dependencies into separate packages, but like
he said, it would be a breaking change.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spf13/viper/issues/707?email_source=notifications&email_token=AABKKZGDXVUUPHPJQJ6IFITP3MMZ7A5CNFSM4HPVMEF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYELVAQ#issuecomment-503888514,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABKKZETA75T3LDOZFO5MJ3P3MMZ7ANCNFSM4HPVMEFQ
.
spf13
on 20 Jun 2019
Hi, I need some assistance but is remote config reader library xordataexchange/crypt deprecated?
zarraozaga
on 26 Jun 2019
etcd 3.5 will improve this significantly: client packages will be released as separate modules allowing to import only what's necessary for client functions. See etcd-io/etcd#12330 and etcd-io/etcd#12498
Sadly, there is no ETA for the release.
I prepared a draft PR for the crypt library: bketelsen/crypt#12
As you can see there is a significant change in dependencies.
sagikazarmark
on 11 Jan 2021
@sagikazarmark it seems that etcd is sort of not-happening anytime soonish.
In https://github.com/spf13/viper/issues/867 I had made a suggestion that split remote config from local config by simply moving remote config to a separate module. This breaks compatibility but seems a low-hanging fruit since it doesn't require extensive changes and is easy to adapt (I may have broken it though).
Would you consider that suggestion as a short-term intermediate step (say v2) before doing larger refactoring in a longer-term v3?
Update I've just recompiled my application with viper 1.7.1 instead of my local fork using go 1.16 and can no longer find a significant difference in binary size. I'm not sure if that was expected but it sort of removes the need for action.
andig
on 15 Mar 2021
Related issues
michaeleekk
·
3Comments
ankur22
·
5Comments
therealfakemoot
·
5Comments
orian
·
4Comments
TimJones
·
4Comments
Most helpful comment
Although tightly coupled dependencies were probably always the case, only recently did we start digging deep into viper after https://github.com/spf13/viper/pull/635. These issues have unfortunately pushed us to remove viper from a couple dozen of our projects and reinvent the wheel (github.com/knadh/koanf).
The following example, although not a 1:1 comparison, is illustrative of the effects of hardcoded dependencies (go1.12 linux/amd64).
This produces a 2.4 MB binary.
Here, simply including viper, irrespective of the features used, produces a 12 MB binary.
I agree with @sagikazarmark . The only probable solution is to split dependencies into separate packages, but like he said, it would be a breaking change.