Viper: Consider env vars when unmarshalling

[aeneasr]

Is this possible?

UPDATE: yes it is, see https://github.com/spf13/viper/issues/188#issuecomment-255519149

type Config struct {
    BindPort int `mapstructure:"port" yaml:"port,omitempty"`
}

var c Config

// ...
viper.AutomaticEnv()

if err := viper.ReadInConfig(); err != nil {
    // ...
}

if err := viper.Unmarshal(&c); err != nil {
    // ...
}

// I want c.BindPort == viper.Get("PORT") == os.Getenv("PORT")

[rybit]

I am having the same problem, it isn't fixed directly by the #195. You have to register defaults it seems before viper will load the values. I have been experimenting with some terrible reflection code to go set defaults from a given structure, but I'd like to keep reflection out where possible.

Consider the following code:

type Config struct {
  CAFiles  []string `mapstructure:"ca_files"
  CertFile string   `mapstructure:"cert_file"
  KeyFile  string   `mapstructure:"key_file"
  Servers  []string `mapstructure:"servers"`
}


func configuredViperInstance() *viper.Viper {
  v := viper.New()
  v.SetConfigName("config")
  v.AddConfigPath("./")

  v.SetEnvPrefix("test")
  v.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
  v.AutomaticEnv()

  if err := v.ReadInConfig(); err != nil && !os.IsNotExist(err) {
    panic(err)
  }
  return v
}

func main() {
  noDefaults := configuredViperInstance()
  config := new(Config)
  if err := noDefaults.Unmarshal(config); err != nil {
    panic(err)
  }

  fmt.Printf("%+v\n", config)
  fmt.Printf("config-cert_file: %s\n", config.CertFile)
  fmt.Printf("cert_file: %s\n", noDefaults.GetString("cert_file"))

  fmt.Println("-------------------------")

  withDefaults := configuredViperInstance()
  withDefaults.SetDefault("cert_file", "")
  config = new(Config)
  if err := withDefaults.Unmarshal(config); err != nil {
    panic(err)
  }
  fmt.Printf("%+v\n", config)
  fmt.Printf("config-cert_file: %s\n", config.CertFile)
  fmt.Printf("cert_file: %s\n", noDefaults.GetString("cert_file"))
}

I then call it passing in this json

{
  "servers": ["server-1", "server-2"],
  "key_file": "key-file"
}

TEST_CERT_FILE="env-certfile" go run nested_conf_testing.go

I get the response

&{CAFiles:[] CertFile: KeyFile:key-file Servers:[server-1 server-2]}
config-cert_file:
cert_file: env-certfile
-------------------------
&{CAFiles:[] CertFile:env-certfile KeyFile:key-file Servers:[server-1 server-2]}
config-cert_file: env-certfile
cert_file: env-certfile

[aeneasr]

@rybit you need to do

type Config struct {
    BindPort int `mapstructure:"PORT" yaml:"port,omitempty"`
}

var c Config

// ...
viper.AutomaticEnv()

viper.BindEnv("PORT")
viper.SetDefault("PORT", 4444)

if err := viper.ReadInConfig(); err != nil {
    // ...
}

if err := viper.Unmarshal(&c); err != nil {
    // ...
}

to get this working.

[aeneasr]

If BindEnv or SetDefault is missing, it won't work with the latest beggining with PR #195

[benoitmasson]

@rybit I don't think this is a bug, I'd say that this is the expected behavior: how should Viper know that a struct field has to be mapped to an environment variable, if you don't tell it to?

If you ask for config value foo with viper.Get(), Viper knows it has to look for a variable FOO in the environment and viper.AutomaticEnv() will be sufficient to handle that. But on the other side, viper.AllSettings() (used by viper.Unmarshal()) can't be aware that a config value for foo is expected (unless all environment values are added to the map, which wouldn't be reasonable).

As for me, automatically adding defaults or environment binding with reflection is the way to go, if done carefully. You could then use structure tags to have the programmer define his own default value, which is very handy and flexible (I do it myself :-) )

@arekkas From what I remember, if AutomaticEnv is on, then SetDefault() is enough (this tells Viper that such a configuration value exists). BindEnv is useful only when AutomaticEnv is off, and in that case, SetDefault() shouldn't be needed.
Are you sure both are necessary? [EDIT: just saw your other post in the PR, I'll investigate this]

[burdiyan]

I wonder if it's possible to make @arekkas' example working without explicitly doing BindEnv and just doing AutomaticEnv.

Normally we use the following workflow in our applications:

1. Define global configuration structure as a Go struct.
2. Expose flags that can modify this configuration and bind the flag set to viper.
3. Allow to modify the configuration via file, flags or env variables.

Inside the application code we only deal with the config struct (to avoid dealing with arbitrary string keys in Viper). And we explicitly pass the config, it is not a global structure.

Basically the default values of the config are defined in flags. So I would like to avoid explicitly doing BindEnv for each of the env vars. It seems like it should be doable to have this with AutomaticEnv directly.

[burdiyan]

Oh, it seems like doing viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_")) makes it possible.

So maybe adding some documentation about it can make this issue fixed.

[from-nibly]

@burdiyan were you able to confirm that your solution with the SetEnvKeyReplacer works? I am unable to get it to work in my project. Maybe there is another thing I am missing?

calling viper.GetString("foo.bar") gets me the value but calling viper.UnmarshalKey("foo", &cfg) uses the value found in my config file.

[from-nibly]

nvm I figured out you can fix it with this work around

for _, key := range viper.AllKeys() {
    val := viper.Get(key)
    viper.Set(key, val)
}

[aarondl]

@from-nibly did you end up using BindEnv or SetDefault to ensure you had keys to reference? I'm looking at this problem and it seems like I'm going to have to crawl through the env variables and set defaults/values for them in order to make Unmarshal work correctly.

    viper.SetEnvPrefix("application")
    viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
    viper.AutomaticEnv()

Even with this setup the methods AllKeys and AllSettings return nothing, which Unmarshal uses directly to try to prepare a structure for the mapstructure library to work.

[from-nibly]

here is the full code I used to create a viper instance that I could use to Unmarshal an object that used env variables

func LoadConfig() *viper.Viper {
    conf := viper.New()

    conf.AutomaticEnv()
    conf.SetEnvPrefix("ps")
    conf.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))

    conf.SetConfigName("application")
    conf.AddConfigPath("/etc/my-app")
    err := conf.ReadInConfig()

    if err != nil {
        switch err.(type) {
        default:
            panic(fmt.Errorf("Fatal error loading config file: %s \n", err))
        case viper.ConfigFileNotFoundError:
            logger.Warning("No config file found. Using defaults and environment variables")
        }
    }

    // workaround because viper does not treat env vars the same as other config
    for _, key := range conf.AllKeys() {
        val := conf.Get(key)
        conf.Set(key, val)
    }

    return conf
}

[aarondl]

Odd. I do the same thing on the latest master branch and I still wind up with nothing being loaded into my struct. There's nowhere you actually define defaults? Or maybe your config file has empty values in it? When I try this AllKeys is totally empty unless I put things in the config file.

[from-nibly]

oh yes sorry just looked at it again. I define each property as "empty string" in my config file (which gives me the list of property names in AllKeys)

[krak3n]

Edits: Syntax

I've also run into this problem so I wrote a quickie (and perhaps dirty) recursive function which manually calls BindEnv on a struct so you can have a partially complete config file:

func BindEnvs(iface interface{}, parts ...string) {
    ifv := reflect.ValueOf(iface)
    ift := reflect.TypeOf(iface)
    for i := 0; i < ift.NumField(); i++ {
        v := ifv.Field(i)
        t := ift.Field(i)
        tv, ok := t.Tag.Lookup("mapstructure")
        if !ok {
            continue
        }
        switch v.Kind() {
        case reflect.Struct:
            BindEnvs(v.Interface(), append(parts, tv)...)
        default:
            viper.BindEnv(strings.Join(append(parts, tv), "."))
        }
    }
}

Usage:

// Config holds configuration
type Config struct {
    Log    Log    `mapstructure:"log"`
    DB     DB     `mapstructure:"mongo"`
    Server Server `mapstructure:"server"`
}

// Log configuration
type Log struct {
    Format string `mapstructure:"format"`
}

// DB configuration
type DB struct {
    URL    string `mapstructure:"url"`
    Name  string `mapstructure:"db"`
}

// Server holds HTTP server configuration
type Server struct {
    Listen          string        `mapstructure:"listen"`
    ShutdownTimeout time.Duration `mapstructure:"shutdown_timeout"`
}

// set up env key replacer and prefix if applicable
func init() {
    viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
    viper.SetEnvPrefix("MY_APP")
}

// Builds config - error handling omitted fore brevity
func config() Config {
    c := Config{} // Populate with sane defaults
    viper.ReadInConfig()
    BindEnvs(c)
    viper.Unmarshal(&c)
    return c
}

// Get and use config
func main() {
    c := config()
    fmt.Println(c.DB.URL)
}

I guess you could also do this in init:

func init() {
    viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
    viper.SetEnvPrefix("MY_APP")
    BindEnvs(Config{})
}

[natefinch]

@krak3n - thanks for the hack to get this to work. I was considering doing the same thing, and happy to have someone else work out the fiddly bits.

For reference, mine defaults to use the lowercase name of the field instead of requiring the mapstructure tag (since almost all of mine are just straight strings):

func bindEnvs(v *viper.Viper, iface interface{}, parts ...string) {
    ifv := reflect.ValueOf(iface)
    ift := reflect.TypeOf(iface)
    for i := 0; i < ift.NumField(); i++ {
        fieldv := ifv.Field(i)
        t := ift.Field(i)
        name := strings.ToLower(t.Name)
        tag, ok := t.Tag.Lookup("mapstructure")
        if ok {
            name = tag
        }
        path := append(parts, name)
        switch fieldv.Kind() {
        case reflect.Struct:
            bindEnvs(v, fieldv.Interface(), path...)
        default:
            v.BindEnv(strings.Join(path, "."))
        }
    }
}

[supershal]

all the options suggested above either required to set config file with empty values or bind each environment variables. These options are prone to errors.

I tried following simple workaround based on comments from this issue. It does not require to bind env values or set defaults or set file with all config set to EMPTY VALUE or reflection.

type Config struct {
    Name string `yaml:"name"`
    Port int `yaml:"port"`  
}
func NewConfig() *Config {
    return &Config{Port: 8080, Name: "foo"}
}

var cfgFile = "/tmp/overwrite.yaml" // contents: name: bar
var config *Config
func initConfigE() error {
    v := viper.New()
    // set default values in viper.
    // Viper needs to know if a key exists in order to override it.
    // https://github.com/spf13/viper/issues/188
    b, err := yaml.Marshal(NewConfig())
    if err != nil {
        return err
    }
    defaultConfig := bytes.NewReader(b)
    v.SetConfigType("yaml")
    if err := v.MergeConfig(defaultConfig); err != nil {
        return err
    }
    // overwrite values from config
    v.SetConfigFile(cfgFile)
    if err := v.MergeInConfig(); err != nil {
        if _, ok := err.(viper.ConfigParseError); ok {
            return err
        }
        // dont return error if file is missing. overwrite file is optional
    }
    // tell viper to overrwire env variables
    v.AutomaticEnv()
    v.SetEnvPrefix(envVarPrefix)
    v.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))

    // refresh configuration with all merged values
    config = &Config{}
    return v.Unmarshal(&config)
}

[krak3n]

@supershal nice approach, nice way to avoid reflection. I'll see if this can work with toml formatted configs tool :+1:

[unmultimedio]

Hello, any updates on this? getKeys func still does not read from ENVwhen no config file or defults are set.

[sagikazarmark]

Please see and follow the issue above. ☝️