Viper: Does viper require a config file in order to use ENV variables?

[jsirianni]

Hi, I have an application that can use both a config file and env variable.

My initConfig() method looks like this:

if err := viper.ReadInConfig(); err == nil {
    fmt.Println("Using config file:", viper.ConfigFileUsed())
} else {
    fmt.Println("Config file not found:", cfgFile)
}

viper.SetEnvPrefix("GC")
viper.AutomaticEnv()

err := viper.Unmarshal(&config)
if err != nil {
    fmt.Println(err.Error())
    os.Exit(1)
}

unmarshal drops the config into a struct, and it works well.

I am facing an issue where if my config file does not exist (or is empty), viper seems to completely ignore any environment variables. I have verified that viper can see the env variables, by printing them to standard out. My workaround is to deploy the config file, with all the correct keys but no values. Viper then correctly overrides the config file with my environment variables.

Am I wrong to assume that Viper should be able to use environment variables when no config file is found?

[matejkramny]

+1 just spent a lot of time finding why env variables don't bind to the config file. Simply adding the key to the yaml file made it work

[jsirianni]

Looking at viper.go, I find this comment:

// Viper is a prioritized configuration registry. It
// maintains a set of configuration sources, fetches
// values to populate those, and provides them according
// to the source's priority.
// The priority of the sources is the following:
// 1. overrides
// 2. flags
// 3. env. variables
// 4. config file
// 5. key/value store
// 6. defaults
//
// For example, if values from the following sources were loaded:
//
//  Defaults : {
//      "secret": "",
//      "user": "default",
//      "endpoint": "https://localhost"
//  }
//  Config : {
//      "user": "root"
//      "secret": "defaultsecret"
//  }
//  Env : {
//      "secret": "somesecretkey"
//  }
//
// The resulting config will have the following values:
//
//  {
//      "secret": "somesecretkey",
//      "user": "root",
//      "endpoint": "https://localhost"
//  }

The example shows secret being in all three configuration sources, and being overridden accordingly. What they do not seem to mention is that config is required. I find it hard to believe that it should be this way, considering values found in a config work just fine even if you have not specified defaults anywhere.

It would be nice to know if this is desired behavior, so I can build my applications accordingly. My main driver for avoiding a configuration file is that I do not wish to have configurations baked into my docker containers. I can deploy empty configurations, however, that is an additional (and unnecessary) stop in my build process.

[renannprado]

@jsirianni totally agree with you. I was surprised actually that I'm forced to configure AddConfigPath, SetConfigName, SetConfigType, AutomaticEnv and set a default value like this (viper.SetDefault("TEST", "")) or via file.
IMO the Unmarshal should bind out of the box to the environment variables, i.e. running AutomaticEnv should be enough for Unmarshal to work properly. I'm not sure if easy or feasible (given the maintainers want to keep backwards compatibility probably), but I'd say this sounds like the logical behavior to me.

[sagikazarmark]

The problem is actually Unmarshal in this case. If you don't set a config and call viper.Get* you will receive the desired value. The AutomaticEnv feature (when turned on) will check the environment variable even if it isn't registered.

Unmarshal works a bit differently: it collects the registered keys and passes them with their values to the mapstructure package which binds the values to the appropriate keys of the structure. In this case there is no "request" nothing to compare against.

I agree that this is not optimal, but given Unmarshal was added later I can hardly categorize this as a regression.

Unfortunately it's not easy to fix as environment variables are transformed in various ways:

• a prefix is added
• env key replacer is applied

Even if the prefix can be transformed back, the env key replacer cannot be "reversed".

~It would probably worth checking to see if somehow we can get the keys from mapstructure or hook into the process there somehow.~ (Update: unfortunately keys are not available in the decode hooks)

(An alternative could be doing a "reverse decode": grab an empty struct and "decode" it into a map[string]interface{}. Recursively go through the map to collect the keys, flatten them and bind those keys. It's not a nice solution, but at least you can somewhat automate it.)

In the meantime, using BindEnv manually is the only way unfortunately (or manual unmarshaling).

Normally I define default configuration wherever possible, so it's really just those few keys I have to manually bind (database host, port, user, password, name; redis host and password).

[sagikazarmark]

Please see and follow the issue above. ☝️

[Niondir]

I'm modeling my whole config as struct. To make all environment variables work, even when the config file does not contain the key I first load my defaults from a struct like this:

func Init() {
    initDefaultValues()
    /* ... */
    viper.AutomaticEnv()
    /* ... */
    viper.ReadInConfig()
}


func initDefaultValues() {
    defaultConfig = &Config{}
    defaultConfig.Server.Port = 1272

    loadConfigFromStruct(defaultConfig)
}

func loadConfigFromStruct(cfg interface{}) {
    cfgMap := make(map[string]interface{})
    err := mapstructure.Decode(cfg, &cfgMap)
    if err != nil {
        logrus.WithError(err).Fatalf("failed to marshal default config")
        os.Exit(1)
    }

    cfgJsonBytes, err := json.Marshal(&cfgMap)
    if err != nil {
        logrus.WithError(err).Fatalf("failed to marshal default config")
        os.Exit(1)
    }

    viper.SetConfigType("json")
    err = viper.ReadConfig(bytes.NewReader(cfgJsonBytes))
    if err != nil {
        logrus.WithError(err).Fatalf("failed to load default config")
        os.Exit(1)
    }
}