Airflow: Config value interpolation not working when set as environment variable

[BasPH]

Apache Airflow version:
1.10.10

Kubernetes version (if you are using kubernetes) (use kubectl version):

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:
I set store_serialized_dags as an environment variable: export AIRFLOW__CORE__STORE_SERIALIZED_DAGS=True.

The default value of store_dag_code is %(store_serialized_dags)s, so I expect it to now also be True. However, it's not (it's False).

What you expected to happen:
I expect store_dag_code to have the same value as store_serialized_dags.

It's because the interpolation is applied to the value set in .cfg, but not when it's set via an env var.

How to reproduce it:

Was able to reproduce it in a test (add this to test_configuration.py):

    def test_interpolation_from_env_var(self):
        """Test if interpolation works when the substituted value is set by an environment variable."""
        test_config = '''[test]
key1 = testme
key2 = %(key1)s
'''
        with mock.patch.dict('os.environ', AIRFLOW__TEST__KEY1="something_else"):
            test_conf = AirflowConfigParser(default_config=test_config)
            key2 = test_conf.get("test", "key2")
            self.assertEqual(key2, "something_else")

So far I haven't been able to make any sense of airflow/configuration.py, to fix it.

Anything else we need to know:

[BasPH]

Dug a bit more into this issue. It seems Airflow builds its configuration in a rather odd way. First there is an AirflowConfigParser, which is the class you get when importing from airflow.configuration import conf. In addition to overriding e.g. ConfigParser.get(), to support fetching configuration from environment variables and others, the AirflowConfigParser also holds an attribute airflow_defaults with is a "standard" ConfigParser by itself, used for looking up default values if none of the custom ways for setting config are used.

This makes it difficult to fix this issue. Interpolation of config values is done in an Interpolation class, which can be overridden to also support fetching values from the custom config. However, since there a (standard) ConfigParser inside the AirflowConfigParser for the default values, we'd have to hack around to also support the custom config fetching ways in there.

All in all, it might be technically possible to fix, but I'd rather opt for rewriting the whole configuration module, to avoid more hacking and take the opportunity to create something simple to understand.

[ashb]

Ooooooh yeah. Tricky one. Tricky tricky one.

[BasPH]

@ashb WDYT about rewriting the entire module? To me the whole module reads like hack on hack on hack. Putting some thought into it and rewriting it completely seems like the better alternative than to add another hack to get this interpolation issue to work.

The coming month I'm completely full so I'm not able to spend any time on it soon though.

[kaxil]

+1 to rewriting for 2.0. However, for 1.10.11, I think I will try to fix or add another _hack_ for store_dag_code - let'see

[potiuk]

I think that one will be the last remaining in 1.10.11 milestone (I margin the other one shortly and it does no have to be released as it is just matter documentation)

[vikramkoka]

@kaxil Does this belong in the 2.0.0 milestone anymore?

[kaxil]

Moved it to 2.1