Hydra: [Feature Request] Use JSON as an alternative to YAML

[faroit]

🚀 Feature Request

hydra seem to only support YAML files for configurations. For various reasons, many people prefer other formats such as JSON.

Motivation

YAML and JSON are very similar and support for most data types are present in both. This allows to implement a JSON parser that supports the same features as the YAML version. Furthermore JSON parsers in Python are typically faster than YAML.

[omry]

Hi, thanks for the feature request.
YAML is compatible which JSON: valid JSON is (almost always) valid YAML.
this means you can put json strings inside YAML files and they will be parsed properly.

When composing configurations with your defaults:

defaults:
  foo: bar

Hydra will look for foo/bar.yaml in your config search path. this means adding additional extensions would involve checking for all supported extensions in some order. I don't think this is worth the overhead and the extra complexity at this point.

I know existing systems may be using JSON. In my opinion YAML is a better format for configuration and I would like to keep it as the primary supported one.

Hydra 1.0.0 will add configuration source abstraction which can come handy here to add opt-in support for json files.
I will keep this open for now to look into that possibility later, but in general I am not thrilled about adding json support to hydra-core.

[faroit]

Hydra 1.0.0 will add configuration source abstraction which can come handy here to add opt-in support for json files.
I will keep this open for now to look into that possibility later, but in general I am not thrilled about adding json support to hydra-core.

sounds good to me.

[omry]

Thinking about it some more config source may be inappropriate (it adds support for additional locations for configs, not for additional types - which are orthogonal).
the rest stands. I will consider this but it's not very likely.

[festeh]

There's also a Jsonnet language, which is compared to YAML here. For example, it's used in AllenNLP library. Would it be possible to let user write custom data loading functions in configuration source abstraction in order to support such files?

[omry]

Thanks for the pointer to Jsonnet.
There are a few things here:

1. OmegaConf is a data model and not a language parser. it does add runtime features like interpolations and resolvers (and obviously merging of objects).
   It is somewhat agnostic to the underlying format, but at the moment there is built in support for YAML and dataclasses. (new in OmegaConf 2.0.0, not yet in Hydra).
   I am planning to add support for simple expressions to OmegaConf here. Such support would not work properly with Jsonnnet interpolation support means this has to happen at runtime.

2. Hydra got some magical behavior that are making it hard, I described those in the comment above. (defaults list not containing the file extension and what the implications are).

3. Frameworks alike AllenNLP will have to be adapted to use Hydra to benefit from it. some of those changes involves potentially using a different configuration language. but there could be other deeper changes.

I am keeping this feature request option to potentially implement support for other config formats in the future (user provided or not, this is challenging).
As I said this is a low priority.

[faroit]

I know existing systems may be using JSON. In my opinion YAML is a better format for configuration and I would like to keep it as the primary supported one.

YAML may indeed be a better format for a human readable config format. However, I believe that the fact that it cannot be easily parsed and validated in python without installing additional dependencies (also true for other languages), makes it significantly harder to use in real world.

[omry]

@faroit, This is a very weak justification in the context of Hydra.
If you want to use Hydra, you can use it with pyyaml installed.
Hydra is a MUCH bigger dependency to add than pyyaml. by using Hydra you buy into accepting it's dependencies.

[faroit]

I don't understand why Hydra would be heavy on dependencies. I only see OmegaConf which in itself only depends on PyYAML - hence with json, Hydra could be almost pure python.

If you want to use Hydra, you can use it with YAML

It seems that json support is not seen as a viable option for this project and feedback is not appreciated so I'm closing this.

[omry]

@faroit I am not referring to the request but to the argument that yaml as a dependency is a problem for Hydra:
pyyaml is a hard dependency for OmegaConf, OmegaConf is a hard dependency for Hydra.

if you want to use Hydra, you will have a pyyaml dependency even this is implemented and there is native support for json files.
I see no reason to work toward eliminating this dependency.

The reason I commented here is that I saw your last comment which I missed before when I went through all the existing tasks.
Allow me to be the one closing issues. Refer to my earlier comment for reasoning as to why this is still open.