Pytorch-lightning: Lightweight Hyperparameter Datastructure

[awaelchli]

🚀 Feature

A simple and flexible way to store hyperparameters in a dict/Namespace-like object.

Motivation

Pitch

An object that behaves like this:

# just like Namespace
hparams = Hyperparameters(x=1, y=2)  
# or from a dict
hparams = Hyperparameters({"x": 1, "y": 2})  
# it could support nesting
hparams = Hyperparameters({"x": 1, "y": {"a": 3}}) 

# Namespace-like look up
x = hparams.x
a = hparams.y.a
# or like a dict
x = hparams["x"]
a = hparams["y"]["a"]

# we will have to check for invalid keys
hparams["batch_size"]  # ok
hparams["batch-size"]  # error
hparams["batch size"]  # error

Optional features:

# could support reading from files
# useful for checkpoint loading
hparams = Hyperparameters.from_yaml("file.yml")

# could support flattening/sanitizing nested structure
# useful for logging to TensorBoard
# or to make it pickleable
clean = hparams.flatten().sanitize()

# note that these internal methods will prevent these keys:
hparams.flatten = True  # Problem, because not hyperparameter!

Pro:

• Behaves like dict and Namespace in one, so it will work out of the box with PL since internally we treat it as dict anyway (saving checkpoint).

Contra:

• Any convenience methods/attributes we add to the class will collide with hyperparameters names. See example above.
• Need to spend extra effort making sure user does not pick bad dict keys

Considerations

• "Hyperparameters" may be a too specific name for such a general datastructure.

Additional context

Discussed on slack and idea has popped up in other isses as well.

Related to #1841, #1735 and would solve #1737

[awaelchli]

Also post more ideas here.

[Borda]

About the name we can have inside auto replacement of any separator (, ,) by _

[yukw777]

Should we just use OmegaConf for this? It might make it easier to integrate Hydra down the line too.

[miriaford]

Completely agree with @yukw777 . OmegaConf does all of the above and better, PL shouldn't reinvent the wheel.

[awaelchli]

I guess the idea was to provide a basic datastructure that works without dependency on another library. A simple wheel that rolls and does nothing more, created with tools that we already have :)

[yukw777]

Maybe we could use ConfigParser? It's part of the standard library, so no external dependency. Its interface is not as simple as what @awaelchli has envisioned though.

[Borda]

We need to do our arg casting so it would be nice to wrap it in one object
https://github.com/PyTorchLightning/pytorch-lightning/blob/981169cacc5da17af8d796d94b747c17566ef0bc/pytorch_lightning/trainer/trainer.py#L723

[omry]

We need to do our arg casting so it would be nice to wrap it in one object

https://github.com/PyTorchLightning/pytorch-lightning/blob/981169cacc5da17af8d796d94b747c17566ef0bc/pytorch_lightning/trainer/trainer.py#L723

What do you get from the Namespace object that you are not getting from OmegaConf DictConfig?

In other words, DictConfig is most likely a drop-in replacement to Namespace. (with the exception of explicit type checks).

[awaelchli]

I didn't know OmegaConf when I opened the issue.
Anyway, the PL team decided to simplify the way hyperparameters are passed into the model so there is no need for an object like this anymore. Closing this. Thanks for the discussion.

[drozzy]

Maybe pydantic? (just my 2c).