Hydra: Allow configuring the search path via the config file

[omry]

Proposal:

hydra:
  search_path:
     - path: /foo                 # support /path or pkg:// style.
       action: append|prepand     # default to append
       anchor: hydra              # optional anchor

Consider implementing this in a recursive manner, where any included config file can contribute to the search path, but initially this should work for the primary config file.

[asherp]

Not sure if this is the feature I'm waiting for, but I want my users to install my app and easily point to a new config directory that overrides my defaults. My app is based on the hydra_app_example. Please advise.

[asherp]

Ok, I think I figured it out. First add a default search path to config.yaml:

search_path: config.yaml

Then merge it using OmegaConf

import hydra
from omegaconf import OmegaConf

@hydra.main(config_path='conf/config.yaml', strict = False)
def main(cfg):
    if cfg.search_path is not None:
        override_path = hydra.utils.to_absolute_path(cfg.search_path)
        override_conf = OmegaConf.load(override_path)
        cfg = OmegaConf.merge(cfg, override_conf)

Now the user can run your app from any directory containing a config.yaml, or specify their own:

myapp search_path=path/to/user/config.yaml

[omry]

That's a pretty good workaround. I would call this something like config_override and not search_path, as search path is an internal concept in Hydra that will eventually be exposed.

Asher, can you share a bit about your project? I am curious about how people are using Hydra and for what.

[asherp]

@omry It's for a project called Kamodo, which is a functional, publication-focused API for space weather models and data (though it can be used for much more!). Kamodo handles function composition, unit conversion, and automatic plot generation via function inspection, which should minimize the effort needed to create analysis pipelines and perform data-model comparisons. Kamodo is built with sympy and plotly.

Most of our science users are comfortable with command-line but are inexperienced with modern programing languages. The CLI for kamodo will allow them to configure their data sources and any equations they want to use for post processing and quickly generate plots. Our users will often test different models or data sources for science relevance and will want to easily reproduce those results, so Hydra's ability to compose several overlapping configurations and have timestamped output directories is perfect for our use case.

Github - https://github.com/nasa/Kamodo
Official site - https://ccmc.gsfc.nasa.gov/Kamodo/

I'm also considering Hydra for my other project Hourly, which uses git "clock in" and "clock out" messages to track hours worked, generate time sheets, and produce work charts. Since I started freelancing, I've logged at least 1000 sessions across 5 different projects using Hourly.

Currently, Hourly's cli is based on click, but I need to provide default functionality specific to a particular repo (to ignore errant clock-in/outs, etc). I'd also like to generate time sheets from related projects, so config composition would come in handy. Finally, I'm usually the only one who uses hourly, but eventually I'll need to iterate over all contributors to a project (assuming I get more users!).

Project page - https://asherp.github.io/hourly/index.html
Github - https://github.com/asherp/hourly

[omry]

Kamodo looks awesome, and I am really happy to start seeing serious projects that are making use of Hydra.

A few notes:

• Can you add Hydra as a dependency of Kamodo (in setup.py/cfg or requirements.txt). this will allow GitHub to detect it as a project using Hydra.
• Did you see the new compose API? it lets you compose configs with Hydra anywhere, not just in the main. This is specifically useful for Jupyter notebooks. here is an example (you can launch the binder to play with it online).
• Take a look at Pytorch, it might be very useful for Kamodo as an alternative to numpy. It allows highly optimized vector/matrix operations for both CPU and GPU, as well as automatic differentiation.
• Hourly looks like a nice and useful project for freelancers, it would be great if you use Hydra there as well, and I would love to know about your experience porting from Click to Hydra. Click is super popular but I think Hydra is much more powerful and it can probably be used as a replacement for Click for most use cases.
• I bet NASA got some serious compute clusters. While there are no official public plugins for it yet, we are using Hydra at FAIR to launch to our internal cluster. Hydra has a plugin mechanism that can support any cluster (including private clusters with non public APIs).
  I would be happy to guide you in writing a plugin that will allow you to use Hydra to launch to your internal cluster from the command line.

I am considering creating a "Who is using Hydra" section in the docs, would you agree to write a short testimonial saying why you find it useful?

[omry]

One more thing:
As a framework, Kamodo might want to provide some configuration to its users.
For example, some default configurations to instantiate commonly used models etc.
The best way to do it with Hydra is to add those configs to the search path.

You can do it programmatically and automatically via the search plugin API.
I don't yet have a super clean example for this in hydra/plugins/examples, but the colorlog plugin comes very close.
Essentially this is all its doing. check it out here.

[Stonesjtu]

I have the similar use cases as @asherp .

• The major interface of my program is in binary form, most users don't write any python codes.
• The default value is defined in structured config rather than a default config file
• Users write their own config file in their work dir, and runs through python -m xxx.main

@omry
I think the current version 1.0rc supports specify config_path, but the path is resolved relative to the main.py file, so I have to specify absolute path to make it work.

Can we resolve the config_path relative to the $pwd, which makes more sense in CLI cases

[omry]

You can probably achieve that now via a config searchpath plugin.
example.

[omry]

by the way, take a look at the Application Packaging page.

1. You CAN use config files with your application even if it's installed.
2. You can get a native command to execute it instead of the clunky python -m module.

[odelalleau]

I have a related use case I'm not sure can be supported easily right now:

I would like to have an application coming with its own config file (packaged with the app, not to be edited), that would also let the user override some parameters through their own config file (stored in the directory where they use the app).

I haven't looked closely yet at the config search path plugin, but reading the doc it sounds like it's going to stop at the first matching file (instead of merging configurations from multiple files).
It looks like I could use the compose API but that would require some extra manual work.

Just curious if there's an option I'm not seeing, or something I misunderstood?

[jieru-hu]

I have a related use case I'm not sure can be supported easily right now:
I would like to have an application coming with its own config file (packaged with the app, not to be edited), that would also let the user override some parameters through their own config file (stored in the directory where they use the app).
I haven't looked closely yet at the config search path plugin, but reading the doc it sounds like it's going to stop at the first matching file (instead of merging configurations from multiple files).
It looks like I could use the compose API but that would require some extra manual work.
Just curious if there's an option I'm not seeing, or something I misunderstood?

It seems like something Hydra may already support? As an example, all of Hydra's plugin gets their config added by SearchPathPlugin, example here: https://github.com/facebookresearch/hydra/blob/master/plugins/hydra_colorlog/hydra_plugins/hydra_colorlog/colorlog.py#L6-L9
I imagine in your case you can create multiple SearchPathPlugins pointing all the configs you are interested in and have Hydra merge them all.
It would be helpful if you can provide a minimal example of what the config structure would look like, that way I may better help you :)

[odelalleau]

I imagine in your case you can create multiple SearchPathPlugins pointing all the configs you are interested in and have Hydra merge them all.

Maybe -- I haven't looked closely at it, the reason why I thought it wouldn't work is because the doc says "When a config is requested, The first matching config in the search path is used", suggesting that there is no merging (just use whichever is found first). But I may not be understanding properly.

As an example I may want to have a directory structure like this:

├── my_app
│   └── config
│       └── config.yaml
└── user_code
    └── .my_app
        └── config.yaml

And I would like both config files to be merged (in my case the config found under "user_code" would only modify keys already existing in the "my_app" config, not add new keys).

I managed to make it work with the Compose API but (1) I had to drop @hydra.main(), and (2) it's a bit cumbersome because I need to declare my defaults list in the "user_code" config even if I would like to reuse the one from "my_app". So overall not so convenient...

[omry]

Configuring the defaults list (the topic of this issue) has little to do with your question.

First of all, if you have two configs in the same name and the same group, only one will be visible.
This is not changing (and in fact enabling replacing an existing be prepending a searchpath containing a config with the same name.

In 1.0, you only get one defaults list - in the primary config.
That primary config can refer to a file from your app:

library/conf
  app_config.yaml
  ...

user_app/conf
  user_overrides.yaml # contains things the user want to override in app_config.
  user_conf.yaml 
    defaults:
      - app_config
      - user_overrides  # this is needed by default the composition order is placing this file before things in the defaults list.

The chunkiness in this solution will be addressed once recursive default supports coming in 1.1.
Refer to this design doc to see what's coming. Feedback is welcome.

[odelalleau]

Configuring the defaults list (the topic of this issue) has little to do with your question.

It may not have been what you had in mind when you created this issue, but https://github.com/facebookresearch/hydra/issues/274#issuecomment-568036695 is pretty close to what I want to achieve.

The chunkiness in this solution will be addressed once recursive default supports coming in 1.1.
Refer to this design doc to see what's coming. Feedback is welcome.

I had a look, I may not understand it all, but I'm not sure it would work for my use case.

To make things more concrete, let's take this example from Hydra docs: https://hydra.cc/docs/next/tutorials/structured_config/static_schema
Let's say I have a packaged app with config files as shown in this example. I would like to let a user of my app create their own config file(s) to override only some of these settings. For instance they could create a file .my_app/config/db/staging.yaml that would contain:

host: mysql007.staging

And then by running python /path/to/my_app.py --config-dir .my_app/config, the app would run with the user-specified host when using the staging db, instead of the default one.

The reason why I'm doubtful about recursive defaults enabling this is because of this example in the doc you shared:

defaults:
 - d2/model: resnet50  # model from an installed framework
 - _self_
 - db: mysql

  # This will override the value provided by the framework

d2:
  model:
    num_layers: 100

Here, num_layers will be set to 100 regardless of which model is used, but what if I want to set it to 100 for resnet50 and to 200 for resnet101?

[odelalleau]

Just an additional thought: if there's a way to do it via config files it'd make sense to also be able to do it via a command line override. Something like:

python /path/to/my_app.py db/staging.host=mysql007.staging

[omry]

The reason why I'm doubtful about recursive defaults enabling this is because of this example in the doc you shared:
That example is just one of many ways to use recursive defaults.

Here is another example from the docs of the next version (both the logic this is documenting and the docs themselves are work in progress):
https://hydra.cc/docs/next/advanced/defaults_list#config-inheritance-via-composition

[omry]

Please move the discussion to a dedicated issue.
It has nothing at all to do with this issue.

[odelalleau]

Please move the discussion to a dedicated issue.
It has nothing at all to do with this issue.

Sorry for the hijack :) Thanks for the pointers, I think I have a better understanding now of how things currently are and how they will be in the future. I believe I can solve my problem with a slightly different approach which makes more sense in my situation, so I won't open another issue.