Dvc: experiments: consider using hydra for CLI params syntax

[pmrowla]

Follow up to #4780

python-benedict for parsing raw yaml syntax works for the moment, but as noted by @skshetry something like hydra would probably be more user-friendly

[sjawhar]

Just to clarify: python-benedict does not parse raw yaml. That is being handled by ruamel.YAML here:
https://github.com/iterative/dvc/blob/251a3269c2c5f7e39938274b06fca319de1bf79b/dvc/repo/experiments/run.py#L22-L30
https://github.com/iterative/dvc/blob/251a3269c2c5f7e39938274b06fca319de1bf79b/dvc/utils/serialize/_yaml.py#L69-L70

Benedit handles putting the parsed param value into the params dict using the dot-separated key path. Hydra definitely looks awesome, but it looks like its use case is a bit different than what you're looking for here (and seems to only have experimental support for what this issue would need: https://hydra.cc/docs/next/experimental/compose_api)

[johnnychen94]

Here's an example. To properly set up GitLab CI, I need to modify parameters with a much much smaller network and dataset, then I ran into this...

# .gitlab-ci.yml
test:gpu:
  stage: build
  variables:
    julia_version: "1"
  extends:
    - .julia
  script:
    - pip3 install dvc[s3] -q
    - dvc version
    - dvc exp run --params models.gaussian_15.data.max_num_patches=1000,models.gaussian_15.train.model_width=8,models.gaussian_15.train.model_depth=5,models.gaussian_15.train.batch_size=32,models.gaussian_15.train.epochs=2,models.gaussian_25.data.max_num_patches=1000,models.gaussian_25.train.model_width=8,models.gaussian_25.train.model_depth=5,models.gaussian_25.train.batch_size=32,models.gaussian_25.train.epochs=2,models.gaussian_50.data.max_num_patches=1000,models.gaussian_50.train.model_width=8,models.gaussian_50.train.model_depth=5,models.gaussian_50.train.batch_size=32,models.gaussian_50.train.epochs=2

Which would be much easier to read if we can reformat them into:

  # will insert one extra space between each param, which doesn't follow the current dvc params syntax
  scripts:
    - dvc exp run --params
      models.gaussian_15.data.max_num_patches=1000,
      models.gaussian_15.train.model_width=8,
      models.gaussian_15.train.model_depth=5,
      models.gaussian_15.train.batch_size=32,
      models.gaussian_15.train.epochs=2,
      models.gaussian_25.data.max_num_patches=1000,
      models.gaussian_25.train.model_width=8,
      models.gaussian_25.train.model_depth=5,
      models.gaussian_25.train.batch_size=32,
      models.gaussian_25.train.epochs=2,
      models.gaussian_50.data.max_num_patches=1000,
      models.gaussian_50.train.model_width=8,
      models.gaussian_50.train.model_depth=5,
      models.gaussian_50.train.batch_size=32,
      models.gaussian_50.train.epochs=2

dvc currently failed to parse the params input because one extra whitespace is inserted when flattening multiline scripts into oneliner...

My current "workaround" to this is to use sed to do a hot replacement, which is okay since it's a sandbox environment.

The params.yaml are written so as to fully utilize the parameterization feature while still preserving flexibility. If it helps, here's my dvc.yaml and params.yaml files:

dvc.yaml and params.yaml

# dvc.yaml

stages:
  extract_data:
    foreach: ${datasets}
    do:
      cmd: mkdir -p ${item.path} && unzip -q ${item.src} "${item.name}/*" -d tmp && mv tmp/${item.name}/* ${item.path}
      deps:
      - ${item.src}
      outs:
      - ${item.path}
  prepare:
    foreach: ${models}
    do:
      cmd: >-
        julia --color=yes --startup=no --project=prepare -e "using Pkg; Pkg.instantiate()" &&
        julia --color=yes --startup=no --project=prepare \
          prepare/main.jl data/train/rawdata/ data/train/prepared \
          --noise-level ${item.data.noise_level} \
          --patch-stride ${item.data.patch_stride} \
          --patch-size ${item.data.patch_size} \
          --max-num-patches ${item.data.max_num_patches}
      deps:
      - data/train/rawdata
      - prepare
      outs:
      - data/train/prepared/${item.name}.h5
  train:
    foreach: ${models}
    do:
      cmd: >-
        julia --color=yes --startup=no --project=train -e "using Pkg; Pkg.instantiate()" &&
        julia --color=yes --startup=no --project=train \
          train/main.jl data/train/prepared/${item.name}.h5 models/${item.name} \
          --epochs ${item.train.epochs} \
          --batch-size ${item.train.batch_size} \
          --initial-lr ${item.train.initial_learning_rate} \
          --lr-decay-ratio ${item.train.learning_rate_decay_ratio} \
          --lr-decay-freq ${item.train.learning_rate_decay_frequency} \
          --model-width ${item.train.model_width} \
          --model-depth ${item.train.model_depth} \
          --padding-size ${item.train.padding_size} \
          --kernel-size ${item.train.kernel_size} \
          --batch-size ${item.train.batch_size} \
          --log-freq ${item.train.log_frequency} \
          --use-gpu ${item.train.use_gpu}
      deps:
      - data/train/prepared/${item.name}.h5
      - train
      outs:
      - models/${item.name}
  evaluate:
    foreach: ${testcases}
    do:
      cmd: >-
        julia --color=yes --startup=no --project=evaluate -e "using Pkg; Pkg.instantiate()" &&
        julia --color=yes --startup=no --project=evaluate evaluate/main.jl \
          models/${item.model_name} \
          data/test/${item.dataset} \
          results/${item.model_name}/${item.dataset} \
          ${item.noise_level}
      deps:
      - data/test/${item.dataset}
      - models/${item.model_name}
      - evaluate
      outs:
      - results/${item.model_name}/${item.dataset}/processed_images
      plots:
      - results/${item.model_name}/${item.dataset}/checkpoints.csv
      - results/${item.model_name}/${item.dataset}/metrics.csv
  summary:
    cmd: >-
      julia --color=yes --startup=no --project=evaluate -e "using Pkg; Pkg.instantiate()" &&
      julia --color=yes --startup=no --project=evaluate evaluate/summary.jl results metrics.json
    deps:
    - results
    - evaluate
    metrics:
    - metrics.json

# params.yaml
models:
  gaussian_15:
    name: gaussian_15
    data:
      noise_type: gaussian
      noise_level: 15
      patch_size: 41
      patch_stride: 10
      max_num_patches: 400_000
    train:
      epochs: 50
      initial_learning_rate: 0.001
      learning_rate_decay_ratio: 0.1
      learning_rate_decay_frequency: 30
      model_width: 64
      model_depth: 17
      padding_size: 1
      kernel_size: 3
      batch_size: 128
      log_frequency: 20
      use_gpu: 'true'

  gaussian_25:
    name: gaussian_25
    data:
      noise_type: gaussian
      noise_level: 25
      patch_size: 41
      patch_stride: 10
      max_num_patches: 400_000
    train:
      epochs: 50
      initial_learning_rate: 0.001
      learning_rate_decay_ratio: 0.1
      learning_rate_decay_frequency: 30
      model_width: 64
      model_depth: 17
      padding_size: 1
      kernel_size: 3
      batch_size: 128
      log_frequency: 20
      use_gpu: 'true'

  gaussian_50:
    name: gaussian_50
    data:
      noise_type: gaussian
      noise_level: 50
      patch_size: 41
      patch_stride: 10
      max_num_patches: 400_000
    train:
      epochs: 50
      initial_learning_rate: 0.001
      learning_rate_decay_ratio: 0.1
      learning_rate_decay_frequency: 30
      model_width: 64
      model_depth: 17
      padding_size: 1
      kernel_size: 3
      batch_size: 128
      log_frequency: 20
      use_gpu: 'true'


datasets:
  train:
    name: train
    src: data/DnCNN.zip
    path: data/train/rawdata
  Set12:
    name: Set12
    src: data/DnCNN.zip
    path: data/test/Set12
  Set68:
    name: Set68
    src: data/DnCNN.zip
    path: data/test/Set68


testcases:
  set12_gaussian_15:
    model_name: gaussian_15
    noise_level: 15
    dataset: Set12
  set12_gaussian_25:
    model_name: gaussian_25
    noise_level: 25
    dataset: Set12
  set12_gaussian_50:
    model_name: gaussian_50
    noise_level: 50
    dataset: Set12

  set68_gaussian_15:
    model_name: gaussian_15
    noise_level: 15
    dataset: Set68
  set68_gaussian_25:
    model_name: gaussian_25
    noise_level: 25
    dataset: Set68
  set68_gaussian_50:
    model_name: gaussian_50
    noise_level: 50
    dataset: Set68

[pmrowla]

For running an experiment with this many modified parameters, I would recommend editing params.yaml directly rather than using the dvc exp run --params option. dvc exp run will include any uncommitted changes to params.yaml in the experiment.

Whether you do this via sed or a separate script that writes params.yaml would be up to you.

[johnnychen94]

Does it sound good to have dvc exp run --params test_params.yaml which loads all the params in that file?

[turian]

I also would be interested in Hydra support for DVC. I am using Hydra but would like to use DVC too.