Dvc: Add --show-json support for dvc exp show

[rogermparent]

I'm a bit light on understanding the actual DVC end of this currently, but for the VS Code plugin and other similar inter-language applications, a machine-friendly interface will be needed to send tables between the two processes, and eventually satisfy iterative/vscode-dvc#1.

Outputting the tables to JSON and parsing DVC shell calls is a simple, drop-in solution that, at the very least, can get the extension usable prior to a more advanced solution like a daemon with IPC.

I'll update this ticket with more details as I get more familiar, and any other relevant info on dvc expo show and how its data is structured is also appreciated.

[efiop]

@pmrowla Could you please add this ASAP to unblock the guys?

[pmrowla]

from the PR:

$ dvc exp show --no-pager
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Experiment            ┃ Created      ┃     auc ┃ featurize.max_features ┃ featurize.ngrams ┃ prepare.seed ┃ prepare.split ┃ train.n_estimators ┃ train.seed ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ workspace             │ -            │ 0.57756 │ 2000                   │ 2                │ 20170428     │ 0.2           │ 50                 │ 20170428   │
│ 11-bigrams-experiment │ Jun 20, 2020 │ 0.61314 │ 1500                   │ 2                │ 20170428     │ 0.2           │ 50                 │ 20170428   │
│ ├── 1df77f7           │ 02:29 PM     │ 0.51676 │ 500                    │ 2                │ 20170428     │ 0.2           │ 50                 │ 20170428   │
│ ├── 1dad0d2           │ 02:29 PM     │ 0.57756 │ 2000                   │ 2                │ 20170428     │ 0.2           │ 50                 │ 20170428   │
│ └── *5a9d6dc          │ 02:29 PM     │       - │ 2000                   │ 2                │ 20170428     │ 0.2           │ 50                 │ 20170428   │
└───────────────────────┴──────────────┴─────────┴────────────────────────┴──────────────────┴──────────────┴───────────────┴────────────────────┴────────────┘

Indented output for dvc exp show --show-json in the same project:

{
  "workspace":{
    "baseline":{
      "timestamp":null,
      "params":{
        "params.yaml":{
          "prepare":{
            "split":0.2,
            "seed":20170428
          },
          "featurize":{
            "max_features":2000,
            "ngrams":2
          },
          "train":{
            "seed":20170428,
            "n_estimators":50
          }
        }
      },
      "queued":false,
      "metrics":{
        "scores.json":{
          "auc":0.5775633054725381
        }
      }
    }
  },
  "9e0cd8a4f9eff4dd184470721da42ac4eb64fd2c":{
    "baseline":{
      "timestamp":"2020-06-20T06:45:46",
      "params":{
        "params.yaml":{
          "prepare":{
            "split":0.2,
            "seed":20170428
          },
          "featurize":{
            "max_features":1500,
            "ngrams":2
          },
          "train":{
            "seed":20170428,
            "n_estimators":50
          }
        }
      },
      "queued":false,
      "metrics":{
        "scores.json":{
          "auc":0.6131382960762474
        }
      },
      "name":"11-bigrams-experiment"
    },
    "1df77f701b1f73e79df68513676631db8bd59636":{
      "timestamp":"2020-10-09T14:29:23",
      "params":{
        "params.yaml":{
          "prepare":{
            "split":0.2,
            "seed":20170428
          },
          "featurize":{
            "max_features":500,
            "ngrams":2
          },
          "train":{
            "seed":20170428,
            "n_estimators":50
          }
        }
      },
      "queued":false,
      "metrics":{
        "scores.json":{
          "auc":0.5167645657589519
        }
      }
    },
    "1dad0d2bd1ee6dced3a676e5cb672522e9e6406a":{
      "timestamp":"2020-10-09T14:29:49",
      "params":{
        "params.yaml":{
          "prepare":{
            "split":0.2,
            "seed":20170428
          },
          "featurize":{
            "max_features":2000,
            "ngrams":2
          },
          "train":{
            "seed":20170428,
            "n_estimators":50
          }
        }
      },
      "queued":false,
      "metrics":{
        "scores.json":{
          "auc":0.5775633054725381
        }
      }
    },
    "5a9d6dc5056efa1310cd9afb61f437680f38b1af":{
      "timestamp":"2020-10-09T14:29:53",
      "params":{
        "params.yaml":{
          "prepare":{
            "split":0.2,
            "seed":20170428
          },
          "featurize":{
            "max_features":2000,
            "ngrams":2
          },
          "train":{
            "seed":20170428,
            "n_estimators":50
          }
        }
      },
      "queued":true
    }
  }
}

[pmrowla]

Output is just nested dictionaries.

• At the top level, the first set of keys is commit SHAs from the original/parent DVC repo or workspace special case for the local workspace.
• At the next level, keys are experiment commit SHAs or baseline special case containing the values from the original parent/baseline commit.
• Within each experiment dict you can have:
  
  
  
  • name - Optional git branch/tag name string
  
  
  • timestamp - ISO8601 formatted timestamp string
  
  
  • params - nested dict of DVC parameters (mapping params_file: {key: value, ...})
  
  
  • metrics - nested dict of DVC metrics (mapping metrics_file: {key: value, ...})
  
  
  • queued - bool, True if experiment is queued and not yet run (same as asterisk from the regular table)
  
  
  • checkpoint_tip - Optional commit SHA string pointing to the tip of this checkpoint experiment branch. Only used for checkpoints, can be used to group checkpoint commits - commits from a checkpoint run will all share the same SHA.
  
  

[pmrowla]

--all-commits/--all-branches/--all-tags can be used as needed to get more commits and derived experiments in the output