Tensorboard: Feature request: plot different scalar summaries together based on regexes

[leandro-gracia-gil]

Currently, combining multiple summaries into one sample plot requires the use of different summary writers with summaries coming from the same graph variable, but optionally different inputs like train and test sets.

However, the requirement of coming from the same variable prevents merging plots in cases where this is not possible. For example, the A3C reinforcement learning algorithm uses multiple workers running structurally identical graphs, but each with its own set of trainable variables. This leads to different output variables regardless of the input used, and therefore prevents plotting together the outputs of multiple workers despite they conceptually represent the same.

I'd like to propose a simple way to solve this. As an extension of the current behavior, would it be possible to provide a set of regexes to Tensorboard such that any summaries matching one of them are plotted together? Something like "plot regexes", where each regex defines what gets merged in one plot. This would produce a simple yet very flexible way to manage scalar plots beyond the name uniqueness restrictions that Tensorflow imposes.

Another perhaps slightly more complex but also more powerful option would be to define replacement regexes, which then would merge in a same plot everything that has the same name after applying the replacements. For example, the case of the A3C algorithm above would be solved with something as simple as 's/worker_[0-9]+//' to remove any worker-specific prefixes and merge any worker plots together. In fact, if we consider regexes of the form 's/filter_regex/plot_name/' what we get is a generalization of the proposal above, allowing to combine multiple regexes into one same plot if desired.

Any other suggestions or ideas are most welcome.

[wchargin]

Yes, this is possible, and preprocessing the tag space by taking its quotient set under a regex-defined equivalence relation sounds like a reasonable idea.

This isn't a priority for us right now, but if you send us a PR we'd be happy to take a look! The main file of interest will surely be tf-scalar-dashboard.html; the accompanying tf-scalar-chart.html may also be relevant. You might also want to take a look at categorizationUtils.ts, though I'd be more hesitant to change that file.

You should be aware that difficulties arise when the set of charts displayed is not static due to design flaws in Polymer; https://github.com/tensorflow/tensorboard/issues/170 describes some of these issues. However, https://github.com/tensorflow/tensorboard/pull/189 and https://github.com/tensorflow/tensorboard/pull/204 should help hide this complexity for the specific case of the scalar dashboard, so you should only need to worry about this if you start seeing bizarre behavior.

[leandro-gracia-gil]

I'm pretty busy at the moment, but if I happen to have the time I might give it a try. Any suggestions / recommendations about the UI to enter the regexes? Like for example, how to do it in a way that avoids confusion with existing regex filters, how should we support introducing regex replacement rules, and so on.

[chihuahua]

Proposal: The Custom Scalars Plugin

Overview

We propose a plugin (short name custom-scalars) that lets the user make several custom scalar plots. The lines shown in each plot will depend on

• The runs selected on the left hand side (per the practice in other plugins).
• The regex for filtering tags specific to each plot.

The need for this plugin is noted both above as well as in #597 - users would like to customize which tags appear in individual plots. Currently, the scalars dashboard only shows 1 plot per tag. Several researchers would like to try out this plugin.

Implementation Details

Dashboard Design

The dashboard will show a "+ Add Plot" button at the top. When the user clicks it, an empty chart is added along with a regex input field on top of it. When the user inputs a regex, the chart shows lines for tags that are captured by the regex. A legend below clarifies which tags are captured. Tags within the same run are differentiated by different markers. Runs are differentiated by colors as expected.

A low-fidelity paper prototype:

As you see, this plugin does not rely on runs for categorization of plots.

Frontend Implementation

We refactor the tf-line-chart-data-loader component to accept multiple tags:
https://github.com/tensorflow/tensorboard/blob/d1a5098aa2884b6b8e6194f30f4138e8ee2be2c7/tensorboard/components/tf_line_chart_data_loader/tf-line-chart-data-loader.html#L114

We refactor vz-line-chart to assign a new marker shape whenever the same color is used more than once (which should probably already be the case).
https://github.com/tensorflow/tensorboard/blob/d1a5098aa2884b6b8e6194f30f4138e8ee2be2c7/tensorboard/components/vz_line_chart/vz-line-chart.ts#L669

For each plot, we pass a special color scale from the plugin to tf-line-chart-data-loader that assigns the same color for each run. And then, we construct each data series name by passing in the name of the tag suffixed by (run: foo). The data series name will no longer be just the name of the run.

Backend Design

The constructor of the custom-scalars plugin will take a scalars plugin and use the latter to serve specific scalars data to the frontend. This is similar to how the distributions plugin takes a histograms plugin.

Routes

/tags: This route simply returns a JSON list of all tags with scalar data. The frontend will apply regexes to that list.
/scalar_data?run=foo&tag=bar: This route returns a list of scalar data given a run and tag. This route will be called several times by the frontend depending on how many tags a user's regex captures.

[georgedahl]

I would find this useful. Especially if there was a way to maintain the grouping using some config file so as soon as I launched tensorboard for my runs the plots I wanted were already there (set up once and save a config.json file or something?).

[jart]

Maybe have a --scalars=foo.pbtxt flag where foo.pbtxt is like:

chart: {
  title: "My Custom Plot"
  tag: "foo/bar"
  tag: "lol/cat"
}
chart: {
  title: "Another Custom Plot"
  tag: "foo/bar"
  tag: "lol/cat"
}

Then the existing scalars dashboard, if it gets that flag, just puts those custom charts at the top of the dashboard.

[chihuahua]

Hmm, yeah, how about a JSON file --scalars=foo.json that lists regexes, ie

[
  {
    "title": "Happiness over Time",
    "tagRegex": "/lol/bar*"
  },
  {
    "title": "Plot Foo",
    "tagRegex": "/(meep|quux)/bar*"
  }
]

[wchargin]

Protobuf has stability and documentation advantages over JSON. You can look at a proto definition and see the exact grammar for the configuration file. The same is not true for JSON, and I'm not aware of any advantages that JSON has over protobuf in this case.

[chihuahua]

Is it easy for open source users to pick up pbtxt? The graph has taken a pbtxt for a long time, and we've encountered problems with people trying to manually create the pbtxt and failing due to annoying subtle issues (missing spaces / tabs). The formatting has to be just right.

If there were a python-native parser for pbtxt, that would mitigate this problem.

[jart]

I've always found it far more challenging to make sense of a schemaless JSON configuration (cough cough Kubernetes) than to work out differences in syntax. Configs are generally things people copy and modify from examples anyway. If the pbtxt parser is a little too fussy, then maybe the best thing to do is send protobuf some PRs making it better.

[chihuahua]

K, I'd be fine with pbtxt if we have clear docs. Although pbs have schemas (which trivially comprise of string types here), I'd say they're less clear to create than JSON. I don't think pbtxt was meant to be generally used for data transmission. The graph explorer sort of abused pbtxt.

If someone told me to create a pbtxt file today, I'd have to copy and paste the format from google3 examples. There are few pbtxts in the open source world. I like recognition over recall.

[decentralion]

Re pbtxt vs json: Note there is a canonical mapping between proto3 and json so it may be that you can get best of both worlds by defining the scheme in a .proto file, but then using the JSON-mapped version for convenience and familiarity.

[decentralion]

For the API design: I think it would be nice if the vz-line-chart allowed explicit association between (tag,run) combinations and the marker choice, and the code auto-generated a sensible correspondence. That opens the doors to allowing the user to create their own explicit mapping, if that's a feature that there's demand for.

I think having the plugin save the config file in the root log directory sounds good. I recommend building a re-usable API for plugins to save/load named config files, rather than doing it in a one-off manner. That will be useful for other plugins too, and it will allow us to migrate all plugins that use this system en masse to the new backened, when the time comes.

[chihuahua]

Some good ideas from @jart:

• TensorBoard will show the custom scalars dashboard (instead of the current scalars one) if a configuration file is provided.
• The configuration file will explicitly list tags instead of regexes. This lets the user clearly configure their TensorBoard home page over time.
• We can consider folding this plugin into the current scalars plugin. I think I might prototype this as a separate one first (the frontend's easier to implement that way) and then attempt combining.

[chihuahua]

Here's a screenshot of the current frontend:

It doesn't yet incorporate all the ideas on this thread, but I'm working on it.

I'm going to code the backend now (reading the config file) and then solicit some people to try the custom scalars plugin for feedback.

[agrinh]

I have an application where being able to examine plots per epoch is very useful for understanding and sanity-checking the evolution of the model (similar to the images plugin where you can choose which epoch to see an image from through a slider).

@chihuahua Would it make sense to support something like that with this request perhaps?

[RadZaeem]

How is the status of PR?
Is there a way for now to plot train/validation/test on same plot?

[nfelt]

I believe this PR is addressed by Chi's work implementing the custom scalars plugin and dashboard (primarily in #664). @chihuahua can you confirm and close if so?

[nfelt]

Closing this as largely addressed by the custom scalars dashboard. See also #59 about being able to control the summary name, which may end up being another way to address the original problem.