Tensorboard: Documentation on Tensorboard HParams plugin

[rishabhsahrawat]

I am looking for documentation on Tensorboard HParams plugin, in particular, I want to know about hp.Discrete & hp.RealInterval and if there are also some other functions like these available. I tried to look for documentation on these but could not find anything. I would be thankful if someone can share a link of the documentation.

[wchargin]

Hi @rishabhsahrawat! We have a [tutorial notebook] with a demo that
shows these APIs. You can run the demo in Colab or download it as an
IPython notebook. You can also read the [top-level API docs] in
api.py, and of course the individual docstrings on the various classes
and functions. You can also consult the [hparams demo script] for an
end-to-end example.

I want to know about hp.Discrete & hp.RealInterval and if there
are also some other functions like these available.

The three domain types are the two that you list plus IntInterval. The
demo script uses all of these.

Let us know if you have any further questions!

[zfang98]

Hi

Hi @rishabhsahrawat! We have a tutorial notebook with a demo that
shows these APIs. You can run the demo in Colab or download it as an
IPython notebook. You can also read the top-level API docs in
api.py, and of course the individual docstrings on the various classes
and functions. You can also consult the hparams demo script for an
end-to-end example.

I want to know about hp.Discrete & hp.RealInterval and if there
are also some other functions like these available.

The three domain types are the two that you list plus IntInterval. The
demo script uses all of these.

Let us know if you have any further questions!

Hi I looked and tried the tutorial notebook in your note.
I have some questions about this line:
HP_DROPOUT = hp.HParam('dropout', hp.RealInterval(0.1, 0.2))
It seems hparams ran the model with only two values for dropout_rate :0.1, 0.2 instead of using values inbetween 0.1 and 0.2.
How can I let hparams to try values in a range?

Thanks
Fang

[wchargin]

This is described in the text and code of section 3:

For simplicity, use a grid search: try all combinations of the
discrete parameters and just the lower and upper bounds of the
real-valued parameter. For more complex scenarios, it might be more
effective to choose each hyperparameter value randomly (this is called
a random search).

and

  for dropout_rate in (HP_DROPOUT.domain.min_value, HP_DROPOUT.domain.max_value):
    ...

You can see that we’re explicitly picking the two endpoints of the
domain only. You could instead use something like

  for dropout_rate in tf.linspace(
      HP_DROPOUT.domain.min_value,
      HP_DROPOUT.domain.max_value,
      11,
  ):
    ...

for a higher-resolution grid search, or

  for dropout_rate_iteration in range(10):
    dropout_rate = HP_DROPOUT.domain.sample_uniform()
    ...

for a random search.

More generally, the TensorBoard hparams plugin doesn’t tune the model
for you. You bring your own tuner, or write a simple one inline, as in
the tutorial; the hparams dashboard lets you visualize the results of
tuning along with the rest of your metrics.

[thisismygitrepo]

I highly recommend that these details be added to the notebook, cause the it leaves much to be desired.

[thisismygitrepo]

My TF says:
AttributeError: 'RealInterval' object has no attribute 'sample_uniform'

[thisismygitrepo]

Additionally, I got this error when I tried tf.linspace as you explained
TypeError: Object of type 'EagerTensor' is not JSON serializable

[wchargin]

The sample_uniform function requires TensorBoard 1.14.

You’ll need to convert tensors to plain values before serializing them
as hparams: either

  for dropout_rate_tensor in tf.linspace(min_, max_, n):
    dropout_rate = dropout_rate_tensor.numpy()
    ...

or

  for dropout_rate in tf.linspace(min_, max_, n).numpy():
    ...

or just use the numpy linspace:

  for dropout_rate in np.linspace(min_, max_, n):
    ...

[thisismygitrepo]

Hi. Thanks, the suggestions worked. Apart from sample_uniform method. I'm using TF beta1 which is the latest, and it comes with tensorboard 1.14.0a20190603
Still says undefined method.

[wchargin]

@kernelizd: sample_uniform was added in #2259, which merged on
2019-06-03, so it should have first appeared in the 2019-06-04 nightly.
Please upgrade to either latest tb-nightly or latest tensorboard.

[VCBE123]

HParams failed with hparams['layers'] = 7, of unsupported type

[OscarVanL]

Hi @wchargin , you suggested using for dropout_rate in tf.linspace(min_, max_, n).numpy(), which gives a np.float32 value. The problem is that when I use this in my hparams dictionary as in the tutorial notebook you linked, when it reaches the line hp.hparams(hparams) or the callback hp.KerasCallback(log_dir, hparams), it gives this error:

TypeError: Object of type float32 is not JSON serializable

In normal cases since it's being serialised to a JSON file, I'd just wrap this in str(), but in this case it's being used as a hyperparameter so I can't do that.

What would you suggest?

[OscarVanL]

The way I've temporarily got around that problem is having for dropout_rate in tf.linspace(min_, max_, n).numpy(), then running str(dropout_rate) before adding dropout_rate to my hparams dictionary.

Then whenever I want to use dropout_rate as a parameter I use float(dropout_rate) to make it a float again.

However, even this isn't a good workaround, because now my Tensorboard HPARAMS window is not displaying any data, and the console is spamming errors: HParams error: Cannot use an interval filter for a value of type: <class 'str'>, Value: 0.1 whenever I open it.

[wchargin]

@OscarVanL: Thanks for pointing this out! As noted on #3057, the issue
was that np.float64s get serialized natively but np.float32s require
extra treatment. As of #3059, hp.hparams and hp.hparams_pb now
accept any Numpy scalars (np.float32, np.int64, etc.) and will do
the conversion for you.

[jarednielsen]

All the documented methods seem to use Keras callbacks. Is there a way to manually log a metric at the end of a run? I have everything working except for the metrics, which are not logged.

HP_MODEL_TYPE = hp.HParam("model_type", hp.Discrete(["albert", "bert"]))
HP_LEARNING_RATE = hp.HParam("learning_rate", hp.RealInterval(1e-5, 1e-1))

METRIC_LOSS = "my_metric"
hp.hparams_config(
    hparams=[HP_MODEL_TYPE, HP_LEARNING_RATE],
    metrics=[hp.Metric(METRIC_LOSS, display_name="my_metric")],
)
hp.hparams({
    HP_MODEL_TYPE: args.model_type,
    HP_LEARNING_RATE: args.learning_rate,
})
tf.summary.scalar(METRIC_LOSS, 0.23, step=1)

[formigone]

Hey, guys. hparams is incredible. Thanks for that. Quick question: how do I update the "status" of my training sessions so those checkboxes correspond more accurately? Also, I've been using this a fair bit, and I feel I understand it well. What's the best way I can contribute documentation for it?

[manivaradarajan]

@formigone Can you open a new issue with your question?

We would be delighted for you to contribute documentation. Please open a PR against the tutorial notebook or API docs (see the comment above (https://github.com/tensorflow/tensorboard/issues/2348#issuecomment-502171358) for links.