Ray: What are the auto-filled metrics in step() output? [tune]

[ptyshevs]

What is your question?

Reading through the user-guide, I've stumbled upon section about stop argument of tune.run:
https://docs.ray.io/en/latest/tune/user-guide.html#stopping-trials

There is the following code sample:

# training_iteration is an auto-filled metric by Tune.
tune.run(
    my_trainable,
    stop={"training_iteration": 10, "mean_accuracy": 0.98}
)

Where can I find the extensive list of what metrics were auto-filled by Tune?

Ray version and other system information (Python version, TensorFlow version, OS):
Ray version: 0.8.7

[richardliaw]

Hmm we have them here:

https://docs.ray.io/en/master/tune/tutorials/overview.html#what-are-all-these-extra-outputs

But let me know where we should move this!

[ptyshevs]

@richardliaw
It appear that there are two version of docs:

• Built with sphinx: https://docs.ray.io/en/latest/tune/user-guide.html#stopping-trials
• This new fancy one: https://docs.ray.io/en/master/tune/tutorials/overview.html#what-are-all-these-extra-outputs

The problem is that only the first one is indexed in Google, so I've never actually seen the one you've sent.
Moreover, I'm working with the Tune docs for the last couple of months and I've never seen any reference to the docs that you've sent.

Can you please explain how we ended up with two version of generated documentation?

btw, thanks, that's exactly what I was looking for. You can close the issue now

[ptyshevs]

There is also a small problem. according to the doc mentioned in the issue we have training_iteration auto-filled, but I don't see this specific name in https://docs.ray.io/en/master/tune/tutorials/overview.html#what-are-all-these-extra-outputs.

Whom should I trust?

[richardliaw]

Hmm I think it is there -

-

training_iteration: The number of times tune.report() has been called

I think on your version of the docs you will see the same here
https://docs.ray.io/en/latest/tune/api_docs/trainable.html#ray.tune.Trainable.train

FYI the new docs are maybe “too new” (not reflected on the pip release) -
so maybe don’t use them yet ;)

The next release should be next week. Hope that helps!

On Thu, Sep 17, 2020 at 12:09 PM Pavel Tyshevskyi [e-mail]
wrote:

There is also a small problem. according to the doc mentioned in the issue
we have training_iteration auto-filled, but I don't see this specific
name in
https://docs.ray.io/en/master/tune/tutorials/overview.html#what-are-all-these-extra-outputs
.

Whom should I trust?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ray-project/ray/issues/10862#issuecomment-694440202,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABCRZZLQWLXF26T332FNCX3SGJNFLANCNFSM4RQ2AHYA
.

[amogkam]

Auto filled metrics are also listed here on latest docs: https://docs.ray.io/en/latest/tune/user-guide.html#reporting-metrics.

And on master: https://docs.ray.io/en/master/tune/user-guide.html#reporting-metrics

[ptyshevs]

@richardliaw you're totally right, I probably need to get some sleep. I'm looking forward to the release.

@amogkam thank you, I don't know how I missed that.