[RLlib] Docs do-over (new API stack): New MetricsLogger API rst page.
#49538
Conversation
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
…s_redo_metrics_logger
…_redo_metrics_logger
sven1977
requested review from
maxpumperla,
simonsays1980 and
a team
as code owners
sven1977
changed the title
MetricsLogger API rst page.MetricsLogger API rst page.
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
…_redo_metrics_logger Signed-off-by: sven1977 <[email protected]> # Conflicts: # rllib/BUILD
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
…_redo_metrics_logger
Signed-off-by: sven1977 <[email protected]>
sven1977
changed the title
MetricsLogger API rst page.MetricsLogger API rst page.
sven1977
added
rllib
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
doc/source/rllib/metrics-logger.rst
Outdated
| .. note:: | ||
| So far, RLlib components owning a :py:class:`~ray.rllib.utils.metrics_logger.MetricsLogger` | ||
| instance are :py:class:`~ray.rllib.algorithms.algorithm.Algorithm`, :py:class:`~ray.rllib.env.env_runner.EnvRunner`, | ||
| :py:class:`~ray.rllib.core.learner.learner.Learner`, and all :py:class:`~ray.rllib.connectors.connector_v2.ConnectorV2` classes. |
There was a problem hiding this comment.
EpisodeReplayBuffers
doc/source/rllib/metrics-logger.rst
Outdated
|
|
||
| - Users can log scalar values over time, such as losses or rewards. | ||
| - Thereby, users can configure different reduction types, in particular ``mean``, ``min``, ``max``, ``sum``, or ``None``. | ||
| - Users can also specify sliding windows, over which these reductions take place, for example ``window=100`` to average over the |
There was a problem hiding this comment.
Maybe say something about clear_on_reduce and how aggregates are held in the history instead of accumulating large lists.
| which doesn't alter any underlying values. | ||
|
|
||
|
|
||
| Instead of providing a flat key, you can also log a value under some nested key through passing in a tuple: |
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
|
||
| In case you are logging a nested structure of values, for example | ||
| ``{"time_s": 0.1, "lives": 5, "rounds_played": {"player1": 10, "player2": 4}}`` and all values have the exact same log settings |
There was a problem hiding this comment.
A dict that is for any reason empty, will be reduced away, which is for collecting result dicts for a table is challenging.
There was a problem hiding this comment.
Ok, let me try this right now and fix, if possible ...
There was a problem hiding this comment.
Yeah, an empty dict is not logged at all. Meaning ...
metrics.log_dict({}, key="abc")
... results in metrics not undergoing any changes, which I think makes a lot of sense. There are simply no keys to be logged and the underlying tree.map_structure doesn't have any nested structure to loop through.
doc/source/rllib/metrics-logger.rst
Outdated
|
|
||
|
|
||
|
|
||
| Timing things |
There was a problem hiding this comment.
"Timing processes"?
There was a problem hiding this comment.
Fine, renamed into Timing code blocks. :D
doc/source/rllib/metrics-logger.rst
Outdated
| # EMA should be ~1.1sec. | ||
| assert 1.15 > logger.peek("my_block_to_be_timed") > 1.05 | ||
|
|
||
| Also note that the default logging behavior is through exponential mean averaging (EMA), with a default coefficient of 0.01. |
There was a problem hiding this comment.
Maybe describing how to use a simple mean instead?
There was a problem hiding this comment.
Actually, we don't support this yet :D
You can do a simple mean over a window, yes, but not over all lifetime, b/c we don't have a proper mechanism yet to keep the list of values small. You could do logger.log_value([key], [value], reduce="mean", window=float("inf")), but that would cause a mem leak at some point. We need to fix this by keeping track of the number of items logged thus far and then keeping the internal list smaller, but still providing the proper lifetime average. To be completely honest, not sure how useful a lifetime average would be (as opposed to an EMA, which makes more sense to report). But we should fix this either way ...
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
…_redo_metrics_logger
Signed-off-by: sven1977 <[email protected]>
…_redo_metrics_logger Signed-off-by: sven1977 <[email protected]> # Conflicts: # doc/source/rllib/package_ref/algorithm.rst
Signed-off-by: sven1977 <[email protected]>
Signed-off-by: sven1977 <[email protected]>
Docs do-over (new API stack): New
MetricsLoggerAPI rst page.Why are these changes needed?
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.