Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib] Cleanup examples folder (new API stack) vol 31: Add hierarchical training example script. #49127

Merged
merged 36 commits into from
Dec 11, 2024
Merged

[RLlib] Cleanup examples folder (new API stack) vol 31: Add hierarchical training example script. #49127

merged 36 commits into from
Dec 11, 2024

Conversation

sven1977
Copy link
Contributor

@sven1977 sven1977 commented Dec 6, 2024 •
edited
Loading

Cleanup examples folder (new API stack) vol 31:

  • Add new example env (six room env) with two versions: flat and hierarchical.
  • Add hierarchical training example script.

Why are these changes needed?

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

sven1977 added 15 commits November 4, 2024 18:14
@sven1977
wip
10718e7 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into clea…
f6caa54 
…nup_examples_folder_31_hierarchical_training
@sven1977
wip
45d16fa 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
e02f5ad 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into clea…
2e507ec 
…nup_examples_folder_31_hierarchical_training
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into clea…
4c04b04 
…nup_examples_folder_31_hierarchical_training
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into clea…
36fb8d4 
…nup_examples_folder_31_hierarchical_training
@sven1977
wip
d3da672 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
b8d502f 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into clea…
542d22a 
…nup_examples_folder_31_hierarchical_training
@sven1977
wip
724d350 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
408f633 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
a632872 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into clea…
6185746 
…nup_examples_folder_31_hierarchical_training
@sven1977
wip
1a99237 
Signed-off-by: sven1977 <[email protected]>
@sven1977 sven1977 added rllib RLlib related issues rllib-env rllib env related issues rllib-docs-or-examples Issues related to RLlib documentation or rllib/examples rllib-newstack rllib-oldstack-cleanup Issues related to cleaning up classes, utilities on the old API stack labels Dec 6, 2024
@sven1977
wip
c42b435 
Signed-off-by: sven1977 <[email protected]>
@sven1977 sven1977 requested review from maxpumperla and a team as code owners December 6, 2024 12:20
sven1977
sven1977 commented Dec 6, 2024
View reviewed changes
doc/source/rllib/rllib-examples.rst Outdated
Hierarchical Training
+++++++++++++++++++++

- `Policy Inference after Training <https://github.com/ray-project/ray/blob/master/rllib/examples/inference/policy_inference_after_training.py>`__:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, still WIP ... please ignore this change here for now

sven1977
sven1977 commented Dec 6, 2024
View reviewed changes
rllib/env/utils/infinite_lookback_buffer.py Outdated
@@ -533,9 +533,18 @@ def _get_int_index(
):
data_to_use = self.data
if _ignore_last_ts:
data_to_use = self.data[:-1]
if self.finalized:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these were all bugs

@sven1977
wip
36bea65 
Signed-off-by: sven1977 <[email protected]>
sven1977
sven1977 commented Dec 6, 2024
View reviewed changes
rllib/env/multi_agent_episode.py Outdated
return {
key: sub_buffer.get(
indices=index_incl_lookback - sub_buffer.lookback,
neg_index_as_lookback=True,
fill=fill,
_add_last_ts_value=hanging_val,
_add_last_ts_value=(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another bug fix

@sven1977
wip
466f53d 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into clea…
9f3e607 
…nup_examples_folder_31_hierarchical_training
@sven1977
wip
1aa4ab0 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
9f8c33a 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'fix_further_bugs_in_multi_agent_episode' into cleanup_e…
874183c 
…xamples_folder_31_hierarchical_training

# Conflicts:
#	rllib/env/multi_agent_episode.py
#	rllib/env/single_agent_episode.py
#	rllib/env/tests/test_multi_agent_episode.py
#	rllib/env/utils/infinite_lookback_buffer.py
@sven1977 sven1977 requested a review from a team as a code owner December 10, 2024 18:45
@sven1977
wip
240720a 
Signed-off-by: sven1977 <[email protected]>
@sven1977
LINT
6715808 
Signed-off-by: sven1977 <[email protected]>
@sven1977
running fine w/o crashes
d289127 
Signed-off-by: sven1977 <[email protected]>
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into clea…
281c30a 
…nup_examples_folder_31_hierarchical_training

Signed-off-by: sven1977 <[email protected]>

# Conflicts:
#	rllib/env/multi_agent_episode.py
@sven1977
wip
98c5e4f 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
5e71845 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
8d46aec 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
72bec12 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
a1369a7 
Signed-off-by: sven1977 <[email protected]>
@sven1977
wip
78b36fc 
Signed-off-by: sven1977 <[email protected]>
@sven1977 sven1977 enabled auto-merge (squash) December 11, 2024 12:29
@github-actions github-actions bot disabled auto-merge December 11, 2024 12:29
@github-actions github-actions bot added the go add ONLY when ready to merge, run all tests label Dec 11, 2024
@sven1977
wip
81f7a57 
Signed-off-by: sven1977 <[email protected]>
@sven1977 sven1977 enabled auto-merge (squash) December 11, 2024 12:36
@sven1977
wip
cd61bb3 
Signed-off-by: sven1977 <[email protected]>
@github-actions github-actions bot disabled auto-merge December 11, 2024 13:31
@sven1977 sven1977 enabled auto-merge (squash) December 11, 2024 13:45
@sven1977
wip
5209af7 
Signed-off-by: sven1977 <[email protected]>
@github-actions github-actions bot disabled auto-merge December 11, 2024 14:44
@sven1977 sven1977 enabled auto-merge (squash) December 11, 2024 16:27
@sven1977 sven1977 merged commit f98ce02 into ray-project:master Dec 11, 2024
6 checks passed
@sven1977 sven1977 deleted the cleanup_examples_folder_31_hierarchical_training branch December 11, 2024 18:27
simonsays1980 pushed a commit to simonsays1980/ray that referenced this pull request Dec 12, 2024
@sven1977 @simonsays1980
[RLlib] Cleanup examples folder (new API stack) vol 31: Add hierarchi…
4783f33 
…cal training example script. (ray-project#49127)
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Dec 17, 2024
@sven1977 @ujjawal-khare
[RLlib] Cleanup examples folder (new API stack) vol 31: Add hierarchi…
a401270 
…cal training example script. (ray-project#49127)

Signed-off-by: ujjawal-khare <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simonsays1980 simonsays1980 simonsays1980 approved these changes

@maxpumperla maxpumperla Awaiting requested review from maxpumperla maxpumperla is a code owner

Assignees

@simonsays1980 simonsays1980

Labels
go add ONLY when ready to merge, run all tests rllib RLlib related issues rllib-docs-or-examples Issues related to RLlib documentation or rllib/examples rllib-env rllib env related issues rllib-newstack rllib-oldstack-cleanup Issues related to cleaning up classes, utilities on the old API stack
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sven1977 @simonsays1980