[runtime_env] Change default working_dir behavior to replace local cached version entirely

[ujjawal-khare]

Summary

The runtime environment working directory is not refreshed upon multiple job submissions to the same Ray cluster due to a local cached version of the working directory existing and not fetching the updated version of the directory on S3. This PR changes the default behavior to delete and fetch from S3 again rather than use the cached local directory fetched on the first job submission. This change only applies to the case where multiple jobs are submitted in a row with the same working_dir url.

Related issue number

Closes #49036

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[dayshah]

there's a merge conflict, can you update

[rynewang]

LGTM, needs Train approval @justinvyu

[justinvyu]

Why are these changes needed for this PR? Converting to abspath before doing relpath is pretty confusing for non-local filesystems (such as S3) where abs path doesn't make sense.

Let's revert this change.

[ujjawal-khare-27]

@justinvyu The test case was failing Buildkite Link due to an issue with path handling. Specifically, the file system mount was unable to locate the directory because of a mismatch between file_info.path and fs_path. To resolve this, I converted the paths to abs_path to ensure consistency and proper resolution.

[justinvyu]

Where do you see that this test fails due to this line of code? The change above seems unrelated to this code.

As I mentioned above, we should not use abs path for this method anyways.

[ujjawal-khare-27]

@justinvyu have reverted the change and tests are passing as well. Can you please check?

[justinvyu]

@ujjawal-khare-27 Sorry for missing this -- after the revert my approval is no longer needed. I'll just merge the PR now.