Skip to content

Preserve terminal observation in BraxAutoResetWrapper for correct truncation bootstrapping - #351

Open
andreolf wants to merge 1 commit into
google-deepmind:mainfrom
andreolf:fix/autoreset-final-obs-305
Open

Preserve terminal observation in BraxAutoResetWrapper for correct truncation bootstrapping#351
andreolf wants to merge 1 commit into
google-deepmind:mainfrom
andreolf:fix/autoreset-final-obs-305

Conversation

@andreolf

Copy link
Copy Markdown

Summary

Fixes #305. BraxAutoResetWrapper overwrote state.obs with the reset observation on done without preserving the terminal (pre-reset) observation anywhere, which makes correct value bootstrapping on truncation impossible for downstream RL (every actor-critic method needs the terminal obs to distinguish termination from truncation when computing TD/GAE targets — the same reason Gymnasium exposes final_observation).

Change

  • Expose the terminal observation in state.info['AutoResetWrapper_final_obs'].
  • Set it in step() from the true post-step state.obs (before it's blended with the reset obs).
  • Initialize it in reset() so the info pytree structure matches step() (important under jax.lax.scan).
  • Extend test_auto_reset_wrapper to assert the key exists after reset and that, on a done step, the preserved terminal obs differs from the returned reset obs.

Non-breaking: only adds a new info key.

Notes

  • I don't have a local JAX/MJX GPU environment to run the suite end-to-end, so I'd appreciate a maintainer running wrapper_test.py. The change is small and mirrors the existing *_first_obs bookkeeping.
  • I'll sign the CLA — flagging that the CLA bot will comment.

…d#305)

BraxAutoResetWrapper overwrote state.obs with the reset observation on done
without keeping the terminal (pre-reset) obs anywhere, making correct value
bootstrapping on truncation impossible for downstream RL. Expose it in
state.info['AutoResetWrapper_final_obs'] (initialized in reset() for pytree
consistency, set in step()). Extends the existing wrapper test.
@google-cla

google-cla Bot commented Aug 19, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BraxAutoResetWrapper discards terminal observation, breaking correct value bootstrapping on truncation

1 participant