Congratulations to the hardworking folks at UPenn!

Thank you Edward for including me and for all the nice discussions.

🌐 penn-pal-lab.github.io/aawr
📝 openreview.net/forum?id=Rkd...
💻 github.com/penn-pal-lab...

More theory details in Appendix A-E and on slide 30 (orbi.uliege.be/handle/2268/...)

As seen from the results and videos, AAWR improves significantly on (i) foundation policies, (ii) behavior cloning policies, and (iii) AWR policies, providing good policies even in partially observable environments with non Markovian inputs.

It is the case here, where we use additional cameras, position estimates, or bounding boxes from pretrained models.

These features i are used as additional input of the critic Q(i, z, a) to provide a better advantage estimate and policy improvement direction.

In fact, it is a common assumption in asymmetric RL, which distinguishes the execution information from the training information.

In practice, while we do not always know the exact state, it is common to have more information available about the state at training time.

Moreover, (s, z) is shown to be the Markovian state of an equivalent MDP, allowing us to rely on the Bellman equation and TD learning instead of MC learning.

Now, how realistic is it to assume that we know the state s in addition to the input z?

Unfortunately, we show that we cannot just learn the symmetric critic Q(z, a) = E[G | z, a]. Instead, we need an asymmetric critic Q(s, z, a) = E[G | s, z, a] for a valid policy iteration.

This is because, unlike for policy gradients, the AWR objective is not linear in Q.

To learn a good policy (for this specific input z), we may want to rely on existing RL algorithms such as policy gradient or policy iteration.

Here, because we perform offline-to-online training, we rely on AWR, a policy iteration algorithm going offline to online seamlessly.

When learning in real-world scenarios, it is common to have constraints on the input available to the policy at execution time (e.g., last observation only, wrist camera only, etc).

In the general case (POMDP), the input z is a function of the observation history h: z = f(h).

Interestingly, this distribution can be learned from off-policy samples with a TD-like update. And this, even when encouraging the visitation of features of future states (possibly aliased).

arxiv.org/abs/2412.06655

4) Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures.

With Adrien Bolland and Damien Ernst, we propose a new intrinsic reward. Instead of encouraging visiting states uniformly, we encourage visiting *future* states uniformly, from every state.

This view offers interesting insights for the design of intrinsic rewards, by providing four criteria.

arxiv.org/abs/2402.00162

3) Behind the Myth of Exploration in Policy Gradients.

With Adrien Bolland and Damien Ernst, we decided to frame the exploration problem for policy-gradient methods from the optimization point of view.

By adapting a finite-time bound, we uncover an interesting tradeoff between informativeness of the additional information and complexity of the resulting value function.

openreview.net/forum?id=wNV...

2) Informed Asymmetric Actor-Critic: Theoretical Insights and Open Questions.

With Daniel Ebi and Damien Ernst, we looked for a reason why asymmetric actor-critic was performing better, even when using RNN-based policies with the full observation history as input (no aliasing).

In AsymAC, while the policy maintains an agent state based on observations only, the critic also takes the state as input. Its better performance is linked to eventual "aliasing" in the agent state, hurting TD learning in the symmetric case only.

arxiv.org/abs/2501.19116

1) A Theoretical Justification for Asymmetric Actor-Critic Algorithms.

With Damien Ernst and Aditya Mahajan, we looked for a reason why asymmetric actor-critic algorithms are performing better than their symmetric counterparts.