Adaptive Predictive Reinforcement Learning: Entropy-Driven Adaptive Prediction Horizons

1. Overview

Entropy-guided horizons for faster, safer RL

Fixed prediction horizons lead to wasted computation in easy scenes and brittle planning in complex ones. This project uses policy entropy to adapt the depth of world-model rollouts on the fly. Early results show faster convergence in LunarLander-v2 and stronger success/return trade-offs in LiteSocNavGym, while keeping inference latency suitable for robotics.

Time

53.48 h to 500k steps

Entropy horizon vs 59.54 h fixed (LunarLander)

100%

Success rate

SAC + CGLSTM (adaptive) in LiteSocNavGym

5.62 days

Full training

LiteSocNavGym vs 28.41 days for DreamerV3 + CGLSTM

Adaptive horizon pipeline: policy entropy controls how many CGLSTM rollouts are generated before the SAC update.

2. Research Motivation

• Fixed horizons in world-model RL either over-plan (wasting compute when the scene is simple) or under-plan (failing when dynamics are chaotic).
• Policy entropy provides a continuous indicator of decision uncertainty; high entropy → longer lookahead, low entropy → shorter horizon.
• Target applications include social navigation and aerial robotics, where scene complexity changes rapidly and latency budgets are strict.

3. Adaptive Architecture

The adaptive controller sits between the CGLSTM world model and the reinforcement learner. At each step it samples multiple horizon candidates, evaluates expected entropy reduction, and commits to the smallest depth that meets a confidence threshold.

Replacing the LayerNorm GRU in DreamerV3 with CGLSTM gives higher fidelity latent rollouts before the adaptive controller is applied.

Adaptive horizon inside LiteSocNavGym: when policy entropy spikes, additional CGLSTM rollouts are appended before the SAC update.

4. Implementation Notes

• Horizon candidates: {1,2,4,6,8} with entropy thresholds at 0.2, 0.35, 0.5.
• Entropy is computed from the SAC policy logits; if entropy > threshold_k, expand horizon to next candidate.
• CGLSTM hidden state is reused across horizon branches to avoid recomputing encodings.

def choose_horizon(entropy, candidates=(1, 2, 4, 6, 8), thresholds=(0.2, 0.35, 0.5, 0.65)):
    horizon = candidates[0]
    for h, t in zip(candidates[1:], thresholds):
        if entropy > t:
            horizon = h
        else:
            break
    return horizon

entropy = policy_entropy(obs)
steps_ahead = choose_horizon(entropy)
rollout = world_model.rollout(obs, steps=steps_ahead)
value, action = sac.update(rollout)

5. LunarLander Experiments (500k steps)

5.1 Window-slide selection for CGLSTM

Entropy controller uses a 32-step window (middle curve) – fastest loss decay without the instability seen for 16.

5.2 Horizon candidate comparison

Candidate set {1,2,3,4,5} converges quickly; {1,2,4,6,8} overtakes after 200k steps. Set B selected for final runs.

5.3 Training performance

Entropy-driven horizon (cyan) matches DreamerV3 + CGLSTM convergence while remaining stable after 350k steps.

5.4 Wall-time savings

Entropy controller reaches 500k steps in 53.48 hours (blue) vs 59.54 hours for the fixed horizon baseline (orange).

6. LiteSocNavGym Experiments (4M steps)

6.1 Training reward progression

Adaptive horizon (cyan) and DreamerV3 + CGLSTM (brown) are the only agents sustaining positive reward after 3M steps.

6.2 Success rate vs average return

Adaptive horizon with CGLSTM achieves ~0.48 average return at ~1.6 ms inference, outperforming PPO/TD3 in both axes.

6.3 Training cost profile

Adaptive horizon needs 5.62 days to train (hidden size 256), versus 28.41 days for DreamerV3 + CGLSTM.

6.4 Success vs return scatter (aggregate)

Scatter from viva slides: adaptive SAC variants dominate the top-right corner (high success, high return).

6.5 Dreamer comparison

DreamerV3 + CGLSTM (orange marker) obtains the best success/return compromise, justifying its role as a teacher for adaptive SAC.

7. Current Status & Next Steps

Completed so far:

• Integrated entropy-driven controller with SAC + CGLSTM on LunarLander-v2 and LiteSocNavGym.
• Benchmarked wall-time, training-time, and inference-time trade-offs versus DreamerV3 and classical baselines.
• Selected stable hyperparameters (CGLSTM window = 32, horizon set {1,2,4,6,8}).

Next steps:

• Extend to full SocNavGym with richer social scenarios and compare against predictive-world-model agent.
• Evaluate robustness under sensor noise and partial observability.
• Port adaptive horizon logic into DreamerV3 for end-to-end latent adaptation.
• Prepare submission for journal special issue on Adaptive RL in Robotics.

8. Resources & Links

Technologies: Python • PyTorch • Gymnasium • SAC • DreamerV3 • CGLSTM

• SocNavGym Environment
• Cosine-Gated LSTM
• Predictive World Models
• Collaboration Interest Form