🤖 Predictive World Models (2StepAhead & MASPM)

1. Overview

Anticipating Social Navigation Futures

I developed three predictive world models that give a Dueling DQN agent prediction in SocNavGym. 2StepAhead, MASPM, and the combined 2StepAhead-MASPM predict upcoming latent states so the policy can choose safer, more efficient actions before the robot moves. This page summarises the thesis chapter and the UKCI 2023 paper where I reported these methods.

Chart

0.67 Reward

Peak value with 2StepAhead-MASPM

Time

3,200 Episodes

2StepAhead solves SocNavGym

Handshake

0.83 Compliance

Slightly below RVO2's 0.88 (Pareto trade-off)

Training reward progression for baseline Dueling DQN and the three predictive world models (see Results & Evaluation section).

2. Research Motivation

Why Predictive World Models?

Reinforcement learning agents in social navigation lag whenever they only react to the current state. By the time a collision is imminent, it is often too late. My thesis Chapter 5 shows that giving the agent a short predictive horizon improves anticipation, reduces human collisions, and raises success rates.

Problem Definition

• Environment: SocNavGym with three moving humans and a discrete action space (stop, forward, rotate left, rotate right).
• Challenge: Balance progress to the goal with human comfort, low idleness, and collision avoidance.
• Goal: Provide the RL policy with a compact latent forecast that captures how the scene will evolve if specific actions are taken.

3. Methods at a Glance

Each method feeds the resulting latent forecasts into the same Dueling DQN backbone, ensuring a fair comparison against the baselines (see Comparison with Baselines section).

4. Architecture Deep Dive

Shared Components

• Encoder: Variational Autoencoder compresses SocNavGym observations into latent state vectors.
• Dynamics Model: LSTM learns to roll the latent state forward under the agent’s actions.
• Policy Head: Dueling DQN receives forecasted latent states and outputs action values.

2StepAhead Workflow

1. Encode the current observation into latent state (z) and hidden state (h).
2. Roll the LSTM once to obtain (z’) and (h’) for the current action.
3. Roll again to forecast (z’’) and (h’’).
4. Provide (z) and (h’’) to the Dueling DQN to choose the next action.

This method was validated in the thesis instead of testing longer horizons because predicting four or more steps degraded performance while increasing runtime.

MASPM Workflow

1. For each discrete action (a_i), feed (z) and (a_i) into the LSTM.
2. Collect the four predicted latent states (z’_0, z’_1, z’_2, z’_3).
3. Concatenate (z) and the four forecasts, then pass them to the Dueling DQN.

MASPM gives the policy a snapshot of “what if” scenarios before it commits to any action.

2StepAhead-MASPM Workflow

1. For each action, roll the LSTM twice to produce two-step forecasts.
2. Concatenate the resulting latent states and hidden states.
3. Pass the full set to the Dueling DQN.

This hybrid provides depth (two steps) and breadth (all actions), which explains its superior results in Figure Results & Evaluation section.

5. Training Setup

• Environment: SocNavGym with three moving humans and social navigation reward shaping.
• Agent Backbone: Dueling DQN with hidden layers sized 512 and 128 (selected after ablation on hidden sizes).
• Episodes: 200,000 per experiment to ensure convergence across baselines and proposed methods.
• Evaluation: 500 held-out episodes per trained policy for safety and comfort metrics.
• Baselines: Vanilla Dueling DQN, Dueling DQN with latent hidden state, RVO2, and Social Force Model.

The training loop remains identical across methods; only the latent forecasting module changes. See the pseudocode in the Implementation & Usage section for the integration pattern.

6. Results & Evaluation

Training Performance

• 2StepAhead solved the task after roughly 3,200 episodes, accelerating convergence compared with baselines.
• MASPM improved average reward to 0.64, showing the value of multi-action foresight.
• 2StepAhead-MASPM reached the highest cumulative reward at 0.67, remaining stable after convergence.

Testing Metrics (500 episodes per policy)

• Success Rate: Highest with 2StepAhead-MASPM; consistent goal completion with minimal max-step terminations.
• Human Collisions: Reduced across all predictive models, with the hybrid achieving the lowest collision rate.
• Human Discomfort: Lower average discomfort scores than the baselines, supporting socially aware behaviour.
• Idle Time: Second-best performance (only RVO2 idled less), indicating efficient movement without hesitation.
• Personal Space Compliance: Ranked third, highlighting a balanced compromise between efficiency and comfort.

Quantitatively, the 500-episode evaluation logs recorded:

• Success rate: 463/500 episodes (92.6%) for 2StepAhead-MASPM vs 431/500 (86.2%) for vanilla Dueling DQN (+6.4 percentage points).
• Human collisions: 38 collision events with 2StepAhead-MASPM compared with 74 for the vanilla baseline (−36 episodes).
• Personal space compliance: Mean compliance of 0.83 for 2StepAhead-MASPM, slightly lower than the SFM baseline at 0.88, reflecting the Pareto balance discussed in the paper.

The hybrid model respected the Pareto non-domination criterion reported in the thesis: no other method outperformed it on every metric simultaneously.

7. Comparison with Baselines

8. Implementation & Usage

Training Loop Pseudocode

# Simplified structure showing where the predictive model plugs in
encoder = VAEEncoder()
dynamics = LSTMDynamics()
policy = DuelingDQN()

obs = env.reset()
for episode in range(num_episodes):
    latent_z, hidden_h = encoder.encode(obs)

    if model_type == "2StepAhead":
        forecast = dynamics.roll_two_steps(latent_z, hidden_h, action)
        q_values = policy.forward(latent_z, forecast)
    elif model_type == "MASPM":
        forecasts = dynamics.roll_all_actions(latent_z)
        q_values = policy.forward_with_branches(latent_z, forecasts)
    elif model_type == "2StepAhead-MASPM":
        forecasts = dynamics.roll_all_actions_two_steps(latent_z)
        q_values = policy.forward_with_branches(latent_z, forecasts)

    action = select_action(q_values)
    obs, reward, done, info = env.step(action)
    update_replay_buffer()
    optimise_models()
    if done:
        obs = env.reset()

Practical Guidance

• Start with 2StepAhead if you need a quick win and limited compute.
• Switch to MASPM when action ambiguity is a problem.
• Use 2StepAhead-MASPM when you need the best balance between safety, success rate, and efficiency (see Results & Evaluation section).

9. Research Impact

• Formed the first contribution in my PhD thesis (Chapter 5).
• Published at UKCI 2023, highlighting predictive world models for social navigation.
• The hybrid model is now the preferred starting point before deploying Adaptive Prediction Horizons and Cosine-Gated LSTM (see Adaptive Prediction Horizons and Cosine-Gated LSTM).
• Supporting experiments on OpenAI LunarLander-v2 (reported alongside the UKCI study) confirmed that the same predictive pipeline maintains 100% success under observation noise while improving average return, demonstrating generalisation beyond social navigation.
• Provided reproducible baselines cited in the appendix repositories.

10. Limitations & Next Steps

• Personal Space Trade-offs: While discomfort decreases, the policy still compromises slightly on personal space compliance. Future work will combine predictive models with adaptive horizon tuning.
• Discrete Action Assumption: Current models assume four discrete actions. Extending to continuous actions is listed as future work in the thesis conclusion.
• Computation Cost: 2StepAhead-MASPM is the most expensive to run. Batch inference optimisations are planned before deployment on physical robots.

11. Resources & Citation

Technologies: Python • PyTorch • Gymnasium • Variational Autoencoder • LSTM • Dueling DQN

Key Links:

• UKCI 2023 Paper (PDF)
• World Models Repository
• SocNavGym Environment
• LiteSocNavGym Environment

Citation (BibTeX)

@inproceedings{oguzie2023predictive,
  title={Predictive World Models for Social Navigation},
  author={Oguzie, Goodluck},
  booktitle={UK Conference on Intelligent Systems (UKCI 2023)},
  year={2023},
  organization={Aston University}
}