Enhancing Robot Social Navigation
with Reinforcement Learning
and Advanced Predictive Models
Student: Goodluck Oguzie
190212683@aston.ac.uk
Supervisor: Dr. Luis J. Manso
Associate-Supervisor: Prof. Dr. Aniko Ekart
๐ Presentation Outline
- Context, Motivation & Problem
- Reinforcement Learning Basics
- Predictive Models Overview
- Environments
- Key Contributions:
- Predictive World Models
- Cosine-Gated LSTM (CGLSTM)
- Adaptive Prediction Horizons
- Conclusion & Future Work
๐ค What is Social Robot Navigation?
๐ก Definition:
Robots navigating among humans while respecting social norms, ensuring safety, and operating efficiently.
โ๏ธ Importance:
Social robot navigation is crucial for seamless integration into environments such as healthcare, hospitality, and public spaces. Advanced predictive decision-making is essential to manage inherent unpredictability.
| Approach | โ Strength | โ ๏ธ Weakness |
|---|---|---|
| Rule Base (Arkin, 1998) | Simple & efficient | Fails in dynamic settings |
| Social Force Models (Helbing & Molnar, 1995) | Good for crowd modeling | Limited adaptability |
| RVO2 (Van den Berg et al., 2011) | Real-time collision avoidance; effective in dynamic scenarios | Limited in long-term planning and human comfort prediction |
| Reinforcement Learning (Mnih et al., 2015) | Learns dynamically | Requires extensive training |
๐ก Our Research: Integrates reinforcement learning with advanced predictive models to overcome these limitations.
๐ The Problem:
"Robots struggle to navigate in social environments due to unpredictable human behavior."
Predict vs. React
๐ Research Questions
โ Q1: How do predictive world models improve decision-making in Social Robot Navigation (SocNav)?
โ Q2: What challenges arise when transitioning Reinforcement Learning applications from discrete to continuous action spaces in Social Robot Navigation, and how can we address the challenges?
RL Fundamentals
๐ Reinforcement Learning (RL) is a method by which an agent learns to make optimal decisions by taking actions in an environment to maximize cumulative reward (Sutton & Barto, 2018).
| RL Algorithm | ๐ฏ Strengths | โ ๏ธ Limitations |
|---|---|---|
| DQN (Mnih et al., 2015) | Efficient in discrete tasks | Not suited for continuous or complex action spaces |
| DDPG (Lillicrap et al., 2015) | Effective in continuous control tasks | Sensitive to hyperparameters and exploration noise |
| PPO (Schulman et al., 2017) | Stable learning; good sample efficiency | Sensitive to hyperparameter tuning |
| TD3 (Fujimoto et al., 2018) | Addresses overestimation bias in DDPG | Still requires careful hyperparameter tuning |
| SAC (Haarnoja et al., 2018) | Robust for continuous control; promotes exploration | Higher computational demands |
| DreamerV3 (Hafner et al., 2023) | Learns a latent world model for planning | Sensitive to model inaccuracies and observation noise |
Predictive Models
๐ค Predictive Models are machine learning models that learn from sequential data to forecast future states.
| Model | Description | ๐ฏ Pros | โ ๏ธ Cons |
|---|---|---|---|
|
RNN (Rumelhart et al., 1986) |
Basic recurrent net passing hidden states across time | ๐ฏ Simple, good for short sequences | โ ๏ธ Vanishing/exploding gradients for long sequences |
|
LSTM (Hochreiter & Schmidhuber, 1997) |
Adds gating (input, forget, output) for long-term dependencies | ๐ฏ Mitigates vanishing gradients; better for lengthy sequences | โ ๏ธ More parameters, heavier computation |
|
GRU (Cho et al., 2014) |
A simplified LSTM with fewer gates | ๐ฏ Fewer parameters, faster training than LSTM | โ ๏ธ May capture slightly less nuance for very long sequences |
|
Transformers (Vaswani et al., 2017) |
Employ self-attention for global context | ๐ฏ Excels at capturing long-range dependencies | โ ๏ธ High memory & compute requirements |
World Models (Ha & Schmidhuber, 2018)
Dreamer: A Step Forward, but Still Not Adaptive Enough
- โ Builds on World Models
- โ ๏ธ Does not consider all the possible actions
- โ ๏ธ Fixed Prediction Horizon (one step ahead)
- โฑ Limited Real-Time Prediction Horizon Adaptation
Environments Used
โ๏ธ FallingBallEnv
๐ Purpose: Toy environment for sequence prediction.
๐ช LunarLanderโv2
๐ Purpose: A Simple Continuous control task for RL.
๐ SocNavGym
๐ Purpose: Benchmark for social navigation tasks.
๐ท LiteSocNavGym
๐ Purpose: Lightweight variant for quicker iteration.
Predictive World Models for Social Navigation (Contribution 1)
- The goal is to mitigate limitations of World Models:
- ๐ single-step prediction
- ๐ limited actions
- ๐ฏ enabling multi-step, adaptive decision-making in dynamic social environments.
Paper PDF / Extended Explanation
๐งฉ 2StepAhead Predictive World Model
๐ Predicts hidden & latent states two steps ahead.
๐ Multi Action State Predictive Model (MASPM)
๐ Predicts outcomes for all available actions.
๐ 2StepAhead-MASPM (Hybrid Model)
๐ Combines 2StepAheadโs efficiency with MASPMโs comprehensive evaluation.
โฑ Training Metrics
โฑ Full convergence reached in ~200k episodes with top rewards ~0.75.
๐ฏ Testing Metrics (1)
โ
Testing results indicate higher rewards and success rates for our methods.
๐ฏ Testing Metrics (2)
๐ฏ 2StepAhead-MASPM demonstrates superior multi-metric performance.
Cosine-Gated LSTM (CGLSTM)
Contribution 2:
- ๐ค Aims to improve sequence prediction by focusing on directional alignment.
- ๐ Builds on limitations observed in previous LSTM experiments (e.g., in SocNavGym).
Paper PDF / Extended Explanation
๐Limitations
๐ They struggle with dynamic environments when predicting further ahead.
๐ก Why Cosine-Gated LSTM?
- ๐ค Emphasizes directional alignment in sequence prediction.
- ๐ Addresses the limitations of vanilla LSTM in handling long term prediction .
๐๏ธ CGLSTM Architecture
โ๏ธ Dynamically weighs new inputs vs. hidden states based on similarities.
โฑ FallingBallEnv Results
๐ CGLSTM outperforms LSTM, GRU, and Transformer in the falling ball predictions.
๐ Model Parameter Count
๐ง CGLSTM balances performance and efficiency with fewer parameters than the Transformer.
๐ฏ Benchmark Tasks (MNIST, etc.)
โ
Highest accuracy overall.
โฒ๏ธ Training & Testing Times
๐ GRU is faster, but CGLSTM uses ~5% more time for superior accuracy.
๐ค SocNavGym: Realistic Scenario
๐ Achieves the lowest MAE across prediction horizons (k=1 to k=10).
๐ฏ Multi-Model Prediction Comparison
๐ฏ CGLSTM shows tighter and more accurate trajectory predictions vs LSTM, GRU, and Transformer.
EntropyโDriven Adaptive Prediction Horizons
Contribution 3:
- โ Propose an entropy-driven adaptive Prediction horizon within SAC.
- ๐ค Integrate CGLSTM into DreamerV3 for enhanced predictive modeling.
- ๐ Benchmark vs. SAC, PPO, TD3, DDPG, DreamerV3 variants.
๐ Proposed Methods
- ๐ DreamerV3 + CGLSTM
- ๐ 2StepAhead-MASPM (Fixed Horizon)
- ๐ค SAC + CGLSTM (Adaptive Prediction Horizon)
๐ Proposed: DreamerV3 + CGLSTM
๐ค Enhances State Prediction.
๐ Proposed: 2StepAhead-MASPM (Fixed Horizon)
โ๏ธ Provides a strong baseline for adaptive Prediction horizon comparison.
๐ค Proposed: SAC + CGLSTM (Adaptive Prediction Horizon)
Dynamically adjusts prediction horizon via action entropy.
๐งช Preliminary Experiments Overview
- โ๏ธ Tested window-slide sizes: 16, 32, 64 for trade-offs in cost vs. accuracy.
- ๐ฏ Compared horizon sets {1,2,3,4,5} vs. {1,2,4,6,8} vs. fixed 8-step horizon.
๐ Window-Slide Selection
โ
Slide=64 yields lowest training loss.
๐ฏ Adaptive Prediction Horizon Selection
๐ฏ Adaptive Prediction Horizon Selection Result.
๐ Adaptive vs. Fixed Horizon
Adaptive {1,2,4,6,8} vs. fixed 8-step Prediction horizon.
โฑ Training Results: LunarLander
โฑ Full convergence reached in ~500k episodes.
โฑ Training Results: LiteSocNavGym
โฑ Full convergence reached in ~4 Million episodes.
๐ LunarLander Results (1)
โ๏ธ Success โ Return: Models trade off landing quality, time, and efficiency.
๐ LunarLander Results (2)
Note: Gaussian noise (ฯ = 0.085) added during testing.
๐ Adaptive Prediction horizon SAC + CGLSTM achieves 175.22 average return.
๐ LunarLander Results (3)
๐ Highest weighted success for adaptive horizon: 99.73%.
๐ LiteSocNavGym Results (3)
โ๏ธ โ
Not just reaching the goal โ how you get there matters.
๐ LiteSocNavGym Results (1)
๐ DreamerV3 + CGLSTM achieves 96% success.
๐ LiteSocNavGym Results (2)
๐ Adaptive horizon yields balanced success & cost.
โฑ Training Time per Model
โฑ DreamerV3 models take ~5โ10ร longer to train.
๐ Reward vs. Wall Clock
๐ Steady improvement over time โ some models converge faster than others.
โ๏ธ Inference Time vs. Return
โ๏ธ High-performing models often require higher inference cost โ balance matters.
๐ Conclusion
- ๐ Novel Predictive Models: 2StepAhead, MASPM, and CGLSTM address short-horizon and reactive limitations in social robot navigation.
- โ Enhanced Performance: Demonstrated higher safety, human comfort, and success rates in diverse simulation environments.
- ๐ Adaptive Horizons in RL: Introduced an entropyโdriven approach that surpasses fixed-horizon methods and enhances continuous-control capabilities.
๐ Future Work
Deploy RL models on a real robot.
Publications Arising from this Thesis
- ๐ฏ Oguzie, Goodluck, e Anikรณ Ekรกrt (2023). "Predictive World Models for Social Navigation." In: Advances in Computational Intelligence Systems, Contributi Presentati al 22ยฐ UK Workshop on Computational Intelligence (UKCI). Advances in Intelligent Systems and Computing (AISC). Springer. (In stampa)
- ๐ Oguzie, Goodluck. "Cosine-Gated LSTM." In: 2024 IEEE 5ยช Conferenza Internazionale su Pattern Recognition e Machine Learning (PRML), pp. 8โ15. IEEE, 2024.
- ๐ Oguzie, Goodluck. "Adaptive Prediction Horizons in RL." (In Progress).
๐ References (1โ8)
- R. C. Arkin, Behavior-Based Robotics. MIT Press, 1998.
- D. Helbing and P. Molnar, โSocial force model for pedestrian dynamics,โ Phys. Rev. E, vol. 51, no. 5, p. 4282, 1995.
- J. van den Berg, S. J. Guy, M. Lin, and D. Manocha, โReciprocal n-body collision avoidance,โ in Robotics Research. Springer, 2011.
- V. Mnih, K. Kavukcuoglu, D. Silver, et al., โPlaying Atari with deep reinforcement learning,โ arXiv preprint arXiv:1312.5602, 2013.
- R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. MIT Press, 2018.
- T. P. Lillicrap, J. Hunt, A. Pritzel, et al., โContinuous control with deep reinforcement learning,โ arXiv preprint arXiv:1509.02971, 2015.
- J. Schulman, F. Wolski, P. Dhariwal, et al., โProximal policy optimization algorithms,โ arXiv preprint arXiv:1707.06347, 2017.
- S. Fujimoto, H. van Hoof, and D. Meger, โAddressing function approximation error in actor-critic methods,โ in Proc. 35th Int. Conf. Machine Learning (ICML), 2018.
๐ References (9โ15)
- T. Haarnoja, A. Zhou, K. Hartikainen, et al., โSoft actor-critic algorithms and applications,โ arXiv preprint arXiv:1812.05905, 2018.
- D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba, โMastering diverse domains through world models,โ arXiv preprint arXiv:2301.04104, 2023.
- D. E. Rumelhart, G. E. Hinton, and R. J. Williams, โLearning representations by back-propagating errors,โ Nature, vol. 323, pp. 533โ536, 1986.
- S. Hochreiter and J. Schmidhuber, โLong short-term memory,โ Neural Computation, vol. 9, no. 8, pp. 1735โ1780, 1997.
- K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio, โOn the properties of neural machine translation: Encoder-decoder approaches,โ arXiv preprint arXiv:1409.1259, 2014.
- A. Vaswani, N. Shazeer, N. Parmar, et al., โAttention is all you need,โ in Advances in Neural Information Processing Systems (NIPS), 2017.
- D. Ha and J. Schmidhuber, โRecurrent world models facilitate policy evolution,โ in Advances in Neural Information Processing Systems (NIPS), 2018.