SimScale: Learning to Drive via Real-World Simulation at Scale
Haochen Tian, Tianyu Li, Haochen Liu, Jiazhi Yang, Yihang Qiu, Guang Li, Junli Wang, Yinfeng Gao, Zhang Zhang, Liang Wang, Hangjun Ye, Tieniu Tan, Long Chen, Hongyang Li
2025-12-03
Summary
This paper focuses on improving self-driving cars by making them better at handling unusual or dangerous situations, even ones they haven't specifically been trained on.
What's the problem?
Self-driving cars learn from data collected by humans driving, but rare and critical events like near-accidents aren't well represented in this data. It's hard to get enough real-world examples of these situations to train the car to react safely, creating a data diversity problem.
What's the solution?
The researchers created a sophisticated computer simulation environment. They take existing driving data and use it to generate many new, realistic scenarios, including challenging ones. The simulation can change things like weather, other cars, and even the car's own path to create these new situations. They also created a way for the simulated car to 'learn' the best actions to take in these new scenarios, essentially creating a virtual expert. Finally, they trained the self-driving car using a combination of real-world data and this newly generated simulated data.
Why it matters?
This work is important because it allows self-driving cars to become much safer and more reliable. By using simulation to fill in the gaps in real-world data, the cars can learn to handle a wider range of situations, even those they've never encountered before. The fact that improvements continue with more simulation data, even without more real-world data, is a big step towards truly autonomous driving.
Abstract
Achieving fully autonomous driving systems requires learning rational decisions in a wide span of scenarios, including safety-critical and out-of-distribution ones. However, such cases are underrepresented in real-world corpus collected by human experts. To complement for the lack of data diversity, we introduce a novel and scalable simulation framework capable of synthesizing massive unseen states upon existing driving logs. Our pipeline utilizes advanced neural rendering with a reactive environment to generate high-fidelity multi-view observations controlled by the perturbed ego trajectory. Furthermore, we develop a pseudo-expert trajectory generation mechanism for these newly simulated states to provide action supervision. Upon the synthesized data, we find that a simple co-training strategy on both real-world and simulated samples can lead to significant improvements in both robustness and generalization for various planning methods on challenging real-world benchmarks, up to +6.8 EPDMS on navhard and +2.9 on navtest. More importantly, such policy improvement scales smoothly by increasing simulation data only, even without extra real-world data streaming in. We further reveal several crucial findings of such a sim-real learning system, which we term SimScale, including the design of pseudo-experts and the scaling properties for different policy architectures. Our simulation data and code would be released.