Letian Wang 王乐天

Hi there! I'm a Ph.D. student at the University of Toronto, where I am fortunate to be supervised by Prof. Steven Waslander (Nicest Steve Ever!) in the Toronto Robotics and AI Lab. I'm affiliated with Vector Institute. I have been fortunate to do research at/with:

  • Google DeepMind, with
  • NVIDIA Research, with
  • and as a Research Assistant at/with
  • CMU Robotics Institute, with Prof. Changliu Liu (2021)
  • UC Berkeley, with Prof. Masayoshi Tomizuka (2020)
  • I obtained bachelor's degree with highest honor (President Award) at Beihang University.

    I'm fascinated by intelligent systems that can generalize beyond their training experience and adapt to the open world, much like humans do. I started my research in robotics, and this curiosity has since taken me across multimodal learning, LLM agents, and reinforcement learning. Most recently, I've been working on general-purpose video models [1] and long-term memory for LLM agents [2]. I'm particularly drawn to simple, unified approaches that scale effectively, and I do not get tired when trying to understand how scaling works across data, compute, model architectures, and task complexity.

    I have authored 2 books [3] [4], was the awardee of Qualcomm Fellowship 2025, the winner of 2022 CARLA autonomous driving challenge, and won the best paper award honorable mention at RA-L 2021, first prize in the National Challenge Cup, China.


    G. Scholar     LinkedIn     Twitter

    I'm looking for industrial positions from 2026.

    letianwang0 at gmail dot com

    Innovation GenCeption is released with code/weight! Also see my thoughts after building it at Google DeepMind
    Press WallStreet Journal and Google AI blog post on our video foundation models (THFM and GenCeption).
    Awardee Qualcomm Fellowship Proposal: Spatiotemporal Reasoning for Unified Perception and Prediction World Models
    Contributor MiroThinker A state-of-the-art open-source LLM deep research model
    2022.09 - Present
    UofT

    PhD student at University of Toronto, supervised by Steven Waslander. We're interested in perception and decision-making algorithms that can generalize and adapt to the open world.

    2025 - 2026
    DeepMind

    Student Researcher at Google DeepMind, with Misha Andriluka, Andrei Zanfir, Eduard Bazavan, João Carreira, Andrew Zisserman, Kaiming He. I am working on unified text-video models.

    2025 -
    MiriMind AI

    Research intern at MiroMind, with Jifeng Dai and Lidong Bing. I worked on LLM deep research agents, long-term memory, long context and efficient sparse attention. Contributor to MiroThinker, the state-of-the-art open-source deep research model.

    2023.09 - 2024.06
    NVIDIA

    Research intern at NVIDIA Research, with Peter Karkus, Seung Wook Kim, Boris Ivanovic, Yue Wang, Sanja Fidler, and Marco Pavone. I worked on self-supervised 3D-aware visual pretraining via generalizable neural radiance field [NeurIPS'24].

    2020.10 - 2022.05
    Carnegie Mellon University

    Research assistant at the Robotics Institute of Carnegie Mellon University, with Changliu Liu and Yeping Hu. I worked on generalizable motion prediction algorithms in different scenarios, and social interaction for autonomous driving [NeurIPS'21, AAAI'22, Book]

    2019.10 - 2020.10
    UC Berkeley

    Research assistant at UC Berkeley, with Masayoshi Tomizuka, Liting Sun, and Wei Zhan. I worked on socially-compatible behavior generation for autonomous driving [RA-L'21] (Best Paper Award Honorable Mention)

    2018.07 - 2018.09
    Hong Kong University of Science and Technology

    Research assistant at the HKUST RI, with Fu Zhang. I worked on unified control strategy for vertically take-off and landing UAVs.

    See my Google Scholar profile for a more up-to-date list.
    * denotes equal contribution

    Video Generation Models are General-Purpose Vision Learners

    Letian Wang, Chuhan Zhang, Rishabh Kabra, Jasper Uijlings, Steven Waslander, Andrew Zisserman, Joao Carreira, Kaiming He, Misha Andriluka, Eduard Gabriel Bazavan, Andrei Zanfir, Cristian Sminchisescu
    European Conference on Computer Vision (ECCV 2026)
    Can video generation do for vision what LLM did for language? Yes
    Webpage  •  PDF  •  Code

    BeyondSight: Object Permanence for End-to-End Autonomous Driving

    Sandro Papais, Letian Wang, M Jain, B Rezaei, Steven L. Waslander
    European Conference on Computer Vision (ECCV 2026)
    PDF

    LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving

    Hao Shao, Letian Wang, Yang Zhou, Y Hu, Zhuofan Zong, Steven L. Waslander, Wei Zhan, Hongsheng Li
    European Conference on Computer Vision (ECCV 2026)
    PDF

    VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning

    J Xu, R Wang, F Pu, M Wang, R Ji, Tong Zhou, C Gu, J Zuo, Han Xiao, Y Geng, Letian Wang, et al.
    arXiv preprint arXiv:2608.26105
    PDF

    Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives

    Letian Wang, M A Lavoie, Sandro Papais, B Nisar, Y Chen, W Ding, Boris Ivanovic, et al.
    Foundations and Trends in Robotics 13 (1-2), 2026 (Book)
    PDF

    DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving

    Yang Zhou, Hao Shao, Letian Wang, Zhuofan Zong, Hongsheng Li, Steven Waslander
    International Conference on Learning Representations (ICLR 2026)
    PDF

    Grounded World Model for Semantically Generalizable Planning

    Q Li, L Feng, H Zhang, W Li, Letian Wang, A Alahi, H Soh
    arXiv preprint arXiv:2604.11751
    PDF

    DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning

    Yang Zhou, X Wang, Hao Shao, Letian Wang, G Zhao, J Shao, J Zhu, T Yu, Z Zhu, et al.
    arXiv preprint arXiv:2604.01765
    PDF

    THFM: A Unified Video Foundation Model for 4D Human Perception and Beyond

    Letian Wang, Andrei Zanfir, Eduard Gabriel Bazavan, Misha Andriluka, Cristian Sminchisescu
    arXiv preprint arXiv:2603.25892
    PDF

    SCATR: Mitigating New Instance Suppression in LiDAR-based Tracking-by-Attention via Second Chance Assignment and Track Query Dropout

    Brian Cheong, Letian Wang, Sandro Papais, Steven L. Waslander
    IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2026)
    PDF

    MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling

    MiroMind Team
    arXiv preprint arXiv:2511.11793
    Strongest Open-Source LLM Deep Research at the time
    PDF

    OmniGen: Unified Multimodal Sensor Generation for Autonomous Driving

    T Tang, E Ma, X Zhou, Letian Wang, T Yan, X Zhang, K Zhan, P Jia, X Lang, et al.
    ACM International Conference on Multimedia (ACM MM 2025)
    PDF

    ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting

    Sandro Papais, Letian Wang, Brian Cheong, Steven L. Waslander
    IEEE/CVF International Conference on Computer Vision (ICCV 2025)
    PDF

    OpenNav: Open-World Navigation with Multimodal Large Language Models

    M Yuan, Letian Wang, Steven L. Waslander
    IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2025)
    PDF

    SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction

    Yang Zhou*, Hao Shao*, Letian Wang*, Steven L. Waslander, Hongsheng Li, Yu Liu
    International Conference on Learning Representations (ICLR 2025)
    Exploring the Scaling Laws in Motion Prediction
    PDF  •  

    DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features

    Letian Wang, Seung Wook Kim, Jiawei Yang, Cunjun Yu, Boris Ivanovic, Steven L Waslander, Yue Wang, Sanja Fidler, Marco Pavone, Peter Karkus
    Advances in Neural Information Processing Systems (NeurIPS 2024)
    Exploring Self-Supervised 3D-Aware Visual Pretraining.
    Webpage  •   PDF  •   Video

    Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

    Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, Hongsheng Li
    Advances in Neural Information Processing Systems (NeurIPS 2024, Spotlight)
    PDF  •  

    SmartRefine: An Scenario-Adaptive Refinement Framework for Efficient Motion Prediction

    Yang Zhou, Hao Shao, Letian Wang, Steven L Waslander, Hongsheng Li, Yu Liu
    IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024)
    Outperform all published ensemble-free works on Argoverse 2 leaderboard (single agent track).
    PDF  •   Code

    LmDrive: Closed-Loop End-to-End Driving with Large Language Models

    Hao Shao, Yuxuan Hu, Letian Wang, Steven L Waslander, Yu Liu, Hongsheng Li
    IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024)
    First work to bring LLM into closed-loop end-to-end autonomous driving.
    Webpage  •   PDF  •   Code

    Accelerating Reinforcement Learning for Autonomous Driving using Task-Agnostic and Ego-Centric Motion Skills

    Tong Zhou*, Letian Wang*, Ruobing Chen, Wenshuo Wang, Yu Liu
    IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2023)
    PDF  •  

    Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors

    Letian Wang, Jie Liu, Hao Shao, Wenshuo Wang, Ruobing Chen, Yu Liu, Steven L Waslander
    Robotics: Science and Systems (RSS 2023)
    PDF  •   Code Talk

    ReasonNet: End-to-End Driving with Temporal and Global Reasoning

    Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslander, Hongsheng Li, Yu Liu
    Conference on Computer Vision and Pattern Recognition (CVPR 2023)
    Winner of CARLA Autonomous Driving Challenge 2022
    PDF  •   Code

    Social Interactions for Autonomous Driving: A Review and Perspectives

    Wenshuo Wang, Letian Wang, Chengyuan Zhang, Changliu Liu, Lijun Sun
    Foundation and Trends in Robotics (Book)
    PDF  •  

    Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer

    Hao Shao*, Letian Wang*, Ruobing Chen, Hongsheng Li, Yu Liu
    Conference on Robot Learning 2022
    First Place on the CARLA Leaderboard
    PDF  •   Code

    Efficient Game-Theoretic Planning with Prediction Heuristic for Socially-Compliant Autonomous Driving

    Chenran Li, Tu Trinh, Letian Wang, Changliu Liu, Masayoshi Tomizuka, Wei Zhan
    IEEE Robotics and Automation Letters 2021
    PDF  •  

    Human Instruction Following: Graph Neural Network Guided Object Navigation

    Hongyi Chen, Letian Wang, Yuhang Yao, Ye Zhao, Patricio Vela
    The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022, Workshop on Embodied AI
    PDF  •  

    Transferable and Adaptable Driving Behavior Prediction

    Letian Wang, Yeping Hu, Liting Sun, Wei Zhan, Masayoshi Tomizuka, Changliu Liu
    arxiv
    PDF  •  

    Online Adaptation of Neural Network Models by Modified Extended Kalman Filter for Customizable and Transferable Driving Behavior Prediction

    Letian Wang, Yeping Hu, Changliu Liu
    AAAI Conference on Artificial Intelligence, Workshop on Human-Centric Self-Supervised Learning
    PDF  •  

    Hierarchical Adaptable and Transferable Networks (HATN) for Driving Behavior Prediction

    Letian Wang, Yeping Hu, Liting Sun, Wei Zhan, Masayoshi Tomizuka, Changliu Liu
    Conference on Neural Information Processing Systems (NeurIPS 2021), Workshop on Machine Learning for Autonomous Driving (Spotlight)
    PDF  •  

    Socially-Compatible Behavior Design of Autonomous Vehicles with Verification on Real Human Data

    Letian Wang, Liting Sun, Masayoshi Tomizuka, Wei Zhan
    IEEE Robotics and Automation Letters 2021 Best Paper Award - Honorable Mention
    PDF  •  

    Overall Design and Control of Coaxial Tilt Rotor Vertically Take-off-and-Landing UAV

    Letian Wang, Yuhan Lu, Yibo Liu, Yicong Fu, Bonan Xu, Jingyu Zhao, Qi Qian, Yifan Yan, Weijun Wang
    First prize in National Challenge Cup 2017 (全国挑战杯一等奖, known as the Sci./Tech. Olympics among universities in China).

    Some of my slides can be found here

    2024

    Toronto TechTalk

    2021

    NeurIPS Workshop on Machine Learning for Autonomous Driving

    2020

    INFORMS Annual Meeting

    2020+

    Reviewer: IJRR, RSS, NeurIPS, CVPR, ICRA, IROS, ML4AD, TNNLS, TVT, ITS, IV