Chinese Research Team Reworks Driving VLA Training Around Spatial Cues
A joint research team from Beihang University, Tsinghua University's Institute for AI Industry Research , DiDi and Communication University of China has developed DriveTeach-VLA, a new training framework designed to address a fundamental weakness in vision-language-action models for autonomous driving — the disconnect between multimodal perception and trajectory planning. The paper has been accepted at ECCV 2026, with code and models publicly released.
The Core Problem: Language Supervision Alone Is Not Enough
Vision-language-action (VLA) models have attracted growing interest in autonomous driving research. Built on multimodal large language model (MLLM) backbones such as Qwen, these systems can process camera feeds, interpret driving instructions and produce natural language explanations of road scenes. Their advantage over conventional end-to-end models lies in the ability to leverage large-scale visual-language pretraining.
But a structural gap persists. Pretraining equips VLA models with scene description capabilities; trajectory regression — the ability to predict a continuous path through physical road space — must be learned from scratch during post-training. The research team found that a directly fine-tuned VLA baseline scatters attention across the entire image during inference, suggesting that standard post-training with visual question answering (VQA) and chain-of-thought supervision fails to anchor the model to the visual elements that actually govern driving decisions: pedestrians, vehicles, lane markings and traffic signals.
As the team frames it, knowing that "there is a pedestrian ahead" is a linguistic fact. Knowing which side the pedestrian occupies, whether the vehicle must maneuver around it, and how the drivable area will shift — these are the spatial relationships that determine control output. Bridging this gap is the central objective of DriveTeach-VLA.
Two Targeted Interventions
The framework introduces two complementary training components.
The first, Driving-aware Vision Distillation (DVD), recalibrates the visual encoder using a teacher-student distillation scheme. A teacher branch receives images overlaid with bounding boxes generated by GroundingDINO for traffic-relevant objects — vehicles, pedestrians, road barriers, traffic signs and signals. A student branch receives only the original image and learns to replicate the teacher's feature representations through patch-level alignment. This approach embeds driving-relevant visual priors directly into the encoder without requiring additional question-answer annotations. Ablation experiments showed DVD delivers approximately 1.4 points of improvement on the PDMS metric, and the gains remain stable even after introducing 20% bounding box noise through random dropping and positional jitter.
The second component, 2D Trajectory-Guided Prompt (2D-TGP), addresses the coordinate mismatch between expert trajectories and model inputs. Autonomous driving datasets record expert trajectories in bird's-eye view (BEV) or vehicle coordinate systems — representations suited for planning error computation but misaligned with the front-view images a VLA model processes. DriveTeach-VLA projects future BEV trajectories onto the image plane using camera intrinsic and extrinsic parameters, converting them into sequences of 2D image coordinates. These coordinates are then formatted as text prompts, giving the model a spatially grounded intermediate representation that connects visual regions to future motion.
A Two-Model Architecture
The full system uses two separate models, both built on Qwen2.5-VL-3B. A TGP-Prompter predicts the 2D image-plane trajectory from the current image, vehicle state and driving instruction. A TGP-Planner receives all of the above plus the 2D-TGP and generates the final BEV trajectory. Separating these functions avoids competition between text instructions, image coordinates and BEV trajectory outputs within a single model, stabilizing training. A final GRPO reinforcement learning stage aligns outputs with driving quality objectives spanning collision avoidance, road compliance, comfort and traffic rule adherence.
Incremental ablation results illustrate the contribution of each component. The Qwen2.5-VL-3B baseline achieves a PDMS of 84.8. Adding VQA and chain-of-thought supervision raises this to 86.4. DVD brings it to 87.3. Incorporating 2D-TGP without GRPO reaches 88.2, and the complete training pipeline reaches 90.4.
Benchmark Results
On the NAVSIM Navtest benchmark, DriveTeach-VLA achieves 90.4 PDMS and 85.4 EPDMS using a single front-facing camera and a 3B-parameter backbone. On nuScenes open-loop trajectory prediction, the model records an L2 error of 0.30 and a collision rate of 0.12. Attention analysis confirms that DVD's effects extend beyond final scores: the attention mass on traffic-critical regions increases from 2.28×10⁻² in the baseline to 3.19×10⁻², indicating a measurable shift in how the model allocates visual focus during planning.
With multi-candidate sampling (N=12) and the DrivorR selector, performance reaches 92.7 PDMS and 89.0 EPDMS — a result the authors characterize as an exploration of the upper bound of autoregressive VLA sampling rather than a direct comparison to single-inference figures.
Deployment Efficiency
On an H100 GPU, the dual-model system incurs a total inference latency of 3.03 seconds and requires 17.2 GiB of memory. Compared to a single-model multi-turn QA baseline, this represents an increase of 0.16 seconds in latency and 8.6 GiB in memory, while delivering a 1.2-point PDMS improvement. The authors note that the 17.2 GiB memory footprint falls within practical deployment constraints.
The lead author, Yang Yuguang, is a doctoral candidate at Beihang University whose prior work includes CuriousVLA, described as the first MLLM-native autonomous driving model, and contributions to the CLOVER planning model. DriveTeach-VLA is presented as a continuation of that research line.
Related Coverage:
Didi Autonomous Driving Start 24/7 Fully Driverless Testing in Guangzhou