Zhiding Yu

I conduct research in multimodal learning and intelligent data strategies. I lead the Eagle VLM project which develops a family of frontier vision-language models with public training/data recipes and state-of-the-art performance matching or outperforming existing top-tier VLMs. Our work has laid the core VLM foundation and data strategy behind several flagship NVIDIA products/projects, including Llama-Nemotron-VL, Nemo Retriever Multimodal Embedding, GR00T N1, and GR00T N1.5.

Participated in a multi-org effort to design NVIDIA’s next-generation AV system. Led a team to design and develop a Transformer-based 3D perception system for joint 3D object detection, tracking and online mapping at long distances. Developed Transformer-based neural planner and DriveVLM for E2E driving, with community-recognized works such as BEV-Planner, OmniDrive, and Hydra-MDP.

Led the earliest effort to develop Vision Transformers at NVIDIA which partially shaped the landscape of NVIDIA’s internal AI product. Applications of my work include scene understanding, robust general purpose backbone, autonomous driving perception and scalable auto-labeling pipelines. Proposed multiple Transformer-based bird’s-eye view (BEV) perception frameworks with SOTA results in 3D object detection, tracking, and 3D occupancy prediction. Some works from this period with community impact include SegFormer (Most Influential NeurIPS Papers, 3K Stars), VoxFormer (CVPR23 Highlight, 1.1K Stars), FB-BEV/FB-OCC (ICCV23, 735 Stars), and FocalFormer3D (ICCV23, Ranked 1st on nuScenes LiDAR 3D Detection and Tracking Leaderboard (Mar. 2023)). Successful tech transfers to numerous NVIDIA products, including AV and NVIDIA TAO Toolkit.

Worked extensively on label-efficient learning and transfer learning. Proposed weakly supervised, semi-supervised and self-supervised learning frameworks, with SOTA performance in visual recognition applications. Proposed various unsupervised domain adaptation and synthetic-to-real generalization methods for improved model robustness and generalization “in the wild”.

Deep category-aware semantic edge detection Proposed an end-to-end deep semantic edge learning framework with state-of-the-art performance One paper accepted to CVPR 2017

Deep CNN based static facial expression recognition Partly integrated to the Microsoft Emotion Recognition API under Project Oxford First Runner Up at the EmotiW-SFEW Challenge 2015 Full-length paper accepted at ACM-ICMI 2015 as Oral

Intern Project: PixelTone