Yu Rong

Working on multimodal AI agents, embodied AI, video generation, human avatar creation, and 3D computer vision. - Building toward real-time multimodal embodied AI agents that understand text, speech, facial expression, and visual signals, and generate speech plus photorealistic digital human behavior. - Developed components across the multimodal GenAI stack, including unified RVQ tokenizers for human motion and semantic behavior modeling, LLM-based speech / TTS model extensions with human-latent modalities, audio-controlled avatar animation models with flow matching, and WAN-based video generation models controllable from audio and text. - Work involving data preprocessing and curation, model training, evaluation pipeline development, and training / inference infrastructure for multimodal avatar agent research. - Built human-tracking foundation models for large-scale image and video understanding, using ViT backbones pretrained on billion-scale image data and post-trained with synthetic data. Added prediction heads for human motion and dense landmark estimation, and owned model training and pipeline curation for million-scale video processing. - Developed high-fidelity offline face-tracking algorithms and production-scale processing pipelines for indoor multi-view and in-the-wild datasets, including algorithm design, implementation, quantitative analysis, and large-scale GPU processing. - Built a live AR avatar-driving system that takes desk-mounted camera inputs and predicts headposes and facial-expression latents in real time. Work involving raw data collection, training data generation, deep tracking model training, and live deployment.

We use SMPL-X to represent whole body 3D motion including face, hands and body. Separate modules is adopt to predict independent body part first. Part motion predictions are then composed by the integration module to obtain unified whole body motion results. We design three different integration modules which trade off between accuracy and speed. Our system, both fast mode and slow-yet-precise mode, largely surpass the previous methods for in-the-wild scenarios. Demo, code and models are available at the project page.