Jianwei Yang

Build Multimodal Omni Models.

Build multimodal agentic AI foundation

1. Led development of multimodal agentic models at MSR, achieving state-of-the-art on both multimodal image and video understanding, and agentic tasks like UI navigation and robot manipulation. 2. Co-led the development of Phi-3-V with Microsoft GenAI, including model architecture design and experimenting, achieving the state-of-the-art for multimodal image understanding tasks with 4B model size. 3. Invented Set-of-Mark (SoM) prompting and co-developed OmniParser for GPT-4V, which fostered recent huge progress of multimodal agent development such as UI navigation and robotics manipulation 4. Led and contributed to LLaVA variants, including LLaVA-Med, LLaVA-Plus, LLaVA-Grounding, LlaVA Interactive, to enhance the medical understanding, grounding and tool use capability of multimodal LLMs.

1. Led the development of uni ed and interactive dense vision models, including X-Decoder, SEEM, Semantic SAM, achieving the SoTA performance for various image segmentation tasks. 2. Led and contributed to the development of advanced grounding vision models, RegionCLIP, GLIP and GroundingDINO, achieving SoTA performance for zero-shot and open-vocabulary object detection. 3. Co-led the development of Microsoft Florence with Azure AI by contributing the core learning algorithm Unified Contrastive Learning (UniCL), achieved SoTA on 40+ vision and multimodal benchmarks. 4. Invented two vision architectures, Focal Transformer and Focal Modulation Networks, which achieved SoTA image classi cation and object detection performance on ImageNet-1K and COCO Detection.