Di Zhang

Research on RL and Multi-modal Reasoning of Vision Language Models (VLMs) • Pioneered a diverse, domain-specific dataset framework to tackle challenges in visual reasoning for small-scale (2B-level) VLM training. This framework helped clarify which tasks were suitable for reasoning and enabled effective multi-stage distillation, allowing our 2B model to achieve performance comparable to much larger models on benchmarks like MMMU and MathVista. • Contributed this refined dataset to NVIDIA’s Nemotron-VL and Issac Gr00t series, making it a core component of their reasoning training pipeline and thus enhancing reasoning capabilities across NVIDIA’s entire LLM suite.

Chemical Large Language Models and Agents • Innovatively applied LLMs to chemistry at a time when existing models struggled with basic molecule generation. We worked directly with chemists to understand their needs, then built a template-driven dataset from sources like PubChem and reaction databases. This allowed our model to handle chemistry Q&A far better than GPT-3.5 on domain tasks. • Extended these methods to multimodal tasks, enabling the LLM to interpret chemical diagrams and visual inputs using LLaVA. This made it a powerful tool in both text and visual chemistry domains, outperforming previous models on our custom test sets.

Recommendation System Algorithm Engineer at Alibaba Inc. • Acted as a primary algorithm designer for the LLM-based product listing assistant at Xianyu (Idle Fish), China’s largest C2C second-hand marketplace. This innovation automated product descriptions and title optimization, significantly lowering the entry barrier for sellers and improving listing quality and matching efficiency. • Led the enhancement of RAG-driven search, enabling buyers with different price preferences to find the most suitable products faster. This not only streamlined the listing process but also boosted transaction match rates across the marketplace.

Research on Embodied human-computer interaction of humanoid robot • Explored the security aspects of embodied human-robot interaction and integrated advanced NLP and dialogue systems to enhance robot autonomy. Contributed to the "Jiajia" service robot and the “Xiaochuan” Giant Panda Robot projects, aligning with modern trends in embodied learning for more intuitive and context-aware human-robot collaboration.