Renrui Zhang

Education

• [2017-2021] 🎉 I received my B.E. degree from Peking University, awarded Outstanding Graduate (Top 5%).
• [2020-2021] I worked as a visiting student in University of Pennsylvania, supervised by Prof. Jianbo Shi.
• [2021-Now] 🎓 I'm pursuing my Ph.D. in MMLab, CUHK, supervised by Prof. Hongsheng Li and Prof. Xiaogang Wang.
• [2021-2024] I worked as a research intern at Shanghai AI Lab, supervised by Dr. Peng Gao.
• [2024-2025] I worked as a research intern at LLaVA team, ByteDance, Seattle, supervised by Dr. Chunyuan Li.
• [2025-Now] 💪 I joined SEED (Multimodal Interaction & World Model), ByteDance, San Jose.

Biography

📌 My research interests include Large Multimodal Models, Vision-language Learning, Emboided AI, and 3D Vision.

✉️ I'm looking for undergraduate and graduate students for academic cooperation. Discussions are welcome!

News

• [2025-05] 🔥 We release "T2I-R1", introducing R1 into image generation domains for CoT reasoning.
• [2025-05] One paper accepted by ICML 2025
• [2025-03] 🔥 We release "HybridVLA", the first work unifying Autoregression and Diffusion in VLA models.
• [2025-02] Three papers accepted by CVPR 2025, one Highlight 🎉
• [2025-01] Five papers accepted by ICLR 2025, two Spotlight 🎉
• [2025-01] 🔥 We release "Image Generation with CoT", the first work investigating CoT strategies (e.g., Test-time Scling, RL, and Reflection) in autoregressive text-to-image generation.
• [2025-01] 🎉 "Video-MME", is thrilled to be selected as One of the 14 Groundbreaking Stuides in 2024.
• [2024-08] 🔥 We release "LLaVA-OneVision", the latest LLaVA model for image, video, and image-text interleaved scenarios with superior performance.
• [2024-07] Four papers accepted by ECCV 2024
• [2024-07] 🔥 We release "LLaVA-NeXT-Interleave" for multi-image instruction tuning and "MAVIS" for multimodal mathematical reasoning.
• [2024-05] Three papers accepted by ICML 2024
• [2024-03] Seven papers accepted by CVPR 2024, two Highlight 🎉
• [2024-03] 🔥 We release "MathVerse", a novel mathematical benchmark with the first CoT evaluation strategy.
• [2024-01] Four papers accepted by ICLR 2024

Selected Projects

* Equal Contribution    # Project Lead

♠ o1/R1-like Chain-of-Thought (CoT) Reasoning

• 🔥 Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step
  Ziyu Guo*, Renrui Zhang#*, Chengzhuo Tong*, Zhizheng Zhao*, Haoquan Zhang, Manyuan Zhang, Peng Gao, Hongsheng Li, Pheng-Ann Heng
  The first work investigating CoT strategies (e.g., Test-time Scling, RL, and Reflection) in autoregressive T2I
  CVPR 2025
  
  [Paper] [Code 🌟500+]

• 🔥 MME-CoT: Benchmarking CoT in LMMs for Reasoning Quality, Robustness, and Efficiency
  Dongzhi Jiang*, Renrui Zhang#*, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, Bo Zhang, Chaoyou Fu, Peng Gao, Hongsheng Li
  The first-ever comprehensive benchmark to evaluate CoT reasoning capabilities in LMMs
  arXiv 2025
  
  [Paper] [Project] [Code]

• MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine
  Renrui Zhang#*, Xinyu Wei*, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, Peng Gao, Chunyuan Li, Hongsheng Li
  The first specialized LMM for multimodal mathematical problem-solving (CLIP-Math + CoT SFT + DPO)
  ICLR 2025
  
  [Paper] [Code]

• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
  Renrui Zhang#*, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, Hongsheng Li
  A novel benchmark assessing visual comprehension and CoT capabilities of LMMs for mathematics
  ECCV 2024
  
  [Paper 🎓100+] [Project] [Code]

♠ Large Language & Multimodal Models (LLMs & LMMs)

• 🔥 LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-initialized Attention
  Renrui Zhang*, Jiaming Han*, Dongyang Liu*, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, Peng Gao
  The pioneering work performing visual instruction tuning for LLMs, prior to LLaVA and MiniGPT-4
  ICLR 2024
  
  [Paper 🎓800+] [Code 🌟5.8K+]

• 🔥 LLaVA-OneVision: Easy Visual Task Transfer
  Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, Chunyuan Li
  The latest, open-sourced LLaVA model superior at image, video, and image-text interleaved scenarios
  TMLR 2025
  
  [Paper 🎓400+] [Project] [Code 🌟3.6K+]

• 🔥 LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
  Feng Li*, Renrui Zhang*, Hao Zhang*, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, Chunyuan Li
  The upgraded LLaVA model handling image-text interleaved, multi-image, video, and 3D tasks
  ICLR 2024 Spotlight 🎉
  
  [Paper 🎓100+] [Project] [Code 🌟3.6K+]

• Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following
  Ziyu Guo*, Renrui Zhang#*, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, Pheng-Ann Heng
  Point-LLM, the first LLM in 3D world, prior to 3D-LLM and PointLLM, and Point-Bind, a multi-modal 3D large model
  arXiv 2024
  
  [Paper 🎓100+] [Code 🌟400+]

• Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis
  Chaoyou Fu#, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, Xing Sun
  A comprehensive and thorough benchmark for evaluating LMMs in video understanding
  CVPR 2025
  
  🎉 Selected as One of the 14 Groundbreaking Stuides in 2024
  
  [Paper 🎓200+] [Project] [Code 🌟400+]

• MMSearch: Unveiling the Potential of Large Models as Multi-modal Search Engines
  Dongzhi Jiang*, Renrui Zhang#*, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, Yu Liu, Chunyuan Li, Hongsheng Li
  ICLR 2025
  

• SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models
  Dongyang Liu*, Renrui Zhang*, Longtian Qiu*, Siyuan Huang*, Weifeng Lin*, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, Wenqi Shao, Chao Xu, Conghui He, Junjun He, Hao Shao, Pan Lu, Hongsheng Li, Yu Qiao, Peng Gao
  ICML 2024
  

• ImageBind-LLM: Multi-modality Instruction Tuning
  Jiaming Han*, Renrui Zhang*, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, Xudong Lu, Shuai Ren, Yafei Wen, Xiaoxin Chen, Xiangyu Yue, Hongsheng Li, Yu Qiao
  arXiv 2023
  

♠ Large Vision Models

• 🔥 Personalize Segment Anything Model with One Shot
  Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, Hongsheng Li
  Customize SAM model to segment personal concepts in a zero-shot or parameter-efficient manner
  ICLR 2024
  
  [Paper 🎓200+] [Code 🌟1.6K+]

• 🔥 SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Manners
  Ziyu Guo*, Renrui Zhang#*, Xiangyang Zhu, Chengzhuo Tong, Peng Gao, Chunyuan Li, Pheng-Ann Heng
  The "Segment Anything Moment" in 3D world for 3D objects, indoor & outdoor scenes, and raw LiDARs
  arXiv 2024
  
  [Paper] [Project] [Live Demo 🎨] [Code 🌟300+]

♠ Emboided AI & Robotics

• 🔥 HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model
  Jiaming Liu*, Hao Chen*, Pengju An, Zhuoyang Liu, Renrui Zhang#, Chenyang Gu, Xiaoqi Li, Ziyu Guo, Sixiang Chen, Mengzhen Liu, Chengkai Hou, Mengdi Zhao, KC Zhou, Pheng-Ann Heng, Shanghang Zhang
  The first unified autoregression and diffusion framework in VLA models
  arXiv 2025
  
  [Paper] [Project] [Code 🌟100+]

• Lift3D Foundation Policy: Lifting 2D Large-scale Pretrained Models for Robust 3D Robotic Manipulation
  Yueru Jia*, Jiaming Liu*#, Sixiang Chen*, Chenyang Gu, Zhilue Wang, Longzan Luo, Lily Lee, Pengwei Wang, Zhongyuan Wang, Renrui Zhang#, Shanghang Zhang
  Adapting frozen 2D pre-trained models effectively for robust 3D robotic manipulation
  CVPR 2025
  
  [Paper] [Project] [Code 🌟100+]

• RoboMamba: Multimodal State Space Model for Efficient Robot Reasoning and Manipulation
  Jiaming Liu*#, Mengzhen Liu*, Zhenyu Wang, Lily Lee, Kaichen Zhou, Pengju An, Senqiao Yang, Renrui Zhang#, Yandong Guo, Shanghang Zhang
  Applying Mamba architecture with a simple policy head for efficient robotic manipulation
  NeurIPS 2024
  
  [Paper] [Project] [Code 🌟100+]

♠ Vision-language Learning

• PointCLIP: Point Cloud Understanding by CLIP
  Renrui Zhang*, Ziyu Guo*, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, Hongsheng Li
  CVPR 2022
  

• Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
  Renrui Zhang*, Wei Zhang*, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, Hongsheng Li
  ECCV 2022
  

• Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners
  Renrui Zhang*, Xiangfei Hu*, Bohao Li, Siyuan Huang, Hanqiu Deng, Hongsheng Li, Yu Qiao, Peng Gao
  CVPR 2023
  

• PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning
  Xiangyang Zhu*, Renrui Zhang#*, Bowei He, Ziyu Guo, Ziyao Zeng, Zipeng Qin, Shanghang Zhang, Peng Gao
  ICCV 2023
  

• Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement
  Xiangyang Zhu*, Renrui Zhang#*, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, Peng Gao
  ICCV 2023
  

• CLIP-Adapter: Better Vision-language Models with Feature Adapters
  Peng Gao*, Shijie Geng*, Renrui Zhang*, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, Yu Qiao
  IJCV 2024
  

• Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation
  Yulu Gan, Xianzheng Ma, Yihang Lou, Yan Bai, Renrui Zhang, Nian Shi, Lin Luo
  AAAI 2023 Best Student Paper 🎉
  

♠ 3D Vision & Autonomous Driving

• MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection
  Renrui Zhang, Han Qiu, Tai Wang, Xuanzhuo Xu, Ziyu Guo, Yu Qiao, Peng Gao, Hongsheng Li
  ICCV 2023
  

• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training
  Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, Hongsheng Li

  NeurIPS 2022
  

• Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders
  Renrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao, Hongsheng Li

  CVPR 2023
  

• Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysis
  Renrui Zhang, Liuhui Wang, Yali Wang, Peng Gao, Hongsheng Li, Jianbo Shi
  CVPR 2023
  

• No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentation
  Xiangyang Zhu*, Renrui Zhang#*, Bowei He, Ziyu Guo, Jiaming Liu, Han Xiao, Chaoyou Fu, Hao Dong, Peng Gao

  CVPR 2024 Hightlight 🎉
  

Selected Awards

• [2021-06] Outstanding Graduate, Peking University (Top 5%)
• [2020-09] Academic Excellent Scholarship (Ranked 1^st/73)
• [2020-09] Merit Student PaceSetter, Peking University (Ranked 1^st/73)
• [2019-09] Academic Excellent Scholarship (Ranked 4^th/73)
• [2019-09] Merit Student, Peking University (Ranked 4^th/73)
• [2016-07] China Youth Technology Innovation Award (The Only 1 in Province)
• [2016-10] 1^st Prize in Provincial Chinese Physics Olympiad (Ranked 18^th in Province)
• [2015-10] 2^nd Prize in The Chinese 15^th Awarding Program for Future Scientist (Ranked 1^st in Province)
• [2013-03] 1^st Prize in Provincial China Adolescent Robotics Competition (Ranked 1^st in Province)

Hobbies

Soccer ⚽️, Moive 🎬, Singing 🎤, Piano 🎹, Violin 🎻, Snorkeling 🤿, HotToys 🦸‍♂️, FC Online 🎮, PUBG 🐓