Renrui Zhang Ph.D. candidate Multimedia Laboratory (MMLab), The Chinese University of Hong Kong Office: SHB 304, CUHK, Hong Kong S.A.R., China Email: 1700012927@pku.edu.cn, zhangrenrui@pjlab.org.cn [Google Scholar], [GitHub]

📌 My research interests are multi-modal learning, large language/vision models, and 3D computer vision

✉️ I anticipate graduating in 2025 and am open to both academic and industrial research positions in North America and Asia. If you are interested, please feel free to contact me.

✉️ I'm also looking for self-motivated undergraduate and graduate students for academic cooperation.

Education

• [2017-2021] 🎉 I received my B.E. degree from Peking University, awarded Outstanding Graduate (Top 5%).

• [2020-2021] I worked as a visiting student in University of Pennsylvania, supervised by Prof. Jianbo Shi.

• [2021-Now] 💪 I'm pursuing my Ph.D. in MMLab, CUHK, supervised by Prof. Hongsheng Li and Prof. Xiaogang Wang.

• [2021-2024] I worked as a research intern at Shanghai AI Lab, supervised by Dr. Peng Gao.

• [2024-Now] I'm working as a research intern at ByteDance, Seattle, supervised by Dr. Chunyuan Li.

News

• [2024-07] Four papers accepted by ECCV 2024

• [2024-05] Three papers accepted by ICML 2024

• [2024-03] Seven papers accepted by CVPR 2024, two of them gain Highlight 🎉

• [2024-02] One paper accepted by ICRA 2024

• [2024-01] Four papers accepted by ICLR 2024

• [2023-12] Four papers accepted by AAAI 2024

• [2023-09] One paper accepted by NeurIPS 2023

• [2023-08] Two papers accepted by IJCV 2023

• [2023-07] Five papers accepted by ICCV 2023

• [2023-04] One paper accepted by IJCAI 2023

• [2023-02] Six papers accepted by CVPR 2023

• [2022-11] Two papers accepted by AAAI 2023, one gains the Best Student Paper 🎉

• [2022-09] One paper accepted by NeurIPS 2022

• [2022-07] Three papers accepted by ECCV 2022

• [2022-03] One paper accepted by CVPR 2022

• [2021-09] One paper accepted by NeurIPS 2021

Selected Preprints and Projects

* Equal contribution, # Corresponding author

• LLaVA-OneVision: Easy Visual Task Transfer
  B Li, Y Zhang, D Guo, R Zhang, F Li, H, Zhang, K Zhang, Y Li, Z Liu, C Li
  🔥 The new generation of LLaVA models for multi-modal learning

  🔥 Code [1.4k+ Stars 🌟]
  

• MAVIS: Mathematical Visual Instruction Tuning
  R Zhang*, X Wei*, D Jiang, Y Zhang, Z Guo, C Tong, J Liu, A Zhou, B Wei, S Zhang, P Gao, H Li
  📐 The first public large-scale multi-modal mathematical dataset for tuning large models

  🔥 Code [50+ Stars 🌟]
  

• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
  F Li*, R Zhang*, H Zhang*, Y Zhang, B Li, W Li, Z Ma, C Li
  💪 The official latest and powerful LLaVA version excelling in diverse multi-modal scenarios

  🔥 Code [1.4k+ Stars 🌟]
  

• MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines
  D Jiang*, R Zhang*#, Z Guo, Y Wu, J Lei, P Qiu, P Lu, Z Chen, G Song, Y Liu, P Gao, C Li, H Li
  🔥 The first multi-modal search engine pipeline and benchmark, surpassing Perplexity Pro

  🔥 Code [350k+ Stars 🌟]
  

• Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following
  Z Guo*, R Zhang*#, X Zhu, Y Tang, X Ma, J Han, K Chen, P Gao, X Li#, H Li, P Heng
  🧠 A 3D multi-modal model for general 3D learning, Point-Bind, and the first 3D large language model, Point-LLM

  🔥 Code [300+ Stars 🌟]
  

• A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
  C Fu*, R Zhang*, H Lin*, Z Wang*, T Gao, Y Luo, Y Huang, Z Zhang, L Qiu, ...
  🚀 The first technical report for Gemini vs GPT-4V. A total of 128 pages. Completed within one week of the Gemini API opening

  🔥 Code [6.4k+ Stars 🌟]
  

• X-Accessory Code Base
  🧠 An open-source toolkit for the deployment of Large Language Models (LLMs) and multi-modal LLMs

  🔥 Code [2.0k+ Stars 🌟]
  

Selected Publications

* Equal contribution, # Corresponding author

• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
  R Zhang*, D Jiang*, Y Zhang*, H Lin, Z Guo, P Qiu, A Zhou, P Lu, KW Chang, P Gao, H Li
  📐 The first benchmark to evaluate the real capabilities of MLLMs for visual mathematical reasoning

  ECCV 2024, 🔥 Code [100k+ Stars 🌟]
  

• LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
  R Zhang*, J Han*, A Zhou, X Hu, S Yan, P Lu, H Li, P Gao#, Y Qiao
  🚀 Fine-tune LLaMA for instruction-following capacity and multi-modal reasoning by PEFT methods

  ICLR 2024, 🔥 Code [5.2k+ Stars 🌟]
  

• Personalize Segment Anything Model with One Shot
  R Zhang, Z Jiang*, Z Guo*, S Yan, J Pan, H Dong, P Gao, H Li#
  🚀 Efficiently personalize SAM for segmenting specific objects and improving DreamBooth

  ICLR 2024, 🔥 Code [1.3k+ Stars 🌟]
  

• MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection
  R Zhang, H Qiu, T Wang, Z Guo, Z Cui, Y Qiao, P Gao, H Li#
  🚀 The first DETR-based network for camera-based 3D object detection

  ICCV 2023, Code [250+ Stars 🌟]
  

• Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysis
  R Zhang, L Wang, Y Wang, P Gao, H Li, J Shi#
  🚀 The first 0-parameter and 0-training method for 3D analysis

  CVPR 2023, Code [350+ Stars 🌟]
  

• Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners
  R Zhang*, X Hu*, B Li, S Huang, H Deng, H Li, Y Qiao, P Gao#
  🚀 Collaborate large models (GPT, CLIP, DINO, DALL-E) for image understanding

  CVPR 2023, Code [300+ Stars 🌟]
  

• Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoders
  R Zhang, L Wang, Y Qiao, P Gao, H Li#
  🚀 Transfer pre-trained 2D knowledge into 3D space for point cloud learning

  CVPR 2023, Code [150+ Stars 🌟]
  

• Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptation
  Y Gan, X Ma, Y Lou, Y Bai, R Zhang, N Shi, L Luo#
  🚀 Visual prompting tuning for superior continual test-time adaptation

  AAAI 2023 Best Student Paper 💥

• Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training
  R Zhang, Z Guo, P Gao, R Fang, B Zhao, D Wang, Y Qiao, H Li#
  🚀 Hierarchical 3D transformer for MAE-style self-supervised point cloud pre-training

  NeurIPS 2022, Code [150+ Stars 🌟]
  

• Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
  R Zhang*, W Zhang*, R Fang, P Gao, K Li, J Dai, Y Qiao, H Li#
  🚀 Propose a training-free cache model to adapt CLIP for downstream Visual recognition

  ECCV 2022, Code [400+ Stars 🌟]
  

• PointCLIP: Point Cloud Understanding by CLIP
  R Zhang*, Z Guo*, W Zhang, K Li, X Miao, B Cui, Y Qiao, P Gao, H Li#
  🚀 The first work to extend CLIP for zero-shot 3D understanding

  CVPR 2022, Code [250+ Stars 🌟]
  

Selected Awards

• [2021-06] Outstanding Graduate, Peking University (Top 5%)

• [2020-09] Academic Excellent Scholarship (Ranked 1st/73)

• [2020-09] Merit Student PaceSetter, Peking University (Ranked 1st/73)

• [2019-09] Academic Excellent Scholarship (Ranked 4th/73)

• [2019-09] Merit Student, Peking University (Ranked 4th/73)

• [2016-07] China Youth Technology Innovation Award (The Only 1 in Province)

• [2016-10] 1st Prize in Provincial Chinese Physics Olympiad (Ranked 18th in Province)

• [2015-10] 2nd Prize in The Chinese 15th Awarding Program for Future Scientist (Ranked 1st in Province)

• [2013-03] 1st Prize in Provincial China Adolescent Robotics Competition (Ranked 1st in Province)

Hobbies

Soccer ⚽️, Moive 🎬, Singing 🎤, Piano 🎹, Violin 🎻, Games 🎮, Snorkeling 🤿, HotToys 🦸‍♂️