CV

Education

• Bachelor of Computer Science and Technology, University of Science and Technology of China (USTC), Sept. 2018 - June 2022
• Ph.D student at Chinese Academy of Sciences, Sept. 2022 - present

Work experience

• Aug. 2021 - Jan. 2023 & Aug. 2023 - Present: Research intern
  • GenAI group, Microsoft Research Asia
  • Supervisor: Dr. Furu Wei and Shuming Ma
• Feb. 2021 - July 2021: Teaching Assistant for Mathematical analysis B2, undergraduate course
  • USTC
  • Supervisor: Professor Yelong Zheng
• Sept. 2020 - Jan. 2021: Teaching Assistant for Mathematical analysis B1, undergraduate course
  • USTC
  • Supervisor: Professor Yelong Zheng

Awards

• Outstanding Teaching Assistant at USTC, 2021
• Silver Award of Outstanding Student Scholarship at USTC, 2020
• Huawei Scholarship, 2020
• Silver Award of Outstanding Student Scholarship at USTC, 2019

Services

• Journal Reviewer: IEEE Transactions on Affective Computing.
• Conference Reviewer: ICLR, ECCV, EMNLP, ACL Rolling Review.

Talks

• (08/2024) The Era of 1-bit LLM at MSR Asia Intern Tech Fest.
• (06/2024) The Era of 1-bit LLM at Cohere for AI, and SIGMA Lab, Tsinghua University.
• (04/2024) BitNet b1.58 at MiraclePlus.
• (08/2023) Magneto at Microsoft Research Asia.

Preprints & Publications

• BitNet a4.8: 4-bit Activations for 1-bit LLMs. Hongyu Wang*, Shuming Ma*, Furu Wei
• 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs. Jinheng Wang, Hansong Zhou, Ting Song, Shaoguang Mao, Shuming Ma, Hongyu Wang, Yan Xia, Furu Wei
• Q-Sparse: All Large Language Models can be Fully Sparsely-Activated. Hongyu Wang*, Shuming Ma*, Ruiping Wang, Furu Wei
• DeepNet: Scaling Transformers to 1,000 Layers. Hongyu Wang*, Shuming Ma*, Li Dong, Shaohan Huang, Dongdong Zhang, Furu Wei, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI 2024)
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. Shuming Ma*, Hongyu Wang*, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei
• BitNet: Scaling 1-bit Transformers for Large Language Models. Hongyu Wang*, Shuming Ma*, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
• Magneto: A Foundation Transformer. Hongyu Wang*, Shuming Ma*, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, Furu Wei. International Conference on Machine Learning (ICML), 2023.
• TorchScale: Transformers at Scale. Shuming Ma*, Hongyu Wang*, Shaohan Huang, Wenhui Wang, Zewen Chi, Li Dong, Alon Benhaim, Barun Patra, Vishrav Chaudhary, Xia Song, Furu Wei