News · October 11, 2021
Microsoft and Nvidia train a 530-billion-parameter language model
Microsoft and Nvidia have trained one of the largest language models yet, using a combination of their open-source training tools. The race for scale continues, and so do concerns about bias.
What happened
- Microsoft and Nvidia announced on October 11, 2021 the Megatron-Turing NLG model with 530 billion parameters, which they said has about three times as many parameters as the largest existing model of its type, according to a Microsoft Research blog post.
- According to the post, training ran on Nvidia's Selene supercomputer of 560 DGX A100 servers and combined Nvidia's Megatron-LM with Microsoft's DeepSpeed library.
- The authors wrote that the model picks up stereotypes and biases from its training data and that any production use would need safeguards in line with Microsoft's responsible AI principles.
- Synced reported the announcement the same day, describing the model as the result of a research collaboration between the two companies.
Why it matters
Few companies will train models of this size, and the post does not describe a product. The more useful parts are the open-source training tools behind it, which are already used to train smaller models more efficiently.
For most businesses the practical question is not which model is largest but which one is good enough, affordable and controllable for a specific task. Bigger models still repeat biases from their data, so testing for harmful outputs remains part of any rollout.
Sources
- Microsoft Research Blog: Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, the World's Largest and Most Powerful Generative Language Model
- Synced: 530 Billion Parameters! Microsoft and NVIDIA Trained the Largest Generative Language Model
This post is CMM’s own summary and commentary on publicly reported news. Facts are attributed to the sources linked above; please read them for full details. CMM is not affiliated with these publishers or with the companies mentioned, and trademarks belong to their owners. Prepared with AI assistance. Corrections or requests: [email protected]. See our editorial policy.
Want help with this?
Tell us what you need, and we will suggest the right team model on a short call.
Talk to us