Accelerating Training of Large Machine-Learning Models with LiGO Technique

The new LiGO technique speeds up the training of large machine learning models

ChatGPT from OpenAI has many incredible features. For example, it can debug code or write poetry similar to Shakespearean sonnets. ChatGPT’s massive machine-learning engine is responsible for these abilities. Researchers found that these models can have extraordinary abilities when they are large enough.

The training of larger models takes more time and resources. In order to train a model, hundreds of billions examples are shown. The process of gathering so much information is complex. There are also the environmental and monetary costs associated with running powerful computers for weeks or days to train a complex model that could have billions parameters.

It’s estimated that training models of the size of ChatGPT could cost millions of dollars for just one training run. Can we make these methods more efficient so that we get better models in less time, and at a lower cost? Yoon Kim is an assistant professor at MIT’s Department of Electrical Engineering and Computer Science and a CSAIL member. He proposes to leverage smaller language models previously trained.

Source:
https://techxplore.com/news/2023-03-ligo-technique-large-machine-learning.html