Right now, we’re at a point where AI models are starting to show signs of human intelligence. Companies pour thousands of hours and billions of dollars into training their models, but they soon realized that pumping their models with more data won’t produce the results that they want. AI companies are working to pass a certain limitation with training models.
No one can fathom the sheer amount of data that’s been fed into AI models up until this point. Companies have scraped data from numerous sources such as news sites, YouTube, social media platforms, and many more. These companies scraped much of this data without our knowledge, and this continues today.
The tactic to build better AI models has been to train them on ever-increasing amounts of data, so companies have been finding new ways to gather more. For example, companies like OpenAI, Meta, and Google have been signing license deals with several media companies to use their data.
AI companies are trying to get past the limitation with training large models
Billions of people throughout the world create online content each day. So, it’s inconceivable that all the world’s data could be scraped to train AI models. However, according to a report, much of the easily accessible data in the world might have already been used to train models.
This is a pretty big issue because as models get bigger, they’ll require more training data. If companies are already running into data limitations this early on, then there’s no chance that they’ll achieve AGI (Artificial General Intelligence).
OpenAI and Safe Superintelligence co-founder Ilya Sutskever said that the stage of training AI models has plateaued. Larger and more data-hungry AI models just aren’t producing the kinds of results that companies are looking for. So, stuffing these models with more and more data just isn’t the thing to do.
It’s not just the data
There are other factors that lead to this limitation. Firstly, AI doesn’t only require data, it also requires power. Running massive data centers with thousands of chips and servers can really run up the light bill. This is one reason why Google, OpenAI, and Meta are looking into nuclear power. Well, if companies are looking to train larger models, they’re going to have to take into account how much energy it will cost.
Next, as stated by Reuters, the hardware powering these models is bound to fail from being run constantly. Well, as models increase in size, the strain on the chips and other components will only increase.
As such, companies like OpenAI, Google, and others have been working on other ways of improving their models without dumping data into them. For example, Noam Brown, an OpenAI researcher said, “having a bot think for just 20 seconds in a hand of poker got the same boosting performance as scaling up the model by 100,000x and training it for 100,000 times longer.”
So, scaling up might not be the way to AGI. Companies are going to be looking for inventive ways of achieving smarter AI. Right now, we don’t know what other companies are going to do, but we’re sure that the AI landscape won’t be the same.