Cloud Hosting

Recent progress in AI has been startling. Barely a weeks gone by without a new algorithm, application, or implication making headlines. But OpenAI, the source of much of the hype, only recently completed their flagship algorithm, GPT-4, and according to OpenAI CEO Sam Altman, its successor, GPT-5, hasnt begun training yet.

Its possible the tempo will slow down in coming months, but dont bet on it. A new AI model as capable as GPT-4, or more so, may drop sooner than later.

This week, in an interview withWill Knight, Google DeepMind CEO Demis Hassabis said their next big model, Gemini, is currently in development, a process that will take a number of months. Hassabis said Gemini will be a mashup drawing on AIs greatest hits, most notably DeepMinds AlphaGo, which employed reinforcement learning to topple a champion at Go in 2016, years before experts expected the feat.

At a high level you can think of Gemini as combining some of the strengths of AlphaGo-type systems with the amazing language capabilities of the large models, Hassabis told Wired. We also have some new innovations that are going to be pretty interesting. All told, the new algorithm should be better at planning and problem-solving, he said.

Many recent gains in AI have been thanks to ever-bigger algorithms consuming more and more data. As engineers increased the number of internal connectionsor parametersand began to train them on internet-scale data sets, model quality and capability increased like clockwork. As long as a team had the cash to buy chips and access to data, progress was nearly automatic because the structure of the algorithms, called transformers, didnt have to change much.

Then in April, Altman said the age of big AI models was over. Training costs and computing power had skyrocketed, while gains from scaling had leveled off. Well make them better in other ways, he said, but didnt elaborate on what those other ways would be.

GPT-4, and now Gemini, offer clues.

Last month, at Googles I/O developer conference, CEO Sundar Pichai announced that work on Gemini was underway. He said the company was building it from the ground up to be multimodalthat is, trained on and able to fuse multiple types of data, like images and textand designed for API integrations (think plugins). Now add in reinforcement learning and perhaps, as Knight speculates, other DeepMind specialties in robotics and neuroscience, and the next step in AI is beginning to look a bit like a high-tech quilt.

But Gemini wont be the first multimodal algorithm. Nor will it be the first to use reinforcement learning or support plugins. OpenAI has integrated all of these into GPT-4 with impressive effect.

If Gemini goes that far, and no further, it may match GPT-4. Whats interesting is whos working on the algorithm. Earlier this year, DeepMind joined forces with Google Brain. The latter invented the first transformers in 2017; the former designed AlphaGo and its successors. Mixing DeepMinds reinforcement learning expertise into large language models may yield new abilities.

In addition, Gemini may set a high-water mark in AI without a leap in size.

GPT-4 is believed to be around a trillion parameters, and according to recent rumors, it might be a mixture-of-experts model made up of eight smaller models, each a fine-tuned specialist roughly the size of GPT-3. Neither the size nor architecture has been confirmed by OpenAI, who, for the first time, did not release specs on its latest model.

Similarly, DeepMind has shown interest in making smaller models that punch above their weight class (Chinchilla), and Google has experimented with mixture-of-experts (GLaM).

Gemini may be a bit bigger or smaller than GPT-4, but likely not by much.

Still, we may never learn exactly what makes Gemini tick, as increasingly competitive companies keep the details of their models under wraps. To that end, testing advanced models for ability and controllability as theyre built will become more important, work that Hassabis suggested is also critical for safety. He also said Google might open models like Gemini to outside researchers for evaluation.

I would love to see academia have early access to these frontier models, he said.

Whether Gemini matches or exceeds GPT-4 remains to be seen. As architectures become more complicated, gains may be less automatic. Still, it seems a fusion of data and approachestext with images and other inputs, large language models with reinforcement learning models, the patching together of smaller models into a larger wholemay be what Altman had in mind when he said wed make AI better in ways other than raw size.

Hassabis was vague on an exact timeline. If he meant training wouldnt be complete for a number of months, it could be a while before Gemini launches. A trained model is no longer the end point. OpenAI spent months rigorously testing and fine-tuning GPT-4 in the raw before its ultimate release. Google may be even more cautious.

But Google DeepMind is under pressure to deliver a product that sets the bar in AI, so it wouldnt be surprising to see Gemini later this year or early next. If thats the case, and if Gemini lives up to its billingboth big question marksGoogle could, at least for the moment, reclaim the spotlight from OpenAI.

Image Credit: Hossein Nasr / Unsplash

See more here:
Here's Why Google DeepMind's Gemini Algorithm Could Be Next-Level AI - Singularity Hub