Desperate times! Google Gemini AI Looks Quite Remarkable, But It’s Still Behind OpenAI

Google Gemini AI model is only marginally better than the one from OpenAI that’s been out for eight months.

Google’s table shows Gemini Ultra beating GPT-4 on most standard benchmarks. (Google)

(Bloomberg Opinion) -- The days between Thanksgiving and Christmas are normally a dead zone for launching new technology, but these are desperate times for Alphabet Inc.'s Google. The lumbering search giant was caught on the back foot by ChatGPT one year ago, and it's been eager to paint a picture of itself speeding ahead. After reports of a delay, on Wednesday it suddenly announced Gemini AI, a new AI model that could spot sleight-of-hand magic tricks and ace an accountancy exam. A demo video released by Google has wowed social media — but it's a feat in spin. From a technical standpoint, Google is still chasing OpenAI from behind.

Let's start with the technicalities. Here's the table Google released showing how Gemini ranks against OpenAI's top model, GPT-4:

Google's table shows Gemini Ultra beating GPT-4 on most standard benchmarks. These test AI models on things like high school physics, professional law and moral scenarios, and the current AI race is defined almost entirely by such capabilities.

But on most of the benchmarks, Gemini Ultra beat OpenAI's GPT-4 model by only a few percentage points. In other words, Google's top AI model has only made narrow improvements on something that OpenAI completed work on at least a year ago. And Ultra is still under wraps. 

If it's released in early January, as Google has suggested, Gemini Ultra might not stay the top model for very long. In the time it has taken Google to catch up to OpenAI, the nimbler player has had almost a year to work on its next AI model, GPT-5.

Then there's the video demo below that technologists described as “jaw-dropping” on X, the site formerly known as Twitter:

On first viewing, this is impressive stuff. The model's ability to track a ball of paper from under a plastic cup, or to infer that a dot-to-dot picture was a crab before it is even drawn, show glimmers of the reasoning abilities that Google's DeepMind AI lab have cultivated over the years. That's missing from other AI models. But many of the other capabilities on display are not unique and can be replicated by ChatGPT Plus(3), as Wharton professor Ethan Mollick has demonstrated here and here.  

Google also admits that the video is edited. “For the purposes of this demo, latency has been reduced and Gemini outputs have been shortened for brevity,” it states in its YouTube description, raising questions about how much prompting it actually gave the model.  

The video also doesn't specify that this is (probably) Gemini Ultra, the model that's not here yet. Fudging such details points to the broader marketing effort here: Google wants us remember that it's got one of the largest teams of AI researchers in the world and access to more data than anyone else. It wants to remind us, as it did on Wednesday, how vast its deployment network is by bringing less-capable versions of Gemini to Chrome, Android and Pixel phones. 

But being everywhere isn't always the advantage it seems in tech. Early mobile kings Nokia Oyj  and Blackberry Ltd. learned that the hard way in the 2000s when Apple jumped in with the iPhone, a more capable and intuitive product, and ate their lunches. In software, market success comes from having the best-performing systems.

Google's showboating is almost certainly timed to capitalize on all the recent turmoil at OpenAI. When a board coup at the smaller AI startup temporarily ousted CEO Sam Altman and put the company's future in doubt, Google swiftly launched a sales campaign to persuade OpenAI's corporate customers to switch to Google, according to a report in The Wall Street Journal. Now it seems to be riding that wave of uncertainty with the launch of Gemini. 

But impressive demos can only get you so far, and Google has demonstrated uncanny new tech before that didn't go anywhere. (Remember Duplex?) Google's gargantuan bureaucracy and layers of product managers have kept it from shipping products as nimbly as OpenAI till now. As society grapples with AI's transformative effects, that's no bad thing. But take Google's latest show of sprinting ahead with a pinch of salt. It's still coming up from behind.

(1) Google, being Google, has made matters a little complicated by releasing Gemini in three parts: Nano, Pro and Ultra. The Nano model can run on Pixel phones without an internet connection, while Pro is being integrated into Bard and is about as good as GPT-3.5. Remember, that's the one OpenAI released back in November 2022.

(2) Although OpenAI released GPT-4 in March 2023, it had previously spent six months testing it for harmful side effects. That suggests the company likely completed its work on the model some time around September 2022.

(3) ChatGPT Plus is powered by OpenAI's most recent language model, GPT-4.