Google’s Gemini AI launch marred by questions over capabilities

Google unveiled its much-anticipated artificial intelligence system Gemini on Wednesday, touting benchmarks suggesting it could compete with OpenAI’s industry-leading GPT-4 model in reasoning abilities. But the launch has quickly been overshadowed by accusations that the tech giant overstated Gemini’s capabilities.

In a tightly choreographed video demonstration, Google showed Gemini interacting with visual data through a camera mounted above a desk, fielding questions and reasoning through problems as a human assistant manipulated objects. The slick presentation implied Gemini could serve as an intelligent digital assistant capable of sophisticated conversation and assistance with daily tasks. 

Yet tech experts analyzing the underlying technology behind the scenes say Gemini may fail to live up to Google’s lofty aspirations. The company is rolling out Gemini in three versions — Gemini Pro, Gemini Light and Gemini Ultra. But early reviews of the mid-range Pro version made public on Wednesday indicate it still struggles with tasks that should be routine for a state-of-the-art AI system.

“I’m extremely disappointed with Gemini Pro on Bard,” said Victor de Lucca, an early tester of the Bard update, in an X.com post showing that the AI system was not able to correctly list the 2023 Oscar winners. “It still gives very, very bad results to questions that shouldn’t be hard anymore with RAG.”

Others pointed out discrepancies between the capabilities Google claimed in its benchmark testing and what appears possible with the publicly available Pro version. 

“Google Gemini Ultra [is] only 4% better…using different prompts versus GPT-4-0613?” asked developer Nick Dobos in a widely shared post on X.com, suggesting the comparison was misleading. 

The slick Gemini video also came under fire after a Google spokesperson confirmed to Bloomberg that the footage was pre-recorded and narrated after the fact, rather than representing a live conversational demo.

The controversy illustrates the challenges Google faces in marketing AI systems to consumers. While techies eagerly dissect benchmark numbers and academic papers, the general public responds more to inspirational videos promising a revolutionary future.

This disconnect has tripped up big tech companies before, perhaps most infamously in 2016 when Microsoft’s Tay chatbot was yanked offline after learning hate speech from Twitter users. This is also the second time Google Bard has been accused by the tech community of falling short of the company’s promise. In September, VentureBeat reported that Google Bard was still failing to deliver on its promise — even after major updates.

Google is, of course, aiming to recover quickly, promising to make Gemini more widely available to developers and researchers who can fully put it through its paces. But the rocky start shows the tech giant still has work to do if it wants its AI assistant to measure up to the hype.