When large language models (LLMs) hallucinate, developers typically assume the model lacks the required facts. Engineering teams diagnose the error as missing knowledge. The standard response is to increase model size, expand training data, or build complex retrieval architectures.
A new study by researchers at Google Research and Technion demonstrates that the knowledge is often not missing. The model has the information encoded parametrically but fails to surface it during generation.
Their experiments show that frontier models like GPT-5 and Gemini-3 encode 95-98% of tested facts. This indicates that in many cases, recall, rather than encoding, is the primary bottleneck for factual accuracy.
By understanding how to unlock existing knowledge through inference-time computation, engineering teams can build more reliable applications without necessarily relying on larger models or external databases.
Knowledge profiling: measuring what models actually know
To map this gap between storage and retrieval, the researchers propose shifting the evaluation focus from question-level accuracy to fact-level profiling. Instead of simply scoring whether an LLM answers an isolated prompt right or wrong, fact-level profiling tests a single underlying piece of information across multiple conditions, evaluating whether the fact is stored in the model’s parameters at all, whether it can be queried from different directions and phrasings, and what computational effort is required to retrieve it.
This framework distinguishes between whether a fact is parametrically “encoded” and whether it is “known”. A model encodes a fact if it can accurately reproduce it when primed with its original training context. A model knows a fact if it can reliably answer questions about it across varied phrasings and directions.
“Encoding and recall failures are indistinguishable under accuracy metrics, yet they imply different limitations and solutions,” the researchers write. “Encoding failures call for pre-training interventions, such as scaling model size or data coverage. Recall failures suggest post-training interventions that often improve how models utilize what they already encode.”
The paper illustrates this using a sample fact: Oasis played their first gig at the Boardwalk club. Based on how models process this information, the study categorizes knowledge into five distinct profiles:
-
Direct recall: The model encodes the fact and readily accesses it to answer direct questions without extra inference compute.
-
Encoding failure (empty shelves): The model neither encodes nor knows the fact. It cannot complete a Wikipedia-style sentence about Oasis’s early days, nor can it answer questions about the event. This signals a need for more pre-training data or greater model capacity.
-
Recall failure (lost keys): The model has the fact encoded but cannot access it. It can seamlessly complete the original training text about Oasis, but fails to answer “Where did Oasis play their first show?” even when given time to think.
-
Recall with thinking: The fact is encoded, but inaccessible to direct generation. It is only successfully recalled when the model uses inference-time computation, such as Chain-of-Thought, to bridge the gap. The researchers refer to this mechanism as recall facilitation. The model might initially fail to answer the direct question. By generating intermediate thoughts about the band’s early history in Manchester, it structurally primes itself to locate and recall the locked answer.
-
Inference without encoding: The model never explicitly encoded the Oasis fact. Instead, it successfully answers the question by making an educated guess or reasoning across other encoded facts it does know. It might deduce the answer by chaining together separate data points, such as “Oasis formed in Manchester,” “the Boardwalk was a famous 90s music club there,” and “the Boardwalk hosted early gigs by emerging bands.”
Scaling illusions, long-tails, and tip-of-the-tongue recoveries
The researchers evaluated 13 LLMs on over 4 million responses. They used WikiProfile, a benchmark containing 2,150 facts extracted from Wikipedia, testing each fact across formats ranging from exact context completion to multiple-choice verification.
For frontier models like GPT-5 and Gemini-3, encoding is nearing saturation. These models successfully encode 95-98% of the tested facts. However, they still fail to directly recall 26-34% of those encoded facts without thinking.
Inference-time thinking acts as a vital recovery mechanism. Providing models with extra computational effort successfully retrieves 40-65% of the encoded facts that models initially fail to directly recall. The researchers compare this to the human tip-of-the-tongue state, where deliberate effort, such as mentally retracing context, eventually helps remember the information.
Scaling up model size does not automatically resolve this gap. In fact, companies often mistakenly try to solve recall failures by fine-tuning larger internal models—an expensive architectural misstep.
“When facts come out wrong, the go-to move is to scale, meaning train a larger model or add more data,” Nitay Calderon, Research Scientist at Google, told VentureBeat. “Both are expensive, and if the facts are already encoded, neither helps.”
For example, the researchers found that scaling the Gemma3 model from 1 billion to 27 billion parameters largely filled the “empty shelves” by decreasing encoding failures from 85% to 23%. But at the same time, the share of recall failures increased, peaking at 40% without thinking.
This suggests that scaling mainly solves the storage problem rather than the access problem. As the model memorizes vastly more facts, a larger pool of knowledge becomes trapped in an “encoded but inaccessible” state. The bulk of model errors shifts from missing data to failed recall.
“Our findings suggest that recall is tightly coupled to the conditions under which facts were learned, degrading when queries diverge from training-time patterns,” the researchers write. How a user asks a question directly dictates whether the model can unlock the stored answer.
For example, the experiments showed that rare facts are encoded at rates similar to popular facts. Yet they found a large recall gap between long-tail and highly popular facts that exceeds 25% for frontier models.
Similarly, models struggle to generate answers to reverse questions (i.e., asking for the subject instead of the object). For example, a model might easily answer that Oasis played their first gig at the Boardwalk club, but fail to answer who played their first gig at that same club. At the same time, the same models show that they know the correct answer when given the same question in multiple-choice format.
“Whereas these failures are often interpreted as limitations of memorization or bidirectional encoding, our results suggest a different picture: rare facts are often encoded but inaccessible, and reverse facts can be recognized even when they cannot be generated,” the researchers write. “This reframes both phenomena as recall failures rather than ‘missing knowledge.'”
The ROI of thinking and tips for developers
The high encoding rates of frontier models require a shift in how developers approach factuality and pipeline architecture.
Don’t treat every factual failure as a retrieval problem: The default enterprise reaction to hallucinations is often to deploy Retrieval-Augmented Generation (RAG), scale up vector databases, or ingest more domain documents. While RAG is the right call for fresh or internal data, using it as a blanket fix for hallucinations adds latency and costs to facts the model already has locked in its parametric memory.
“A lot of what teams solve with RAG are facts the model can already answer from memory, so you’re paying extra latency and per-call cost for nothing,” Calderon said. “If a fact is truly missing, RAG can be the right fix. But if the fact is encoded and the model just can’t recall it, RAG and scaling the model only add cost on top of the real problem.”
Use inference-time reasoning selectively: Thinking recovered 40–65% of encoded facts that models failed to directly recall. However, because only 10-20% of facts actually require thinking, turning it on globally wastes your compute budget. The challenge is dynamically routing queries, as models lack the self-awareness to reliably diagnose when they are about to fail.
“To use the compute well, the model has to sense ahead of time that a plain answer is about to fail, so it can escalate before answering,” Calderon said. “That self-awareness is its own skill, and today’s models aren’t reliably good at it.” This metacognitive bottleneck is why Google researchers are developing frameworks like “faithful uncertainty” to allow models to accurately gauge their own confidence and trigger deeper reasoning rather than hallucinating.
Deploy generate-then-verify pipelines: Because models are better at recognizing facts (verification) than generating them from scratch, developers can build architectural loops where a model generates a response and is then prompted to explicitly reflect on and verify its own claims. “Since recognizing a correct answer is easier than generating one, a verify pass over the model’s own output could catch mistakes that plain generation misses and add some factual improvement on top,” Calderon said.
Test semantic access, not just benchmark accuracy: Standard accuracy metrics mask underlying model capabilities. Evaluation sets should probe the same underlying fact across different phrasings, contexts, and directions to truly understand what a model knows versus what it can reliably access.
Leverage query reformulation and retries: Because recall is highly context-dependent, query framing dictates success. Changing the structure of a prompt, generating relevant intermediate context, or prompting the model to generate a reasoning chain before answering are legitimate reliability mechanisms that surface information direct prompts miss.
Limitations and practical takeaways
The WikiProfile benchmark relies on encyclopedic Wikipedia facts. These findings might not perfectly generalize to proprietary or highly specialized enterprise domains. A model’s ability to store and recall a niche internal company metric may behave differently than its handling of public encyclopedic data.
Fully profiling a frontier model on the WikiProfile suite costs approximately $500. Developers can significantly reduce this cost by omitting multiple-choice variants or using fewer response samples per question.
Teams can access the WikiProfile benchmark on Hugging Face to evaluate their own systems. Because the benchmark includes the exact prompts used to build it, enterprise data engineering teams can recreate the pipeline on their own internal corpora to diagnose whether their bespoke agents are suffering from missing data or missing keys. However, teams should manage their expectations when moving away from encyclopedic data.
“The pipeline is built to be applied on a new corpus, and we provide all the prompts we used,” Calderon said. “The one thing to expect: on Wikipedia it was mostly a recall problem. Domain-specific facts may genuinely not be encoded in the model.”
Ultimately, this shift toward knowledge usage levels the playing field for enterprise AI stacks. “For companies that don’t build models from scratch, this is good news,” Calderon said. “Pre-training is hugely expensive and out of reach for most, but the levers that matter now are not: post-training can help with little data and few steps, and inference-time tools like thinking, verify steps, and retrieval are already what most teams use.”
This story was updated to include remarks from Google.
Credit: Source link
