I have spent the past several years building AI into products where a wrong answer has a price attached: an AI-powered payroll compliance platform for the Australian market, where an error lands in someone’s bank account, and a college admissions product, where a poor match can shape a student’s next four years. The question that decided the architecture was the same in both. How far should the language model be allowed to go before something deterministic takes over?
One in five U.S. payrolls contains an error, at an average cost of $291 each before litigation enters the picture. The EU AI Act puts AI used for admissions decisions and employment management in its high-risk categories.
Let the Rules Own the Numbers
Deterministic rules own anything with a dollar figure or a legal consequence attached, and the model interprets and explains around those rules, and nothing more.
On the payroll platform, our goal was a clear boundary between the payroll rules and the AI. Payroll experts first define pay rules in a structured format. The system then processes those rules and generates every possible scenario that can result from them. That complete set of scenarios becomes the model’s source of truth. The model cannot create new rules, change existing ones, or decide which sources to use. It can only retrieve the relevant rule set, look up rate tables, and cite the scenario it used. Rule creation, scenario generation, and source selection stay outside the AI layer entirely.
The admissions product follows the same logic: location requirements, budget limits, institution status, and eligibility thresholds are applied as rules before the model can influence a ranking. If a student says they must attend college in California, an out-of-state school cannot override that because the model thought it was otherwise a strong fit. The model handles softer preferences and explanations, and deterministic logic is the guardrail wherever approximately right is not acceptable.
Retrieval Grounds the Model, Up to a Point
Retrieval-augmented generation keeps answers tied to the organization’s own policies rather than whatever the model absorbed in training. It is also oversold. When Stanford’s RegLab tested commercial legal research tools built on RAG, they still hallucinated between 17% and 33% of the time. Retrieval narrows what the model can say, and the model can still get it wrong.
I learned that directly. On the payroll platform, the system once found the correct employment award for a case, and the model still interpreted the overtime rules incorrectly. The right source had been retrieved, yet the answer was wrong, so after that we became much stricter about what the model could decide on its own. Important payroll rules moved into explicit business logic, every answer had to be supported by source material, and we added tests built around the case that had failed. Access to the right information only helps if you also control how the model uses it.
Make Every Answer Traceable
Before a response is generated, the system records exactly which rulebook and version were used, along with the model and other request details. Saying that an AI response was based on company policy is difficult to verify. Saying that it was generated using version 7 of a specific agreement, which has been retained and can be compared with other versions, is much stronger evidence. We also trace every model and tool call: what the user asked, what was retrieved, which tools ran, what the model returned, and any errors along the way. When a support issue comes in, we can see exactly where an incorrect answer came from. The 2024 Air Canada ruling, where a tribunal made the airline honor a discount its chatbot had invented, shows what the alternative costs.
Put Humans Where the Consequences Are
NIST’s Generative AI Profile treats the line between what a system does alone and what requires a person as a governance decision. We route a case to a human when the system is uncertain, when information conflicts, or when the decision could meaningfully affect the user. In admissions, an unusual student profile, or a recommendation that does not clearly satisfy a student’s hard requirements, gets flagged for counselor review. That step has caught recommendations that looked reasonable on the surface but missed a constraint in the student’s profile, and each one fed back into the system. Neither product finalizes anything on its own. Counselor input is required before a recommendation is relied on, and every pay stub the payroll system generates is flagged for approval before it goes out. Human review is not meant for every request. The goal is to automate routine cases confidently and involve a person when the system has a good reason to be uncertain.
Deciding What the Model Can Own Today
There is a practical test I apply to any workflow. If a wrong output can be caught and undone before it reaches a person, the model can own that step. If the answer can be checked against a rule, the rule owns it and the model explains it. If a wrong output would reach a paycheck or an application decision before anyone sees it, that step is not ready.
Language models are the best interpreters we have ever put in front of a rulebook. In domains that punish mistakes, interpretation is where they should stop.
Credit: Source link
