Hallucinations, Bias, and the Deterministic Trap: Why AI Confidently Lies to You

Published on 18 Aug 2026 by Shri Krishnan

An essay from 9horsemen's founder. For practical help running a business, see the blog.

Part 4 of 4 in the AI for Founders series. See all parts.

Listen to this episode of Why Your Software Sucks on Spotify.

AI hallucinates because it predicts the most likely next words, not the true ones, and it carries bias because it learns from the past. Neither is a bug the next version will fix. Here’s why, with real cases, and how to build around it.

What does an AI hallucination look like?

It looks like a confident, fluent answer with made-up facts in it, delivered in exactly the same tone as a true one.

Picture this scenario.

You are interviewing a candidate for a senior operations role at your company. They walk in wearing a tailored suit. Their posture is perfect, their eye contact is unwavering, and their handshake is firm.

You hand them a printout of your company’s logistics ledger and ask: “Can you review these three international shipping contracts and confirm our tax liability under the revised customs tariff?”

The candidate doesn’t blink. They immediately speak with total fluency:

“Certainly. Under Section 14-B of the Maritime Harmonized Code, your liability is capped at 4.2% because your freight forwarder operates out of Rotterdam. This was reaffirmed in the landmark corporate precedent Vance v. Global Logistics Corp (2019).”

They sound articulate. They cite specific sections. They name a court case with zero hesitation.

Now imagine you open your legal database to double-check their citation. You discover:

Section 14-B of that code does not exist.
The Rotterdam exemption is a fabrication.
The court case Vance v. Global Logistics Corp was completely invented. Neither the judge, the lawyers, nor the shipping company ever existed in the history of the world.

When you confront the candidate with this evidence, they do not apologise, blush, or look embarrassed. With the exact same calm smile, they reply:

“You are completely correct. My mistake! Under Section 18-C of the Northern Freight Treaty…”—and proceed to invent an entirely new lie just as smoothly as the first.

Would you hand this person the keys to your company’s bank account? Would you let them draft legally binding customer agreements or calculate your employee payroll?

Of course not. You would escort them out of the building.

Yet every single day, thousands of business founders, executives, and department heads hook Large Language Models directly to their core business operations and let that exact same charismatic fabricator speak directly to their clients, their databases, and their ledgers.

Welcome to the twin landmines of modern artificial intelligence: Hallucination and Bias.

If you do not understand why these failures occur—and why Silicon Valley cannot simply “code them away” with the next software update—you are building your business on digital quicksand.

Why does AI hallucinate?

Because a language model predicts the most likely next word, not the true one, so when it doesn’t know, it produces something that sounds right.

The biggest marketing lie sold to non-technical entrepreneurs is the idea that AI “hallucination” is an engineering defect that will disappear in the next version of the software.

Silicon Valley venture capitalists love to promise: “GPT-4 hallucinates less than GPT-3, and GPT-5 will hallucinate even less, until eventually the error rate drops to zero.”

This statement demonstrates a deliberate ignorance of how generative artificial intelligence works.

+-------------------------------------------------------------------------+
|                  THE CORE MISUNDERSTANDING OF LLMs                      |
|                                                                         |
|  WHAT FOUNDERS THINK IT IS:                                             |
|  [ User Question ] ===> [ Search Internal Fact Database ] ===> [ Truth ]|
|                                                                         |
|  WHAT IT ACTUALLY IS:                                                   |
|  [ User Context ]  ===> [ Calculate Next Most Likely Word ] ===> [ Text]|
+-------------------------------------------------------------------------+

As we unpacked in Series 2, a Large Language Model is not a database of verified facts. It is not an electronic encyclopedia. It does not possess a model of objective reality.

An LLM is a probabilistic next-token predictor.

When you ask a model: “What is the capital of France?”, it does not look out a window, consult an atlas, or check a government registry. It calculates that in its vast corpus of training data, the word that statistically appears most frequently after the sequence “The capital of France is” is the word “Paris.”

It gets the answer right not because it knows what France is, but because the statistical relationship between those words is nearly 100% consistent across the internet.

Now, look at what happens when you ask that same model a question with lower statistical density—a question about a specific, obscure edge case in your business:

“What is the warranty policy for our commercial Model-4 water pumps installed in humid environments after 18 months?”

If that exact scenario was not repeated fifty thousand times in the training data, the model does not say: “I do not know.”

Remember: the model is mathematically optimised to produce a fluent answer, not a truthful one.

It evaluates the mathematical geometry of your prompt:

It knows what warranties sound like.
It knows the typical sentence structures of industrial equipment manuals.
It knows the legal terminology commonly found in commercial manufacturing disclosures.

So, it generates words that match the expected shape, rhythm, and tone of a warranty clause:

“The Model-4 pump carries an extended 24-month environmental warranty covering corrosion, provided routine descaling is documented.”

It sounds immaculate. It reads like it was drafted by a corporate attorney. But every word was assembled based on mathematical probability rather than empirical truth.

The Reality: Creativity and Hallucination Are the Same Circuitry

Here is the uncomfortable reality that founders must accept:

Hallucination is the exact same computational mechanism as creativity.

+-------------------------------------------------------------------------+
|                    THE DUAL NATURE OF GENERATIVE MATH                   |
|                                                                         |
|   SAME ENGINE: Next-Token Probability Matrix                            |
|                                                                         |
|   When it invents a clever marketing slogan:    ===> "GENIUS CREATIVITY"|
|   When it invents a nonexistent warranty rule:  ===> "HALLUCINATION"    |
+-------------------------------------------------------------------------+

When you ask an AI model to write a fictional short story about an astronaut stranded on Mars, you want it to invent characters, plot twists, and dialogue that never existed in the real world. You praise that output as “creative.”

When you ask that same model to summarise a 40-page supplier contract, and it invents a clause about early termination fees, you curse the output as a “hallucination.”

To the neural network, those two tasks are computationally identical. It is running the exact same maths: selecting words that look natural in that context.

An AI cannot “stop hallucinating” without destroying its ability to generate novel text altogether. You can constrain it, you can point it at specific documents, and you can build verification checks around it. But you cannot eliminate the fundamental nature of a probabilistic engine.

What are some real examples of AI hallucinations?

Two well-known ones: Air Canada’s chatbot gave a grieving passenger a bereavement fare policy that didn’t exist, and a tribunal made the airline pay; and a New York lawyer filed a brief citing court cases ChatGPT had made up.

If you think hallucinations are merely an academic curiosity or an amusing social media screenshot, study these corporate disasters.

Case 1: The Air Canada Bereavement Debacle

In 2022, a grieving passenger named Jake Moffatt needed to book a last-minute flight from Vancouver to Toronto following his grandmother’s death.

Moffatt visited Air Canada’s official website and interacted with their customer-facing AI chatbot.

The passenger asked the chatbot how Air Canada’s bereavement policy functioned. The AI chatbot replied with clear, comforting instructions:

“If you need to travel immediately or have already purchased your ticket and will be submitting your ticket within 90 days of the date your ticket was issued, you can apply for your bereavement refund retroactively.”

Relying on the chatbot’s explicit statement, Moffatt purchased a full-fare ticket for over $1,600 CAD.

Weeks later, he submitted his paperwork for the retroactive discount. Air Canada flatly rejected the claim. Their human agents pointed to the airline’s official, static written policy, which stated clearly that bereavement rates could never be claimed retroactively after travel had occurred.

Moffatt took Air Canada to the Civil Resolution Tribunal in British Columbia.

Air Canada’s defence was astonishing. The airline argued in legal filings that the chatbot was “a separate legal entity” responsible for its own actions, and that Air Canada could not be held liable for information provided by its automated software agent.

The tribunal judge dismissed the airline’s argument:

The court ruled that Air Canada is a single corporate entity.
The chatbot was presented as an official representative on the company’s website.
A reasonable consumer has no reason to assume that an official tool provided by an enterprise is lying to them.

The court ordered Air Canada to pay Moffatt damages and court fees.

More importantly, Air Canada became a global case study in corporate embarrassment. They deployed a probabilistic text generator into a customer-facing role where strict, non-negotiable policy adherence was required.

Case 2: The Lawyer Who Cited Phantom Precedents Roberto Mata v. Avianca, Inc., 1:22-cv-01461 (S.D. New York 5/4/23) ( (Castel, J.)

In early 2023, a seasoned New York personal injury attorney filed a legal brief in federal court on behalf of his client, Roberto Mata, who was suing the airline Avianca for an injury sustained during a flight.

The legal brief was dense, formal, and authoritative. It cited several major court precedents:

Varghese v. China Southern Airlines Co Ltd., 925 F.3d 1339 (11th Cir. 2019)
Shaboon v. Egyptair, 2013 IL App (1st) 111279-U (Ill. App. Ct. 2013)
Martinez v. Delta Airlines, Inc., 2019 WL 4639462 (Tex. App. Sept. 25, 2019)

When the federal judge and Avianca’s defence attorneys attempted to pull the referenced court transcripts, none of them could be found in any legal archive.

The judge issued an order demanding an explanation.

The attorney was forced to submit an affidavit admitting that he had used ChatGPT to conduct his legal research. He had never used the software before. When he asked the AI if the cases were real, the model assured him: “Yes, these cases are documented in the Federal Reporter.”

The court sanctioned the attorneys with fines, and their professional reputations were permanently tarnished across national media.

The lawyer made the fatal error: he confused fluency with authority. Because the AI sounded like a seasoned Supreme Court justice, he assumed it was operating with the integrity of an audited database.

Why is AI biased?

Because it learns from historical data and repeats whatever patterns that data holds, including old prejudices.

If hallucination is the problem of AI inventing things that don’t exist, bias is the problem of AI institutionalising the worst parts of things that did exist.

When non-technical founders hear the word “bias,” they often assume it refers to conscious political opinions or malicious intent from the engineers who built the system.

In machine learning, bias is far simpler and far more dangerous:

Machine learning models are mathematical mirrors. They do not show us the world as it ought to be; they reflect the mathematical distribution of the historical data they were trained on.

+-------------------------------------------------------------------------+
|                     HOW ALGORITHMIC BIAS FORMS                          |
|                                                                         |
|   1. HISTORICAL REALITY (Flawed, unequal, prejudiced data)              |
|              |                                                          |
|              v                                                          |
|   2. MACHINE LEARNING ENGINE (Finds strongest mathematical patterns)    |
|              |                                                          |
|              v                                                          |
|   3. AUTOMATED DEPLOYMENT (Replicates past patterns at massive scale)   |
|              |                                                          |
|              v                                                          |
|   4. SYSTEMIC AMPLIFICATION ("The computer says this is optimal")       |
+-------------------------------------------------------------------------+

An algorithm does not hate anyone. It possesses no malice, no ideology, and no moral compass. It simply detects statistical patterns. If the historical data you feed it contains patterns of human discrimination, systematic neglect, or historical disparities, the algorithm identifies those patterns as the “correct” baseline for future decisions.

Case: The Classic Case Study: Amazon’s Scrapped Recruiting Engine

In the mid-2010s, Amazon set out to build an engineering marvel: an AI engine that could review thousands of incoming software developer resumes and automatically rank the top candidates.

The goal was noble and efficient: eliminate human recruiter fatigue and surface the best technical talent based purely on objective qualifications.

The engineering team trained the model on ten years of historical resumes submitted to Amazon. The machine analysed patterns among candidates who were hired versus those who were rejected.

Within a short period, engineers discovered a fatal pattern: the engine was systematically downgrading resumes that included the word “women’s.”

A resume that said “President of the Women’s Computer Science Club” was penalised.
Resumes from graduates of two specific all-women’s colleges were automatically scored lower.

Why did the machine do this? Did the algorithm hate women?

No. The machine simply looked at the historical training data. Over the preceding ten years, the global tech industry was overwhelmingly dominated by men. The vast majority of resumes that had historically been submitted, hired, and promoted at Amazon belonged to male engineers.

The algorithm did not understand gender, civil rights, or workplace equality. It performed cold mathematical pattern recognition:

Variable A: Words appearing frequently on successful historical resumes (male names, male sports, non-gender-specific clubs).
Variable B: Words appearing rarely on historical engineering resumes (“women’s”).

The algorithm concluded: “The word ‘women’s’ has a negative statistical correlation with historically successful hires at this company. Therefore, penalise any resume that contains it.”

Amazon attempted to patch the software with explicit rules, but the underlying statistical bias kept finding new ways to surface. The project was quietly abandoned.

The Real-World Impact for Small Businesses

For an everyday founder, algorithmic bias is not a theoretical ethics debate. It is a direct commercial and regulatory risk:

Automated Credit & Lending Decisions: If you deploy an automated machine learning model to evaluate small-business credit applications, and your historical training data reflects decades of redlining or banking disparities, your model will systematically deny credit or charge higher interest rates to minority-owned enterprises. You are legally liable for that disparate impact under consumer protection laws, whether you intended it or not.
Algorithmic Pricing Disparities: Ride-sharing and delivery algorithms have frequently been exposed for charging higher surge pricing or offering fewer service routes in lower-income zip codes, because the algorithms optimised for “historical profitability” and “perceived driver safety scores.”
Facial Recognition in Retail: Off-the-shelf computer vision systems used for loss prevention and shoplifting detection consistently show drastically higher false-positive error rates on darker skin tones. Accusing a customer of theft because a 92%-accurate vision model misidentified them is the fastest path to a devastating public relations crisis and a civil rights lawsuit.

What is the difference between probabilistic and deterministic software?

Deterministic software gives the same answer every time; probabilistic software, which is what AI is, gives the most likely answer, which can change and can be wrong.

Understanding hallucination and bias brings us to the central architectural concept every founder must master if they want to survive the next decade of technology:

The divide between Probabilistic Software and Deterministic Software.

+-------------------------------------------------------------------------+
|                  PROBABILISTIC vs. DETERMINISTIC SYSTEMS                |
+------------------------------------+------------------------------------+
|   PROBABILISTIC (AI Models)        |   DETERMINISTIC (Traditional Code) |
+------------------------------------+------------------------------------+
| • Outputs are based on probability | • Outputs are guaranteed by logic  |
| • Ask the same question 10 times,  | • Execute the same code 10 times,  |
|   get 10 slightly different answers|   get the exact same result 10/10  |
| • Optimized for fluency & synthesis| • Optimized for accuracy & safety  |
| • "Probably correct" (92% - 98%)   | • "Provably correct" (100.00%)     |
| • Black-box reasoning              | • Auditable, step-by-step logs     |
+------------------------------------+------------------------------------+

Traditional software is deterministic.

If you write 2 + 2 in a Python script, it will equal 4. It will equal 4 today, it will equal 4 tomorrow, and it will equal 4 on a server running in Tokyo at 3:00 AM during a thunderstorm.
If you set a sales tax rule in your billing system: IF state == 'CA', THEN tax = 7.25%, that rule will execute with 100% mathematical fidelity across ten million transactions.

Artificial Intelligence is probabilistic.

It does not execute a calculation; it rolls a multidimensional mathematical die.
If an LLM tells you that 2 + 2 = 4, it does so because 4 has a 99.999% probability of following that sequence. But on the 100,000th attempt, under strange conversational context, it can output: “2 + 2 is approximately 5 in theoretical non-Euclidean accounting spaces.”

The Founder’s Trap

The trap occurs when a business takes an inherently probabilistic tool and assigns it to a deterministic task.

                           THE DANGEROUS PATH
                                   |
                                   v
             [ Probabilistic Model (LLM / Generative AI) ]
                                   |
         Directly managing unmonitored operational systems:
         - Invoice Calculations
         - Contract Authoring
         - Refund Issuance
         - Diagnostic Health Advice
                                   |
                                   v
                  [ Catastrophic Operational Rupture ]
                  (Hallucinations, Lawsuits, Lost Margin)

Look at where companies get burned:

The Accounting Department: Asking an LLM to extract financial figures from fifty scanned receipts and sum them together. The LLM does not calculate; it predicts what a financial summary should look like. It will randomly miss lines, invent round numbers, and misplace decimal points.
The Legal Department: Asking a generative model to draft a non-disclosure agreement or vendor contract from scratch without human legal review. The model will assemble legal-sounding phrasing that omits critical local jurisdiction mandates or includes contradictory liability clauses.
The Inventory Department: Asking an open-ended conversational model to handle customer returns and warranty authorisations without connecting it to a hard-coded inventory database.

When an AI system is 95% accurate, engineers celebrate. A 95% accuracy score on a benchmark test is considered a breakthrough in academic computer science.

In business operations, a 95% accuracy rate is a catastrophic failure.

If an e-commerce company processes 10,000 orders a day, a 95% accuracy rate means:

500 customers receive the wrong order.
500 customers are charged an incorrect price.
500 packages are routed to the wrong shipping address.

Every single day. Your customer service team will drown, your reputation will collapse, and your profit margins will be wiped out by chargebacks.

Can you stop AI from hallucinating?

Not completely, but you can contain it: let AI handle messy inputs, let tested code handle facts, money and rules, and have a person sign off on anything high-stakes.

Does this mean founders should avoid AI altogether? Not at all. It means you must change how you architect your systems.

You must never let a probabilistic model run free inside your organisation. You must build what we call a Deterministic Software Cage around the probabilistic engine.

+-------------------------------------------------------------------------+
|                    THE DETERMINISTIC SOFTWARE CAGE                      |
|                                                                         |
|   [ UNSTRUCTURED INPUT ]                                                |
|   (Messy customer email, long PDF, open-ended question)                 |
|             |                                                           |
|             v                                                           |
|   +-----------------------------------------------------------------+   |
|   |  OUTER CAGE: Deterministic Guardrails (Hard Rules in Code)      |   |
|   |  - Authenticate user credentials                                |   |
|   |  - Check policy boundaries (Max refund: $100)                   |   |
|   |  - Strip sensitive PII / Payment info                           |   |
|   |         |                                                       |   |
|   |         v                                                       |   |
|   |  [ INNER CORE: Probabilistic Engine (LLM) ]                     |   |
|   |  - Extracts context & intent                                    |   |
|   |  - Synthesizes unstructured text into structured JSON           |   |
|   |  - Drafts natural, empathetic language response                 |   |
|   |         |                                                       |   |
|   |         v                                                       |   |
|   |  OUTER CAGE: Deterministic Verification (Hard Code Audits)      |   |
|   |  - Validates JSON against strict database schema                |   |
|   |  - Runs exact math calculations via SQL / Python (Not AI)       |   |
|   |  - Confirms claims against verified knowledge-base IDs          |   |
|   +-----------------------------------------------------------------+   |
|             |                                                           |
|             v                                                           |
|   [ SAFE, AUDITED BUSINESS ACTION ]                                     |
+-------------------------------------------------------------------------+

In a well-engineered business architecture:

The AI handles the squishy, messy human inputs: It reads unstructured emails, understands conversational accents, summarises 50-page documents, and drafts initial email replies.
Deterministic software handles the truth, the money, and the rules: Traditional, unbending code runs the calculations, queries the audited database, validates that outputs match business policies, and prevents unauthorised actions.

The Three Pillars of Safe AI Architecture

1. Retrieval-Augmented Generation (RAG) with Source Grounding

Never ask an LLM to answer a business question from its “memory” (its internal training weights). Its memory is where hallucinations live.

Instead, use RAG.

Think of RAG as giving the AI an open-book exam:

You take your company’s audited, human-verified documentation (your actual employee handbook, product spec sheets, pricing guides).
When a user asks a question, your software searches your database for the three exact paragraphs that contain the answer.
You feed those three verified paragraphs into the LLM alongside a strict instruction:

“Answer the user’s question using ONLY the provided text below. If the answer cannot be found in this text, state clearly: ‘I do not have that information.’ Under penalty of system shutdown, do not extrapolate, assume, or invent details.”

By constraining the model to a closed set of verified facts, you reduce the hallucination surface area by over 90%.

2. Structured JSON Outputs Instead of Free-Form Text

If an AI needs to interact with your business systems, never let it write prose to your database.

Instruct the model to output its analysis exclusively as structured data (like a JSON object) with strict field requirements:

JSON

{
  "customer_intent": "refund_request",
  "order_id": 40921,
  "claimed_defect": "broken_zipper",
  "sentiment_urgency_score": 0.85
}

Once the model outputs this structured data, your traditional deterministic software takes over:

Traditional code checks if Order #40921 was purchased within the last 30 days.
Traditional code checks if the customer spent more than $50.
Traditional code calculates the exact refund amount to two decimal places.

The AI acted as an interpreter, but deterministic business logic controlled the checkbook.

3. Human-in-the-Loop (HITL) Guardrails

For any action involving legal liability, medical guidance, employee termination, or financial movement over a specified threshold, build a hard-coded checkpoint that requires a human signature.

The AI prepares the research, identifies the discrepancy, and drafts the email or transaction.

A human employee looks at the screen, reviews the primary source citations, and clicks: “Approve and Send.” You get 80% of the time savings of automation without surrendering operational control to an algorithm.

What should you check before using AI in your business?

Before you deploy any artificial intelligence feature or approve an internal AI workflow, print this checklist and force your technical leads or vendors to answer these five questions:

+-------------------------------------------------------------------------+
|                  THE FOUNDER'S AI GOVERNANCE FILTER                     |
|                                                                         |
|   [ ] 1. PROBABILISTIC ISOLATION                                        |
|       Is this AI model directly executing financial, legal, or          |
|       database changes, or is it mediated by deterministic code?        |
|                                                                         |
|   [ ] 2. SOURCE GROUNDING                                               |
|       Can every claim or summary generated by the system be traced back |
|       to a verifiable, human-audited source document via an API link?   |
|                                                                         |
|   [ ] 3. HISTORICAL BIAS AUDITING                                       |
|       What historical inequalities or reporting biases exist in our     |
|       underlying data, and what mathematical guardrails counteract them?|
|                                                                         |
|   [ ] 4. FAILURE MODE COST EVALUATION                                   |
|       If this system hallucinates a convincing lie with 100% confidence,|
|       what is our exact dollar loss and legal exposure?                 |
|                                                                         |
|   [ ] 5. HUMAN-IN-THE-LOOP SAFETY VALVES                                |
|       Does a real human employee sign off on high-stakes outputs before |
|       they reach a customer's eyes or impact a balance sheet?           |
+-------------------------------------------------------------------------+

Does setting temperature to zero stop AI hallucinations?

Setting your model’s temperature to zero does not make it deterministic; it merely makes it predictably wrong when its statistical weights diverge. When failure carries real-world consequences, software cannot rely on polite chatbot promises—it requires deterministic assertions, rigorous schemas, and intentional systems design.

Over this four-part series, we have stripped away the marketing theatre of Silicon Valley:

In Part 1, we saw that AI is not a synthetic consciousness waking up; it is an elaborate mixing board of statistical dials trained on historical data.
In Part 2, we learned that Predictive AI is a digital actuary that sorts numbers, while Generative AI is an autocomplete parrot that predicts language.
In Part 3, we saw how Agentic systems and physical robotics are moving AI out of the chat box and into real-world operational workflows.
And here in Part 4, we uncovered why probabilistic systems lie, why historical data encodes human prejudice, and why deterministic software rules must remain the backbone of any serious business enterprise.

The entrepreneurs who build lasting, highly profitable companies over the next twenty years will not be the naive techno-optimists who fire their staff and trust open-ended chatbots to run their business.

Nor will they be the cynical luddites who bury their heads in the sand and pretend artificial intelligence is a passing fad.

The winners will be the pragmatic systems architects.

They will use Generative AI to eliminate administrative friction and amplify human creativity. They will deploy Predictive AI to sharpen their operational instincts. They will harness Agentic automation to scale lean, highly efficient teams.

And they will lock every single bit of it inside an unbending, auditable cage of deterministic software and human judgement.

Build on bedrock, not on probabilities.

That standard is how we build at 9horsemen: AI where a wrong guess is cheap, fixed and tested code wherever bookings, customers or money are involved.

Our first product, Gallop, works this way. AI reads a description of a local service business; tested code builds its website, online booking and business app. If you run a service business, it’s free to build and takes a few minutes.