How to Catch Hallucinations in AI-Generated Code

A code hallucination is any output where an AI coding agent produces code that looks correct but references something that does not exist or behaves in a way it claims but does not deliver. The agent imports a package that was never published, calls a method that is not in the API, or tells you a button now saves the form when clicking it does nothing. The code is syntactically valid and reads as plausible, which is exactly why it survives review.

To catch hallucinations in AI-generated code, you sort them into three classes and apply the detection method that fits each. Hallucinated dependencies and APIs are caught at build time with resolution checks and type analysis. Hallucinated behavior, where the code runs but does the wrong thing, is caught only by executing the code against the real application and observing what happens. Silent breakage, where a hallucinated change quietly regresses a flow it was not supposed to touch, is caught by regression tests that exercise the whole journey. The first class is the easiest to catch and gets most of the attention. The last two are where the expensive failures hide.

This matters because AI-generated code arrives faster than a human can read it line by line. A reviewer confirms the imports resolve and the types check, then approves a 400-line diff without ever running the feature: static tooling catches the fabricated import, not the fabricated behavior. Knowing which class you have tells you which layer of verification will surface it.

Why hallucinations are a distinct failure class

It is tempting to file hallucinations under "bugs," but they differ. A normal bug tries to do the right thing and gets it wrong. A hallucination is code confidently built on a false premise: a library that does not exist, a signature the model imagined, or a claim about the UI that was never checked against the UI. The model does not flag the uncertainty. It writes import fastjson with the same fluency as a real import, and reports "the modal now closes on save" whether or not it looked. Because the code is well-formed, hallucinations pass any check that assumes broken code looks broken. It is part of why AI-generated code carries more bugs than reviewed human code.

The three categories, and how to catch each

1. Hallucinated dependencies and APIs

This is the most-studied category and the one with hard numbers behind it. A USENIX Security 2025 study generated 576,000 code samples across 16 code-generating models and found that 19.7 percent of the packages the models recommended did not exist: over 205,000 unique hallucinated package names. Open-source models fabricated packages at around 21.7 percent versus 5.2 percent for commercial models, and 58 percent of hallucinated names reappeared across repeated runs rather than being one-off noise.

The persistence turns a nuisance into an attack surface. Security researchers named the resulting threat slopsquatting: an attacker watches for a commonly hallucinated package name, registers a real malicious package under it, and waits for the next developer whose agent suggests that same import. The term was coined by Seth Larson, the Python Software Foundation's developer-in-residence. Unlike typosquatting, which relies on a human typo, slopsquatting relies on the model making the same mistake for everyone.

The same failure shows up one level down as hallucinated APIs: a call to response.getJSON() when the real method is response.json(), or a parameter the function does not accept.

How to catch this class:

• Resolve every dependency before it merges. Fail the build on any import that does not resolve to a published, pinned package the lockfile has seen.
• Lean on the type system and the linter. In typed languages, a hallucinated method or wrong signature is a compile error. In dynamic languages, a strict linter with import resolution catches most fabricated names.
• Pin and allowlist dependencies, so a novel package name is a decision a human makes, not a default the agent takes. Confirm a first-time package's age, downloads, and maintainer before trusting it.

The good news: this class is largely catchable before the code ever runs. The harder classes survive compilation.

2. Hallucinated behavior

This is the class static checks cannot reach. The imports resolve, the types check, the diff reads cleanly, and the agent's summary says the feature works. Then you run it and the save button does not save, the modal opens over the wrong content, the validation message never fires, or the new filter silently returns the unfiltered list.

Nothing here is a fabricated symbol; everything the code references exists. What is hallucinated is the behavior the agent claimed and never confirmed. Research on hallucinations in LLM-generated code puts this beyond functional-correctness bugs, into a category where code is plausible but does not do what it purports to do. For anything with a user interface, "does what it purports to do" is a runtime property. You cannot read it off the diff.

The only reliable way to catch a behavioral hallucination is to execute the change against the real application and observe the actual result: open the app in a real browser, drive the exact interaction the agent said it fixed, and assert on what the page does, not on what the agent reported. This is the same verification gap covered in detecting hidden bugs in AI-generated code. The failures that matter require running the software, and they are invisible to every tool that only inspects it.

This is where Shiplight fits the loop. It plugs into the coding agent as an MCP server and gives it eyes and hands in a real browser, so immediately after editing a UI, the agent opens the app, performs the interaction, and checks the outcome before claiming the change is done. Instead of asserting "the modal now closes on save," it opens the modal, clicks save, and reports whether the modal actually closed. The claim collapses the moment it is checked against a running browser. The full pattern is in verifying AI-written UI changes and how to verify AI-generated code. The design point: assert on observed behavior, never on the agent's narration, because an agent that hallucinated the behavior will just as happily hallucinate a passing self-report.

3. Silent breakage

The third class is a hallucination with blast radius. The agent changes a shared component or a piece of global state to satisfy the feature in front of it, confident the change is contained when it is not. The feature works, but a flow three screens away that depended on the old behavior is now broken, and nobody looked because it is not in the diff. This is a hallucination about scope, and you only catch it by running the parts of the app the agent did not touch.

How to catch this class:

• Keep a regression suite that covers your critical journeys end to end, not just the code paths in the current change. The point is to exercise what the agent believed it left alone.
• Run that suite on every pull request as a blocking gate. A regression caught at the commit that introduced it is cheap; the same regression found by a user is not.
• Write those tests from intent, not brittle selectors, so the suite does not shatter every time the agent refactors. With Shiplight, the agent authors these tests by walking the app, and they run locally with npx shiplight test alongside existing Playwright.

Layer the defenses to match the classes

No single tool catches all three classes, and treating them as one problem is why hallucinations reach production. Build the layers in the order they can fire:

Static checks are the cheap first line and clear the fabricated-symbol class almost entirely. But they define "correct" as "well-formed," and a behavioral hallucination is perfectly well-formed. The classes that pass compilation are the ones that require execution to catch, which is the layer where AI-generated code fails in front of users.

Key Takeaways

• Code hallucinations are confidently-wrong output, not ordinary bugs. They pass review because plausible code does not look broken.
• Split them into three classes: hallucinated dependencies and APIs, hallucinated behavior, and silent breakage.
• Fabricated packages and methods are largely catchable before runtime with resolution checks, types, linting, and allowlists.
• Behavior and scope hallucinations survive compilation and code review. Only running the code against the real app surfaces them, so ground behavioral checks in a real browser and assert on observed results, never on the agent's own report.

Frequently Asked Questions

How do you catch AI code hallucinations?

Sort them into three classes and match a method to each. Catch hallucinated dependencies and APIs at build time with strict dependency resolution, type checking, linting, and package allowlists. Catch hallucinated behavior by running the change against the real application in a browser and asserting on what actually happens. Catch silent breakage with an intent-based end-to-end regression suite that runs on every pull request. Static tooling handles the first class; only execution handles the other two.

What is a hallucination in AI-generated code?

It is output where the agent produces valid-looking code built on a false premise: a package or method that does not exist, or behavior it claims but never verified. The code compiles and reads as reasonable, which is why hallucinations pass review that assumes broken code looks broken.

What is slopsquatting?

Slopsquatting is a supply-chain attack that exploits package hallucinations. Because models fabricate the same non-existent package names repeatedly, an attacker can register a real malicious package under a commonly hallucinated name and wait for developers whose agents suggest that import. The term was coined by the Python Software Foundation's Seth Larson. A 2025 USENIX study found 58 percent of hallucinated names recurred across runs, which is what makes the attack practical.

Why do static analysis and code review miss code hallucinations?

Both evaluate whether code is well-formed, and a behavioral hallucination is well-formed. The imports resolve, the types check, and the diff reads cleanly, so both checks pass. Whether the save button actually saves is a runtime property that neither a linter nor a reviewer scanning a large diff observes. You have to run the feature.

Can unit tests catch hallucinated behavior in AI code?

Unit tests catch logic errors in isolated functions but miss hallucinated UI behavior and cross-flow breakage. A handler can be correct in isolation while the interaction it powers does nothing in the browser, or while a shared change silently breaks a different flow. End-to-end verification that drives the real application is required to catch behavioral hallucinations and silent regressions.

---

References: Socket: The Rise of Slopsquatting, USENIX Security 2025: We Have a Package for You! (code and data), Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code (arXiv 2404.00971), Playwright Documentation