Your Prompt System Has No Tests, And That Is Why You Cannot Tell It Is Broken

Tags: ai, python, testing, showdev

Code fails loudly. A prompt system fails in silence, and it fails while still producing something that looks completely fine.

I found this out the slow way. I had built a multi-skill agent system: 15 skills, nine commands, each one writing structured JSON that the next one reads. It worked for weeks. Then it did not, and I could not tell you when it stopped, because nothing ever threw. A skill quietly stopped writing one field. The next skill read a null and carried on. The final score came out a few points off, in a document that read exactly as convincing as it had the week before.

Plausible output is the one thing these models are never bad at. That is precisely the problem.

What a test even means here

You cannot assert on the prose. Run the same prompt twice and you get different words, and that is fine, because the words are not the contract. Something else is.

Three things turned out to be testable, and together they catch nearly everything:

The arithmetic. My system scores six weighted dimensions and applies a penalty when any dimension falls below a floor. That is deterministic. The model produces the dimension values, but the final number is a function of them, and a function is something a checker can recompute without going anywhere near the model. If the number on disk disagrees with the number the checker computes, one of them is lying and it does not matter which.

The shape. Every skill writes a file with an expected structure. Required fields, enums for anything constrained, explicit nullability, conditional requirements where one field's presence forces another. This is schema validation, and it is unglamorous, and it caught more real regressions than anything else I wrote.

The prose rules that are actually numbers. A memo has to sit inside a word budget. It has to contain its required sections. It has to cite at least three URLs that are shaped like URLs. None of that judges quality, and all of it catches drift, because the specific way a model degrades is by getting longer, vaguer and less sourced.

The checker

About 750 lines of Python. No dependencies, standard library only, runs in well under a second.

I want to defend the no-dependencies part, because it was not laziness. This checker runs inside a hook on every file write. If it needs a virtualenv, it will not be there when someone clones the repo, and a test that does not run is worse than no test because it lets you believe you are covered.

The contract lives in one JSON file keyed by output filename, with six directives: required, enums, nullable, conditional, nested_required and one_of. That last one earns its place. Some outputs are valid in more than one shape, for example a business-to-business analysis carries fields a consumer analysis does not, and one_of lets the contract say "this group, or that group" without branching the whole schema.

The part most people skip

A checker that never fails is indistinguishable from a checker that does not work.

So the suite runs against ten fixtures, six valid and four deliberately broken. The broken ones do not merely have to fail. They have to fail with exactly 23, 3, 1 and 2 errors respectively. If a fixture starts producing 22 errors, something in the checker stopped looking.

Then I mutation-tested the assertions themselves: take a passing fixture, break one specific thing, confirm the harness catches it. This is where I found that one of my own tests was passing for the wrong reason. My mutation removed a string case-sensitively while the assertion lowercased the text first, so a capitalised copy survived and the check never fired. The assertion was correct. My test of the assertion was wrong. I would never have found it by reading the code.

If you take one thing from this article, take that. Test the tests. In a system where the output is generated, your checker is the only thing standing between you and confident nonsense, and an untested checker is a smoke alarm with the battery out.

Wiring it to the agent

The last piece is making the agent fix its own output without a human in the loop.

Claude Code has a PostToolUse hook that fires after a file write. It receives JSON on stdin including the path. My hook extracts the path, returns immediately if it is outside a managed folder, runs the checker on that folder, and exits 2 with the errors on stderr when something fails. Exit code 2 on this hook does not block anything, it just surfaces stderr back to the model, and the model reads the errors and corrects the file before it moves on.

The whole hook is about 40 lines, and the most important line in it is the try/except that makes it exit 0 on any unexpected exception. It fails open, always. A hook that crashes and blocks work would get uninstalled by the end of the first week, and then there would be no checking at all. A checker nobody runs protects nothing.

What it caught that I did not expect

I built this to catch the model. It caught me.

Running over my accumulated analyses, one pattern was impossible to miss. My competition estimates before research were systematically too optimistic, by 20 to 40 points, on all five ideas I had run. Never once too pessimistic.

That is not a bug in the model. That is a bug in the rubric, and specifically in letting an unresearched number be treated as a number at all. So I capped unresearched competition estimates at 45 and relabelled them an upper bound rather than a score.

The harness did not find that by being clever. It found it because writing every intermediate value to disk in a checkable format meant there was finally something to look across.

If you want to steal this

The whole thing is MIT and open. The checker is tests/validate_memory.py, the contract is tests/schemas.json, the fixture generator is tests/make_fixtures.py, and the hooks are in .claude/hooks/.

https://github.com/Latifox/find-me-saas

The project it belongs to is a startup-idea validator, but the testing pattern has nothing to do with startup ideas. If your agent writes structured output that another step consumes, this shape transfers directly. Recompute what is deterministic. Validate the shape. Assert the numeric properties of the prose. Test the tests. Fail open.

Then go and look at what the record says about you, because that is where the actual finding will be.