01
a prompt change is a behaviour change
Once an LLM feature is in production, its prompt is business logic. Changing a sentence in it, swapping the model, raising the reasoning effort or loosening the output schema all change what customers get, often in ways that only show up on inputs nobody tried before merging. Yet prompts often live as strings scattered through handlers or as rows edited in an admin panel, with no review, no history and no quick way back.
The fix borrows three habits from ordinary software delivery. Prompts live in versioned files in git, together with everything that affects their behaviour, and a linter checks them. Every change runs an eval suite in CI that fails on explicit thresholds. And production resolves prompts through an alias in a small registry, so rolling back is a data change that takes effect in seconds. This post builds each piece in Python; the same shape works in any stack.
02
one file holds the whole contract
A prompt file should contain everything that changes the output: the model, its settings, the input variables, the output schema and the text itself. Then a pull request that changes behaviour shows up as a diff a reviewer can read.
There's no temperature in that file on purpose. OpenAI's GPT-6 models reject it at their default reasoning settings (the API returns "Unsupported parameter: 'temperature' is not supported with this model"), and Anthropic removed temperature, top_p and top_k in API version 2026-10-21. On current models the knobs that matter are the model ID, the reasoning effort and the output limit, so those are what the file pins.
The output schema is enforced by the provider rather than hoped for. OpenAI's Responses API takes it as text.format with type: "json_schema" and strict: true, and Anthropic's Messages API has the equivalent under output_config.format. That removes a whole class of tests about markdown fences and prose around the JSON, although your code should still validate the result, because a refusal or a truncated response won't match the schema. A call that turns the file into a request looks like this:
A linter catches the mistakes that don't need a model call to find. This one checks for required keys, a moving alias in the model field, a schema that allows extra fields, template variables that don't match the declared inputs and a file name that doesn't match the version:
On a deliberately broken file it prints every problem and exits non-zero, which is all CI needs:
03
temperature 0 was never a determinism switch
Old prompt-testing advice says to set temperature to 0 and compare outputs with stored snapshots. Even on models that still accept the parameter, that doesn't make outputs repeatable. Horace He's write-up at Thinking Machines Lab from September 2025 sampled 1,000 completions at temperature 0 from Qwen3-235B and got 80 distinct ones, which diverged at the 103rd token. The cause is that GPU kernels give slightly different numerical results depending on batch size, and batch size depends on how busy the server is, which no API user controls. With batch-invariant kernels all 1,000 matched, but hosted APIs don't promise that.
So exact-string snapshots of paid model calls make a flaky suite that people learn to rerun until it's green. What holds up is asserting on the structured fields that matter, running each case several times and gating on the pass rate, and keeping the raw outputs of failures so a human can read them.
04
evals in ci with explicit thresholds
An eval case is an input plus the values each field is allowed to take. Real production inputs make the best cases, with personal data removed, and every bug a user reports should become one.
The thresholds go in a file next to the prompts, so a change to the bar is reviewed like any other change:
The runner calls the model for every case several times, validates against the schema, checks the expected fields and writes a JSON artifact for every failure. The gate compares the result with the absolute thresholds and with the pass rate of the version currently in production:
The suite itself is a normal pytest file. In CI the call_model argument is the real client above. Here it's a stub that returns an incomplete answer with chatter in front of the JSON once in fifteen calls, which is enough to fail the 0.99 valid-output threshold and to check that the failure artifact contains what someone debugging it needs:
Running the real suite costs money and takes minutes, so trigger it only on pull requests that touch the prompts, the eval cases or the model client, and upload the artifacts when it fails:
Pick the number of samples per case with the cost in front of you. Five samples of 200 cases is 1,000 calls per run, which is cheap on a small model and noticeable on a large one, and it's the reason the trigger paths above are narrow.
05
contracts for retrieval and attribution
In a retrieval-augmented feature, most regressions come from the retrieval side (a new chunker, a re-embedded corpus, changed index settings) while the generated answers still read fluently. Test that boundary directly, against a fixed corpus version, without calling the generating model at all. Three contracts catch most breakage: the expected document appears in the top k results, the offsets on each hit point at exactly the text the hit carries, and every citation in an answer quotes text from a chunk that was actually retrieved.
Assert on rank rather than on absolute similarity scores, because any change to the embedding model shifts the scores even when the ranking is fine. When the attribution contract fails, the message should show the claim and the chunk side by side, as pytest does here for an answer that cites a refund policy saying the opposite of what it claims:
Tie the golden retrieval file to a corpus version that records the extractor, chunker, embedding model and index settings. When a test fails after a rebuild, you then know the rebuild caused it and can diff the chunks instead of arguing about whether the prompt or the provider changed.
06
aliases make rollback a data change
The last piece is how production finds the prompt. Store every released version in a table, and let the application ask for lead_qualifier@production, where production is an alias pointing at a version. Moving the alias is the deploy, moving it back is the rollback, and a log table records who did which.
Pointing production at version 4, then back at version 3, resolves to the matching model and text each time, leaves an audit trail and refuses a version that was never released, because of the foreign key:
With a 30-second cache per process, a rollback reaches every worker within half a minute without a build or a container restart. The CI pipeline inserts the version row when the eval gate passes, and a person or a script moves the alias. Log the resolved prompt version, the configured model and the model field the provider returns on every request, so that when someone reports a bad answer you can say exactly which prompt and model produced it. A second alias such as candidate, served to a small share of traffic, gives you a canary with the same mechanism, and a kill switch that points the feature at a rules-based fallback is worth having for flows where a wrong answer writes bad data downstream.