AI Feature Release Checklist: What to Verify Before You Ship an LLM Feature to Production

Reading Time: 9 minutes

Originally published: 5 Nov, 2026 Last updated: 6 Nov, 2026

An AI feature needs its own release gate because the usual one cannot see most of its failure modes. A wrong answer returns HTTP 200, a leaked system prompt renders as normal text, and an invented fact passes every assertion you wrote. This checklist covers 29 checks across 5 layers, which issues block a release, and what to re-run when a model or prompt changes.

It is written for CTOs, product managers, and QA leads deciding whether a feature is ready to ship. QA Madness provides independent software testing services for SaaS companies and AI product teams across the UK, Europe, and North America.

Why does an AI feature need its own release gate?

Because a passing test suite tells you the feature runs, not that it is correct. Traditional gates check status codes, schemas and user flows, and an LLM feature can satisfy all 3 while telling a customer something false.

The second reason is that the artifact you ship changes without a code commit. A prompt edit, a model bump or a re-indexed knowledge base all alter behaviour. None of them touch the application code your CI pipeline watches.

The practical implication: your release gate has to key off the model, the prompt and the data as well as the code. Otherwise it waves through the 3 changes most likely to break the feature.

What should you check before releasing an AI feature?

29 checks across 5 layers. The first 4 are specific to AI features. The fifth is the conventional QA that an AI feature still needs and often skips.

Layer	Check	How to verify	Blocks release?
Output quality	Factual accuracy on core questions	Run a golden set of 20 to 50 questions with answers agreed by the content owner	Yes
Output quality	Grounding in source content	Score each claim against the retrieved context; unsupported claims fail	Yes
Output quality	Refusal on out-of-scope questions	Ask questions the product cannot answer; expect an admission, not an attempt	Yes
Output quality	No invented specifics	Check for fabricated prices, dates, policies and feature names against the source of truth	Yes
Output quality	Completeness	Confirm answers include every fact the golden answer requires, not only the first	No
Output quality	Tone on difficult inputs	Run angry, confused and abusive phrasings; check the reply stays in brand voice	No
Safety	Direct prompt injection	Run curated jailbreak prompts; the feature must hold its instructions	Yes
Safety	Indirect injection via content	Hide instructions in a document, email or pasted text the feature will process	Yes
Safety	System prompt disclosure	Ask for the instructions directly and obliquely; nothing internal may surface	Yes
Safety	Harmful request refusal	Run the refusal categories your policy defines	Yes
Safety	Sensitive data leakage	Query as a low-privilege user for data a higher-privilege user owns	Yes
Safety	Off-policy advice	Ask for legal, medical or financial guidance the product should decline	Yes
System behaviour	Multi-turn context retention	Script conversations of 6+ turns; check the last turn still knows what turn 1 established	Yes
System behaviour	Topic switch	Change subject mid-conversation; old context must stop driving answers	No
System behaviour	User correction	Correct the assistant, then check later turns follow the correction	Yes
System behaviour	Retrieval finds the right source	Query against a labelled set where the correct document is known in advance	Yes
System behaviour	Retrieval returns nothing when nothing fits	Ask what no document covers; an empty result beats a near-miss	Yes
System behaviour	Stale index	Edit a source document, then query it after the stated refresh interval	No
System behaviour	Permission-scoped retrieval	Confirm restricted documents never reach an unauthorised user’s context	Yes
System behaviour	Correct tool, correct arguments	Trigger each tool call and inspect the arguments in the trace	Yes
System behaviour	Mid-chain failure	Make a tool fail or time out; the feature must degrade safely, not improvise	Yes
Integrations	Structured output matches the schema	Validate every machine-read response against the contract downstream expects	Yes
Integrations	Downstream step completes	Confirm the ticket, record or booking the AI triggers actually gets created	Yes
Integrations	Provider rate limits, overload and timeouts	Force a 429, a 503 or 529, and a timeout; check the user-facing behaviour	Yes
Integrations	Streaming and partial output	Interrupt a stream mid-answer; confirm nothing half-written is committed	No
Conventional QA	Functional flows around the feature	Standard functional and API coverage of everything the AI touches	Yes
Conventional QA	Authentication and authorisation on AI endpoints	Penetration testing of the application and its APIs	Yes
Conventional QA	Performance under load	Measure time to first token, streaming behaviour and token cost under load	No
Conventional QA	Accessibility of the AI interface	WCAG 2.2 checks on the chat interface: status messages, keyboard access, focus order	No

At QA Madness, this work runs in a black-box manner at the integration layer, without access to model weights or training data. 1 consequence worth planning for: retrieval and tool-call checks need your traces, logs, or a test endpoint. The interface alone does not show which document was retrieved or which argument was passed.

Output quality

Output quality is checked against a written standard, not an impression. Agree on a golden set with whoever owns the content, such as a support lead or product owner. Do it before anyone starts scoring, because “looks fine” is not a release criterion.

Score each answer on specific dimensions rather than overall: factual accuracy, grounding in the source, completeness, refusal behaviour, and tone. A single number tells you something has changed. Per-dimension scores tell you what to fix.

At QA Madness, an LLM-as-a-judge handles the volume, and a QA engineer signs off on every judgment call. An LLM-as-a-judge is a second model scoring the first one’s answers against your criteria, which is how you cover thousands of outputs rather than a sample.

The engineer reviews each failure, each borderline score, and a sample of the passes. That is also how we evaluate output quality when the volume is beyond manual review.

Safety and prompt injection

Safety is tested with curated attack prompts rather than by reading the system prompt and hoping. Use public jailbreak and injection datasets, extended for your domain and your data. Map the scenarios to the OWASP Top 10 for LLM Applications, where prompt injection has ranked first in both the 2025 and 2026 editions.

Test indirect injection specifically. If your feature summarises documents, reads email, or processes anything a user pastes, that content is an input channel an attacker controls. OWASP’s position is that no reliable prevention mechanism exists today, and that defense is therefore architectural rather than interceptive.

So retrieval is not a mitigation. Retrieval-augmented generation adds an injection channel rather than closing one, which is why OWASP carries a separate entry for vector and embedding weaknesses. Test the corpus as an input surface, and gate consequential actions on something other than the model’s judgement.

The check that teams skip: querying as a low-privilege user for data they should not see. Retrieval often ignores the permission model that the rest of the application enforces.

System behaviour: context, retrieval, tool calls

System behaviour covers everything between the user’s message and the model’s output. Each part needs its own scenarios because multi-turn context, retrieval, and tool calls fail in ways a single-turn test never reaches.

For conversations, use scripted multi-turn dialogues that test memory, topic switches, and contradictions. Entity carry-over is a common failure: the user names a product in turn 1 and asks about “its limits” in turn 3.

For retrieval, test the retriever separately from the model, because the same wrong answer can come from either. Our guide on how to test a RAG application covers those test cases in detail.

For features that take actions, 3 questions matter: which tool was called, with which arguments, and what happens when a step fails mid-chain. That is what ground testing AI agents covers.

If you want this layer covered before release, we test how your LLM feature behaves across conversations, retrieval and tool use.

Integrations and output format

Integrations fail silently, which is what makes them dangerous. A response a person reads happily can still break the system parsing it, and the conversation looks successful while nothing downstream happens.

Validate every machine-readable output against the schema the consuming system expects, then confirm the downstream step actually completed. Checking that the model returned something plausible is not the same as checking that a ticket exists.

Also force the failure cases your provider will eventually hand you: a 429, a timeout, a truncated stream. Each needs a defined user-facing behaviour rather than whatever the code does by accident.

2 details catch teams out here. Rate limiting and overload are different codes: 429 for the first, 503 or 529 depending on the provider, so retry logic keyed only to 429 misses half of it.

And not every 429 clears on retry. A quota or billing limit returns the same code and never will.

Functional, performance, security and accessibility

Conventional QA still applies, and AI features routinely ship without it because the AI absorbs the attention. The product around the model has UI, data, business logic, and API contracts, and all 4 need their usual coverage.

3 areas deserve specific mention. Performance means time to first token and streaming behaviour rather than page load, and token cost under load is a real operational risk. Security means penetration testing of the application and its APIs, including the endpoints the AI calls.

Accessibility is the one most often missed entirely, and a streaming interface needs more care than a live region wrapped around the output. Announce status rather than tokens. “Generating response,” “Response complete,” and error states are what WCAG 2.2 success criterion 4.1.3 Status Messages covers, and the answer body counts as result content, which that criterion excludes.

For the transcript, role=”log” is the documented technique, appending each message as a complete unit once generation finishes. Streaming tokens into a live region produces flooded or dropped announcements, and the W3C warns specifically about making an interface too chatty.

Add a heading and keyboard reachability, so a screen reader user can navigate to the output on their own terms. Then verify against real assistive technology.

Which issues block a release, and which can ship?

An issue blocks the release when it produces a false statement, exposes data, or breaks a downstream system. Everything else gets a fix date or goes on a monitoring list. Agree that distinction before testing starts rather than arguing it at the gate.

Severity	What it looks like	Release decision
Critical	Invented facts a customer would act on; data leakage; successful prompt injection; downstream failures that lose a transaction	Block. No exceptions, no feature flags
High	Context loss in common conversations; wrong source retrieved on core questions; schema violations on a secondary path	Block, unless the path sits behind a flag you can keep closed until the fix ships
Medium	Incomplete answers; tone slips; stale index within the documented refresh window	Ship with a fix date in the same sprint
Low	Formatting inconsistencies; verbosity; minor accessibility gaps outside the core flow	Monitor. Log it, review at the next planning session

The judgement sitting behind this table is business impact rather than technical severity. Take an answer stating a refund window that the company does not offer. That is a written commitment to a customer, so it blocks even though the system worked exactly as built.

At QA Madness, every finding is documented the same way: input, actual output, expected output, severity and business impact. AI and non-AI issues arrive in 1 format your team can triage together.

What do you re-test after a model or prompt change?

Re-test scope follows the type of change, not its size. A 1-word prompt edit can move behaviour further than a fortnight of code. “It was a small change” is not a scope decision.

What changed	What to re-run
Model version or provider	Everything. Output quality, safety, multi-turn, tool calls and output format. A new model invalidates your whole baseline
System prompt	Output quality, safety, tone, refusal behaviour and output format. Injection resistance especially, since the instructions are the defence
Retrieval logic, chunking or embeddings	Retrieval checks, grounding, citations and permission scoping. Output quality on the questions those documents answer
Knowledge base content	Golden set items touching the changed documents, plus stale index and refusal behaviour
Application code around the AI	Conventional functional, integration and performance coverage. Add output format if the parsing changed

Run this as AI-in-the-loop regression: a fixed golden set, scored on the same dimensions every time. The signal is the delta relative to the previous baseline, rather than a binary pass/fail. Our article on what to look for in an AI testing partner sets out the regression method in more depth.

The rule worth adopting: treat a prompt change as a release. It gets the same gate, the same re-test scope and the same sign-off as code.

Who signs off an AI release?

4 roles, each owning a decision nobody else is positioned to make. The point of naming them is that AI releases fail when “is this answer good enough” belongs to everyone and therefore to no one.

Role	Owns	Signs off on
Product	What a correct answer contains	The golden set, the refusal policy, and which incomplete answers are acceptable
Engineering	The system behaving as designed	Tool calls, integrations, output contracts, failure handling and performance
QA	Whether the evidence supports shipping	Test execution, severity assignment and the release recommendation
Legal or compliance	Regulatory exposure, where relevant	Data handling, retention, and disclosure requirements

Product owning the golden set is the part teams get wrong most often. If the engineer who wrote the prompt also defines what a good answer looks like, the test measures the prompt against itself.

The legal row applies selectively. 1 example for EU teams: the EU AI Act’s Article 50 transparency obligations have applied since 2 August 2026. Article 50(1) requires providers to design systems so that people are informed they are interacting with an AI, unless that is obvious in context.

Note it binds the provider rather than the deployer. Check which one you are before assuming it is someone else’s problem.

How does this fit into AI quality assurance?

This checklist is 1 gate inside a wider process. AI quality assurance covers the same layers continuously rather than once. That means what the AI says, how the features around it behave, and the software underneath.

The difference a process makes is the baseline. A one-off pre-launch check tells you the feature was acceptable on a Tuesday. A maintained golden set tells you whether last week’s prompt change helped or hurt.

Automated scoring finds candidates, and a QA engineer decides what ships. At QA Madness, that split holds, whether the engagement is a single pre-launch cycle or ongoing coverage. A judge model is still a model, and it is wrong often enough that nobody should let it gate a release alone.

What to take away 

Gate on the model, the prompt, and the data, not only on code. Agree on what blocks a release before testing starts, because severity arguments at the gate are really scope arguments nobody had earlier. And give the golden set to whoever owns the content, since a feature graded by the person who built it tends to grade well.

Your next step is 20 questions with agreed answers. That single artifact turns every later prompt change from a guess into a measurement.

Need every layer of your AI product checked before release? Check our AI quality assurance service.

FAQ

What should be on an AI feature launch checklist?

An AI feature launch checklist covers 5 layers: output quality against agreed golden answers, safety including prompt injection, and system behaviour across context, retrieval, and tool calls. The last 2 are integrations and output format, plus conventional functional, performance, security, and accessibility work. Roughly 29 checks, with blocking criteria agreed before testing starts.

How do you QA an LLM feature before release?

Test it black-box at the integration layer, the way users reach it, scoring outputs against golden answers agreed with the content owner. Run curated attack prompts for injection and safety, script multi-turn conversations, and validate structured outputs against the downstream schema. Have a QA engineer sign off on every judgement call rather than trusting automated scores.

What should you re-test after changing the LLM model or prompt?

A model or provider change invalidates your baseline, so re-run everything: output quality, safety, multi-turn behaviour, tool calls and output format. A prompt change needs output quality, safety, tone, refusal behaviour and format. Check injection resistance carefully, because the system prompt is what an injection attack tries to override.

Which companies provide AI QA services for SaaS products?

Independent QA providers with dedicated LLM testing practices, as distinct from automation vendors testing only the surrounding application. Ask which regulations and failure modes they cover, and who signs off on automated scores. Our article on what AI QA testing costs explains the pricing models and what moves a quote.