LLM testing checks how your application holds up with real users: long conversations, retrieval, tool calls, integrations, and deliberate misuse. Black-box by default, with first findings in week 1.
Black-box testing
we test as your users do, reading tool calls from your traces
Multi-turn, not single-shot
scripted conversations with topic switches and contradictions
Tested separately
each integration checked on its own, not only end to end
Re-tested on every change
model, prompt, or retrieval update
Trusted by teams shipping AI to production
LLM testing verifies that software built on a large language model behaves correctly in real use. It applies to chatbots, in-app assistants, AI search, content generation, and agentic features that take actions on a user’s behalf. The checks cover the whole system: the application keeps context, retrieves the right source, calls the right tool, returns output the next system can read, and refuses what it should refuse.
QA Madness provides LLM testing services for SaaS companies, AI product teams, and enterprise software vendors across the UK, USA, Canada, and Europe.
This page covers testing the application and its integrations. Scoring the quality of individual answers is a separate service, linked below.
How is LLM testing different from LLM evaluation?
LLM testing checks the behavior of the application and everything wired into it.
LLM evaluation scores the quality of the model’s answers.
The difference matters in practice: a system can score well on single prompts and still fall apart in a five-turn conversation or a broken tool chain. Finding that is this service.
| Traditional software testing | LLM Testing | |
|---|---|---|
| Input | Defined, predictable | Free-form, multi-turn, sometimes hostile |
| Expected result | One fixed output | Correct behaviour across a valid range of outputs, asserted on structure and meaning and reported as a pass rate over repeated runs |
| Where it breaks | UI, logic, API contracts | Plus context, retrieval, tool calls, output format, safety |
| Re-test trigger | Code change | Model, prompt, retrieval, or code change |
LLM testing covers the system around the model.
It runs alongside your functional and regression testing, and pairs with the evaluation of answer quality
Conversations, retrieval, tool use, integrations, safety
Faithfulness, answer correctness, safety of content, tone, completeness
AI output testing as one layer of your existing QA
Full QA for AI products: functional, regression, performance, security, accessibility
We build LLM features into our own tooling — and test them the same way we test yours.
QA Madness engineers integrate LLMs into our own products, including QAM Hub, our test management platform.
We meet the same prompt regressions and broken tool calls as builders, then break them systematically as QA. That view from both sides is what a team applying standard test cases does not have.
Functional QA is done, but nobody has tested long conversations, misuse, or what happens when a user goes off script.
With RAG, the model answers from your help center or docs. You need to know what the system does when the right source isn't found or the index has gone stale.
It calls APIs, creates tickets, or updates records, and a step failing mid-chain has consequences a user can see.
A small change moves how the system handles edge cases and safety boundaries, usually without telling anyone.
7 areas, each one a failure mode standard test cases are not designed to reach.
We test memory, topic switches, and contradictions across whole conversations. Users stop repeating themselves and stop abandoning the chat.
We test the retrieval layer apart from the model: was the right source found, and what happens when it is stale or missing. Whether the answer stays faithful to the source it was given is scored in LLM evaluation.
We check which tool gets called, with which arguments, and how a failure mid-chain is handled. We also test whether the model can be talked into an action or a data request outside the user's permissions.
We verify that the system refuses harmful requests, protects sensitive data, and resists jailbreak attempts. Scenarios are mapped to the OWASP Top 10 for LLM Applications, including prompt injection, sensitive information disclosure, improper output handling, excessive agency, and system prompt leakage.
We confirm outputs are schema-valid for whatever consumes them: parseable JSON, correct markdown, length, structure. This is a deterministic check and sits here rather than in evaluation.
Conflicting instructions, off-topic requests, typos, and several languages in one message. Real traffic looks like this.
After a change to a model, system prompt, or retrieval, we retest the affected scenarios. Behavior that was stable in the last release stays stable in this one.
Black-box, on the integration layer - no access to your model weights or training data required. We test the AI the way your users hit it.
Multi-turn dialogues built to test memory, topic switches, and contradictions. You find out where the conversation breaks, not only whether one prompt works.
Each scenario runs several times. We assert on behaviour, structure, and meaning rather than exact wording, and report a pass rate per scenario.
Public jailbreak and injection datasets, extended with attacks specific to your domain and your data, and mapped to the OWASP Top 10 for LLM Applications.
Scenarios where the source is present, missing, or out of date, and checks of which tool was called with which arguments. We read these from your traces or logs — for example LangSmith, Langfuse, or OpenTelemetry — or from a test endpoint, since neither is visible through the UI alone.
LLM-graded checks wired into your Playwright or WebdriverIO suites and CI pipeline: every model or prompt change runs against the golden set before release.
Input → actual output → expected → severity and business impact. Your developers reproduce each one the same day.
Automated checks are what give us coverage at scale.
Every result that involves judgment is reviewed and signed off by a QA engineer before it reaches your report.
No surprises. Clear milestones from kickoff to final report
We learn what your application does, which AI components it uses, what it connects to, and which risks your team needs answered.
Engineers review your documentation, system instructions, retrieval setup, tools, and intended use cases, then identify the highest-risk scenarios. We also agree how we will read retrieval and tool-call behaviour: your traces, your logs, or a test endpoint.
We design the conversations, edge cases, attack inputs, and tool-chain scenarios that target known LLM failure modes.
Engineers run the application through every scenario. Each finding includes the exact input, the actual output, the expected behaviour, severity, and business impact.
After your team ships fixes, we retest the affected scenarios and run regression checks. On project-based work, the scenario library and regression suite stay with your team for reuse.
Case Study
A customer-facing assistant for a mid-market SaaS, built on an LLM via API with retrieval over the help centre. Functional QA was green. Nobody could say how good the answers were
hallucinated answers invented pricing tiers and a refund window that does not exist
prompt-injection bypasses assistant disclosed its system instructions on crafted input
of multi-turn chats lost context forgot the product named two turns earlier
malformed-JSON outputs broke the downstream ticket-creation step
Outcome:
Roughly 3 weeks, added to the QA scope already running: 140 scenarios across 4 failure modes. Every high-severity finding was resolved before public launch, and the scenario library stayed with the team to rerun against the next model update.
Case Study
Every issue is documented so your team can reproduce and fix it the same day.
Failure mode Prompt Injection
Severity: HighFailure mode Retrieval
Severity: High“QA Madness has established a smooth workflow through effective communication. The team is trustworthy, efficient, and hardworking.”
“Thanks to QA Madness’s efforts, we are able to resolve technical issues and keep our platforms optimized and bug-free.”
“QA Madness was seriously professional. They listened to our needs and gave us the kind of work we expected. As a result of their efforts, we can locate a bug in the test environment, which prevents issues from entering production. I would recommend them, 100%.”
"They’ve always been very professional, prompt, and available when we needed them. We’ve never had any issues or needed to go back and teach them how to meet our standards."
Answered by QA Madness AI testing engineers
LLM testing verifies that an application built on a large language model behaves correctly. It checks conversations, retrieval, tool calls, output format, safety, and regressions: the behaviour of the whole system, not only the model’s answers.
LLM testing checks how the LLM-powered system behaves. LLM evaluation scores the quality of the model’s outputs against agreed criteria. In practice, we split them like this: retrieval, adversarial safety, and schema validity are tested here; faithfulness to the source, answer correctness, and the safety of the content itself are scored in evaluation. Most AI products need both.
Chatbots, in-app assistants, AI search, content generation, RAG-based features, and assistants that take actions through APIs or tools. AI chatbot testing is the most common request.
No access to model weights or your provider account is needed. We work black-box by default, the way your users reach the AI. For retrieval and tool-call testing we use your traces or logs — for example LangSmith, Langfuse, or OpenTelemetry — or a test endpoint, because those layers are not visible through the interface alone. It works with products built on OpenAI GPT models, Claude, Gemini, or any other model.
RAG testing covers the retrieval layer separately from the model: the right source, a stale index, and a missing source. For agentic AI testing we check that tools are called with the right arguments, that a failure mid-chain is handled safely, and that the model cannot be persuaded into an action beyond the user’s permissions. Both need traces, logs, or a test endpoint rather than the UI alone.
Prompt injection testing runs public jailbreak and injection datasets, extended with attacks specific to your domain. It includes indirect injection: instructions hidden inside content your users legitimately paste or upload. Scenarios are mapped to the OWASP Top 10 for LLM Applications.
Each scenario runs several times rather than once. We assert on behaviour, structure, and meaning rather than exact wording, and report a pass rate per scenario instead of a single pass or fail.
After any change to the model, system prompt, or retrieval logic. We retest the affected scenarios to confirm stable behaviour has not shifted.
You see preliminary findings from a first scenario sample in week 1. A focused engagement typically runs two to four weeks, depending on how many conversations, integrations, and attack scenarios are in scope.
No. It runs alongside functional and regression testing. If you want every layer covered by one team, see AI Quality Assurance.
QA Madness works with clients across the UK, USA, Canada, and Europe, from our headquarters in Warsaw, Poland.
Tell us about your application, and we'll map the highest-risk conversations, integrations, and attack scenarios, then give you a clear picture of what testing scope makes sense.
