LLM evaluation scores your AI’s answers for accuracy, grounding, safety, completeness, and tone, against criteria agreed before scoring starts. A QA engineer signs off on every judgment call. Preliminary scores in week 1.
Criteria agreed with you
your definition of a good answer, written down before scoring begins
Scored at scale
hundreds of prompt variants behind every scenario
Human-reviewed
a QA engineer reviews every failure, every borderline score, and a sample of passes
Re-evaluated on change
rescored after every model, prompt, retrieval, or source content change
Trusted by teams shipping AI to production
LLM evaluation is the structured scoring of a language model’s outputs against quality criteria defined for your product. It answers one question: is this answer right, safe, and useful here?
An LLM can answer the same question differently each time. Evaluation handles this by judging a range of acceptable answers against agreed criteria, so a valid rewording still passes while an invented fact does not.
QA Madness provides LLM evaluation services for SaaS companies, AI product teams, and enterprise software vendors across the UK, USA, Canada, and Europe. This page covers the scoring of AI outputs. Testing the system around the model is a separate service, linked below.
What is the difference between LLM Evaluation and LLM Testing?
Evaluation scores what the model says. LLM testing assesses how the system around the model behaves: conversation flow, retrieval, tool calls, integrations, and what happens when a step fails. Most products need both. Evaluation is the part that turns “the answer looks fine to me” into a score you can put in a release report.
| LLM Evaluation | LLM Testing | |
|---|---|---|
| Question it answers | Is the answer good? | Does the system behave correctly? |
| What's checked | Faithfulness to the retrieved context, answer correctness, safety of content, tone, completeness | Retrieval, adversarial safety, schema validity, tool calls, conversation flow |
| Pass / fail | Score against agreed criteria and golden answers | Expected behaviour per scenario |
| Re-run trigger | Model, prompt, retrieval, or source content change | Model, prompt, retrieval, or code change |
LLM evaluation is the measurement layer.
It sits on top of the testing you already run and reports the one thing those tests cannot: whether the answers coming back clear your quality bar.
Accuracy, grounding, safety, tone, format of outputs
Conversations, retrieval, tool use, integrations
AI output testing as one layer of your existing QA
Full QA for AI products: functional, regression, performance, security, accessibility
We evaluate LLM outputs in our own products — and hold yours to the same standard
QA Madness engineers ship AI features inside our own tooling, including QAM Hub, our test management platform.
We handle prompt changes, scoring criteria, and model updates in our own product, so we score your outputs against written criteria rather than impressions.
Functional QA is green, but no one has measured accuracy, grounding, or tone before those answers reach users.
You need to know which direction the answer quality moved, and by how much, before the change ships.
Help center, docs, or a knowledge base: you need proof that the answers come from your material.
Enterprise procurement increasingly requests documented AI quality and safety checks, often late in the deal.
8 criteria, scored separately so a report shows you exactly which one broke
We check whether answers hold up as fact for your product and flag anything stated confidently but wrongly. You find out before a user does.
Where the AI answers from your content, we verify that every claim in the answer is supported by the context it was given. An answer that adds a detail the source never contained is a defect, however plausible it sounds. Whether the right source was retrieved in the first place is covered by LLM testing.
We assess whether the response answers the question asked and includes what the user needs to act on it. Fewer follow-up tickets.
We score the content of outputs at scale for harmful advice, sensitive data, and statements your policy does not permit. Adversarial attacks against the system are run in LLM testing.
We check that answers sound like your brand and use your product's terms consistently. Your AI stops contradicting your documentation.
We check that the output reads correctly for its purpose: sensible length, usable markdown, and the right level of detail. Strict schema validity — whether the JSON parses — is a deterministic check and belongs to LLM testing.
We ask the same thing many ways, and run the same prompt several times, to see whether quality holds across phrasings and across runs. Results are reported as a pass rate, not a single verdict.
We rescore affected scenarios after changes to a model, prompt, or retrieval. Silent drops in quality surface before release. When you use retrieval, we report standard RAG metrics such as faithfulness, answer relevancy, and context precision.
Black-box, on the integration layer - no access to your model weights or training data required. We test the AI the way your users hit it.
For each scenario we write a golden answer and its acceptance criteria, specific to your product. "Correct" becomes a written standard your team approved.
Accuracy, safety, tone, and policy are scored as separate dimensions, so you see which one failed and by how much.
A second model scores every output. We first calibrate it against human-labelled samples, then a QA engineer reviews every failure, every borderline score, and a sample of passes. The judge model can run on your own provider account, so your data stays within the vendors you already approved.
Typos, mixed languages, frustrated users, off-topic detours. Real users do not write clean prompts, so we do not test with clean prompts.
LLM-graded checks wired into your Playwright or WebdriverIO suites and CI pipeline: every model or prompt change runs against the golden set before release.
Input → actual output → expected → severity and business impact. Your developers can reproduce each finding the same day.
Automated scoring is what gives us coverage at scale.
Every score that involves judgment is reviewed and signed off by a QA engineer.
A model does not grade its own output unchecked.
Clear milestones from kickoff to your 1st quality scores
We learn what your AI does, who it talks to, and which quality questions your team needs settled.
Engineers read your documentation, system instructions, and real use cases, then agree with you on what counts as a good output.
We build the golden answers and prompt sets: ordinary use cases, edge cases, and adversarial inputs.
Outputs are scored against the criteria and reviewed by QA engineers. Each finding arrives with input, actual output, expected output, and severity.
After fixes or updates, we rescore the affected scenarios. On ongoing engagements, evaluation runs in your CI: every model, prompt, or retrieval change is scored against the golden set before release. On project-based work, the scenario library stays with your team for reuse.
Case Study
A customer-facing assistant for a mid-market SaaS, built on an LLM via API with retrieval over the help centre. Functional QA was green. Nobody could say how good the answers were
hallucinated answers invented pricing tiers and a refund window that does not exist
prompt-injection bypasses assistant disclosed its system instructions on crafted input
of multi-turn chats lost context forgot the product named two turns earlier
malformed-JSON outputs broke the downstream ticket-creation step
Outcome:
Roughly 3 weeks, added to the QA scope already running: 140 scenarios across 4 failure modes. Every high-severity finding was resolved before public launch, and the scenario library stayed with the team to rerun against the next model update.
Case Study
Each low score is documented so your team can reproduce and fix it the same day.
Failure mode Accuracy / Hallucination
Severity: HighFailure mode Grounding / Faithfulness
Severity: High“QA Madness has established a smooth workflow through effective communication. The team is trustworthy, efficient, and hardworking.”
“Thanks to QA Madness’s efforts, we are able to resolve technical issues and keep our platforms optimized and bug-free.”
“QA Madness was seriously professional. They listened to our needs and gave us the kind of work we expected. As a result of their efforts, we can locate a bug in the test environment, which prevents issues from entering production. I would recommend them, 100%.”
"They’ve always been very professional, prompt, and available when we needed them. We’ve never had any issues or needed to go back and teach them how to meet our standards."
Answered by QA Madness AI testing engineers
LLM evaluation scores a language model’s outputs against quality criteria agreed for your product: accuracy, grounding, safety, tone, and format. It tells you whether the answers your product gives are good enough to reach users.
Accuracy and hallucination risk, faithfulness to your source content, relevance and completeness, safety and policy compliance, tone and terminology, output format, consistency across phrasings and runs, and whether quality holds after an update. Eight criteria, each scored separately.
We define golden answers and acceptance criteria with you, run many prompt variants per scenario, score the outputs automatically, and have a QA engineer review every judgment-call result.
Evaluation judges the quality of the model’s answers. LLM testing checks how the whole LLM-powered system behaves: conversations, retrieval, tool calls, and integrations.
We agree on acceptance criteria up front and judge each response against them. A range of valid answers passes. An invented fact or an off-policy statement fails, however well written it is.
Yes. We evaluate outputs at the integration layer, whichever model or provider sits behind it. No access to model weights or your provider account is needed. For grounding checks we use retrieval traces or logs, or a test endpoint that returns the sources used.
A score per criterion for every scenario, a pass rate by criterion, severity for each finding, and score movement after a fix or a model update. Where you use retrieval, we also report standard RAG metrics: faithfulness, answer relevancy, and context precision.
You see preliminary scores on a first scenario sample in week 1. A focused engagement typically runs two to four weeks, depending on how many scenarios and criteria are in scope.
A report of reproducible findings with severity ratings and recommended fixes, plus the scenario library so you can rescore after future updates.
QA Madness works with clients across the UK, USA, Canada, and Europe, from our headquarters in Warsaw, Poland.
Most teams find out from a user. Tell us what your AI does, and we'll suggest the evaluation criteria and scenarios that matter most for your product
