LLM evaluation means measuring whether your AI answers correctly, and measuring it again on every prompt or model change. I build the test set, automatic measurement and production tracking of quality and cost, including for AI someone else built.

Without a test set every prompt change is a feeling. With one it's a number.

Who it is for

  • Companies that already use AI but don't know how well it works
  • Teams where something quietly broke after a model or prompt change
  • Companies that have to prove AI quality to management or an auditor

How it works

  1. I collect real questions and cases and turn them, with correct answers, into a test set.
  2. I set up metrics: accuracy, citations, refusals, speed and cost.
  3. I wire the measurement into development so it runs on every change.
  4. I set up production monitoring and a monthly report.

What you get

Test set

Real questions and cases with correct answers that you keep.

Metrics

Accuracy, citations, refusals, speed and cost per answer.

Gate on change

A prompt or model change that makes the numbers worse doesn't go through.

Abuse testing

Attempts at hidden instructions and data leaks, measured like quality.

Production monitoring

Quality and cost over time, with an alert when they drop.

In practice

Construction Graph RAG has 128 questions with gold answers and a metrics gate in CI. It showed me that a knowledge graph made retrieval worse (MRR 76.1 → 54.8), and it tracked prompt-injection success falling from 16.7 % to 0 %. ClaimPilot is measured on 40 development and 20 test claims.

What changes in the company

A number, not a hunch, decides whether a model or prompt change ships. You see a problem in the report before a customer reports it.

Technology and data security

The test set can use anonymised data. Measurement runs on your side or in an EU region.

Price

€900–2,160, 1–3 weeks. Ongoing monitoring from €190 a month. Pricing

FAQ

What is LLM evaluation?

Measuring a language model's quality on a set of questions with correct answers. It shows accuracy, citations, refusals and cost, measured the same way on every change.

Can you measure AI that someone else built?

Yes. I only need access to its inputs and outputs; I don't need to change its code.

How many questions do we need?

50 to 150 real questions to start. Construction Graph RAG had 128.

How much does it cost?

€900–2,160 for the test set and measurement, 1–3 weeks. Production monitoring from €190 a month.

Related