A wrong note got past review
Example · wrong-answer check · AI scribe · 20 scenarios × 8 runs
160 runs · model version recorded on eachReviewed by a named person
How we test your AI
Most AI testing checks the answer. We test what happens next.
Model benchmarks tell you how a model performs in general. They cannot tell you what your product does when that model hallucinates, leaves something out or changes after an update. We test exactly that, on your staging environment, and hand you a record anyone can reproduce.
Your staging environment · one engineer from your side · no live user data
A model update broke a safety check
Example · model-update re-check · same 20 scenarios re-run
160 runs re-checkedEvery difference reviewed by a named person
Illustration. Example values from a fictional project, not a client engagement.
The method
One place we touch: the AI seam.
Every AI feature has a seam: the point where the model’s answer enters your product. We stand in for the model there, return the answer we choose, and watch what your product does next.
How one test run works
- Your system, unchanged
- Where we stand in
- What is under test
01Unchanged
Your product
Step 1 of 5
Your product
On your staging environment. Same code, same prompts, same settings. Nothing is modified for the test.
01Unchanged
Your product
The AI seam
Prompt, sent as normal
The answer we choose
02We stand in
The AI service
- Source
- “No chest pain”
- Returned
- “Chest pain”
Example fault · negative dropped
Step 2 of 5
The AI service
Your model provider, replaced on staging by a scripted answer we control: wrong, partial, slow, empty or unavailable.
- Omission
- Reversed negative
- Invented detail
- Unsafe value
- Hidden instruction
- Timeout
01Unchanged
Your product
“Chest pain”
02We stand in
The AI service
Step 3 of 5
The answer enters your product
The answer we chose is now inside your product, exactly as a real model answer would be. Nothing else about your system has changed.
03Under test
Your safeguards
- Marked as a draft until reviewed
- Review that cannot be skipped
- Unsafe values blocked
- Model version recorded
Step 4 of 5
Your safeguards
Every scenario runs 8 times
Caught: held and recorded
Got through: logged as a finding, with steps to reproduce
Only if it passes
04
The person using it
Step 5 of 5
The outcome
Should only ever see an answer your safeguards have checked.
Recorded on every run
So anyone can re-run it and get the same result.
Model
Provider, model and snapshot date
Prompt
Version and fingerprint
Fault
Exactly what we supplied
Outcome
What your product did
Evidence
Screenshots and trace
Review
A named reviewer
We never change your code, your model or your prompts. We control one thing, what your product receives, and record everything that happens after it.
Where this sits
Not red teaming. Not model evaluation.
You may already have both. They answer different questions from the one we answer.
-
Red teaming asks
Can your AI be made to fail? Hostile inputs go in, and someone sees whether the model can be pushed off course.
-
Model evaluation asks
How good is the model in general? Benchmarks and scores, measured away from your product.
-
We ask
When the AI is wrong, what does your product do? We supply the wrong answer ourselves and test whether anything stops it.
Twelve ways we make your AI wrong.
Each one is scripted, repeatable and tied to a named risk. Your product either catches it or it does not.
-
Omission
Something left out
A detail present in the source is missing from the answer. We check whether anything flags the gap.
-
A negative reversed
“Not approved” becomes “approved”. One dropped word, and the meaning is the opposite.
-
Hallucination
A detail invented
A name, a date or a figure that was never in the source appears in the answer.
-
A value swapped
The right kind of value, but the wrong one: another name, another amount, another account.
-
An unsafe or out-of-scope suggestion
The answer recommends something your product is not meant to offer, or outside the limits it enforces.
-
The wrong format
The answer arrives in a shape your code does not expect. We watch what breaks downstream.
-
A refusal
The AI declines when it should answer. Your product should say so, not show a blank.
-
The wrong source
The answer is built on the wrong document or record. We check the source is shown and can be checked.
-
Prompt injection
An instruction planted inside ordinary user text. We check whether your product obeys it.
-
An empty answer
Nothing comes back. A silent empty field where an answer should be is its own risk.
-
A slow answer
The answer arrives late. We check timeouts, retries and what the user is told while waiting.
-
The service down
The model is unavailable. Your product should fail safely, and visibly.
What we check
Six questions we ask of your product.
Is the answer marked as unreviewed?
A wrong answer that looks finished is more dangerous than one that looks provisional. We check that draft status is visible everywhere the answer appears, including exports and notifications.
Must a person review it, and can that step be skipped?
We try to skip it: direct links, keyboard shortcuts, bulk actions, the interface your integrations use. If any route gets past the review, the review is not a safeguard.
Is anything unsafe blocked outright?
Values outside the limits you define should not be saveable at all. We test the boundary, either side of it, and what the user is told when it is refused.
What happens when the AI is slow, empty or down?
We delay the answer, empty it, and take the service away in the middle of a request. Your product should fail visibly and safely, never quietly.
Is the model version stored with the output?
When someone asks later which model produced an answer, your product should know. We check the version is recorded with every output.
Are instructions hidden in user text ignored?
We plant instructions inside ordinary input: a note, an uploaded file, a pasted message. Your product should treat them as content, never as commands.
-
12
Ways we make it wrong
-
6
Safeguard questions
-
8
Runs per scenario
Every result is a pass or a fail against a named risk, graded by severity against your own hazard log, with the evidence attached. No impressions, and no single score standing in for the detail.
Book a free session
Check 2 · Model change
When the model changes, you will know what changed.
Providers retire and replace models on their own schedule. Your code does not change, your tests still pass, and the answers are different. The same applies to a prompt rewrite or a change of provider.
-
Baseline
A fixed set of cases run on today’s model, every case repeated, with the full run recorded so it can be reproduced.
-
Re-run
After a model, provider or prompt change, the same set runs again under exactly the same conditions.
-
Review
Every difference is read by a person and recorded as better, worse or new. No single score stands in for that.
-
Decide
You get the comparison before you release, so the decision to ship is made on evidence, not on hope.
Why repeat every case? Because the same input does not always produce the same answer. A case that passes eight times out of eight is a different thing from one that passes once, and your evidence should say which you have.
Check 3 · Correctness
Right against what? Your experts decide.
We do not decide whether your product’s answers are correct. Your experts do, and we build the measurement around their judgement.
Reference answers
Correctness is only measured against answers your own experts define, or a reviewer agreed with you in advance. Never from invented data alone.
A checked scorer
Two of your experts label a set of answers independently, and a third settles disagreements. We check the machine scorer against them, per type of failure, and report the agreement corrected for chance. Omissions are checked against a reference list, never left to the scorer.
What you receive
Evidence that holds up.
Everything is produced by running tests on your real product, so it answers the question whoever is asking: a buyer, an investor, an assessor or your own board.
-
The scenario map and test set
Every failure scenario, the risk it belongs to, the safeguard meant to catch it, and the reference answers your experts wrote. Yours to keep.
-
Results by failure type
What held, what did not, and how often, so your engineers know exactly where to go.
-
The run record
A manifest for every run: model, prompt, data, scoring, repeats and reviewer. Reproducible by anyone who reads it.
-
The comparison report
For every model, provider or prompt change: better, worse, new. Reviewed and signed by a named person.
What is included
Included, and not included.
So you can see exactly where this sits next to the testing you already do.
Included
Not included
Hallucinations and invented details
Bias and accent testing
Omissions, checked against reference lists
Monitoring of live answers once released
Reversed negatives and swapped values
Security penetration testing
Prompt injection
Real-world accuracy rates
Slow, empty and unavailable AI responses
Data leakage and privacy testing
Whether review and other safeguards can be bypassed
Making your model better
Regression after every model, provider or prompt change
A certificate or a safety judgement
Scoring against your experts, with the scorer checked
The evidence is tested on a fixed set of scenarios, measured against your own experts. It is not a measure of real-world accuracy, and it does not replace monitoring once your product is in use.
Working with us
Questions your engineering and security leads will ask.
What access do you need?
A staging environment and one engineer as a point of contact. We work in your environment, under your control. We do not need access to production.
Do you handle real user data?
No live user data. Testing runs on scripted and synthetic cases. Anything that needs real records runs inside your environment, under your control, and we ask for written permission before any adversarial testing.
Do you change our code, our model or our prompts?
No. We control one thing: what your product receives at the AI seam, on staging. Everything else is untouched.
How long does it take?
The first session is 45 minutes and it is free. The first engagement is a fixed 10 days. A re-check after a change is quicker, because the test set and the baseline already exist.
Who does the work?
A testing team with 20+ years in regulated industries, with a named reviewer for every result. Where correctness needs a subject expert, that is your expert, or one agreed with you in advance.
What do you need from our team?
Your definition of a wrong or unsafe answer for each scenario, from the people who know your domain. That is the one thing only you can give us.
What is this not?
Not a certificate, not a safety judgement, and not monitoring of live answers. You decide whether to release. We produce the evidence you decide with.
Do you work in regulated sectors?
Yes. Regulated-industry testing is where this team comes from. For health tech, the same testing feeds your compliance evidence; see the Healthtech and Compliance pages.
Start with the free session.
45 minutes with your product owner and one engineer. We map the five ways your AI could be wrong that matter most for one feature, and the safeguard meant to catch each one. You keep the map, whatever you decide next.