A wrong note got past review
Example · wrong-answer check · AI scribe · 20 scenarios × 8 runs
160 runs · model version recorded on eachReviewed by a named person
Getting started
From first call to evidence in your hands.
Three stages: a free session, a fixed 10-day first engagement, and a re-check every time your model, provider or prompt changes. You always know what happens next and what it needs from you.
Fixed scope · your staging environment · a named reviewer on every result
A model update broke a safety check
Example · model-update re-check · same 20 scenarios re-run
160 runs re-checkedEvery difference reviewed by a named person
Illustration. Example values from a fictional project, not a client engagement.
The three stages
Start small. Stay covered.
Each stage stands on its own. You can stop after any of them and keep everything produced so far.
-
The free session
45 minutes. Five failure scenarios for one AI feature, the safeguard for each, and what is tested today. You keep the map.
-
The 10-day first engagement
The full scenario set, a recorded baseline, every safeguard tested at the seam, and the evidence pack handed over.
-
The re-check
When your model, provider or prompt changes, the same set runs again and every difference is reviewed by a person.
The first engagement
What happens in the ten days.
A typical plan. The scope is fixed before day one, so the ten days do not grow.
-
Days 1 to 2
Build the set
We turn the scenario map into a full test set, with your experts’ definition of a wrong answer for every scenario.
-
Days 3 to 4
Baseline
The set runs on today’s model, every case repeated, with a manifest recorded for every run.
-
Days 5 to 8
Break it on purpose
At the seam, we feed your product wrong answers and test whether each safeguard holds, run after run.
-
Days 9 to 10
Review and hand over
A named person reviews every result. You receive the pack and a walk-through of each finding.
What we need from you
Three things, and only one is hard.
A staging environment
We work in your environment, under your control. We do not need access to production, and nothing we do touches your live users.
One engineer as a point of contact
Someone who can show us where the model’s answer enters the product, and answer questions as they come up. A few hours across the ten days, not a full-time commitment.
Your definition of a wrong answer
For each scenario, from the people who know your domain. This is the hard one, and it is the one thing only you can give us. We bring the structure; you bring the judgement.
Written permission for adversarial tests
Before we plant instructions in user text or take the AI service away mid-request, we ask for your permission in writing, scenario by scenario.
-
45
Minutes to start
-
10
Days, fixed
-
1
Engineer from your side
No live user data at any stage. Anything that needs real records runs inside your environment, under your control.
Book a free session
What you do not need to do
No rebuild. No new system to learn.
The work fits around a team that is already busy shipping.
-
Change your code
We control what your product receives at the seam, on staging. Your code, your model and your prompts are untouched.
-
Hand over user data
Testing runs on scripted and synthetic cases. No live user data leaves your environment.
-
Learn another system
The pack is written to be read: plain documents and records your engineers, your buyers and your board can follow without us in the room.
The re-check
Your product will change. The evidence should keep up.
A baseline is only useful if it is used. Three things should trigger a re-check, and each one is easy to miss.
A new model version
Your provider retires a model or you choose to upgrade. Nothing in your code changes, but the answers do.
A new provider
A move between providers changes format, tone and failure habits all at once.
A new prompt
A wording change is a release. It deserves the same check as a code change.
What comes back: the same set, re-run under the same conditions, with every difference marked better, worse or new, reviewed and signed by a named person before you release.
Start with the free session.
45 minutes with your product owner and one engineer. Five failure scenarios for one AI feature, the safeguard meant to catch each, and what is tested today. You keep the map, whatever you decide next.