Does this detector find enough of the attacks?
An AI-assisted detector has passed its alert review. The team wants to promote it based on that result.
The question
Larkspur Systems and this offline benchmark are fictional. The fixed set has 100 events. Twenty generated alerts; analysts have confirmed five as attacks and fifteen as benign. The other eighty events have not yet been adjudicated. This is not a production sample.
| Events | |
|---|---|
| Confirmed attacks | 5 |
| Benign alerts | 15 |
| Events without alerts, truth not yet checked | 80 |
Every event is counted once. Analysts will finish the truth labels before a release decision.
On this fixed benchmark, precision must be at least 20% and recall at least 90%. AI assistance is allowed for this practice case, but all calculations and conclusions must be explainable.
“Choose your next action and explain the evidence behind it.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: What can you calculate now?
What would you do here? Open the move you’d make.
Calculate alert precision
The strong move. The denominator is alerts, not all events.
Say accuracy is 25% and the detector therefore failed.
Tempting, but it misses something. Accuracy needs the truth for non-alerted events too.
See how a strong candidate says it
“Precision is 5 / 20 = 25%: the share of alerts that are true attacks.”
Why it works: The denominator is alerts, not all events.
Step 2: Can you approve recall?
What would you do here? Open the move you’d make.
Finish the benchmark labels
The strong move. Reviewed alerts cannot reveal false negatives.
Assume every non-alert is benign.
Tempting, but it misses something. That assumes the very errors the evaluation must measure.
See how a strong candidate says it
“No. We must adjudicate the eighty non-alerted events to find missed attacks before applying the recall gate.”
Why it works: Reviewed alerts cannot reveal false negatives.
Step 3: Does the full benchmark pass?
Analysts finish all eighty non-alerted events: four are attacks and seventy-six are benign. The original twenty alert labels do not change.
What would you do here? Open the move you’d make.
Apply both gates
The strong move. Precision passes, but recall does not.
Approve it because 81 of 100 events are classified correctly.
Tempting, but it misses something. Overall accuracy hides four missed attacks and does not satisfy the recall gate.
See how a strong candidate says it
“There are nine attacks in total, and five were detected. Recall is 5 / 9, about 55.6%, so the 90% gate fails.”
Why it works: Precision passes, but recall does not.
Step 4: What should change next?
What would you do here? Open the move you’d make.
Investigate misses before promotion
The strong move. The failed benchmark is evidence for improvement, not proof of production coverage.
Tune until this same benchmark passes and claim production recall.
Tempting, but it misses something. Optimising to one set can overfit it and cannot establish production performance.
See how a strong candidate says it
“Keep promotion on hold, inspect the missed cases, tune within the approved scope and evaluate on a separate labelled set.”
Why it works: The failed benchmark is evidence for improvement, not proof of production coverage.
The answer, in short
- Decision
- Do not promote this detector on the supplied gate.
- Because
- Its fixed-benchmark recall is about 55.6%, despite 81% accuracy.
- What would change it
- An independently labelled evaluation meeting both gates could support a new decision.
Common traps
- Precision and recall use different denominators.
- A full offline benchmark is still not production proof.
Check your answer
- I used alerts as the precision denominator.
- I looked for missed attacks.
- I applied both release gates.
- I kept production claims narrower than the evidence.
Last reviewed Nov 10, 2026