Latest Insights

How to Measure AI Note-Taking Application Quality

Introduction to Evaluating AI Note-Taking Applications

Evaluating an AI note-taking application is not about generating one quality score and calling it done. A note-taking pipeline has multiple components, each capable of failing in a different way. A single blended number tells you almost nothing about which one actually broke.

This blog covers the AI note-taking evaluation metrics that matter at each layer of the pipeline, how to set thresholds for them, and how we’ve applied these metrics in practice while testing a real AI medical scribe as part of Testiva’s evaluation framework. These metrics form an important part of healthcare AI testing by helping teams assess transcription performance and clinical note accuracy.

The Two Critical Layers of AI Note-Taking Application Evaluation

Every note-taking pipeline runs in two stages:

ASR stage – turns audio into a transcript, with speakers attributed
Generation stage – turns that transcript into a structured note

These layers fail independently and for different reasons. A transcription error compounds into a generation error if it isn’t caught first, and a generation error can occur on a perfectly clean transcript. Measuring only the final note conflates these two failure sources. The metrics below are split by layer on purpose, and should be reported separately rather than merged into one score.

ASR Metrics

Word Error Rate (WER)

WER = (Substitutions + Deletions + Insertions) / Total Reference Words

This is the baseline transcription accuracy metric, but it treats every word equally, which is exactly where it can mislead you.

Example: Comparing the same script through an uploaded file versus a live microphone, WER moved from 0.98% to 10.75% on one recording, and from 3.32% to 19.48% on another, purely from the capture method, with everything else held constant.

Medical Term WER

The same formula, computed only over a clinical-term subset of the transcript, since an error here carries more downstream risk than an error on an ordinary word.

Example: On one recording, two anticoagulant drug names (“warfarin,” “apixaban”) were deleted outright and replaced with a generic phrase, pushing medical term WER from 5% to 11.1%, even though the rest of the transcript was largely intact. On another, “EKG” became “ECG,” a clinically harmless substitution that still registered as a 25% medical term WER on a four-term sample. This metric needs context, not just a raw threshold, to interpret correctly.

Negation Preservation Rate

The percentage of negated statements (“no pain,” “not improving”) correctly preserved.

Example: One recording dropped from 100% to 92.9% negation preservation purely because the phrase “without assistance” was lost under degraded audio. That single miss was enough to fail the run, even though WER alone was still under threshold. On a noisier recording of the same scenario, negation preservation fell as low as 55.6%, with four separate negated statements dropped or truncated.

Diarisation Accuracy

The percentage of speaker turns attributed to the correct speaker. We treat 90% as the pass threshold.

Example: This dropped to roughly 83% in a parent-child consultation, where the model failed to detect speaker changes between the child and the accompanying parent in close succession. Capture method also moved this metric in opposite directions: improving on a two-speaker recording captured by microphone, while degrading on a three-speaker recording captured the same way. Diarisation accuracy isn’t safely predictable from one test condition to another.

WER Drift

The change in WER between two runs that should otherwise produce comparable results. Our threshold was 2%. This metric exists specifically to catch inconsistency rather than one-off error.

Example: We saw drift fail at 2.85% and at 9.8%, both well above threshold, on capture-method comparisons where the underlying script and conditions were identical except for upload versus microphone. The degradation wasn’t isolated to one bad transcription run.

Clinical Note Generation Metrics for AI Note-Taking Applications

Faithfulness / Groundedness Score

The ratio of claims in the note taking testing is traceable back to the transcript. This was the most consequential metric in our testing.

The most pervasive defect we found, examination findings hallucination, showed up in four distinct forms across 13 separate test cases:

A default “no abnormalities reported” fill when no exam occurred
A patient-reported symptom restated as a clinical finding
A fabricated finding cited to justify a diagnosis
Exam results invented for an exam the transcript never described as completed

We also caught content from another patient’s case leaking into the current note during a brief interruption, and a diagnosis fabricated from over-inference, where the model documented a more severe, distinct clinical entity than what the patient actually described.

Completeness

The ratio of clinically relevant transcript content reflected in the note.

This metric needs careful interpretation: we found it misfiring in a specific, recurring way, docking a note for “missing” content, like safety-netting or follow-up language, that was never actually discussed in the source transcript. We confirmed this exact pattern on four separate test case pairs. A completeness failure needs to be checked against the transcript before it’s logged as a real defect, not assumed correct because an automated grader produced it.

Factual Accuracy

Agreement between specific facts, values, and conditional logic in the note versus the transcript.

We caught two distinct failures here:

A referral letter written as a definitive action when the doctor had explicitly made it conditional on an unresolved test result
A medication the doctor explicitly retracted mid-consultation that still appeared in the plan as a confirmed treatment item

Both cases pass a simple “was this topic mentioned” check, since the topic genuinely was mentioned. Factual accuracy needs to evaluate certainty and framing, not just topic presence.

Dosage/Terminology Fidelity

Exact-match checking on drug names, doses, and units, including deliberately injected unit traps (mg vs mcg, lbs vs kg).

The same retracted-medication case above also failed here, since the note carried forward an exact dose (levothyroxine 50 mcg) for a drug the doctor had explicitly walked back. We also saw a subtler version of this: terminology that’s clinically equivalent but textually divergent, like paracetamol appearing in a note where the transcript said acetaminophen, traced back to a locale shift introduced somewhere upstream in the microphone-capture pipeline.

Template Field Mapping

Whether content lands in the structurally correct section for a given template.

We saw this fail when a referral letter’s reference field rendered as a garbled placeholder instead of the actual subject line. A lower-confidence version of the same failure: content technically present but flagged as needing more clinical paraphrasing, worth weighting differently in an overall score than a true mapping error.

Setting Evaluation Thresholds for AI Note-Taking Applications

Metric	Threshold
WER	5%
WER Drift	2%
Diarisation Accuracy	90%

Each of these is evaluated independently rather than folded into one number, and that separation matters in practice. On one test pair, the overall asr_passed flag flipped to false purely because negation preservation missed, while WER was still comfortably under its own 5% threshold.

If we’d only looked at a single blended overall_score, values like 96.31 versus 96.05 across a capture-method comparison, that kind of single-metric failure would have been invisible inside a score that still looks fine at a glance. Thresholds need to live per-metric, and a pass/fail flag should always be inspectable down to which specific check actually failed.

Building a Comprehensive AI Note-Taking Evaluation Strategy

At Testiva, we don’t rely on any single metric or evaluator. Our approach supports scalable healthcare AI testing by combining multiple evaluation methods to measure both transcription performance and note quality. Our evaluation framework combines layer-specific metrics, multiple evaluation methods, and human validation to provide a more reliable picture of AI note-taking quality.

We scored every generated note across all seven template formats per test case (SOAP, Prose, CHAP, Referral, each in short and long variants) rather than sampling one or two, since a defect like the four-form hallucination pattern above doesn’t reproduce identically across every template. A narrow sample would have undercounted it.

We also ran identical scripts through two capture paths, upload and microphone, specifically to separate what the model contributes from what the recording pipeline contributes. This is how Testiva identified that the capture method affects different metrics in different, non-uniform directions.

Alongside the programmatic ASR metrics, we scored the notes layer using two separate LLM judges rather than one, and cross-checked their verdicts against each other on the same note. Where the two judges disagreed, particularly on hallucination and completeness, that disagreement itself became a signal for further investigation rather than something to average away.

On top of that, we layered manual human review on a sample of results, especially anything either judge flagged as failing, checking the cited reason directly against the transcript before treating it as a confirmed defect.

This combination of programmatic ASR metrics, multiple LLM judges, and targeted human review forms the core of Testiva’s AI note-taking evaluation framework, enabling teams to identify transcription errors, hallucinations, and note-quality issues with greater confidence.

Conclusion

A robust evaluation framework is essential for building trustworthy AI note-taking applications. Measuring each stage of the pipeline independently, setting metric-specific thresholds, and validating automated evaluations helps uncover transcription errors, hallucinations, and other quality issues before they impact users.

At Testiva, we apply these best practices through a comprehensive evaluation framework that combines layer-specific metrics, targeted testing, automated evaluation, and human validation. This helps teams gain clear insights into model performance, improve clinical note accuracy, and build confidence in real-world deployments through structured healthcare AI testing.

Ready to evaluate your AI note-taking application with confidence? Contact Testiva today for a demo of our evaluation framework and learn how structured QA can improve accuracy, reduce hallucinations, and strengthen the reliability of your AI-generated notes.