Guide

How to validate AI screening before using it in your review protocol

Before you write AI screening into a systematic review protocol, check the evidence, pilot the tool on your own kind of question, and pre-specify how you will audit it. Here is a practical way to do all three.

Why validate before the protocol, not after

A protocol commits you to a method. If AI screening is part of that method, reviewers, editors and peer reviewers will want to know why you trusted it and how you will check it. Average accuracy figures from a vendor are a starting point, not an answer: performance varies from review to review. One vendor reported that an LLM screening model's recall varied widely between reviews, and that performance "depended heavily on how eligibility criteria were written" [6]. Our own benchmark shows the same pattern: 96.5% recall overall across 310 public review datasets, but fewer than 95% in 31 of them [7]. So plan two things: a pilot before you commit, and a check during the review.

Step 1: Read the tool's published validation critically

Step 2: Pilot it on a review you have already finished

The most convincing evidence is how the tool behaves on your kind of question. Take a completed review in your field where you know the final includes, run the tool with the same criteria you would write for the new review, and measure:

MeasureWhat it tells you
RecallShare of the known includes the tool kept (did not auto-exclude). Report it with a confidence interval [3].
Records auto-excludedHow much screening the tool would remove.
WSS@95Work saved over sampling at 95% recall, the usual workload measure [2].
Missed includes, by nameRead every missed study. Are they borderline, or core studies that would change your conclusions?

If the misses trace back to criteria wording, fix the wording and re-run once. Do not keep tuning until the pilot looks perfect; that only overfits to one review.

Step 3: Pre-specify the method in your protocol

Write down, before screening starts:

  1. The tool, version or date used, and the setting or threshold (fixed, not tuned during the review).
  2. Which decisions the tool makes alone (for example, auto-excluding only very low-probability records) and which always go to a human.
  3. How AI exclusions will be checked: a blind random sample of a stated size, with a recorded seed [3].
  4. A stopping or fallback rule: what you will do if the check finds eligible records, such as widening the sample or screening all exclusions by hand. Statistical stopping criteria give a more formal version of this [4].
  5. How you will report it, including the PRISMA 2020 flow diagram and selection process (item 8) [1].
Example protocol wording: "Title/abstract records will be scored by [tool] against our eligibility criteria at its default setting, fixed before screening. Records below the tool's exclusion threshold will be excluded automatically; all other records will be screened by two independent reviewers. Before full-text screening, a reviewer blinded to the tool's decisions will screen a random sample of [n] automatically excluded records (seed recorded). If any eligible record is found, all automatically excluded records will be screened manually. Results will be reported with a 95% Wilson confidence interval."

Step 4: Check during the review

Run the pre-specified blind sample of AI exclusions and calculate the miss rate with a Wilson interval. Our guide to auditing AI exclusions walks through the arithmetic and shows how sample size limits what a clean sample can prove. Keep a human second reviewer where you can: single screening misses more eligible studies than dual screening [5], and an automated exclusion is effectively a single decision.

Step 5: Report it

Report the pilot (if done), the tool and setting, the number of records excluded automatically, the audit sample and its result, and any deviation from the protocol. See our guide to reporting AI-assisted screening in PRISMA 2020 for wording.

Where SyRev fits

SyRev publishes its validation on 310 public datasets with fixed settings, its weakest results and a citable report [7] (see Validation). You can replay a review you have already finished to see recall and work saved on your own topic, and every review comes with a validation pack: a blind random sample of auto-excluded records with a Wilson confidence interval, a decision log, a methods paragraph and an AI-use statement. The steps above work with any tool and a spreadsheet.

References

  1. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi:10.1136/bmj.n71
  2. Cohen AM, Hersh WR, Peterson K, Yen PY. Reducing workload in systematic review preparation using automated citation classification. J Am Med Inform Assoc 2006;13:206–219. doi:10.1197/jamia.M1929
  3. Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc 1927;22:209–212. doi:10.1080/01621459.1927.10502953
  4. Callaghan MW, Müller-Hansen F. Statistical stopping criteria for automated screening in systematic reviews. Syst Rev 2020;9:273. doi:10.1186/s13643-020-01521-4
  5. Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol 2019;19:132. doi:10.1186/s12874-019-0782-0
  6. Covidence. How Covidence decides what AI to release (and what to hold back). covidence.org/blog/ai-screening-automation-systematic-reviews (accessed 15 Oct 2026).
  7. SyRev Team. SyRev title/abstract screening: validation on 310 public systematic-review datasets (v1.0). Zenodo, 2026. doi:10.5281/zenodo.23399663

Frequently asked questions

How many AI-excluded records should I check?

It depends on how many records the tool excluded and how much uncertainty you can accept. With zero eligible records found, a sample of 300 gives an upper 95% Wilson bound of about 1.3% on the miss rate; 1,000 gives about 0.38%. Decide the number in your protocol before you look.

Is a vendor's published recall enough to justify AI screening?

Usually not on its own. Performance varies between reviews, so pair published validation with a pilot on a completed review in your field and a blind check of exclusions in the new review.

What recall should I aim for?

Many automation studies use 95% as a working benchmark for work saved (WSS@95), but there is no universal threshold. State your own target and fallback rule in the protocol.

Can I change the AI threshold during the review?

Avoid it. Fix the setting before screening starts, as you would any other method, and report any change as a protocol deviation.

Written by the SyRev Team · 15 October 2026