Guide
How to validate AI screening before using it in your review protocol
Before you write AI screening into a systematic review protocol, check the evidence, pilot the tool on your own kind of question, and pre-specify how you will audit it. Here is a practical way to do all three.
Why validate before the protocol, not after
A protocol commits you to a method. If AI screening is part of that method, reviewers, editors and peer reviewers will want to know why you trusted it and how you will check it. Average accuracy figures from a vendor are a starting point, not an answer: performance varies from review to review. One vendor reported that an LLM screening model's recall varied widely between reviews, and that performance "depended heavily on how eligibility criteria were written" [6]. Our own benchmark shows the same pattern: 96.5% recall overall across 310 public review datasets, but fewer than 95% in 31 of them [7]. So plan two things: a pilot before you commit, and a check during the review.
Step 1: Read the tool's published validation critically
- Is it public and reproducible? Look for named datasets, licences, and a citable report rather than a single headline number.
- Is recall reported, not just accuracy? In screening, most records are ineligible, so accuracy can look high even when relevant studies are lost. Recall (the share of truly included studies the tool kept) is the number that matters.
- Is the spread shown? Ask for the worst reviews, not only the average, and whether any benchmark resembles your topic and study designs.
- Were settings fixed in advance? Thresholds tuned per dataset after seeing the answers will overstate performance.
- Is the denominator honest? Check how records with no abstract were handled. For example, our report states that 26 relevant records without a title or abstract were not scored; counting them as missed lowers recall from 96.5% to 96.2% [7].
Step 2: Pilot it on a review you have already finished
The most convincing evidence is how the tool behaves on your kind of question. Take a completed review in your field where you know the final includes, run the tool with the same criteria you would write for the new review, and measure:
| Measure | What it tells you |
|---|---|
| Recall | Share of the known includes the tool kept (did not auto-exclude). Report it with a confidence interval [3]. |
| Records auto-excluded | How much screening the tool would remove. |
| WSS@95 | Work saved over sampling at 95% recall, the usual workload measure [2]. |
| Missed includes, by name | Read every missed study. Are they borderline, or core studies that would change your conclusions? |
If the misses trace back to criteria wording, fix the wording and re-run once. Do not keep tuning until the pilot looks perfect; that only overfits to one review.
Step 3: Pre-specify the method in your protocol
Write down, before screening starts:
- The tool, version or date used, and the setting or threshold (fixed, not tuned during the review).
- Which decisions the tool makes alone (for example, auto-excluding only very low-probability records) and which always go to a human.
- How AI exclusions will be checked: a blind random sample of a stated size, with a recorded seed [3].
- A stopping or fallback rule: what you will do if the check finds eligible records, such as widening the sample or screening all exclusions by hand. Statistical stopping criteria give a more formal version of this [4].
- How you will report it, including the PRISMA 2020 flow diagram and selection process (item 8) [1].
Step 4: Check during the review
Run the pre-specified blind sample of AI exclusions and calculate the miss rate with a Wilson interval. Our guide to auditing AI exclusions walks through the arithmetic and shows how sample size limits what a clean sample can prove. Keep a human second reviewer where you can: single screening misses more eligible studies than dual screening [5], and an automated exclusion is effectively a single decision.
Step 5: Report it
Report the pilot (if done), the tool and setting, the number of records excluded automatically, the audit sample and its result, and any deviation from the protocol. See our guide to reporting AI-assisted screening in PRISMA 2020 for wording.
Where SyRev fits
SyRev publishes its validation on 310 public datasets with fixed settings, its weakest results and a citable report [7] (see Validation). You can replay a review you have already finished to see recall and work saved on your own topic, and every review comes with a validation pack: a blind random sample of auto-excluded records with a Wilson confidence interval, a decision log, a methods paragraph and an AI-use statement. The steps above work with any tool and a spreadsheet.
References
- Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi:10.1136/bmj.n71
- Cohen AM, Hersh WR, Peterson K, Yen PY. Reducing workload in systematic review preparation using automated citation classification. J Am Med Inform Assoc 2006;13:206–219. doi:10.1197/jamia.M1929
- Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc 1927;22:209–212. doi:10.1080/01621459.1927.10502953
- Callaghan MW, Müller-Hansen F. Statistical stopping criteria for automated screening in systematic reviews. Syst Rev 2020;9:273. doi:10.1186/s13643-020-01521-4
- Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol 2019;19:132. doi:10.1186/s12874-019-0782-0
- Covidence. How Covidence decides what AI to release (and what to hold back). covidence.org/blog/ai-screening-automation-systematic-reviews (accessed 15 Oct 2026).
- SyRev Team. SyRev title/abstract screening: validation on 310 public systematic-review datasets (v1.0). Zenodo, 2026. doi:10.5281/zenodo.23399663
Frequently asked questions
How many AI-excluded records should I check?
It depends on how many records the tool excluded and how much uncertainty you can accept. With zero eligible records found, a sample of 300 gives an upper 95% Wilson bound of about 1.3% on the miss rate; 1,000 gives about 0.38%. Decide the number in your protocol before you look.
Is a vendor's published recall enough to justify AI screening?
Usually not on its own. Performance varies between reviews, so pair published validation with a pilot on a completed review in your field and a blind check of exclusions in the new review.
What recall should I aim for?
Many automation studies use 95% as a working benchmark for work saved (WSS@95), but there is no universal threshold. State your own target and fallback rule in the protocol.
Can I change the AI threshold during the review?
Avoid it. Fix the setting before screening starts, as you would any other method, and report any change as a protocol deviation.
Written by the SyRev Team · 15 October 2026