Guide
How to audit AI exclusions in systematic review screening
If an AI tool removed records from your screening queue, you need evidence that it did not throw out eligible studies. Here is a simple, defensible way to check, and how to report the result.
Why audit the exclusions specifically
Records the tool keeps still pass through human screening, so errors there are usually caught. Records the tool excludes are never seen again unless you look. Single-reviewer screening is known to miss more eligible studies than dual screening [1], and an automated exclusion is effectively a single, unchecked decision. An audit puts a number on that risk.
Step by step
- Decide the sample size before you look. Write it in your protocol or a dated note. A few hundred records is typical; the table below shows what different sizes can tell you.
- Draw a simple random sample of AI-excluded records. Use a recorded seed so the sample can be reproduced.
- Screen the sample blind. Mix the sampled exclusions with some records the tool kept, or at least hide the tool's decision, so the reviewer is not anchored by it.
- Count eligible records found using the same title/abstract criteria as the main screen. Resolve borderline cases the same way you would in the main screen.
- Calculate the miss rate with a confidence interval. Use the Wilson score interval [2], which behaves well when the count is zero or very small.
- Translate it into missed studies. Multiply the upper bound of the interval by the total number of AI exclusions. That is the worst plausible number of eligible records left in the excluded pile.
- Act on the result. If any eligible record turns up, or the upper bound is too high for your review, widen the check or re-screen the exclusions by hand. Decide this threshold in advance.
What a clean sample can and cannot tell you
Upper 95% Wilson bound on the miss rate when the sample finds no eligible records:
| Records checked | Eligible found | Upper 95% bound | If the tool excluded 4,000 records |
|---|---|---|---|
| 100 | 0 | 3.7% | up to about 148 eligible records |
| 200 | 0 | 1.9% | up to about 75 |
| 500 | 0 | 0.76% | up to about 30 |
| 1,000 | 0 | 0.38% | up to about 15 |
Two things follow. A clean sample never proves zero misses, and a small sample can leave a lot of room when the excluded pile is large. Because most records in a typical search are ineligible, even a few missed studies can matter a great deal to recall, so report the interval rather than just "no errors found".
Related checks
- Known includes. If you have studies you already know are eligible (from earlier reviews or seed papers), check whether the tool kept all of them. This is a quick test of recall, but these studies are often easier than average.
- Stopping rules. If you used active learning and stopped screening early, a statistical stopping criterion gives a more formal guarantee than a fixed sample [3].
- Workload saved. Work saved over sampling (WSS) is the usual way to express how much screening the tool saved at a given recall [4]. Report it alongside the audit, not instead of it.
How to report it
Put the sampling method, seed, sample size and result in a supplementary appendix, and refer to it from the selection process section of your methods (PRISMA 2020 item 8) [5]. For wording around the rest of the AI workflow, see our guide to reporting AI-assisted screening.
Where SyRev fits
SyRev builds this check in: its validation pack draws a blind sample of AI exclusions for a human to screen and reports the result with a Wilson confidence interval, ready to include in your appendix. You can also run the same steps by hand with any tool and a spreadsheet.
References
- Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol 2019;19:132. doi:10.1186/s12874-019-0782-0
- Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc 1927;22:209–212. doi:10.1080/01621459.1927.10502953
- Callaghan MW, Müller-Hansen F. Statistical stopping criteria for automated screening in systematic reviews. Syst Rev 2020;9:273. doi:10.1186/s13643-020-01521-4
- Cohen AM, Hersh WR, Peterson K, Yen PY. Reducing workload in systematic review preparation using automated citation classification. J Am Med Inform Assoc 2006;13:206–219. doi:10.1197/jamia.M1929
- Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi:10.1136/bmj.n71