Guide

How to audit AI exclusions in systematic review screening

If an AI tool removed records from your screening queue, you need evidence that it did not throw out eligible studies. Here is a simple, defensible way to check, and how to report the result.

Why audit the exclusions specifically

Records the tool keeps still pass through human screening, so errors there are usually caught. Records the tool excludes are never seen again unless you look. Single-reviewer screening is known to miss more eligible studies than dual screening [1], and an automated exclusion is effectively a single, unchecked decision. An audit puts a number on that risk.

Step by step

  1. Decide the sample size before you look. Write it in your protocol or a dated note. A few hundred records is typical; the table below shows what different sizes can tell you.
  2. Draw a simple random sample of AI-excluded records. Use a recorded seed so the sample can be reproduced.
  3. Screen the sample blind. Mix the sampled exclusions with some records the tool kept, or at least hide the tool's decision, so the reviewer is not anchored by it.
  4. Count eligible records found using the same title/abstract criteria as the main screen. Resolve borderline cases the same way you would in the main screen.
  5. Calculate the miss rate with a confidence interval. Use the Wilson score interval [2], which behaves well when the count is zero or very small.
  6. Translate it into missed studies. Multiply the upper bound of the interval by the total number of AI exclusions. That is the worst plausible number of eligible records left in the excluded pile.
  7. Act on the result. If any eligible record turns up, or the upper bound is too high for your review, widen the check or re-screen the exclusions by hand. Decide this threshold in advance.

What a clean sample can and cannot tell you

Upper 95% Wilson bound on the miss rate when the sample finds no eligible records:

Records checkedEligible foundUpper 95% boundIf the tool excluded 4,000 records
10003.7%up to about 148 eligible records
20001.9%up to about 75
50000.76%up to about 30
1,00000.38%up to about 15

Two things follow. A clean sample never proves zero misses, and a small sample can leave a lot of room when the excluded pile is large. Because most records in a typical search are ineligible, even a few missed studies can matter a great deal to recall, so report the interval rather than just "no errors found".

Related checks

How to report it

A reviewer blinded to the tool's decisions screened a random sample of 300 of the 4,120 records excluded by [tool]. No eligible records were found (miss rate 0%, 95% Wilson CI 0–1.3%).

Put the sampling method, seed, sample size and result in a supplementary appendix, and refer to it from the selection process section of your methods (PRISMA 2020 item 8) [5]. For wording around the rest of the AI workflow, see our guide to reporting AI-assisted screening.

Where SyRev fits

SyRev builds this check in: its validation pack draws a blind sample of AI exclusions for a human to screen and reports the result with a Wilson confidence interval, ready to include in your appendix. You can also run the same steps by hand with any tool and a spreadsheet.

References

  1. Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol 2019;19:132. doi:10.1186/s12874-019-0782-0
  2. Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc 1927;22:209–212. doi:10.1080/01621459.1927.10502953
  3. Callaghan MW, Müller-Hansen F. Statistical stopping criteria for automated screening in systematic reviews. Syst Rev 2020;9:273. doi:10.1186/s13643-020-01521-4
  4. Cohen AM, Hersh WR, Peterson K, Yen PY. Reducing workload in systematic review preparation using automated citation classification. J Am Med Inform Assoc 2006;13:206–219. doi:10.1197/jamia.M1929
  5. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi:10.1136/bmj.n71