Guide

Dual screening and Cohen's kappa: a practical guide

Why two independent reviewers catch more eligible studies, how to run dual screening without it taking over your timeline, and how to calculate, interpret and report Cohen's kappa.

Why screen in duplicate

A methodological review found that single-reviewer screening misses more eligible studies than conventional double screening [6], and a study of second reviewers found that they added eligible studies that the first reviewer had missed [7]. PRISMA 2020 asks you to report how many reviewers screened each record and whether they worked independently [8].

How to run it

  1. Pilot first. Both reviewers screen the same small set, compare decisions and tighten unclear criteria before the main screen.
  2. Keep reviewers blind. Neither should see the other's votes until both have screened a record.
  3. Resolve conflicts with a fixed rule. Use discussion, then a third reviewer if needed. Decide in advance and write it in the protocol.
  4. Measure agreement after the pilot and again at the end of each stage.

Calculating Cohen's kappa

Kappa compares observed agreement with the agreement expected by chance [1]: κ = (po − pe) / (1 − pe).

Worked example: two reviewers screen 1,000 records.

B includeB excludeTotal
A include402060
A exclude10930940
Total509501,000

Raw agreement is 97%, but kappa is about 0.71. That's because most records are obvious excludes that any two reviewers would agree on.

Interpreting kappa

Landis and Koch's bands (0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect) are widely quoted but were offered as arbitrary benchmarks [3]. McHugh argues they are lenient for health research and suggests stricter readings [2]. Use them as a guide, not a pass mark.

Beware the prevalence paradox. When almost all records are excludes, kappa can be low even when raw agreement is very high [4, 5]. Report both numbers, plus the counts in the 2×2 table, so readers can judge for themselves.

What to do with a low kappa

How to report it

Two reviewers independently screened all titles and abstracts after piloting the criteria on [n] records. Reviewers were blinded to each other's decisions, and disagreements were resolved by discussion or by a third reviewer. Agreement was [x]% (Cohen's κ = [y]).

Where SyRev fits

SyRev supports dual review at title/abstract and full-text screening, with screening kept blinded until both reviewers have screened and disagreements going to a resolve step. It reports raw agreement and Cohen's kappa. The second reviewer can be a human workspace member (on Team and institutional plans) or the AI. If you use the AI as the second reviewer, describe it as such in your methods. It isn't the same as two independent human reviewers.

References

  1. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37–46. doi:10.1177/001316446002000104
  2. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012;22(3):276–282. doi:10.11613/BM.2012.031
  3. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159–174. doi:10.2307/2529310
  4. Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990;43(6):543–549. doi:10.1016/0895-4356(90)90158-L
  5. Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol 1993;46(5):423–429. doi:10.1016/0895-4356(93)90018-V
  6. Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol 2019;19:132. doi:10.1186/s12874-019-0782-0
  7. Stoll CRT, Izadi S, Fowler S, Green P, Suls J, Colditz GA. The value of a second reviewer for study selection in systematic reviews. Res Synth Methods 2019;10(4):539–545. doi:10.1002/jrsm.1369
  8. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi:10.1136/bmj.n71