Guide
Dual screening and Cohen's kappa: a practical guide
Why two independent reviewers catch more eligible studies, how to run dual screening without it taking over your timeline, and how to calculate, interpret and report Cohen's kappa.
Why screen in duplicate
A methodological review found that single-reviewer screening misses more eligible studies than conventional double screening [6], and a study of second reviewers found that they added eligible studies that the first reviewer had missed [7]. PRISMA 2020 asks you to report how many reviewers screened each record and whether they worked independently [8].
How to run it
- Pilot first. Both reviewers screen the same small set, compare decisions and tighten unclear criteria before the main screen.
- Keep reviewers blind. Neither should see the other's votes until both have screened a record.
- Resolve conflicts with a fixed rule. Use discussion, then a third reviewer if needed. Decide in advance and write it in the protocol.
- Measure agreement after the pilot and again at the end of each stage.
Calculating Cohen's kappa
Kappa compares observed agreement with the agreement expected by chance [1]: κ = (po − pe) / (1 − pe).
Worked example: two reviewers screen 1,000 records.
| B include | B exclude | Total | |
|---|---|---|---|
| A include | 40 | 20 | 60 |
| A exclude | 10 | 930 | 940 |
| Total | 50 | 950 | 1,000 |
- Observed agreement po = (40 + 930) / 1,000 = 0.970
- Chance agreement pe = (0.060 × 0.050) + (0.940 × 0.950) = 0.003 + 0.893 = 0.896
- κ = (0.970 − 0.896) / (1 − 0.896) = 0.074 / 0.104 ≈ 0.71
Raw agreement is 97%, but kappa is about 0.71. That's because most records are obvious excludes that any two reviewers would agree on.
Interpreting kappa
Landis and Koch's bands (0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect) are widely quoted but were offered as arbitrary benchmarks [3]. McHugh argues they are lenient for health research and suggests stricter readings [2]. Use them as a guide, not a pass mark.
Beware the prevalence paradox. When almost all records are excludes, kappa can be low even when raw agreement is very high [4, 5]. Report both numbers, plus the counts in the 2×2 table, so readers can judge for themselves.
What to do with a low kappa
- Look at the disagreements, not just the number. They usually cluster on one or two criteria.
- Rewrite the unclear criterion, re-pilot, and record the change and its date.
- Don't re-screen only until the number looks good. Report the kappa you actually got.
How to report it
Where SyRev fits
SyRev supports dual review at title/abstract and full-text screening, with screening kept blinded until both reviewers have screened and disagreements going to a resolve step. It reports raw agreement and Cohen's kappa. The second reviewer can be a human workspace member (on Team and institutional plans) or the AI. If you use the AI as the second reviewer, describe it as such in your methods. It isn't the same as two independent human reviewers.
References
- Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37–46. doi:10.1177/001316446002000104
- McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012;22(3):276–282. doi:10.11613/BM.2012.031
- Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33(1):159–174. doi:10.2307/2529310
- Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol 1990;43(6):543–549. doi:10.1016/0895-4356(90)90158-L
- Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol 1993;46(5):423–429. doi:10.1016/0895-4356(93)90018-V
- Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Med Res Methodol 2019;19:132. doi:10.1186/s12874-019-0782-0
- Stoll CRT, Izadi S, Fowler S, Green P, Suls J, Colditz GA. The value of a second reviewer for study selection in systematic reviews. Res Synth Methods 2019;10(4):539–545. doi:10.1002/jrsm.1369
- Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi:10.1136/bmj.n71