Validation: how SyRev performs on public systematic-review datasets

We test SyRev on published, openly licensed screening datasets where the right answers are already known. Here is what we found, the reviews where it did less well, and how you can check the results yourself.

Last updated: 15 October 2026

Citable report: doi.org/10.5281/zenodo.23399663. Suggested citation: SyRev Team (2026). SyRev title/abstract screening: validation on 310 public systematic-review datasets (v1.0). Zenodo. https://doi.org/10.5281/zenodo.23399663

Headline numbers (Careful, our default setting)

How we test

Datasets. We use public datasets from published systematic reviews. Each record has a known answer: whether the study ended up in the review. The answers come from the review authors or the dataset creators, not from us. Every dataset we report is openly licensed, and we list each licence below.

What SyRev does. For each record, SyRev estimates how likely it is to meet each eligibility criterion, reading the title and abstract, or the full text where noted. Records that are very unlikely to meet the criteria are auto-excluded. Everything else goes to a person. These results come from the same screening model and the same settings that SyRev uses today.

What we measure.

How the thresholds were chosen. SyRev has two fixed settings, and they are the same for every dataset:

Not suitable for pharmacovigilance or regulatory use without a separate validation.

We did not adjust either setting for any dataset.

How the criteria were written. For the datasets built from published systematic reviews, the criteria were generated automatically from each review's published selection criteria. For the environmental-health, preclinical and SYNERGY sets, we wrote them by hand from the published descriptions. None were changed after seeing results.

Large pools. For most search pools, we scored every record. In two benchmarks (CLEF 2019 Intervention and the SWIFT-Review sets), we scored a random sample of records and weighted it back to the full pool. We scored every included study, except in the two largest SWIFT-Review sets (Transgenerational and Neuropain), where we sampled 300 included studies each. We checked that this works. For CLEF 2018 and Chan et al., we ran both a sample and every record. The auto-excluded share agreed within 0.6 points and WSS@95 within 0.2 points. Recall is the same in both by design, because the sample contains every included study.

A person always reviews. SyRev never makes the final include decision. Uncertain records and every candidate include go to a reviewer. If our main screening AI is unavailable, screening pauses and you can try again later; nothing is charged for a failed run.

Results by dataset

Careful setting unless stated. The records column shows records scored, with the full pool they represent in brackets where we sampled.

DatasetWhat it isReviewsRecords scored (pool)Relevant studiesRecallAuto-excludedPrecisionWSS@95Reviews under 95% recall
Chan et al. 2025, manual-search setFull search pools from published systematic reviews, every record22142,49976897.4%77.8%2.4%86.0%2 of 22
CLEF TAR 2019, InterventionAbstract screening of Boolean search results from published systematic reviews198,818 (41,935)93096.3%77.9%9.6%70.2%5 of 19
CLEF TAR 2019, diagnostic, prognosis and qualitativeSearch pools from published systematic reviews, every record (includes one extra intervention topic)1240,48375295.9%73.8%6.8%68.8%2 of 12
CLEF TAR 2018, diagnostic test accuracySearch pools from published systematic reviews, every record30217,4943,96494.9%79.0%8.2%75.1%8 of 30
SWIFT-Review (OHAT and CAMARADES)Environmental health and preclinical, 5 sets57,497 (96,353)85799.7%47.6%11.9%74.2%0 of 5
SYNERGY and Appenzeller-Herzog 2020Medicine and psychology reviews, every record919,03369296.4%65.9%10.3%69.3%2 of 9
Bannach-Brown et al. 2019Preclinical animal models of depression11,99328099.6%10.7%15.7%68.5%—
Bannach-Brown et al. 2019, held-out setThe authors' separate validation set11,251163100%11.0%14.6%81.9%—
CSMeD-FT, title and abstractRecords from published systematic reviews that had already passed abstract screening2113,1401,30197.7%5.6%42.9%28.2%12 of 211
CSMeD-FT, full textThe same records, read in full2113,1801,32399.5%5.1%43.6%32.1%5 of 211

The headline figures combine every title-and-abstract row above: 310 review datasets from 308 distinct reviews. The CSMeD-FT full-text row is a second reading of the same records, so it is reported on its own. 1,078 records had no title or abstract, so they could not be scored and are left out of every figure on this page, including the 442,208 records scored. 1,002 of them were in CLEF 2018. 26 were relevant studies, all in CSMeD-FT (17 reviews), and they are not counted in the 9,707 relevant studies. Counting those 26 as missed would put overall recall at 96.2% instead of 96.5%.

Careful and Rapid compared (relevant studies kept / auto-excluded):

DatasetCareful (default)Rapid
All title-and-abstract datasets96.5% kept91.5% kept
Chan et al. 202597.4% / 77.8%91.9% / 88.1%
CLEF TAR 2019, Intervention96.3% / 77.9%90.9% / 86.8%
CLEF TAR 2019, diagnostic, prognosis and qualitative95.9% / 73.8%87.2% / 85.1%
CLEF TAR 2018, diagnostic test accuracy94.9% / 79.0%88.6% / 87.6%
SWIFT-Review99.7% / 47.6%99.5% / 58.0%
SYNERGY and Appenzeller-Herzog 202096.4% / 65.9%90.3% / 76.6%
Bannach-Brown 201999.6% / 10.7%98.2% / 24.6%
Bannach-Brown 2019, held-out set100% / 11.0%100% / 28.1%
CSMeD-FT, title and abstract97.7% / 5.6%96.4% / 12.7%
CSMeD-FT, full text99.5% / 5.1%98.9% / 10.4%

What the results tell you

In progress

Reproduce the results or ask for the data

SyRev is a screening aid. Final eligibility decisions stay with your review team.