Validation: how SyRev performs on public systematic-review datasets
We test SyRev on published, openly licensed screening datasets where the right answers are already known. Here is what we found, the reviews where it did less well, and how you can check the results yourself.
Last updated: 15 October 2026
Citable report: doi.org/10.5281/zenodo.23399663. Suggested citation: SyRev Team (2026). SyRev title/abstract screening: validation on 310 public systematic-review datasets (v1.0). Zenodo. https://doi.org/10.5281/zenodo.23399663
Headline numbers (Careful, our default setting)
- 96.5% of relevant studies kept across 310 public review datasets: 9,363 of 9,707 relevant studies were kept and 344 were missed. We scored 442,208 records.
- 74–79% of records auto-excluded in full search pools from published systematic reviews (four benchmarks, 83 reviews), while 94.9–97.4% of relevant studies were kept.
- 69–86% work saved at 95% recall (WSS@95) in those search pools, and 74.2% on environmental-health sets.
- 99.5% of included studies kept when SyRev reads the full text (CSMeD-FT, 211 published systematic reviews at the full-text stage).
- Our lowest results are shown too. In 31 of the 310 review datasets, fewer than 95% of relevant studies were kept. Our weakest benchmark is diagnostic test accuracy (CLEF 2018), at 94.9%. Details are below.
How we test
Datasets. We use public datasets from published systematic reviews. Each record has a known answer: whether the study ended up in the review. The answers come from the review authors or the dataset creators, not from us. Every dataset we report is openly licensed, and we list each licence below.
What SyRev does. For each record, SyRev estimates how likely it is to meet each eligibility criterion, reading the title and abstract, or the full text where noted. Records that are very unlikely to meet the criteria are auto-excluded. Everything else goes to a person. These results come from the same screening model and the same settings that SyRev uses today.
What we measure.
- Recall (relevant studies kept): of the studies that were really included in the review, the share SyRev did not auto-exclude. This is the number that matters most for a systematic review.
- Auto-excluded: the share of all records SyRev removed, so nobody has to screen them.
- Precision: of the records SyRev kept, the share that were really included. Precision is low in large search pools because SyRev keeps anything that might be relevant and leaves the decision to people.
- Work saved at 95% recall (WSS@95): if records are read in SyRev's order until 95% of the relevant studies are found, the share of reading saved compared with screening everything. Ties count against SyRev.
How the thresholds were chosen. SyRev has two fixed settings, and they are the same for every dataset:
- Careful auto-excludes a record only when its estimated chance of inclusion is below 0.1%. It is the default, and all our recall claims use it.
- Rapid uses 0.5%. It saves more work and may miss more.
Not suitable for pharmacovigilance or regulatory use without a separate validation.
We did not adjust either setting for any dataset.
How the criteria were written. For the datasets built from published systematic reviews, the criteria were generated automatically from each review's published selection criteria. For the environmental-health, preclinical and SYNERGY sets, we wrote them by hand from the published descriptions. None were changed after seeing results.
Large pools. For most search pools, we scored every record. In two benchmarks (CLEF 2019 Intervention and the SWIFT-Review sets), we scored a random sample of records and weighted it back to the full pool. We scored every included study, except in the two largest SWIFT-Review sets (Transgenerational and Neuropain), where we sampled 300 included studies each. We checked that this works. For CLEF 2018 and Chan et al., we ran both a sample and every record. The auto-excluded share agreed within 0.6 points and WSS@95 within 0.2 points. Recall is the same in both by design, because the sample contains every included study.
A person always reviews. SyRev never makes the final include decision. Uncertain records and every candidate include go to a reviewer. If our main screening AI is unavailable, screening pauses and you can try again later; nothing is charged for a failed run.
Results by dataset
Careful setting unless stated. The records column shows records scored, with the full pool they represent in brackets where we sampled.
| Dataset | What it is | Reviews | Records scored (pool) | Relevant studies | Recall | Auto-excluded | Precision | WSS@95 | Reviews under 95% recall |
|---|---|---|---|---|---|---|---|---|---|
| Chan et al. 2025, manual-search set | Full search pools from published systematic reviews, every record | 22 | 142,499 | 768 | 97.4% | 77.8% | 2.4% | 86.0% | 2 of 22 |
| CLEF TAR 2019, Intervention | Abstract screening of Boolean search results from published systematic reviews | 19 | 8,818 (41,935) | 930 | 96.3% | 77.9% | 9.6% | 70.2% | 5 of 19 |
| CLEF TAR 2019, diagnostic, prognosis and qualitative | Search pools from published systematic reviews, every record (includes one extra intervention topic) | 12 | 40,483 | 752 | 95.9% | 73.8% | 6.8% | 68.8% | 2 of 12 |
| CLEF TAR 2018, diagnostic test accuracy | Search pools from published systematic reviews, every record | 30 | 217,494 | 3,964 | 94.9% | 79.0% | 8.2% | 75.1% | 8 of 30 |
| SWIFT-Review (OHAT and CAMARADES) | Environmental health and preclinical, 5 sets | 5 | 7,497 (96,353) | 857 | 99.7% | 47.6% | 11.9% | 74.2% | 0 of 5 |
| SYNERGY and Appenzeller-Herzog 2020 | Medicine and psychology reviews, every record | 9 | 19,033 | 692 | 96.4% | 65.9% | 10.3% | 69.3% | 2 of 9 |
| Bannach-Brown et al. 2019 | Preclinical animal models of depression | 1 | 1,993 | 280 | 99.6% | 10.7% | 15.7% | 68.5% | — |
| Bannach-Brown et al. 2019, held-out set | The authors' separate validation set | 1 | 1,251 | 163 | 100% | 11.0% | 14.6% | 81.9% | — |
| CSMeD-FT, title and abstract | Records from published systematic reviews that had already passed abstract screening | 211 | 3,140 | 1,301 | 97.7% | 5.6% | 42.9% | 28.2% | 12 of 211 |
| CSMeD-FT, full text | The same records, read in full | 211 | 3,180 | 1,323 | 99.5% | 5.1% | 43.6% | 32.1% | 5 of 211 |
The headline figures combine every title-and-abstract row above: 310 review datasets from 308 distinct reviews. The CSMeD-FT full-text row is a second reading of the same records, so it is reported on its own. 1,078 records had no title or abstract, so they could not be scored and are left out of every figure on this page, including the 442,208 records scored. 1,002 of them were in CLEF 2018. 26 were relevant studies, all in CSMeD-FT (17 reviews), and they are not counted in the 9,707 relevant studies. Counting those 26 as missed would put overall recall at 96.2% instead of 96.5%.
Careful and Rapid compared (relevant studies kept / auto-excluded):
| Dataset | Careful (default) | Rapid |
|---|---|---|
| All title-and-abstract datasets | 96.5% kept | 91.5% kept |
| Chan et al. 2025 | 97.4% / 77.8% | 91.9% / 88.1% |
| CLEF TAR 2019, Intervention | 96.3% / 77.9% | 90.9% / 86.8% |
| CLEF TAR 2019, diagnostic, prognosis and qualitative | 95.9% / 73.8% | 87.2% / 85.1% |
| CLEF TAR 2018, diagnostic test accuracy | 94.9% / 79.0% | 88.6% / 87.6% |
| SWIFT-Review | 99.7% / 47.6% | 99.5% / 58.0% |
| SYNERGY and Appenzeller-Herzog 2020 | 96.4% / 65.9% | 90.3% / 76.6% |
| Bannach-Brown 2019 | 99.6% / 10.7% | 98.2% / 24.6% |
| Bannach-Brown 2019, held-out set | 100% / 11.0% | 100% / 28.1% |
| CSMeD-FT, title and abstract | 97.7% / 5.6% | 96.4% / 12.7% |
| CSMeD-FT, full text | 99.5% / 5.1% | 98.9% / 10.4% |
What the results tell you
- Savings depend on the review, and we show the full range. In full search pools, about three-quarters of records leave the queue. Where records have already passed abstract screening, as in CSMeD-FT, only about 5–6% leave, because most of the pile is genuinely relevant. At that stage, SyRev's value is confirming includes rather than cutting volume.
- Averages hide spread, so we list the lowest reviews. At Careful, fewer than 95% of relevant studies were kept in 31 of 310 review datasets (29 of the 166 with at least 5 relevant studies). Examples:
- One CSMeD-FT review on asthma kept 3 of 9.
- One CLEF topic on platelet transfusion kept 3 of 6.
- One CLEF 2018 review on rapid tests for malaria kept 186 of 297.
- One Chan review on testosterone kept 44 of 52.
- Clear criteria give the best results. Most of the lowest results came from how the criteria were worded, not from the model misreading abstracts.
- The malaria review was limited to non-falciparum malaria. Most of the missed studies (79 of 111) were about falciparum malaria only, which the written criteria exclude.
- In the testosterone review, the published criteria excluded men with hypogonadism, yet most included trials were in late-onset hypogonadism. SyRev followed the written rule.
- Writing specific criteria, and checking them on a few known includes first, makes the biggest difference.
- Benchmarks have errors too. In the asthma review above, most of the "missed" studies have titles unrelated to the review. They look like record-linkage errors in the dataset.
- Our weakest benchmark is CLEF 2018 (diagnostic test accuracy), at 94.9%. The malaria review above accounts for 111 of its 201 missed studies. Without that review, the benchmark's recall would be 97.5%. In CLEF 2019, the diagnostic reviews reached 96.6%, and the one prognosis review kept 176 of 192.
- Careful protects recall, and Rapid trades some of it for speed. On the full search pools, Rapid kept 87–92% of relevant studies against 95–97% at Careful. That is why Careful is the default.
- Labels are often final-review decisions. SyRev scores titles and abstracts, while many gold labels reflect full-text decisions. This makes the test harder than normal abstract screening, so perfect recall is never guaranteed.
- The best evidence is your own review. Run a review you have already finished to see recall and hours saved on your topic. Then use the validation pack's blind sample to check SyRev's exclusions.
In progress
- Monitor (living reviews): validation on published systematic review updates.
- Data extraction: newly launched; public accuracy validation to follow.
Reproduce the results or ask for the data
- Datasets: all are public. Links and licences:
- CLEF TAR 2018 and 2019: github.com/CLEF-TAR/tar (MIT).
- CSMeD and CSMeD-FT: github.com/WojciechKusa/systematic-review-datasets (Apache-2.0 code; CSMeD-FT data CC BY 4.0).
- SWIFT-Review sets: Howard et al. 2016, doi.org/10.1186/s13643-016-0263-z (via CSMeD).
- SYNERGY: github.com/asreview/synergy-dataset (CC0).
- Appenzeller-Herzog 2020: doi.org/10.5281/zenodo.3625931 (CC BY 4.0).
- Bannach-Brown et al.: records at zenodo.org/records/151190 (CC BY 4.0). Held-out labels at github.com/abannachbrown/The-use-of-text-mining-and-machine-learning-algorithms-in-systematic-reviews (GPL-3.0).
- Chan et al. 2025: data.mendeley.com/datasets/7sgmg89zb6/1 (CC BY 4.0).
- What we can share: the score for every record, the per-review results, the criteria we used for each review, the sampling seeds and the scoring scripts. The model version is recorded with each result.
- Ask us: email privacy@syrev.ai and tell us which dataset you want to check.
SyRev is a screening aid. Final eligibility decisions stay with your review team.