Guide
How to check for retracted studies in a systematic review
Retracted papers keep being cited, and some end up inside systematic reviews and meta-analyses. Here is a practical routine for catching them at each stage, and how to report what you checked.
Why it matters
Studies of the literature show that retracted trials continue to be cited after retraction, including by systematic reviews and guidelines [3]. Re-analyses suggest that removing retracted studies can change the results of some meta-analyses, though in many cases the effect is small [2]. Bakker and colleagues set out practical steps for review teams, journals and database providers to reduce this "residue" of retracted data in evidence synthesis [1].
When to check
| Stage | What to do |
|---|---|
| After import | Flag retracted records and expressions of concern before screening starts, so reviewers see them in context. |
| Included studies | Check every included study again before data extraction and analysis. |
| Before submission | Re-run the check shortly before you submit, and again at the proofs stage if there's a long delay. Retractions can happen at any time. |
| Updates | For living or updated reviews, re-check earlier includes at each update. |
How to check
- Use identifiers, not titles. Collect DOIs and PMIDs for every record. Matching by identifier is far more reliable than title searching.
- Check the Retraction Watch database through Crossref. Crossref acquired the Retraction Watch database in 2023 and makes the data openly available [4], so a DOI lookup covers retractions recorded by both publishers and Retraction Watch.
- Check PubMed. PubMed labels retracted articles and links them to the retraction notice. For a set of PMIDs, you can search them together and filter by the "Retracted Publication" publication type.
- Use your reference manager. Zotero, for example, flags items that appear in the Retraction Watch data [5]. This is a useful second net, but it only covers items stored in your library.
- Look beyond retractions. Record expressions of concern and major corrections too. They don't automatically exclude a study, but your team should decide in advance how to handle them.
- Decide and document. Exclude retracted studies from the analysis (or report a sensitivity analysis if you keep any data for a stated reason), and record the date and sources of each check.
Example methods sentence
PRISMA 2020 asks you to describe the methods used to decide which studies were included [6]. Stating what was checked for retraction, and when, makes that decision auditable.
Common mistakes
- Checking only at the search stage and not before submission.
- Relying on the publisher's PDF you downloaded months ago, which may not carry the retraction notice.
- Ignoring expressions of concern.
- Not recording the date of the check.
Where SyRev fits
When records are imported into a SyRev review or a Monitor update, SyRev looks up each record's DOI in Crossref (which includes the Retraction Watch database) and its PMID in PubMed. It then flags retracted studies and expressions of concern in the screening queue. You can re-check a review at any time, retraction status is included in exports and the audit pack, and SyRev drafts a methods sentence stating what was checked and when. Only DOIs and PMIDs are sent for the check. This is available on every plan.
References
- Bakker CJ, Boughton SL, Faggion CM, Fanelli D, Kaiser K, Schneider J. Reducing the residue of retractions in evidence synthesis: ways to minimise inappropriate citation and use of retracted data. BMJ Evid Based Med 2024;29(2):121–126. doi:10.1136/bmjebm-2022-111921
- Fanelli D, Wong J, Moher D. What difference might retractions make? An estimate of the potential epistemic cost of retractions on meta-analyses. Account Res 2022;29(7):442–459. doi:10.1080/08989621.2021.1947810
- Avenell A, Stewart F, Grey A, Gamble G, Bolland M. An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature. BMJ Open 2019;9(10):e031909. doi:10.1136/bmjopen-2019-031909
- Crossref. News: Crossref and Retraction Watch (blog, 2023). crossref.org/blog/news-crossref-and-retraction-watch
- Zotero. Retracted item notifications (blog). zotero.org/blog/retracted-item-notifications
- Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi:10.1136/bmj.n71