Deduplicating multi-database search exports

Citation managers are for the library you live in. This is the step before that: you have three RIS files, they all claim to be the same review, and you still do not know how many unique records you actually retrieved.

That number is the first box in the PRISMA 2020 identification flow. If you mash the files together and click “remove duplicates,” you can lose a trial or count the same paper twice. Both errors show up later as a screening total nobody can defend.

This is not a tour of EndNote. It is the merge order I use on multi-database exports, and how those two failure modes change the counts you report.

Keep the piles labelled

Export each database into its own file. Name it so a stranger can reconstruct the search: PubMed_20260903_protocol-v3.ris, not final2.ris.

Do not concatenate until every row still knows its source. When a merge is wrong, you need to know whether the extra copy came from Embase or from CENTRAL. A single flattened library cannot tell you that.

Typical shape (made-up numbers, useful as a worked example):

• PubMed 1,842
• Embase 2,410
• CENTRAL 310
• other 95
• identified 4,657

Those 4,657 records are not 4,657 studies. They are rows.

Match in this order, then stop auto-merging

Run exact keys first. Fuzzy matching last. Human review on anything that is close rather than identical.

1. Exact DOI. Normalise https://doi.org/10.…, doi:, and case. If the DOI is the same, it is the same citable object. Auto-merge.
2. PMID and other database IDs. Same PMID, Embase accession, or trial ID (NCT, ISRCTN) is a merge. Conflicting IDs on the same title are not a merge. They are a flag.
3. Normalised title + year + first author. Strip punctuation, case, and trailing “a randomised trial.” If all three agree, auto-merge only when no ID conflict exists. If the title is close but the year or first author disagrees, do not auto-merge.
4. Residuals. Everything left is human review. Title-only matching is how you fuse two different 2022 insulin papers.

Software that only does DOI then title+year will still leave PMID-only duplicates as splits. That is expected. Treat those as residuals, not as unique studies.

False merge versus false split

False merge (too aggressive). Two trials share a similar title and year. The tool keeps one. Your “records screened” number drops. A real study never reaches full text. The review looks cleaner than it was.

False split (too timid). The same paper is in PubMed with a PMID and in Embase with no DOI. You screen it twice. Identification looks inflated. Disagreement between reviewers is often the same record in two jackets.

I would rather have a slightly larger residual pile than a silent merge. You can still collapse a residual. You cannot un-delete a trial you never screened.

Write the PRISMA numbers from the audit, not from a toast message

For the example above, suppose the audit says 1,218 rows were removed as duplicates. Then:

• Records identified: 4,657
• Duplicates removed: 1,218
• Records screened: 3,439

Those three numbers have to come from the same log: which rule fired, which record was kept, which source files were in the group. If your tool cannot export that, keep the groups in a spreadsheet before you empty the recycle bin.

Do not reuse the “identified” total as the screening denominator. Do not hide database-specific duplicates inside “other reasons” at screening.

What I actually run

I keep the labelled exports, run an automated pass for DOI (and title+year when there is no DOI), then inspect the residual groups by hand. SRT’s deduplication tool does that first pass and writes a group audit (kept row, match method, source file). It does not replace PMID matching or residual review. If a group has two valid DOIs, do not let title+year glue them together.

If you already live in EndNote or Zotero, the order does not change. The software is just the place you click. The PRISMA box still needs the audit.

When you are done

You should be able to answer, from files, not from memory: how many rows each database produced, which rule removed each duplicate, and which residuals you judged by hand. That is the whole job.