Remove Duplicate Lines from a List
Paste a list and get it back with duplicates gone, order kept, counts shown, case and spacing handled sensibly.
What was removed
- Lines in
- –
- Unique out
- –
Free forever, no sign-up and no limits, and this tool installs on its own so you can keep it on your home screen.
Paste a list, get it back with every repeat removed, first occurrence kept, original order untouched, and the number of duplicates found shown as the headline. Case and stray spaces are treated as the same line unless you say otherwise.
How to use it
- Paste the listEmails, order numbers, keywords, names, one per line.
- Check the togglesIgnore case and Ignore spaces are on, so 'Anna ' and 'anna' count as one. Turn either off for strict matching.
- Copy the clean listThe unique lines, in the order they first appeared.
Where duplicates come from
Nobody types a list with duplicates; they merge their way in. Two exports of overlapping date ranges. A mailing list combined with last year's. Keyword research from three tools that mostly agree. The overlap is invisible in a thousand-line paste, and the cost is real, double emails sent, inflated counts reported, the same work done twice.
The headline number here is the duplicates found, not the survivors, because that is the diagnostic: paste a "clean" list and a big number tells you the upstream process is merging something it should not. The report spells it out, removed N duplicates from M lines, and the expandable panel below it lists each removed line with its occurrence count so the cleanup is auditable.
Forgiving matching, faithful output
Strict equality is the wrong default for pasted data. The same email arrives as Name@Example.com from one export and name@example.com from another; spreadsheet copies carry trailing spaces that no eye will ever see. The default toggles treat those as the duplicates they really are, while the line that survives is your original, byte for byte. Matching is normalised; your data is not rewritten.
When the difference is the data, case-sensitive codes, IDs where trailing space is meaningful, both toggles switch off and matching becomes exact.
Order is information
Plenty of dedupe methods sort as a side effect, which quietly destroys whatever the order meant: priority, chronology, the sequence a customer signed up in. This tool never reorders: with Keep first, the first appearance wins its place and keeps it, and with Keep last, the final appearance does, either way in its own original position. Deduplication and sorting are different decisions, and you can take the result next door to sort-lines when you want both.
Questions people ask
Which copy of a duplicate is kept?
Your choice: the Keep control switches between the first occurrence (the default) and the last. Either way the surviving line keeps its own position and the list order is otherwise unchanged. Keep last is the one you want when later entries carry corrections, as in a re-imported export. Nothing is sorted behind your back; for sorting too, the sort-lines tool has a dedupe option built in.
Can I see what was removed?
Yes. The What was removed panel under the headline lists every line that appeared more than once, with how many times it occurred, so you can audit the cleanup instead of trusting it. A line shown with ×3 appeared three times and lost two copies.
Are 'Anna' and 'anna ' really duplicates?
With the default toggles, yes: case and surrounding spaces are ignored for matching, because in pasted lists those differences are almost always noise from different sources. The kept line is still your original text, matching is forgiving, output is untouched.
Why did my count of removed lines seem high?
Usually invisible whitespace: lines that differ only by a trailing space or a tab collapse into one when Ignore spaces is on. That is normally the correct outcome, turn the toggle off to see the strict count.
What happens to blank lines?
Dropped. A deduplicated list with holes in it is rarely what anyone wants, and blank lines in pasted data are almost always copying artefacts.
Can I dedupe a comma-separated list?
Put each item on its own line first, paste into the line-breaks tool and split there, or use your editor's find-and-replace to turn commas into newlines. This tool deliberately works line-wise so that what counts as an item is never ambiguous.
How big a list can it handle?
Tens of thousands of lines without noticing. Matching uses a hash set, so doubling the list barely changes how long it takes, and the answer still arrives as fast as you can paste.