Remove Duplicate Lines
Remove duplicate lines from text
About Remove Duplicate Lines
Remove Duplicate Lines keeps only the first copy of each line that appears more than once in a list or text, in the original order. It suits mailing lists, keyword lists, and log extracts pasted one item per line. Three options change the comparison: ignore upper and lower case, ignore spaces around a line, and remove empty lines. Lines are compared after Unicode normalization, so é typed as one character or as e plus an accent match. It compares whole lines, not words, and runs in your browser.
Example
With "Ignore case" and "Ignore surrounding spaces" checked; the page then shows 3 lines removed.
List
apple Banana apple Apple banana cherry
Result
apple Banana cherry
How to remove duplicate lines
- 1Paste your list into the box, one item per line.
- 2Check "Ignore case" when Apple and apple are the same item, "Ignore surrounding spaces" for stray spaces, and "Remove empty lines" to drop blank lines.
- 3Click "Remove duplicates"; the number of lines removed appears above the button.
- 4Copy the cleaned list with "Copy", or keep it as a file with "Download" (deduplicated.txt).
Formats and limits
- Input
- Pasted text; Windows, old Mac, and Unix line breaks are all read
- Kept line
- The first occurrence, exactly as written, with its own spaces and case
- Empty lines
- Compared like other lines, so only the first one stays; Remove empty lines drops them all
- Output
- Kept lines in their original order with Unix line breaks, ready to copy or to save as deduplicated.txt
Where your text is processed
The comparison runs in JavaScript within this page, using a list of seen lines held in memory. Your lines never leave the browser and are forgotten when you close the tab. If copying the result fails, the page shows "Copy failed" and our error log receives the message "Copy to the clipboard failed." with the tool name and browser version, but no line of your list.
Frequently asked questions
Are Apple and apple counted as duplicates?
No, unless you check "Ignore case". With "Ignore surrounding spaces" checked, a line with spaces before or after it also matches the same line without them. In every case the line kept is the first one, as you typed it.
Does it keep the blank lines between paragraphs?
No. Empty lines are compared like any other line, so only the first one survives and later paragraph breaks are removed as duplicates. Check "Remove empty lines" to drop them all; the removed count then includes them.
Can it remove duplicate words inside a line?
No. It compares whole lines only. Two lines holding the same words in another order, or with different punctuation, are both kept. To see how two versions differ word by word, use Compare Text instead.
Does it work with lists copied from Excel or Windows?
Yes. Windows (CRLF), old Mac (CR), and Unix (LF) line endings are all recognized, so the last line of a pasted column matches its copy elsewhere. The result is written with Unix line breaks.
Tips and tricks
- ✓Sort the cleaned list afterward with Sort Text Lines, whose A–Z order puts item 2 before item 10.