Productivity
How to Remove Duplicate Lines and Clean Up Messy Lists
Email lists, keyword lists and exported data often repeat the same item many times. Removing the repeats sounds easy. But lines that look the same to you can look different to a computer.
Where hidden duplicates come from
- Capital letters. Apple and apple are different unless you ignore case.
- Stray spaces. A space at the end of a line is invisible but it stops two lines from matching.
- Blank lines. Empty lines repeat too and you usually don't want them.
A process that works
- Paste your list into Remove Extra Spaces to trim line ends and delete empty lines.
- Copy the clean result into the Duplicate Line Remover.
- Decide if Apple and apple count as the same item and set case sensitivity to match.
- Check the counts for original, unique and removed lines. If a number surprises you look at the output before you use it.
Which copy stays?
The first one. Later copies are dropped and the order of your list doesn't change. If you want to keep a certain spelling put it higher up in the list.
When a spreadsheet is better
Sometimes a row is a duplicate only when several columns match. For that use the duplicate feature in your spreadsheet app. Excel has Remove Duplicates and Google Sheets has a data cleanup option. A line tool only compares whole lines.