Home / Free Utility Tools / Free Remove Duplicate Lines

Free Remove Duplicate Lines

Remove repeated lines from a list, keeping the first copy. Optionally ignore case, trim spaces and sort the result.

Duplicates that look different, and duplicates that do not

Removing exact duplicates is easy. The interesting part of cleaning a list is deciding what counts as the same. anna@example.com and Anna@Example.com are the same address to every mail system, but not the same string of characters. example.com/page and example.com/page/ are one page to you and two lines to a computer. This tool lets you decide, with options for case, spaces and empty lines, and then keeps one copy of each line you consider identical.

A worked example

Here is a list pasted from three sources:

anna@example.com
Ben@Example.com
anna@example.com 
ben@example.com

carla@example.com

With case sensitivity on and trimming off, nothing is removed, because every line differs in some small way. Untick Case sensitive, keep Trim spaces ticked and tick Remove empty lines, and the result is three clean lines: anna, the first Ben and carla. The trailing space and the capital letters no longer matter, and the blank line is gone.

What each option does

Case sensitive. When off, upper and lower case letters are treated as identical.
Trim spaces around lines. Ignores spaces at the start and end of each line, which are a common hidden cause of apparent differences.
Remove empty lines. Deletes blank lines from the result.
Sort. Keep the original order, or sort from A to Z or Z to A.
Keep first or last occurrence. Decides which copy survives, which matters for order and exact spelling.

Choosing first or last

If your list is built up over time and you want to preserve the order in which things were first added, keep the first occurrence. If newer lines are added at the end and a later version is more accurate, for example a corrected spelling or a newer record, keep the last occurrence. When the lines are identical it makes no difference, but when they differ only in case or spacing, you are choosing which version you want to keep.

Everyday jobs it is good for

Email and contact lists merged from several sources before importing them into a mailing tool.
Keyword lists collected from multiple research tools.
URL lists, for example before building a sitemap with the XML sitemap generator.
Log files or error messages, to see the distinct ones.
Student names, product codes or tags copied from several spreadsheets.

Near-duplicates need your eyes

The tool removes lines that match exactly under your settings. It cannot know that Smith, John and John Smith are one person, or that a web address with www and one without point to the same page. After removing duplicates, sort the list and scan it, because near-duplicates end up next to each other and are easy to spot. For URLs, make the formats consistent first, with the same scheme, the same www choice and the same trailing slash habit.

A tidy workflow for keyword research

Export keyword ideas from three different tools, paste all three lists into the box, turn off Case sensitive, trim spaces and remove empty lines, and sort A to Z. Duplicates collapse, and near-variants line up beside each other for easy scanning. Copy the clean list into a spreadsheet, add volume data and decide which to target. A messy hour becomes five minutes.

Cleaning a mailing list

Before you import a contact list into a newsletter tool, lowercase it with the case converter, then remove duplicates here. This avoids sending the same person two copies and keeps your subscriber count honest. Check for obviously invalid entries by eye afterwards, since the tool does not verify addresses.

What about very large lists?

Tens of thousands of lines are usually fine in a modern browser. For millions, use a spreadsheet or a command line tool, since browsers have memory limits. The tool works entirely on your device, so nothing is uploaded, which is useful for private data.

Comparing instead of cleaning

If you want to see how two lists differ, instead of cleaning one, use the text diff checker. To tidy the case of the lines before you remove duplicates, the case converter can make everything lowercase first, which makes matching simpler. Your lists stay in your browser, so personal data such as email addresses is never uploaded.