Remove duplicate lines
Paste your list and remove all duplicate lines in one click. The tool keeps the first occurrence of each unique line and discards the rest. Optionally trim whitespace before comparing, sort the result alphabetically, and remove empty lines. All processing runs locally in your browser.
Input
Sorted output
SORT OPTIONS
How it works
The input is split on newlines to produce an array of lines. Trim whitespace applies String.trim() to each line. Remove duplicates uses a Set keyed on the lowercased line to preserve the first occurrence. Sort modes use Array.sort() with localeCompare() for alphabetical ordering or a length comparison for length-based modes. Shuffle uses the Fisher-Yates algorithm with Math.random(). The result array is joined back with newlines. All processing runs in JavaScript in your browser.
Processing runs in your browser
All processing happens locally in your browser tab. Our servers are not involved at any point.
Related operations
For swapping recurring substrings across the list, try find and replace. To compare two sorted lists side by side, use text diff. For checking item counts after deduplication, see the word counter.
Frequently asked questions
- How are duplicates identified?
- Two lines are considered duplicates if they match after optional whitespace trimming. The comparison is case-insensitive, so 'Hello' and 'hello' are treated as the same line and only the first is kept.
- Does the order of unique lines change?
- By default, the output preserves the original order of the first occurrence of each line. If you enable sorting, the unique lines are then sorted alphabetically after deduplication.
- Can I remove nearly-duplicate lines?
- This tool does exact matching only. For fuzzy or partial deduplication (e.g. removing lines that are similar but not identical), use the regex replace tool with a pattern that normalises variations before deduplicating.
- Is my data sent to a server?
- All processing runs locally in your browser. No data is sent to our servers.