0 lines
·
0 KB
Remove duplicate lines, online
Paste any list. Toggle four rules. Get a clean output that keeps the order, case, and whitespace you actually wanted. Built for data teams that ship.
Paste something on the left — results stream in here.
Or press Ctrl+Enter
Input lines 0
Blank lines 0
Duplicate groups 0
Removed 0
Unique remaining0
What people clean here
04 — use casesClean exports from CRM, Mailchimp, or HubSpot before importing — without nuking your column order.
Audit huge keyword sheets, find duplicates across campaigns before launch.
Strip duplicates from CSV / TSV before INSERT — case-insensitive, whitespace-aware, up to 80 MB.
Collapse repeated stack traces and noisy log lines into a clean, scannable list.
How it works
three steps-
01
Paste your list or upload a .txt
Live preview up to 100K lines on desktop / 5K on mobile. Larger inputs switch to big-file mode automatically — same engine, the page stays responsive and delivers a .txt download.
-
02
Toggle the four rules
Case: treat ABC and abc as equal or not. Trim: ignore leading/trailing whitespace when comparing. Empty: drop blank lines. Shuffle: randomize the output after dedup.
-
03
Hit Run
Duplicates collapse to a single row; the first occurrence is kept. The output card shows the cleaned list in its original order.
Under the hood
engine notes- Algorithm
- O(n) hash-table dedup · Object.create(null) prototype-free map
- Threshold split
- Desktop: 2 MB / 100K lines · Mobile: 300 KB / 5K lines · Hard cap: 80 MB
- Order preservation
- First-occurrence index recorded in the dedup pass
Total complexity O(n + u log u + s log s) where n = input lines, u = unique lines, s = output length. The earlier version called Object.keys(freqMap).length inside the per-line loop — O(n²), made 156K lines take 5 minutes. The current implementation does ~500 ms for the same input.
Above the live threshold the input never enters the textarea (browser native word-wrap is O(n) on insert) — processing moves to a Web Worker, the page stays interactive, and you get a .txt download.
Excel's "Remove Duplicates" and Google Sheets' UNIQUE() both re-sort columns under the hood — we keep the original order by default. Toggle Shuffle if you want a Fisher-Yates randomized output instead.
Privacy & limits
what stays whereEverything runs in your browser. The list you paste is never uploaded — the page makes zero fetch calls with your data, and the analytics layer cannot see what you typed. After page load you can disconnect from the network and the tool still works. 80 MB hard cap per run protects against accidentally locking the tab on a multi-gigabyte log.
Frequently asked
answeredDoes case-insensitive dedup ignore case in the output too?
No. The case toggle only affects comparison — "Apple" and "APPLE" are treated as duplicates and only the first one is kept, but the kept row preserves its original capitalization. Output is never lowercased.
Can I keep the LAST occurrence of each duplicate instead of the first?
Yes — chain two tools. Run the input through the Reverse Lines tool first, then paste the reversed list into the deduper. You can reverse the final output one more time if you need the original direction back. Both passes are local, no upload.
How is this different from Excel's Remove Duplicates?
Three differences: (1) we run in the browser — paste from anywhere without opening Excel, (2) we keep the original case and whitespace exactly, while Excel sometimes auto-normalizes cells, and (3) Excel's Remove Duplicates often re-sorts the column under the hood; we keep the original order by default.
Does it handle CSV with quoted fields containing commas?
It treats each line as a single string for comparison — quoted fields containing commas are matched as part of the whole line. If you need column-level dedup ("dedupe rows where column B matches"), this tool isn't the right one; use Google Sheets UNIQUE() or a Power Query step instead.
What's the file size limit?
80 MB hard cap per run. Above 2 MB on desktop (300 KB on mobile) the tool switches to big-file mode — your text doesn't go through the textarea, processing happens in a Web Worker, and the result downloads as .txt instead of rendering in the output card.
Is my data sent anywhere?
No. The whole pipeline — paste, dedupe, render — lives in this browser tab. There is no upload step, and the telemetry layer never sees what you pasted. Disconnect from Wi-Fi after the page loads and the tool still works.