A practical walkthrough of cleaning messy text online — handling whitespace, line breaks, HTML tags and stray characters.
A step-by-step method for cleaning pasted text: collapse extra spaces, remove line breaks and empty lines, delete duplicates and strip HTML tags safely.
Remove line breaks to rejoin the lines, then remove extra spaces to fix the doubled spacing at the joins. Watch for words hyphenated across the original line breaks.
They differ by case or by a trailing space. Lowercase the list and remove extra spaces first, then de-duplicate.