Real-world text is messy

Cleaning messy text step by step

1<p>I LOVE this!!! 
2https://ex.com 
3a@b.com</p>

1import re
2raw = "<p>I LOVE this!!! https://ex.com a@b.com</p>"
3
4text = re.sub(r'<.*?>', '', raw)       # Step 1
5text = re.sub(r'http\S+', '', text)    # Step 2
6text = re.sub(r'\S+@\S+', '', text)
7text = ' '.join(text.split())          # Step 3
8
9print(text)  # "I LOVE this!!!"

Worked example: Complete pipeline

Full transformation:

Key decision points:

• Keep emoji ? Yes for sentiment analysis (conveys emotion)
• Keep punctuation !!!? Maybe (emphasis, but noisy)
• Lowercase? Depends on task (lose "LOVE" emphasis)

Different tasks need different preprocessing

The web is a massive source of text data

Web scraping with Beautiful Soup

Basic example:

1from bs4 import BeautifulSoup
2import requests
3
4# Fetch webpage
5url = "https://example.com/article"
6response = requests.get(url)
7html_content = response.content
8
9# Parse HTML
10soup = BeautifulSoup(html_content, 'html.parser')
11
12# Extract text from specific elements
13title = soup.find('h1').get_text()
14article = soup.find('article').get_text()
15clean_text = ' '.join(article.split())  # Remove extra whitespace
16print(f"Title: {title}\nArticle: {clean_text[:200]}...")

Advanced Beautiful Soup techniques

1from bs4 import BeautifulSoup
2
3html = """<div class="article">
4 <h2>Breaking News</h2>
5 <p class="content">First paragraph.</p>
6 <p class="content">Second paragraph.</p>
7 <div class="ads">Advertisement</div>
8</div>"""
9
10soup = BeautifulSoup(html, 'html.parser')
11
12paragraphs = soup.find_all('p', class_='content')  # Find content paragraphs
13text = ' '.join([p.get_text() for p in paragraphs])
14
15for ad in soup.find_all('div', class_='ads'):      # Remove unwanted sections
16    ad.decompose()
17
18article_text = soup.get_text(separator=' ', strip=True)
19print(article_text)

Web scraping often produces mixed encodings

1import chardet
2
3raw_bytes = b'Caf\xe9 au lait \x96 delicious!'  # Unknown encoding
4
5result = chardet.detect(raw_bytes)              # Detect encoding
6print(f"Detected: {result}")  # {'encoding': 'Windows-1252', 'confidence': 0.73}
7
8text = raw_bytes.decode(result['encoding'])     # Decode with detected encoding
9print(text)                   # "Café au lait – delicious!"
10
11import ftfy                                     # Alternative: fix broken Unicode
12broken = "Cafà au lait €" delicious!"
13fixed = ftfy.fix_text(broken)
14print(fixed)                  # "Café au lait – delicious!"

Pro tip: Save all files as UTF-8 to avoid these headaches!

Five common preprocessing steps

Preprocessing example

1import re
2
3def preprocess_text(text):
4    text = re.sub(r'http\S+|www.\S+', '', text)  # Remove URLs
5    text = re.sub(r'\S+@\S+', '', text)          # Remove emails
6    text = re.sub(r'<.*?>', '', text)            # Remove HTML tags
7    text = ' '.join(text.split())                # Remove extra whitespace
8    text = text.lower()                          # Lowercase
9    return text
10
11raw = "Check out https://example.com! <b>Amazing</b> deals!!"
12print(preprocess_text(raw))  # "check out amazing deals!!"

Stemming is fast but crude; lemmatization is accurate

Stemming with NLTK

1from nltk.stem import PorterStemmer, SnowballStemmer
2porter = PorterStemmer()
3snowball = SnowballStemmer('english')
4
5words = ['running', 'runs', 'runner', 'ran', 'easily', 'fairly']
6
7print("Word         Porter       Snowball")
8for word in words:
9    print(f"{word:12} {porter.stem(word):12} {snowball.stem(word)}")
10
11# Output:
12# running      run          run
13# runs         run          run
14# runner       runner       runner
15# ran          ran          ran
16# easily       easili       easili
17# fairly       fairli       fair

Lemmatization with spaCy

1import spacy
2nlp = spacy.load("en_core_web_sm")
3
4text = "The cats are running faster than dogs ran yesterday"
5doc = nlp(text)
6
7print(f"{'Word':<12} {'Lemma':<12} {'POS'}")
8print("-" * 36)
9for token in doc:
10    print(f"{token.text:<12} {token.lemma_:<12} {token.pos_}")
11
12# Output:
13# Word         Lemma        POS
14# cats         cat          NOUN
15# running      run          VERB
16# dogs         dog          NOUN
17# ran          run          VERB

Stop words: common words with little semantic value

Practical tips for data cleaning

Complete preprocessing pipeline example

1import spacy, re
2
3class TextPreprocessor:
4    def __init__(self, remove_stopwords=False):
5        self.nlp = spacy.load("en_core_web_sm")
6        self.remove_stopwords = remove_stopwords
7
8    def clean(self, text):
9        text = re.sub(r'http\S+', '', text)   # Remove URLs
10        text = re.sub(r'<.*?>', '', text)     # Remove HTML
11        text = ' '.join(text.split())         # Normalize whitespace
12        doc = self.nlp(text)
13        tokens = [t.lemma_ for t in doc if not (self.remove_stopwords and t.is_stop)]
14        return ' '.join(tokens)

Preprocessing checklist

Tools and libraries

Primary references

Key takeaways

1. Preprocessing is crucial but task-dependent — No universal pipeline; adapt to your needs!
2. Balance cleaning vs. information loss — More preprocessing does not always mean better
3. Stemming vs. Lemmatization — Stemming: fast, approximate; Lemmatization: slow, accurate
4. Modern models are robust — Transformers can handle messy text better than older models
5. Always validate — Inspect data before and after preprocessing