Introduction to NLP: Tokenization, Stemming, and Lemmatization

Learn the fundamentals of Natural Language Processing (NLP). Understand text preprocessing, tokenization, stemming, lemmatization, and stop-word removal.

Introduction

Computers excel at reading tables of numbers, but they struggle with human language. Humans communicate using words, idioms, sarcasm, slang, and messy grammar.

Natural Language Processing (NLP) is the branch of Artificial Intelligence that bridges the gap between human language and computer understanding. It enables algorithms to read, decipher, and extract meaning from raw text. However, before an algorithm can analyze text, the raw strings must undergo a rigorous cleaning process called Text Preprocessing.

What You Will Learn

  • The core pipeline of NLP text preprocessing.
  • The differences between Tokenization, Stemming, and Lemmatization.
  • How to filter out useless characters and stop words.
  • How to implement a preprocessing pipeline in Python using nltk (Natural Language Toolkit).

Why This Topic Matters

Raw text is full of noise (HTML tags, punctuation, uppercase/lowercase mismatches, typos, and filler words like "the" and "is"). If you feed raw, messy text directly into a machine learning model, the model will struggle to learn. Proper preprocessing reduces vocabulary size, eliminates noise, and standardizes words, boosting model accuracy significantly.

Prerequisites

Detailed Explanation

The NLP preprocessing pipeline transforms messy human sentences into clean, standardized units. Let's examine the main stages in order:

graph TD
    A[Raw Text] --> B[Lowercase Conversion]
    B --> C[Noise & Punctuation Removal]
    C --> D[Tokenization]
    D --> E[Stop Words Removal]
    E --> F[Stemming or Lemmatization]
    F --> G[Clean Preprocessed Tokens]

1. Lowercase Conversion & Noise Removal

We convert all text to lowercase so that "Running", "running", and "RUNNING" are treated identically. Punctuation (commas, periods, exclamation marks) and numbers are usually stripped out unless they contain critical semantic meaning.

2. Tokenization

Tokenization is the process of splitting a continuous string of text into individual units called Tokens (usually words or subwords).

  • Input: "AI is transforming recruitment."
  • Output: ["AI", "is", "transforming", "recruitment"]

3. Stop Words Removal

Stop words are frequently occurring words that carry very little semantic meaning (e.g., "a", "an", "the", "in", "is", "at"). By filtering these out, we focus purely on the core informational keywords.

  • Input: ["AI", "is", "transforming", "recruitment"]
  • Output: ["AI", "transforming", "recruitment"]

4. Stemming vs. Lemmatization

To group different forms of the same word together, we reduce words to their base form. There are two ways to do this:

A. Stemming

Stemming is a crude, rule-based process that chops off the ends of words. It is fast but often results in non-dictionary words.

  • Algorithm: Porter Stemmer.
  • Example: "running", "runs", "ran" $\rightarrow$ "run".
  • Error: "studies" $\rightarrow$ "studi", "arguing" $\rightarrow$ "argu" (chopped incorrectly).

B. Lemmatization

Lemmatization uses vocabulary and morphological analysis to return the dictionary base form of a word, known as the Lemma. It requires a dictionary database (like WordNet) and takes part-of-speech (POS) tags into account.

  • Example: "better" $\rightarrow$ "good" (Stemming would fail here).
  • Example: "was" $\rightarrow$ "be".

Visual Diagram (Mermaid)

graph LR
    subgraph Stemming vs Lemmatization
        A[studies] -->|Porter Stemmer| B[studi]
        A -->|WordNet Lemmatizer| C[study]
        
        D[worst] -->|Porter Stemmer| E[worst]
        D -->|WordNet Lemmatizer| F[bad]
    end
    style B fill:#F59E0B,stroke:#fff,color:#fff
    style C fill:#10B981,stroke:#fff,color:#fff

Python Code Examples

We will write a complete preprocessing script in Python using the nltk library to clean a sample sentence.

import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer

# Download necessary NLTK datasets (safely handled in production)
nltk.download('punkt', quiet=True)
nltk.download('stopwords', quiet=True)
nltk.download('wordnet', quiet=True)

# 1. Input Raw Text
text = "The AI agents are running, analyzing, and studying recruitment patterns!"
print("Original:", text)

# 2. Lowercase and Tokenize
tokens = word_tokenize(text.lower())
print("Tokens:", tokens)

# 3. Remove Punctuation and Stop Words
stop_words = set(stopwords.words('english'))
filtered_tokens = [w for w in tokens if w.isalnum() and w not in stop_words]
print("Filtered (No stop words/punctuation):", filtered_tokens)

# 4. Apply Stemming
stemmer = PorterStemmer()
stems = [stemmer.stem(w) for w in filtered_tokens]
print("Stems (Stemmed):", stems)

# 5. Apply Lemmatization
lemmatizer = WordNetLemmatizer()
lemmas = [lemmatizer.lemmatize(w, pos='v') for w in filtered_tokens] # 'v' for verb
print("Lemmas (Lemmatized):", lemmas)

Industry Use Cases

  • Search Engines: Google preprocesses queries so that searching for "running shoes" also matches pages containing "runs shoes" or "run shoe".
  • Spam Filtering: Clean and tokenize emails to analyze keyword frequencies (e.g., extracting lemmas like "viagra" or "claim").
  • Customer Feedback Aggregators: Stemming customer reviews to cluster complaints into categories (e.g., grouping "complaining", "complained", "complaints").

Advantages & Limitations

Advantages

  • Reduces Vocabulary Size: Compresses the unique word count, preventing overfitting.
  • Removes Noise: Standardizes text and discards grammatical filler words.

Limitations

  • Loss of Context: Stemming and stop-word removal can destroy sentence structure, which is problematic for advanced contextual models like transformers.
  • Language Dependency: Stemming rules and lemmatizers are language-specific. Rules for English do not translate to languages like Japanese or Arabic.

FAQs

Q: When should I NOT remove stop words? A: You should keep stop words when training deep sequence models (like Transformers or RNNs) for translation or text generation, because removing words like "not" or "him" completely alters the syntactic context and meaning of the sentence.

Q: What is a token? A: A token is the smallest unit of text processed by an NLP model. In basic setups, it is a word, but in modern models (like GPT), subwords or characters are used.

Summary

NLP bridges human communications and algorithmic operations. The foundation of text processing begins with Preprocessing: converting text to lowercase, tokenizing strings into discrete units, removing meaningless stop words, and using Stemming (fast rule-based chopping) or Lemmatization (dictionary-based semantic root lookup) to normalize word forms.

Next Topic

Once text is cleaned into standardized tokens, how do we convert these strings of characters into numbers that mathematical machine learning models can read? Let's check: Text Vectorization: Bag of Words, TF-IDF, and N-grams.

← Back to Syllabus