Text Vectorization: Bag of Words, TF-IDF, and N-grams

Master Text Vectorization. Learn how to convert text to numbers using Bag of Words (BoW), Term Frequency-Inverse Document Frequency (TF-IDF), and N-grams.

Introduction

If you want to train a Decision Tree, an SVM, or a Neural Network, you must feed it numbers. You cannot pass the string "recruitment" to a mathematical equation.

Text Vectorization is the process of converting textual information into numerical vectors that machine learning algorithms can ingest. How we perform this conversion determines whether the model understands which words are important, or if it treats all words identically.

What You Will Learn

  • How Bag of Words (BoW) represents text as count vectors.
  • How N-grams preserve basic word order.
  • The mathematics behind the TF-IDF (Term Frequency-Inverse Document Frequency) formula.
  • How to implement vectorizers in Python using scikit-learn.

Why This Topic Matters

Simple count vectorizers fail because they treat common words (like "the" or "and") as important just because they appear frequently. TF-IDF fixes this by downscaling words that appear across all documents, highlighting the rare, high-information terms. It is the core algorithm behind document search engines, text classifiers, and recommendation algorithms.

Prerequisites

Detailed Explanation

Let's explore the three foundational numerical representations of text.

1. Bag of Words (BoW)

The Bag of Words model counts how many times each word appears in a document.

  1. Build Vocabulary: Create a list of all unique words across all documents.
  2. Count Frequencies: For each document, count occurrences of each word in the vocabulary, yielding a vector.

Example:

  • Document 1: "AI is great."
  • Document 2: "Recruitment is great."
  • Vocabulary: ["AI", "is", "great", "recruitment"]
  • Doc 1 Vector: [1, 1, 1, 0]
  • Doc 2 Vector: [0, 1, 1, 1]

Limitation: BoW completely ignores word order. "Not good, very bad" and "very good, not bad" would yield identical vectors.

2. N-grams

To capture word order, we group adjacent words into sequences of size $N$:

  • Unigram ($N=1$): ["AI", "is", "great"]
  • Bigram ($N=2$): ["AI is", "is great"]
  • Trigram ($N=3$): ["AI is great"]

By using bigrams and trigrams, our vectorizer preserves basic context (e.g., separating "not happy" from "happy").

3. TF-IDF (Term Frequency-Inverse Document Frequency)

Instead of counting raw occurrences, TF-IDF weights words based on their importance:

$$\text{TF-IDF}(t, d, D) = \text{TF}(t, d) \times \text{IDF}(t, D)$$

A. Term Frequency (TF)

Measures how frequently a term $t$ appears in a document $d$: $$\text{TF}(t, d) = \frac{\text{Count of } t \text{ in } d}{\text{Total words in } d}$$

B. Inverse Document Frequency (IDF)

Measures how common or rare a term is across the entire corpus of documents $D$: $$\text{IDF}(t, D) = \log \left( \frac{\text{Total number of documents } |D|}{\text{Number of documents containing } t + 1} \right)$$ (We add 1 to the denominator to avoid division by zero if a word isn't in the corpus).

  • If a word appears in every document (like "the"), $\frac{|D|}{|{d \in D : t \in d}|} \approx 1$. Since $\log(1) = 0$, the word's IDF weight becomes $0$.
  • If a word is rare (like "quantum"), the numerator is large, the denominator is small, resulting in a high IDF value.

Visual Diagram (Mermaid)

graph TD
    A[Text Documents] --> B[TF-IDF Vectorizer]
    B --> C[Compute Term Frequency: Local Importance]
    B --> D[Compute Inverse Document Frequency: Global Rarity]
    C & D --> E[Multiply TF * IDF]
    E --> F[Output Vector: Numeric Matrix representation]

Python Code Examples

We will implement both Bag of Words (CountVectorizer) and TF-IDF in Python using scikit-learn.

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

# Sample corpus
documents = [
    "AI is transforming the recruitment process.",
    "Recruitment software relies on AI algorithms.",
    "The weather is nice today."
]

# 1. Bag of Words (using CountVectorizer)
count_vectorizer = CountVectorizer()
bow_matrix = count_vectorizer.fit_transform(documents)

print("Vocabulary:", count_vectorizer.get_feature_names_out())
print("BoW Representation for Document 1:")
print(bow_matrix.toarray()[0])

# 2. TF-IDF Vectorizer (filtering stop words, adding bigrams)
tfidf_vectorizer = TfidfVectorizer(stop_words='english', ngram_range=(1, 2))
tfidf_matrix = tfidf_vectorizer.fit_transform(documents)

print("\nTF-IDF Features:", tfidf_vectorizer.get_feature_names_out())
print("TF-IDF Vector for Document 1 (showing non-zero weights):")
# Mapping features to weights
for feature, weight in zip(tfidf_vectorizer.get_feature_names_out(), tfidf_matrix.toarray()[0]):
    if weight > 0:
        print(f" - {feature}: {round(weight, 4)}")

Industry Use Cases

  • Search Engine Retrieval: Matching search queries with web pages. Pages containing high-weight query terms (calculated via TF-IDF) are ranked higher.
  • Support Ticket Classification: Converting customer emails to TF-IDF vectors to classify them into "Billing", "Tech Support", or "Sales".
  • Document Clustering: Grouping research papers by vectorizing text and applying K-Means clustering.

Advantages & Limitations

Advantages

  • Simple & Fast: Highly efficient to compute and store.
  • Highlights Important Words: TF-IDF successfully identifies domain-specific keywords.

Limitations

  • No Semantic Meaning: Treats words as isolated symbols. It does not know that "king" and "queen" are related, or that "car" and "automobile" mean the same thing.
  • Sparse Matrices: If your vocabulary has 50,000 words, each document vector has 50,000 columns, mostly filled with zeros. This consumes a massive amount of memory.

FAQs

Q: What is the difference between CountVectorizer and TfidfVectorizer? A: CountVectorizer simply counts raw frequencies, meaning common words dominate. TfidfVectorizer scales down frequent words and scales up rare words to highlight high-information features.

Q: How does ngram_range=(1, 2) work in Scikit-Learn? A: It tells the vectorizer to extract both individual words (unigrams) and pairs of adjacent words (bigrams), allowing the model to capture short word sequences.

Summary

Text vectorization converts raw words into numerical arrays. While Bag of Words models simply count frequencies, they can be enriched using N-grams to keep local word ordering. TF-IDF improves upon raw counts by scaling down common grammatical terms and prioritizing rare informational words, producing sparse matrices for downstream classifiers.

Next Topic

How do we capture the true semantic meaning of words (e.g., realizing that "cat" and "kitten" are similar) using dense, continuous vectors? Let's check: Word Embeddings: Word2Vec, GloVe, and Semantic Space.

← Back to Syllabus