Word Embeddings: Word2Vec, GloVe, and Semantic Space

Explore Word Embeddings in NLP. Learn about Word2Vec (CBOW and Skip-gram architectures), GloVe, cosine similarity, and vector math.

Introduction

In TF-IDF, the vectors for "car" and "automobile" are completely perpendicular, meaning their mathematical correlation is exactly zero. The model has no idea they refer to the same object.

To solve this, linguist J.R. Firth in 1957 suggested: "You shall know a word by the company it keeps."

This philosophy inspired Word Embeddings—dense, low-dimensional vectors where words with similar meanings or contexts are mapped to nearby coordinates in a continuous Semantic Space. Under this paradigm, "car" and "automobile" are represented by similar vectors, allowing algorithms to understand actual language semantics.

What You Will Learn

  • The difference between sparse one-hot vectors and dense embeddings.
  • The two architectures of Word2Vec: CBOW and Skip-gram.
  • The concept of Vector Arithmetic (e.g., King $-$ Man $+$ Woman $=$ Queen).
  • How to measure semantic similarity using Cosine Similarity.
  • How to load and query pre-trained embeddings in Python.

Why This Topic Matters

Word embeddings are the cornerstone of modern deep NLP. By representing words as dense vectors (usually 100 to 300 dimensions), they compress vocabulary sizes, capture complex synonym relationships, and serve as the standard input representation for advanced systems like LSTMs, Transformers, and LLMs.

Prerequisites

Detailed Explanation

To understand embeddings, we must look at how Word2Vec utilizes context to train dense vectors.

Sparse vs. Dense Vectors

If our vocabulary has 10,000 words, One-Hot Encoding represents each word as a vector of size 10,000 with a single 1 and 9,999 0s. Word Embeddings map these same words to dense vectors of size 300, filled with continuous floating-point numbers.

Word        One-Hot Vector (Sparse)         Embedding Vector (Dense)
--------------------------------------------------------------------------
king        [1, 0, 0, 0, 0, ... 0]          [0.25, -0.42, 0.81, ... 0.12]
queen       [0, 1, 0, 0, 0, ... 0]          [0.24, -0.40, 0.83, ... 0.11]

The Word2Vec Architectures

Developed by Tomas Mikolov at Google in 2013, Word2Vec is a shallow neural network trained to predict context. It has two training architectures:

graph TD
    A[Word2Vec Training]
    A --> B[CBOW: Continuous Bag of Words <br> Predict target word using surrounding context]
    A --> C[Skip-gram <br> Predict surrounding context using a target word]

1. Continuous Bag of Words (CBOW)

  • Goal: Predict a missing target word based on its context (surrounding words).
  • Example: Given ["the", "cat", "___", "on", "the", "mat"], CBOW tries to predict "sat".
  • Behavior: Faster to train; works well for frequent words.

2. Skip-gram

  • Goal: Predict the surrounding context words given a single target word.
  • Example: Given the target word "sat", Skip-gram tries to predict context words like ["cat", "on", "mat"].
  • Behavior: Performs better on small datasets and represents rare words more accurately.

Semantic Vector Arithmetic

Because these vectors capture meaning, we can perform algebraic operations on them. The most famous example demonstrates gender translation:

$$\vec{v}{\text{king}} - \vec{v}{\text{man}} + \vec{v}{\text{woman}} \approx \vec{v}{\text{queen}}$$

Similarly: $$\vec{v}{\text{Paris}} - \vec{v}{\text{France}} + \vec{v}{\text{Germany}} \approx \vec{v}{\text{Berlin}}$$

Cosine Similarity

To find how similar two words are, we measure the angle between their vectors in the 300-dimensional space using Cosine Similarity:

$$\text{Similarity}(\vec{A}, \vec{B}) = \cos(\theta) = \frac{\vec{A} \cdot \vec{B}}{||\vec{A}|| \cdot ||\vec{B}||}$$

  • Cosine = 1: Vectors point in the exact same direction (synonyms).
  • Cosine = 0: Vectors are orthogonal (uncorrelated words).
  • Cosine = -1: Vectors point in opposite directions (antonyms).

Visual Diagram (Mermaid)

graph LR
    subgraph CBOW
        C[Context Words] -->|Predict| T[Target Word]
    end
    subgraph Skip-gram
        Target[Target Word] -->|Predict| Context[Context Words]
    end
    style T fill:#10B981,stroke:#fff,color:#fff
    style Target fill:#3B82F6,stroke:#fff,color:#fff

Python Code Examples

We will use the gensim library to load a mini pre-trained Word2Vec model and perform semantic similarity queries.

import gensim.downloader as api

# 1. Load a small, pre-trained Word2Vec model (glove-wiki-gigaword-50)
# This model represents words as 50-dimensional vectors
print("Loading Word2Vec model...")
model = api.load("glove-wiki-gigaword-50")
print("Model loaded successfully!")

# 2. Find Cosine Similarity
similarity = model.similarity("car", "automobile")
print(f"\nSimilarity between 'car' and 'automobile': {round(similarity, 4)}")

# 3. Perform Vector Math: king - man + woman = ?
result = model.most_similar(positive=["king", "woman"], negative=["man"], topn=1)
print(f"Result of 'king - man + woman': {result[0][0]} (score: {round(result[0][1], 4)})")

# 4. Find anomalies (which word doesn't match?)
odd_word = model.doesnt_match(["apple", "banana", "orange", "computer"])
print(f"Odd word in [apple, banana, orange, computer]: {odd_word}")

Industry Use Cases

  • Recommendation Systems: Vectorizing user search terms to recommend semantically related products (e.g., searching for "running sneakers" shows results for "athletic footwear").
  • Sentiment Analysis: Pre-training words to catch nuances (e.g., mapping "dreadful" close to "bad").
  • Language Translation: Aligning semantic spaces across different languages to translate words with similar vector structures.

Advantages & Limitations

Advantages

  • Captures Semantics: Captures word analogies, synonyms, and context.
  • Dimensionality Reduction: Replaces huge sparse matrices with small, dense, 300-dimension vectors.

Limitations

  • Polysemy Issue: A word can only have one vector. Therefore, Word2Vec cannot separate the meaning of "bank" (river bank) from "bank" (financial bank). It merges both meanings into a single vector.
  • Static Context: The vector remains identical regardless of the sentence it appears in. (Modern transformers like BERT solve this).

FAQs

Q: What is GloVe? A: GloVe (Global Vectors for Word Representation) is another word embedding method developed by Stanford. While Word2Vec uses local context windows in a neural network, GloVe works by optimizing factorization of global word co-occurrence matrices.

Q: How do we handle Out-Of-Vocabulary (OOV) words? A: Standard Word2Vec cannot generate vectors for words it hasn't seen during training. Algorithms like FastText solve this by breaking words into character n-grams, allowing them to construct vectors for unseen words.

Summary

Word embeddings map words into dense, continuous vector spaces where geometric proximity reflects semantic meaning. Trained on context clues using CBOW or Skip-gram models, Word2Vec vectors enable mathematical operations, cosine similarities, and contextual modeling that form the backbone of modern text processing.

Next Topic

How do neural networks process these embedding vectors sequentially over time to understand whole sentences? Let's explore: Recurrent Neural Networks (RNNs) for Sequence Modeling.

← Back to Syllabus