Decision Trees in Machine Learning

Introduction

Linear and Logistic Regression are mathematically beautiful, but they share a massive flaw: they are linear. They try to draw straight lines through data. Real-world human decisions are rarely straight lines; they are often a series of "If-This-Then-That" questions. To replicate human-like decision making, Artificial Intelligence uses Decision Trees, a powerful algorithm capable of learning highly complex, non-linear patterns.

What You Will Learn

• How a Decision Tree mimics human logic.
• The terminology: Root, Nodes, and Leaves.
• The mathematics of splitting data: Entropy and Gini Impurity.
• Why Decision Trees are prone to Overfitting.

Why This Topic Matters

Decision Trees are the most "explainable" algorithm in Machine Learning. In highly regulated industries like Banking and Healthcare, you cannot use a "Black Box" neural network because it is illegal to deny someone a loan without explaining exactly why. A Decision Tree allows you to print out a flowchart and show the customer exactly which "If/Else" branch caused their denial.

Prerequisites

• Supervised Learning

Detailed Explanation

A Decision Tree is exactly what it sounds like: an upside-down tree that makes decisions.

It starts at the top with the entire dataset and asks a True/False question about one specific feature. It splits the data into two branches based on the answer. It keeps asking questions and splitting until it reaches a final decision at the bottom.

Tree Terminology

1. Root Node: The very top of the tree. The single best question that splits the dataset most perfectly.
2. Internal Nodes: The branches. Subsequent questions asked based on previous answers.
3. Leaf Nodes: The absolute bottom of the tree. The final prediction (e.g., "Approved" or "Denied").

The Math: How does it choose the questions?

How does the AI know which question to ask first? Should it split the data by "Age" or by "Income"?

It calculates Gini Impurity or Entropy. These are mathematical formulas that measure how "messy" a dataset is.

• If a group has 50 Cats and 50 Dogs, the Entropy is high (Very messy).
• If a group has 100 Cats and 0 Dogs, the Entropy is 0 (Perfectly pure).

The AI loops through every single feature and calculates: "If I split the data using this feature, how much does the Entropy drop?" (This drop is called Information Gain). It always chooses the question that provides the highest Information Gain, forcing the data to become as pure as possible, as fast as possible.

Visual Diagram (Mermaid)

graph TD
    A[Root Node: <br> Is Income > $50,000?]
    
    A -- Yes --> B{Node: <br> Credit Score > 700?}
    A -- No --> C{Node: <br> Has Co-Signer?}
    
    B -- Yes --> D[Leaf: APPROVED]
    B -- No --> E[Leaf: DENIED]
    
    C -- Yes --> F[Leaf: APPROVED]
    C -- No --> G[Leaf: DENIED]
    
    style D fill:#10B981,stroke:#fff,color:#fff
    style F fill:#10B981,stroke:#fff,color:#fff
    style E fill:#EF4444,stroke:#fff,color:#fff
    style G fill:#EF4444,stroke:#fff,color:#fff

Python Code Examples

Decision Trees are incredibly easy to implement using scikit-learn.

from sklearn.tree import DecisionTreeClassifier
import numpy as np

# Dataset: [Income (in 1000s), Credit Score]
X = np.array([
    [40, 600],  # Low income, low score
    [80, 750],  # High income, high score
    [45, 720],  # Low income, high score
    [90, 620]   # High income, low score
])

# Labels: 0 = Denied, 1 = Approved
y = np.array([0, 1, 1, 0])

# Instantiate the model
# We set max_depth to stop the tree from growing infinitely
model = DecisionTreeClassifier(max_depth=3)

# Train the model
model.fit(X, y)

# Predict a new applicant: $50k income, 680 credit score
new_applicant = np.array([[50, 680]])
prediction = model.predict(new_applicant)

print(f"AI Decision: {'Approved' if prediction[0] == 1 else 'Denied'}")

Industry Use Cases

• Medical Triage: ER systems use Decision Trees to quickly triage patients. ("Is patient breathing?" -> "Is heart rate > 120?" -> Route to Trauma).
• Loan Approvals: As seen above, replacing human loan officers with an automated, perfectly consistent, unbiased set of rules.

Advantages

• Explainability: Literally draws a flowchart that a 5-year-old can understand.
• No Data Scaling: Linear Regression requires you to scale your data (e.g., turning $100,000 into 1.0). Decision Trees don't care about the scale; they just look for cut-off thresholds.
• Handles Non-Linearity: Can easily solve complex datasets where variables interact in weird, non-straight-line ways.

Limitations

• Massive Overfitting: This is the achilles heel of Decision Trees. If you do not force them to stop growing (by setting a max_depth), the tree will grow a unique leaf for every single row in your training data. It will achieve 100% training accuracy but fail instantly in the real world because it memorized the noise.
• Instability: Changing just one row in the training dataset can cause the math to calculate a completely different Root Node, resulting in an entirely different tree being built.

FAQs

Q: Can a Decision Tree do Regression (predict numbers)? A: Yes! A Decision Tree Regressor splits the data just like classification, but the final Leaf Node predicts the average value of all the data points that landed in that leaf.

Summary

Decision Trees are powerful, non-linear algorithms that make predictions by splitting data into smaller and smaller chunks using a series of mathematically optimized True/False questions. While highly interpretable, their severe tendency to Overfit makes them dangerous to use alone.

Next Topic

How do we fix the massive Overfitting problem of a single Decision Tree? We plant an entire forest. Move on to: Random Forests.