Linear Discriminant Analysis (LDA) in Machine Learning Explained
You have a classification problem with labeled data and too many features. Your first instinct might be PCA: reduce the dimensions, keep the variance, then classify. But PCA does not know about your labels. It finds the direction of maximum spread in your data, which is sometimes the exact wrong direction for separating classes. The two classes might be stretched along the same axis, and projecting onto that axis mixes them together instead of pulling them apart.
Linear Discriminant Analysis solves exactly this problem. It is a supervised technique that finds the projection making classes as separable as possible ,pushing class means apart while keeping each class tight. Unlike PCA, LDA knows your labels and uses them. And unlike most dimensionality reduction methods, LDA also works directly as a classifier. This dual role — reducer and classifier in one algorithm , is what makes it worth understanding on its own terms before comparing it to anything else.
Before we go further, a disambiguation: LDA in machine learning means Linear Discriminant Analysis. In natural language processing, LDA means Latent Dirichlet Allocation, which is a completely different technique for topic modeling in text. They share an acronym and nothing else. This page is about the classification and dimensionality reduction technique.
What Is Linear Discriminant Analysis (LDA)?
Linear Discriminant Analysis is a supervised machine learning technique that projects data onto a lower-dimensional space in a way that maximizes the separation between classes. It was introduced by Ronald Fisher in 1936 in the paper "The Use of Multiple Measurements in Taxonomic Problems", where he applied it to the Iris dataset — the same dataset we will use in the code section below.
LDA serves two purposes simultaneously:
- Dimensionality reduction: Project high-dimensional data onto a lower-dimensional space (at most C−1 dimensions for C classes) while preserving class separability
- Classification: Model each class as a Gaussian distribution with shared covariance and classify new points by their posterior probability
Most techniques do one or the other. PCA reduces dimensions but does not classify. Logistic Regression classifies but does not reduce dimensions. LDA does both, which is why it appears in both the dimensionality reduction toolkit and the classification algorithm toolkit.
For the broader landscape of where LDA sits among other machine learning algorithms, it bridges the generative-modeling approach (like Naive Bayes) and the discriminative approach (like Logistic Regression).
Stop learning AI in fragments—master a structured AI Engineering Course with hands-on GenAI systems with IIT Roorkee CEC Certification
:::
AI Engineering Course Advanced Certification by IIT-Roorkee CEC
A hands on AI engineering program covering Machine Learning, Generative AI, and LLMs - designed for working professionals & delivered by IIT Roorkee in collaboration with Scaler.
Enrol Now
The Intuition: Finding the Axis That Separates Classes
Here is the geometric picture that makes LDA obvious. Imagine a 2D dataset with two classes: red points and blue points. The red points are clustered on the left, the blue points on the right, but both classes are stretched diagonally from bottom-left to top-right.
If you apply PCA, it finds the direction of maximum variance. That direction is the diagonal the axis along which the data is most spread out. But projecting onto the diagonal mixes the two classes together because both red and blue points span that axis.
LDA asks a different question: "Which projection axis makes the class means as far apart as possible while keeping each class as tight as possible?" The answer is roughly the horizontal axis perpendicular to the direction that would mix the classes. Projecting onto that axis pulls the red mean and the blue mean apart while compressing each class's spread.
This is Fisher's criterion in geometric terms: maximize the distance between class means relative to the spread within each class.
Fisher formalized this in 1936 with the Iris dataset, showing that three species of iris flowers could be separated by projecting their four measurements (sepal length, sepal width, petal length, petal width) onto a single discriminant axis. The dataset he used is the same one that ships with scikit-learn today.
Transform Your Career
Choose from our industry-leading programs designed for career success
Modern Software and AI Engineering Program
Master full-stack development with AI integration
+1000 moreModern Data Science and ML with specialisation in AI
Advanced data science techniques with AI specialization
+1000 moreAdvanced AIML with Specialisation in Agentic AI
Deep dive into AIML with focus on Agentic systems
+1000 moreDevOps, Cloud & AI Platform Engineering
Build and manage AI-powered cloud infrastructure
+1000 moreAI Engineering Advanced Certification by IIT-Roorkee
Premier AI engineering certification from IIT-Roorkee
How LDA Works: Between-Class and Within-Class Scatter
The math behind LDA is a ratio of two scatter matrices. Here is what each one measures and why their ratio gives you the optimal projection.
The Two Scatter Matrices
- Between-class scatter matrix (S_B): Measures how far apart the class means are from the overall mean. A large S_B means the classes are spread far apart which is what you want for separation.
- Within-class scatter matrix (S_W): Measures how spread out the points are within each class. A small S_W means each class is tight and compact which is also what you want for separation.
Fisher's Criterion
The optimal projection direction w maximizes the ratio:
J(w) = (w^T × S_B × w) / (w^T × S_W × w)
In words: find the direction where the between-class separation is large relative to the within-class spread. The solution is the eigenvector of S_W⁻¹ × S_B corresponding to the largest eigenvalue.
The Algorithm in Four Steps
- Compute class means: Calculate the mean vector for each class and the overall mean
- Compute scatter matrices: Build S_B (between-class) and S_W (within-class)
- Solve the eigenvalue problem: Find the eigenvectors of S_W⁻¹ × S_B
- Project: Select the top k eigenvectors and project the data onto them
For a C-class problem, S_B has rank at most C−1, which means you can extract at most C−1 discriminant directions. For the 3-class Iris dataset, that means at most 2 LDA components.
Comfortable with eigenvectors and scatter matrices? If not, Scaler's free Mathematics for Machine Learning course covers the linear algebra behind LDA, PCA, and other ML algorithms with structured exercises.
LDA's Dual Role: Dimensionality Reduction AND Classification
Most dimensionality reduction techniques stop at reduction. PCA gives you a lower-dimensional representation, and then you feed that into a separate classifier. LDA can do both in one step.
As a Dimensionality Reducer
Call fit_transform() on the LDA object and you get a lower-dimensional representation of your data. For a C-class problem, you can reduce to at most C−1 dimensions. This is a hard mathematical limit based on the rank of the between-class scatter matrix.
- Binary classification (2 classes): at most 1 LDA component
- 3-class problem (like Iris): at most 2 LDA components
- 10-class problem: at most 9 LDA components
This limit is both a constraint and a feature. It means LDA is not suitable for reducing to very low dimensions when you have few classes, but it also means the components it gives you are guaranteed to be the most discriminative ones possible.
As a Classifier
Call predict() on the LDA object and it classifies directly. Under the hood, LDA models each class as a multivariate Gaussian distribution with a shared covariance matrix. For a new data point, it computes the posterior probability of each class using Bayes' theorem and assigns the point to the class with the highest probability.
The decision boundary between any two classes is linear (hence "linear" discriminant analysis) because the shared covariance assumption means the quadratic terms cancel out when you compare posteriors.
This generative approach connects LDA to Naive Bayes both model class-conditional distributions and classify via Bayes' theorem. The difference is that Naive Bayes assumes feature independence while LDA assumes shared covariance.
For the broader context of how LDA fits into feature extraction pipelines, the feature extraction guide maps where LDA, PCA, and other methods fit in a typical ML workflow.
The dimensionality reduction overview covers why reducing dimensions matters for both computational cost and model performance.
LDA vs PCA: The Definitive Comparison
This is the comparison every ML student needs to internalize. The two techniques look similar (both project data onto new axes) but optimize completely different objectives.
| Aspect | PCA | LDA |
|---|---|---|
| Supervision | Unsupervised — ignores labels | Supervised — uses class labels |
| Objective | Maximize variance along projection axes | Maximize separation between class means relative to within-class spread |
| What it finds | Directions of maximum data spread | Directions of maximum class discrimination |
| Max components | min(n_features, n_samples) | n_classes − 1 |
| When it wins | Unlabeled data, compression, visualization, noise reduction | Labeled data with a classification goal |
| When it fails for classification | Max-variance axis may mix classes (the wrong-axis problem) | Small sample sizes, non-Gaussian classes, different class covariances |
| Common pipeline | PCA → any classifier | LDA as reducer → classifier, or LDA as classifier directly |
The rule of thumb: If you have labels and your end goal is classification, try LDA first. If you have no labels, or your goal is compression or visualization without a classification target, use PCA. In practice, many pipelines use PCA first (to reduce noise and dimensionality) and then LDA (to find the discriminative directions in the reduced space).
For a deeper dive into PCA's mechanics, the PCA guide covers eigendecomposition, explained variance, and when PCA helps versus hurts model performance.
LDA in Python: Iris Walkthrough (with PCA Side-by-Side)
Here is the complete code that applies both LDA and PCA to the Iris dataset and compares the projections visually.
Expected output:
LDA Classification Accuracy: 1.000
LDA explained variance ratios: [0.9912 0.0088]
Two things to notice in the side-by-side plots:
- PCA's first component explains most of the variance (around 92 percent) but the projection does not separate the classes as cleanly as LDA. Versicolor and Virginica overlap more in PCA space.
- LDA's first component explains 99.1 percent of the class separability and the three species are nearly perfectly separated in the 2D LDA projection. Setosa is completely isolated, and Versicolor and Virginica have minimal overlap.
The scikit-learn LDA documentation covers the three solver options: svd (default, does not compute the scatter matrices explicitly), lsqr (supports shrinkage), and eigen (the classical approach described above).
Scaler Placement Report and Statistics
Scaler learners achieved 2.5x salary growth with average post-Scaler CTC reaching ₹23L.
Google
Amazon
Microsoft
Flipkart
Adobe1200+ moreAssumptions and When LDA Fails
LDA makes specific statistical assumptions. When they hold, LDA is powerful and efficient. When they are violated, performance degrades and you need alternatives.
The Three Core Assumptions
1. Gaussian class distributions: LDA assumes each class is drawn from a multivariate Gaussian distribution. In practice, LDA is reasonably robust to mild deviations from normality, but heavily skewed or multimodal class distributions will reduce its effectiveness.
2. Equal covariance matrices: LDA assumes all classes share the same covariance matrix. This is what produces linear decision boundaries. If different classes have very different spreads or orientations in the feature space, the shared-covariance assumption is violated. Quadratic Discriminant Analysis (QDA) relaxes this assumption by allowing each class its own covariance matrix, producing quadratic (curved) decision boundaries instead of linear ones.
3. Independence of features (partial): While LDA does not assume strict feature independence like Naive Bayes, highly correlated features can make the within-class scatter matrix S_W singular or near-singular, especially when the number of features exceeds the number of samples. Regularized LDA or dimensionality reduction before LDA addresses this.
Failure Modes and Alternatives
| Failure Mode | What Goes Wrong | Alternative |
|---|---|---|
| Non-Gaussian classes | Decision boundary does not follow the true class shape | Kernel LDA, QDA, or non-parametric classifiers |
| Different class covariances | Linear boundary cannot separate classes with different spreads | QDA (quadratic boundaries) |
| Small sample size (p >> n) | S_W becomes singular, eigendecomposition fails | Regularized LDA (shrinkage='auto' in sklearn) or PCA before LDA |
| Non-linear class boundaries | Linear projection cannot capture the separation | Kernel LDA, or non-linear methods (t-SNE, UMAP for visualization; SVM with kernel for classification) |
| More dimensions needed than C−1 | LDA's component limit is too restrictive | PCA for additional dimensions, then LDA on top |
Sensitivity to Outliers
LDA is sensitive to outliers because they distort the class means and inflate the within-class scatter. A few extreme points can shift the discriminant direction away from the optimal axis. Robust variants of LDA exist that use robust estimates of the mean and covariance, but in practice, outlier detection and removal before LDA is the simpler approach.
LDA vs Other Classifiers and Key Takeaways
LDA sits in an interesting position relative to other classification algorithms:
LDA vs Logistic Regression: Both produce linear decision boundaries, but they arrive there differently. Logistic Regression is discriminative it models the probability of the class given the features directly. LDA is generative it models the distribution of features within each class and derives the decision boundary from Bayes' theorem. On well-behaved data that meets LDA's assumptions, both perform similarly. On data that violates the Gaussian assumption, Logistic Regression is usually more robust because it makes fewer assumptions. The Logistic Regression guide covers the discriminative approach in detail.
LDA vs Naive Bayes: Both are generative classifiers that use Bayes' theorem. The difference is in the feature-dependency assumption. Naive Bayes assumes features are conditionally independent given the class (which is almost never true but works well for text). LDA assumes features follow a multivariate Gaussian with shared covariance (which captures feature correlations but assumes a specific distributional form).
Turn Learning into Career Growth
Key Takeaways
- LDA is supervised: it uses class labels to find the most discriminative projection, unlike PCA which ignores labels
- LDA has a dual role: it reduces dimensions (up to C−1 components) and classifies directly via Gaussian class models
- Fisher's criterion is the core idea: maximize between-class separation relative to within-class spread
- The three assumptions (Gaussian classes, equal covariance, sufficient samples) determine when LDA works and when to switch to alternatives
- In practice, LDA is a strong baseline for classification with moderate-dimensional data and a useful preprocessing step before more complex classifiers
Build intuition for every classical ML algorithm from LDA to XGBoost: Scaler's AI and ML Program covers the mathematical foundations, practical implementations, and production considerations across the full ML algorithm landscape, with mentor-led guidance and industry-relevant projects.
FAQs
What is Linear Discriminant Analysis in simple terms?
Linear Discriminant Analysis is a supervised technique that finds the best way to project your data so that different classes end up as far apart as possible while each class stays as tight as possible. It uses class labels to find discriminative directions, unlike PCA which ignores labels and just looks for maximum variance. LDA can be used both to reduce the number of features in your data and to classify new data points directly.
What is the difference between LDA and PCA?
PCA is unsupervised and finds the directions of maximum variance in your data, regardless of class labels. LDA is supervised and finds the directions that maximize separation between classes. For a classification task with labeled data, LDA's projection directions are usually more useful because they are optimized for discrimination. However, PCA can reduce to any number of dimensions while LDA is limited to at most (number of classes minus 1) components. A common pipeline uses PCA first for noise reduction and then LDA for finding discriminative directions.
Is LDA a classifier or a dimensionality reduction technique?
It is both. When you call fit_transform(), LDA acts as a dimensionality reducer, projecting your data onto at most (C−1) discriminant axes where C is the number of classes. When you call predict(), LDA acts as a classifier by modeling each class as a Gaussian distribution with shared covariance and assigning new points to the class with the highest posterior probability. This dual role makes LDA unique among classical ML techniques.
What are the assumptions of LDA?
LDA assumes three things: that each class follows an approximately Gaussian distribution, that all classes share the same covariance matrix (which produces linear decision boundaries), and that there are enough samples to estimate the scatter matrices reliably. When these assumptions are violated, alternatives exist: Quadratic Discriminant Analysis relaxes the equal-covariance assumption, Kernel LDA handles non-linear boundaries, and regularized LDA addresses small-sample issues.
How many dimensions can LDA reduce to?
At most (number of classes minus 1). For a binary classification problem, LDA can produce only 1 discriminant component. For the 3-class Iris dataset, it produces at most 2 components. This is a mathematical limit based on the rank of the between-class scatter matrix, not a tunable parameter. If you need more dimensions than this limit allows, you can use PCA before LDA to first reduce to a manageable number of features.
Is Linear Discriminant Analysis the same as Latent Dirichlet Allocation?
No. They share the acronym "LDA" but are completely different techniques. Linear Discriminant Analysis is a supervised classification and dimensionality reduction method for numerical data. Latent Dirichlet Allocation is an unsupervised topic modeling technique for text data that discovers hidden topics in document collections. In machine learning contexts, LDA almost always refers to Linear Discriminant Analysis. In NLP and text mining contexts, it usually refers to Latent Dirichlet Allocation.
