Stochastic Gradient Descent (SGD): How It Works, Formula & Examples
Stochastic Gradient Descent (SGD) is a gradient-based optimisation algorithm that updates a model's parameters using one randomly selected training example at a time.
In this article, we will look into how stochastic gradient descent works, the SGD update formula, how it differs from batch and mini-batch gradient descent, and its advantages, limitations, and practical applications in machine learning.
What is Gradient Descent?
gradient descent in machine learning is an optimisation algorithm used to reduce a model's prediction error during training. It works by calculating the prediction error, measuring how each model parameter contributes to that error, and updating those parameters in a direction that reduces the overall loss. This process is repeated until the model reaches a point where further updates produce little or no improvement.
Consider a property listing platform training a model to estimate house prices. During each training cycle, the model predicts house prices, compares them with the actual selling prices, calculates the prediction error, and updates its parameters to improve the next set of predictions. In standard gradient descent, this update is performed only after evaluating the entire training dataset.
To understand how prediction error is calculated and how gradients are propagated through a neural network, you can also explore this Gradient Descent and Backpropagation.
Build an AI-First Career, Master the Complete Skillset
Choose from our industry-leading programs designed for career success
Modern Software and AI Engineering Program
Master full-stack development with AI integration
+1000 moreModern Data Science and ML with specialisation in AI
Advanced data science techniques with AI specialization
+1000 moreAdvanced AIML with Specialisation in Agentic AI
Deep dive into AIML with focus on Agentic systems
+1000 moreDevOps, Cloud & AI Platform Engineering
Build and manage AI-powered cloud infrastructure
+1000 moreAI Engineering Advanced Certification by IIT-Roorkee
Premier AI engineering certification from IIT-Roorkee
AI Forward Deployed Engineer Program
Full-stack engineering, production AI and client-facing consulting
+1000 morewhat is stochastic gradient descent?
Stochastic Gradient Descent (SGD) follows the same optimisation process but updates the model after processing one randomly selected training example instead of the complete dataset.
Using the same property listing example, imagine the training dataset contains thousands of houses. Rather than waiting to evaluate every property before making a single update, SGD predicts the price of one house, calculates its loss, updates the model, and immediately moves to another randomly selected training example. After every training example has been processed once, one training epoch is complete.
Because model parameters are updated much more frequently, SGD usually reaches useful solutions faster than standard gradient descent. The trade-off is that the optimisation path becomes less stable, as each update is based on only one training example rather than the entire dataset. The prediction error used for every update is determined by a loss function, making the choice of loss function an important part of model training.
Check Out: Advanced AI & Machine Learning Course with Agentic AI
Stop learning AI in fragments—master a structured AI Engineering Course with hands-on GenAI systems with IIT Roorkee CEC Certification
:::
AI Engineering Course Advanced Certification by IIT-Roorkee CEC
A hands on AI engineering program covering Machine Learning, Generative AI, and LLMs - designed for working professionals & delivered by IIT Roorkee in collaboration with Scaler.
Enrol Now
How Stochastic Gradient Descent Works
Stochastic Gradient Descent trains a model by updating its parameters after processing one training example at a time. Here’s how sgd works step-by-step:
Step 1: Initialise the model parameters
Training begins by assigning initial values to the model parameters (weights and bias). These values are usually chosen randomly, so the model's first predictions are unlikely to be accurate.
Step 2: Shuffle the training dataset
Before each training cycle (epoch), the training examples are randomly shuffled. Shuffling prevents the model from learning patterns based on the order of the data and helps produce more reliable parameter updates. If the data were processed in the same order during every epoch, the model could repeatedly learn from similar patterns in sequence, slowing convergence and making the optimisation process less reliable.
Step 3: Select one training example
SGD processes only one training example at a time. For example, a house-price prediction model may use a single property with its features and actual selling price for one training step.
Step 4: Calculate the prediction error
The model predicts the selling price of the selected house and compares it with the actual selling price. The difference between these values is measured using a loss function, which quantifies the prediction error for that training example.
Step 5: Compute the gradient
Using the calculated loss, the algorithm determines how each model parameter contributed to the prediction error. These partial derivatives form the gradient and indicate how the parameters should be adjusted to reduce the loss.
Step 6: Update the model parameters
The model parameters are updated immediately using the SGD update rule. Since the update is based on only one training example, it is computationally inexpensive and can be performed much more frequently than in standard gradient descent.
Step 7: Repeat until one epoch is complete
Steps 3 to 6 are repeated for every training example in the shuffled dataset. Once all examples have been processed, one epoch is complete. The dataset is then shuffled again, and the process continues for multiple epochs until the model converges or reaches the desired level of accuracy.
Example: One SGD update
Suppose a property listing model predicts the price of a house as ₹82 lakh, while the actual selling price is ₹78 lakh. The loss function calculates the prediction error, the gradient determines how each model parameter contributed to that error, and the SGD update rule adjusts the parameters before the model moves to the next property in the dataset. This process is repeated for every training example until the model's predictions become more accurate.
SGD Formula and Notation
The sgd formula determines how a model's parameters are adjusted after processing each training example. The update equation is:
w := w − η · (∂J/∂w)
Where:
| Symbol | Meaning |
|---|---|
| w | Current model parameter (weight) before the update |
| η (eta) | Learning rate, which controls the size of each update |
| J | Loss function that measures the prediction error |
| ∂J/∂w | Gradient of the loss function with respect to the parameter w |
Note: Every parameter in the model is updated using the same SGD update rule during training. Modern neural networks often contain millions of such parameters.
After computing the gradient for a single training example, SGD updates the parameter by moving it in the opposite direction of the gradient. If the gradient is positive, the parameter value decreases. If the gradient is negative, the parameter value increases. Repeating this process across multiple training examples gradually reduces the overall loss and improves the model's predictions. SGD computes the gradient using only one training example at a time. This enables faster parameter updates but also introduces more variation in the optimisation path.
Example:
Suppose a house-price prediction model has the following values after processing one training example:
| Parameter | Value |
|---|---|
| Current weight (w) | 2.50 |
| Learning rate (η) | 0.10 |
| Gradient (∂J/∂w) | 0.80 |
Using the SGD update rule:
w = w − η × (∂J/∂w)
Substituting the values:
w = 2.50 − (0.10 × 0.80)
w = 2.50 − 0.08
w = 2.42
After processing this training example, the model updates the weight from 2.50 to 2.42. During training, the same calculation is performed for every model parameter after each training example, gradually reducing the overall loss.
Note: This example shows the update for a single model parameter. But when it comes to working with it, machine learning models often contain thousands or even millions of parameters, and the SGD update rule is applied to each of them during training.
SGD vs Batch vs Mini-Batch Gradient Descent
This difference in sgd vs batch gradient descent affects training speed, memory usage, convergence behaviour, and computational efficiency.
| Feature | Batch Gradient Descent | Stochastic Gradient Descent (SGD) | Mini-Batch Gradient Descent |
|---|---|---|---|
| Training data used per update | Entire training dataset | One training example | A small batch of training examples (commonly 32–512) |
| Parameter updates | One update after processing the complete dataset | One update after every training example | One update after every mini-batch |
| Training speed | Slow for large datasets | Fast | Faster than batch, more stable than SGD |
| Memory requirement | High | Low | Moderate |
| Convergence behaviour | Smooth and stable | Noisy with frequent fluctuations | More stable while retaining faster updates |
| Best suited for | Small datasets | Large datasets and online learning | Most modern deep learning applications |
Did you know?
The mini-batch gradient descent is the preferred optimisation approach for training most machine learning and deep learning models. It combines the computational efficiency of stochastic gradient descent with the stability of batch gradient descent, allowing models to train faster without introducing excessive fluctuations during parameter updates.
Standard batch gradient descent is still used for smaller datasets where computational cost is less of a concern, while stochastic gradient descent remains a practical choice for online learning and applications where data arrives continuously.
How Scaler Transformed Careers in Different Fields
Scaler learners achieved 2.5x salary growth with average post-Scaler CTC reaching ₹23L.
Google
Amazon
Microsoft
Flipkart
Adobe1200+ moreAdvantages and Disadvantages of SGD
Like every optimisation algorithm, stochastic gradient descent involves a trade-off between training speed and optimisation stability. The advantages of sgd depends on the size of the dataset, the learning rate, and the complexity of the machine learning model.
Advantages of Stochastic Gradient Descent
- Faster parameter updates: Since SGD updates the model after processing each training example, it begins improving the model immediately instead of waiting for the entire dataset to be evaluated.
- Scales well to large datasets: Processing one training example at a time keeps the computational cost of each update low, making SGD suitable for training models on datasets containing millions of observations.
- Requires less memory: Unlike batch gradient descent, SGD does not need to load the complete training dataset to compute every update, reducing memory requirements during training.
- Can escape shallow local minima: The randomness introduced by frequent parameter updates helps the optimisation process continue exploring the loss surface instead of settling too early into suboptimal solutions.
Turn Learning into Career Growth
Disadvantages of Stochastic Gradient Descent
- Less stable convergence: Because every update is based on a single training example, the optimisation path fluctuates more than batch gradient descent and may take longer to settle near the minimum.
- Sensitive to the learning rate: A learning rate that is too high may cause the algorithm to overshoot the optimal solution, while a very small learning rate can slow down convergence significantly.
- May require more training epochs: Although each update is computationally inexpensive, the algorithm often needs multiple passes through the training dataset before convergence.
- Performance depends on proper tuning: SGD is commonly used with techniques such as momentum or adaptive optimisers to achieve faster and more stable convergence.
The Role of Learning Rate (and Beyond: Momentum, Adam)
The learning rate (η) determines how much the model's parameters change after every SGD update. Since stochastic gradient descent updates the model after each training example, selecting an appropriate learning rate is essential for achieving stable and efficient training.
- A high learning rate makes larger parameter updates. While this may speed up learning initially, the model can overshoot the optimal solution and fail to converge.
- A low learning rate produces smaller updates, resulting in a more stable optimisation process. However, training becomes slower and may require many more epochs to reach a good solution.
When you work with ML enough, you’ll soon realize that the learning rate is also reduced gradually as training progresses. This approach, known as learning rate scheduling, allows the model to take larger steps during the early stages of training and smaller, more precise updates as it moves closer to the minimum.
Read More: Model Optimization
Choosing an appropriate learning rate often requires experimentation, as the optimal value depends on the dataset, model architecture, and optimisation problem.
Beyond SGD: Momentum and Adam
Several optimisation algorithms build upon stochastic gradient descent to improve convergence.
- Momentum reduces unnecessary oscillations by incorporating information from previous parameter updates. This allows the optimiser to move more consistently toward the minimum, particularly on uneven loss surfaces.
- Adam (Adaptive Moment Estimation) extends SGD by combining momentum with adaptive learning rates for individual parameters. Instead of using a single learning rate throughout training, Adam automatically adjusts the update size for each parameter, often leading to faster and more stable convergence.
Although SGD remains a widely used optimisation algorithm, momentum and Adam are commonly preferred for training deep neural networks because they generally converge faster and require less manual tuning of the learning rate.
Conclusion
| Optimisation Method | Best Used When |
|---|---|
| Batch Gradient Descent | The dataset is small and stable parameter updates are more important than training speed. |
| Stochastic Gradient Descent (SGD) | The dataset is large or arrives continuously, and faster parameter updates are preferred. |
| Mini-Batch Gradient Descent | Most modern machine learning and deep learning applications, as it balances training speed and stable convergence. |
Start your learning journey today with Data Science & ML Course with AI Specialization
FAQs
1. What is stochastic gradient descent?
Stochastic Gradient Descent (SGD) is an optimisation algorithm that updates a model's parameters after processing one training example at a time. These frequent updates make it computationally efficient for training machine learning and deep learning models on large datasets.
2. What is the difference between SGD and batch gradient descent?
Batch gradient descent calculates the gradient using the entire training dataset before updating the model, whereas SGD updates the model after processing a single training example. As a result, batch gradient descent produces smoother updates, while SGD trains faster but follows a noisier optimisation path.
3. What is the SGD update formula?
The SGD update rule is w := w − η · (∂J/∂w), where w represents the model parameter, η is the learning rate, and ∂J/∂w is the gradient of the loss function with respect to that parameter.
4. What is mini-batch gradient descent?
Mini-batch gradient descent updates model parameters using a small batch of training examples instead of the entire dataset or a single example. It combines the computational efficiency of SGD with the stability of batch gradient descent, making it the most commonly used optimisation approach in modern deep learning.
5. Why is stochastic gradient descent called "stochastic"?
The term stochastic refers to randomness. SGD processes training examples in a random order, so every parameter update is based on a different training example rather than the complete dataset.
6. What is the role of the learning rate in SGD?
The learning rate determines the size of each parameter update. A high learning rate may cause the model to overshoot the optimal solution, while a low learning rate can slow convergence and increase the training time.
