How Does a Neural Network Learn? Weights, Backpropagation & Gradient Descent
A neural network learns by iteratively adjusting its internal parameters, called weights and biases, to minimize the difference between its predictions and the actual ground truth. This process involves forward propagation to make a prediction, a loss function to quantify the error, and backpropagation with gradient descent to update the parameters.
The Core Concept of Learning: Minimizing Error
In the context of machine learning, "learning" is a systematic process of optimization. For a neural network, this is not a process of memorizing data but rather of tuning its internal configuration to create an accurate and generalizable mathematical model. The network starts as a blank slate, making random, uninformed predictions. By repeatedly being shown examples from a training dataset and being "told" how wrong its predictions are, it gradually refines its parameters.
This entire process is fundamentally about minimizing error. It can be broken down into three conceptual stages:
- Prediction: The network uses its current parameters (weights and biases) to make a prediction for a given input. This is known as the forward pass or forward propagation.
- Error Calculation: The prediction is compared to the correct, ground-truth label using a loss function. This function outputs a single scalar value representing the magnitude of the error. A high value signifies a poor prediction, while a value near zero signifies a good one.
- Parameter Update: The network's parameters are adjusted in a way that is likely to reduce the error on the next prediction. This update mechanism is the core of the learning process and is achieved through the combined use of backpropagation and an optimization algorithm like gradient descent.
This cycle of prediction, error calculation, and parameter update is repeated thousands or even millions of times, allowing the network to converge on a set of parameters that accurately maps inputs to outputs.
Stop learning AI in fragments—master a structured AI Engineering Course with hands-on GenAI systems with IIT Roorkee CEC Certification
AI Engineering Course Advanced Certification by IIT-Roorkee CEC
A hands on AI engineering program covering Machine Learning, Generative AI, and LLMs - designed for working professionals & delivered by IIT Roorkee in collaboration with Scaler.
Enrol Now
The Anatomy of a Prediction: Forward Propagation
Before a network can learn from its mistakes, it must first be able to make a prediction. This is accomplished through a process called forward propagation, where input data is fed through the network's layers, undergoing a series of transformations until a final output is produced. Understanding this process requires an understanding of the network's fundamental components.
Neurons, Weights, and Biases
An artificial neural network is composed of interconnected nodes, or neurons, organized into layers. Each connection between neurons has an associated weight, and each neuron (except those in the input layer) has a bias.
- Weights (w): These are the most critical parameters in a neural network. A weight associated with a connection determines the strength and sign of that connection. A large positive weight means the input from that connection will strongly excite the receiving neuron, while a large negative weight means it will strongly inhibit it. In essence, weights encode the importance of each input feature.
- Biases (b): A bias is an extra parameter associated with each neuron that is added to the weighted sum of inputs. Its role is to provide a trainable offset, allowing the activation function's output to be shifted to the left or right. This is analogous to the y-intercept in a linear equation y = mx + b and is crucial for fitting data that does not pass through the origin.
For a single neuron, the first step is to compute the weighted sum of its inputs and add the bias. This is represented by the equation:
z = (w₁x₁ + w₂x₂ + ... + wₙxₙ) + b
In vector notation, this simplifies to:
z = w ⋅ x + b
Here, x is the vector of inputs, w is the vector of corresponding weights, b is the bias, and z is the pre-activation output.
Build an AI-First Career, Master the Complete Skillset
Choose from our industry-leading programs designed for career success
Modern Software and AI Engineering Program
Master full-stack development with AI integration
+1000 moreModern Data Science and ML with specialisation in AI
Advanced data science techniques with AI specialization
+1000 moreAdvanced AIML with Specialisation in Agentic AI
Deep dive into AIML with focus on Agentic systems
+1000 moreDevOps, Cloud & AI Platform Engineering
Build and manage AI-powered cloud infrastructure
+1000 moreAI Engineering Advanced Certification by IIT-Roorkee
Premier AI engineering certification from IIT-Roorkee
AI Forward Deployed Engineer Program
Full-stack engineering, production AI and client-facing consulting
+1000 moreActivation Functions: Introducing Non-Linearity
The value z is a linear combination of inputs. If this were the final output of the neuron, the entire network, no matter how many layers deep, would behave as a simple linear model. To learn complex patterns, such as those in image recognition or natural language, the network must be able to model non-linear relationships.
This is the role of the activation function, σ(z). This function is applied to the pre-activation output z to produce the neuron's final output, a.
a = σ(z) = σ(w ⋅ x + b)
This non-linear transformation allows the network to approximate arbitrarily complex functions. Common activation functions include:
- Sigmoid: σ(z) = 1 / (1 + e⁻ᶻ). It squashes its input into a range between 0 and 1. It was historically popular but is less used in hidden layers today due to the vanishing gradient problem.
- Hyperbolic Tangent (Tanh): tanh(z) = (eᶻ - e⁻ᶻ) / (eᶻ + e⁻ᶻ). It squashes its input into a range between -1 and 1. It is zero-centered, which can be advantageous over Sigmoid.
- Rectified Linear Unit (ReLU): ReLU(z) = max(0, z). It outputs the input directly if it is positive, and 0 otherwise. It is computationally efficient and is the most widely used activation function in modern deep learning models.
[IMAGE: A graph showing the curves of the Sigmoid, ReLU, and Tanh activation functions on the same axes. The x-axis is labeled 'z (Input)' and the y-axis is labeled 'a (Output)'. The Sigmoid curve is S-shaped from 0 to 1. The Tanh curve is S-shaped from -1 to 1. The ReLU curve is a flat line at y=0 for x<0 and a diagonal line y=x for x>0.]
The Forward Pass: From Input to Output
The forward pass is the complete process of an input vector x traversing the entire network to produce a final prediction ŷ (y-hat).
- The input data x is presented to the input layer. The outputs of this layer are simply the feature values themselves.
- The outputs from the input layer are passed to the first hidden layer. Each neuron in this hidden layer calculates its weighted sum z and applies its activation function to produce its output a.
- The outputs (a) from the first hidden layer serve as the inputs for the second hidden layer, and this process repeats for all subsequent hidden layers.
- Finally, the outputs from the last hidden layer are passed to the output layer. The neurons in the output layer perform their calculations. The activation function used here depends on the task (e.g., Sigmoid for binary classification, Softmax for multi-class classification, or no activation for regression).
- The final value(s) from the output layer constitute the network's prediction, ŷ.
Quantifying Mistakes: The Role of the Loss Function
After the forward pass produces a prediction ŷ, the next step is to evaluate how "wrong" that prediction is. This is accomplished using a loss function (also known as a cost or error function). A loss function takes the network's prediction (ŷ) and the true target value (y) and computes a single scalar value that quantifies the error. The overarching goal of the entire training process is to find the set of weights and biases that minimizes this loss value.
The choice of loss function is critical and depends on the nature of the problem being solved (e.g., regression or classification).
Become the Ai engineer who can design, build, and iterate real AI products, not just demos with an IIT Roorkee CEC Certification
AI Engineering Course Advanced Certification by IIT-Roorkee CEC
A hands on AI engineering program covering Machine Learning, Generative AI, and LLMs - designed for working professionals & delivered by IIT Roorkee in collaboration with Scaler.
Enrol NowCommon Loss Functions for Different Tasks
-
Mean Squared Error (MSE) for Regression: Used when predicting continuous values, such as the price of a house or the temperature. It measures the average of the squares of the errors. Squaring the error ensures that the value is always positive and penalizes larger errors more heavily.
- Formula: L(y, ŷ) = (1/n) * Σ(ŷᵢ - yᵢ)²
- Where n is the number of training examples, ŷᵢ is the prediction for the i-th example, and yᵢ is the true value.
-
Cross-Entropy Loss for Classification: Used when predicting a class label from a set of discrete possibilities. It measures the performance of a classification model whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability diverges from the actual label.
- Formula (Binary Cross-Entropy): L(y, ŷ) = -[y * log(ŷ) + (1 - y) * log(1 - ŷ)]
- Here, y is the true label (0 or 1) and ŷ is the predicted probability of the class being 1.
The output of the loss function is the signal that guides the learning process. The next step is to use this signal to figure out how to adjust the weights and biases to reduce the loss.
The Optimization Engine: Gradient Descent
Once we have a measure of the network's error, we need a mechanism to systematically adjust the weights and biases to minimize it. This is the job of the optimization algorithm. The most fundamental and widely used optimizer is Gradient Descent. It is an iterative algorithm that finds the local minimum of a function—in this case, the loss function.
Imagine the loss function as a vast, hilly landscape where the altitude at any point represents the loss for a given set of weights. Our goal is to find the lowest point in this landscape. Gradient Descent does this by starting at a random point (randomly initialized weights) and taking small steps in the direction of the steepest descent.
Understanding Gradients and the Derivative
The "direction of steepest descent" is found using calculus. The gradient of the loss function, denoted ∇L, is a vector containing the partial derivatives of the loss L with respect to each weight w and bias b in the network (∂L/∂w₁, ∂L/∂w₂, ...).
- A derivative dy/dx measures the instantaneous rate of change of a function y with respect to a variable x. It tells us how much y will change for a tiny change in x.
- A partial derivative ∂L/∂w tells us how much the total loss L will change for a tiny change in a specific weight w, while holding all other weights constant.
- The gradient ∇L points in the direction of the steepest ascent of the loss function. Therefore, to minimize the loss, we must move in the direction opposite to the gradient.
The Gradient Descent Algorithm
The core of Gradient Descent is its update rule. After calculating the gradient for each parameter, the algorithm updates the parameter by taking a small step in the negative gradient direction.
The update rule for a single weight w is:
w_new = w_old - α * (∂L/∂w)
Let's break down this crucial formula:
- w_old: The current value of the weight.
- w_new: The updated value of the weight for the next iteration.
- α (alpha): The learning rate. This is a small positive hyperparameter (e.g., 0.01, 0.001) that controls the size of the step we take. Choosing an appropriate learning rate is critical:
- If α is too small, the training process will be very slow.
- If α is too large, the algorithm might overshoot the minimum and fail to converge, with the loss value oscillating or even diverging.
- ∂L/∂w: The partial derivative of the loss with respect to the weight w. This is the calculated gradient component that tells us the direction in which to adjust the weight.
This update rule is applied simultaneously to every weight and bias in the entire network during each training step.
Variants of Gradient Descent
The term "Gradient Descent" can refer to a few different variants, which differ in how much data is used to compute the gradient of the loss function at each step.
| Variant | Description | Pros | Cons |
|---|---|---|---|
| Batch Gradient Descent | Calculates the gradient using the entire training dataset for each parameter update. | - Guaranteed to converge to the global minimum for convex loss surfaces and to a local minimum for non-convex surfaces. - Stable, non-noisy updates. | - Very slow and computationally expensive for large datasets. - Requires the entire dataset to fit in memory. |
| Stochastic Gradient Descent (SGD) | Performs a parameter update for each training example. The gradient is computed using only a single sample. | - Much faster computationally. - The noisy updates can help escape shallow local minima. | - High variance in parameter updates, leading to a noisy convergence path. - Can overshoot the minimum and never fully converge. |
| Mini-Batch Gradient Descent | A compromise between the two. Performs an update for every mini-batch of training examples (e.g., 32, 64, 128 samples). | - Provides a balance between the stability of Batch GD and the efficiency of SGD. - Allows for highly optimized matrix operations. | - Introduces an additional hyperparameter: the batch size. |
In modern deep learning, Mini-Batch Gradient Descent is the most commonly used variant due to its efficiency and stable convergence properties.
How Scaler Transformed Careers in Different Fields
Scaler learners achieved 2.5x salary growth with average post-Scaler CTC reaching ₹23L.
Google
Amazon
Microsoft
Flipkart
Adobe1200+ moreThe Engine's Fuel: Backpropagation
We now understand that we need the gradient ∂L/∂w for every weight in the network to perform the Gradient Descent update. But how do we calculate these millions of gradients efficiently? A naive approach would be computationally intractable.
This is where the Backpropagation algorithm comes in. Backpropagation, short for "backward propagation of errors," is a highly efficient algorithm for computing the gradients needed to train a neural network. It is essentially a practical application of the chain rule from calculus to the nested structure of a neural network.
The Chain Rule in Calculus: A Quick Refresher
The chain rule is used to find the derivative of a composite function. If a variable L depends on a, which in turn depends on z, which in turn depends on w, the chain rule states that the derivative of L with respect to w is the product of the individual derivatives:
∂L/∂w = (∂L/∂a) * (∂a/∂z) * (∂z/∂w)
A neural network is a massive composite function. The final loss L depends on the output of the final layer, which depends on its weights and the outputs of the previous layer, and so on, all the way back to the first layer's weights. Backpropagation applies the chain rule systematically to "un-peel" these layers of functions and calculate each weight's contribution to the final error.
The Backpropagation Algorithm in Action
Backpropagation consists of a forward pass followed by a backward pass.
-
Forward Pass: As described earlier, an input is fed into the network. At each layer, the pre-activation z and the activation a are calculated and stored for every neuron. This continues until the final prediction ŷ is made, and the overall loss L is calculated.
-
Backward Pass: This is where the gradients are computed, starting from the end of the network and moving backward.
- At the Output Layer: The algorithm first computes the gradient of the loss L with respect to the output layer's activations (∂L/∂a). Then, using the chain rule, it calculates the gradient with respect to the output layer's weights.
- Propagating to Hidden Layers: The algorithm then propagates the error backward to the last hidden layer. It calculates the gradient of the loss with respect to this layer's activations by using the gradients already computed for the output layer. This process is repeated, moving backward one layer at a time.
- At Each Layer: For any given layer, backpropagation calculates the gradient of the loss with respect to that layer's weights (∂L/∂w) and biases (∂L/∂b) by using the gradient that was passed back from the layer in front of it.
By reusing the gradient calculations from the subsequent layer, backpropagation avoids redundant computations and makes the process of training deep networks with millions of parameters computationally feasible. It is the cornerstone algorithm that enabled the deep learning revolution.
The Complete Neural Network Training Process Loop
By combining all these concepts, we can define the full, iterative neural network training process. This loop forms the basis of how nearly all deep learning models are trained.
-
Initialization: The process begins by initializing all the weights w and biases b in the network. This is typically done using small random numbers to break symmetry and ensure that different neurons in the same layer learn different features.
-
Training Loop: The algorithm then enters a loop that runs for a predefined number of epochs. An epoch is one complete pass through the entire training dataset. Inside this loop, the following steps are repeated for each mini-batch of data: a. Forward Propagation: The mini-batch of input data is passed through the network to generate predictions ŷ. All intermediate values (z and a for each neuron) are cached for use in the backward pass. b. Loss Calculation: The loss function L is used to compute the error between the predictions ŷ and the true labels y. c. Backward Propagation: The backpropagation algorithm is executed to compute the gradient of the loss function with respect to every weight and bias in the network (∂L/∂w and ∂L/∂b). d. Weight Update: The optimizer (e.g., Gradient Descent) updates all weights and biases using the computed gradients and the learning rate α. w = w - α * ∂L/∂w b = b - α * ∂L/∂b
-
Repetition: This inner loop over mini-batches continues until the entire dataset has been seen (completing one epoch). The outer loop over epochs continues until a stopping criterion is met, such as the model's performance on a separate validation dataset no longer improving, or a fixed number of epochs is completed.
Here is a simplified pseudo-code representation of the process:
Advanced Optimizers and Hyperparameters
While Mini-Batch Gradient Descent is the foundational optimizer, several more advanced algorithms have been developed to improve its performance. These optimizers adapt the learning rate during training and help accelerate convergence.
Turn Learning into Career Growth
Momentum, RMSprop, and Adam
- Momentum: This method helps accelerate SGD in the relevant direction and dampens oscillations. It adds a fraction of the previous update vector to the current one, simulating "momentum" that pushes the update in a consistent direction.
- RMSprop (Root Mean Square Propagation): This optimizer adapts the learning rate for each parameter separately. It divides the learning rate by an exponentially decaying average of squared gradients, effectively decreasing the learning rate for parameters with large gradients and increasing it for those with small gradients.
- Adam (Adaptive Moment Estimation): This is arguably the most popular and effective optimization algorithm. It combines the ideas of both Momentum and RMSprop. It stores an exponentially decaying average of past squared gradients (like RMSprop) and an exponentially decaying average of past gradients (like Momentum). It is often the default choice for training deep neural networks.
Key Hyperparameters in Training
The entire training process is governed by several critical hyperparameters—settings that are not learned by the network but are configured by the engineer before training begins.
- Learning Rate (α): As discussed, this controls the step size of the parameter updates.
- Number of Epochs: The number of times the entire training dataset is passed through the network.
- Batch Size: The number of training examples used in one iteration of the training loop.
- Network Architecture: The number of hidden layers and the number of neurons in each layer.
Conclusion: Learning as a Process of Iterative Refinement
The process of a neural network learning is not an act of magic but a principled, mathematical procedure of iterative refinement. It begins with a randomly configured model that makes poor predictions. Through a continuous loop, it performs a forward pass to see the input and make a guess, calculates its error using a loss function, and then uses backpropagation to efficiently determine how each of its millions of parameters contributed to that error. Finally, an optimizer like Gradient Descent uses this information to make small, intelligent adjustments to those parameters—the weights and biases. Repeated over many epochs, this cycle transforms the network from an uninformed system into a powerful model capable of solving complex tasks.
FAQ
What is the difference between an epoch, a batch, and an iteration?
- Epoch: One full pass through the entire training dataset. If your dataset has 10,000 images, one epoch is completed when the model has seen all 10,000 images.
- Batch (or Mini-Batch): A small subset of the training dataset. Instead of processing the entire dataset at once, the data is divided into smaller batches.
- Iteration: A single update of the model's parameters. In mini-batch gradient descent, one iteration corresponds to processing one batch of data. If the dataset has 10,000 images and the batch size is 100, then one epoch consists of 100 iterations (10,000 / 100).
Why can't we just solve for the optimal weights mathematically instead of using gradient descent?
For a very simple model like linear regression, it is possible to solve for the optimal weights directly using a closed-form solution (the Normal Equation). However, neural networks are highly non-linear and non-convex functions with millions of parameters. This creates an incredibly complex loss landscape with many local minima, saddle points, and plateaus. There is no analytical, closed-form equation to find the global minimum for such a complex function. Therefore, we must use an iterative optimization technique like gradient descent to navigate this landscape and find a sufficiently good set of weights.
What happens if the learning rate is too high or too low?
- Too Low: The model will learn very slowly, requiring many epochs to converge to a good solution. It may also get stuck in a shallow local minimum because it lacks the "momentum" to overcome small humps in the loss landscape.
- Too High: The model's training will be unstable. The parameter updates will be so large that they will consistently overshoot the minimum of the loss function. This can cause the loss to oscillate wildly or even diverge (increase indefinitely), preventing the model from learning anything.
What is the "vanishing gradient" problem?
The vanishing gradient problem is a major challenge in training very deep neural networks, particularly those using activation functions like Sigmoid or Tanh. During backpropagation, gradients are calculated by multiplying derivatives from layer to layer (via the chain rule). If these derivatives are consistently small (less than 1), their product can become exponentially small as it is propagated backward through many layers. Consequently, the gradients for the early layers of the network become nearly zero, causing their weights to update very slowly or not at all. This effectively stops the early layers from learning. The widespread adoption of the ReLU activation function, whose derivative is a constant 1 for positive inputs, was a key breakthrough in mitigating this problem.
TEST_SECTION
This is a test section to verify if the script is working.
