Performance Metrics in Machine Learning Explained

You trained a model. Your notebook says 96 percent accuracy. You push it to production, and within a week the engineering team is pinging you on Slack: "Your model is taking 800 milliseconds per prediction and we need it under 100. Also, the infra cost is 4x what we budgeted." Welcome to the part of machine learning that most courses skip entirely.

Performance metrics in machine learning fall into two completely different families, and mixing them up or ignoring one of them is the fastest way to build something that works in a notebook and fails in production. The first family measures prediction quality: how correct your model is. Accuracy, precision, recall, F1, RMSE, R-squared. These are the metrics you learn in every course and tutorial. The second family measures runtime behavior: how fast and how cheaply your model serves predictions. Latency, throughput, model size, cost per inference. These are the metrics that determine whether your model actually ships.

This page is a quick reference for both families. Every metric is a card: the formula, a one-line meaning, when to use it, and the trap that catches people in interviews and production reviews. If you want full walkthroughs with worked examples and the reasoning behind each metric, the evaluation metrics in machine learning guide is the deep-dive companion. This page is the cheat sheet you keep open during a project and review before an interview.

Model Performance: Two Kinds of "Fast and Good"

Most ML courses teach one family of performance metrics: the ones that measure prediction quality. Accuracy, precision, recall, F1, RMSE, R-squared. These answer the question "how correct is my model?"

There is a second family that almost no course covers: runtime performance metrics. Inference latency, throughput, model size, cost per prediction. These answer the question "can my model actually run in production?"

Both families matter. A model with 97 percent accuracy that takes 800 milliseconds per prediction is the wrong choice for a real-time fraud system that needs sub-100ms responses. A model with 93 percent accuracy that responds in 15 milliseconds is the one that ships. This page covers both families because production teams track both, and interviews at product companies increasingly ask about the second family to separate candidates who have only built notebooks from candidates who understand deployed ML.

Regression Metric Cards

MAE (Mean Absolute Error)

| Formula | (1/N) × Σ |yᵢ − ŷᵢ| | | :---- | :---- | | One-line meaning | Average absolute difference between predicted and actual values | | Use when | You want an interpretable error metric in the same units as your target variable | | Trap | MAE treats all errors linearly. A prediction that is off by 100 contributes the same as ten predictions off by 10. If large errors are disproportionately bad, use RMSE. | | scikit-learn | mean_absolute_error |

MSE (Mean Squared Error)

RMSE (Root Mean Squared Error)

R² (Coefficient of Determination)

Adjusted R²

MAPE (Mean Absolute Percentage Error)

| Formula | (100%/N) × Σ |(yᵢ − ŷᵢ) / yᵢ| | | :---- | :---- | | One-line meaning | Average percentage error relative to actual values | | Use when | You want to communicate error in percentage terms to non-technical stakeholders | | Trap | Undefined when actual values are zero. Asymmetric: overpredicting and underpredicting by the same amount give different MAPE values. Consider SMAPE or MASE as alternatives (Hyndman & Koehler, 2006). |

Measuring Reliably: Validation Reminders

The best metric computed on the wrong split is worse than a mediocre metric computed correctly. Three rules:

• Never evaluate on training data. Always hold out a test set that the model has never seen during training or hyperparameter tuning.
• Use k-fold cross-validation for small datasets where a single train-test split is unreliable. The k-fold cross-validation guide covers the implementation and the common mistake of leaking information between folds.
• Use stratified splits for classification with imbalanced classes to ensure each fold preserves the class distribution.
• scikit-learn provides every metric above as a one-liner. The classification_report function gives precision, recall, and F1 for all classes in a single call. The cross_val_score function handles k-fold evaluation with any scorer.

Build metric evaluation into your workflow from day one: Scaler's free Supervised Learning course covers classification and regression metrics with hands-on exercises using real datasets.

Hiring Partners:

GoogleAmazonMicrosoftFlipkartAdobe1200+ more

The Production Layer: Latency, Throughput, and Size

This is the section that separates notebook practitioners from production engineers. Once a model is deployed, teams track metrics that have nothing to do with prediction quality but everything to do with whether the system can serve real users at real scale.

Inference Latency (P50, P95, P99)

What it measures: How long a single prediction takes, from request received to response sent.

Why percentiles matter: A P50 latency of 25ms sounds good. But if P99 is 2,400ms, one in a hundred users is waiting over two seconds. Production teams set SLOs (Service Level Objectives) on P95 or P99, not on averages.

Reference benchmarks for context:

• Real-time user-facing systems (search ranking, recommendation, fraud): P95 target of 50–100ms is standard for web applications. Google's published research on user-facing latency shows that delays above 100ms begin to measurably impact user engagement (Google, "Speed Matters," 2009; Aberdeen Group research found that a 1-second delay in page load time reduces conversions by 7 percent).
• Batch scoring (offline predictions, nightly jobs): Seconds to minutes per prediction are acceptable.
• Edge and mobile inference: Sub-30ms for real-time applications like on-device speech recognition.

What drives latency: Model size, input preprocessing complexity, feature computation at serving time, network overhead between the application and the model server, and hardware (CPU vs GPU vs TPU).

Throughput (Queries Per Second)

What it measures: How many predictions the system can serve per second.

Why it matters: A model that takes 10ms per prediction on a single thread serves 100 QPS. If your application receives 5,000 requests per second at peak, you need horizontal scaling (more instances) or a faster model.

Reference context: Typical production ML serving infrastructure targets 1,000–10,000 QPS per instance for lightweight models (logistic regression, small decision trees) and 100–500 QPS for deep learning models on GPU, depending on model size and input complexity. TensorFlow Serving and TorchServe both publish benchmark suites for their respective frameworks (TensorFlow Serving benchmarks; TorchServe benchmarks).

Model Size and Memory Footprint

What it measures: The disk size of the serialized model and the RAM required to load and run it.

Why it matters: A 2GB model requires expensive GPU instances to serve. A 50MB model runs on CPU and fits on mobile devices. Model size directly determines infrastructure cost.

Reference context:

• scikit-learn models (logistic regression, random forests): typically 1–100MB serialized
• Small deep learning models (distilled BERT, MobileNet): 50–200MB
• Large language models (GPT-2, Llama 7B): 500MB to 14GB+
• Quantized models (INT8 instead of FP32): roughly 4x size reduction with 1–3 percent accuracy loss, as documented in Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference" (CVPR 2018)

Cost Per Prediction

What it measures: The infrastructure cost of serving one prediction, combining compute time, memory, and any API costs.

Why it matters: At 10 million predictions per day, the difference between ₹0.001 and ₹0.01 per prediction is ₹90,000 per day. Production teams track this because model serving is an ongoing operational cost, not a one-time training expense.

The Accuracy-Latency Trade-Off (and Monitoring Over Time)

Here is the decision production teams make that courses never discuss: should we sacrifice accuracy to gain latency?
The answer, in most deployed systems, is yes. Here is why.

A BERT-large model might give you 97 percent accuracy on text classification but take 400ms per prediction. Distilling it into a DistilBERT model brings accuracy to 95.5 percent and latency down to 90ms. For a real-time content moderation system serving millions of requests, that 1.5 percent accuracy drop is worth the 4.4x speed improvement because the user experience degradation from 400ms latency costs more than the 1.5 percent additional misclassification rate.

The standard techniques for this trade-off are:

• Knowledge distillation: Train a smaller model to mimic a larger model's outputs. Hinton et al. introduced this in "Distilling the Knowledge in a Neural Network" (2015). DistilBERT, for example, retains 97 percent of BERT's performance at 60 percent of the size and 40 percent less inference time (Sanh et al., 2019).
• Quantization: Reduce model weights from 32-bit floating point to 8-bit integers. Cuts model size by 4x with typically 1–3 percent accuracy loss. Supported natively by TensorFlow Lite and PyTorch.
• Pruning: Remove low-importance weights from a trained model. Can reduce model size by 50–90 percent depending on the model and the acceptable accuracy threshold.

Monitoring Over Time: Metric Decay

The metrics you measured during development will degrade after deployment. This is not a failure of your model. It is a fact about how data distributions change.

Data drift means the input distribution at serving time diverges from the training distribution. A fraud model trained on 2023 transaction patterns sees new patterns in 2026.
Concept drift means the relationship between inputs and outputs changes. A pricing model trained in a low-inflation economy produces bad predictions in a high-inflation one.

Production teams monitor both families of metrics continuously:

• Quality metrics on a sampled subset of predictions where ground truth labels are available (usually with a delay of hours to days)
• Runtime metrics on every prediction in real time (latency percentiles, error rates, throughput)
• Drift metrics comparing serving-time input distributions to training-time distributions using statistical tests like KL divergence, Population Stability Index (PSI), or Wasserstein distance

The MLOps pipeline guide covers how to build monitoring infrastructure that catches drift before it becomes a business problem.
For a broader view of where performance metrics fit in the ML lifecycle, the machine learning topics hub connects this reference to model building, evaluation, and deployment resources.

Quality metrics get models built. Performance metrics get them shipped. Learn both in Scaler's AI and ML Program from evaluation fundamentals to production monitoring, with mentor-led guidance across the full ML lifecycle.