Evaluation Metrics
Choose the right score for the job: accuracy, precision, recall, R-squared.
A model needs a metric to say how good it is, and the right metric depends on the task.
Regression metrics
- MAE / MSE — average prediction error.
- R² — fraction of variance explained (1.0 is perfect).
Classification metrics
- Accuracy — fraction correct. Misleading on imbalanced data.
- Precision — of predicted positives, how many were right.
- Recall — of actual positives, how many were found.
Example
from sklearn.metrics import accuracy_score, precision_score, recall_score
y_true = [1, 0, 1, 1, 0, 1]
y_pred = [1, 0, 0, 1, 0, 1]
print(accuracy_score(y_true, y_pred)) # 0.833
print(precision_score(y_true, y_pred)) # 1.0
print(recall_score(y_true, y_pred)) # 0.75When to use it
- A fraud-detection engineer tracks recall (not accuracy) because missing a real fraud (false negative) is far more costly than a false alarm.
- An ML team reports both precision and F1-score for a medical diagnosis model because class imbalance makes raw accuracy misleading.
- A regression model's performance is reported as RMSE in the original dollar units rather than R2, so non-technical stakeholders can interpret the error magnitude.
More examples
Classification: accuracy, precision, recall
Computes all four common classification metrics on a held-out test set, enabling comparison across dimensions beyond simple accuracy.
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2)
model = RandomForestClassifier().fit(X_tr, y_tr)
preds = model.predict(X_te)
print('Accuracy:', accuracy_score(y_te, preds).round(3))
print('Precision:', precision_score(y_te, preds).round(3))
print('Recall:', recall_score(y_te, preds).round(3))
print('F1:', f1_score(y_te, preds).round(3))Regression: MAE, RMSE, R2
Reports three complementary regression metrics: MAE for average absolute error, RMSE for penalising large errors, and R2 for explained variance.
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.linear_model import LinearRegression
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
import numpy as np
X, y = make_regression(n_samples=200, noise=20, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y)
model = LinearRegression().fit(X_tr, y_tr)
preds = model.predict(X_te)
print('MAE:', mean_absolute_error(y_te, preds).round(2))
print('RMSE:', np.sqrt(mean_squared_error(y_te, preds)).round(2))
print('R2:', r2_score(y_te, preds).round(3))ROC-AUC for ranking quality
Computes ROC-AUC using predicted probabilities rather than hard labels, measuring the model's ability to rank positive examples above negative ones regardless of threshold.
from sklearn.metrics import roc_auc_score
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2)
model = LogisticRegression(max_iter=5000).fit(X_tr, y_tr)
probs = model.predict_proba(X_te)[:, 1]
print('ROC-AUC:', roc_auc_score(y_te, probs).round(4))
Discussion