Classification Metrics & Decision Thresholds

Start from the confusion matrix

For a binary classifier, the confusion matrix counts true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). Precision is TP / (TP + FP): among predicted positives, how many were correct? Recall is TP / (TP + FN): among actual positives, how many were found? Specificity is TN / (TN + FP). F1 is the harmonic mean of precision and recall. Accuracy can look high when the majority class dominates, even if the model rarely finds the positive class.

from sklearn.metrics import classification_report, confusion_matrix

labels = classifier.predict(X_valid)
print(confusion_matrix(y_valid, labels))
print(classification_report(y_valid, labels, zero_division=0))

Ranking metrics and operating thresholds answer different questions

ROC-AUC measures how well scores rank positives above negatives across thresholds. The precision-recall curve focuses on positive-class retrieval and is often more informative when positives are rare. Average precision summarizes the precision-recall curve. These metrics do not select the business threshold for you. Use validation probabilities to choose a threshold that matches relative error costs or capacity, record that choice, then evaluate once on the untouched test data.