Model Interpretation & Error Analysis

Inspect model reliance, probability quality and error patterns without confusing feature importance with causation.

By the end of this lesson

You will be able to

Explain permutation feature importance and its dependence on a scoring metric.
Distinguish feature importance from causal effect.
Inspect false positives, false negatives and subgroup metrics.
Recognize when predicted probabilities need calibration.

Permutation importance asks what the model relies on

Permutation importance shuffles one feature column and measures how much a fitted model's score degrades. If shuffling a feature damages the score, the model relied on information in that column. Measure importance on held-out data with a metric relevant to the task. Correlated features can share or mask importance, and the result is specific to the model, sample and metric. It is not a causal effect or a ranking of what matters in the world.

from sklearn.inspection import permutation_importance

result = permutation_importance(
    fitted_pipeline, X_valid, y_valid,
    scoring="average_precision",
    n_repeats=20,
    random_state=42,
    n_jobs=-1,
)

Inspect the errors before declaring success

Review false positives and false negatives, then compare error rates across meaningful groups such as campaign month or customer age bands. Use enough observations per group and report uncertainty; small groups can produce unstable metrics. Check calibration when probability estimates drive decisions: among examples assigned probability near 0.7, roughly 70% should be positive over a sufficiently large comparable sample. Performance differences may reflect data collection, historical choices or distribution shifts. Do not claim a feature caused the outcome based on importance or correlation.

Apply what you learned

Exercise

Write a model error report

Choose a fitted classifier and produce a short report that explains its dependencies and important failure patterns.

Calculate permutation importance on validation or test data using a task-relevant score.
Inspect false positives and false negatives and identify recurring patterns.
Compare precision or recall across at least two sufficiently sized groups.
Inspect a calibration curve if probabilities will inform a decision.
Write three limitations: causality, data/population shift and subgroup performance.

Done when

The report uses held-out data, names its metric and sample limitations, treats importance as model reliance rather than causality and identifies concrete error patterns.

Free references

Study resources

2 links

01
Permutation Feature Importance Article
Read the definition and example. Note why importance changes with the scoring metric and evaluation data.

Done
02
Probability Calibration Article
Read calibration curves and calibrated classifiers. Focus on probability quality rather than only label accuracy.

Done

Check your understanding

Self-test

2 questions

Question 1

What does permutation importance measure?
Question 2

Does high feature importance mean the feature causes the target?

Completion check

Move on when

You can use held-out permutation importance, inspect errors and explain why model reliance and subgroup metrics must be interpreted cautiously.

Back to module