Module 04 · Applied project
Bank Subscription Prediction Lab
Build and evaluate a complete tabular classification workflow using UCI Bank Marketing bank-additional-full.csv. Predict whether a client subscribes to a term deposit (y) using only information available before the call. The UCI documentation warns that duration is highly predictive but is only known after the call; exclude it from the realistic feature set. Preserve the original row order because this release is ordered by date, then make chronological train/validation/test splits. Compare a prior baseline, logistic regression and a tree ensemble under one consistent evaluation protocol.
Implementation plan
Build it in 7 steps
- 01 Download the UCI Bank Marketing archive and use
bank-additional-full.csv. Record the file name, checksum if practical, row count, column names, target distribution and the UCI note aboutduration. - 02 Preserve supplied row order and create chronological partitions: first 70% for training, next 15% for validation and final 15% for testing. Do not shuffle rows before splitting. Report the positive-class share in each partition.
- 03 Define
yas the subscription outcome and excludedurationfromX. Inspect missing values and data types. Separate numeric and categorical columns; treat thepdayssentinel value deliberately and document your choice. - 04 Build a
ColumnTransformerwith numeric imputation and scaling plus categorical imputation and one-hot encoding. Put it inside aPipelineso each model and validation fold learns preprocessing only from its training rows. - 05 Fit a
DummyClassifier(strategy='prior'), a regularizedLogisticRegressionpipeline and aRandomForestClassifierpipeline. Use training-only cross-validation or a time-aware validation procedure to select sensible hyperparameters; do not inspect test labels during selection. - 06 Choose a decision threshold using validation probabilities and a stated operational trade-off. Evaluate the final model once on the untouched test partition with a confusion matrix, precision, recall, F1, ROC-AUC and average precision; compare candidates on the same test rows.
- 07 Run permutation importance on held-out data, inspect false positives and false negatives, compare performance across sufficiently sized campaign or age bands, and write a concise report describing leakage controls, performance, limitations and monitoring needs.
What to submit
Deliverables
- A reproducible Python project or notebook with clear environment and dataset instructions.
- A data audit recording file identity, row count, target balance, types, missing values and the excluded
durationfeature. - A chronological split manifest and a description of the prediction moment and valid feature set.
- A preprocessing and model pipeline for the prior baseline, logistic regression and random forest candidates.
- A model comparison table containing precision, recall, F1, ROC-AUC and average precision on the same final test rows.
- A confusion matrix and precision-recall or ROC plot for the selected model at a threshold chosen on validation data.
- A permutation-importance plot and a brief false-positive / false-negative and subgroup error analysis.
- A short model card documenting intended use, limitations, potential leakage, subgroup caveats and what should be monitored before real-world use.
Quality gate
Before you finish
- dataThe split preserves supplied time order, and `duration` is excluded because it is only available after the call.
- leakageImputers, scalers and encoders are fitted within training pipelines; validation and test rows are only transformed.
- baselineEvery model is compared with the same prior baseline on identical final test rows.
- evaluationThe threshold is chosen from validation data, while the final test partition is used once for the report.
- interpretationPermutation importance and subgroup errors are presented as diagnostics, not causal explanations or proof of fairness.
Follow the module
Lessons for this project
- 04.01 ML Problem Framing & Learning TypesTranslate a vague prediction idea into a concrete machine-learning task with a defined target, available features and evaluation objective.
- 04.02 Data Splits, Baselines & LeakageSeparate model fitting from model selection and final evaluation, and identify leakage before it inflates a metric.
- 04.03 Feature Preparation & PipelinesPrepare mixed numeric and categorical columns with imputers, scaling and encoding inside a single scikit-learn pipeline.
- 04.04 Linear Models & RegularizationUse linear models as strong tabular baselines and control overfitting with Ridge, Lasso and Elastic Net penalties.
- 04.05 Logistic Regression & ProbabilitiesTurn a linear score into a class probability and understand how logistic regression separates two classes.
- 04.06 Classification Metrics & Decision ThresholdsEvaluate classifiers with confusion matrices, precision, recall, F1, ROC-AUC and average precision, especially when classes are imbalanced.
- 04.07 k-Nearest NeighborsUnderstand lazy learning and how distance, feature scale and neighbor count affect KNN predictions.
- 04.08 Naive Bayes ClassifiersApply Bayes' rule to classification and understand the conditional-independence assumption behind Naive Bayes.
- 04.09 Decision TreesUnderstand how decision trees partition feature space and how depth and leaf constraints control overfitting.
- 04.10 Bagging & Random ForestsExplain how bootstrap aggregation and random feature subsets produce a more stable ensemble of decision trees.
- 04.11 Boosting for Tabular DataUnderstand gradient boosting as a stage-wise additive model and tune its capacity and learning rate carefully.
- 04.12 Support Vector MachinesUnderstand maximum-margin classification and how kernels allow support vector machines to model nonlinear boundaries.
- 04.13 Clustering with K-MeansGroup observations without target labels and understand K-means initialization, inertia, scaling and evaluation limits.
- 04.14 Dimensionality Reduction & PCAUse principal component analysis to represent correlated numeric features with orthogonal directions that capture decreasing amounts of variance.
- 04.15 Cross-Validation & Hyperparameter SearchEstimate model performance across folds and tune hyperparameters using a split strategy that matches the observations.
- 04.16 Model Interpretation & Error AnalysisInspect model reliance, probability quality and error patterns without confusing feature importance with causation.
How to assess your work
Project rubric
Problem and data contract
The target, prediction moment and allowed feature set are clearly defined; duration is excluded.
The report explains temporal ordering, class balance, sentinel values and limits of historical campaign data.
Preprocessing and leakage control
Mixed-type preprocessing is contained in a pipeline and fitted only on training rows.
Cross-validation and model search reuse the same pipeline without leaking validation or test information.
Model comparison
A prior baseline, logistic regression and a tree ensemble are compared on the same rows.
The comparison discusses performance, cost and model complexity rather than declaring a winner from one metric.
Evaluation quality
Precision, recall, F1, ROC-AUC and average precision are reported with a confusion matrix.
The operating threshold is justified on validation data and final test evaluation is performed once.
Interpretation and limits
Permutation importance and several errors are inspected; at least three limitations are documented.
The report examines sufficiently sized subgroups, discusses calibration and avoids unsupported causal or fairness claims.
Reproducibility
The data source, environment, random settings and run instructions are recorded.
A clean run reproduces tables, plots and the model card from the original dataset.