k-Nearest Neighbors

Understand lazy learning and how distance, feature scale and neighbor count affect KNN predictions.

By the end of this lesson

You will be able to

Describe the neighbor-voting procedure.
Explain why KNN is sensitive to feature scaling.
Describe how k affects bias and variance.
Recognize computational cost and the curse of dimensionality.

Prediction comes from nearby examples

For a new row, KNN calculates distances to stored training rows, selects the k nearest neighbours and predicts from their labels. In classification, it uses a vote; distance weighting gives closer neighbours more influence. The method has little explicit training work, but prediction can be expensive because distances to many stored examples may need to be calculated.

from sklearn.neighbors import KNeighborsClassifier

knn = Pipeline([
    ("prepare", preprocess),  # numeric columns are scaled
    ("model", KNeighborsClassifier(n_neighbors=7, weights="distance")),
])
knn.fit(X_train, y_train)

Distance is meaningful only when scales are considered

If one feature ranges from 0 to 100,000 and another from 0 to 1, the first can dominate Euclidean distance. Scaling numeric features is therefore important for KNN. Increasing k generally smooths local decisions and can increase bias; a very small k can be noisy and overfit. As dimensionality grows, distances can become less discriminative. KNN is a useful benchmark, but it is not automatically a good choice for wide or very large datasets.

Apply what you learned

Exercise

Tune the neighborhood size

Compare several KNN neighborhood sizes on a small classification dataset using a scaled pipeline.

Create a preprocessing pipeline that scales numeric features.
Fit KNN models with k values such as 3, 7, 15 and 31.
Compare validation F1 or average precision for the same split.
Compare uniform voting with distance weighting.
Explain one reason many features or many training rows can make KNN difficult to use.

Done when

Every KNN variant receives the same preprocessing and validation rows, and the comparison explains the impact of k and scale.

Free references

Study resources

2 links

01
K-nearest neighbors, Clearly Explained Video
Watch the individual K-nearest-neighbors explanation. Focus on neighbor voting and the role of k.

Done
02
Nearest Neighbors Article
Read KNeighborsClassifier and the practical notes on distance metrics and scaling.

Done

Check your understanding

Self-test

2 questions

Question 1

What happens when k is very small?
Question 2

Why is scaling especially important for KNN?

Completion check

Move on when

You can fit KNN through a scaled pipeline and explain how k, distance and dimensionality affect behavior.

Mark passed and continue Back to module