K-Nearest Neighbors
Classify a point by the majority vote of its closest neighbours.
Syntax
KNeighborsClassifier(n_neighbors=5)KNN makes no model up front. To classify a new point it finds the k closest training points and takes their majority label.
Key points
- Small
kis sensitive to noise; largeksmooths boundaries. - It relies on distance, so features must be scaled first.
Example
from sklearn.neighbors import KNeighborsClassifier
import numpy as np
X = np.array([[1], [2], [3], [7], [8], [9]])
y = np.array([0, 0, 0, 1, 1, 1])
model = KNeighborsClassifier(n_neighbors=3)
model.fit(X, y)
print(model.predict([[2.5]])) # [0]
print(model.predict([[7.5]])) # [1]When to use it
- A recommendation engine uses KNN to find the five most similar users to a target customer based on their purchase vectors.
- A data scientist uses KNN as a fast non-parametric baseline on a small image classification task before trying deeper models.
- A retailer applies KNN regression to predict a product's expected price based on the prices of the k most similar items in the catalogue.
More examples
KNN classifier on Iris
Trains a 5-NN classifier after scaling, which is essential for KNN since it is distance-based and sensitive to feature magnitude.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X, y = load_iris(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
scaler = StandardScaler()
X_tr_s = scaler.fit_transform(X_tr)
X_te_s = scaler.transform(X_te)
knn = KNeighborsClassifier(n_neighbors=5).fit(X_tr_s, y_tr)
print('Accuracy:', knn.score(X_te_s, y_te).round(3))Choose k with cross-validation
Evaluates multiple values of k using 5-fold cross-validation to identify the optimal number of neighbours without touching the test set.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_digits
import numpy as np
X, y = load_digits(return_X_y=True)
for k in [1, 3, 5, 10, 20]:
knn = KNeighborsClassifier(n_neighbors=k)
cv = cross_val_score(knn, X, y, cv=5).mean()
print(f'k={k}: CV accuracy={cv:.3f}')KNN regression
Uses KNN for regression by averaging the target values of the 7 nearest neighbours, showing the algorithm works equally well for continuous outputs.
from sklearn.neighbors import KNeighborsRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
X, y = make_regression(n_samples=300, n_features=5, noise=10, random_state=2)
X_tr, X_te, y_tr, y_te = train_test_split(X, y)
knn = KNeighborsRegressor(n_neighbors=7).fit(X_tr, y_tr)
print('R2:', r2_score(y_te, knn.predict(X_te)).round(3))
Discussion