K-Nearest Neighbors (KNN) Algorithm Explained with Examples

Quick answer: Learn the K-Nearest Neighbors algorithm including how it works, choosing the value of K, distance metrics, and its strengths and weaknesses.

What is K-Nearest Neighbors?

K-Nearest Neighbors, or KNN, is a simple Machine Learning algorithm that classifies a new data point based on the majority class among its K closest points in the training data. For regression, it predicts the average value of those K nearest neighbours instead.

How KNN Works, Step by Step

Choose a value for K, the number of neighbours to consider
Calculate the distance from the new point to every point in the training data
Select the K closest points
For classification, take a majority vote among those K points' classes; for regression, average their values

from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train_scaled, y_train)
predictions = model.predict(X_test_scaled)

Why Feature Scaling Matters So Much for KNN

KNN relies entirely on distance calculations. If one feature has a much larger numeric range than another, such as income in thousands versus age in years, it will dominate the distance calculation regardless of its actual importance. Scaling features to a comparable range, using StandardScaler or MinMaxScaler, is essential before applying KNN.

Distance Metrics

Metric	Description
Euclidean distance	Straight-line distance, the most common default choice
Manhattan distance	Sum of absolute differences along each dimension
Minkowski distance	A generalisation that includes both Euclidean and Manhattan as special cases

Choosing the Right Value of K

A small K, such as 1, makes the model sensitive to noise and prone to overfitting. A very large K smooths predictions too much and can underfit, potentially ignoring genuinely important local patterns. K is typically chosen using cross validation, testing several values and picking the one with the best validation performance. Using an odd number for K in binary classification avoids tied votes.

Strengths and Weaknesses

Strengths	Weaknesses
Simple to understand and implement	Slow at prediction time on large datasets, since it compares against all training points
No training phase, it stores the data directly	Sensitive to irrelevant features and unscaled data
Naturally handles multi-class problems	Performs poorly in very high-dimensional spaces

The Curse of Dimensionality

As the number of features grows very large, distances between points become less meaningful, since points tend to spread out and become nearly equidistant from each other. This makes KNN less effective on high-dimensional data unless dimensionality is first reduced.

Common Interview Questions

Why is feature scaling so important for KNN?

KNN relies purely on distance calculations, so a feature with a much larger numeric range would dominate the distance measure regardless of its actual predictive importance, unless all features are scaled to a comparable range first.

What happens if K is set too small or too large?

Too small a K makes the model sensitive to noise in individual points and prone to overfitting. Too large a K oversmooths the decision boundary and can underfit, missing genuinely important local patterns.

FAQ

Frequently Asked Questions

What is the K-Nearest Neighbors algorithm?

A Machine Learning algorithm that classifies a new data point based on the majority class among its K closest points in the training data, or averages nearby values for regression.

Why does feature scaling matter so much for KNN?

KNN relies entirely on distance calculations, so a feature with a much larger numeric range would dominate the distance measure unless all features are scaled to a comparable range first.

How do you choose the right value of K?

Typically through cross validation, testing several values and choosing the one with the best validation performance, using an odd number for binary classification to avoid tied votes.

What is a major limitation of KNN?

It is slow at prediction time on large datasets, since it must compare a new point against every point in the training data, and it performs poorly in very high-dimensional data.