Lecture 02: KNN, ML Vocabulary, and K-Means – CS 189, Fall 2026

In this notebook we build our way from raw data to our first machine learning model, and then to our first unsupervised model. We start with the tools we need to look at data (pandas, numpy, and plotting), then introduce the simplest model we can think of, k-nearest neighbors. Along the way KNN will force us to invent the ideas of generalization, the train/test split, and hyperparameters. We close with k-means, which solves a related problem without any labels at all.

The main body of this notebook is what we walk through in lecture. The Appendix at the end goes much deeper on pandas, numpy, and visualization syntax; work through it after lecture.

In [1]:
# Download data & Install Dependencies
import os
import requests

os.makedirs("data", exist_ok=True)
data_files = {
    "data/penguins.csv":
        "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv",
}
for path, url in data_files.items():
    if not os.path.exists(path):
        r = requests.get(url)
        r.raise_for_status()
        with open(path, "wb") as f:
            f.write(r.content)
        print(f"Downloaded {path}")
    else:
        print(f"Found {path}")

Found data/penguins.csv

/Users/nargesnorouzi/Library/Python/3.9/lib/python/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
In [2]:
import numpy as np
import pandas as pd
import plotly.express as px

px.defaults.width = 800
pd.set_option("plotting.backend", "plotly")

os.makedirs("images", exist_ok=True)

Part 1: Looking at the Data

Before any modeling, look at the data. This section is a fast tour of the tools; the Appendix has the full treatment of every function used here.

The Palmer Penguins Dataset

Measurements of 344 penguins from three islands in the Palmer Archipelago, Antarctica. Each row is one penguin. We will use this single dataset for everything today.

In [3]:
penguins = pd.read_csv("data/penguins.csv")
penguins.head()
Out[3]:
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g sex
0 Adelie Torgersen 39.1 18.7 181.0 3750.0 MALE
1 Adelie Torgersen 39.5 17.4 186.0 3800.0 FEMALE
2 Adelie Torgersen 40.3 18.0 195.0 3250.0 FEMALE
3 Adelie Torgersen NaN NaN NaN NaN NaN
4 Adelie Torgersen 36.7 19.3 193.0 3450.0 FEMALE
In [4]:
penguins.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 344 entries, 0 to 343
Data columns (total 7 columns):
 #   Column             Non-Null Count  Dtype
---  ------             --------------  -----
 0   species            344 non-null    object
 1   island             344 non-null    object
 2   bill_length_mm     342 non-null    float64
 3   bill_depth_mm      342 non-null    float64
 4   flipper_length_mm  342 non-null    float64
 5   body_mass_g        342 non-null    float64
 6   sex                333 non-null    object
dtypes: float64(4), object(3)
memory usage: 18.9+ KB
In [5]:
# describe() summarizes the numeric columns
penguins.describe()
Out[5]:
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
count 342.000000 342.000000 342.000000 342.000000
mean 43.921930 17.151170 200.915205 4201.754386
std 5.459584 1.974793 14.061714 801.954536
min 32.100000 13.100000 172.000000 2700.000000
25% 39.225000 15.600000 190.000000 3550.000000
50% 44.450000 17.300000 197.000000 4050.000000
75% 48.500000 18.700000 213.000000 4750.000000
max 59.600000 21.500000 231.000000 6300.000000

How much data do we have, and what are we looking at?

  • shape gives (rows, columns)
  • value_counts() counts the occurrences of each value in a column
In [6]:
print("shape:", penguins.shape)

shape: (344, 7)
In [7]:
penguins["species"].value_counts()
Out[7]:
species
Adelie       152
Gentoo       124
Chinstrap     68
Name: count, dtype: int64

Selecting Data: `loc` and `iloc`

Two ways to pull a subset out of a DataFrame. The distinction matters constantly, so it is worth getting straight now.

  • iloc[] selects by integer position, like indexing a numpy array. The end of a slice is excluded.
  • loc[] selects by label, meaning the index value and the column name. The end of a slice is included.
In [8]:
# iloc: by position. Rows 0-4, first three columns.
penguins.iloc[0:5, 0:3]
Out[8]:
species island bill_length_mm
0 Adelie Torgersen 39.1
1 Adelie Torgersen 39.5
2 Adelie Torgersen 40.3
3 Adelie Torgersen NaN
4 Adelie Torgersen 36.7
In [9]:
# loc: by label. Rows 0-4 (inclusive!), named columns.
penguins.loc[0:4, ["species", "island", "bill_length_mm"]]
Out[9]:
species island bill_length_mm
0 Adelie Torgersen 39.1
1 Adelie Torgersen 39.5
2 Adelie Torgersen 40.3
3 Adelie Torgersen NaN
4 Adelie Torgersen 36.7
In [10]:
# A single column is a Series; a list of columns is a DataFrame
print(type(penguins["bill_length_mm"]))
print(type(penguins[["bill_length_mm"]]))

<class 'pandas.core.series.Series'>
<class 'pandas.core.frame.DataFrame'>

Filtering and Missing Values

Real data has missing information. isna() finds them and dropna() removes the rows with missing NULL values.

In [11]:
penguins.isna().sum()
Out[11]:
species               0
island                0
bill_length_mm        2
bill_depth_mm         2
flipper_length_mm     2
body_mass_g           2
sex                  11
dtype: int64
In [12]:
# Boolean filtering: which penguins are both heavy and long-flippered?
mask = (penguins["body_mass_g"] > 5000) & (penguins["flipper_length_mm"] > 220)
penguins[mask].head()
Out[12]:
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g sex
221 Gentoo Biscoe 50.0 16.3 230.0 5700.0 MALE
237 Gentoo Biscoe 49.2 15.2 221.0 6300.0 MALE
239 Gentoo Biscoe 48.7 15.1 222.0 5350.0 MALE
250 Gentoo Biscoe 47.3 15.3 222.0 5250.0 MALE
253 Gentoo Biscoe 59.6 17.0 230.0 6050.0 MALE
In [13]:
df = penguins.dropna()
print(len(penguins), "rows ->", len(df), "rows after dropping missing values")
df.head()

344 rows -> 333 rows after dropping missing values
Out[13]:
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g sex
0 Adelie Torgersen 39.1 18.7 181.0 3750.0 MALE
1 Adelie Torgersen 39.5 17.4 186.0 3800.0 FEMALE
2 Adelie Torgersen 40.3 18.0 195.0 3250.0 FEMALE
4 Adelie Torgersen 36.7 19.3 193.0 3450.0 FEMALE
5 Adelie Torgersen 39.3 20.6 190.0 3650.0 MALE

Grouping and Summarizing

groupby() splits the data into groups, applies a function to each, and combines the results.

In [14]:
df.groupby("species")[["bill_length_mm", "flipper_length_mm", "body_mass_g"]].mean().round(1)
Out[14]:
bill_length_mm flipper_length_mm body_mass_g
species
Adelie 38.8 190.1 3706.2
Chinstrap 48.8 195.8 3733.1
Gentoo 47.6 217.2 5092.4

`numpy`: the Array Underneath

pandas is built on numpy. Models in scikit-learn want numpy arrays, so this is how we hand our data over.

In [15]:
FEATURES = ["bill_length_mm", "flipper_length_mm"]
X = df[FEATURES].to_numpy()      # feature matrix, shape (n_samples, n_features)
y = df["species"].to_numpy()     # labels, shape (n_samples,)

print("X shape:", X.shape, "| y shape:", y.shape)
print("X dtype:", X.dtype, "\n")
print("The first three rows of X:\n", X[:3])

X shape: (333, 2) | y shape: (333,)
X dtype: float64

The first three rows of X:
 [[ 39.1 181. ]
 [ 39.5 186. ]
 [ 40.3 195. ]]
In [16]:
# Vectorized operations act on the whole array at once, with no Python loop.
print("column means:", X.mean(axis=0).round(2))
print("column stds: ", X.std(axis=0).round(2))

column means: [ 43.99 200.97]
column stds:  [ 5.46 13.99]

Visualizing: Can We See the Species?

If the species separate visually in these two dimensions, then a model has a chance.

In [17]:
fig = px.scatter(
    df, x="bill_length_mm", y="flipper_length_mm", color="species",
    title="Palmer Penguins: bill length vs flipper length",
    labels={
        "bill_length_mm": "Bill length (mm)",
        "flipper_length_mm": "Flipper length (mm)",
        "species": "Species",
    },
    height=520,
)
# Plotly's default legend sits off to the right and often gets clipped in notebooks.
fig.update_layout(
    showlegend=True,
    legend=dict(
        title_text="Species",
        x=0.01,
        y=0.99,
        xanchor="left",
        yanchor="top",
        bgcolor="white",
        bordercolor="lightgray",
        borderwidth=1,
    ),
)
fig.show()

Look at what this plot is telling us. The three species occupy different regions. Nothing separates them perfectly, but points near each other tend to share a species.

Part 2: The Simplest Possible Model

We want to predict a penguin's species from its bill length and flipper length. We ask ourselves, what is the least clever thing that could possibly work?

K-Nearest Neighbors (KNN) with k = 1

To predict the species of a new penguin, find the penguin in our data that is closest to it, and copy that penguin's species.

That is the entire algorithm. Notice what it does not have any optimization. "Training" is nothing more than storing the data.

In [18]:
from sklearn.neighbors import KNeighborsClassifier

knn1 = KNeighborsClassifier(n_neighbors=1)
knn1.fit(X, y)          # "training": store the data
print("Model is trained.")

Model is trained.
In [19]:
# Inference: predict the species of a new penguin
new_penguin = np.array([[45.0, 200.0]])   # bill 45mm, flipper 200mm
print("Predicted species:", knn1.predict(new_penguin)[0])

Predicted species: Chinstrap

How Good Is It?

We ask the model to predict the species of every penguin in our dataset, and count how often it is right.

In [20]:
accuracy = knn1.score(X, y)
print(f"Accuracy on our data: {accuracy:.1%}")

Accuracy on our data: 100.0%

100%.¶

We just built a perfect classifier out of the dumbest idea.

Something is wrong. What is it?

In [21]:
# Why 100%? Ask which point is nearest to the first penguin in our data.
distances, indices = knn1.kneighbors(X[:1], n_neighbors=1)
print("Nearest neighbour of penguin 0 is penguin index:", indices[0][0])
print("at a distance of:", distances[0][0])

Nearest neighbour of penguin 0 is penguin index: 0
at a distance of: 0.0

The nearest neighbour of every penguin in our dataset is itself, at distance zero. We asked the model to answer questions it had already memorized the answers to.

This tells us nothing about whether the model has learned anything. We need to evaluate it on penguins it has never seen.

Part 3: Generalization and the Train/Test Split

Generalization is the ability to perform well on new, unseen data drawn from the same distribution. To measure it, we hold data back.

In [22]:
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(f"Training set: {X_train.shape[0]} penguins")
print(f"Test set:     {X_test.shape[0]} penguins")

Training set: 266 penguins
Test set:     67 penguins
In [23]:
knn1 = KNeighborsClassifier(n_neighbors=1).fit(X_train, y_train)

print(f"Training accuracy: {knn1.score(X_train, y_train):.1%}")
print(f"Test accuracy:     {knn1.score(X_test, y_test):.1%}")

Training accuracy: 100.0%
Test accuracy:     95.5%

Training accuracy is still 100%, and it always will be for $k=1$, because every training point is still its own nearest neighbour. The test accuracy is the honest number, and it is meaningfully lower.

The gap between those two numbers is the thing we spend the rest of the semester managing.

Part 4: k Is a Choice

Why look at only one neighbour? Let each of the $k$ nearest neighbours vote, and take the majority.

Seeing k in the Decision Boundary

A decision boundary shows what the model would predict at every point in the feature space. It makes the effect of $k$ visible.

In [24]:
import plotly.graph_objects as go
from sklearn.preprocessing import LabelEncoder

def decision_boundary_figure(k, X_tr, y_tr, resolution=250):
    le = LabelEncoder().fit(y_tr)
    model = KNeighborsClassifier(n_neighbors=k).fit(X_tr, le.transform(y_tr))

    pad = 1.0
    xs = np.linspace(X_tr[:, 0].min() - pad, X_tr[:, 0].max() + pad, resolution)
    ys = np.linspace(X_tr[:, 1].min() - pad, X_tr[:, 1].max() + pad, resolution)
    xx, yy = np.meshgrid(xs, ys)
    zz = model.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

    fig = go.Figure()
    fig.add_trace(go.Heatmap(x=xs, y=ys, z=zz, showscale=False, opacity=0.30,
                             colorscale=[[0, "#002675"], [0.5, "#FDB515"], [1, "#028842"]]))
    for i, cls in enumerate(le.classes_):
        m = y_tr == cls
        fig.add_trace(go.Scatter(
            x=X_tr[m, 0], y=X_tr[m, 1], mode="markers", name=cls,
            marker=dict(size=7, line=dict(width=1, color="white"),
                        color=["#002675", "#FDB515", "#028842"][i])))
    fig.update_layout(title=f"KNN decision boundary, k = {k}",
                      xaxis_title="Bill length (mm)", yaxis_title="Flipper length (mm)",
                      width=800, height=520)
    return fig

decision_boundary_figure(1, X_train, y_train).show()

k = 1 gives a noisy boundary with little islands around individual points. The model is contorting itself to get every single training penguin right, including the ones that are unusual. That is overfitting: fitting the noise as well as the signal.

In [25]:
decision_boundary_figure(15, X_train, y_train).show()
In [26]:
decision_boundary_figure(100, X_train, y_train).show()

k = 15 smooths the boundary into something that looks like a real trend.

k = 100 smooths it so much that the model is barely paying attention to the data. That is underfitting.

So $k$ controls a trade-off, and somewhere in between is a good value.

Finding a Good k

We cannot use the test set to pick $k$. The moment we tune anything against the test set, it stops measuring generalization and becomes just another training set.

So we split again: a validation set, carved out of the training data.

In [27]:
X_tr, X_val, y_tr, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42, stratify=y_train
)
print(f"train {len(X_tr)} | validation {len(X_val)} | test {len(X_test)}")

train 199 | validation 67 | test 67
In [28]:
ks = list(range(1, 101))
train_acc, val_acc = [], []
for k in ks:
    m = KNeighborsClassifier(n_neighbors=k).fit(X_tr, y_tr)
    train_acc.append(m.score(X_tr, y_tr))
    val_acc.append(m.score(X_val, y_val))

best_k = ks[int(np.argmax(val_acc))]
print(f"Best k on the validation set: {best_k}  (validation accuracy {max(val_acc):.1%})")

Best k on the validation set: 3  (validation accuracy 95.5%)
In [29]:
fig = go.Figure()
fig.add_trace(go.Scatter(x=ks, y=train_acc, name="Training accuracy",
                         line=dict(color="#002675", width=3)))
fig.add_trace(go.Scatter(x=ks, y=val_acc, name="Validation accuracy",
                         line=dict(color="#FDB515", width=3)))
fig.add_vline(x=best_k, line_dash="dash", line_color="#028842",
              annotation_text=f"best k = {best_k}")
fig.update_layout(title="Training and validation accuracy as k increases",
                  xaxis_title="k (number of neighbours)", yaxis_title="Accuracy",
                  width=800, height=520)
fig.show()

Read this plot carefully, because its shape recurs all semester.

  • On the left (small $k$) training accuracy is perfect and validation accuracy is lower. Overfitting.
  • On the right (large $k$) both accuracies fall together. Underfitting.
  • The sweet spot is where validation accuracy peaks.

$k$ is a hyperparameter: a value we choose before training, as opposed to a parameter, which is learned from the data during training. KNN is unusual in having no parameters at all, only this one hyperparameter. That makes it a clean place to see the distinction.

In [30]:
# Only now, having chosen k, do we touch the test set. Once.
final = KNeighborsClassifier(n_neighbors=best_k).fit(X_train, y_train)
print(f"Final test accuracy with k={best_k}: {final.score(X_test, y_test):.1%}")

Final test accuracy with k=3: 97.0%

A Wrinkle: Distance Depends on Units

KNN is built entirely on distance, so it cares about the scale of the features. Flipper length spans roughly 172-231 mm and bill length roughly 32-60 mm, so flipper length contributes far more to the distance simply because its numbers are bigger.

In [31]:
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

scaled = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=best_k))
scaled.fit(X_train, y_train)

print(f"Unscaled test accuracy: {final.score(X_test, y_test):.1%}")
print(f"Scaled test accuracy:   {scaled.score(X_test, y_test):.1%}")

Unscaled test accuracy: 97.0%
Scaled test accuracy:   97.0%

Standardization rescales each feature to have mean 0 and standard deviation 1, so every feature contributes to the distance on equal terms. For any distance-based model this is not optional.

Part 5: The Same Idea for Regression

So far we predicted a category (species). What if we want to predict a number, like body mass?

The algorithm barely changes. Find the k nearest neighbours, and instead of taking a majority vote, take their average.

In [32]:
from sklearn.neighbors import KNeighborsRegressor

Xr = df[["flipper_length_mm"]].to_numpy()
yr = df["body_mass_g"].to_numpy()

Xr_train, Xr_test, yr_train, yr_test = train_test_split(Xr, yr, test_size=0.2, random_state=42)
print("Predicting body mass (g) from flipper length (mm)")

Predicting body mass (g) from flipper length (mm)
In [33]:
grid = np.linspace(Xr.min(), Xr.max(), 400).reshape(-1, 1)

fig = go.Figure()
fig.add_trace(go.Scatter(x=Xr_train.ravel(), y=yr_train, mode="markers", name="Training data",
                         marker=dict(color="lightgray", size=6)))
for k, colour in [(1, "#002675"), (25, "#FDB515"), (200, "#028842")]:
    reg = KNeighborsRegressor(n_neighbors=k).fit(Xr_train, yr_train)
    fig.add_trace(go.Scatter(x=grid.ravel(), y=reg.predict(grid), mode="lines",
                             name=f"k = {k}", line=dict(width=3, color=colour)))
fig.update_layout(title="KNN regression: body mass vs flipper length",
                  xaxis_title="Flipper length (mm)", yaxis_title="Body mass (g)",
                  width=800, height=520)
fig.show()

The same trade-off, in a different costume. $k=1$ is a jagged step function chasing every point. $k=200$ is nearly a flat line. The middle is where the real trend lives.

Overfitting and underfitting are not facts about classification. They are facts about model complexity.

In [34]:
for k in [1, 25, 200]:
    reg = KNeighborsRegressor(n_neighbors=k).fit(Xr_train, yr_train)
    print(f"k={k:>3}  train R^2 = {reg.score(Xr_train, yr_train):.3f}   "
          f"test R^2 = {reg.score(Xr_test, yr_test):.3f}")

k=  1  train R^2 = 0.646   test R^2 = 0.491
k= 25  train R^2 = 0.785   test R^2 = 0.802
k=200  train R^2 = 0.422   test R^2 = 0.427

At k=1 the training score is far above the test score: the model is memorizing. At k=200 the two scores collapse together, and both are bad: the model is too simple to capture the trend. The same story the classification curve told, told again.

Part 6: What If We Had No Labels?

Everything so far was supervised: every penguin came with its species attached.

Now suppose a field researcher hands us the same measurements with no species labels at all, and asks: are there natural groups in here?

This is unsupervised learning, and specifically clustering.

In [35]:
# Throw the labels away.
fig = px.scatter(df, x="bill_length_mm", y="flipper_length_mm",
                 title="The same penguins, with no labels",
                 labels={"bill_length_mm": "Bill length (mm)",
                         "flipper_length_mm": "Flipper length (mm)"},
                 height=520)
fig.update_traces(marker=dict(color="gray", size=7))
fig.show()

K-Means in scikit-learn

K-means partitions the data into K clusters, each represented by a centroid. It assigns each point to the nearest centroid, then moves each centroid to the mean of its points, and repeats.

In [36]:
from sklearn.cluster import KMeans

Xs = StandardScaler().fit_transform(X)   # distance-based again, so standardize

kmeans = KMeans(n_clusters=3, n_init=10, random_state=42)
cluster = kmeans.fit_predict(Xs)

df_c = df.copy()
df_c["cluster"] = cluster.astype(str)
px.scatter(df_c, x="bill_length_mm", y="flipper_length_mm", color="cluster",
           title="K-means with K = 3",
           labels={"bill_length_mm": "Bill length (mm)",
                   "flipper_length_mm": "Flipper length (mm)"},
           height=520).show()

Lloyd's Algorithm, Step by Step

Watch the centroids move. Each iteration is two steps: assign, then update.

In [37]:
def lloyd_steps(Xs, K=3, n_iter=6, seed=1):
    rng = np.random.default_rng(seed)
    centres = Xs[rng.choice(len(Xs), K, replace=False)].copy()
    history = []
    for _ in range(n_iter):
        d = ((Xs[:, None, :] - centres[None, :, :]) ** 2).sum(axis=2)
        assign = d.argmin(axis=1)                       # assignment step
        history.append((centres.copy(), assign.copy()))
        for j in range(K):                              # update step
            if (assign == j).any():
                centres[j] = Xs[assign == j].mean(axis=0)
    return history

history = lloyd_steps(Xs)
print(f"{len(history)} iterations recorded")

6 iterations recorded
In [38]:
palette = ["#002675", "#FDB515", "#028842"]
fig = go.Figure()
frames = []
for step, (centres, assign) in enumerate(history):
    data = []
    for j in range(3):
        m = assign == j
        data.append(go.Scatter(x=Xs[m, 0], y=Xs[m, 1], mode="markers",
                               marker=dict(color=palette[j], size=6), name=f"cluster {j}"))
    data.append(go.Scatter(x=centres[:, 0], y=centres[:, 1], mode="markers",
                           marker=dict(color="black", size=18, symbol="x"), name="centroids"))
    frames.append(go.Frame(data=data, name=str(step)))

fig.add_traces(frames[0].data)
fig.frames = frames
fig.update_layout(
    title="Lloyd's algorithm: assign, then update",
    xaxis_title="Bill length (standardized)", yaxis_title="Flipper length (standardized)",
    width=800, height=560,
    updatemenus=[dict(type="buttons", showactive=False,
                      buttons=[dict(label="Play", method="animate",
                                    args=[None, dict(frame=dict(duration=900, redraw=True),
                                                     fromcurrent=True)])])])
fig.show()

Choosing K

K is a hyperparameter, exactly like k in KNN. But there is no validation accuracy to optimize, because there are no labels. The common heuristic is the elbow method: plot the within-cluster sum of squares (inertia) against K and look for the bend.

In [39]:
Ks = range(1, 11)
inertia = [KMeans(n_clusters=k, n_init=10, random_state=42).fit(Xs).inertia_ for k in Ks]

fig = px.line(x=list(Ks), y=inertia, markers=True,
              title="Elbow method: inertia vs K",
              labels={"x": "K (number of clusters)", "y": "Within-cluster sum of squares"})
fig.update_traces(line=dict(color="#002675", width=3))
fig.update_layout(width=800, height=480)
fig.show()

The bend is around K=3, which is encouraging. But note how much judgment that reading takes. The elbow is a heuristic, not a criterion.

Part 7: Clusters Are Not Labels

We have run two models on the same two columns. One was told the species; one was not. How much did the labels actually buy us?

In [40]:
comparison = pd.crosstab(df_c["cluster"], df_c["species"])
comparison
Out[40]:
species Adelie Chinstrap Gentoo
cluster
0 141 5 0
1 1 4 118
2 4 59 1
In [41]:
fig = px.scatter(df_c, x="bill_length_mm", y="flipper_length_mm",
                 color="species", symbol="cluster",
                 title="True species (colour) vs discovered clusters (symbol)",
                 labels={"bill_length_mm": "Bill length (mm)",
                         "flipper_length_mm": "Flipper length (mm)"},
                 height=560)
fig.show()

K-means, with no access to the labels at all, largely rediscovered the species.

But be careful about what that means. The clusters are not species. K-means found groups of penguins that are close together in bill and flipper measurements, and in this dataset those groups happen to line up with species. Change the features and the clusters change. Nothing in the algorithm knows what a species is, and nothing guarantees the groups it finds correspond to anything you care about.

Appendix: Going Deeper

The material below was not covered in lecture. Work through it on your own; it is the reference for the syntax used above and for Homework 1.

A1. `pandas` Data Structures

A DataFrame is a 2-dimensional table. A Series is a single column. Both are built on an Index.

In [42]:
s = pd.Series([3750, 3800, 3250], index=["p0", "p1", "p2"], name="body_mass_g")
print(s)
print("\nindex:", s.index.tolist())
print("values:", s.values)

p0    3750
p1    3800
p2    3250
Name: body_mass_g, dtype: int64

index: ['p0', 'p1', 'p2']
values: [3750 3800 3250]
In [43]:
frame = pd.DataFrame({
    "species": ["Adelie", "Gentoo", "Chinstrap"],
    "bill_length_mm": [39.1, 46.1, 46.5],
    "island": ["Torgersen", "Biscoe", "Dream"],
})
frame
Out[43]:
species bill_length_mm island
0 Adelie 39.1 Torgersen
1 Gentoo 46.1 Biscoe
2 Chinstrap 46.5 Dream

A2. Exploring a `DataFrame`

In [44]:
print(penguins.head(3))          # first rows
print(penguins.tail(3))          # last rows
print(penguins.sample(3))        # random rows
print(penguins.columns.tolist()) # column names
print(penguins.dtypes)           # column types
print(penguins["island"].unique())

  species     island  bill_length_mm  bill_depth_mm  flipper_length_mm  \
0  Adelie  Torgersen            39.1           18.7              181.0
1  Adelie  Torgersen            39.5           17.4              186.0
2  Adelie  Torgersen            40.3           18.0              195.0

   body_mass_g     sex
0       3750.0    MALE
1       3800.0  FEMALE
2       3250.0  FEMALE
    species  island  bill_length_mm  bill_depth_mm  flipper_length_mm  \
341  Gentoo  Biscoe            50.4           15.7              222.0
342  Gentoo  Biscoe            45.2           14.8              212.0
343  Gentoo  Biscoe            49.9           16.1              213.0

     body_mass_g     sex
341       5750.0    MALE
342       5200.0  FEMALE
343       5400.0    MALE
    species     island  bill_length_mm  bill_depth_mm  flipper_length_mm  \
287  Gentoo     Biscoe            49.5           16.2              229.0
14   Adelie  Torgersen            34.6           21.1              198.0
142  Adelie      Dream            32.1           15.5              188.0

     body_mass_g     sex
287       5800.0    MALE
14        4400.0    MALE
142       3050.0  FEMALE
['species', 'island', 'bill_length_mm', 'bill_depth_mm', 'flipper_length_mm', 'body_mass_g', 'sex']
species               object
island                object
bill_length_mm       float64
bill_depth_mm        float64
flipper_length_mm    float64
body_mass_g          float64
sex                   object
dtype: object
['Torgersen' 'Biscoe' 'Dream']

A3. `loc` and `iloc` in Full

The rule to remember: iloc is positional and excludes the endpoint, loc is label-based and includes it.

In [45]:
print(penguins.iloc[0])              # row 0 as a Series
print(penguins.iloc[0:3])            # rows 0,1,2
print(penguins.iloc[:, 2])           # column at position 2
print(penguins.iloc[[0, 5, 10], [0, 2]])   # arbitrary rows and columns

species                 Adelie
island               Torgersen
bill_length_mm            39.1
bill_depth_mm             18.7
flipper_length_mm        181.0
body_mass_g             3750.0
sex                       MALE
Name: 0, dtype: object
  species     island  bill_length_mm  bill_depth_mm  flipper_length_mm  \
0  Adelie  Torgersen            39.1           18.7              181.0
1  Adelie  Torgersen            39.5           17.4              186.0
2  Adelie  Torgersen            40.3           18.0              195.0

   body_mass_g     sex
0       3750.0    MALE
1       3800.0  FEMALE
2       3250.0  FEMALE
0      39.1
1      39.5
2      40.3
3       NaN
4      36.7
       ...
339     NaN
340    46.8
341    50.4
342    45.2
343    49.9
Name: bill_length_mm, Length: 344, dtype: float64
   species  bill_length_mm
0   Adelie            39.1
5   Adelie            39.3
10  Adelie            37.8
In [46]:
print(penguins.loc[0:2])                          # rows labelled 0,1,2 (inclusive)
print(penguins.loc[:, "species":"bill_length_mm"]) # column slice by name
print(penguins.loc[penguins["species"] == "Gentoo", "body_mass_g"].mean())

  species     island  bill_length_mm  bill_depth_mm  flipper_length_mm  \
0  Adelie  Torgersen            39.1           18.7              181.0
1  Adelie  Torgersen            39.5           17.4              186.0
2  Adelie  Torgersen            40.3           18.0              195.0

   body_mass_g     sex
0       3750.0    MALE
1       3800.0  FEMALE
2       3250.0  FEMALE
    species     island  bill_length_mm
0    Adelie  Torgersen            39.1
1    Adelie  Torgersen            39.5
2    Adelie  Torgersen            40.3
3    Adelie  Torgersen             NaN
4    Adelie  Torgersen            36.7
..      ...        ...             ...
339  Gentoo     Biscoe             NaN
340  Gentoo     Biscoe            46.8
341  Gentoo     Biscoe            50.4
342  Gentoo     Biscoe            45.2
343  Gentoo     Biscoe            49.9

[344 rows x 3 columns]
5076.016260162602

A4. Modifying and Sorting

In [47]:
tmp = df.copy()
tmp["mass_kg"] = tmp["body_mass_g"] / 1000              # add a column
tmp["bill_ratio"] = tmp["bill_length_mm"] / tmp["bill_depth_mm"]
tmp = tmp.drop(columns=["bill_ratio"])                  # drop a column
tmp.sort_values("body_mass_g", ascending=False).head()
Out[47]:
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g sex mass_kg
237 Gentoo Biscoe 49.2 15.2 221.0 6300.0 MALE 6.30
253 Gentoo Biscoe 59.6 17.0 230.0 6050.0 MALE 6.05
297 Gentoo Biscoe 51.1 16.3 220.0 6000.0 MALE 6.00
337 Gentoo Biscoe 48.8 16.2 222.0 6000.0 MALE 6.00
299 Gentoo Biscoe 45.2 16.4 223.0 5950.0 MALE 5.95

A5. Aggregation, `groupby`, and Pivot Tables

In [48]:
print(df.groupby("species")["body_mass_g"].agg(["mean", "std", "count"]).round(1))
print()
print(df.groupby(["species", "island"])["flipper_length_mm"].mean().round(1))
print()
print(df.pivot_table(index="species", columns="island",
                     values="body_mass_g", aggfunc="mean").round(0))

             mean    std  count
species
Adelie     3706.2  458.6    146
Chinstrap  3733.1  384.3     68
Gentoo     5092.4  501.5    119

species    island
Adelie     Biscoe       188.8
           Dream        189.9
           Torgersen    191.5
Chinstrap  Dream        195.8
Gentoo     Biscoe       217.2
Name: flipper_length_mm, dtype: float64

island     Biscoe   Dream  Torgersen
species
Adelie     3710.0  3701.0     3709.0
Chinstrap     NaN  3733.0        NaN
Gentoo     5092.0     NaN        NaN

A6. Joining `DataFrames`

In [49]:
islands = pd.DataFrame({
    "island": ["Torgersen", "Biscoe", "Dream"],
    "latitude": [-64.77, -65.43, -64.73],
})
merged = df.merge(islands, on="island", how="left")   # try how="inner" and how="outer"
merged[["species", "island", "latitude"]].head()
Out[49]:
species island latitude
0 Adelie Torgersen -64.77
1 Adelie Torgersen -64.77
2 Adelie Torgersen -64.77
3 Adelie Torgersen -64.77
4 Adelie Torgersen -64.77

A7. `numpy` Essentials

In [50]:
a = np.arange(12).reshape(3, 4)
print(a)
print("shape:", a.shape, "| ndim:", a.ndim)
print("row sums:", a.sum(axis=1))
print("col means:", a.mean(axis=0))
print("boolean mask:", a[a > 6])
print("broadcasting:", (a - a.mean(axis=0)).round(2))

[[ 0  1  2  3]
 [ 4  5  6  7]
 [ 8  9 10 11]]
shape: (3, 4) | ndim: 2
row sums: [ 6 22 38]
col means: [4. 5. 6. 7.]
boolean mask: [ 7  8  9 10 11]
broadcasting: [[-4. -4. -4. -4.]
 [ 0.  0.  0.  0.]
 [ 4.  4.  4.  4.]]
In [51]:
# Euclidean distance by hand, which is exactly what KNN computes
p, q = X[0], X[1]
print("manual :", np.sqrt(((p - q) ** 2).sum()))
print("numpy  :", np.linalg.norm(p - q))

manual : 5.015974481593781
numpy  : 5.015974481593781

A8. Visualization Reference

matplotlib and seaborn are the static plotting standards; plotly gives interactive figures, which is why we use it in lecture.

In [52]:
import matplotlib.pyplot as plt
import seaborn as sns

fig, axes = plt.subplots(1, 2, figsize=(11, 4))
axes[0].hist(df["body_mass_g"], bins=25, color="#002675")
axes[0].set_title("Body mass"); axes[0].set_xlabel("g")
sns.scatterplot(data=df, x="bill_length_mm", y="flipper_length_mm",
                hue="species", ax=axes[1])
axes[1].set_title("By species")
plt.tight_layout(); plt.show()
No description has been provided for this image
In [53]:
# Plotly: histogram, box plot, and a faceted scatter
px.histogram(df, x="body_mass_g", color="species", nbins=30, height=420).show()
px.box(df, x="species", y="flipper_length_mm", color="species", height=420).show()
px.scatter(df, x="bill_length_mm", y="flipper_length_mm",
           color="species", facet_col="island", height=420).show()