question on exercise03 from phases/01-math-foundations/10-dimensionality-reduction

[jubeater]

PCA Elbow Question

as the exercise quote:

Take a dataset with 50 features where only 5 are informative (generate one with sklearn.datasets.make_classification). Apply PCA and check whether the explained variance curve correctly identifies that the data is effectively 5-dimensional.

I ran PCA on a synthetic dataset created with sklearn.datasets.make_classification using:

• n_samples = 5000
• n_features = 50
• n_informative = 5

Then I plotted the cumulative explained variance ratio from PCA.

What I observed

• The curve increases smoothly and looks almost linear.
• I do not see a clear elbow near 5 components.
• I also tried sklearn PCA and got the same result, so the issue does not seem to be my PCA implementation.

What I want to ask

Why does the explained variance curve look so linear in this case?

Is it because:

• n_informative = 5 only means 5 features are useful for predicting the label, not that the data is actually 5-dimensional in the PCA sense?
• PCA measures variance, not label informativeness(only use X not y in my case)?

Are there ways to generate or transform the dataset so that PCA shows a clearer elbow near 5 components like different make_classification settings ?

If not, what method could I use to be able to find the number of informative features is 5 from the generated datasets?

Current cumulative variance output

[0.79001528 0.80883941 0.81594031 0.82063076 0.82527467 0.82985169
 0.83439929 0.83889553 0.8433546  0.84779575 0.85217141 0.85650316
 0.8607992  0.86506236 0.86929372 0.87349747 0.87768152 0.88182503
 0.88595956 0.89007338 0.89415834 0.89822124 0.90224452 0.90625165
 0.91022895 0.91418556 0.91811477 0.92201461 0.92588088 0.92970524
 0.93349004 0.9372697  0.94101989 0.94474171 0.94843535 0.95211684
 0.95578337 0.9594258  0.96302392 0.96661829 0.97016817 0.97367212
 0.97716275 0.98063472 0.98406645 0.98743854 0.99074949 0.99403477
 0.99727858 1.        ]

Code I ran using sklearn

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import make_classification
from sklearn.decomposition import PCA

X_sample, y_sample = make_classification(
    n_samples=5000,
    n_features=50,
    n_informative=5,
    n_redundant=0,
    n_repeated=0,
    n_clusters_per_class=1,
    class_sep=10.0,
    flip_y=0.0,
    shuffle=False,
    random_state=42,
)

pca_sample = PCA(n_components=50)
pca_sample.fit(X_sample)

cum_ratio = np.cumsum(pca_sample.explained_variance_ratio_)
print(f"list of the cum ratio: {cum_ratio}")
plt.figure(figsize=(8, 5))
plt.plot(range(1, 51), cum_ratio, marker="o")
plt.axhline(0.95, color="red", linestyle="--", label="95% variance")
plt.xlabel("Number of components")
plt.ylabel("Cumulative explained variance")
plt.title("PCA Elbow Plot")
plt.grid(True)
plt.legend()
plt.show()

[skyrain888]

@claude 有问题吗

[eshitakundu]

Both of your guesses are right. There's also a third reason, and I think it explains your numbers exactly.

Your curve matches one cluster per class. I tried to reproduce your output, and with only the settings you listed I get a first component of about 0.11, not 0.79. But n_clusters_per_class=1, class_sep=7 matches your first 10 cumulative values to within 0.005:

Did you set those (or something similar)? If so, here's why that gives one big component and then a flat line.

1. PCA only looks at variance in X. n_informative=5 means 5 features help predict y, not that the data stretches out in 5 directions. PCA never sees y, so it can't know which features are "informative". It just ranks directions by spread. The 43 noise features (50 minus 5 informative minus the default n_redundant=2) each have a variance of about 1, so once the real signal is used up, every remaining direction looks the same, and the cumulative curve becomes a straight line.

2. The class signal only spans (number of cluster centres - 1) directions. make_classification builds the 5 informative features by placing one Gaussian blob per class per n_clusters_per_class. The classes differ in where those blob centres sit, and a set of points can only spread in so many directions: 2 points lie on a line (1 direction), 3 on a plane (2), 4 in 3D (3), and so on. So even though the blobs live in a 5-feature space, with:

• 1 cluster per class × 2 classes = 2 centres → the signal is a single direction. That's your 0.79 first component.
• the default 2 × 2 = 4 centres → 3 directions, so the elbow is at 3.
• 2 × 4 = 8 centres → enough to fill all 5, so the elbow is at 5.

from sklearn.datasets import make_classification
from sklearn.decomposition import PCA

kw = dict(n_samples=5000, n_features=50, n_informative=5, n_redundant=0, class_sep=4, random_state=0)
for k in [1, 2, 4]:
    X, _ = make_classification(**kw, n_clusters_per_class=k)
    print(2 * k, "centres:", PCA().fit(X).explained_variance_ratio_[:7].round(3))

# 2 centres: [0.559 0.026 0.022 0.01  0.01  0.01  0.01 ]   <- elbow after 1
# 4 centres: [0.357 0.125 0.066 0.018 0.013 0.011 0.011]   <- elbow after 3
# 8 centres: [0.244 0.146 0.112 0.064 0.052 0.01  0.01 ]   <- elbow after 5

So to see the elbow at 5, use at least 6 centres (n_clusters_per_class=4, or n_classes=3, n_clusters_per_class=2), a class_sep large enough for the signal to stand out from the noise, and n_redundant=0. Plotting the individual ratios, as above, shows the elbow much more clearly than the cumulative curve.

3. Finding the 5 informative features. Even when PCA gives you the elbow, it only tells you how many directions the data uses. Each component is a mix of all 50 columns, so it never says which columns. Since "informative" means "useful for predicting y", a method that looks at y is the right tool. With shuffle=False, the informative features are columns 0 to 4:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import f_classif, mutual_info_classif

X, y = make_classification(n_samples=5000, n_features=50, n_informative=5, n_redundant=0, shuffle=False, random_state=0)
F, _ = f_classif(X, y)
mi = mutual_info_classif(X, y)
rf = RandomForestClassifier(200).fit(X, y)
for name, score in [("ANOVA F", F), ("mutual info", mi), ("random forest", rf.feature_importances_)]:
    print(name, np.argsort(-score)[:5])

# ANOVA F [ 1  2  0  4 33]
# mutual info [ 1  2  0  4 20]
# random forest [1 2 0 4 3]

All three find 4 of the 5 straight away, and the forest gets all 5. Feature 3 is weaker in this sample, which is normal, since each informative feature gets a random share of the signal. If you want the method to choose how many features to keep, RFECV or SelectFromModel will do that.