Permutation feature importance answers a narrow but useful question: how much does a fitted model's score on unseen data change when one feature is made uninformative by random shuffling? That makes it useful for data scientists, machine learning engineers, and applied analysts who already have a trained model and want to understand which inputs the model depends on for held-out performance.
It does not prove that a feature causes the outcome. It measures model-score sensitivity under a specific perturbation, on a specific dataset, with a specific scoring metric.
What permutation feature importance answers
Permutation feature importance evaluates a fitted estimator by comparing its baseline score with the score after one feature has been permuted. The scikit-learn user guide on permutation feature importance describes the method as shuffling a feature and measuring how much the model score decreases from the baseline. Christoph Molnar's Interpretable Machine Learning chapter on permutation feature importance frames the same idea as measuring the increase in prediction error, or the decrease in model score, after a feature's values are shuffled.
For example, if a classifier has a held-out accuracy of 0.86 and accuracy falls to 0.76 after shuffling feature A, feature A has a larger apparent importance than feature B if shuffling feature B leaves accuracy at 0.85. The importance is relative to that fitted model, that held-out sample, and that metric.
Why measure it on held-out data
You can compute permutation feature importance on training data, but held-out data is usually the better choice when the question is about generalization. A validation or test split asks whether the model still depends on the feature when predicting cases it did not fit on.
That distinction matters. A feature can look useful on training data because the fitted model memorized idiosyncrasies. Measuring on held-out data better aligns the importance calculation with model performance on unseen observations. The scikit-learn guide explicitly notes that permutation importance can be computed on held-out data, which helps assess feature importance for model performance beyond the training set.
The result is still sample-dependent. If the held-out set is small, shifted, or not representative of the intended use case, the importances can be unstable or misleading.
Inputs you need before running the calculation
Before computing permutation feature importance, prepare these inputs:
| Input | Why it matters |
|---|---|
| Fitted model | Permutation importance evaluates an already fitted estimator. It is not a feature-selection method independent of the model. |
| Held-out feature matrix | This is the data whose columns will be shuffled one at a time. |
| Held-out target values | The scoring metric needs true labels or target values. |
| Scoring metric | Importance is defined as the change in this metric, so accuracy, R², and other metrics can produce different rankings. |
| Number of repeats | Repeating the shuffle helps show how much the score drop varies because of random permutation. |
| Random seed | A seed makes the random shuffling reproducible for review and reporting. |
| Feature names | Names make the output interpretable and easier to audit. |
Step-by-step procedure
Use this workflow for each feature:
- Score the fitted model on held-out data. This is the baseline score.
- Pick one feature column in the held-out feature matrix.
- Randomly shuffle that column's values across rows, leaving all other columns unchanged.
- Score the same fitted model on the modified held-out data.
- Compute the score decrease: baseline score minus permuted score.
- Repeat the shuffle several times for that feature.
- Summarize the mean score decrease and the repeat-to-repeat variation.
- Repeat for every feature and rank features by mean decrease.
The scikit-learn permutation_importance function implements this process by repeatedly shuffling individual columns and measuring the change in a specified scoring metric.
Python example using scikit-learn
The following example is illustrative and reproducible in structure. It uses synthetic data and a fitted model only to show the mechanics. The exact numeric output depends on the environment, library versions, random process, and the data generated when you run it.
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.model_selection import train_test_split
# Synthetic example data for demonstration only.
X, y = make_classification(
n_samples=1000,
n_features=8,
n_informative=4,
n_redundant=2,
random_state=42
)
feature_names = [f'feature_{i}' for i in range(X.shape[1])]
X = pd.DataFrame(X, columns=feature_names)
X_train, X_holdout, y_train, y_holdout = train_test_split(
X,
y,
test_size=0.25,
random_state=42
)
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)
result = permutation_importance(
model,
X_holdout,
y_holdout,
scoring='accuracy',
n_repeats=20,
random_state=42
)
importance_table = pd.DataFrame({
'feature': feature_names,
'importance_mean': result.importances_mean,
'importance_std': result.importances_std
}).sort_values('importance_mean', ascending=False)
print(importance_table)
A hypothetical reference table might look like this:
| feature | importance_mean | importance_std |
|---|---|---|
| feature_3 | 0.120 | 0.025 |
| feature_1 | 0.075 | 0.020 |
| feature_6 | 0.012 | 0.018 |
| feature_4 | -0.004 | 0.010 |
This table is not a reported benchmark and should not be treated as evidence about your model. It is a reference answer for how to read the structure:
- feature_3 has the largest average accuracy drop after shuffling, so this fitted model appears to rely on it most among the listed features on this held-out split.
- feature_1 also appears useful, but with a smaller mean drop.
- feature_6 is close to zero relative to its variation, so the evidence for usefulness is weak in this run.
- feature_4 is slightly negative, which can happen from random variation or because shuffling a noisy feature improves the selected score in a particular sample. Treat that cautiously rather than as a universal property of the feature.
How to interpret the results
Permutation feature importance is easiest to interpret when the scoring metric is familiar. With accuracy, a mean importance of 0.05 means the model's held-out accuracy fell by an average of 5 percentage points when that feature was shuffled, assuming the baseline and permuted scores are both accuracy values.
Use three parts of the output together:
| Output | Interpretation |
|---|---|
| Mean importance | Average decrease in the selected score after shuffling the feature. Larger positive values indicate greater model-score sensitivity. |
| Standard deviation or repeat variation | How much the measured decrease changes across random shuffles. High variation means the estimate is less stable. |
| Rank | A convenient ordering, not a guarantee that adjacent features are meaningfully different. |
Positive importance means the selected score usually got worse after shuffling. Near-zero importance means the model's score changed little in this held-out sample. Negative importance means the score improved after shuffling in the repeated experiment; this can occur because of random variation, a noisy feature, or the particular data split and metric.
Every value is conditional. Change the fitted model, the held-out split, or the scoring metric, and the importance values can change.
A practical decision rule
A reasonable operating rule is:
Review features whose mean permutation importance is meaningfully above zero and large relative to repeat-to-repeat variation. Treat near-zero or negative values as candidates for investigation, not automatic deletion.
That rule has important limits. Before removing or promoting a feature, check whether it is correlated with other predictors, required for monitoring, operationally important, regulated, or relevant to fairness analysis. Also validate any downstream change separately. Permutation importance tells you how the current fitted model responds to a shuffled column; it does not prove that removing the column will improve a future model.
Failure case: correlated features can mask each other
Suppose a credit-risk model includes two highly similar variables:
- current_balance
- average_balance_last_30_days
If the model can use either one, shuffling current_balance may not reduce accuracy much because average_balance_last_30_days still carries similar information. Shuffling the second feature may produce the same pattern. Both features can appear weak even though the group of related balance features matters to the model.
This is a known limitation. The scikit-learn guide warns that correlated features can make permutation importance misleading because permuting one feature may leave similar predictive information available in another feature. Molnar's Interpretable Machine Learning chapter also notes that when features are dependent, independent permutation can create unrealistic combinations of feature values, which complicates interpretation.
The implication is practical: do not read a low individual importance score as proof that a feature is useless when correlated or dependent features are present. Consider grouped review, domain knowledge, and separate validation before changing the feature set.
Limitations and safeguards
Use these safeguards when reporting or acting on permutation feature importance:
- Do not treat it as causality. A high score drop means the fitted model relied on the feature under the permutation experiment. It does not show that changing the real-world feature would cause the outcome to change.
- Match the metric to the decision. A feature can be important for one metric and less important for another.
- Report the held-out split. Importance measured on one validation or test set may not hold on another sample.
- Use repeated permutations. The measured decrease can vary with the random shuffle, so a single permutation is less informative.
- Inspect correlated and dependent features. Individual shuffling can understate importance when similar information remains in other columns, and it can create unrealistic records when features normally move together.
- Be cautious with small held-out sets. Limited evaluation data can make ranks unstable.
- Separate interpretation from action. Feature removal, monitoring changes, or production updates should be validated beyond the importance table.
Reporting template
Use a compact table like this when sharing results with reviewers or stakeholders:
| Field | Example entry |
|---|---|
| Model | Fitted classifier or regressor name |
| Held-out data | Validation split from a specified date range or sampling process |
| Metric | Accuracy, R², or another selected score |
| Repeats | 20 permutations per feature |
| Random seed | 42 |
| Feature | feature_3 |
| Mean importance | 0.120 score decrease |
| Variation | 0.025 standard deviation across repeats |
| Interpretation | The model's held-out score is sensitive to this feature under permutation |
| Caveat | Check correlated features and validate decisions separately |
Bottom line
Permutation feature importance is a practical way to measure how much a fitted model's held-out score depends on each feature. Run it on validation or test data, repeat the shuffling, rank by mean score decrease, and interpret the result alongside variability, metric choice, and feature dependence. Use it to guide model review, not to claim causality or make irreversible feature decisions by itself.
