Feature Selection Techniques in Machine Learning: What Made Our Models 3x Faster

Adeel AslamLast Updated on November 4, 2026

15-16 Min Read TimeAdd as preferred on Google

Key Takeaways

Running cheap filters, then statistical tests, then RFECV cut one churn model from 234 to 34 features, with faster training and latency and flat accuracy.

• On the churn model, training fell from 45 to 8 minutes and latency from 120 to 35 ms, while accuracy moved from 84.2% to 84.7%.
• Accuracy often stays flat and sometimes drops after feature cuts.
• Run cheap filters first, then statistical tests, then a model-based search such as RFECV, because each expensive step then works on a shorter list.
• Fit every selector on training data only. In scikit-learn's random-label example, selecting on the full data first gave 0.76 test accuracy instead of 0.5.
• Selection can remove signal found only in feature interactions or rare cases.
• We alert when a feature's importance changes by more than 30% and release new feature sets to 10% of traffic first.

Start by splitting your data and measuring what the variance and correlation filters alone save.

A churn model with 234 features took 45 minutes to train and 120 ms to return a prediction. Our team cut it to 34 features. Training then took 8 minutes, latency fell to 35 ms, and accuracy moved slightly up, from 84.2% to 84.7%.

Feature selection techniques in machine learning remove inputs that add cost without adding signal. They fall into three families: filter methods that screen features with statistics, wrapper methods that retrain a model on candidate subsets, and embedded methods that rank features during training. This guide covers the staged workflow we run, scikit-learn code for each stage, the preprocessing that comes first, the leakage traps, and the cases where cutting features made a model worse.

What do feature selection techniques in machine learning do?

Feature selection techniques pick the smallest set of input columns that keeps a model's predictive quality and drop the rest before training. Filter methods score each feature on its own. Wrapper methods test subsets by retraining a model, and embedded methods let the model's own training rank the features. Most production pipelines combine them in sequence.

Feature count drives cost well beyond training time. Every feature needs a data source, transformation code, a quality check and documentation, and each one is a place where an upstream change can quietly break predictions.

One of our older churn models shows the effect. It had 847 features, needed 6 hours of training on our largest GPU instance and used 12GB of RAM per inference instance. Within 3 months, 30% of its features had become unreliable. When the model suddenly predicted that 95% of customers would churn, the team spent 8 hours tracing the problem to a single corrupted feature.

Removing features does not guarantee better accuracy. The dependable gains are faster training, lower latency and memory use, and simpler debugging, while accuracy often stays flat and sometimes drops.

How do filter, wrapper and embedded methods compare?

Filter methods are the cheapest and least precise. Wrapper methods are the most precise and the most expensive, and embedded methods sit between them because selection happens inside a single model fit. The table compares the families as scikit-learn implements them.

Filter methods

Filter methods rank features using a statistic computed without a predictive model, such as variance, correlation, a chi-square score or mutual information with the target. In scikit-learn they are VarianceThreshold and SelectKBest with a scoring function.

Filters fit the first pass. They run in seconds on hundreds of columns and remove constant, duplicated and unrelated features before any expensive step.

Filters judge one feature at a time, which means they miss features that only matter in combination. The scikit-learn User Guide (Feature selection) also states that the chi-square test should only be applied to non-negative features such as frequencies, and that mutual information methods need more samples for an accurate estimate.

Wrapper methods

Wrapper methods train a model on candidate subsets and keep the subset with the best validation score. Recursive feature elimination (RFE) fits a model, prunes the least important features and repeats until it reaches a target count, while RFECV runs that loop inside cross-validation and picks the count that scores best. SequentialFeatureSelector adds or removes one feature at a time.

Wrappers suit the final stage. They measure what the model you will deploy actually gains from each feature.

Cost is the drawback. The same scikit-learn guide notes that in backward sequential selection, going from m features to m minus 1 with k-fold cross-validation requires fitting m × k models, where RFE needs a single fit for that step.

Embedded methods

Embedded methods select features as a by-product of training. L1-penalized linear models such as Lasso or logistic regression push many coefficients to zero, tree ensembles report feature importances, and SelectFromModel keeps the features whose importance clears a threshold.

Embedded selection is the practical middle option for large feature sets. One fit gives a ranking, and the penalty strength (C for logistic regression, alpha for Lasso) controls how many features survive.

Tree importances can mislead. The scikit-learn guide on permutation importance states that impurity-based importance is strongly biased toward high-cardinality features, so an ID-like column can rank near the top without predicting anything on new data. Confirm the top features with permutation importance on held-out rows.

Our preprocessing pipeline for large ML datasets

Selection scores are only as reliable as the data behind them. Every project we run passes through the same five-stage preprocessing pipeline first, built for datasets too large for one machine, such as a 500GB Apache Parquet dataset processed on Azure.

1. The data loader reads the source files (Parquet in most of our projects, CSV or JSON otherwise) and streams them in chunks, which keeps any one stage from needing the whole dataset in memory.
2. The data cleaner removes duplicates, handles missing values and checks data types and value ranges.
3. The transformer normalizes, encodes and engineers features, running operations on GPUs where they help.
4. The validator checks four layers in order: schema, statistical properties, business rules and drift against reference data.
5. The output formatter writes processed data in the shape the training job or feature store expects.

Profiling happens before any of this, on a sample. Instead of loading a 100GB dataset to count null values, we sample 100,000 rows and extrapolate. The estimate is close enough to plan memory and compute.

Memory is the usual bottleneck. We downcast numeric types where precision allows, convert repeated strings to categories, drop unused columns as early as possible and use sparse representations for mostly-zero data. We also enforce one schema from the first chunk, because a column read as integers in one chunk and as strings in the next fails late and in confusing ways.

One preprocessing rule carries straight into feature selection. Any statistic the pipeline learns, such as an imputation mean or a scaling range, must come from the training data only.

How we cut a churn model from 234 to 34 features in three stages

The workflow runs cheap filters first, statistical tests second and a model-based search last, which means each expensive step works on a shorter list. On the 234-feature churn model, the count went to 180 after filtering, 89 after statistical tests and 34 after the model-based search.

Step 1. Split the data, then drop constant and correlated features

Splitting comes first. Every later step learns from the data, and any step that sees test rows leaks information into the score. After the split, VarianceThreshold removes features with the same value in every row, and a correlation check removes one feature from each pair above 0.95.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.feature_selection import VarianceThreshold

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

# Drop constant columns (fitted on training data only)
vt = VarianceThreshold(threshold=0.0).fit(X_train)
keep = X_train.columns[vt.get_support()]
X_train, X_test = X_train[keep], X_test[keep]

# Drop one feature from each pair with |correlation| > 0.95
corr = X_train.corr().abs()
upper = corr.where(np.triu(np.ones(corr.shape, dtype=bool), k=1))
to_drop = [col for col in upper.columns if (upper[col] > 0.95).any()]
X_train, X_test = X_train.drop(columns=to_drop), X_test.drop(columns=to_drop)

In the churn data, this step removed 31 zero-variance features, mostly old promotional flags, and 23 highly correlated ones, leaving 180. One pair, total_orders and order_count, had a correlation of 1.0. We kept total_orders because it was easier to explain.

Step 2. Score the remaining features with statistical tests

Statistical tests rank each surviving feature by its relationship with the target. We used chi-square for categorical features and mutual information for numerical ones, with a significance threshold of p < 0.05, and reached 89 features.

Chi-square needs non-negative inputs such as counts or one-hot flags. If other columns go through it, scale them with MinMaxScaler inside the pipeline so the scaler is fitted on training data alone.

Step 3. Pick the final feature count with RFECV

RFECV with a random forest tested feature counts from 10 to 100. Performance peaked at 34 features (AUC 0.847) and plateaued above that. The code below simplifies Step 2 to mutual information for all columns and chains it with RFECV in one Pipeline.

from sklearn.pipeline import Pipeline
from sklearn.feature_selection import SelectKBest, mutual_info_classif, RFECV
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold

selector = Pipeline([
    ("stats", SelectKBest(mutual_info_classif, k=89)),
    ("rfecv", RFECV(
        estimator=RandomForestClassifier(n_estimators=200, random_state=42, n_jobs=-1),
        step=5,
        min_features_to_select=10,
        cv=StratifiedKFold(n_splits=5),
        scoring="roc_auc",
    )),
])
selector.fit(X_train, y_train)

print("Selected features:", selector.named_steps["rfecv"].n_features_)
X_test_selected = selector.transform(X_test)

On this model, training fell from 45 minutes to 8, inference latency from 120 ms to 35 ms, and accuracy rose from 84.2% to 84.7%, with 70% fewer monitoring alerts from the feature pipeline. Treat these as one data point. A model that starts with less redundancy will gain less.

How do you keep feature selection free of data leakage?

Fit every selector on training data only, and refit it inside each fold when cross-validating. A selector that sees test rows picks features that happen to fit those rows. The test score then looks better than the model is.

The scikit-learn Common pitfalls guide shows how large the effect can be. It builds a binary classification problem with 200 samples, 10,000 randomly generated features and random labels, where accuracy should sit around 0.5, and selecting 25 features with SelectKBest on the full data before splitting produced a test accuracy of 0.76. Splitting first and selecting on the training set alone gave 0.5.

Wrapping selection and the model in one Pipeline is the simplest guard. Tools such as cross_val_score and GridSearchCV then refit the selector on each training fold automatically. The same rule covers scalers fitted before chi-square, imputation values and target statistics computed for high-cardinality columns. Keep a final hold-out set that no selection step has touched, and report that score.

When does feature selection make a model worse?

Feature selection hurts when it removes signal that only appears in combination with other features or in rare cases the filters treat as noise. These failures came from our own projects:

• In a pricing model, product category features looked weak one at a time, but their interactions with customer segments carried real signal.
• A correlation threshold of 0.8 instead of 0.95 removed complementary features that described different aspects of customer behavior.
• In fraud detection, correlation filtering dropped indicators that only correlated during fraudulent transactions, and variance thresholding dropped features that mattered only for rare fraud patterns.
• Automated selection removed features that business experts knew were critical during specific market conditions.

Business constraints come before any score. Ask product and compliance owners which features must stay, for regulatory or explainability reasons, and lock those in before the search starts. Our view is that stability also outweighs a small accuracy gain. A model that holds 85% accuracy month after month is easier to run and trust than one that swings between 90% and 75% as the data changes.

Feature selection for categorical, high-cardinality, time series and imbalanced data

Standard filters assume independent numeric columns. Four common data types break that assumption and need adjusted methods.

How do you select one-hot encoded categorical features as a group?

Score one-hot columns as a group. Dummy columns from the same category are correlated by construction, and dropping only some of them leaves the category half-encoded. We train a small model on each category group alone and keep or drop the whole group depending on whether it beats the baseline score.

High-cardinality ID columns

Columns such as user_id or product_id with thousands of values rarely work as raw inputs. We replace them with per-category aggregates (the target mean, the target standard deviation and how often the category appears), computed only for categories with at least 100 rows and filled with global values for the rest. These aggregates use the target, so compute them on training folds only, then check their value with permutation importance on held-out data.

Selecting time series features with Granger causality tests

Time series features need lag-aware tests. Plain correlation compares values at the same timestamp and misses a feature that predicts the target several periods later. We run Granger causality tests from statsmodels at lags up to 12 periods, keep features with p < 0.05 at any lag, and validate with time-ordered splits such as TimeSeriesSplit. Testing 12 lags per feature raises the chance of a false positive, so confirm the survivors on a later time window.

Imbalanced classes

Selection scores on imbalanced data are dominated by the majority class, and features that separate the rare class can rank low. We average mutual information over 10 balanced bootstrap samples, each pairing all minority rows with an equal-size random draw of majority rows. When the minority class is too small for stable scores, the fix is more training data; our guide to data augmentation techniques for improving model robustness covers how to create controlled variations of existing examples.

How to monitor feature selection in production

A selected feature set needs monitoring after release. We track each feature's importance against its training baseline, re-run selection when it drifts, and send any new set to a small share of traffic before full rollout.

Our drift monitor stores baseline importances at training time and compares them on every retraining run. A relative change above 30% in any feature's importance raises an alert. The re-selection pipeline then checks data quality (under 10% missing values and under 5% duplicate rows) and runs the same three-stage workflow on fresh data.

Overlap decides what happens next. When the new set shares at least 70% of its features with the current one, the update goes through automatically. Below 70%, a person reviews it, because a change that large may point to an upstream data problem rather than a shift in customer behavior.

New feature sets go to 10% of traffic first, with users assigned by a stable hash. Python's built-in hash() salts string hashes with a random value per process, which means the same user can land in a different group after a restart. A hashlib digest gives the same bucket every time.

import hashlib

def assign_group(user_id, treatment_share=0.10):
    digest = hashlib.sha256(str(user_id).encode()).hexdigest()
    bucket = int(digest, 16) % 100
    return "treatment" if bucket < treatment_share * 100 else "control"

We also review importances every quarter even when no alert fires. Slow drift rarely crosses a threshold.

Feature selection results from three production models

Across three of our models, cutting features made training or inference at least 3x faster, and accuracy or recommendation quality held or improved in each case. The table lists the figures and the step that made each one work.

In the recommender, quality went up after removing 95% of the features because the model had been overfitting to noise in rarely used ones. The size of any speedup depends on how much redundancy the original feature set carried, so these numbers show what is possible on our data and will differ on yours.

A four-week plan for adding feature selection to an existing model

A month is enough to move one production model through the full workflow without rushing validation.

1. In week 1, count features, record baseline training time, latency, memory and accuracy, and apply the variance and correlation filters to the training split.
2. In week 2, add statistical tests inside a Pipeline and check the reduced set on hold-out data.
3. In week 3, run RFECV or SelectFromModel with the production algorithm and compare the result against the current model.
4. In week 4, release the new set to 10% of traffic, turn on drift monitoring and document why each remaining feature stayed.

Frequently asked questions

Does feature selection always improve model accuracy?

Not always. Accuracy often stays about the same, and aggressive filtering can lower it by removing interaction effects or rare-class signals. The reliable gains are faster training, lower latency and fewer pipeline failures. Measure accuracy on a hold-out set the selection never touched before accepting any cut.

Is PCA a feature selection technique?

PCA is feature extraction. It builds new components from combinations of the original columns, so every original feature still has to be collected, and the components are harder to explain. Feature selection keeps a subset of the original columns, which is why teams that need explainable models usually prefer it.

Which feature selection method is fastest?

Filter methods such as VarianceThreshold and SelectKBest are fastest because they score features without training a model.

How many features should a production model keep?

There is no fixed number. Let cross-validation choose it: RFECV scores each candidate count and shows where performance peaks. In our churn model the curve flattened at 34 of 234 features. Pick the smallest count whose score is within noise of the best, since every extra feature adds pipeline and monitoring work.

How often should features be re-selected in production?

Re-select when monitoring shows drift, for example when a feature's importance moves more than 30% from its training baseline, and review the full set at least once a quarter otherwise.

Getting help with a slow or costly ML model

Start with the cheapest test. Split your data, run the variance and correlation filters on the training set, and measure how much training time and latency change.

If the filters alone remove a large share of columns, the model is a strong candidate for the full set of feature selection techniques in machine learning covered here, from RFECV to production monitoring. Keep the hold-out set untouched throughout, and lock in any features the business requires before the search begins. For teams that want outside help with a production model that has grown slow or expensive, Arbisoft's AI and machine learning team builds right-sized machine learning solutions, starting with a check of data quality and readiness.