📘 Why do features make the model?

What you’ll learn

1. Why Features MatterExplain what features are, how the feature matrix is structured, and why feature representation can determine model performance.This lesson defines a feature as a measurable input property and frames feature engineering as crafting representations that expose signal to a model. It introduces the X feature matrix and y target convention and stresses that a model can only use information its features make accessible. It contrasts feature requirements across model families, noting that distance- and gradient-based models need scaling while tree models do not. It positions domain knowledge and an iterative feature-model loop as central to applied work.
2. Numeric TransformsApply scaling, log/power transforms, and binning correctly while fitting all parameters on the training set.This lesson covers standardization to zero mean and unit variance, min-max scaling to a fixed range, and robust scaling with the median and IQR for outlier-prone features. It explains log and power transforms such as Box-Cox and Yeo-Johnson for reducing skew and changing distribution shape. It introduces binning into equal-width or quantile buckets to capture nonlinear effects at the cost of granularity. Throughout, it emphasizes that every statistic is computed on training data and applied unchanged to test data.
3. Categorical and Missing DataChoose appropriate categorical encodings and missing-value strategies, fitting all parameters on training data to avoid leakage.This lesson distinguishes one-hot encoding for nominal categories from ordinal encoding for genuinely ordered levels, and introduces target (mean) encoding for high-cardinality features along with its overfitting and leakage risks and safeguards like out-of-fold encoding. It then covers missing data: why most algorithms require complete inputs, simple mean, median, or most-frequent imputation, missing-value indicator columns for informative missingness, and model-based methods such as kNN and iterative imputation. Both encoders and imputers learn their parameters from training data and apply them unchanged to test data. The unifying theme is treating every such step as a fitted transform with a defined policy for unseen or absent values.
4. Time and Text FeaturesEngineer informative datetime and text features, including cyclical encoding and TF-IDF, with training-fit vocabularies.This lesson decomposes timestamps into calendar components and uses sine-cosine encoding to represent cyclical variables on a circle so adjacent times stay close. It introduces the bag-of-words model as a sparse vector of term counts over a training-derived vocabulary. It explains TF-IDF weighting, which downweights ubiquitous words and emphasizes distinctive ones, and reviews preprocessing choices like tokenization, stop words, and n-grams. It reinforces that vocabularies and preprocessing parameters are fit on training data.
5. Interactions and SelectionCreate interaction and polynomial features and apply filter, wrapper, and embedded feature-selection methods.This lesson explains interaction features that capture joint effects linear models cannot otherwise represent and polynomial features that let linear models fit curves at the cost of dimensionality. It categorizes feature selection into filter methods using model-agnostic statistics, wrapper methods like recursive feature elimination, and embedded methods such as L1 (lasso) regularization that zeroes coefficients during fitting. It connects selection to the curse of dimensionality, where sparse high-dimensional data weakens distance-based reasoning. It motivates parsimony as a path to better generalization.
6. Guided Project: Build a Feature PipelineBuild a leakage-safe feature pipeline that fits on training data, transforms train and test consistently, and is cross-validated end to end.This guided project assembles a column-typed feature pipeline on a mixed tabular dataset, splitting data before computing any statistics. It routes numeric, categorical, and datetime columns to appropriate impute-and-transform branches via a column transformer, with each step exposing fit and transform. It fits the full pipeline on training data only and applies it to test data, then cross-validates the entire pipeline so every transform refits within each fold. The deliverable is a fitted, serialized pipeline plus a data dictionary documenting the engineered features.

Questions this course answers

Grounded in trusted sources

• Alice Zheng and Amanda Casari, Feature Engineering for Machine Learning (O’Reilly)
• scikit-learn User Guide — preprocessing and pipelines, https://scikit-learn.org/stable/modules/preprocessing.html
• Max Kuhn and Kjell Johnson, Feature Engineering and Selection (CRC / online book)
• Pedro Domingos, “A Few Useful Things to Know about Machine Learning,” CACM — features beat algorithms often
• Google Rules of ML — feature hygiene and training-serving skew notes, https://developers.google.com/machine-learning/guides/rules-of-ml

Every Wunder lesson is built from real, reputable sources — never invented.

Related courses