Glossary

Terms are listed alphabetically. The bracketed units say where the term is developed: U1 to U5 are the five units of the course (Chapters 1 to 14; see the Preface for the mapping).

Accuracy paradox
on imbalanced data the useless majority-class model scores high accuracy, so optimising accuracy pushes you towards it. (U2, U4)
AUC
area under the ROC curve; the probability a random positive scores above a random negative. (U4)
Back-propagation
the chain rule organised so each layer's error signal is computed once and reused. (U4)
Bias (statistical)
error from a model too rigid to represent the pattern; shows as high training error. (U1, U3)
Bias term
the intercept \(\theta _{0}\) or \(b\). A different meaning of the same word. (U3)
Centroid
the mean of the points in a cluster. (U5)
Chaining
single linkage joining two real groups through a thin line of intermediate points. (U5)
Classification
supervised learning with a categorical target. (U1)
Confusion matrix
the \(2 \times 2\) table of TP, FP, FN, TN. (U4)
Cost function \(J(\theta )\)
the loss averaged over the training set; a function of the parameters. (U3)
Cross-validation
k-fold: each fold is held out once; \(k \times\) candidates model fits in a search. (U3)
Curse of dimensionality
as \(n\) grows, points become sparse and distances become nearly equal, which breaks k-NN and k-means. (U2)
Decision boundary
the surface in feature space where the predicted class changes. (U4)
Dendrogram
the tree of merges; the height of a join is the distance at which it happened. (U5)
Design matrix \(X\)
\(m \times (n+1)\), one row per example, first column all ones. (U3)
Elbow
the last \(k\) that buys a large drop in WCSS. (U5)
Ensemble
combining several models (bagging, random forests, boosting); beyond this course but the usual next step after trees.
Entropy
\(-\sum p \log_{2}p\), the impurity of a node in bits. (U4)
Epoch
one pass over the whole training set. (U4)
Feature
one input column. The target is not a feature. (U1)
Feature extraction
building new features from combinations of the old (PCA). Contrast selection. (U2)
Feature selection
keeping a subset of the original columns; filter, wrapper or embedded. (U2)
Generalisation
performance on data not used in training. The only thing that counts. (U1)
Gini index
\(1 - \sum p^{2}\), an impurity measure without a logarithm. (U4)
Gradient descent
\(\theta\) := \(\theta - \alpha \nabla J\), repeated, with all parameters updated simultaneously. (U3)
Hinge loss
\(\max(0, 1 -\) y·f(x)); exactly zero beyond the margin, which is what makes the SVM sparse. (U4)
Hyper-parameter
set before training and chosen on validation data: \(\alpha , \lambda\), k, depth, \(C, \gamma\). (U1)
Hypothesis space
the set of functions an algorithm may choose from. (U1)
Imputation
filling missing values; the statistic must come from the training fold. (U2)
Inductive bias
the preference an algorithm has before seeing data, without which learning is impossible. (U1)
Information gain
impurity before a split minus the weighted impurity after it. (U4)
Instance / example
one row of the dataset. (U1)
k-means++
initialisation that spreads the starting centroids out, reducing poor local optima. (U5)
Lasso (L1)
penalty \(\lambda \sum |\theta |\); sets coefficients exactly to zero, so it selects features. (U3)
Lazy learner
one that stores the data and does the work at prediction time; k-NN. (U4)
Leakage
information from the test set influencing training; the score becomes optimistic by an unmeasurable amount. (U2)
Learning curve
error against training-set size; a persistent gap means variance, two flat high curves mean bias. (U3)
Linkage
how the distance between two clusters is defined: single, complete, average, Ward. (U5)
Log-loss
−[y log \(h + (1-y) \log(1-h)\)]; convex, and unbounded for confident mistakes. (U4)
Margin
the empty corridor around an SVM boundary, of width \(2/\Vert w\Vert\). (U4)
Masking
an outlier inflating \(\sigma\) until it hides inside its own band. (U2)
Multicollinearity
correlated features making the split of credit between their coefficients unstable. (U3)
Normal equation
\(\theta = (X^{T}X)^{-1}X^{T}y\); exact, no \(\alpha\), but \(O(n^{3})\). (U3)
One-hot encoding
one binary column per category; drop one level for a linear model. (U2)
Overfitting
fitting noise; training error low, test error much higher. (U1)
PCA
rotate to the eigenvectors of the covariance matrix and keep the largest; unsupervised, scale-sensitive, not feature selection. (U2)
Perceptron
a single threshold unit; converges iff the classes are linearly separable. (U4)
Precision
\(\text{TP}/(\text{TP} + \text{FP})\). Of what I flagged, how much was real. (U4)
Pruning
cutting a grown tree back (post-) or stopping it early (pre-). (U4)
PVE
proportion of variance explained, \(\lambda _{i}/\sum \lambda\). (U2)
\(R^{2}\)
fraction of variance removed relative to predicting \(\bar{y}\); can be negative on test data. (U3)
Recall
\(\text{TP}/(\text{TP} + \text{FN})\). Of what was real, how much I caught. (U4)
Regression
supervised learning with a continuous target. (U1)
Regularisation
a penalty on parameter size that trades a little bias for a lot of variance. (U3)
ReLU
\(\max(0\), z); derivative 1 while active, which is why deep networks train. (U4)
Residual
\(y - \hat{y}\); sums to zero when the model has an intercept. (U3)
Ridge (L2)
penalty \(\lambda \sum \theta ^{2}\); shrinks smoothly, never to exactly zero, and makes \(X^{T}X + \lambda I\) always invertible. (U3)
ROC curve
TPR against FPR over every threshold. (U4)
Silhouette
(\(b -\) a)/max(a, \(b\)) per point; unlike WCSS it has a genuine maximum over \(k\). (U5)
SMOTE
synthetic minority points interpolated between neighbours; training fold only. (U2)
Softmax
the multi-class generalisation of the sigmoid; outputs sum to 1. (U4)
Standardisation
\(z = (x - \mu )/\sigma\); the default scaling. (U2)
Stratified split
one that preserves the class proportions in each part. (U2)
Support vector
a training point on or inside the margin; the only points the SVM solution depends on. (U4)
Test set
opened once, at the end. Looking at it twice makes it a validation set. (U1)
\(\text{WCSS} /\) inertia
\(\sum \Vert x - \mu \Vert ^{2}\) within clusters; falls at every \(k\), so it cannot choose \(k\). (U5)
Underfitting
the model cannot represent the pattern; both errors high. (U1)
Validation set
used to choose hyper-parameters and model family. (U1)
Vanishing gradient
\(\sigma ' \le 0.25\) multiplied once per layer, so early layers stop learning. (U4)
Variance (statistical)
how much the fitted model changes with a different training sample; shows as a train–test gap. (U1, U3)