Glossary¶
Terms are listed alphabetically. The bracketed units say where the term is developed: U1 to U5 are the five units of the course (Chapters 1 to 14; see the Preface for the mapping).
- Accuracy paradox
- on imbalanced data the useless majority-class model scores high accuracy, so optimising accuracy pushes you towards it. (U2, U4)
- AUC
- area under the ROC curve; the probability a random positive scores above a random negative. (U4)
- Back-propagation
- the chain rule organised so each layer's error signal is computed once and reused. (U4)
- Bias (statistical)
- error from a model too rigid to represent the pattern; shows as high training error. (U1, U3)
- Bias term
- the intercept \(\theta _{0}\) or \(b\). A different meaning of the same word. (U3)
- Centroid
- the mean of the points in a cluster. (U5)
- Chaining
- single linkage joining two real groups through a thin line of intermediate points. (U5)
- Classification
- supervised learning with a categorical target. (U1)
- Confusion matrix
- the \(2 \times 2\) table of TP, FP, FN, TN. (U4)
- Cost function \(J(\theta )\)
- the loss averaged over the training set; a function of the parameters. (U3)
- Cross-validation
- k-fold: each fold is held out once; \(k \times\) candidates model fits in a search. (U3)
- Curse of dimensionality
- as \(n\) grows, points become sparse and distances become nearly equal, which breaks k-NN and k-means. (U2)
- Decision boundary
- the surface in feature space where the predicted class changes. (U4)
- Dendrogram
- the tree of merges; the height of a join is the distance at which it happened. (U5)
- Design matrix \(X\)
- \(m \times (n+1)\), one row per example, first column all ones. (U3)
- Elbow
- the last \(k\) that buys a large drop in WCSS. (U5)
- Ensemble
- combining several models (bagging, random forests, boosting); beyond this course but the usual next step after trees.
- Entropy
- \(-\sum p \log_{2}p\), the impurity of a node in bits. (U4)
- Epoch
- one pass over the whole training set. (U4)
- Feature
- one input column. The target is not a feature. (U1)
- Feature extraction
- building new features from combinations of the old (PCA). Contrast selection. (U2)
- Feature selection
- keeping a subset of the original columns; filter, wrapper or embedded. (U2)
- Generalisation
- performance on data not used in training. The only thing that counts. (U1)
- Gini index
- \(1 - \sum p^{2}\), an impurity measure without a logarithm. (U4)
- Gradient descent
- \(\theta\) := \(\theta - \alpha \nabla J\), repeated, with all parameters updated simultaneously. (U3)
- Hinge loss
- \(\max(0, 1 -\) y·f(x)); exactly zero beyond the margin, which is what makes the SVM sparse. (U4)
- Hyper-parameter
- set before training and chosen on validation data: \(\alpha , \lambda\), k, depth, \(C, \gamma\). (U1)
- Hypothesis space
- the set of functions an algorithm may choose from. (U1)
- Imputation
- filling missing values; the statistic must come from the training fold. (U2)
- Inductive bias
- the preference an algorithm has before seeing data, without which learning is impossible. (U1)
- Information gain
- impurity before a split minus the weighted impurity after it. (U4)
- Instance / example
- one row of the dataset. (U1)
- k-means++
- initialisation that spreads the starting centroids out, reducing poor local optima. (U5)
- Lasso (L1)
- penalty \(\lambda \sum |\theta |\); sets coefficients exactly to zero, so it selects features. (U3)
- Lazy learner
- one that stores the data and does the work at prediction time; k-NN. (U4)
- Leakage
- information from the test set influencing training; the score becomes optimistic by an unmeasurable amount. (U2)
- Learning curve
- error against training-set size; a persistent gap means variance, two flat high curves mean bias. (U3)
- Linkage
- how the distance between two clusters is defined: single, complete, average, Ward. (U5)
- Log-loss
- −[y log \(h + (1-y) \log(1-h)\)]; convex, and unbounded for confident mistakes. (U4)
- Margin
- the empty corridor around an SVM boundary, of width \(2/\Vert w\Vert\). (U4)
- Masking
- an outlier inflating \(\sigma\) until it hides inside its own band. (U2)
- Multicollinearity
- correlated features making the split of credit between their coefficients unstable. (U3)
- Normal equation
- \(\theta = (X^{T}X)^{-1}X^{T}y\); exact, no \(\alpha\), but \(O(n^{3})\). (U3)
- One-hot encoding
- one binary column per category; drop one level for a linear model. (U2)
- Overfitting
- fitting noise; training error low, test error much higher. (U1)
- PCA
- rotate to the eigenvectors of the covariance matrix and keep the largest; unsupervised, scale-sensitive, not feature selection. (U2)
- Perceptron
- a single threshold unit; converges iff the classes are linearly separable. (U4)
- Precision
- \(\text{TP}/(\text{TP} + \text{FP})\). Of what I flagged, how much was real. (U4)
- Pruning
- cutting a grown tree back (post-) or stopping it early (pre-). (U4)
- PVE
- proportion of variance explained, \(\lambda _{i}/\sum \lambda\). (U2)
- \(R^{2}\)
- fraction of variance removed relative to predicting \(\bar{y}\); can be negative on test data. (U3)
- Recall
- \(\text{TP}/(\text{TP} + \text{FN})\). Of what was real, how much I caught. (U4)
- Regression
- supervised learning with a continuous target. (U1)
- Regularisation
- a penalty on parameter size that trades a little bias for a lot of variance. (U3)
- ReLU
- \(\max(0\), z); derivative 1 while active, which is why deep networks train. (U4)
- Residual
- \(y - \hat{y}\); sums to zero when the model has an intercept. (U3)
- Ridge (L2)
- penalty \(\lambda \sum \theta ^{2}\); shrinks smoothly, never to exactly zero, and makes \(X^{T}X + \lambda I\) always invertible. (U3)
- ROC curve
- TPR against FPR over every threshold. (U4)
- Silhouette
- (\(b -\) a)/max(a, \(b\)) per point; unlike WCSS it has a genuine maximum over \(k\). (U5)
- SMOTE
- synthetic minority points interpolated between neighbours; training fold only. (U2)
- Softmax
- the multi-class generalisation of the sigmoid; outputs sum to 1. (U4)
- Standardisation
- \(z = (x - \mu )/\sigma\); the default scaling. (U2)
- Stratified split
- one that preserves the class proportions in each part. (U2)
- Support vector
- a training point on or inside the margin; the only points the SVM solution depends on. (U4)
- Test set
- opened once, at the end. Looking at it twice makes it a validation set. (U1)
- \(\text{WCSS} /\) inertia
- \(\sum \Vert x - \mu \Vert ^{2}\) within clusters; falls at every \(k\), so it cannot choose \(k\). (U5)
- Underfitting
- the model cannot represent the pattern; both errors high. (U1)
- Validation set
- used to choose hyper-parameters and model family. (U1)
- Vanishing gradient
- \(\sigma ' \le 0.25\) multiplied once per layer, so early layers stop learning. (U4)
- Variance (statistical)
- how much the fitted model changes with a different training sample; shows as a train–test gap. (U1, U3)