Evaluation and leakage
Introduced
Vocabulary
| English | Meaning |
|---|---|
| confusion matrix | A table counting predictions for each combination of actual and predicted class. |
| accuracy/ˈækjʊrəsi/ | The number of correct predictions divided by the total number of predictions. |
| precision/prɪˈsɪʒn/ | True positives divided by all predicted positives, when that denominator is nonzero. |
| recall/ˈriːkɔːl/ | True positives divided by all actual positives, when that denominator is nonzero. |
The split and the matrix
- Train/validation/test; the test set is spent once. The confusion matrix counts TP, FP, FN, TN; accuracy, precision, recall read off it.
{exercise:uni-cs1306-split}
Practice
The test set is —
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Spent once: it becomes a second validation set otherwise.
The metrics, from the matrix
- Course matrix (test 200): TP 36, FP 10, FN 6, TN 148 — accuracy $(36+148)/200 = 0.92$; precision $36/46 = 0.783$; recall $36/42 = 0.857$; $F_1 = 0.818$. Base rate $21\%$ means the trivial always-on-time predictor scores 0.79: the recall number is where the meaning lives.
- 5-fold CV on Overnight: 0.905, 0.925, 0.915, 0.900, 0.935 — mean 0.916, sd 0.013: a competing 0.921 is within noise.
{exercise:uni-cs1306-metrics}
Practice
From TP 36, FP 10, FN 6, TN 148: precision is —
Difficulty 1/5 · 100 coins + a small bonus on first completion today
36 / (36 + 10) = 0.783; recall 36/42 = 0.857.
Leakage is plumbing
- A midnight status-code feature encodes the label: training 0.99, honest held-out 0.90 — the label was whispering through the feature. The data card's provenance column is the defence.
- Classic traps: spending the test set on model choice; scaling before splitting; averaging fold accuracies over unequal folds. A score without a resampling story is an anecdote.
{exercise:uni-cs1306-leakage}
Practice
A midnight status-code feature causes —
Difficulty 1/5 · 100 coins + a small bonus on first completion today
The label whispers through the feature - provenance is the defence.