Evaluation and leakage
Introducido
Vocabulario
| English | Significado |
|---|---|
| confusion matrix | confusion matrix |
| accuracy/ˈækjʊrəsi/ | accuracy |
| precision/prɪˈsɪʒn/ | precision |
| recall/ˈriːkɔːl/ | recall |
English
The split and the matrix
- Train/validation/test; the test set is spent once. The confusion matrix counts TP, FP, FN, TN; accuracy, precision, recall read off it.
{exercise:uni-cs1306-split}
Práctica
The test set is —
Dificultad 1/5 · 100 monedas + un pequeño bono en el primer intento de hoy
Spent once: it becomes a second validation set otherwise.
English
The metrics, from the matrix
- Course matrix (test 200): TP 36, FP 10, FN 6, TN 148 — accuracy $(36+148)/200 = 0.92$; precision $36/46 = 0.783$; recall $36/42 = 0.857$; $F_1 = 0.818$. Base rate $21\%$ means the trivial always-on-time predictor scores 0.79: the recall number is where the meaning lives.
- 5-fold CV on Overnight: 0.905, 0.925, 0.915, 0.900, 0.935 — mean 0.916, sd 0.013: a competing 0.921 is within noise.
{exercise:uni-cs1306-metrics}
Práctica
From TP 36, FP 10, FN 6, TN 148: precision is —
Dificultad 1/5 · 100 monedas + un pequeño bono en el primer intento de hoy
36 / (36 + 10) = 0.783; recall 36/42 = 0.857.
English
Leakage is plumbing
- A midnight status-code feature encodes the label: training 0.99, honest held-out 0.90 — the label was whispering through the feature. The data card's provenance column is the defence.
- Classic traps: spending the test set on model choice; scaling before splitting; averaging fold accuracies over unequal folds. A score without a resampling story is an anecdote.
{exercise:uni-cs1306-leakage}
Práctica
A midnight status-code feature causes —
Dificultad 1/5 · 100 monedas + un pequeño bono en el primer intento de hoy
The label whispers through the feature - provenance is the defence.