Evaluation and leakage

Vocabulaire

English	Sens
confusion matrix	A table counting predictions for each combination of actual and predicted class.
accuracy/ˈækjʊrəsi/	The number of correct predictions divided by the total number of predictions.
precision/prɪˈsɪʒn/	True positives divided by all predicted positives, when that denominator is nonzero.
recall/ˈriːkɔːl/	True positives divided by all actual positives, when that denominator is nonzero.

English

The split and the matrix

Train/validation/test; the test set is spent once. The confusion matrix counts TP, FP, FN, TN; accuracy, precision, recall read off it.

{exercise:uni-cs1306-split}

Pratique

The test set is —

Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui

English

Course matrix (test 200): TP 36, FP 10, FN 6, TN 148 — accuracy $(36+148)/200 = 0.92$; precision $36/46 = 0.783$; recall $36/42 = 0.857$; $F_1 = 0.818$. Base rate $21\%$ means the trivial always-on-time predictor scores 0.79: the recall number is where the meaning lives.
5-fold CV on Overnight: 0.905, 0.925, 0.915, 0.900, 0.935 — mean 0.916, sd 0.013: a competing 0.921 is within noise.

{exercise:uni-cs1306-metrics}

Pratique

From TP 36, FP 10, FN 6, TN 148: precision is —

Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui

English

A midnight status-code feature encodes the label: training 0.99, honest held-out 0.90 — the label was whispering through the feature. The data card's provenance column is the defence.
Classic traps: spending the test set on model choice; scaling before splitting; averaging fold accuracies over unequal folds. A score without a resampling story is an anecdote.

{exercise:uni-cs1306-leakage}

Pratique

A midnight status-code feature causes —

Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui