A bank wants a Model to flag fraudulent card payments. Of every 100 transactions, about 1 is fraud and 99 are fine, shown as the thin red slice of the ring. The model reads each transaction and answers "fraud" or "fine", and the Flags it raises go to a human reviewer.
How do we decide whether the model is any good? The obvious number is accuracy: the share of all its answers that were correct. It is simple and often useful. This lesson shows the situation where it quietly lies, and what to measure instead.
💡Good to know
Fraud, disease and equipment faults are all rare events, so this problem turns up everywhere.