Does your confidence match your track record?
Confidence can be checked against outcomes over many predictions. If events you label 80% likely occur about eight times out of ten across a large enough set, that level is well calibrated. One failed prediction does not by itself disprove the estimate, because 80% still leaves room for failure.
Go deeper
Treat confidence as something measurable
Suppose you record a series of forecasts and label some outcomes 80% likely. Across many comparable forecasts, roughly 80% of those outcomes should occur if that confidence level is well calibrated.
Calibration compares stated probability with observed frequency. It cannot be judged reliably from a single result or a tiny sample.
If nearly every event labelled 80% happens, the forecasts may be too cautious. If only half happen, they may be too confident.
Calibration also differs from usefulness. Predicting the usual rate every time can be calibrated while failing to distinguish easy cases from difficult ones. The measure answers a specific question: when you express a particular level of confidence, how closely does reality match it over time?