Should Scoring Rules be 'Effective'?

by Robert F. Nau (Management Science 31:5 527-535)

Abstract: A scoring rule is a reward function for eliciting or evaluating forecasts expressed as discrete or continuous probabilities distributions. A rule is strictly proper if it encourages the forecaster to state his true subjective distributions, and effective if it is associated with a metric on the set of probability distributions. Recently, the property of effectiveness (which is stronger than strict properness) has been proposed as a desideratum for scoring rules for continuous probabilities, since in practice the forecast must be chosen from a low-dimensional set of "admissible" distributions. It is shown in this paper that what effectiveness implies, beyond strict properness, is not a monotonicity property but transitivity property, which is difficult to justify behaviorally. The logarithmic scoring rule is shown to violate the transitivity property, and hence it is not effective. The L-1 and L-infinity metrics are shown to admit no effective scoring rules. Some potential difficulties in interpreting admissible forecasts are also discussed.

Key words: Probability forecasting, evaluation of forecasts, proper scoring rules, effectiveness of scoring rules