**Should Scoring Rules be 'Effective'?**

by Robert F. Nau (*Management Science* **31**:5 527-535)

**Abstract:** A scoring rule is a reward function for eliciting
or evaluating forecasts expressed as discrete or continuous probabilities
distributions. A rule is *strictly proper* if it encourages
the forecaster to state his true subjective distributions, and
*effective* if it is associated with a metric on the set
of probability distributions. Recently, the property of effectiveness
(which is stronger than strict properness) has been proposed as
a desideratum for scoring rules for continuous probabilities,
since in practice the forecast must be chosen from a low-dimensional
set of "admissible" distributions. It is shown in this
paper that what effectiveness implies, beyond strict properness,
is not a monotonicity property but *transitivity* property,
which is difficult to justify behaviorally. The logarithmic scoring
rule is shown to violate the transitivity property, and hence
it is not effective. The L-1 and L-infinity metrics are shown
to admit no effective scoring rules. Some potential difficulties
in interpreting admissible forecasts are also discussed.

**Key words: **Probability forecasting, evaluation of forecasts,
proper scoring rules, effectiveness of scoring rules