214 Performance-Based Expert Judgement Weighting Using Moment Methods (PSAM-0264)

When we want to express the uncertainty about the value of a quantity of interest, we can do this by describing a probability distribution for it. Alternatively, we can provide information on the moments of the uncertain quantity, as the probability distribution of an uncertain quantity can be approximated arbitrarily well by describing enough moments for it. When considering the case where the information about the moments is obtained through expert judgment elicitation, we are confronted with the problem of what to do when different experts provide different assessments.

In a previous paper we addressed the problem of what a rational Decision Maker's (DM) assessment of moments should be when this DM bases his/her assessment only on the experts' moment assessments. We demonstrated that that the linear pool can be justified in an analogous but technically different way to linear pools for probability assessments. Cooke [1] argues that when rational consensus is pursued, a methodology using expert judgment should at least be reproducible, accountable, subject to empirical control, neutral and fair. In the paper we have described how these five principles can be satisfied when aggregating subjective assessments of moments. To satisfy the principle of empirical control, we proposed the use of performance based weighting, like Cooke does with the classical model when combining probability assessments.

In this paper we apply the moment model developed in the previous paper on expert judgment data gathered in various applications of the classical model, and compare the outcomes with the classical model. Both the classical and the moment model use a linear opinion pool to aggregate experts' assessments and base the choice of expert weights on their performance, but apply different measures to determine this expert performance. The five cases studied in this paper provide a mixed picture: neither method has a distinctly better performance, even though in all cases the models suggest completely different sets of weights to assign to the experts. Although the number of cases studied in this comparison is limited, the results at least suggest performance based weighting might be a good alternative to choosing the equal weights the DM linear opinion pool. This also holds from a theoretical point of view, i.e. to make the expert assessments susceptible to empirical control.