Crowdsourcing the Evaluation of Design Concepts With Empathic Priming

The evaluation of design concepts is a time consuming and resource intensive process. Crowdsourcing evaluations has been examined in previous work as a means to reduce the need for expert raters, while achieving similar evaluation results. This paper examines the impact of empathically priming novice raters on their evaluation of alternative design concepts. The rating system is based on a pairwise comparison method that requires minimal training of novice raters. In a pilot study the pairwise method for crowdsourcing evaluations is compared with crowdsourced evaluations using non-pairwise rating scales and with the evaluations of expert raters. The primary study incorporates written and empathic priming strategies to determine their impact on novice raters’ evaluation of concepts. Raters are asked to consider several criteria, including novelty, feasibility, clarity (of the concept), usefulness, ease of use, and overall worthiness of further development. Results offer insight into the criteria that are most relevant to novice raters and whether empathic priming has a significant effect on those evaluations.