Understanding the Impact of Expertise on Design Outcome: An Approach Based on Concept Inventories and Item Response Theory

Research on expertise in design has focused primarily on understanding expert-novice differences. Although it is well established that experts perform better than novices, there is a lack of formal methods to quantify the potential impact of expertise on the quality of design outcomes. The research question addressed in this paper is: How can the impact of expertise on the quality of design solutions be quantified? Quantifying such impacts can be of particular importance in product development, recruitment processes and design competitions. We utilize an approach based on Item Response Theory (IRT) and Concept Inventories (CI) for expertise quantification. We then investigate and validate the impact of expertise on solution quality through a behavioral experiment involving a track design problem. The results highlight the usefulness of the proposed approach and provide a functional relationship between expertise and solution quality. We also observe behavioral differences between participants with varying scores on a test taken in the behavioral experiment. The proposed approach could be used in the future work to quantify learning.