Question-by-Question Interrater Analysis and Suggestions for Improvements of a Functional Model Scoring Rubric

Functional modeling is a tool used for system abstraction. By divorcing system function from component structure, functional modeling allows designers to more easily identify design opportunities and compartmentalize product functions, which can lead to innovation during the ideation process. In this paper, we examine the reliability of a rubric used to evaluate student-generated functional models by comparing interrater reliabilities on a question-by-question basis from a previous study where an examination of the reliability of each question was not assessed. We then suggest changes to the rubric in order to improve the rubric’s overall interrater reliability as well as its question-by-question interrater reliability. These rubric alterations include clarification of vague language, inclusion of examples and counter examples, and a procedure for handling nonexistent functional components as opposed to incorrect or “nonsensical” functional components. This work is in contribution to the ongoing development of this functional modeling rubric as an education instrument. As functional modeling becomes more widely accepted in the design community and in engineering curricula, it is important to have a validated evaluation metric with which to assess student-generated functional models.