Past studies have identified the following cognitive skills relevant to conceptual design: divergent thinking, spatial reasoning, visual thinking, abstract reasoning, and problem formulation (PF). Standardized tests are being developed to assess these skills. The tests on divergent thinking and visual thinking are fully developed and validated; this paper focuses on the development of a test of abstract reasoning in the context of engineering design. Similar to the two previous papers, this paper reports on the theoretical and empirical basis for skill identification and test development. Cognitive studies of human problem solving and design thinking revealed four indicators of abstract reasoning: qualitative deductive reasoning (DR), qualitative inductive reasoning (IR), analogical reasoning (AnR), and abductive reasoning (AbR). Each of these is characterized in terms of measurable indicators. The paper presents test construction procedures, trial runs, data collection, norming studies, and test refinement. Initial versions of the test were given to approximately 250 subjects to determine the clarity of the test problems, time allocation and to gauge the difficulty level. A protocol study was also conducted to assess test content validity. The beta version was given to approximately 100 students and the data collected was used for norming studies and test validation. Analysis of test results suggested high internal consistency; factor analysis revealed four eigenvalues above 1.0, indicating assessment of four different subskills by the test (as initially proposed by four indicators). The composite Cronbach’s alpha for all of the factors together was found to be 0.579. Future research will be conducted on criterion validity.

References

References
1.
Wilde
,
G.
,
1983
, “
The Skills and Practices of Engineering Designers Now and in the Future
,”
Des. Stud.
,
4
(1), pp.
21
34
.10.1016/0142-694X(83)90005-4
2.
Newstetter
,
W.
, and
Khan
,
S.
,
1997
, “
A Developmental Approach to Assessing Design Skills and Knowledge
,”
27th FIE Conference
, Pittsburgh, PA, Nov. 5–8, Vol.
2
.
3.
Lewis
,
W.
, and
Bonollo
,
E.
,
2002
, “
An Analysis of Professional Skills in Design: Implications for Education and Research
,”
Des. Stud.
23
(4), pp.
385
406
.10.1016/S0142-694X(02)00003-0
4.
Dym
,
C.
,
Agogino
,
A.
,
Eris
,
O.
,
Frey
,
D.
, and
Leifer
,
L.
,
2005
, “
Engineering Design Thinking, Teaching and Learning
,”
J. Eng. Educ.
,
94
(
1
), pp.
103
120
.10.1002/j.2168-9830.2005.tb00832.x
5.
Shah
,
J.
,
2005
, “
Identification, Measurement, and Development of Design Skills in Engineering Education
,”
International Conference on Engineering Design (ICED 05)
,
Melbourne, Australia
. Aug 15–18, Melbourne, Australia.
6.
Shah
,
J.
,
Smith
,
S.
, and
Woodward
,
J.
,
2009
, “
Development of Standardized Tests for Design Skills
,”
International Conference on Engineering Design (ICED 09)
,
Stanford, CA
, Aug. 24–27.
7.
Shah
,
J.
,
Millsap
,
R.
,
Woodward
,
J.
, and
Smith
,
S.
,
2012
, “
Applied Tests of Design Skills—Part 1: Divergent Thinking
,”
ASME J. Mech. Des.
,
134
(
2
), pp. p. 021005.10.1115/1.4005594
8.
Shah
,
J.
,
Woodward
,
J.
, and
Smith
,
S.
,
2013
, “
Applied Tests of Design Skills—Part II: Visual Thinking
,”
ASME J. Mech. Des.
,
135
(
7
), p.
071004
.10.1115/1.4024228
9.
Shah
,
J.
,
Millsap
,
R.
,
Woodward
,
J.
, and
Smith
,
S.
,
2010
, “
Applied Tests of Design Skills: Divergent Thinking Data Analysis and Reliability Studies
,”
2010 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
, Quebec, Canada Aug. 15–18.
10.
Shah
,
J.
,
Woodward
,
J.
, and
Smith
,
S.
,
2011
, “
Applied Tests of Engineering Design Skills: Visual Thinking Characterization, Test Development and Validation
,”
Proceedings of the 18th International Conference on Engineering Design (ICED 11)
,
Copenhagen, Denmark
, Aug. 15–19.
11.
Langer
,
S.
,
1953
,
Feeling and Form: A Theory of Art Developed From Philosophy in a New Key
,
Charles Scribner's Sons
,
New York
, p. 90.
12.
Khorshidi
,
M.
,
Woodward
,
J.
, and
Shah
,
J.
,
2012
, “
Towards a Comprehensive Test of Qualitative Reasoning Skill in Design
,”
2012 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
, Chicago, IL Aug. 12–15.
14.
Goel
,
A. K.
,
1997
, “
Design, Analogy, and Creativity
,”
IEEE Expert
,
12
(3), pp.
62
70
.10.1109/64.590078
15.
Schunn
,
C.
, and
Klahr
,
D.
,
1995
, “
A 4-Space Model of Scientific Discovery
,”
Proceedings of the 17th Annual Conference of the Cognitive Science Society
, Erlbaum, Hillsdale, NJ.
16.
Chiu
,
M.
,
2003
, “
Design Moves in Situated Design With Case-Based Reasoning
,”
Des. Stud.
,
24
(1), pp.
1
25
.10.1016/S0142-694X(02)00007-8
17.
Jonassen
,
D.
,
2000
, “
Toward a Design Theory of Problem Solving
,”
Educ. Technol. Res. Dev.
,
48
(
4
), pp.
63
85
.10.1007/BF02300500
20.
Campbell
,
J.
, and
Wolstencroft
,
J.
,
1990
, “
Structure and Significance of Analogical Reasoning
,”
Artif. Intell. Med.
,
2
(2), pp.
103
118
.10.1016/0933-3657(90)90032-M
21.
Roozenburg
,
N.
, and
Eekels
,
N.
,
1996
,
Product Design: Fundamentals and Methods
,
Wiley
,
Chichester, UK
.
22.
Torrens
,
D.
, and
Thompson
,
V. A.
,
1999
, “
Individual Differences and the Belief Bias Effect: Mental Models, Logical Necessity, and Abstract Reasoning
,”
Thinking Reasoning
,
5
, pp.
1
28
.10.1080/135467899394066
23.
Markovits
,
H.
,
Doyon
,
C.
, and
Simoneau
,
M.
,
2002
, “
Individual Differences in Working Memory and Conditional Reasoning With Concrete and Abstract Content
,”
Thinking Reasoning
,
8
(1), pp.
97
107
.10.1080/13546780143000143
24.
Staub
,
F.
, and
Stern
,
E.
,
1997
, “
Abstract Reasoning With Mathematical Constructs
,”
Int. J. Educ. Res.
,
27
(
1
), pp.
63
75
.10.1016/S0883-0355(97)88444-3
25.
Johnson-Laird
,
P.
,
1999
, “
Deductive Reasoning
,”
Annu. Rev. Psychol.
,
50
, pp.
109
135
.10.1146/annurev.psych.50.1.109
26.
Oaksford
,
M.
, and
Hahn
,
U.
,
2007
, “
Induction, Deduction, and Argument Strength in Human Reasoning and Argumentation
,”
Inductive Reasoning: Experimental, Developmental, and Computational Approaches
,
A.
Feeney
and
E.
Heit
, eds.,
Cambridge University
,
Cambridge
, pp.
269
327
.
27.
Holland
,
J.
,
Holyoak
,
K.
,
Nisbett
,
R.
, and
Thagard
,
P.
,
1986
,
Induction: Process of Inference, Learning, and Discovery
,
MIT
,
Cambridge
, MA.
28.
Klauer
,
K.
,
1989
, “
Teaching for Analogical Transfer as a Means of Improving Problem-Solving, Thinking and Learning
,”
Instructional Sci.
,
18
, pp.
179
192
.10.1007/BF00053358
29.
de Koning
,
E.
,
Sijtsma
,
K.
, and
Hamers
,
J.
,
2003
, “
Construction and Validation of a Test for Inductive Reasoning
,”
Eur. J. Psychol. Assess.
,
19
(
1
), pp.
24
39
.10.1027//1015-5759.19.1.24
30.
Throne
,
S.
,
2000
, “
Data Analysis in Qualitative Research
,”
Evidence-Based Nurs.
,
3
(3), pp.
68
70
.10.1136/ebn.3.3.68
31.
Brian
,
W.
,
1994
, “
Inductive Reasoning and Bounded Rationality
,”
Am. Econ. Rev.
,
84
(
2
), pp. 406–411.10.2307/2117868
32.
Nisbett
,
R.
,
Krantz
,
D.
,
Jepson
,
C.
, and
Kunda
,
Z.
,
1983
, “
The Use of Statistical Heuristics in Everyday Inductive Reasoning
,”
Psychol. Rev.
,
90
(
4
), pp.
339
363
.10.1037/0033-295X.90.4.339
33.
Klix
,
F.
,
2001
, “
Problem Solving: Deduction, Induction, and Analogical Reasoning
,”
Int. Encycl. Social Behav. Sci.
, pp.
12123
12130
.10.1016/B0-08-043076-7/00544-1
34.
Kedar-Cabelli
, S. T.,
1985
, “
Analogy-from a Unified Perspective
,” Department of Computer Science Laboratory for Computer Science Research, New Brunswick, NJ, Technical Report No. ML-TR-3.
35.
Findler
,
N.
,
1981
, “
Analogical Reasoning in Design Process
,”
Des. Stud.
,
2
(
1
), pp. 45–51.10.1016/0142-694X(81)90029-6
36.
Fu
,
K.
,
Chan
,
J.
,
Cagan
,
J.
,
Kotovsky
,
K.
,
Shunn
,
C.
, and
Wood
,
K.
,
2012
, “
The Meaning of Near and Far: The Impact of Structuring Design Database and the Effect of Distance of Analogy on Design Output
,”
ASME J. Mech. Des.
,
135
(
2
), p.
021007
.10.1115/1.4023158
37.
Casakin
,
H.
, and
Goldschmidt
,
G.
,
1999
, “
Expertise and the Use of Visual Analogy: Implications for Design Education
,”
Des. Stud.
,
20
(2), pp.
153
175
.10.1016/S0142-694X(98)00032-5
38.
Visser
,
W.
,
1996
, “
Two Functions of Analogical Reasoning in Design: A Cognitive-Psychology Approach
,”
Des. Stud.
,
17
(4), pp.
417
434
.10.1016/S0142-694X(96)00020-8
39.
Peirce
,
C.
,
1931–1958
,
Collected Papers of Charles Sanders Peirce
,
C.
Hartshorne
and
P.
Weiss
, eds., Harvard University Press, Cambridge, MA.
40.
Charniak
,
E.
, and
McDermott
,
P.
,
1985
,
Introduction to Artificial Intelligence
,
Addison Wesley
,
Menlo Park, CA
.
41.
Lawson
,
A.
,
2000
, “
Reasoning: the Generality of Hypothetico-Deductive Reasoning: Making Scientific Thinking Explicit
,”
Am. Biol. Teach.
,
62
(
7
), pp.
482
495
.10.1662/0002-7685(2000)062[0482:TGOHDR]2.0.CO;2
42.
Aliseda
,
A.
,
2003
, “
Mathematical Reasoning vs. Abductive Reasoning: A Structural Approach
,”
Synthese
,
134
(1–2), pp.
25
44
.10.1023/A:1022127429205
43.
Ng
,
H.
, and
Mooney
,
R.
,
1990
, “
On the Role of Coherence in Abductive Explanation
,”
Proceedings of the 8th National Conference on Artificial Intelligence
, Bostin, MA, Jul. 29–Aug. 3, pp.
337
342
.
44.
Stern
,
W.
,
1912
,
Psychologische Methoden der Intelligenz-Prüfung
,
T.
Barth
, ed., Barth,
Leipzig
,
Germany
.
45.
Raven
,
J.
,
Raven
,
J.
, and
Court
,
J.
,
2003
,
Manual for Raven's Progressive Matrices and Vocabulary Scales
,
Harcourt Assessment
,
San Antonio, TX
.
46.
Watson
,
G.
, and
Glaser
,
E. M.
,
1980
, Critical Thinking Appraisal, Psych Corp, Cleveland, OH.
47.
Hicks
,
R.
, and
Southey
,
G.
,
1990
, “
The Watson-Glaser Critical Thinking Appraisal and Performance of Business Management Students
,”
Psychcol. Test Bull.
,
3
, pp.
74
81
.
48.
Torrance
,
E.
,
1966
,
Torrance Tests of Creativity
,
Personnel
,
Princeton, NJ
.
49.
Kendal
,
I.
, and
Izard
,
J.
,
1988
, “
ACER mechanical reasoning test (revised)
,”
Psych. Bull.
,
1
.
50.
Arasteh
,
A.
, and
Arasteh
,
J.
,
1976
,
Creativity in Human Development: An Interpretive and Annotated Bibliography
,
Schenkman Pub. Co., Cambridge
,
MA and New York
.
51.
Murray
,
R. E.
,
1979
, “
Variability in MAT Results Within the Field of Education
,”
Psychol. Rep.
,
45
(
2
), pp.
665
666
.10.2466/pr0.1979.45.2.665
52.
Lawson
,
A.
,
1978
, “
Development and Validation of the Classroom Test of Formal Reasoning
,”
J. Res. Sci. Teach.
,
15
(
1
), pp.
11
24
.10.1002/tea.3660150103
53.
Klauer
,
K.
, and
Phye
,
G.
,
2008
, “
Inductive Reasoning: A Training Approach
,”
Rev. Educ. Res.
,
78
(
1
), pp.
85
123
.10.3102/0034654307313402
54.
Holyoak
,
K.
, and
Koh
,
K.
,
1987
, “
Surface and Structural Similarity in Analogical Transfer
,”
Memory Cognit.
,
15
(4), pp.
332
340
.10.3758/BF03197035
55.
Keane
,
M.
,
1987
, “
On Retrieving Analogues When Solving Problems
,”
Q. J. Exp. Psychol. A
,
39
(1), pp.
29
41
.10.1080/02724988743000015
56.
Gick
,
M.
, and
Holyoak
,
K.
,
1980
, “
Analogical Problem Solving
,”
Cognit. Psychol.
,
12
(3), pp.
306
355
.10.1016/0010-0285(80)90013-4
57.
Bearman
,
C.
,
Ball
,
L.
, and
Ormerod
,
T.
,
2002
, “
An Exploration of Real-World Analogical Problem Solving in Novices
,”
Proceeding of the 24th Conference of the Cognitive Science Society
, Fairfax, VA, Aug. 7–10.
58.
Urbina
,
S.
,
2004
,
Essentials of Psychological Testing
,
Wiley
, Hoboken, NJ.
59.
Kaplan
,
R.
, and
Saccuzzo
,
D.
,
2012
,
Psychological Testing, Principles, Applications, and Issues
,
8th ed.
,
Jon-David Hague Publishing
, Independence, KY.
60.
Anastasi
,
A.
, and
Urbina
,
S.
,
1997
,
Psychological Testing
,
7th ed.
,
Prentice Hall
, Upper Saddler River, NJ, p. 113.
61.
Khorshidi
,
M.
,
Shah
,
J.
, and
Woodward
,
J.
,
2013
, “
Rethinking the Comprehensive Test on Qualitative Reasoning for Designers
,”
ASME
Paper No. DETC2013-12403
.10.1115/DETC2013-12403
62.
Horn
,
J. L.
, and
Cattell
,
R. B.
,
1966
, “
Refinement and Test of the Theory of Fluid and Crystallized General Intelligences
,”
J. Educ. Psychol.
,
57
(
5
), pp.
253
–270.10.1037/h0023816
63.
Wilde
,
D. J.
,
2008
,
Teamology: The Construction and Organization of Effective Teams
,
Springer
, Berlin, Heidelberg.
You do not currently have access to this content.