As we continue to model more complex systems, the validation of dynamical responses has come to the forefront of modeling and simulation. One form of dynamic response is when the output is a function of time. The proper evaluation of functional data over an array of desired input parameters is critical to achieving a robust validation assessment of a simulation model. We extend the correlation analysis (CORA) objective rating system to validate functional data across experimental regions. Functional regression analysis is used to generate surrogate estimations of the system response functions at points within the region where experimental observations are absent. These CORA scores provide a measure of disagreement at each desired parameter configuration. An overall score for model validity is achieved using a weighted linear combination of the individual CORA scores. Finally, an improved CORA size scoring metric is introduced.

References

References
1.
ISO,
2014
, “Road Vehicles—Objective Rating Metric for Non-Ambiguous Signals,”
International Organization for Standardization
,
Geneva, Switzerland
, Standard No.
ISO/TS 18571:2014
.https://www.iso.org/standard/62937.html
2.
Storm
,
S.
,
Hill
,
R. R.
, and
Pignatiello
,
J. J.
,
2013
, “
A Response Surface Methodology for Modeling Time Series Response Data
,”
Qual. Reliab. Eng. Int.
,
29
(
5
), pp.
771
778
.
3.
Balci
,
O.
,
1994
, “
Validation, Verification, and Testing Techniques Throughout the Life Cycle of a Simulation Study
,”
Ann. Oper. Res.
,
53
(
1
), pp.
121
173
.
4.
Davis
,
P. K.
,
1992
, “Generalizing Concepts and Methods of Verification, Validation, and Accreditation (VV&A) for Military Simulations,” RAND Corporation, Santa Monica, CA, Technical Report No.
RAND/R-4249-ACQ
.https://www.rand.org/pubs/reports/R4249.html
5.
Sargent
,
R. G.
,
2010
, “
Verification and Validation of Simulation Models
,”
IEEE
Winter Simulation Conference
, Washington, DC, Dec. 9–12, pp.
166
183
.
6.
Law
,
A. M.
,
2007
,
Simulation Modeling and Analysis
,
4th ed.
,
McGraw-Hill
,
New York
.
7.
Shannon
,
R. E.
,
1975
,
Systems Simulation: The Art and Science
,
Prentice Hall
,
Englewood Cliffs, NJ
.
8.
Balci
,
O.
, and
Sargent
,
R. G.
,
1981
, “
A Methodology for Cost-Risk Analysis in the Statistical Validation of Simulation Models
,”
Commun. ACM
,
24
(
4
), pp.
190
197
.
9.
Naylor
,
T. H.
, and
Finger
,
J. M.
,
1967
, “
Verification of Computer Simulation Models
,”
Manage. Sci.
,
14
(
2
), pp.
B92
B101
.
10.
Garratt
,
M.
,
1974
, “
Statistical Validation of Simulation Models
,”
Summer Computer Simulation Conference
, Houston, TX, pp.
915
926
.
11.
Fishman
,
G. S.
, and
Kiviat
,
P. J.
,
1967
, “
The Analysis of Simulation-Generated Time Series
,”
Manage. Sci.
,
13
(
7
), pp.
525
557
.
12.
Gallant
,
A.
,
Gerig
,
T. M.
, and
Evans
,
J.
,
1974
, “
Time Series Realizations Obtained According to an Experimental Design
,”
J. Am. Stat. Assoc.
,
69
(
347
), pp.
639
645
.
13.
Sarin
,
H.
,
Kokkolaras
,
M.
,
Hulbert
,
G.
,
Papalambros
,
P.
,
Barbat
,
S.
, and
Yang
,
R.-J.
,
2010
, “
Comparing Time Histories for Validation of Simulation Models: Error Measures and Metrics
,”
ASME J. Dyn. Syst. Meas. Control
,
132
(
6
), p.
061401
.
14.
Van Horn
,
R. L.
,
1971
, “
Validation of Simulation Results
,”
Manage. Sci.
,
17
(
5
), pp.
247
258
.
15.
Balci
,
O.
,
1998
, “
Verification, Validation, and Testing
,”
Handbook of Simulation
, Vol.
10
,
J.
Banks
, ed.,
Wiley
,
New York
, pp.
335
393
.
16.
Hills
,
R. G.
, and
Trucano
,
T. G.
,
1999
, “Statistical Validation of Engineering and Scientific Models: Background,” Sandia National Laboratories, Albuquerque, NM, Report No.
SAND99-1256
.https://inis.iaea.org/search/search.aspx?orig_q=RN:30047122
17.
Buranathiti
,
T.
,
Cao
,
J.
,
Chen
,
W.
,
Baghdasaryan
,
L.
, and
Xia
,
Z. C.
,
2006
, “
Approaches for Model Validation: Methodology and Illustration on a Sheet Metal Flanging Process
,”
ASME J. Manuf. Sci. Eng.
,
128
(
2
), pp.
588
597
.
18.
Chen
,
W.
,
Baghdasaryan
,
L.
,
Buranathiti
,
T.
, and
Cao
,
J.
,
2004
, “
Model Validation Via Uncertainty Propagation and Data Transformations
,”
AIAA J.
,
42
(
7
), pp.
1406
1415
.
19.
Ghanem
,
R. G.
,
Doostan
,
A.
, and
Red-Horse
,
J.
,
2008
, “
A Probabilistic Construction of Model Validation
,”
Comput. Methods Appl. Mech. Eng.
,
197
(
29–32
), pp.
2585
2595
.
20.
Hills
,
R. G.
,
2006
, “
Model Validation: Model Parameter and Measurement Uncertainty
,”
ASME J. Heat Transfer
,
128
(
4
), pp.
339
351
.
21.
Rebba
,
R.
, and
Mahadevan
,
S.
,
2008
, “
Computational Methods for Model Reliability Assessment
,”
Reliab. Eng. Syst. Saf.
,
93
(
8
), pp.
1197
1207
.
22.
Oberkampf
,
W. L.
, and
Barone
,
M. F.
,
2006
, “
Measures of Agreement Between Computation and Experiment: Validation Metrics
,”
J. Comput. Phys.
,
217
(
1
), pp.
5
36
.
23.
Ferson
,
S.
,
Oberkampf
,
W. L.
, and
Ginzburg
,
L.
,
2008
, “
Model Validation and Predictive Capability for the Thermal Challenge Problem
,”
Comput. Methods Appl. Mech. Eng.
,
197
(
29–32
), pp.
2408
2430
.
24.
Chen
,
W.
,
Xiong
,
Y.
,
Tsui
,
K.-L.
, and
Wang
,
S.
,
2008
, “
A Design-Driven Validation Approach Using Bayesian Prediction Models
,”
ASME J. Mech. Des.
,
130
(
2
), p.
021101
.
25.
Jiang
,
X.
, and
Mahadevan
,
S.
,
2008
, “
Bayesian Wavelet Method for Multivariate Model Assessment of Dynamic Systems
,”
J. Sound Vib.
,
312
(
4–5
), pp.
694
712
.
26.
Jiang
,
X.
, and
Mahadevan
,
S.
,
2008
, “
Bayesian Validation Assessment of Multivariate Computational Models
,”
J. Appl. Stat.
,
35
(
1
), pp.
49
65
.
27.
Rebba
,
R.
,
Huang
,
S.
,
Liu
,
Y.
, and
Mahadevan
,
S.
,
2005
, “
Statistical Validation of Simulation Models
,”
Int. J. Mater. Prod. Technol.
,
25
(
1–3
), pp.
164
181
.
28.
Rebba
,
R.
, and
Mahadevan
,
S.
,
2006
, “
Model Predictive Capability Assessment Under Uncertainty
,”
AIAA J.
,
44
(
10
), pp.
2376
2384
.
29.
Rebba
,
R.
,
Mahadevan
,
S.
, and
Huang
,
S.
,
2006
, “
Validation and Error Estimation of Computational Models
,”
Reliab. Eng. Syst. Saf.
,
91
(
10–11
), pp.
1390
1397
.
30.
Rebba
,
R.
, and
Mahadevan
,
S.
,
2006
, “
Validation of Models With Multivariate Output
,”
Reliab. Eng. Syst. Saf.
,
91
(
8
), pp.
861
871
.
31.
Wang
,
S.
,
Chen
,
W.
, and
Tsui
,
K.-L.
,
2009
, “
Bayesian Validation of Computer Models
,”
Technometrics
,
51
(
4
), pp.
439
451
.
32.
Kleijnen
,
J. P.
,
1992
, “
Regression Metamodels for Simulation With Common Random Numbers: Comparison of Validation Tests and Confidence Intervals
,”
Manage. Sci.
,
38
(
8
), pp.
1164
1185
.
33.
Kleijnen
,
J. P.
,
1995
, “
Verification and Validation of Simulation Models
,”
Eur. J. Oper. Res.
,
82
(
1
), pp.
145
162
.
34.
Kleijnen
,
J. P.
,
1995
, “
Sensitivity Analysis and Optimization of System Dynamics Models: Regression Analysis and Statistical Design of Experiments
,”
Syst. Dyn. Rev.
,
11
(
4
), pp.
275
288
.
35.
Kleijnen
,
J. P.
,
Feelders
,
A. J.
, and
Cheng
,
R. C.
,
1998
, “
Bootstrapping and Validation of Metamodels in Simulation
,”
IEEE
Winter Simulation Conference
, Washington, DC, Dec. 13–16, pp.
701
706
.
36.
Kleijnen
,
J. P.
, and
Sargent
,
R. G.
,
2000
, “
A Methodology for Fitting and Validating Metamodels in Simulation
,”
Eur. J. Oper. Res.
,
120
(
1
), pp.
14
29
.
37.
Kleijnen
,
J. P.
, and
Deflandre
,
D.
,
2006
, “
Validation of Regression Metamodels in Simulation: Bootstrap Approach
,”
Eur. J. Oper. Res.
,
170
(
1
), pp.
120
131
.
38.
Bendat
,
J. S.
, and
Piersol
,
A. G.
,
1980
,
Engineering Applications of Correlation and Spectral Analysis
, Vol.
315
,
Wiley-Interscience
,
New York
, p.
1
.
39.
Jenkins
,
G. M.
,
1961
, “
General Considerations in the Analysis of Spectra
,”
Technometrics
,
3
(
2
), pp.
133
166
.
40.
Donelly
,
B.
,
Morgan
,
R. M.
, and
Eppinger
,
R. H.
,
1983
, “
Durability, Repeatability and Reproducibility of the NHTSA Side Impact Dummy
,”
SAE
Paper No. 831624.
41.
Russell
,
D. M.
,
1997
, “
Error Measures for Comparing Transient Data—Part I: Development of a Comprehensive Error Measure
,”
68th Shock and Vibration Symposium
, Hunt Valley, MD, Nov. 3–6, pp.
175
184
.http://roadsafellc.com/NCHRP22-24/Literature/Papers/Metrics/Russell-ErrorMeasures.pdf
42.
Sprague
,
M.
, and
Geers
,
T.
,
2004
, “
A Spectral-Element Method for Modeling Cavitation in Transient Fluid-Structure Interaction
,”
Int. J. Numer. Methods Eng.
,
60
(
15
), pp.
2467
2499
.
43.
Gehre
,
C.
,
Gades
,
H.
, and
Wernicke
,
P.
,
2009
, “
Objective Rating of Signals Using Test and Simulation Responses
,”
21st International Technical Conference on the Enhanced Safety of Vehicles Conference
(
ESV
), Stuttgart, Germany, June 15–18, pp.
15
18
.
44.
Zhan
,
Z.
,
Fu
,
Y.
, and
Yang
,
R.-J.
,
2011
, “Enhanced Error Assessment of Response Time Histories (EEARTH) Metric and Calibration Process,”
SAE
Paper No. 2011-01-0245.
45.
Jiang
,
X.
,
Yang
,
R.-J.
,
Barbat
,
S.
, and
Weerappuli
,
P.
,
2009
, “
Bayesian Probabilistic PCA Approach for Model Validation of Dynamic Systems
,”
SAE Int. J. Mater. Manuf.
,
2
(
1
), pp.
555
563
.
46.
Lamb
,
D.
,
Castanier
,
M.
,
Pan
,
H.
,
Kokkolaras
,
M.
, and
Hulbert
,
G.
,
2012
, “Model Validation for Simulations of Vehicle Systems,” Ford Motor Company, Dearborn, MI, DTIC Document No.
A566158
.http://www.dtic.mil/docs/citations/ADA566037
47.
Zhan
,
Z.
,
Fu
,
Y.
,
Yang
,
R.-J.
, and
Peng
,
Y.
,
2011
, “
An Enhanced Bayesian Based Model Validation Method for Dynamic Systems
,”
ASME J. Mech. Des.
,
133
(
4
), p.
041005
.
48.
Jiang
,
X.
, and
Mahadevan
,
S.
,
2011
, “
Wavelet Spectrum Analysis Approach to Model Validation of Dynamic Systems
,”
Mech. Syst. Signal Process.
,
25
(
2
), pp.
575
590
.
49.
Cheng
,
Z.
,
Pellettiere
,
J. A.
, and
Wright
,
N. L.
,
2006
, “
Wavelet-Based Test-Simulation Correlation Analysis for the Validation of Biodynamical Modeling
,”
Conference and Exposition on Structural Dynamics
, St Louis, MO, Jan. 30–Feb. 2, pp. 2124–2132.https://pdfs.semanticscholar.org/ef0c/bc3e2f9f09e0a87a6414ffe36a4a93216f60.pdf
50.
Morris
,
J. S.
,
2015
, “
Functional Regression
,”
Annu. Rev. Stat. Appl.
,
2
(
1
), pp.
321
359
.
51.
Faraway
,
J. J.
,
1997
, “
Regression Analysis for a Functional Response
,”
Technometrics
,
39
(
3
), pp.
254
261
.
52.
Faraway
,
J. J.
,
2000
, “Modeling Reach Motions Using Functional Regression Analysis,”
SAE
Paper No. 2000-01-2175.
53.
Untaroiu
,
C.
,
Shin
,
J.
, and
Lu
,
Y.-C.
,
2013
, “
Assessment of a Dummy Model in Crash Simulations Using Rating Methods
,”
Int. J. Autom. Technol.
,
14
(
3
), pp.
395
405
.
54.
Thunert
,
C.
,
2012
, “Cora Release 3.6 User's Manual,”
Partnership for Dummy Technology and Biomechanics
, Ingolstadt, Germany.
55.
Montgomery
,
D. C.
,
2009
,
Design and Analysis of Experiments
,
7th ed.
,
Wiley
,
Hoboken, NJ
.
56.
Myers
,
R. H.
,
Montgomery
,
D. C.
, and
Anderson-Cook
,
C. M.
,
2009
,
Response Surface Methodology: Process and Product Optimization Using Designed Experiments
,
3rd ed.
,
Wiley
,
Hoboken, NJ
.
57.
Gehre
,
C.
, and
Stahlschmidt
,
S.
,
2011
, “
Assessment of Dummy Models by Using Objective Rating Methods
,”
22nd International Technical Conference on the Enhanced Safety of Vehicles
, Washington, DC, June 13–16.
You do not currently have access to this content.