Abstract

In this paper, we introduce a physics-driven regularization method for training of deep neural networks (DNNs) for use in engineering design and analysis problems. In particular, we focus on the prediction of a physical system, for which in addition to training data, partial or complete information on a set of governing laws is also available. These laws often appear in the form of differential equations, derived from first principles, empirically validated laws, or domain expertise, and are usually neglected in a data-driven prediction of engineering systems. We propose a training approach that utilizes the known governing laws and regularizes data-driven DNN models by penalizing divergence from those laws. The first two numerical examples are synthetic examples, where we show that in constructing a DNN model that best fits the measurements from a physical system, the use of our proposed regularization results in DNNs that are more interpretable with smaller generalization errors, compared with other common regularization methods. The last two examples concern metamodeling for a random Burgers’ system and for aerodynamic analysis of passenger vehicles, where we demonstrate that the proposed regularization provides superior generalization accuracy compared with other common alternatives.

References

References
1.
Goodfellow
,
I.
,
Bengio
,
Y.
, and
Courville
,
A.
,
2016
,
Deep Learning
,
MIT Press
,
Cambridge, MA
.
2.
LeCun
,
Y.
,
Bengio
,
Y.
, and
Hinton
,
G.
,
2015
, “
Deep Learning
,”
Nature
,
521
(
7553
), pp.
436
444
. 10.1038/nature14539
3.
Srivastava
,
N.
,
Hinton
,
G.
,
Krizhevsky
,
A.
,
Sutskever
,
I.
, and
Salakhutdinov
,
R.
,
2014
, “
Dropout: a Simple Way to Prevent Neural Networks From Overfitting
,”
J. Mach. Learn. Res.
,
15
(
1
), pp.
1929
1958
.
4.
Guo
,
X.
,
Li
,
W.
, and
Iorio
,
F.
,
2016
, “
Convolutional Neural Networks for Steady Flow Approximation
,”
Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
,
San Francisco, CA
,
Aug. 13–17
,
ACM
, pp.
481
490
.
5.
Hennigh
,
O.
,
2017
, “
Lat-Net: Compressing Lattice Boltzmann Flow Simulations Using Deep Neural Networks
,” arXiv:1705.09036.
6.
Hennigh
,
O.
,
2017
, “
Automated Design Using Neural Networks and Gradient Descent
,” arXiv:1710.10352.
7.
Queipo
,
N. V.
,
Haftka
,
R. T.
,
Shyy
,
W.
,
Goel
,
T.
,
Vaidyanathan
,
R.
, and
Tucker
,
P. K.
,
2005
, “
Surrogate-Based Analysis and Optimization
,”
Prog. Aerosp. Sci.
,
41
(
1
), pp.
1
28
. 10.1016/j.paerosci.2005.02.001
8.
Shanock
,
L. R.
,
Baran
,
B. E.
,
Gentry
,
W. A.
,
Pattison
,
S. C.
, and
Heggestad
,
E. D.
,
2010
, “
Polynomial Regression With Response Surface Analysis: A Powerful Approach for Examining Moderation and Overcoming Limitations of Difference Scores
,”
J. Bus. Psychol.
,
25
(
4
), pp.
543
554
. 10.1007/s10869-010-9183-4
9.
Buhmann
,
M. D.
,
2000
, “
Radial Basis Functions
,”
Acta Numer.
,
9
, pp.
1
38
. 10.1017/S0962492900000015
10.
Wild
,
S. M.
,
Regis
,
R. G.
, and
Shoemaker
,
C. A.
,
2008
, “
Orbit: Optimization by Radial Basis Function Interpolation in Trust-regions
,”
SIAM J. Sci. Comput.
,
30
(
6
), pp.
3197
3219
. 10.1137/070691814
11.
Xiu
,
D.
, and
Karniadakis
,
G. E.
,
2002
, “
The Wiener–askey Polynomial Chaos for Stochastic Differential Equations
,”
SIAM J. Sci. Comput.
,
24
(
2
), pp.
619
644
. 10.1137/S1064827501387826
12.
Marzouk
,
Y. M.
, and
Najm
,
H. N.
,
2009
, “
Dimensionality Reduction and Polynomial Chaos Acceleration of Bayesian Inference in Inverse Problems
,”
J. Comput. Phys.
,
228
(
6
), pp.
1862
1902
. 10.1016/j.jcp.2008.11.024
13.
Simpson
,
T. W.
,
Mauery
,
T. M.
,
Korte
,
J. J.
, and
Mistree
,
F.
,
2001
, “
Kriging Models for Global Approximation in Simulation-based Multidisciplinary Design Optimization
,”
AIAA J.
,
39
(
12
), pp.
2233
2241
. 10.2514/2.1234
14.
Jeong
,
S.
,
Murayama
,
M.
, and
Yamamoto
,
K.
,
2005
, “
Efficient Optimization Design Method Using Kriging Model
,”
J. Aircr.
,
42
(
2
), pp.
413
420
. 10.2514/1.6386
15.
Bouhlel
,
M. A.
, and
Martins
,
J. R.
,
2019
, “
Gradient-Enhanced Kriging for High-Dimensional Problems
,”
Eng. Comput.
,
35
(
1
), pp.
157
173
. 10.1007/s00366-018-0590-x
16.
De Baar
,
J.
,
Scholcz
,
T. P.
,
Verhoosel
,
C. V.
,
Dwight
,
R. P.
,
van Zuijlen
,
A. H.
, and
Bijl
,
H.
,
2012
, “
Efficient Uncertainty Quantification With Gradient-Enhanced Kriging: Applications in FSI
,”
ECCOMAS
,
Vienna
.
17.
Smola
,
A. J.
, and
Schölkopf
,
B.
,
2004
, “
A Tutorial on Support Vector Regression
,”
Stat. Comput.
,
14
(
3
), pp.
199
222
. 10.1023/B:STCO.0000035301.49549.88
18.
Clarke
,
S. M.
,
Griebsch
,
J. H.
, and
Simpson
,
T. W.
,
2005
, “
Analysis of Support Vector Regression for Approximation of Complex Engineering Analyses
,”
ASME J. Mech. Des.
,
127
(
6
), pp.
1077
1087
. 10.1115/1.1897403
19.
Rudy
,
S. H.
,
Brunton
,
S. L.
,
Proctor
,
J. L.
, and
Kutz
,
J. N.
,
2017
, “
Data-Driven Discovery of Partial Differential Equations
,”
Sci. Adv.
,
3
(
4
), p.
e1602614
. 10.1126/sciadv.1602614
20.
Raissi
,
M.
, and
Karniadakis
,
G. E.
,
2018
, “
Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations
,”
J. Comput. Phys.
,
357
, pp.
125
141
. 10.1016/j.jcp.2017.11.039
21.
Wu
,
K.
, and
Xiu
,
D.
,
2019
, “
Numerical Aspects for Approximating Governing Equations Using Data
,”
J. Comput. Phys.
,
384
, pp.
200
221
. 10.1016/j.jcp.2019.01.030
22.
Voss
,
H. U.
,
Kolodner
,
P.
,
Abel
,
M.
, and
Kurths
,
J.
,
1999
, “
Amplitude Equations From Spatiotemporal Binary-Fluid Convection Data
,”
Phys. Rev. Lett.
,
83
(
17
), p.
3422
. 10.1103/PhysRevLett.83.3422
23.
Shan
,
T.
,
Dang
,
X.
,
Li
,
M.
,
Yang
,
F.
,
Xu
,
S.
, and
Wu
,
J.
,
2018
, “
Study on a 3D Possion’s Equation Slover Based on Deep Learning Technique
,”
2018 IEEE International Conference on Computational Electromagnetics (ICCEM)
,
New York
,
Mar. 26–28
,
IEEE
,
New York
, pp.
1
3
.
24.
Nabian
,
M. A.
, and
Meidani
,
H.
,
2018
, “
Deep Learning for Accelerated Seismic Reliability Analysis of Transportation Networks
,”
Comput.-Aided Civ. Infrastruct. Eng.
,
33
(
6
), pp.
443
458
. 10.1111/mice.2018.33.issue-6
25.
Qin
,
T.
,
Wu
,
K.
, and
Xiu
,
D.
,
2018
, “
Data Driven Governing Equations Approximation using Deep Neural Networks
,” arXiv:1811.05537.
26.
Lagaris
,
I. E.
,
Likas
,
A.
, and
Fotiadis
,
D. I.
,
1998
, “
Artificial Neural Networks for Solving Ordinary and Partial Differential Equations
,”
IEEE Trans. Neural Networks
,
9
(
5
), pp.
987
1000
. 10.1109/72.712178
27.
Lee
,
H.
, and
Kang
,
I. S.
,
1990
, “
Neural Algorithm for Solving Differential Equations
,”
J. Comput. Phys.
,
91
(
1
), pp.
110
131
. 10.1016/0021-9991(90)90007-N
28.
Wang
,
L.
, and
Mendel
,
J.
,
1990
, “
Structured Trainable Networks for Matrix Algebra
,”
1990 IJCNN International Joint Conference on Neural Networks
,
San Diego, CA
,
June 17–21
,
IEEE
,
New York
, pp.
125
132
.
29.
Mills
,
K.
,
Spanner
,
M.
, and
Tamblyn
,
I.
,
2017
, “
Deep Learning and the Schrödinger Equation
,”
Phys. Rev. A
,
96
(
4
), p.
042113
. 10.1103/PhysRevA.96.042113
30.
Ehrhardt
,
S.
,
Monszpart
,
A.
,
Mitra
,
N. J.
, and
Vedaldi
,
A.
,
2017
, “
Learning A Physical Long-Term Predictor
,” arXiv:1703.00247.
31.
Zhang
,
Z.
,
Zhang
,
L.
,
Sun
,
Z.
,
Erickson
,
N.
,
From
,
R.
, and
Fan
,
J.
,
2018
, “
Solving Poisson’s Equation Using Deep Learning in Particle Simulation of PN Junction
,” arXiv:1810.10192.
32.
Raissi
,
M.
,
Perdikaris
,
P.
, and
Karniadakis
,
G.
,
2019
, “
Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations
,”
J. Comput. Phys.
,
378
, pp.
686
707
. 10.1016/j.jcp.2018.10.045
33.
Gonzalez-Garcia
,
R.
,
Rico-Martinez
,
R.
, and
Kevrekidis
,
I.
,
1998
, “
Identification of Distributed Parameter Systems: A Neural Net Based Approach
,”
Comput. Chem. Eng.
,
22
, pp.
S965
S968
. 10.1016/S0098-1354(98)00191-4
34.
Sirignano
,
J.
, and
Spiliopoulos
,
K.
,
2018
, “
DGM: A Deep Learning Algorithm for Solving Partial Differential Equations
,”
J. Comput. Phys.
,
375
, pp.
1339
1364
. 10.1016/j.jcp.2018.08.029
35.
Nabian
,
M. A.
, and
Meidani
,
H.
,
2018
, “
A Deep Neural Network Surrogate for High-Dimensional Random Partial Differential Equations
,” arXiv:1806.02957.
36.
Rudd
,
K.
,
2013
, “
Solving Partial Differential Equations using Artificial Neural Networks
,” Ph.D. thesis,
Duke University
,
Durham, NC
.
37.
Weinan
,
E.
,
Han
,
J.
, and
Jentzen
,
A.
,
2017
, “
Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations
,”
Commun. Math. Stat.
,
5
(
4
), pp.
349
380
. 10.1007/s40304-017-0117-6
38.
Han
,
J.
,
Jentzen
,
A.
, and
Weinan
,
E.
,
2018
, “
Solving High-Dimensional Partial Differential Equations Using Deep Learning
,”
Proc. Natl. Acad. Sci. U. S. A.
,
115
(
34
), pp.
8505
8510
. 10.1073/pnas.1718942115
39.
Berg
,
J.
, and
Nyström
,
K.
,
2018
, “
A Unified Deep Artificial Neural Network Approach to Partial Differential Equations in Complex Geometries
,”
Neurocomputing
,
317
, pp.
28
41
. 10.1016/j.neucom.2018.06.056
40.
Raissi
,
M.
,
Wang
,
Z.
,
Triantafyllou
,
M. S.
, and
Karniadakis
,
G. E.
,
2019
, “
Deep Learning of Vortex-Induced Vibrations
,”
J. Fluid Mech.
,
861
, pp.
119
137
. 10.1017/jfm.2018.872
41.
Raissi
,
M.
,
Yazdani
,
A.
, and
Karniadakis
,
G. E.
,
2018
, “
Hidden Fluid Mechanics: A Navier–Stokes Informed Deep Learning Framework for Assimilating Flow Visualization Data
,” arXiv:1808.04327.
42.
Xiu
,
D.
, and
Hesthaven
,
J. S.
,
2005
, “
High-order Collocation Methods for Differential Equations With Random Inputs
,”
SIAM J. Sci. Comput.
,
27
(
3
), pp.
1118
1139
. 10.1137/040615201
43.
Ghanem
,
R. G.
, and
Spanos
,
P. D.
,
2003
,
Stochastic Finite Elements: A Spectral Approach
,
Courier Corporation
,
Chelmsford, MA
.
44.
Fishman
,
G.
,
2013
,
Monte Carlo: Concepts, Algorithms, and Applications
,
Springer Science & Business Media
,
New York City, NY
.
45.
Bottou
,
L.
,
2012
, “Stochastic Gradient Descent Tricks”,
Neural Networks: Tricks of the Trade
, Lecture Notes in Computer Science, vol.
7700
,
G.
Montavon
,
G. B.
Orr
and
K. R.
Müller
, eds.,
Springer
,
Berlin, Heidelberg
, pp.
421
436
.
46.
Kingma
,
D. P.
, and
Ba
,
J.
,
2014
, “
Adam: A Method for Stochastic Optimization
,” arXiv:1412.6980.
47.
Duchi
,
J.
,
Hazan
,
E.
, and
Singer
,
Y.
,
2011
, “
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
,”
J. Mach. Learn. Res.
,
12
(
Jul
.), pp.
2121
2159
.
48.
Zeiler
,
M. D.
,
2012
, “
Adadelta: An Adaptive Learning Rate Method
,” arXiv:1212.5701.
49.
Sutskever
,
I.
,
Martens
,
J.
,
Dahl
,
G.
, and
Hinton
,
G.
,
2013
, “
On the Importance of Initialization and Momentum in Deep Learning
,”
International Conference on Machine Learning
,
Atlanta, GA
,
June 16–21
, pp.
1139
1147
.
50.
Wan
,
L.
,
Zeiler
,
M.
,
Zhang
,
S.
,
Le Cun
,
Y.
, and
Fergus
,
R.
,
2013
, “
Regularization of Neural Networks Using Dropconnect
,”
International Conference on Machine Learning
,
Atlanta, GA
,
June 16–21
, pp.
1058
1066
.
51.
Caruana
,
R.
,
Lawrence
,
S.
, and
Giles
,
C. L.
,
2001
, “
Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping
,”
Advances in Neural Information Processing Systems
, pp.
402
408
.
52.
Salamon
,
J.
, and
Bello
,
J. P.
,
2017
, “
Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification
,”
IEEE Signal Process Lett.
,
24
(
3
), pp.
279
283
. 10.1109/LSP.2017.2657381
53.
Hinton
,
G. E.
,
Srivastava
,
N.
,
Krizhevsky
,
A.
,
Sutskever
,
I.
, and
Salakhutdinov
,
R. R.
,
2012
, “
Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors
,” arXiv:1207.0580.
54.
Dahl
,
G. E.
,
Sainath
,
T. N.
, and
Hinton
,
G. E.
,
2013
, “
Improving Deep Neural Networks for LVCSR using Rectified Linear Units and Dropout
,”
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
,
Vancouver, BC
,
May 26–31
,
IEEE
,
New York
, pp.
8609
8613
.
55.
Krizhevsky
,
A.
,
Sutskever
,
I.
, and
Hinton
,
G. E.
,
2012
, “
Imagenet Classification With Deep Convolutional Neural Networks
,”
Advances in Neural Information Processing Systems
, pp.
1097
1105
.
56.
Pham
,
V.
,
Bluche
,
T.
,
Kermorvant
,
C.
, and
Louradour
,
J.
,
2014
, “
Dropout Improves Recurrent Neural Networks for Handwriting Recognition
,”
14th International Conference on Frontiers in Handwriting Recognition (ICFHR)
,
Heraklion, Greece
,
Sept. 1–4
,
IEEE
,
New York
, pp.
285
290
.
57.
Wager
,
S.
,
Wang
,
S.
, and
Liang
,
P. S.
,
2013
, “
Dropout Training as Adaptive Regularization
,”
Advances in Neural Information Processing Systems
, pp.
351
359
.
58.
Baydin
,
A. G.
,
Pearlmutter
,
B. A.
,
Radul
,
A. A.
, and
Siskind
,
J. M.
,
2018
, “
Automatic Differentiation in Machine Learning: A Survey
,”
J. Mach. Learn. Res.
,
18
, pp.
1
43
.
59.
Griewank
,
A.
, and
Walther
,
A.
,
2008
,
Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation
, Vol.
105
,
SIAM
,
Philadelphia
.
60.
Raissi
,
M.
,
2018
, “
Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations
,” arXiv:1801.06637.
61.
Battles
,
Z.
, and
Trefethen
,
L. N.
,
2004
, “
An Extension of MATLAB to Continuous Functions and Operators
,”
SIAM J. Sci. Comp.
,
25
(
5
), pp.
1743
1770
.
62.
Medjo
,
T. T.
,
1995
, “
Vorticity-velocity Formulation for the Stationary Navier–Stokes Equations: The Three-Dimensional Case
,”
Appl. Math. Lett.
,
8
(
4
), pp.
63
66
. 10.1016/0893-9659(95)00048-U
63.
Taira
,
K.
, and
Colonius
,
T.
,
2007
, “
The Immersed Boundary Method: A Projection Approach
,”
J. Comput. Phys.
,
225
(
2
), pp.
2118
2137
. 10.1016/j.jcp.2007.03.005
64.
Colonius
,
T.
, and
Taira
,
K.
,
2008
, “
A Fast Immersed Boundary Method Using a Nullspace Approach and Multi-Domain Far-Field Boundary Conditions
,”
Comput. Methods Appl. Mech. Eng.
,
197
(
25-28
), pp.
2131
2146
. 10.1016/j.cma.2007.08.014
65.
Kutz
,
J. N.
,
Brunton
,
S. L.
,
Brunton
,
B. W.
, and
Proctor
,
J. L.
,
2016
,
Dynamic Mode Decomposition: Data-Driven Modeling of Complex Systems
,
149
,
SIAM
,
Philadelphia
.
66.
Tu
,
J.
,
Yeoh
,
G. H.
, and
Liu
,
C.
,
2018
,
Computational Fluid Dynamics: A Practical Approach
,
Butterworth-Heinemann
,
Oxford, UK
.
67.
Nabian
,
M. A.
, and
Farhadi
,
L.
,
2016
, “
Multiphase Mesh-free Particle Method for Simulating Granular Flows and Sediment Transport
,”
J. Hydraul. Eng.
,
143
(
4
), p.
04016102
. 10.1061/(ASCE)HY.1943-7900.0001275
68.
Hennigh
,
O.
,
2018
, “
Steady-State-Flow-with-Neural-Nets
,” https://github.com/loliverhennigh/Steady-State-Flow-With-Neural-Nets.git, Accessed October 2, 2018.
69.
Chen
,
S.
, and
Doolen
,
G. D.
,
1998
, “
Lattice Boltzmann Method for Fluid Flows
,”
Annu. Rev. Fluid Mech.
,
30
(
1
), pp.
329
364
. 10.1146/annurev.fluid.30.1.329
70.
He
,
K.
,
Zhang
,
X.
,
Ren
,
S.
, and
Sun
,
J.
,
2016
, “
Deep Residual Learning for Image Recognition
,”
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
,
Las Vegas, NV
,
June 27–30
, pp.
770
778
.
71.
van den Oord
,
A.
,
Kalchbrenner
,
N.
,
Espeholt
,
L.
,
Vinyals
,
O.
, and
Graves
,
A.
,
2016
, “
Conditional Image Generation with PixelCNN Decoders
,”
Advances in Neural Information Processing Systems
, pp.
4790
4798
.
72.
Salimans
,
T.
,
Karpathy
,
A.
,
Chen
,
X.
, and
Kingma
,
D. P.
,
2017
, “
PixelCNN++: A PixelCNN Implementation With Discretized Logistic Mixture Likelihood and Other Modifications
,”
International Conference on Learning Representations (ICLR)
,
Toulon, France
,
Apr. 24 –26
.
73.
Goodfellow
,
I.
,
2016
, “
NIPS 2016 Tutorial: Generative Adversarial Networks
,” arXiv:1701.00160.
74.
Gouk
,
H.
,
Frank
,
E.
,
Pfahringer
,
B.
, and
Cree
,
M.
,
2018
, “
Regularisation of Neural Networks by Enforcing Lipschitz Continuity
,” arXiv:1804.04368.
75.
Gulrajani
,
I.
,
Ahmed
,
F.
,
Arjovsky
,
M.
,
Dumoulin
,
V.
, and
Courville
,
A. C.
,
2017
, “
Improved Training of Wasserstein Gans
,”
Advances in Neural Information Processing Systems
, pp.
5767
5777
.
76.
Miyato
,
T.
,
Kataoka
,
T.
,
Koyama
,
M.
, and
Yoshida
,
Y.
,
2018
, “
Spectral Normalization for Generative Adversarial Networks
,” arXiv:1802.05957.
77.
Gu
,
S.
, and
Rigazio
,
L.
,
2014
, “
Towards Deep Neural Network Architectures Robust to Adversarial Examples
,” arXiv:1412.5068.
78.
Goodfellow
,
I. J.
,
Shlens
,
J.
, and
Szegedy
,
C.
,
2014
, “
Explaining and Harnessing Adversarial Examples
,” arXiv:1412.6572.
You do not currently have access to this content.