The control of shared energy assets within building clusters has traditionally been confined to a discrete action space, owing in part to a computationally intractable decision space. In this work, we leverage the current state of the art in reinforcement learning (RL) for continuous control tasks, the deep deterministic policy gradient (DDPG) algorithm, toward addressing this limitation. The goals of this paper are twofold: (i) to design an efficient charged/discharged dispatch policy for a shared battery system within a building cluster and (ii) to address the continuous domain task of determining how much energy should be charged/discharged at each decision cycle. Experimentally, our results demonstrate an ability to exploit factors such as energy arbitrage, along with the continuous action space toward demand peak minimization. This approach is shown to be computationally tractable, achieving efficient results after only 5 h of simulation. Additionally, the agent showed an ability to adapt to different building clusters, designing unique control strategies to address the energy demands of the clusters studied.

References

References
1.
Amin
,
M.
,
2013
, “
Energy: The Smart-Grid Solution
,”
Nature
,
499
(
7457
), p.
145
.
2.
Odonkor
,
P.
,
Lewis
,
K.
,
Wen
,
J.
, and
Wu
,
T.
,
2016
, “
Adaptive Energy Optimization Toward Net-Zero Energy Building Clusters
,”
ASME J. Mech. Des.
,
138
(
6
), p. 061405.
3.
Korpaas
,
M.
,
Holen
,
A. T.
, and
Hildrum
,
R.
,
2003
, “
Operation and Sizing of Energy Storage for Wind Power Plants in a Market System
,”
Int. J. Electr. Power Energy Syst.
,
25
(
8
), pp.
599
606
.
4.
Perez
,
R.
,
David
,
M.
,
Hoff
,
T. E.
,
Jamaly
,
M.
,
Kivalov
,
S.
,
Kleissl
,
J.
,
Lauret
,
P.
, and
Perez
,
M.
,
2016
, “
Spatial and Temporal Variability of Solar Energy
,”
Found. Trends Renewable Energy
,
1
(
1
), pp.
1
44
.
5.
Sommerfeld
,
J.
,
Buys
,
L.
, and
Vine
,
D.
,
2017
, “
Residential Consumers Experiences in the Adoption and Use of Solar PV
,”
Energy Policy
,
105
, pp.
10
16
.
6.
Crabtree
,
G.
,
2015
, “
Perspective: The Energy-Storage Revolution
,”
Nature
,
526
(
7575
), pp.
S92
S92
.
7.
Lund
,
P. D.
,
Lindgren
,
J.
,
Mikkola
,
J.
, and
Salpakari
,
J.
,
2015
, “
Review of Energy System Flexibility Measures to Enable High Levels of Variable Renewable Electricity
,”
Renewable Sustainable Energy Rev.
,
45
, pp.
785
807
.
8.
Ted
,
C.
,
2016
, “
Tesla Powerwall: Not Just for Solar
,” Hanley Wood Media, Inc., Washington, DC, accessed Oct. 12, 2018, https://goo.gl/QudGAR
9.
Le
,
H. T.
, and
Nguyen
,
T. Q.
,
2008
, “
Sizing Energy Storage Systems for Wind Power Firming: An Analytical Approach and a Cost-Benefit Analysis
,”
IEEE
Power and Energy Society General Meeting-Conversion and Delivery of Electrical Energy in the 21st Century,
Pittsburgh, PA, July 20–24, pp.
1
8
.
10.
Business Wire,
2014
, “
Sonnenbatterie Enters us Market With First Distribution Deal
,” Business Wire, San Francisco, CA, accessed Oct. 12, 2018, https://goo.gl/hFd7ix
11.
Deign
,
J.
,
2015
, “
Sungevity and Sonnenbatterie to Undercut Tesla and SolarCity on Storage Pricing?
,” Greentech Media, Boston, MA, accessed Oct. 12, 2018, https://goo.gl/Yr9kpj
12.
GMP
,
2017
, “
Green Mountain Power
,” GMP, Colchester, VT, accessed Oct. 12, 2018, https://goo.gl/wVhhzq
13.
Alharbi
,
H.
, and
Bhattacharya
,
K.
,
2018
, “
Stochastic Optimal Planning of Battery Energy Storage Systems for Isolated Microgrids
,”
IEEE Trans. Sustainable Energy
,
9
(
1
), pp.
211
227
.
14.
Chakraborty
,
S.
,
Senjyu
,
T.
,
Toyama
,
H.
,
Saber
,
A.
, and
Funabashi
,
T.
,
2009
, “
Determination Methodology for Optimising the Energy Storage Size for Power System
,”
IET Gener., Transm. Distrib.
,
3
(
11
), pp.
987
999
.
15.
Bhandari
,
R.
, and
Stadler
,
I.
,
2009
, “
Grid Parity Analysis of Solar Photovoltaic Systems in Germany Using Experience Curves
,”
Sol. Energy
,
83
(
9
), pp.
1634
1644
.
16.
Kou
,
P.
,
Gao
,
F.
, and
Guan
,
X.
,
2015
, “
Stochastic Predictive Control of Battery Energy Storage for Wind Farm Dispatching: Using Probabilistic Wind Power Forecasts
,”
Renewable Energy
,
80
, pp.
286
300
.
17.
Divya
,
K.
, and
Østergaard
,
J.
,
2009
, “
Battery Energy Storage Technology for Power systems - An Overview
,”
Electric Power Syst. Res.
,
79
(
4
), pp.
511
520
.
18.
Hadjipaschalis
,
I.
,
Poullikkas
,
A.
, and
Efthimiou
,
V.
,
2009
, “
Overview of Current and Future Energy Storage Technologies for Electric Power Applications
,”
Renewable Sustainable Energy Rev.
,
13
(
6–7
), pp.
1513
1522
.
19.
Wang
,
X.
,
Vilathgamuwa
,
D. M.
, and
Choi
,
S. S.
,
2008
, “
Determination of Battery Storage Capacity in Energy Buffer for Wind Farm
,”
IEEE Trans. Energy Convers.
,
23
(
3
), pp.
868
878
.
20.
Venu
,
C.
,
Riffonneau
,
Y.
,
Bacha
,
S.
, and
Baghzouz
,
Y.
,
2009
, “
Battery Storage System Sizing in Distribution Feeders With Distributed Photovoltaic Systems
,”
IEEE
Bucharest PowerTech
, Bucharest, Romania, June 28–July 2, pp.
1
5
.
21.
Castillo-Cagigal
,
M.
,
Caamano-Martín
,
E.
,
Matallanas
,
E.
,
Masa-Bote
,
D.
,
Gutiérrez
,
A.
,
Monasterio-Huelin
,
F.
, and
Jiménez-Leube
,
J.
,
2011
, “
PV Self-Consumption Optimization With Storage and Active Dsm for the Residential Sector
,”
Sol. Energy
,
85
(
9
), pp.
2338
2348
.
22.
Guille
,
C.
, and
Gross
,
G.
,
2009
, “
A Conceptual Framework for the Vehicle-to-Grid (v2g) Implementation
,”
Energy Policy
,
37
(
11
), pp.
4379
4390
.
23.
Wu
,
X.
,
Hu
,
X.
,
Moura
,
S.
,
Yin
,
X.
, and
Pickert
,
V.
,
2016
, “
Stochastic Control of Smart Home Energy Management With Plug-in Electric Vehicle Battery Energy Storage and Photovoltaic Array
,”
J. Power Sources
,
333
, pp.
203
212
.
24.
Haines
,
G.
,
McGordon
,
A.
,
Jennings
,
P.
, and
Butcher
,
N.
,
2009
, “
The Simulation of Vehicle-to-Home Systems—Using Electric Vehicle Battery Storage to Smooth Domestic Electricity Demand
,”
International Conference on Ecologic Vehicles and Renewable Energies
, Monaco, Mar. 26–29.https://www.researchgate.net/publication/228696724_The_Simulation_of_Vehicle-to-Home_Systems-Using_Electric_Vehicle_Battery_Storage_to_Smooth_Domestic_Electricity_Demand
25.
Jian
,
L.
,
Xue
,
H.
,
Xu
,
G.
,
Zhu
,
X.
,
Zhao
,
D.
, and
Shao
,
Z.
,
2013
, “
Regulated Charging of Plug-in Hybrid Electric Vehicles for Minimizing Load Variance in Household Smart Microgrid
,”
IEEE Trans. Ind. Electron.
,
60
(
8
), pp.
3218
3226
.
26.
Fares
,
R. L.
, and
Webber
,
M. E.
,
2017
, “
The Impacts of Storing Solar Energy in the Home to Reduce Reliance on the Utility
,”
Nat. Energy
,
2
(
2
), p.
17001
.
27.
Itron Consulting
,
2016
, “
2016 SGIP Advanced Energy Storage Impact Evaluation
,” Itron Consulting, Davis, CA, accessed on Oct 9, 2018, http://docs.cpuc.ca.gov/PublishedDocs/Published/G000/M194/K599/194599448.PDF
28.
Fridgen
,
G.
,
Gründler
,
A.
, and
Rusic
,
M.
,
2015
, “
Energy Cooperatives as an Application of Microgrids: Multi-Criteria Investment Decision Support
,”
36th International Conference on Information Systems
(
ICIS
), Fort Worth, TX, Dec. 13–16, pp. 1–20.https://aisel.aisnet.org/cgi/viewcontent.cgi?article=1653&context=icis2015&usg=AFQjCNHTIeza7NOWDj1vLJpsn5szRfm9PQ
29.
Hu
,
M.
,
Weir
,
J. D.
, and
Wu
,
T.
,
2012
, “
Decentralized Operation Strategies for an Integrated Building Energy System Using a Memetic Algorithm
,”
Eur. J. Oper. Res.
,
217
(
1
), pp.
185
197
.
30.
Odonkor
,
P.
, and
Lewis
,
K.
,
2016
, “
Optimization of Energy Use Strategies in Building Clusters Using Pareto Bands
,”
ASME
Paper No. IDETC2016-59963.
31.
Ren
,
Y.
,
Bayrak
,
A. E.
, and
Papalambros
,
P. Y.
,
2016
, “
Ecoracer: Game-Based Optimal Electric Vehicle Design and Driver Control Using Human Players
,”
ASME J. Mech. Des.
,
138
(
6
), p.
061407
.
32.
Sexton
,
T.
, and
Ren
,
M. Y.
,
2017
, “
Learning an Optimization Algorithm Through Human Design Iterations
,”
ASME J. Mech. Des.
,
139
(
10
), p.
101404
.
33.
Sutton
,
R. S.
, and
Barto
,
A. G.
,
1998
,
Reinforcement Learning: An Introduction
, Vol.
1
,
MIT Press
,
Cambridge, UK
.
34.
Duch
,
W.
, and
Mandziuk
,
J.
, eds.,
2007
,
Challenges for Computational Intelligence
, Vol.
63
,
Springer Science & Business Media
, Berlin.
35.
Silver
,
D.
,
Schrittwieser
,
J.
,
Simonyan
,
K.
,
Antonoglou
,
I.
,
Huang
,
A.
,
Guez
,
A.
,
Hubert
,
T.
,
Baker
,
L.
,
Lai
,
M.
,
Bolton
,
A.
,
Chen
,
Y.
,
Lillicrap
,
T.
,
Hui
,
F.
,
Sifre
,
L.
,
Van den Driessche
,
G.
,
Graepel
,
T.
, and
Hassabis
,
D.
,
2017
, “
Mastering the Game of Go Without Human Knowledge
,”
Nature
,
550
(
7676
), p.
354
.
36.
Barth-Maron
,
G.
,
Hoffman
,
M. W.
,
Budden
,
D.
,
Dabney
,
W.
,
Horgan
,
D.
,
TB
,
D.
,
Muldal
,
A.
,
Heess
,
N.
, and
Lillicrap
,
T. P.
,
2018
, “
Distributed Distributional Deterministic Policy Gradients
,” CoRR, e-print
arXiv:1804.08617
.https://arxiv.org/abs/1804.08617
37.
Pardo
,
D.
,
Möller
,
L.
,
Neunert
,
M.
,
Winkler
,
A. W.
, and
Buchli
,
J.
,
2016
, “
Evaluating Direct Transcription and Nonlinear Optimization Methods for Robot Motion Planning
,”
IEEE Rob. Autom. Lett.
,
1
(
2
), pp.
946
953
.
38.
Böhme
,
T. J.
, and
Frank
,
B.
,
2017
, “
Direct Methods for Optimal Control
,”
Hybrid Systems, Optimal Control and Hybrid Vehicles
,
Springer
, Berlin, pp.
233
273
.
39.
Lillicrap
,
T. P.
,
Hunt
,
J. J.
,
Pritzel
,
A.
,
Heess
,
N.
,
Erez
,
T.
,
Tassa
,
Y.
,
Silver
,
D.
, and
Wierstra
,
D.
,
2015
, “
Continuous Control With Deep Reinforcement Learning
,” e-print
arXiv:1509.02971
.https://arxiv.org/abs/1509.02971
40.
Mnih
,
V.
,
Kavukcuoglu
,
K.
,
Silver
,
D.
,
Rusu
,
A.
,
Veness
,
J.
,
Bellemare
,
M. G.
,
Graves
,
A.
,
Riedmiller
,
M.
,
Fidjeland
,
A.
,
Ostrovski
,
G.
,
Petersen
,
S.
,
Beattie
,
C.
,
Sadik
,
A.
,
Antonoglou
,
I.
,
King
,
H.
,
Kumaran
,
D.
,
Wierstra
,
D.
,
Legg
,
S.
, and
Hassabis
,
D.
,
2015
, “
Human-Level Control Through Deep Reinforcement Learning
,”
Nature
,
518
(
7540
), p.
529
.
41.
Silver
,
D.
,
Lever
,
G.
,
Heess
,
N.
,
Degris
,
T.
,
Wierstra
,
D.
, and
Riedmiller
,
M.
,
2014
, “
Deterministic Policy Gradient Algorithms
,”
International Conference on Machine Learning
, Beijing, China, June 21–26, pp.
387
395
.http://proceedings.mlr.press/v32/silver14.pdf
42.
Uhlenbeck
,
G. E.
, and
Ornstein
,
L. S.
,
1930
, “
On the Theory of the Brownian Motion
,”
Phys. Rev.
,
36
(
5
), p.
823
.
43.
Keskar
,
N. S.
,
Mudigere
,
D.
,
Nocedal
,
J.
,
Smelyanskiy
,
M.
, and
Tang
,
P. T. P.
,
2016
, “
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
,” CoRR, e-print
arXiv:1609.04836
https://arxiv.org/abs/1609.04836.
44.
City of Austin
,
2017
, “
Residential Electric Rate Schedules
,” City of Austin, Austin, TX, accessed Oct. 12, 2018, https://goo.gl/Atd66W
45.
Andrychowicz
,
M.
,
Wolski
,
F.
,
Ray
,
A.
,
Schneider
,
J.
,
Fong
,
R.
,
Welinder
,
P.
,
McGrew
,
B.
,
Tobin
,
J.
,
Abbeel
,
P.
, and
Zaremba
,
W.
,
2017
, “
Hindsight Experience Replay
,” CoRR, e-print
arXiv:1707.01495
https://arxiv.org/abs/1707.01495.
46.
Tobin
,
J.
,
Fong
,
R.
,
Ray
,
A.
,
Schneider
,
J.
,
Zaremba
,
W.
, and
Abbeel
,
P.
,
2017
, “
Domain Randomization for Transferring Deep Neural Networks From Simulation to the Real World
,”
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),
Vancouver, BC, Canada, Sept. 24–28, pp.
23
30
.
You do not currently have access to this content.