Abstract

The flexibility afforded by distributed energy resources in terms of energy generation and storage has the potential to disrupt the way we currently access and manage electricity. But as the energy grid moves to fully embrace this technology, grid designers and operators are having to come to terms with managing its adverse effects, exhibited through electricity price volatility, caused in part by the intermittency of renewable energy. With this concern however comes interest in exploiting this price volatility using arbitrage—the buying and selling of electricity to profit from a price imbalance—for energy cost savings for consumers. To this end, this paper aims to maximize arbitrage value through the data-driven design of optimal operational strategies for distributed energy resources (DERs). Formulated as an arbitrage maximization problem using design optimization principles and solved using reinforcement learning, the proposed approach is applied toward shared DERs within multi-building residential clusters. We demonstrate its feasibility across three unique building clusters, observing notable energy cost reductions over baseline values. This highlights a capability for generalized learning across multiple building clusters and the ability to design efficient arbitrage policies for energy cost minimization. The scalability of this approach is studied using two test cases, with results demonstrating an ability to scale with relatively minimal additional computational cost, and an ability to leverage system flexibility toward cost savings.

References

References
1.
Tesla
,
2019
, “
Tesla Solar Roof
,” https://goo.gl/Z58MPu. Accessed January 2019.
2.
Tesla
,
2019
, “
Tesla Powerwall
,” https://goo.gl/6XE63R. Accessed January 2019.
3.
Wang
,
H.
, and
Zhang
,
B.
,
2018
, “
Energy Storage Arbitrage in Real-Time Markets Via Reinforcement Learning
,”
IEEE Power Energy Society General Meeting (PESGM), IEEE
,
Portland, OR
,
Aug. 5–9
, pp.
1
5
. 10.1109/pesgm.2018.8586321
4.
Krishnamurthy
,
D.
,
Uckun
,
C.
,
Zhou
,
Z.
,
Thimmapuram
,
P. R.
, and
Botterud
,
A.
,
2017
, “
Energy Storage Arbitrage Under Day-Ahead and Real-Time Price Uncertainty
,”
IEEE Trans. Power Syst.
,
33
(
1
), pp.
84
93
. http://dx.doi.org/10.1109/TPWRS.2017.2685347
5.
Kloess
,
M.
,
2012
, “
Electric Storage Technologies for the Future Power System—An Economic Assessment
,”
9th International Conference on the European Energy Market (EEM), IEEE
,
Florence, Italy
,
May 10–12
, pp.
1
8
. 10.1109/eem.2012.6254729
6.
Steffen
,
B.
,
2012
, “
Prospects for Pumped-Hydro Storage in Germany
,”
Energy Policy
,
45
(
C
), pp.
420
429
. 10.1016/j.enpol.2012.02.052
7.
Walawalkar
,
R.
,
Apt
,
J.
, and
Mancini
,
R.
,
2007
, “
Economics of Electric Energy Storage for Energy Arbitrage and Regulation in New York
,”
Energy Policy
,
35
(
4
), pp.
2558
2568
. 10.1016/j.enpol.2006.09.005
8.
Sioshansi
,
R.
,
Denholm
,
P.
,
Jenkin
,
T.
, and
Weiss
,
J.
,
2009
, “
Estimating the Value of Electricity Storage in PJM: Arbitrage and Some Welfare Effects
,”
Energy Econ.
,
31
(
2
), pp.
269
277
. 10.1016/j.eneco.2008.10.005
9.
Graves
,
F.
,
Jenkin
,
T.
, and
Murphy
,
D.
,
1999
, “
Opportunities for Electricity Storage in Deregulating Markets
,”
Electr. J.
,
12
(
8
), pp.
46
56
. 10.1016/S1040-6190(99)00071-8
10.
Zheng
,
M.
,
Meinrenken
,
C. J.
, and
Lackner
,
K. S.
,
2014
, “
Agent-Based Model for Electricity Consumption and Storage to Evaluate Economic Viability of Tariff Arbitrage for Residential Sector Demand Response
,”
Appl. Energy
,
126
, pp.
297
306
. 10.1016/j.apenergy.2014.04.022
11.
Zheng
,
M.
,
Meinrenken
,
C. J.
, and
Lackner
,
K. S.
,
2015
, “
Smart Households: Dispatch Strategies and Economic Analysis of Distributed Energy Storage for Residential Peak Shaving
,”
Appl. Energy
,
147
, pp.
246
257
. 10.1016/j.apenergy.2015.02.039
12.
Abdulla
,
K.
,
De Hoog
,
J.
,
Muenzel
,
V.
,
Suits
,
F.
,
Steer
,
K.
,
Wirth
,
A.
, and
Halgamuge
,
S.
,
2018
, “
Optimal Operation of Energy Storage Systems Considering Forecasts and Battery Degradation
,”
IEEE Trans. Smart Grid
,
9
(
3
), pp.
2086
2096
. 10.1109/TSG.2016.2606490
13.
Alstone
,
P.
,
Gershenson
,
D.
, and
Kammen
,
D. M.
,
2015
, “
Decentralized Energy Systems for Clean Electricity Access
,”
Nat. Clim. Change
,
5
(
4
), p.
305
. 10.1038/nclimate2512
14.
Adil
,
A. M.
, and
Ko
,
Y.
,
2016
, “
Socio-Technical Evolution of Decentralized Energy Systems: A Critical Review and Implications for Urban Planning and Policy
,”
Renewable Sustainable Energy Rev.
,
57
, pp.
1025
1037
. 10.1016/j.rser.2015.12.079
15.
Hu
,
M.
,
Weir
,
J. D.
, and
Wu
,
T.
,
2012
, “
Decentralized Operation Strategies for An Integrated Building Energy System Using a Memetic Algorithm
,”
Eur. J. Oper. Res.
,
217
(
1
), pp.
185
197
. 10.1016/j.ejor.2011.09.008
16.
Odonkor
,
P.
, and
Lewis
,
K.
,
2016
, “
Optimization of Energy Use Strategies in Building Clusters Using Pareto Bands
,”
ASME 2016 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers
,
Charlotte, NC, IDETC2016-59963
.
17.
Odonkor
,
P.
,
Lewis
,
K.
,
Wen
,
J.
, and
Wu
,
T.
,
2016
, “
Adaptive Energy Optimization Toward Net-zero Energy Building Clusters
,”
ASME J. Mech. Des.
,
138
(
6
), p.
061405
. 10.1115/1.4033395
18.
Upside Energy
,
2019
, “
Energy is Changing
,” https://goo.gl/bx6ao3. Accessed January 2019.
19.
BeeBryte
,
2019
, “
Artificial Intelligence for Smarter Buildings
,” https://goo.gl/PNJcZQ. Accessed January 2019
20.
GridEdge
,
2019
, “
The Way We Use Energy is Changing
,” https://goo.gl/PNJcZQ. Accessed January 2019.
21.
Chen
,
H. Q.
,
Honda
,
T.
, and
Yang
,
M. C.
,
2013
, “
Approaches for Identifying Consumer Preferences for the Design of Technology Products: A Case Study of Residential Solar Panels
,”
ASME J. Mech. Des.
,
135
(
6
), p.
061007
. 10.1115/1.4024232
22.
Odonkor
,
P.
, and
Lewis
,
K.
,
2018
, “
Automated Design of Energy Efficient Control Strategies for Building Clusters Using Reinforcement Learning
,”
ASME J. Mech. Des.
,
141
(
2
), p.
021704
. 10.1115/1.4041629
23.
Odonkor
,
P.
, and
Lewis
,
K.
,
2018
, “
Control of Shared Energy Storage Assets Within Building Clusters Using Reinforcement Learning
,”
ASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
American Society of Mechanical Engineers, QC, Canada
, Paper No. DETC2018-86094. 10.1115/detc2018-86094
24.
Liu
,
N.
,
Liu
,
Y.
,
Logan
,
B.
,
Xu
,
Z.
,
Tang
,
J.
, and
Wang
,
Y.
,
2019
, “
Learning the Dynamic Treatment Regimes From Medical Registry Data Through Deep Q-network
,”
Sci. Rep.
,
9
(
1
), p.
1495
. 10.1038/s41598-018-37142-0
25.
Kalashnikov
,
D.
,
Irpan
,
A.
,
Pastor
,
P.
,
Ibarz
,
J.
,
Herzog
,
A.
,
Jang
,
E.
,
Quillen
,
D.
,
Holly
,
E.
,
Kalakrishnan
,
M.
,
Vanhoucke
,
V.
, and
Levine
,
S.
,
2018
, “
Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
,”
Proceedings of the Second Conference on Robot Learning
,
A. Billard, A. Dragan, J. Peters, and J. Morimoto, eds., Vol. 87 of Proceedings of Machine Learning Research, PMLR
, pp.
651
673
. 10.1109/icra.2018.8461039
26.
Lanham
,
M.
,
2018
, “
Deep Reinforcement Learning for the Cannabis Retail Market
,” https://goo.gl/aEEiV2. Accessed March 2019.
27.
Sutton
,
R. S.
, and
Barto
,
A. G.
,
1998
,
Reinforcement Learning: An Introduction
, Vol.
1
,
MIT Press Cambridge
,
MA
.
28.
Pecan Street Inc.
,
2018
, “
Pecan Street Dataport
,” https://dataport.cloud/. Accessed March 2018.
29.
Duch
,
W.
, and
Mańdziuk
,
J.
, eds.,
2007
,
Challenges for Computational Intelligence
,
Springer
,
Berlin Heidelberg
.
30.
Silver
,
D.
,
Schrittwieser
,
J.
,
Simonyan
,
K.
,
Antonoglou
,
I.
,
Huang
,
A.
,
Guez
,
A.
,
Hubert
,
T.
,
Baker
,
L.
,
Lai
,
M.
,
Bolton
,
A.
,
Chen
,
Y.
,
Lillicrap
,
T.
,
Hui
,
F.
,
Sifre
,
L.
,
Van den
,
Driessche
, and
Hassabis
,
D.
,
2017
, “
Mastering the Game of Go Without Human Knowledge
,”
Nature
,
550
(
7676
), p.
354
. 10.1038/nature24270
31.
Ren
,
Y.
,
Bayrak
,
A. E.
, and
Papalambros
,
P. Y.
,
2016
, “
Ecoracer: Game-Based Optimal Electric Vehicle Design and Driver Control Using Human Players
,”
ASME J. Mech. Des.
,
138
(
6
), p.
061407
. 10.1115/1.4033426
32.
Mnih
,
V.
,
Kavukcuoglu
,
K.
,
Silver
,
D.
,
Rusu
,
A. A.
,
Veness
,
J.
,
Bellemare
,
M. G.
,
Graves
,
A.
,
Riedmiller
,
M.
,
Fidjeland
,
A. K.
,
Ostrovski
,
G.
, and
Petersen
,
S.
,
2015
, “
Human-Level Control Through Deep Reinforcement Learning
,”
Nature
,
518
(
7540
), p.
529
. 10.1038/nature14236
33.
City Of Austin
,
2017
, “
Residential Electric Rate Schedules
,” https://goo.gl/Atd66W. Accessed March 2018.
34.
Cardin
,
M.-A.
, and
Hu
,
J.
,
2015
, “
Analyzing the Tradeoffs Between Economies of Scale, Time-Value of Money, and Flexibility in Design Under Uncertainty: Study of Centralized vs. Decentralized Waste-to-Energy Systems
,”
ASME J. Mech. Des.
,
138
(
1
), p.
011401
. 10.1115/1.4031422
You do not currently have access to this content.