Abstract

This research studies the use of predetermined experimental plans in a live setting with a finite implementation horizon. In this context, we seek to determine the optimal experimental budget in different environments using a Bayesian framework. We derive theoretical results on the optimal allocation of resources to treatments with the objective of minimizing cumulative regret, a metric commonly used in online statistical learning. Our base case studies a setting with two treatments assuming Gaussian priors for the treatment means and noise distributions. We extend our study through analytical and semi-analytical techniques which explore worst-case bounds, the presence of unequal prior distributions, and the generalization to k treatments. We determine theoretical limits for the experimental budget across all possible scenarios. The optimal level of experimentation that is recommended by this study varies extensively and depends on the experimental environment as well as the number of available units. This highlights the importance of such an approach which incorporates these factors to determine the budget.

References

References
1.
Otto
,
K. N.
, and
Antonsson
,
E. K.
,
1993
, “
Extensions to the Taguchi Method of Product Design
,”
ASME J. Mech. Des.
,
115
(
1
), pp.
5
13
.
2.
Siang
,
J. K. K.
,
Chia
,
P. Z.
,
Koronis
,
G.
, and
Silva
,
A.
,
2018
, “
Exploring the Use of a Full Factorial Design of Experiment to Study Design Briefs for Creative Ideation
,”
ASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
American Society of Mechanical Engineers
,
New York
, p.
V007T06A008
.
3.
Frey
,
D. D.
, and
Jugulum
,
R.
,
2003
, “
How One-Factor-at-a-Time Experimentation Can Lead to Greater Improvements Than Orthogonal Arrays
,”
ASME 2003 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
American Society of Mechanical Engineers
,
New York
, pp.
505
513
.
4.
Sasena
,
M.
,
Parkinson
,
M.
,
Goovaerts
,
P.
,
Papalambros
,
P.
, and
Reed
,
M.
,
2002
, “
Adaptive Experimental Design Applied to an Ergonomics Testing Procedure DAC-34091
,”
Proceedings of the 2002 ASME DETC Conferences
,
Montreal, Canada
,
Sept. 29–Oct. 2
.
5.
Picheny
,
V.
,
Ginsbourger
,
D.
,
Roustant
,
O.
,
Haftka
,
R. T.
, and
Kim
,
N.-H.
,
2010
, “
Adaptive Designs of Experiments for Accurate Approximation of a Target Region
,”
ASME J. Mech. Des.
,
132
(
7
), p.
071008
. 10.1115/1.4001873
6.
Sutton
,
R. S.
, and
Barto
,
A. G.
,
1998
,
Reinforcement Learning: An Introduction
, Vol.
1
,
MIT Press
,
Cambridge
.
7.
Burtini
,
G.
,
Loeppky
,
J.
, and
Lawrence
,
R.
,
2015
, “
A Survey of Online Experiment Design With the Stochastic Multi-Armed Bandit
,”
arXiv:1510.00757
.
8.
Montgomery
,
D. C.
,
2017
,
Design and Analysis of Experiments
,
John Wiley & Sons
,
Hoboken, NJ
.
9.
Adcock
,
C. J.
,
1997
, “
Sample Size Determination: A Review
,”
J. R. Stat. Soc.: Ser. D (Statistician)
,
46
(
2
), pp.
261
283
. 10.1111/1467-9884.00082
10.
Lenth
,
R. V.
,
2001
, “
Some Practical Guidelines for Effective Sample Size Determination
,”
Am. Statistician
,
55
(
3
), pp.
187
193
. 10.1198/000313001317098149
11.
Schlaifer
,
R.
, and
Raiffa
,
H.
,
1961
,
Applied Statistical Decision Theory
,
Division of Research, Graduate School of Business Administration
,
Harvard
.
12.
Fraser
,
D. A. S.
, and
Guttman
,
I.
,
1956
, “
Tolerance Regions
,”
Ann. Math. Stat.
,
27
(
1
), pp.
162
179
.
13.
Willan
,
A. R.
,
2008
, “
Optimal Sample Size Determinations From An Industry Perspective Based on the Expected Value of Information
,”
Clin. Trials
,
5
(
6
), pp.
587
594
. 10.1177/1740774508098413
14.
Chaloner
,
K.
, and
Verdinelli
,
I.
,
1995
, “
Bayesian Experimental Design: A Review
,”
Stat. Sci.
,
10
(
3
), pp.
273
304
. 10.1214/ss/1177009939
15.
Even-Dar
,
E.
,
Mannor
,
S.
, and
Mansour
,
Y.
,
2006
, “
Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems
,”
J. Mach. Learn. Res.
,
7
(
Jun
), pp.
1079
1105
.
16.
Ciarleglio
,
M. M.
,
Arendt
,
C. D.
, and
Peduzzi
,
P. N.
,
2016
, “
Selection of the Effect Size for Sample Size Determination for a Continuous Response in a Superiority Clinical Trial Using a Hybrid Classical and Bayesian Procedure
,”
Clin. Trials
,
13
(
3
), pp.
275
285
. 10.1177/1740774516628825
17.
Schönbrodt
,
F. D.
, and
Wagenmakers
,
E. -J.
,
2018
, “
Bayes Factor Design Analysis: Planning for Compelling Evidence
,”
Psychon. Bull. Rev.
,
25
(
1
), pp.
128
142
. 10.3758/s13423-017-1230-y
18.
Sudarsanam
,
N.
,
Pitchai Kannu
,
B.
, and
Frey
,
D. D.
,
2019
, “
Optimal Replicates for Designed Experiments Under the Online Framework
,”
Res. Eng. Des.
,
30
(
3
), pp.
363
379
. 10.1007/s00163-019-00311-x
19.
Bellman
,
R.
,
1956
, “
Dynamic Programming and Lagrange Multipliers
,”
Proc. Natl. Acad. Sci.
,
42
(
10
), pp.
767
769
. 10.1073/pnas.42.10.767
20.
Robbins
,
H.
,
1985
,
Herbert Robbins Selected Papers
,
T. L.
Lai
and
D.
Siegmund
, eds.,
Springer
,
New York
, pp.
169
177
.
21.
Thompson
,
W. R.
,
1933
, “
On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples
,”
Biometrika
,
25
(
3/4
), pp.
285
294
. 10.1093/biomet/25.3-4.285
22.
Auer
,
P.
,
Cesa-Bianchi
,
N.
, and
Fischer
,
P.
,
2002
, “
Finite-Time Analysis of the Multiarmed Bandit Problem
,”
Mach. Learn.
,
47
(
2–3
), pp.
235
256
. 10.1023/A:1013689704352
23.
Perchet
,
V.
,
Rigollet
,
P.
,
Chassang
,
S.
,
Snowberg
,
E.
,
Grünwald
,
P.
,
Hazan
,
E.
, and
Kale
,
S.
,
2015
, “
Batched Bandit Problems
,”
Proceedings of the 28th Conference on Learning Theory
,
Paris
,
July 3–6
, p.
1456
.
24.
Auer
,
P.
, and
Ortner
,
R.
,
2010
, “
UCB Revisited: Improved Regret Bounds for the Stochastic Multi-Armed Bandit Problem
,”
Period. Math. Hungarica
,
61
(
1–2
), pp.
55
65
. 10.1007/s10998-010-3055-6
25.
Pham-Gia
,
T.
, and
Turkkan
,
N.
,
1992
, “
Sample Size Determination in Bayesian Analysis
,”
Statistician
,
41
(
4
), pp.
389
397
. 10.2307/2349003
26.
Joseph
,
L.
, and
Belisle
,
P.
,
1997
, “
Bayesian Sample Size Determination for Normal Means and Differences Between Normal Means
,”
J. R. Stat. Soc.: Ser. D (Statistician)
,
46
(
2
), pp.
209
226
. 10.1111/1467-9884.00077
27.
Machin
,
D.
,
Campbell
,
M. J.
,
Tan
,
S.-B.
, and
Tan
,
S.-H.
,
2011
,
Sample Size Tables for Clinical Studies
,
John Wiley & Sons
,
Englewood, NJ
.
28.
Stein
,
C.
,
1945
, “
A Two-Sample Test for a Linear Hypothesis Whose Power Is Independent of the Variance
,”
Ann. Math. Stat.
,
16
(
3
), pp.
243
258
. 10.1214/aoms/1177731088
29.
Simon
,
R.
,
Wittes
,
R.
, and
Ellenberg
,
S.
,
1985
, “
Randomized Phase II Clinical Trials
,”
Cancer Treat. Rep.
,
69
(
12
), pp.
1375
1381
.
30.
Simon
,
R.
,
1989
, “
Optimal Two-Stage Designs for Phase II Clinical Trials
,”
Contemp. Clin. Trials
,
10
(
1
), pp.
1
10
. 10.1016/0197-2456(89)90015-9
31.
Frey
,
D. D.
, and
Wang
,
H.
,
2006
, “
Adaptive One-Dactor-at-a-Time Experimentation and Expected Value of Improvement
,”
Technometrics
,
48
(
3
), pp.
418
431
. 10.1198/004017006000000075
32.
Arnold
,
B. C.
,
Balakrishnan
,
N.
, and
Nagaraja
,
H. N.
,
1992
,
A First Course in Order Statistics
, Vol.
54
,
SIAM
,
Philadelphia, PA
.
You do not currently have access to this content.