Abstract

Demand is growing for the dense and high-performing IT computing capacity to support artificial intelligence, deep learning, machine learning, autonomous cars, the Internet of Things, etc. This led to an unprecedented growth in transistor density for high-end CPUs and GPUs, creating thermal design power (TDP) of even more than 700 watts for some of the NVIDIA existing GPUs. Cooling these high TDP chips with air cooling comes with a cost of the higher form factor of servers and noise produced by server fans close to the permissible limit. Direct-to-chip cold plate-based liquid cooling is highly efficient and becoming more reliable as the advancement in technology is taking place. Several components are used in the liquid-cooled data centers for the deployment of cold plate-based direct-to-chip liquid cooling like cooling loops, rack manifolds, CDUs, row manifolds, quick disconnects, flow control valves, etc. Row manifolds used in liquid cooling are used to distribute secondary coolant to the rack manifolds. Characterizing these row manifolds to understand the pressure drops and flow distribution for better data center design and energy efficiency is important. In this paper, the methodology is developed to characterize the row manifolds. Water-based coolant Propylene glycol 25% was used as the coolant for the experiments and experiments were conducted at 21 °C coolant supply temperature. Two, six-port row manifolds' P-Q curves were generated, and the value of supply pressure and the flowrate were measured at each port. The results obtained from the experiments were validated by a technique called flow network modeling (FNM). FNM technique uses the overall flow and thermal characteristics to represent the behavior of individual components.

References

1.
Dunlap
,
K.
, and
Rasmussen
,
N.
,
2012
, “
Choosing Between Room, Row, and Rack-Based Cooling for Data Centers
,” APC, South Kingstown, RI, Paper No.
WP-130
.https://2nsystems.com/wp-content/uploads/2019/09/Choosing-Between-Room-Row-and-Rack-based-Cooling-for-Data-Centers.pdf
2.
Evans
,
T.
,
2004
, “
The Different Types of Air Conditioning Equipment for IT Environments
,” APC, South Kingstown, RI, Paper No.
59
.https://cloudserver012718.home.pl/pub/katalogi/schneider/6_systemy_zasilania_gwarantowanego_i_chlodzenia/6_9_dokumenty_white_paper/wp-59.pdf
3.
Kaulgud
,
P.
,
Siddarth
,
A.
,
Simon
,
V. S.
, and
Agonafer
,
D.
,
2022
, “
Characterization of Parallel and Opposed Control Dampers to Observe the Effect on Thermal Mixing of Air Streams in an Air-Cooling Unit
,”
38th Semiconductor Thermal Measurement, Modeling & Management Symposium
(
SEMI-THERM
), San Jose, CA, Mar. 21–25, pp.
62
66
.https://ieeexplore.ieee.org/document/9755903
4.
Nemati
,
K.
,
Alissa
,
H. A.
,
Murray
,
B. T.
,
Schneebeli
,
K.
, and
Sammakia
,
B.
,
2017
, “
Experimental Failure Analysis of a Rear Door Heat Exchanger With Localized Containment
,”
IEEE Trans. Compon., Packag. Manuf. Technol.
,
7
(
6
), pp.
882
892
.10.1109/TCPMT.2017.2682863
5.
Tatchell-Evans
,
M.
,
Kapur
,
N.
,
Summers
,
J.
,
Thompson
,
H.
, and
Oldham
,
D.
,
2017
, “
An Experimental and Theoretical Investigation of the Extent of Bypass Air Within Data Centres Employing Aisle Containment, and Its Impact on Power Consumption
,”
Appl. Energy
,
186
, pp.
457
469
.10.1016/j.apenergy.2016.03.076
6.
Modi
,
H.
,
Chowdhury
,
U.
, and
Agonafer
,
D.
,
2022
, “
Impact of Improved Ducting and Chassis Re-Design for AirCooled Servers in a Data Center
,” 21st IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (
iTherm
), San Diego, CA, May 31–June 3, pp.
1
8
.10.1109/iTherm54085.2022.9899625
7.
Modi
,
H.
,
Shahi
,
P.
,
Chinthaparthy
,
L. S. R.
,
Gupta
,
G.
,
Bansode
,
P.
,
Shalom Simon
,
V.
, and
Agonafer
,
D.
,
2022
, “
Experimental Investigation of the Impact of Improved Ducting and Chassis Re-Design of a Hybrid-Cooled Server
,”
ASME
Paper No. IPACK2022-97587.10.1115/IPACK2022-97587
8.
Breen
,
T. J.
,
Walsh
,
E. J.
,
Punch
,
J.
,
Shah
,
A. J.
,
Bash
,
C. E.
,
Kumari
,
N.
, and
Cader
,
T.
,
2012
, “
From Chip to Cooling Tower Data Center Modeling: Chip Leakage Power and Its Impact on Cooling Infrastructure Energy Efficiency
,”
ASME J. Electron. Packag.
,
134
(
4
), p.
041009
.10.1115/1.4007744
9.
Breen
,
T. J.
,
Walsh
,
E. J.
,
Punch
,
J.
,
Shah
,
A. J.
, and
Bash
,
C. E.
,
2010
, “
From Chip to Cooling Tower Data Center Modeling: Part I Influence of Server Inlet Temperature and Temperature Rise Across Cabinet
,”
12th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems
, Las Vegas, NV, June 2–5, pp.
1
10
.10.1109/ITHERM.2010.5501421
10.
Breen
,
T. J.
,
Walsh
,
E. J.
,
Punch
,
J.
,
Shah
,
A. J.
,
Bash
,
C. E.
,
Rubenstein
,
B.
,
Heath
,
S.
, and
Kumari
,
N.
,
2012
, “
From Chip to Cooling Tower Data Center Modeling: Influence of Air-Stream Containment on Operating Efficiency
,”
ASME J. Electron. Packag.
,
134
(
4
), p.
041006
.10.1115/1.4007110
11.
Kang
,
S.
,
Miller
,
D.
, and
Cennamo
,
J.
,
2007
, “
Closed Loop Liquid Cooling for High Performance Computer Systems
,”
ASME
Paper No. IPACK2007-33870.10.1115/IPACK2007-33870
12.
Gao
,
T.
,
David
,
M.
,
Geer
,
J.
,
Schmidt
,
R.
, and
Sammakia
,
B.
,
2015
, “
Experimental and Numerical Dynamic Investigation of an Energy Efficient Liquid Cooled Chiller-Less Data Center Test Facility
,”
Energy Build.
,
91
, pp.
83
96
.10.1016/j.enbuild.2015.01.028
13.
Gao
,
T.
,
Shao
,
S.
,
Cui
,
Y.
,
Espiritu
,
B.
,
Ingalz
,
C.
,
Tang
,
H.
, and
Heydari
,
A.
,
2017
, “
A Study of Direct Liquid Cooling for High-Density Chips and Accelerators
,” 16th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (
ITherm
), Orlando, FL, May 30–June 2, pp.
565
573
.10.1109/ITHERM.2017.7992537
14.
Manaserh
,
Y. A.
,
Gharaibeh
,
A. R.
,
Tradat
,
M. I.
,
Rangarajan
,
S.
,
Sammakia
,
B. G.
, and
Alissa
,
H. A.
,
2022
, “
Multi-Objective Optimization of 3D Printed Liquid Cooled Heat Sink With Guide Vanes for Targeting Hotspots in High Heat Flux Electronics
,”
Int. J. Heat Mass Transfer
,
184
, p.
122287
.10.1016/j.ijheatmasstransfer.2021.122287
15.
Hoang
,
C. H.
,
Fallahtafti
,
N.
,
Rangarajan
,
S.
,
Gharaibeh
,
A.
,
Hadad
,
Y.
,
Arvin
,
C.
,
Sikka
,
K.
,
Schiffres
,
S. N.
, and
Sammakia
,
B.
,
2021
, “
Impact of Fin Geometry and Surface Roughness on Performance of an Impingement Two-Phase Cooling Heat Sink
,”
Appl. Therm. Eng.
,
198
, p.
117453
.10.1016/j.applthermaleng.2021.117453
16.
Chi
,
Y. Q.
,
Summers
,
J.
,
Hopton
,
P.
,
Deakin
,
K.
,
Real
,
A.
,
Kapur
,
N.
, and
Thompson
,
H.
,
2014
, “
Case Study of a Data Centre Using Enclosed, Immersed, Direct Liquid-Cooled Servers
,”
Semiconductor Thermal Measurement and Management Symposium
(
SEMITHERM
), San Jose, CA, Mar. 9–13, pp.
164
173
.10.1109/SEMI-THERM.2014.6892234
17.
Simon
,
V. S.
,
Modi
,
H.
,
Sivaraju
,
K. B.
,
Bansode
,
P.
,
Saini
,
S.
,
Shahi
,
P.
,
Karajgikar
,
S.
,
Mulay
,
V.
, and
D.
Agonafer
,
2022
, “
Feasibility Study of Rear Door Heat Exchanger for a High Capacity Data Center
,”
ASME
Paper No. IPACK2022-97494.10.1115/IPACK2022-97494
18.
Parida
,
P. R.
,
David
,
M.
,
Iyengar
,
M.
,
Schultz
,
M.
,
Gaynes
,
M.
,
Kamath
,
V.
,
Kochuparambil
,
B.
, and
Chainer
,
T.
,
2012
, “
Experimental Investigation of Water Cooled Server Microprocessors and Memory Devices in an Energy Efficient Chiller-Less Data Center
,”
28th Annual IEEE Semiconductor Thermal Measurement and Management Symposium
(
SEMITHERM
), San Jose, CA, Mar. 18–22, pp.
224
231
.10.1109/STHERM.2012.6188852
19.
Greenberg
,
S.
,
Mills
,
E.
,
Tschudi
,
B.
,
Rumsey
,
P.
, and
Myatt
,
B.
,
2006
, “
Best Practices for Data Centers: Lessons Learned From Benchmarking 22 Data Centers
,”
Proceedings of the ACEEE Summer Study on Energy Efficiency in Buildings
, Asilomar, CA
, Aug. 17–22, pp.
76
87
.https://datacenters.lbl.gov/sites/default/files/aceee-datacenters.pdf
20.
Gao
,
T.
,
Tang
,
H.
,
Cui
,
Y.
, and
Luo
,
Z.
,
2018
, “
A Test Study of Technology Cooling Loop in a Liquid Cooling System
,” 17th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (
ITherm
), San Diego, CA, May 29–June 1, pp.
740
747
.10.1109/ITHERM.2018.8419519
21.
Shahi
,
P.
,
Hurnekar
,
H.
,
Deshmukh
,
A.
,
Saini
,
S.
,
Bansode
,
P.
,
Kasukurthy
,
R.
, and
Agonafer
,
D.
, “
Assessment of Pump Power Savings at Rack Level for Dynamic Direct-to-Chip Liquid Cooling Using a Novel Flow Control Device
,”
J. Enhanced Heat Transfer
, 30(1), pp.
15
33
.10.1615/JEnhHeatTransf.2022044476
22.
Heydari
,
A.
,
Shahi
,
P.
,
Radmard
,
V.
,
Eslami
,
B.
,
Chowdhury
,
U.
,
Sivakumar
,
A.
,
Lakshminarayana
,
A.
, et al.,
2022
, “
Experimental Study of Transient Hydraulic Characteristics for Liquid Cooled Data Center Deployment
,”
ASME
Paper No. IPACK2022-97425.10.1115/IPACK2022-97425
23.
Heydari
,
A.
,
Shahi
,
P.
,
Radmard
,
V.
,
Eslami
,
B.
,
Chowdhury
,
U.
,
Hinge
,
C.
,
Cinthaparthy
,
L. S. R.
, et al.,
2022
, “
A Control Strategy for Minimizing Temperature Fluctuations in High Power Liquid to Liquid CDUs Operated at Very Low Heat Loads
,”
ASME
Paper No. IPACK2022-97434.10.1115/IPACK2022-97434
24.
Shahi
,
P.
,
2022
, “
Cold Plate Based Dynamic Liquid Cooling at Data Center and Chip Level
,”
Ph.D. dissertation
,
The University of Texas at Arlington
, Arlington, TX.https://mavmatrix.uta.edu/mechaerospace_dissertations/136
You do not currently have access to this content.