Abstract

The adoption of single-phase liquid immersion cooling (Sp-LIC) for Information Technology equipment provides an excellent cooling platform coupled with significant energy savings. There are, however, very limited studies related to the reliability of such cooling technology. The accelerated thermal cycling (ATC) test given ATC JEDEC is relevant just for air cooling but there is no such standard for immersion cooling. The ASTM benchmark D3455 with some appropriate adjustments was adopted to test the material compatibility because of the air and dielectric fluid differences in the heat capacitance property and corresponding ramp rate during thermal cycling. For this study, accelerated thermal degradation of the printed circuit board (PCB), passive components, and fiber optic cables submerged in air, white mineral oil, and synthetic fluid at a hoisted temperature of 45 °C and 35% humidity is undertaken. This paper serves multiple purposes including designing experiments, testing and evaluating material compatibility of PCB, passive components, and optical fibers in different hydrocarbon oils for single-phase immersion cooling. Samples of different materials were immersed in different hydrocarbon oils and air and kept in an environmental chamber at 45 °C for a total of 288 h. Samples were then evaluated for their mechanical and electrical properties using dynamic mechanical analyzer (DMA) and a multimeter, respectively. The cross section of some samples was also investigated for their structural integrity using scanning electron microscope (SEM). The literature gathered on the subject and quantifiable data gathered by the authors provide the primary basis for this research document.

Introduction

Nontraditional methods like immersion cooling of high-power information technology (IT) equipment have been adopted by operators to push power densities. It proves to be an efficient and cost-effective technique as it eliminates the need for server fans, elevated flooring, and in particular a computer room air conditioning unit, which constitutes a significant percentage of the traditional cooling cost. In single-phase immersion cooling, servers are immersed in a dielectric coolant, and single-phase forced convection is used to circulate the dielectric liquid across hot electronic components and ultimately rejecting the heat using a heat exchanger. In this kind of system, the hardware enjoys a few benefits like having lower junction temperature, relatively little temperature swings or hot spots, no electrochemical migration and it generally runs more reliably with the improved performance [13]. Thus, overall, single-phase immersion-cooled data centers would result in lower maintenance and operating cost; and a noise reduction compared to air-cooled data centers [4,5].

Single-phase liquid immersion cooling (Sp-LIC) offers significant advantages when compared to forced convection air cooling such as higher thermal mass, a high percentage of heat dissipation due to direct contact of dielectric fluids with every component, improved reliability due to as the information technology equipment is shielded from the impact of contaminants and harsh environment, reduction in CapEx and energy costs as fans and computer room air handler/computer room air conditioning units might not be required [6,7]. At the same time, Sp-LIC also does not require complex liquid distribution manifold infrastructure when compared to direct-to-chip liquid cooling, thus, proving to be a better option for edge and modular data center applications. Higher compaction and simplified data center layouts are possible with a tank-based single-phase immersion cooling, which can further reduce the CapEx. Opportunity for waste heat recovery, reduction in carbon footprint, and water usage reduction are some other added benefits of Sp-LIC [8]. It should however be noted that the quality of the waste heat recovered is higher for both pumped and open bath two-phase systems. Sp-LIC also leapfrogs traditional and more contemporary cooling technologies in terms of cooling efficiency, as shown in Fig. 1 [9]. Figure 1 shows the rack density as a function of Power Usage Effectiveness at the facility level for different system-level cooling technologies. The graph shows that even the most efficient air-cooling-based technologies are limited to 20–30 kW per rack. With its ease of implementation, especially for edge data centers, Sp-LIC can provide 300% more rack power density than air-cooling and almost twice as much as cold-plate-based single-phase direct-to-chip liquid cooling with higher efficiency [9].

Fig. 1
A comparison of various data center-level and system-level cooling technologies based on their cooling efficiencies [9]
Fig. 1
A comparison of various data center-level and system-level cooling technologies based on their cooling efficiencies [9]
Close modal

Submerging most of the servers' heat-generating components in a hydrocarbon dielectric fluid causes reliability issues at the device level [1012]. Most of the fluid suppliers and vendors perform soak tests to determine the material compatibility with hydrocarbon fluids. But they do not consider operational reliability such as mechanical and electrical properties of the components specific to the application work environment. This paper reports on the authors' attempts to explore these issues and ascertain whether they warrant special preventative care measures. There is a need for a framework for testing IT equipment cooled by single-phase immersion, as there is a lack of an established methodology or industry standard for the same. This paper is a first attempt at addressing this challenge by determining the design of experiments to test the reliability of wetted materials for single-phase immersion cooling. Accelerated thermal cycle (ATC) testing, especially the “ATC JEDEC” standard, which is utilized for air-cooling, has a crucial limitation. Due to vastly different heat capacities, there is a considerable difference in the ramp rates and corresponding thermal profile of air and dielectric fluid, making the current JEDEC standard not applicable in its current form for single-phase immersion cooling. ASTM D3455 is an industry standard available for testing the compatibility of various materials with mineral oil. These test methods cover screening for the compatibility of materials of construction with electrical insulating oil for use in electrical equipment [13]. These standards exist for the transformer insulating oils. The oil and winding temperatures of the transformer are much higher than single-phase immersion cooling operating temperatures [2]. With necessary modifications made in ASTM D3455 for immersion-cooled data center environment, it is possible to devise a method for testing material compatibility with different dielectric fluids. For this study, accelerated thermal degradation of the printed circuit board (PCB), passive components, and fiber optic cables submerged in air, white mineral oil, and synthetic fluid at a hoisted temperature of 45 °C and 35% humidity is undertaken.

Since the main constituents of a PCB are polymeric and metallic, a good test for mechanical performance would be one based on the change in elastic modulus of the PCB sample with time. To gage if any aging phenomena are triggered, in addition to the mechanical analysis, a close inspection of the formation of cracks and/or ingress of dielectric fluid in the PCB is necessary. In the case of passive components, a careful inspection of any microstructural cracks, which determines the mechanical and electrical execution of key server functions, is of prime importance.

Optical cables are employed in systems that require higher data transmission rates, thus widely in rack and data center-level connection. Maintaining the integrity of the transmitted signal is critical, and to this end, it is necessary to test if there is a reduction in the mechanical and optical performance of the cables over time when comparing immersed and traditionally air-cooled interconnections. Extensive ingress of dielectric fluid can lead to structural or optical changes at the core-cladding interface, leading to increased attenuation. Existing and even the most recent literature on mechanical testing of water-immersed optical fibers reveal that the effect of diffusion of water into the cable structure is the dominant factor in mechanical degradation, as attested by Glaesemann [14] and Dwivedi [15]. However, there does not exist any documented testing of optical cables by high-temperature aging upon immersion in mineral or synthetic oil. The analysis done in this research serves to fill that gap in knowledge by testing certain assumptions about the extent of change in thermomechanical behavior of thermally aged IT hardware components immersed in single-phase dielectric fluid.

The primary objective of this work is to quantitatively ascertain and characterize the material compatibility of single-phase oil-immersed components. This paper reviews the changes brought about by single-phase immersion in the physical and electrical properties of three types of essential data center server components: PCBs, passive electrical components, and fiberoptic data cables. The changes in the electrical properties of passive components like capacitors, resistors, and transistors are analyzed for single-phase immersion cooling. The electrical properties such as resistance and capacitance were recorded before and after thermal aging in air and hydrocarbon fluids using a multimeter. Scanning electron microscope (SEM) analysis was used to check the structural integrity and fluid ingress in the aged samples. Variation in Young's modulus was determined using dynamic mechanical analyzer (DMA) to quantify the change in mechanical properties of thermally aged samples of PCB and optical cables. It is an exercise in elucidating the reliability concerns of active and passive server components, and optical data cables due to the possible physical or chemical interaction with a dielectric fluid. This paper outlines a comprehensive methodology for testing these components and develops an understanding of the impact of single-phase immersion cooling using hydrocarbon fluids on the reliability of data center components.

Methodology

Sample Preparation

Printed Circuit Board.

Samples of PCB were prepared using an Isomet 1000 precision cutter. The Iso-Met 1000 is a precision segmenting saw intended for cutting different kinds of materials with negligible deformation. This accurate cutting machine provides seamless cutting for fragile parts by just utilizing gravity-fed force. Its extraordinary flexibility in chucking allows into consideration holding a wide range of test shapes and configurations, which makes it an excellent precision cutting machine fit for sectioning both fragile and ductile metals, composites, cement, laminates, plastics, electronic gadgets, and biomaterials. The cut samples prepared have dimensions of 50 mm × 10 mm as per the DMA requirements. The samples were cut in such a way that the copper distribution in all samples was identical.

Passive Components.

The passive components were procured based on the surface mount components commonly used in the servers. The list of components used for this experiment includes thick film resistors, polymer capacitors, electrolytic capacitors, and transistors. Thick film resistors are widely used in the electrical and electronic industry and are readily available and cheap. The thick film resistor used in this experiment has a resistance of 1 kΩ and can be operated at a maximum temperature of 105 °C. An electrolytic capacitor is a form of a polarized capacitor where the anode is formed using pure aluminum and a nonsolid covering electrolyte acts as the cathode. A thin insulating layer of aluminum oxide acts as the dielectric of such capacitors. A paper sheet soaked in the electrolytic solution is placed between them and then the plates are wound around one another and placed into a can. The electrolytic capacitors used in this experiment have a capacitance of 100 μF and 470 μF, respectively, and can operate in a temperature range of −55 °C to 105 °C. The polymer capacitor is like the aluminum electrolytic capacitor in construction: it has an anode foil, a cathode foil, and different paper separators in between. Instead of a liquid electrolyte, the polymer capacitor has a solid polymer. The polymer capacitors used in this experiment have a capacitance of 270 μF and 560 μF, respectively, and can operate in a temperature range of −55 °C to 105 °C. A transistor is a miniature electronic component that can act both as an amplifier and as a switch. Depending on the material used for doping in transistors, we can classify the terminals as N-type and P-type. The Negative-Positive-Negative (NPN) and Positive-Negative-Positive (PNP) junction transistors are quite commonly used in electronic appliances. In this study, both NPN and PNP transistors are used and can operate reliably up to a temperature of 150 °C.

Fiberoptic Cable.

Active optical cables (AOCs), a relatively new choice for data transmission networks, are enjoying growing use in data centers. They are a variant of standard multimode fiber that is preferable for short-reach interconnections. The AOC tested in this work was a standard OM2 category Silica-based two-circular-core multimode fiber (MMF) cable. Three 60 cm long sections of the optical cable were cut from the original 3 m long cable as obtained (pristine cable). The samples were then placed in an environmental chamber for thermal aging in different fluid mediums as shown in Fig. 2. To understand the effect of single-phase immersion on the thermal aging of fiber coating, the ends of cut sections were kept open and immersed to enable penetration of hydrocarbon oils into the cables. This can be understood as a possible real-world scenario where there would be physical contact of oil with the cladding coating that protects the core of an optical cable immersed in a single-phase dielectric fluid. As will be explained in the Experimental Setup and Procedure section, the experiment was divided into four aging cycles. At the end of each cycle, 15 cm of wire sections were cut for mechanical testing from the aged 60 cm sections of the cables from each jar. After this, three specimens of length 5 cm each and an average outside diameter of about 2.9 mm, were carefully cut from the extracted 15 cm sections and tested in the DMA probe. The length of 5 cm for the final specimens was selected based on the maximum length that could fit in the DMA probe. This was done to obtain enough (three) data points for the measurement of properties after each cycle, thus increasing the confidence level of the obtained results.

Fig. 2
The environmental chamber containing the synthetic and mineral oil jars with components
Fig. 2
The environmental chamber containing the synthetic and mineral oil jars with components
Close modal

Experimental Setup and Procedure.

Accelerated thermal aging conditions were produced by immersing all three types of samples in borosilicate jars containing mineral oil, synthetic fluid (EC-100), and exposed to air. The samples were kept at an elevated temperature and relative humidity (RH) setpoint in the environmental chamber for a total duration of 288 h, as shown in Fig. 2. The environmental chamber used for this experiment was a Thermotron SE-600-10-10 (Thermotron Industries, Holland, MI). The samples were prepared and baked per ASTM D3455 standard with some modifications done to suit the standard as per single-phase immersion of IT equipment with dielectric fluids as stated earlier. Initially, the environmental chamber underwent a dry run, immediately followed by the sample baking cycle as described in the standard but with reduced predry time. During the baking cycle, all the components placed inside the chamber had minimal contact with each other.

There were five jars used in this experiment: two jars filled with mineral oil having PCB and passive components in one jar and optical fiber cable in the other. Similarly, two jars were filled with EC-100 containing PCB and passive components in one jar and optical fiber cable in the other. The last jar was exposed to air and contained all three types of samples. A comparison of the physical properties of these fluids, water, and air are shown in Table 1. Optical cable samples were kept in a separate jar for the case of the hydrocarbon fluids to avoid the impact of the plasticizer leaching out from the poly vinyl chloride (PVC) cable on other samples. After the baking cycle, the jars were each filled with dielectric fluids up to the 800 ml mark with keeping one empty for air. All five jars were placed in the environmental chamber and the first 72-h cycle was initiated. Upon completion of the first 72 h, a set of samples from all five jars was taken out using tongs and stored in airtight aluminum packets after which the next cycle for the period of another 72 h was started. This was done for four cycles, thus totaling an aging duration of 288 h. This entailed a ramp time of approximately 1 h and a dwell time of between 70 and 71 h for each cycle. The choice of 45 °C temperature and 35% RH setpoints closely resembled the choice of aging conditions in literatures [1,2]. 35% RH corresponding to a 30 °C dew point in 45 °C dry bulb chamber temperature is low enough to remove moisture as a variable and is relatively close to realistic air-cooled data center operating conditions.

Table 1

Comparison of physical properties of the air, water, white mineral oil, and synthetic fluid (EC-100)

Type of fluidHeat capacity (KJ/kg K)Density (kg/m3)Kinematic viscosity (× 10−6 m2/10−6 m2/s)Heat conductivity (W/m K)
Air1.011.2250.160.02
Water4.1910000.660.58
White mineral oil1.67849.316.020.13
Synthetic fluid (EC-100)2.165803.7813.220.1378
Type of fluidHeat capacity (KJ/kg K)Density (kg/m3)Kinematic viscosity (× 10−6 m2/10−6 m2/s)Heat conductivity (W/m K)
Air1.011.2250.160.02
Water4.1910000.660.58
White mineral oil1.67849.316.020.13
Synthetic fluid (EC-100)2.165803.7813.220.1378
Table 2

Air-exposed versus mineral oil-immersed versus synthetic fluid immersed PNP transistor

PNPBase to emitterBase to collectorEmitter to baseCollector to baseCollector to emitter
Test 1 (72 h Cycle)Air exposedOLOL0.6710.67OL
Mineral oil immersedOLOL0.6710.67OL
Synthetic fluid immersedOLOL0.6710.669OL
Test 2 (144 h cycle)Air exposedOLOL0.6710.67OL
Mineral oil immersedOLOL0.6710.67OL
Synthetic fluid immersedOLOL0.6710.669OL
Test 3 (216 h cycle)Air exposedOLOL0.6710.669OL
Mineral oil immersedOLOL0.6710.669OL
Synthetic fluid immersedOLOL0.6690.669OL
Test 4 (288 h cycle)Air exposedOLOL0.6710.669OL
Mineral oil immersedOLOL0.6670.667OL
Synthetic fluid immersedOLOL0.6710.669OL
PNPBase to emitterBase to collectorEmitter to baseCollector to baseCollector to emitter
Test 1 (72 h Cycle)Air exposedOLOL0.6710.67OL
Mineral oil immersedOLOL0.6710.67OL
Synthetic fluid immersedOLOL0.6710.669OL
Test 2 (144 h cycle)Air exposedOLOL0.6710.67OL
Mineral oil immersedOLOL0.6710.67OL
Synthetic fluid immersedOLOL0.6710.669OL
Test 3 (216 h cycle)Air exposedOLOL0.6710.669OL
Mineral oil immersedOLOL0.6710.669OL
Synthetic fluid immersedOLOL0.6690.669OL
Test 4 (288 h cycle)Air exposedOLOL0.6710.669OL
Mineral oil immersedOLOL0.6670.667OL
Synthetic fluid immersedOLOL0.6710.669OL
Table 3

Air-exposed versus mineral oil-immersed versus synthetic fluid immersed NPN transistor

NPNBase to emitterBase to collectorEmitter to baseCollector to baseCollector to emitter
Test 1 (72 h cycle)Air exposed0.5750.556OLOLOL
Mineral oil immersed0.570.554OLOLOL
Synthetic fluid immersed0.5710.552OLOLOL
Test 2 (144 h cycle)Air exposed0.5740.557OLOLOL
Mineral oil immersed0.5740.556OLOLOL
Synthetic fluid immersed0.5750.557OLOLOL
Test 3 (216 h cycle)Air exposed0.5760.557OLOLOL
Mineral oil immersed0.5730.556OLOLOL
Synthetic fluid immersed0.5740.556OLOLOL
Test 4 (288 h cycle)Air exposed0.5740.556OLOLOL
Mineral oil immersed0.5730.556OLOLOL
Synthetic fluid immersed0.5710.554OLOLOL
NPNBase to emitterBase to collectorEmitter to baseCollector to baseCollector to emitter
Test 1 (72 h cycle)Air exposed0.5750.556OLOLOL
Mineral oil immersed0.570.554OLOLOL
Synthetic fluid immersed0.5710.552OLOLOL
Test 2 (144 h cycle)Air exposed0.5740.557OLOLOL
Mineral oil immersed0.5740.556OLOLOL
Synthetic fluid immersed0.5750.557OLOLOL
Test 3 (216 h cycle)Air exposed0.5760.557OLOLOL
Mineral oil immersed0.5730.556OLOLOL
Synthetic fluid immersed0.5740.556OLOLOL
Test 4 (288 h cycle)Air exposed0.5740.556OLOLOL
Mineral oil immersed0.5730.556OLOLOL
Synthetic fluid immersed0.5710.554OLOLOL

Once all the samples were extracted from the jars, they were put through the relevant mechanical and electrical tests. For PCB samples, Young's modulus was measured using DMA7100 for various temperature and frequency conditions. Young's modulus was chosen since it is an ideal indicator of changes in the mechanical properties of polymeric materials. The temperature range was kept between 30 and 50 °C at varying frequencies of 1, 2, 5, and 10 hertz, and a runtime of 30 min per sample. SEM analysis was then carried out on the cross section of the components to determine the structural integrity of the samples. For passive components, after the thermal cycling was completed, the components were taken out of the jars using a pair of tongs. They were then placed in an aluminum cover carefully to prevent any air contact. Changes in electrical properties such as capacitance, resistance, and barrier voltage were recorded using a multimeter. The material characteristics were recorded by performing SEM imaging of the cut cross section of the components. A summary of the entire procedure followed during the experiments is shown in Fig. 3.

Fig. 3
Flowchart describing experimental procedure followed
Fig. 3
Flowchart describing experimental procedure followed
Close modal

For AOC, the temperature was ramped at a rate of 10 °C per minute ranging from 20 to 80 °C with data sampled for every 3 s interval. The choice of frequency was based on the reports from the literature on viscoelastic properties of data cables tested. A frequency scan required at least an additional frequency of 5 Hz would allow studying molecular weight and distribution trends of the molecular weight. A small sinusoidal force amplitude of 100 mN was applied by the machine on the sample with a strain amplitude of 20 μm (0.1% strain). Data generated by the software for elastic modulus facilitated the characterization of frequency and temperature-dependent viscoelastic properties of the polymeric materials in the optical cable and under sinusoidal excitations. SEM micrograph of the aged samples was analyzed to check the structural defects due to leaching out of plasticizers.

Results and Discussion

Printed Circuit Board.

The brittleness of a component is known to increase with an increase in Young's Modulus. Figure 4 shows the comparison of variation in the Young's modulus of PCB samples for four thermal cycles. Young's modulus values for an air-exposed sample show a consistent trend. The values for mineral and synthetic oils-immersed samples closely follow the same trend as air-exposed samples but show a little increase after the fourth cycle. This is attributed to the fact that prolonged immersion at high temperatures may have little effect on the mechanical and viscoelastic properties of the PCB.

Fig. 4
Comparison of Young's modulus of PCBs exposed to air, immersed in mineral oil or immersed in synthetic fluid
Fig. 4
Comparison of Young's modulus of PCBs exposed to air, immersed in mineral oil or immersed in synthetic fluid
Close modal
Fig. 5
SEM analysis for mineral oil and synthetic fluid immersed PCBs
Fig. 5
SEM analysis for mineral oil and synthetic fluid immersed PCBs
Close modal

Figure 5 shows the images obtained from SEM analysis for mineral oil and synthetic fluid immersed PCB samples. This analysis was performed on the samples after the fourth cycle (288 h) of thermal aging. As seen in the images, there was no sign of any crack or oil penetration in the PCB samples. Also, there was no deformation observed and it can be sufficiently concluded that the structural integrity of the samples is unaffected post-thermal cycling for both the oils.

Passive Components.

The electrical properties of the passive components from each of the fluid medium are measured after every thermal cycle of 72 h. Using the trend data of the electrical properties measured, a comparative variation in the electrical properties of the samples exposed to air, immersed in mineral oil or synthetic fluid is obtained. The change in resistance of the thick film resistor tested in three different mediums is represented below in Fig. 6.

Fig. 6
Change in resistance of samples of surface mount resistors exposed to various fluid media versus number of thermal cycles
Fig. 6
Change in resistance of samples of surface mount resistors exposed to various fluid media versus number of thermal cycles
Close modal

Figure 6 shows the plot of the variation of the resistance of thick film resistors versus the number of thermal aging cycles. It is evident from the plots that the resistance is maintained constant at the reference value (measured before thermal cycling) for all three mediums during the thermal cycling. It can, thus, be concluded that the thick film resistors are as reliable to use in immersion cooling as they are in air cooling. The authors took two different electrolytic capacitors for the experiment and the variation in the capacitance after thermal cycling is illustrated in Figs. 7(a) and 7(b). The value of capacitance reduces slightly for mineral oil but is almost constant for the synthetic fluid for three cycles in one case. For the second capacitor, the value of capacitance also does not vary much past the reference value for two cycles and then shows a slight dip for both mineral and synthetic fluid. The allowable tolerance of the electrolytic capacitor is 20% of its original or reference value. As seen the variation in the capacitance is well within the allowable range and hence it is safe and reliable to use in immersion cooling.

Fig. 7
Change in capacitance of the electrolytic capacitors exposed to various fluid media versus number of thermal cycles
Fig. 7
Change in capacitance of the electrolytic capacitors exposed to various fluid media versus number of thermal cycles
Close modal

For polymer capacitors, the allowable tolerance in capacitance of the polymer capacitor is 20% of its original or reference value. As shown in Figs. 8(a) and 8(b), the value of capacitance varies rather erratically in the air but is consistent for both mineral oil and synthetic fluid for one capacitor. The variation in the capacitance is well within the allowable range and hence it is safe and reliable to use in immersion cooling. It can also be seen that the variation in the capacitance for either type of the capacitors tested is approximately 3% from the base value. Thus, from an operational point of view, both polymer and electrolytic capacitors show good compatibility. Further studies on the impact of prolonged immersion on coolant chemistry due to these types of capacitors should be conducted to assess other reliability issues.

Fig. 8
Change in capacitance of polymer capacitors exposed to various media versus number of thermal cycles
Fig. 8
Change in capacitance of polymer capacitors exposed to various media versus number of thermal cycles
Close modal

Tables 2 and 3 show the comparison of voltage across the junctions of PNP and NPN transistors for all three fluid mediums, where “OL” indicates an “over limit” case. As seen in the tables, for both the PNP and NPN transistors, the values of the voltage across the terminals for all the thermal cycles are within the range specified which was 0.45–0.9 volts. Hence, the transistors can safely operate in oil even at elevated temperatures without any effect on their reliability.

From the SEM images shown in Fig. 9, it can be inferred that there were no observed changes in microscopic structure in any of the passive components. There is no formation of cracks or penetration of fluid on the surface of the components. The rubber sealing used in the Electrolytic capacitor is synthetic rubber EPDMS. These results show that there is no effect of dielectric fluids on the material properties of rubber sealing.

Fig. 9
SEM images of components taken from fourth thermal cycle. (From top left) Thick film resistor immersed in mineral oil, (second from top left) thick film resistor immersed in synthetic fluid, (top right) rubber sealing (ethylene propylene diene monomer (EPDMS)) of electrolytic capacitor immersed in mineral oil, (bottom left) rubber sealing of electrolytic capacitor immersed in synthetic fluid, (second from bottom left) rubber sealing of polymeric capacitor immersed in mineral oil, (bottom right) rubber sealing of polymeric capacitor immersed in synthetic fluid.
Fig. 9
SEM images of components taken from fourth thermal cycle. (From top left) Thick film resistor immersed in mineral oil, (second from top left) thick film resistor immersed in synthetic fluid, (top right) rubber sealing (ethylene propylene diene monomer (EPDMS)) of electrolytic capacitor immersed in mineral oil, (bottom left) rubber sealing of electrolytic capacitor immersed in synthetic fluid, (second from bottom left) rubber sealing of polymeric capacitor immersed in mineral oil, (bottom right) rubber sealing of polymeric capacitor immersed in synthetic fluid.
Close modal

Optical Cables.

For active optic cables, the DMA test results of three specimens after each of the four thermal cycles were recorded and studied. The obtained values of each of the three samples after every cycle were averaged and plotted against increasing sample aging time. For convenience and consistency, the results reported are the ones observed for the 1 Hz frequency. The results of the DMA tests at the DMA furnace temperatures of 40 °C and 22 °C have been presented in Figs. 10 and 11, respectively. Changes in the viscoelastic properties indicate changes in the macromolecular polymer chain structure of either the jacket or the cladding coating or both.

Fig. 10
Evolution of complex/elastic modulus over aging time at DMA furnace temperature of 40 °C
Fig. 10
Evolution of complex/elastic modulus over aging time at DMA furnace temperature of 40 °C
Close modal
Fig. 11
Evolution of complex/elastic modulus over aging time at DMA furnace temperature of 22 °C
Fig. 11
Evolution of complex/elastic modulus over aging time at DMA furnace temperature of 22 °C
Close modal

To better understand the underlying mechanism of the elastic modulus changes in the thermoplastic PVC jacket due to aging by immersion in high temperature, it was deemed that Young's modulus (E) versus time results may prove enough. The rate of change of elastic modulus, which is a strong indicator of thermomechanical behavior, undergoes a reduction by the third cycle. Logarithmic trends fitted to the data in the graphs bolster the hypothesis that the effect of aging is self-limiting and slows down after the first few cycles. PVC jacket hardness is increasing due to plasticizers leaching out and/or ingress through the diffusion of oil. Coating toughness rose in magnitude and may contribute to an increase in the fiber mechanical strength. Stiffness and Young's modulus of cable increased with aging time, as hypothesized, and this agreed with recent and extant literature [16,17]. Reduction in elastic modulus change over aging time is proposed to be due to two competing phenomena:

  1. Increased initial diffusion of oil molecules through PVC jacket, and loss of plasticizers due to elevated aging temperature increasing diffusion coefficient, and aggregation of the PVC molecules increasing microvoids. The loss of plasticizer is causing PVC molecules to aggregate and allow for more voids for the oil ingress. High temperatures damage acrylate coatings, which might deteriorate its adhesion to the fiber, causing delamination and exposing the bare fiber to undergo cracking/fractures.

  2. Increased thermal degradation of fiber coating entails stiffening, reduced ductility, and increased net strength resulting in a reduction in the diffusion of oil molecules, as evident from tanδ peak shifts.

The upward trend in tanδ peaks, as seen in Fig. 12 is indicative of elevation in glass transition temperature (Tg). Since Tg is strongly dependent on the content of the plasticizer and general polymer chemical state, it is reasonable to conclude that some plasticizers leached out into the oil because of aging.

Fig. 12
Variation of tanδ with aging time, indicating peak shifting with increasing time
Fig. 12
Variation of tanδ with aging time, indicating peak shifting with increasing time
Close modal

An inspection of the SEM micrographs shown in Fig. 13 reveals small inorganic filler molecules, plasticizers, and microvoids on the PVC surface. When we compare micrographs of the mineral-oil immersed sample surface morphology with that of the air-exposed, increased microvoids and aggregation of PVC molecules are readily visible, which are potentially causing the coating to toughen. Nevertheless, the coating becoming stiffer is rather a positive outcome as it minimizes transmission losses due to microbending. However, this needs corroboration by optical loss testing.

Fig. 13
SEM micrographs of air-exposed (top) and oil immersed (bottom) AOC samples after the third cycle
Fig. 13
SEM micrographs of air-exposed (top) and oil immersed (bottom) AOC samples after the third cycle
Close modal

Conclusion and Future Work

This study provides a cogent methodology to design experiments for determining the material compatibility and evaluating the material compatibility of common PCBs, passive surface mount components, and active optic cables in single-phase immersion cooling. Evaluation of the material compatibility from this study will provide essential literature and reference data for data center administrators to successfully implement this cooling technology.

Many uncertainties and concerns persist regarding the effects of mineral oil immersion cooling on the reliability of IT equipment both at the component and chassis levels [18]. The issues associated with the mechanical reliability and operational service impact of adopting a single-phase immersion cooling strategy must be addressed before the technology will see widespread adoption. The above-mentioned issues were successfully investigated by utilizing a modified benchmark ASTM standard for the accelerated thermal aging test. The evaluation of mechanical and electrical properties of the samples was conducted post-thermal cycling exposed to air and two hydrocarbon oils used. Images from SEM analysis prove that there is no alteration in the structural integrity or diffusion of oils at a microscopic level in PCBs and passive components. The increase in the Young's modulus and stiffness of AOCs can be attributed to the plasticizer leaching and molecule aggregation in the cables. The electrical integrity tests sufficiently concluded that the variation in the electrical characteristics of the passive components was well within allowable ranges.

It can be reiterated that the field of reliability for single-phase immersion cooling has a lot of scope for future work. Comparable tests need to be repeated at higher aging temperatures and different humidity levels, to determine a fluid-sample interaction-dependent phenomenological exponential Arrhenius model, based on the chemical reaction kinetics of the interaction. The numerical value of the aging factor should be determined for long-term reliability evaluation. Since the present research was performed at a single aging temperature of 45 °C, there are not enough data points to develop a first-order Arrhenius-type model to the temperature dependence of cable performance. Multiphysics simulations can be done using commercially available finite element analysis (FEA) codes to validate the theoretical model. In the future, research should also be carried out to understand the changes in the properties of the fluid due to plasticizer contamination and its impact on thermal performance. Nonetheless, owing to the plateauing of elastic modulus values after longer aging durations as observed for AOC and no significant changes in mechanical and electrical properties of PCB and passive component samples, proves that single-phase immersion cooling is a reliable choice for electronics cooling. It also has huge upsides to address future packaging technologies that utilize heterogeneous integration whether it is in 2.5D or 3D form factor since the fluid can freely be in touch with all components as opposed to cold plate cooling, which requires approaches such as cold rails to cool the entire system. It is also important to extend the study to include widely used and reliable FluorinertTM and NovecTM fluids.

Acknowledgment

This work is supported by NSF IUCRC Award No. IIP-1738811. The authors would like to thank Dr. David Sundin, Chief Scientist at Engineered Fluids, LLC, for providing the synthetic hydrocarbon oil EC-100 for this research.

Funding Data

  • National Science Foundation (NSF) IUCRC (Award No. IIP-1738811; Funder ID: 10.13039/100000001).

Nomenclature

     
  • AOC =

    active optical cables

  •  
  • ASTM =

    American Society for Testing and Materials

  •  
  • ATC =

    accelerated thermal cycling

  •  
  • DMA =

    dynamic mechanical analyzer

  •  
  • E =

    Young's modulus

  •  
  • EPDMS =

    ethylene propylene diene monomer

  •  
  • FEA =

    finite element analysis

  •  
  • GPa =

    giga pascal

  •  
  • JEDEC =

    Joint Electron Device Engineering Council

  •  
  • MMF =

    multimode fiber

  •  
  • PCB =

    printed circuit board

  •  
  • PVC =

    poly vinyl chloride

  •  
  • RH =

    relative humidity

  •  
  • SEM =

    scanning electron microscope

  •  
  • SpLIC =

    single-phase liquid immersion cooling

  •  
  • μF =

    microFarad

References

1.
Shah
,
J.
,
Eiland
,
R.
,
Siddharth
,
A.
, and
Agonafer
,
D.
,
2016
, “
Effects of Mineral Oil Immersion Cooling on IT Equipment Reliability and Reliability Enhancements to Data Center Operations
,” 15th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (
ITherm
), Las Vegas, NV, May 31–June 3, pp.
316
325
.10.1109/ITHERM.2016.7517566
2.
Shah
,
J. M.
,
Eiland
,
R.
,
Rajmane
,
P.
,
Siddarth
,
A.
,
Agonafer
,
D.
, and
Mulay
,
V.
, (April 10,
2019
, “
Reliability Considerations for Oil Immersion-Cooled Data Centers
,”
ASME. J. Electron. Packag.
,
141
(
2
), p.
021007
.10.1115/1.4042979
3.
Shah
,
J. M.
,
2016
, “
Reliability Challenges in Airside Economization and Oil Immersion Cooling
,” Master's thesis,
The University of Texas at Arlington
,
Arlington, TX
.
4.
Bansode
,
P. V.
,
Shah
,
J. M.
,
Gupta
,
G.
,
Agonafer
,
D.
,
Patel
,
H.
,
Roe
,
D.
, and
Tufty
,
R.
,
2018
, “
Measurement of the Thermal Performance of a Single-Phase Immersion Cooled Server at Elevated Temperatures for Prolonged Time
,”
ASME
Paper No. IPACK2018-8432. 10.1115/IPACK2018-8432
5.
Bansode
,
P. V.
,
Shah
,
J. M.
,
Gupta
,
G.
,
Agonafer
,
D.
,
Patel
,
H.
,
Roe
,
D.
, and
Tufty
,
R.
,
2019
, “
Measurement of the Thermal Performance of a Custom-Build Single-Phase Immersion Cooled Server at Various High and Low Temperatures for Prolonged Time
,”
ASME J. Electron. Packag.
,
142
(
1
), p.
11010
.10.1115/1.4045156
6.
Day
,
T.
,
Lin
,
P.
, and
Buger
,
R.
,
2019
, “
Liquid Cooling Technologies for Data Centers and Edge Applications
,” Schneider Electric White Paper No. 265, Andover, MA, accessed Aug. 12, 2018, https://download.schneider-electric.com/files?p_Doc_Ref=SPD_VAVR-AQKM3N_EN
7.
Fangzhi
,
W.
,
2018
, “
Best Practice of Alibaba Datacenter-Immersion Cooling Escorts Cloud Computing
,”
Presented at OCP Global Summit
, Mar. 20–21, San Jose, CA, p. 7, accessed Nov. 10, 2018, https://www.opencompute.org/files/Immersion-Cooling-for-Green-Computing-V1.0.pdf
8.
Dhruv
,
V.
,
2020
, “
Air-Based Cooling vs. Liquid-Based Cooling – Newly Updated
,” Green Revolution Cooling, Austin, TX, accessed Jan. 2021, https://www.grcooling.com/air-based-cooling-vs-liquid-based-cooling/
9.
Wen
,
F.
,
2018
, “
Best Practice of Alibaba Datacenter-Immersion Cooling Escorts Cloud Computing
,”
Presented at the OCP Global Summit
, San Jose, CA, p. 12, accessed Sept. 3, 2021, https://www.opencompute.org/files/Immersion-Cooling-for-Green-Computing-V1.0.pdf
10.
Shah
,
J. M.
,
Rizvi
,
S. H. I.
,
Kota
,
I. S.
,
Nagilla
,
S. R.
,
Thakkar
,
D.
, and
Agonafer
,
D.
,
2016
, “
Design Considerations Relating to Non-Thermal Aspects of Oil Immersion Cooling
,”
ASME
Paper No. IMECE2016-67320.10.1115/IMECE2016-67320
11.
Tulkoff
,
C.
, and
Boyd
,
C.
,
2013
, “
Improved Efficiency & Reliability for Data Center Servers Using Immersion Oil Cooling
,” IPC Electronic System Technologies Conference and Exhibition
(IPC ESTC)
, Las Vegas, NV, May
20
23
.
12.
Shah
,
J. M.
,
Eiland
,
R.
,
Rizvi
,
S. H. I.
, and
Agonafer
,
D.
,
2015
, “
Critical Non-Thermal Considerations for Oil Cooled Data Centers
,”
IMAPS Advanced Technology Workshop and Tabletop Exhibit on Thermal Management
, Los Gatos, CA, Sept. 22–24.
13.
ASTM
,
2011
,
Standard Test Methods for Compatibility of Construction Material With Electrical Insulating Oil of Petroleum Origin
,
ASTM International
, West Conshohocken, PA, Standard No. ASTM D3455-02.
14.
Glaesemann
,
G. S.
,
Gulati
,
S. T.
, and
Helfinstine
,
J. D.
,
1988
, “
Effect of Strain and Surface Composition on Young's Modulus of Optical Fibers
,”
Optical Fiber Communication Conference
,
Optical Society of America
, New Orleans, LA, Jan. 25, p.
TuG5
.10.1364/OFC.1988.TuG5
15.
Dwivedi
,
A.
,
Glaesemann
,
G. S.
, and
Eoll
,
C. K.
,
1994
, “
Optical Fiber Strength, Fatigue and Handleability After Aging in a Cable
,”
Proceedings of International Wire and Cable Symposium (43rd)
, Atlanta, GA, Nov. 14–17, pp.
728
735
.
16.
Glaesemann
,
G. S.
,
1992
, “
The Mechanical Behavior of Large Flaws in Optical Fiber and Their Role in Reliability Predictions
,”
Proceedings of International Wire and Cable Symposium
, Vol.
41
, Reno, NV, Nov. 16–19, pp.
689
704
.
17.
Volotinen
,
T. T.
, and
Gebizlioglu
,
O. S.
,
1996
, “
Mechanical Behavior of Coated Fused Silica Optical Fibers Aged at Elevated Temperature in Air and Filling Compound
,”
SPIE
Paper No. 2611.10.1117/12.230096
18.
Shah
,
J. M.
,
2018
, “
Characterizing Contamination to Expand ASHRAE Envelope in Airside Economization and Thermal and Reliability in Immersion Cooling of Data Centers
,”
Ph.D. dissertation
,
The University of Texas at Arlington
,
Arlington, TX
.