Experimental and Numerical Investigation of Single-Phase Liquid Cooling for Heterogeneous Integration Multichip Module

The world is being inundated with an overwhelming amount of digital data from numerous sources, including social media, blog posts, trading, artificial intelligence, quantum computing, and the Internet of Things. To accommodate and manage this overwhelming amount of data, data centers are required to capture, route, store, evaluate, and retrieve this massive amount of information, resulting in a rise in the energy they consume. An estimated 1.31% of the world's yearly energy consumption is utilized by data centers, with the cooling infrastructure responsible for up to half of that in legacy data centers [1–3]. Enhancing the energy efficiency of power-hungry technologies in data centers is therefore becoming more crucial. In addition, the development of chips along with additional integrated components, together with the expanding need for data processing, will increase the power density of chips and heat dissipation.

Air-cooling solutions, which are the most popular cooling systems in data centers, have several limitations, including poor heat transfer coefficients, inefficient energy use, and high levels of noise [4–6]. Because of this, alternate cooling methods, including single-phase liquid cooling and two-phase cooling, have become more popular. According to research studies [7–10], the challenge of cooling high-heat flux devices can be accomplished by adopting single-phase on-chip liquid cooling systems, as they are efficient, practical, and reliable.

In the area of single-phase liquid-cooled electronics, microchannel heat sinks emerge as a reliable option. Heat sinks are well-established for their efficient cooling solutions for microprocessor devices. Tuckerman and Pease's pioneering study [7] established the groundwork for later studies that have thoroughly studied microchannel liquid cooling, including classic microchannel heat sinks [11], manifold microchannel heat sinks [12–14], and spray and jet cooling approaches [15]. Iyengar [16] analyzed the hydraulic and thermal characteristics of a 1 cm² silicon chip that dissipated 400 W of heat using a direct liquid multijet impingement approach. Ansari [17] developed a hybrid heat sink with pin-fins and microchannels to cool microprocessors with uneven power dispersions. Compared with traditional heat sinks, the suggested heat sink was more effective in improving thermal performance. Wadsworth [18] used a small rectangular slot to guide a two-dimensional jet of dielectric fluid into a channel that is contained between the chip surface and nozzle plate to create a correlation for the heat transfer coefficient that depends on jet width, heater length, flow velocity, and fluid properties. Hadad [19] examined an impingement microchannel cold plate's thermohydraulic performance. Their study showed that, based on where the impinging inlet port is located, impingement cold plates, as opposed to parallel flow cold plates, provide the flexibility to control the maximum chip temperature's location.

Advanced chips typically have several cores integrated into them to improve performance, which can lead to temperature nonuniformity. Without an efficient thermal management system, the package can encounter large temperature gradients, which impose more thermal stress on the chip. Bar-Cohen [20] covered liquid and air-cooling techniques for the thermal management of multichip modules (MCMs). The researcher investigated different approaches for enhancing heat transfer and lowering thermal resistance in MCMs, including the utilization of thermally conductive adhesives and the design of innovative heat sinks. Song [21] investigated the use of case-embedded liquid cooling methods via microchannels inserted into the substrate of the electronic package for high-heat flux multichip arrangements. Zhang [22] developed a novel integrated cooling method for multichip modules by using a three-dimensional (3D)-printed design with embedded microchannels. By using this approach, lowers the operating temperature of high-power electronic devices and permits cooling elements to be replaced and reconfigured, allowing them to be customized for specific sources of heat and thermal demands. Ramakrishnan [23] experimentally and numerically studied using warm DI water to cool MCMs to test the performance of two heat sink designs: one with parallel channels and the other with concentrated channels. According to the findings, the cold plate with a concentrated microchannel design had superior thermohydraulic performance. Additionally, it was discovered that designing cold plates specifically for a given package is essential for achieving the best cooling results. Manaserh [24] designed a 3D-printed liquid-cooled heat sink with guiding vanes that target hotspots in high-heat flux chips using a multi-objective optimization technique. According to the study, which considers thermal and hydraulic performance parameters, the enhanced heat sink design was able to reduce hotspot temperatures by 32% when compared with the baseline design.

In this study, we introduce an innovative MCM experimental setup that was designed and built to assess the state-of-the-art of a commercial cold plate that can be used for applications involving uniform and nonuniform heat fluxes. The novelty of this work lies in its experimental design, as it can be customized to meet the needs of different sources of heat and thermal demands, and in its ability to systematically evaluate alternative MCM heat sink configurations by changing numerous variables and using one heat sink for cooling multiple chips. This involves adjusting the spacing and height of the heaters, as well as the heat flux distribution, both uniform and nonuniform. A single heater test was performed as a functionality test for the experimental setup. Another test was carried out to investigate the effect of varying the coolant flowrate on the heaters' temperature and thermal resistance. A numerical model was developed to validate the experimental results, and this numerical model will be used to perform parametric studies on the microchannel design of the heat sink in future studies.

An experimental rig was designed and built to replicate a real-life scenario of heterogeneous packaging of multichip modules (MCM) in high-heat flux applications, where chips and other integrated components of various heights from different manufacturers are placed closely together. This experimental rig, as seen in Figs. 1(a) and 1(b), consists of four ceramic heaters, each of which has a surface area of 1.2 cm × 1.2 cm and a thickness of 0.25 cm. Each heater has a built-in K-type thermocouple near its surface to measure its temperature, and each heater can deliver a total power of 200 W with a maximum temperature limit of 400 °C. To provide precise movements in the x-, y-, and z-directions, each heater was mounted on an XYZ linear stage. The size of the linear stage platform is 6 cm × 6 cm, and it has three knobs to control the distance in each direction. The stage is made from brass, which is a good thermal conductor. To prevent any thermal interaction that would cause heat losses between the heater and the linear stage, an insulator made from Garolite G-11 that has a low thermal conductivity of 0.288 W/m K was placed between them. Additionally, that insulator will keep the linear stage at an appropriate temperature when the heaters are turned on, ensuring that all the heat will be transferred from the heaters to the heat sink attached to them. A commercial pin-fin heat sink, as seen in Fig. 1(c), was used to cool the four chips, and a thermal grease thermal interface material (TIM) was applied in between.

Four separate power supplies were used, and one was connected to each heater. A liquid-to-air coolant distribution unit with a cooling capacity of 2 kW was used to control the supply fluid temperature. An external pump was used to pump DI through the loop, and the coolant flowrate was varied and controlled through an external power source connected to the pump. An ultrasonic flowmeter was used to measure the coolant flowrate, and two KEYENCE GP-M001T were installed at the inlet and outlet sides to measure the pressure drop across the loop. Additionally, two T-type thermocouples were used to measure the coolant temperature at the inlet and outlet. The experimental data was gathered using a National Instruments data acquisition system (DAQ) and recorded using LABVIEW. Table 1 provides the experimental equipment's specifications.

The heat sink tested in the present study, shown in Fig. 1(c), is composed of two parts: a plastic manifold with a single inlet and outlet and a copper base with an array of 729 square pin fins. A Keyence VK-X1050 laser scanning microscope, a noncontact 3D surface profiler that employs laser scanning to produce high-quality pictures at high magnifications up to 100×, was used to measure the heat sink's exact geometrical dimensions. Figure 2 depicts the microscopic images taken of the fins of the heat sink as well as their heights. Table 2 provides the heat sink's dimensions.

6Sigma ET was used in each simulation run for this investigation. On a staggered grid, the commercial software program discretizes and solves the system of governing equations, including the continuity, Navier–Stokes, and energy equations, employing the finite volume method. This technique is a good option for creating precise models of heat sinks for electronics since it was created expressly for modeling electronic cooling systems.

The following fundamental assumptions were used for solving the numerical models in this study:

1. The flow is 3D, steady, laminar, and incompressible.

2. No heat transfer through the surroundings, and the chips are insulated on all sides except the top, where heat transfers to the heat sink.

3. Gravity and wall roughness are negligible.

4. The effects of viscous and radiative heat transfer are insignificant.

The computational domain used to conduct this investigation, which consists of the copper plate, four chips, TIM, and manifold, is depicted in Fig. 3. These components' geometries were modeled using 6SigmaET. Establishing the proper grid is essential for guaranteeing the CFD model's correctness and dependability. In this investigation, the computational domain was covered by a structured grid with hexahedral cells since it is well-known that this type of grid generation produces grids of excellent quality. The proper boundary conditions were assigned, and the grid was concentrated more in areas with high gradients.

During the simulation, a number of boundary conditions were used, including adiabatic and no-slip surrounding wall surfaces as well as a constant inlet flowrate. The bottom surface was an exception, as the four chips continuously produced heat. Constant pressure was assigned at the outlet. The TIM used to conduct the simulation has the same thermal characteristics as the one used in the experiment, which is Kryonaut TIM. DI water, at an inlet temperature of 32 °C, served as the coolant. Table 3 lists the thermo-physical characteristics of the liquid and solid phases used in the simulation.

A grid sensitivity analysis was carried out to find the optimum number of grids to guarantee correctness and consistency in the CFD model. According to the study, a grid number greater than 21 × 10⁶ produced only minor differences in the heat sink pressure drop and the maximum temperature for the four heaters. Figure 4 demonstrates that increasing the number of grids from 21 to 31 × 10⁶ causes a little variation in the heat sink pressure drop and heater temperature of 0.1% and 0.4%, respectively. Therefore, the number of grids chosen was 22 × 10⁶, and the smallest cell sizes were 12, 7, and 6.2 × 10⁻⁵ in the x-, y-, and z-directions. The flow was in the laminar region for the targeted flow rates from 1 to 2 LPM where Reynolds number ranges from 720 to 1230.

where $qremoval$ is the amount of heat picked up by the coolant, m_liq is the liquid mass flowrate, and $Cpliq$ is the liquid specific heat.

The key objective is to verify the CFD modeling's accuracy so that it can be used confidently for next-level investigations and that the outcomes will be reliable for future studies. One of the experiments was used to validate the numerical model in terms of the heaters' temperature and the heat sink's pressure drop. The experiment was conducted at these boundary conditions: input power of 136 W/heater, fluid inlet temperature of 32 °C, fluid flowrate of 1 LPM, and the distance between each heater was 1 mm, and they were all at the same height. Figure 6 compares the experimental and numerical data. The maximum discrepancy in the heaters' temperature and heat sink pressure drop between the experimental and numerical results was less than 1.4% (less than 1 °C) and 4.2%, respectively.

Before running any high-power tests, some preliminary tests were conducted to test the functionality of the experimental setup apparatus. The heaters' thermocouples were tested at room temperature, and the readings were found to be consistent, as seen in Fig. 7(a). Next, another experiment was conducted by turning on one heater at the specified boundary conditions: the total input power was 163 W, the coolant flowrate was 1 LPM, and the coolant inlet temperature was 28 °C. At steady-state, the heater temperature was approximately 46.7 °C, as seen in Fig. 7(b), and using Eq. (6), the calculated heat removal by the coolant was 160 W, which indicates the amount of heat lost in the system, which was less than 2% compared to the input power from the power supply.

The novelty of this experimental setup is to test different heat sink designs for MCM by varying multiple parameters like coolant flowrate, supply coolant temperature, uniform and nonuniform heat fluxes, and different spacings and heights between the heaters. However, only one experiment was carried out; it was performed by placing the heaters 1 mm apart, and they were all at the same height. Uniform heat flux for all the heaters, the total input power was 544.3 W (∼94.5 W/cm² per heater). The supply fluid temperature was 32 °C, and the coolant flowrate varied from 1 to 2 LPM in steps of 0.25 LPM. Figure 8 shows the heaters' case temperatures after reaching a steady-state as the coolant flowrate varied. Increasing the flowrate has a big effect on the case temperatures because of the increase in the convective heat transfer coefficient. However, at a higher flowrate, the heaters' temperatures were almost the same and more uniform because the flow became more uniform in the heat sink channels, and a higher fluid velocity was observed in the channels with more fluid penetration in the heat sink microchannels, as can be seen from the streamlines and temperature contours for the four heaters in Figs. 9(a) and 9(b). In Fig. 9(b) it can be seen that there is a nonuniformity in the heaters' temperatures at 1 LPM, and when the flowrate increased to 2 LPM the temperatures were more uniform.

The total thermal resistance was calculated for both experiment and numerical data at different flow rates by using Eq. (5) and plotted in Fig. 10. The thermal resistance decreased by 29% because of the decrease in the liquid caloric resistance and the increase in the heat transfer coefficient. Going beyond a flowrate of 2 LPM will result in a minimal effect on thermal resistance, and any further enhancement in thermal performance becomes less noticeable.

The pressure drop across the system was measured by measuring the pressure at the inlet and outlet using KEYENCE GP-M001T pressure sensors capable of measuring in the range of –100 to 100 kPa. Before attaching the heat sink to the loop and connecting the inlet and outlet to it, the pressure drop of the system was obtained at different flow rates by looping the inlet and outlet sides together. After that, the heat sink was connected to the inlet and outlet ports, and pressure readings were taken during the experiments. By subtracting the pressure drop measurements with the heat sink attached from the pressure drop of the looping system, only the pressure drop of the heat sink was computed (Fig. 11).

The development of MCM packaging will allow cold plates of the next generation to manage several hot spots together. Special cold plate configurations are needed for each type of packaging in order to effectively remove heat from the chips. In this work, a novel experimental setup was built to simulate a real-life scenario of MCM by using four ceramic heaters of the same size. The experimental setup was used to characterize a commercially available heat sink and observe the heaters' case temperature at constant heat flux and different flow rates. Furthermore, a numerical model was created to support the findings of the experiment. The flowrate has a big impact on thermal performance; increasing the flowrate decreased the total thermal resistance by 29% at a flowrate of 2 LPM and a pressure drop of 4.9 kPa, and at higher flow rates, the temperatures of the heaters are almost uniform and equal. However, increasing the flowrate will increase the pressure drop, which will cause an increase in the required pumping power.

• NSF IUCRC (Award No. IIIP-2209776; Funder ID: 10.13039/100000001).

• MRI (Award No. CNS1040666; Funder ID: 10.13039/100000001).