Fully immersion of servers in electrically nonconductive (dielectric) fluid has recently become a promising technique for minimizing cooling energy consumption in data centers. The improved thermal properties of these dielectric fluids facilitate considerable savings in both upfront and operating cost over traditional air-cooling. This technology provides an opportunity for accommodating increased power densities. It also minimizes the common operational issues of air cooling technique like overheating and temperature swing in the system, fan failures, dust, air quality, and corrosion. This paper presents various data about the thermal performance of a fully single-phase dielectric fluid immersed server over wide temperature ranges (environment temperatures) from 25°C to 55°C for prolonged periods in an environmental chamber. This work explores the effects of high temperatures on the performance of a server and other components like pump, along with potential issues associated with extreme climatic conditions. The experimental data serves as a means to determine failure criteria for the server and pump by subjecting the system to accelerated thermal aging conditions i.e. around 55°C, consequently simulating the most extreme environmental condition that the server may encounter. Connector seals are inspected for expected degradation upon temperature cycling typically at such extreme conditions. Throttling limit for the server and pump power draw for different temperatures was determined to assess pump performance. Determining the relations between component behavior and operating temperature provides an accurate measure of lifetime of a server. The scope of this paper can be expanded by reviewing the effects of low temperatures on server and component performance. Changes to various performance parameters like power draw of pump and server at the higher and the lower operating temperatures and an understanding of issues like condensation can be used to quantify upper and lower limits for pump and server performance.

The percentage of the energy used by data centers for cooling their equipment has been on the rise. With that, there has been a necessity for exploring new and more efficient methods like airside economization, both from an engineering as well as business point of view, to contain this energy demand. Air cooling especially, free air cooling has always been the first choice for IT companies to cool their equipment. But, it has its downside as well. As per ASHRAE standard (2009b), the air which is entering the data center should be continuously filtered with MERV 11 or preferably MERV 13 filters and the air which is inside the data center should be clean as per ISO class 8. The objective of this study is to design a model data center and simulate the flow path with the help of 6sigma room analysis software. A high-density data center was modelled for both hot aisle and cold aisle containment configurations. The particles taken into consideration for modelling were spherical in shape and of diameters 0.05, 0.1 and 1 micron. The physical properties of the submicron particles have been assumed to be same as that of air. For heavier particles of 1 micron in size, the properties of dense carbon particle are chosen for simulating particulate contamination in a data center. The Computer Room Air Conditioning unit is modelled as the source for the particulate contaminants which represents contaminants entering along with free air through an air-side economizer. The data obtained from this analysis can be helpful in predicting which type of particles will be deposited at what location based on its distance from the source and weight of the particles. This can further help in reinforcing the regions with a potential to fail under particulate contamination.

The rapid growth in cloud computing, the Internet of Things (IoT), and data processing via Machine Learning (ML), have greatly increased our need for computing resources. Given this rapid growth, it is expected that data centers will consume more and more of our global energy supply. Improving their energy efficiency is therefore crucial. One of the biggest sources of energy consumption is the energy required to cool the data centers, and ensure that the servers stay within their intended operating temperature range. Indeed, about 40% of a data center’s total power consumption is for air conditioning[1]. Here, we study how the server air inlet and outlet, as well as the CPU, temperatures depend upon server loads typical of real Internet Protocol (IP) traces. The trace data used here are from Google clusters and include the times, job and task ID, as well as the number and usage of CPU cores. The resulting IT loads are distributed using standard load-balancing methods such as Round Robin (RR) and the CPU utilization method. Experiments are conducted in the Data Center Laboratory (DCL) at the Georgia Institute of Technology to monitor the server outlet air temperature, as well as real-time CPU temperatures for servers at different heights within the rack. Server temperatures were measured by on-line temperature monitoring with Xbee, Raspberry PI, Arduino, and hot-wire anemometers. Given that the temperature response varies with server position, in part due to spatial variations in the cooling airflow over the rack inlet and the server fan speeds, a new load-balancing approach that accounts for spatially varying temperature response within a rack is tested and validated in this paper.

This work experimentally studies the impact of facility cooling failure of a direct liquid cooling (DLC) system on the IT equipment (ITE). The facility side of a DLC system removes the heat from a secondary loop — in direct contact with the ITE — and discard it in a chiller loop or ambient. The CPU utilization and coolant set point temperature (SPT) are varied to understand the effect of failure under different operating conditions. The ITE response is studied in terms of chip temperature and power, and fan speed. It was found that failure of the facility cooling system is not hazardous to the IT operation. The rate of change in temperature after failure is low and is sufficient to turn the ITE off safely. This is attributed to the surrounding air in the data center and the thermal mass of the cooling system.

Given the vital rule of data center availability and since the inlet temperature of the IT equipment increase rapidly until reaching a certain threshold value after which IT starts throttling or shut down because of overheat during cooling system failure. Hence, it is especially important to understand failures and their effects. This study presented experimental investigation and analysis of a facility-level cooling system failure scenario in which chilled water interruption introduced to the data center. Quantitative instrumentation tools including wireless technology such as wireless temperature and pressure sensors were used to measure the discrete air inlet temperature and pressure differential though cold aisle enclosure, respectively. In addition, Intelligent Platform Management Interface (IPMI) and cooling system data during failure/recovery were reported. Furthermore, the IT equipment performance and response for opened and contained environments were simulated and compared. Finally, an experiment based analysis of the Ride Through Time (RTT) of servers during chilled water interruption of the cooling infrastructure presented as well. The results showed that for all three classes of servers tested during the cooling failure, CAC helped keep the server’s cooler for longer. The containment provided a barrier between the hot and cold air streams and caused slight negative pressure to build up, which allowed the servers to pull cold air from the underfloor plenum. In addition, the results show that the effect of CAC in containment solutions on the IT equipment performance and response could vary and depend on the server’s airflow, generation and hence types of servers deployed in cold aisle enclosure. Moreover, it was shown that when compared to the discrete sensors, the IPMI inlet temperature sensors underestimate the Ride Through Time (RTT) by 42% and 12% for the CAC and opened cases, respectively.

The objective of this work is to introduce the application of an artificial neural network (ANN) to assist in the evaporative cooling in data centers. To achieve this task, we employ the neural network algorithms to predict weather conditions outside the data center for direct evaporative cooling (DEC) operations. The predictive analysis helps optimize the cooling control strategy for maximizing the usage of evaporative cooling thereby improving the efficiency of the overall data center cooling system. A typical artificial neural network architecture is dynamic in nature and can perform adaptive learning in minimal computation time. A neural network model of a data center was created using operational historical data collected from a data center cooling control system. The neural network model allows the control of the modular data center (MDC) cooling at optimum configuration in two ways. First way is that the network model minimizes time delay for switching the cooling from one mode to the other. Second way, it improves the reaction behavior of the cooling equipment if an unexpected ambient condition change should come. The data center in consideration is a test bed modular data center that comprises of information Technology (IT) racks, Direct Evaporative cooling (DEC) and Indirect Evaporative Cooling (IEC) modules; the DEC/IEC are used together or in alternative mode to cool the data center room. The facility essentially utilizes outside ambient temperature and humidity conditions that are further conditioned by the DEC and IEC to cool the electronics, a concept know as air-side economization. Various parameters are related to the cooling system operation such as outside air temperature, IT heat load, cold aisle temperature, cold aisle humidity etc. are considered. Some of these parameters are fed into the artificial neural network as inputs and some are set as targets to train the neural network system. After the training the process is completed, certain bucket of data is tested and further used to validate the outputs for various other weather conditions. To make sure the analysis represents real world scenario, the operational data used are from real time data logged on the MDC cooling control unit. Overall, the neural network model is trained and is used to successfully predict the weather conditions and cooling control parameters. The prediction models have been demonstrated for the outputs that are static in nature (Levenberg Marquardt method) as well as the outputs that are dynamic in nature i.e., step-ahead & multistep ahead techniques.

Direct evaporative cooling (DEC) is widely used in the data center cooling units to maintain the air condition inside the data centers. Often, the flow rate of the water over the wet cooling media in this DEC process is frequently varied to maintain the air condition inside the data centers based on changing weather conditions. Though the adopted method helps to control the air temperature and relative humidity, the scale formation occurs on the surface of wet cooling media due to the frequent variation of the flow rate and deposition of minerals present in the water at low flow rate values, which increases the total weight of the wet cooling media and it can lead to a wet cooling media collapse. In this paper an alternative and simplified method to control the air condition is presented. A vertically split wet cooling media is designed and tested in a commercial CFD tool to analyze the temperature and relative humidity parameters of the inlet and outlet air to the wet cooling media, in this approach the sections of the media can either be completely wet or completely dry which can potentially avoid the scale formation on the surface of the wet cooling media. In addition to the temperature and relative humidity parameters against the air flow rates, the pressure drop and cooling efficiency values for varied air flow rates are studied. The vertically split wet cooling media configurations are achieved by sectioning the media in to equal and unequal sections. In the equal configuration, media has been tested for 0%, 50% and 100% wetting conditions, and in the unequal configuration, media has been tested for 0%, 33%, 66% and 100% wetting conditions. The test results are used to emphasis the advantage of this staged wetting method and gives a possible solution to the scale formation problem on the wet cooling media during the direct evaporative cooling process in the data center.

As global demand for data centers grows, so does the size and load placed on data centers, leading constraints on power and space available to the operator. Cooling power consumption is a major part of the data center energy usage. Liquid cooling technology has emerged as a viable solution in the process of optimization of the energy consumed per performance unit. In this data center rack level evaluation, 2OU (Open U) hybrid (liquid+air) cooled web servers are tested to observe the effects of warm water cooling on the server component temperatures, IT power and cooling power. The study discusses the importance of variable speed pumping in a centralized coolant configuration system. The cooling setup includes a mini rack capable of housing up to eleven hybrid cooled web servers and two heat exchangers that exhaust the heat dissipated from the servers to the environment (the test rig data center room). The centralized configuration has two redundant pumps placed in series with heat exchanger at the rack. Each server is equipped with two passive (i.e. no active pump) cold plates for cooling the CPUs while rests of the components are air cooled. Synthetic stress load has been generated on each server using stress-testing tools. Pumps in the servers are separately powered using an external power supply. The pump speed is proportional to the voltage across the armature [1]. The pump rpm has been recorded with input voltages ranging from 11V to 17V. The servers are tested for higher inlet temperatures ranging from 25°C to 45°C which falls within the ASHRAE liquid cooled envelope W4 [2]. Variable pumping has been achieved by using different input voltages at respective inlet temperatures.

There are various designs for segregating hot and cold air in data centers such as cold aisle containment (CAC), hot aisle containment (HAC), and chimney exhaust rack. These containment systems have different characteristics and impose various conditions on the information technology equipment (ITE). One common issue in HAC systems is the pressure buildup inside the HAC (known as backpressure). Backpressure also can be present in CAC systems in case of airflow imbalances. Hot air recirculation, limited cooling airflow rate in servers, and reversed flow through ITE with weaker fan systems (e.g. network switches) are some known consequences of backpressure. Currently there is a lack of experimental data on the interdependency between overall performance of ITE and its internal design when a backpressure is imposed on ITE. In this paper, three commercial 2-rack unit (RU) servers with different internal designs from various generations and performance levels are tested and analyzed under various environmental conditions. Smoke tests and thermal imaging are implemented to study the airflow patterns inside the tested equipment. In addition, the impact leak of hot air into ITE on the fan speed and the power consumption of ITE is studied. Furthermore, the cause of the discrepancy between measured inlet temperatures by internal intelligent platform management interface (IPMI) and external sensors is investigated. It is found that arrangement of fans, segregation of space upstream and downstream of fans, leakage paths, location of sensors of baseboard management controller (BMC) and presence of backpressure can have a significant impact on ITE power and cooling efficiency.

Data centers concern not just energy usage, but also other important overall considerations such as the actual computational work, the energy efficiency and the performance of servers. The server, as one of the key ingredients of a data center, plays an increasingly crucial role in contributing to the overall energy use, especially in cases where the efficiency of the infrastructure has been optimized. Baidu has been exploring the sweet zone between power and performance in efficient rack server design and deployment for their self-built data center energy efficiency optimization from all the aspects. Recent deployment of rack server with distributed backup battery (Li-ion) subsystem (BBS) is one typical example to demonstrate their advanced rack server design for energy efficiency. Compared with lead acid battery based traditional UPS in data center, distributed BBS design in Baidu rack server has an advantage in lower power loss, data center power delivery and topology simplification, data center real estate saving, scalable deployment on demand without overprovision and so on, which overall contributes to a total cost ownership (TCO) reduction on both cap-ex (i.e., power infrastructure investment) and op-ex (i.e., electricity bill). This paper introduces overall architecture and design of Baidu rack server with distributed BBS. Furthermore details energy efficiency design methodology of power peak draw trimming based on workload power characterization; Also the related lab data collection, experiments result, ongoing work and future plan are summarized in the end. This paper also recaps TCO saving points benefiting from distributed BBS design into rack server system.

Continuously increasing demand for higher compute performance is pushing for improved advanced thermal solutions. In high performance computing (HPC) area, most of the end users deploy some sort of direct or indirect liquid cooling thermal solutions. But for the users who have air cooled data centers and air cooled thermal solutions are challenged to cool next generation higher Thermal Design Power (TDP) processors in the same platform form factor without changing environmental boundary conditions. This paper presents several different advanced air cooled technologies developed to cool high TDP processors in the same form factor and within the same boundary conditions of current generation processor. Comparison of thermal performance using different cooling technologies such as Liquid Assist Air Cooling (LAAC) and Loop Heat Pipe (LHP) are presented in this paper. A case study of Intel’s Knights Landing (KNL) processor is presented to show case the increase in compute performance due to different advanced air cooling technologies.

In raised floor data centers, tiles with high open area ratio or complex understructure are used to fulfill the demand of today’s high-density computing. Using more open tiles reduces pressure drop across the raised floor with the potential advantages of increased airflow and lower noise. However, it introduces the disadvantage of increased non-uniformity of airflow distribution. In addition, there are various tile designs available on the market with different opening shapes or understructures. Furthermore, a physical separation of cold and hot aisles (containment) has been introduced to minimize the mixing of cold and hot air. In this study, three types of floor tiles with different open area, opening geometry, and understructure are considered. Experimentally validated detail models of tiles were implemented in CFD simulations to address the impact of tile design on the cooling of IT equipment in both open and enclosed aisle configurations. Also, impacts of under-cabinet leakage on the IT equipment inlet temperature in the provisioned and under-provisioned scenarios are studied. Finally, a predictive equation for the critical under-provisioning point that can lead to a no-flow condition in IT equipment with weaker airflow systems is presented.

In this paper, the impact of direct liquid cooling (DLC) system failure on the IT equipment is studied experimentally. The main factors that are anticipated to affect the IT equipment response during failure are the CPU utilization, coolant set point temperature (SPT) and the server type. These factors are varied experimentally and the IT equipment response is studied in terms of chip temperature and power, CPU utilization and total server power. It was found that failure of the cooling system is hazardous and can lead to data center shutdown in less than a minute. Additionally, the CPU frequency throttling mechanism was found to be vital to understand the change in chip temperature, power, and utilization. Other mechanisms associated with high temperatures were also observed such as the leakage power and the fans speed change. Finally, possible remedies are proposed to reduce the probability and the consequences of the cooling system failure.

The operation of today’s data centers increasingly relies on environmental data collection and analysis to operate the cooling infrastructure as efficiently as possible and to maintain the reliability of IT equipment. This in turn emphasizes the importance of the quality of the data collected and their relevance to the overall operation of the data center. This study presents an experimentally based analysis and comparison between two different approaches for environmental data collection; one using a discrete sensor network, and another using available data from installed IT equipment through their Intelligent Platform Management Interface (IPMI). The comparison considers the quality and relevance of the data collected and investigates their effect on key performance and operational metrics. The results have shown the large variation of server inlet temperatures provided by the IPMI interface. On the other hand, the discrete sensor measurements showed much more reliable results where the server inlet temperatures had minimal variation inside the cold aisle. These results highlight the potential difficulty in using IPMI inlet temperature data to evaluate the thermal environment inside the contained cold aisle. The study also focuses on how industry common methods for cooling efficiency management and control can be affected by the data collection approach. Results have shown that using preheated IPMI inlet temperature data can lead to unnecessarily lower cooling set points, which in turn minimizes the potential cooling energy savings. It was shown in one case that using discrete sensor data for control provides 20% more energy savings than using IPMI inlet temperature data.

With the growth of Internet-based services, data centers must deal with increasing data traffic. Load-balancing technologies can help data centers process data effectively and stably; however, the current load-balancing methods do not take into account the heat generated by servers. Excessive heat can increase the failure rate of IT devices and the energy consumption of air conditioning systems, both of which lead to higher data center maintenance costs. This paper aims to simultaneously increase the coefficient of performance (COP) of the data center’s air-conditioning equipment and decrease the semiconductor-based equipment failure rate. To do so — and, consequently, reduce the operation and maintenance costs — we propose a novel request distribution system based on server-temperature and evaluate the proposed system by creating a thermal model of a data center. As a result, it is suggested that using the proposed load-balancing method the semiconductor failure rate can be reduced by 32 % when compared with the common round-robin distribution method, and by 19 % when compared with a load-balancing method based on CPU utilization. Moreover, the COP of the air-conditioning equipment obtained with the proposed method is recognized to be higher than those obtained with either the round-robin or the CPU-utilization-based methods.

Modern day data centers are operated at high power for increased power density, maintenance, and cooling which covers almost 2 percent (70 billion kilowatt-hours) of the total energy consumption in the US. IT components and cooling system occupy the major portion of this energy consumption. Although data centers are designed to perform efficiently, cooling the high-density components is still a challenge. So, alternative methods to improve the cooling efficiency has become the drive to reduce the cooling cost. As liquid cooling is more efficient for high specific heat capacity, density, and thermal conductivity, hybrid cooling can offer the advantage of liquid cooling of high heat generating components in the traditional air-cooled servers. In this experiment, a 1U server is equipped with cold plate to cool the CPUs while the rest of the components are cooled by fans. In this study, predictive fan and pump failure analysis are performed which also helps to explore the options for redundancy and to reduce the cooling cost by improving cooling efficiency. Redundancy requires the knowledge of planned and unplanned system failures. As the main heat generating components are cooled by liquid, warm water cooling can be employed to observe the effects of raised inlet conditions in a hybrid cooled server with failure scenarios. The ASHRAE guidance class W4 for liquid cooling is chosen for our experiment to operate in a range from 25°C – 45°C. The experiments are conducted separately for the pump and fan failure scenarios. Computational load of idle, 10%, 30%, 50%, 70% and 98% are applied while powering only one pump and the miniature dry cooler fans are controlled externally to maintain constant inlet temperature of the coolant. As the rest of components such as DIMMs & PCH are cooled by air, maximum utilization for memory is applied while reducing the number fans in each case for fan failure scenario. The components temperatures and power consumption are recorded in each case for performance analysis.

Over the past few years, there has been an ever increasing rise in energy consumption by IT equipment in Data Centers. Thus, the need to minimize the environmental impact of Data Centers by optimizing energy consumption and material use is increasing. In 2011, the Open Compute Project was started which was aimed at sharing specifications and best practices with the community for highly energy efficient and economical data centers. The first Open Compute Server was the ‘ Freedom’ Server. It was a vanity free design and was completely custom designed using minimum number of components and was deployed in a data center in Prineville, Oregon. Within the first few months of operation, considerable amount of energy and cost savings were observed. Since then, progressive generations of Open Compute servers have been introduced. Initially, the servers used for compute purposes mainly had a 2 socket architecture. In 2015, the Yosemite Open Compute Server was introduced which was suited for higher compute capacity. Yosemite has a system on a chip architecture having four CPUs per sled providing a significant improvement in performance per watt over the previous generations. This study mainly focuses on air flow optimization in Yosemite platform to improve its overall cooling performance. Commercially available CFD tools have made it possible to do the thermal modeling of these servers and predict their efficiency. A detailed server model is generated using a CFD tool and its optimization has been done to improve the air flow characteristics in the server. Thermal model of the improved design is compared to the existing design to show the impact of air flow optimization on flow rates and flow speeds which in turn affects CPU die temperatures and cooling power consumption and thus, impacting the overall cooling performance of the Yosemite platform. Emphasis is given on effective utilization of fans in the server as compared to the original design and improving air flow characteristics inside the server via improved ducting.

The rapid growth of the global network infrastructure has resulted in a sharp increase in the number and size of data center facilities. Total data center power consumption now represents a significant fraction of global electricity production. To conserve natural resources, and to satisfy the cooling demands of compact, powerful electronics, thermal management strategies with high heat transfer coefficients must be employed. Two-phase liquid immersion cooling is one such strategy that has been gaining momentum in commercial cooling applications over recent years. The work discussed in this paper provides information on two different flow boiling investigations performed on vertically oriented surfaces in a small form factor server model. Two different types of surfaces — bare silicon, and silicon surfaces attached with microfinned heat sinks were tested in this study. Novec 649 dielectric fluid was used as the primary working fluid. The first investigation compares the thermal performance of parallel and impinging flow distribution systems, for different subcooling and flow rate conditions. The second investigation is on nucleation suppression in flow boiling for the parallel and impinging flow distribution systems. In this study, flow rates ranging from 0 ml/min to 1650 ml/min were tested and high-speed imaging was performed to capture the change in bubble characteristics. The resulting observations, including highest heat flux values supported without nucleation activity, are reported and discussed.

Data centers house a variety of compute, storage, network IT hardware where equipment reliability is of utmost importance. Heat generated by the IT equipment can substantially reduce its service life if Tjmax , maximum temperature that the microelectronic device tolerates to guarantee reliable operation, is exceeded. Hence, data center rooms are bound to maintain continuous conditioning of the cooling medium becoming large energy consumers. The objective of this work is to introduce and evaluate a new end-of-aisle cooling design which consists of three cooling configurations. The key objectives of close-coupled cooling are to enable a controlled cooling of the IT equipment, flexible as well as modular design, and containment of hot air exhaust from the cold air. The thermal performance of the proposed solution is evaluated using CFD modeling. A computational model of a small size data center room has been developed. Larger axial fans are selected and placed at rack-level which constitute the rack-fan wall design. The model consists of 10 electronic racks each dissipating a heat load of 8kw. The room is modeled to be hot aisle containment i.e. the hot air exhaust exiting for each row is contained and directed within a specific volume. Each rack has passive IT with no server fans and the servers are cooled by means of rack fan wall. The cold aisle is separated with hot aisle by means of banks of heat exchangers placed on the either sides of the aisle containment. Based on the placement of rack fans, the design is divided to three sub designs — case 1: passive heat exchangers with rack fan walls; case 2: active heat exchangers (HXs coupled with fans) with rack fan walls; case 3: active heat exchangers (hxs coupled with fans) with no rack fans. The cooling performance is calculated based on the thermal and flow parameters obtained for all three configurations. The computational data obtained has shown that the case 1 is used only for lower system resistance IT. However, case 2 and Case 3 can handle denser IT systems. Case 3 is the design that can consume lower fan energy as well as handle denser IT systems. The paper also discusses the cooling behavior of each type of design.

With the rapid growth of data centers worldwide and the global shift towards energy sustainability, deploying new cooling technologies has an utmost importance. Conventional cooling systems such as chilled water system, usually have high capital costs and relatively low energy efficiency, leading to a high PUE and TCO values. Indirect evaporative cooling is a promising technology, which offers air cooling with high efficiency, hygiene air quality, and lower total cost. This paper details the design of a proof-of-concept data center with indirect evaporative cooling, which will be eventually deployed at megawatt-scale Baidu datacenters. BIN data analysis and CFD simulation are performed to optimize the physical design and operating conditions. CFD analysis of the data center room is established to optimize rack placement, air flow management, and cold aisle hot aisle configuration. A comprehensive TCO analysis is established, which shows a total savings of 9% using IDEC technology compared to chilled water system for cooling. In addition, TCO analysis indicates small to negligible effect of air supply temperature. Hence, air supply to the cold aisle is set to 27 °C to improve cooling performance. Finally, ROI sensitivity analysis is performed to measure the sensitivity of ROI on power usage effectiveness of the IDEC unit.