High performance computing server racks are being engineered to contain significantly more processing capability within the same computer room footprint year after year. The processor density within a single rack is becoming high enough that traditional, inefficient air-cooling of servers is inadequate to sustain HPC workloads. Experiments that characterize the performance of a direct water-cooled server rack in an operating HPC facility are described in this paper. Performance of the rack is reported for a range of cooling water inlet temperatures, flow rates and workloads that include actual and worst-case synthetic benchmarks. Power and temperature measurements of all processors and memory components in the rack were made while extended benchmark tests were conducted throughout the range of cooling variables allowed within an operational HPC facility. Synthetic benchmark results were compared with those obtained on a single server of the same design that had been characterized thermodynamically. Neither actual nor synthetic benchmark performances were affected during the course of the experiments, varying less than 0.13 percent. Power consumption change in the rack was minimal for the entire excursion of coolant temperatures and flow rates. Establishing the characteristics of such a highly energy efficient server rack in situ is critical to determine how the technology might be integrated into an existing heterogeneous, hybrid cooled computing facility — i.e., a facility that includes some servers that are air cooled as well as some that are direct water cooled.

With the advent of big data and cloud computing solutions, enterprise demand for servers is increasing. There is especially high growth for Intel based x86 server platforms. Today’s datacenters are in constant pursuit of high performance/high availability computing solutions coupled with low power consumption and low heat generation and the ability to manage all of this through advanced telemetry data gathering. This paper showcases one such solution of an updated rack and server architecture that promises such improvements. The ability to manage server and data center power consumption and cooling more completely is critical in effectively managing datacenter costs and reducing the PUE in the data center. Traditional Intel based 1U and 2U form factor servers have existed in the data center for decades. These general purpose x86 server designs by the major OEM’s are, for all practical purposes, very similar in their power consumption and thermal output. Power supplies and thermal designs for server in the past have not been optimized for high efficiency. In addition, IT managers need to know more information about servers in order to optimize data center cooling and power use, an improved server/rack design needs to be built to take advantage of more efficient power supplies or PDU’s and more efficient means of cooling server compute resources than from traditional internal server fans. This is the constant pursuit of corporations looking at new ways to improving efficiency and gaining a competitive advantage. A new way to optimize power consumption and improve cooling is a complete redesign of the traditional server rack. Extracting internal server power supplies and server fans and centralizing these within the rack aims to achieve this goal. This type of design achieves an entirely new low power target by utilizing centralized, high efficiency PDU’s that power all servers within the rack. Cooling is improved by also utilizing large efficient rack based fans for airflow to all servers. Also, opening up the server design is to allow greater airflow across server components for improved cooling. This centralized power supply breaks through the traditional server power limits. Rack based PDU’s can adjust the power efficiency to a more optimum point. Combine this with the use of online + offline modes within one single power supply. Cold backup makes data center power to achieve optimal power efficiency. In addition, unifying the mechanical structure and thermal definitions within the rack solution for server cooling and PSU information allows IT to collect all server power and thermal information centrally for improved ease in analyzing and processing.

The heat dissipated by high performance IT equipment such as servers and switches in data centers is increasing rapidly, which makes the thermal management even more challenging. IT equipment is typically designed to operate at a rack inlet air temperature ranging between 10 °C and 35 °C. The newest published environmental standards for operating IT equipment proposed by ASHARE specify a long term recommended dry bulb IT air inlet temperature range as 18°C to 27°C. In terms of the short term specification, the largest allowable inlet temperature range to operate at is between 5°C and 45°C. Failure in maintaining these specifications will lead to significantly detrimental impacts to the performance and reliability of these electronic devices. Thus, understanding the cooling system is of paramount importance for the design and operation of data centers. In this paper, a hybrid cooling system is numerically modeled and investigated. The numerical modeling is conducted using a commercial computational fluid dynamics (CFD) code. The hybrid cooling strategy is specified by mounting the in row cooling units between the server racks to assist the raised floor air cooling. The effect of several input variables, including rack heat load and heat density, rack air flow rate, in row cooling unit operating cooling fluid flow rate and temperature, in row coil effectiveness, centralized cooling unit supply air flow rate, non-uniformity in rack heat load, and raised floor height are studied parametrically. Their detailed effects on the rack inlet air temperatures and the in row cooler performance are presented. The modeling results and corresponding analyses are used to develop general installation and operation guidance for the in row cooler strategy of a data center.

The integration of a simulation-based Artificial Neural Network (ANN) with a Genetic Algorithm (GA) has been explored as a real-time design tool for data center thermal management. The computation time for the ANN-GA approach is significantly smaller compared to a fully CFD-based optimization methodology for predicting data center operating conditions. However, difficulties remain when applying the ANN model for predicting operating conditions for configurations outside of the geometry used for the training set. One potential remedy is to partition the room layout into a finite number of characteristic zones, for which the ANN-GA model readily applies. Here, a multiple hot aisle/cold aisle data center configuration was analyzed using the commercial software FloTHERM. The CFD results are used to characterize the flow rates at the inter-zonal partitions. Based on specific reduced subsets of desired treatment quantities from the CFD results, such as CRAC and server rack air flow rates, the approach was applied for two different CRAC configurations and various levels of CRAC and server rack flow rates. Utilizing the compact inter-zonal boundary conditions, good agreement for the airflow and temperature distributions is achieved between predictions from the CFD computations for the entire room configuration and the reduced order zone-level model for different operating conditions and room layouts.

Energy consumption in data center has been drastically increasing in recent years. In data center, server racks are cooled down by air conditioning for the whole room in a roundabout way. This air cooling method is inefficient in cooling and it causes hotspot problem that IT equipments are not cooled down enough, but the room is overcooled. On the other hand, countermeasure against the heat of the IT equipments is also one of the big issues. We therefore proposed new liquid cooling systems which IT equipments themselves are cooled down and exhaust heat is not radiated into the server room. For our liquid cooling systems, three kinds of cooling methods have been developed simultaneously. Two of them are direct cooling methods that the cooling jacket is directly attached to heat source, or CPU in this case. Single-phase heat exchanger or two-phase heat exchanger is used as cooling jackets. The other is indirect cooling methods that the heat generated from CPU is transported to the outside of the chassis through flat heat pipes, and condensation sections of the heat pipes are cooled down by liquid. Verification tests have been conducted by use of real server racks equipped with these cooling techniques while pushing ahead with five R&D subjects which constitute our liquid cooling system, which single-phase heat exchanger, two-phase heat exchanger, high performance flat heat pipes, nanofluids technology, and plug-in connector. As a result, the energy saving effect of 50∼60% comparing with conventional air cooling system was provided in direct cooling technique with single-phase heat exchanger.

It is now common for data center managers to question the impact on server energy usage of two recent impact factors: (1) the rise in the data center inlet air temperature to servers per 2008 ASHRAE guidelines, and (2) the fan speed increase from the use of rack level heat exchangers such as Rear Door Heat Exchangers. To help acquire a deeper understanding of the relevant issues, a system floor thermal test was built on the IBM New York data center benchmark floor which consisted of a standard 19 ″ rack filled with 39 3.0 GHz 1U servers that dissipated between 10–17 kW depending on extent of server utilization. Fan speed, chip temperature metrics, and server power data was collected using product debug codes and server level programs. A simulated air heat load was installed right in front of this server rack to allow the manipulation of air inlet temperature into the servers from 20 °C to 32 °C. Two different rack level configurations were considered for the experiments: (i) a perforated front door and no door at the rear, and (ii) a perforated front door and a Rear Door Heat Exchanger at the rear. An exerciser program was used to vary the CPU utilization from Idle to 70% which represented a typical data center work load. Data was collected for 19 servers of the 39 servers (remaining were in use by Benchmark Lab) for the two rack configurations, for 4 inlet server air temperatures, and for two chip exerciser settings, i.e. 16 experiments. For the 70% exerciser setting (typical operation) and the base line rack configuration without rack level heat exchangers, the rise in server power for an increase in inlet air temperature was 5.2% for the 20 °C to 27 °C change and was 17% for the 20 °C to 31 °C change. For the 70% exerciser setting (typical), the increase in server power from the use of rack level heat exchangers (Rear Door Heat Exchanger) was less than 1.3% for all the conditions. Given the broad range of fan speed algorithms and cooling hardware in server products on the market and their change over each generation, significant further study will be required to characterize each category of systems for these conditions. However, the present study provides a template for quantifying server energy usage in a context that data center managers can understand and use.

Conducting experiments on real high-density computer servers can be an expensive and risky task due to the risks associated with unintended inlet temperatures that exceed the server’s red-line temperature limit. Presented herein is the development of the simulated chassis that mimic real computer servers. Briefly, twelve high-power simulated chassis were designed and built to accurately simulate the actual operating conditions of a real computer chassis in a data center. Each simulated chassis is designed to have approximately 300 Pa pressure drop at a flow rate of 600 cfm to represent a real IBM server chassis. Additionally, the simulated chassis are designed to match the thermal mass of a real server. Eight of the simulated chassis were designed to have constant speed fans and variable heating power while the remaining four chassis were designed to have variable speed fans and variable heating power. Further discussions about the design phase of the simulated chassis are the substantial part of this paper. Underlining the challenges and safety issues with high-power chassis, guidelines for designing and constructing a chassis that simulates the real environment of a typical data center are presented.

The patented [1] rear door heat exchanger mounted to the rear of IT equipment racks was announced in April, 2005 by IBM and has shown improvements in data center energy efficiency and reducing hot spots. It also allows data center operators to more easily implement some of the features of the newly approved ASHRAE data center recommended data center environmental guidelines [2]. This paper will describe several case studies of implementing the rear door heat exchanger in various data center layouts. The implementation of the water cooled rear door in these data centers will show the effects of various failure modes and how the new ASHRAE environmental temperature guidelines are still being met with the failure modes examined.

This paper describes the facility level design of future water cooled supercomputers with an emphasis on the energy efficiency opportunities related to the use of water cooled economizer based facility cooling in lieu of traditional refrigeration chiller architectures. A hybrid cooled 80kW server rack with both air and water cooled components is considered for the analyses. In this system, 90% of the rack heat load is rejected directly to pumped water lines via CPU cold plates and conduction cooling and the remaining 10% rack power is rejected to the room ambient air and is subsequently transferred to the water loop via air to liquid heat exchanger based devices. Air to water heat exchangers are not included in the server racks. The facilities cost for water cooling of high power computer racks has been analyzed for data centers situated in two different geographic locations. The study takes into account both the energy savings resulting from the loop design as well as the duration of time for which these savings can be realized at the selected locations. Test data for the water cooled CPU modules is used to project the maximum chip junction temperature for different chilled water temperatures which are in turn determined using the outdoor air temperatures for different times of the year. After calculating the 5 year energy consumption savings over the traditional plant design for each location, for the two different geographic locations in the US considered, it was found that the outdoor weather permitted the use of the economizer based configuration for 53–76% and 60–80% of the time for A and B locations, respectively, yielding per rack energy related savings of $6–21k and $17–26k, respectively, for the thermal conditions considered.

In 2005, IBM released a water cooled heat exchanger product that significantly enhanced data center cooling capability while also demonstrating substantial energy savings. In 2008, IBM released an enhanced water less solution to cool the electronic racks via a R410A refrigerant based vapor compression system, which is the focus of this paper. The paper provides a detailed description of device and coolant loop construction, the experimental thermal data collected, as well as a discussion of its’ cooling energy efficiency relative to both typical air cooled facilities and water cooled heat exchangers, respectively. A data center level case study was performed with experimental measurements collected and discussed herein. Significant energy savings were realized even when the heat exchanger devices were implemented on a small part of the data center. Based on the test data and the experimental data center study, the CRAC units based loops have a COP of 1.95, while the refrigerant refrigerant heat exchanger loop has a COP of 5.0.

For a drastic energy conservation in data centers (more than 30%), a new advanced concept of thermal management systems of integrated cooling network in server racks is proposed, and a series of developmental studies toward the realization of the concept have been conducted. The concept consists of the following technological items: - Plug-in thermal network cables, - Narrow channel heat exchangers (single-phase and two-phase), - Thin flat-type heat pipes, - Development of nano-fluids for heat transfer enhancement. In this concept, CPUs in server racks are directly cooled with the aid of advanced 1.5 mm thick flat-type heat pipes or 3 mm thick narrow channel heat exchangers, though CPUs in server racks, at present, are cooled down with massive heat sinks by forced air flow in the racks. In the case of heat pipes, condensation regions are cooled by either single-phase or two-phase narrow channel heat sink instead of air-cooled finned heat sink. These cooling units are accommodated in server racks. In addition, plug-in thermal network cables, in which a working fluid for narrow channel heat exchangers is running, are integrated among server racks, and thermal energy dissipated in server racks are inclusively merged and managed by the thermal network cables. The working fluid heated up by the dissipated heat is recuperated in an outdoor cooling system. An attempt at the applications of silver nano-fluids as a working fluid in heat pipes and thermal network is also conducted in the present study. Silver nano-fluids are thermochemically synthesized by a microwave heating technique, which allows for superior stable suspension characteristics.

Information Technology equipment is continuing to increase in power, leading to significant challenges from the room air conditioning perspective. Numerical models are the dominant techniques for designing new data centers and optimizing existing ones. It is thus critical to understand the inherent error in the model prediction of flow and temperature distributions in the data center, with emphasis on the vicinity of the server rack. This study uses a simulated server rack in a small room where detailed temperature (3D), flow, and power measurements were carried out for a single scenario under steady state conditions. A numerical model was constructed to represent the data center cell for this case. Model results were then compared with three dimensional experimentally obtained temperature maps and the discrepancy between the two was quantified as an error in temperature. The difference between test data and model results was found to be the highest in the exhaust air path from the rear of the simulated server rack to the inlet of the air conditioning unit. The dominant sources of error are likely to be in the use of the standard k-ε turbulence model, and the simplistic rack representation in which rack exhaust airflow is assumed uniform. Both modeling limitations tend to pronounce hot and cold spots and inhibit the mixing of hot and cold air streams within the data center. This is the first study on comparison between three dimensional temperature distributions obtained via modeling and from actual data collection.

Due to the increase in the rack power of computer equipment, in recent years, thermal management of data centers has become a challenging problem. Data center facilities with raised floor plenums are the most popular configuration from a ducting configuration perspective. The under floor plenum serves as a distribution chamber for the cooling air. Plenums are also used to route piping, conduits and cables that bring power, server network connections, and supply coolant to the air conditioning units. Computational modeling studies have shown that, these under floor blockages impede the flow of cold air from the air conditioning units and yield unpredictable air flow patterns. In this paper, the impact of under floor blockages on flow distribution in a data center is experimentally investigated. A data center experimental cell with one server rack and one air conditioning unit is examined. A number of perforated tiles and their resistances are altered to achieve variable flow patterns under the raised floor. Using previously established guidelines for the placement of under floor structures (piping, etc.) three different scenarios are studied, i.e. no blockages, blockages in critical flow path and blockages in safe flow path are considered. Experiments were carried out for two different tile designs, with free area ratios of 16% and 30%, respectively, and three tile configurations. A detailed comparison between the experimental and CFD model results are presented for several different cases, showing good agreement. Blockages if placed in critical path are shown to significantly reduce the supply tile flow rates which directly affects the thermal performance of the corresponding server. Blockages are shown to have minimal impact when they are placed in safe path. Based on CFD model and experimental results, guidelines on managing under floor blockages are verified. Emphasis has been given to the routing the blockages from safe paths and to keeping critical paths blockage free.

The most common server racks in data centers are front-to-rear racks, which draw in the cooling air from the front side and discharge it from the backside. In a raised-floor data center the cooling air to these racks is provided by perforated tiles that are placed in front of them. In a high-density data center, these tiles discharge a considerable amount of airflow, which leads to a high-velocity vertical jet in front of the rack. Such a high-velocity jet may bypass the servers located at the bottom of the rack leading to their airflow starvation and potential failure. In this paper the effect of the high-velocity jet on the airflow taken by servers at various heights in the rack is studied. A computer model based on the Computational Fluid Dynamics (CFD) technique is used to predict the airflow distribution through servers stacked in the rack. Two cases are considered. In one case, the rack is placed in the middle of a row of racks in a prefect hot aisle-cold aisle arrangement. The boundary conditions around such a rack is symmetrical. In the other case, the rack is placed in a room with asymmetrical boundary conditions. The characteristics of the servers in the rack are taken from typical 1U and 2U servers manufactured by IBM. It is shown that in general the high-velocity jet has a mild effect on the airflow taken by the servers, and the airflow reduction is limited to servers at the bottom of the rack. Racks in a symmetrical configuration are more susceptible to the airflow starvation. In the most critical conditions, an airflow reduction of 15% is calculated for the server located at the bottom of the rack. Using the result obtained from the computational analysis, a simple model is developed to predict the reduction of the cooling air under the most critical situation for the server placed at the bottom of the rack.

The current study focuses on modeling the server-rack airflow and heat transfer interaction in a data center. In a typical computing facility, the computing requirements are often gradually built-up. For example, in this instance, two servers are placed in a rack designed for a six-server stack. Each server will be separately modeled to the required specifications, and also so that their numbers and placement can be changed. The mass flow rate through the server is determined by examining how pressure profiles develop at the inlet and outlet. This mass flow rate then becomes the input into the rack model. The air inflow temperatures at the front and rear grills were obtained from experiments. The pressure profile into and out of the servers were extracted from the rack model and substituted back into the server model. Iteration continues till an acceptable level of convergence is obtained. To validate the models, experiments were carried out using thermocouples arranged in a 3 × 3 grid on a vertical plane between the exit of the server and the rear cabinet wall of the rack. The results showed that the modeling had captured the essence of the flow and heat transfer interaction. The temperature and pressure profile at the rack inlet and outlet, although in a segmented form, have performed adequately to obtain a good approximation of the flow and temperature distribution within the server/rack. The methodology of passing parameters at the server-rack level using a segmented pressure profile has been established. A similar rack-room level interaction will subsequently be developed. In essence, the methodology is equivalent to replacing the server in the rack and the rack in the room with combined flow network - thermal models. But because of the coupled nature of these two different length scale systems, the models are obtained through an iterative process. The approach enables various combinations of servers and racks to be studied quickly for any undesirable effects of off-design data center operation or layout.

As datacenter aisle noise levels approach OSHA workplace limits, air-cooled server racks share with desktop workstations the ultimate problem of being thermally constrained within a finite packaging volume by acoustic emission restrictions. The primary source of noise in most scalable 1 and 2 EIA rack drawers is an air moving device. Multiple compact dual stage counter-rotating bladed tubeaxial devices are being adopted to overcome the pressure drop of increasingly densely packed electronics. The design problem of maximizing hydraulic output within space and noise constraints naturally leads to the question of whether these many smaller fans could be gainfully replaced by fewer but larger fans. To examine this question, the fan laws for single devices are extended to equivalent systems of multiple parallel and series fans. The principle of impedance matching is applied to hydraulic sources and loads to maximize pumping power. A novel method is introduced to map fan speed, size, and hydraulic performance. The paper provides a proof that equal-staged noise-neutral space-filling planar arrays of parallel homologous devices must deliver the same net rack flow and pressure, and it shows that series arrangements of multiple stages of planar parallel fan arrays will always, under acoustic parity conditions, yield increased pressure at diminishing flow with each successive stage. At equivalent system hydraulic output, larger fans in series are shown to consume greater packaging volume while achieving lower overall acoustic output. Net rack flow and pressure relationships are also quantified for systems of equal total rack fan shaft power input. Examination of actual unit level fan noise measurements suggests that in practice, systems of small fans typically output 4 to 9 dB higher noise levels than larger fans operated at the same net flow and pressure. Finally, a survey of unit level airflow and power input measurements reveals that larger fans exhibit higher total efficiency than smaller fans.

A CFD thermal and air flow analysis is performed for a given telecommunication rack. The commercial software, Flotherm, is employed to perform the system level analysis. The analysis is made to examine the effects of the air filter locations at the upstream and downstream of the fans on the flow distribution within the system. The analysis is then extended to the case with one-fan failure condition.

Parallel optical interconnects (POIs) offer capacity advantages for high-density telecomm requirements. Once an array of multi-channel, high-capacity POIs is placed on a board, shelf or telecom rack, it creates a challenging thermal management task. To develop a thermal management concept for high dissipating racks (over 10 kW), capable of keeping a maximum temperature of 80°C for optical components, with tight temperature uniformity requirements of ±1°C between a receiver (Rx) and transmitter (Tx), a fully functional two-shelf telecommunication rack has been built. A thermal management solution was proposed and was based on the implementation of heat pipes embedded into each board. The performance of embedded heat pipes is characterized for densely packaged, high-speed optics applications requiring temperature uniformity and stringent temperature limits. The proposed solution completely meets requirements of power-dense, high speed POIs at the board/shelf level for the typical telecommunication rack. Moreover, it evolves into an enabling strategy for the reliable control of power-dense, temperature-sensitive optical components.