Thermodynamic Characterization of a Direct Water Cooled Server Rack Running Synthetic and Real High Performance Computing Work Loads

High performance computing server racks are being engineered to contain significantly more processing capability within the same computer room footprint year after year. The processor density within a single rack is becoming high enough that traditional, inefficient air-cooling of servers is inadequate to sustain HPC workloads. Experiments that characterize the performance of a direct water-cooled server rack in an operating HPC facility are described in this paper. Performance of the rack is reported for a range of cooling water inlet temperatures, flow rates and workloads that include actual and worst-case synthetic benchmarks. Power and temperature measurements of all processors and memory components in the rack were made while extended benchmark tests were conducted throughout the range of cooling variables allowed within an operational HPC facility. Synthetic benchmark results were compared with those obtained on a single server of the same design that had been characterized thermodynamically. Neither actual nor synthetic benchmark performances were affected during the course of the experiments, varying less than 0.13 percent. Power consumption change in the rack was minimal for the entire excursion of coolant temperatures and flow rates. Establishing the characteristics of such a highly energy efficient server rack in situ is critical to determine how the technology might be integrated into an existing heterogeneous, hybrid cooled computing facility — i.e., a facility that includes some servers that are air cooled as well as some that are direct water cooled.