Design and Verification of Cray XT- 4 Supercomputer Cooling System

Cray supercomputers are purpose-built to meet the special needs of capability class HPC applications. Recent advances in Cray scalar supercomputer systems resulted in the introduction of the higher density compute cabinet that consumes significant amounts of electrical power and produces an extraordinary amount of heat. Due to the new design requirements, which have tremendous consequences with regard to thermal management, new cooling concepts were required. The XT4 can have up to 30,000 dual-core Opterons, and the SeaStar2 interconnection chip plugs right into the Hyper Transport links on the Opteron processors to make a very fast mesh of processors and memory for applications. Such a machine consumes about 5.2 megawatts of power to deliver around 1.15 petaflops. That is twice the size of Red Storm, but more than a factor of 10 more performance. The extra power density required additional modification to the cooling system. This article describes the design, simulations and verification of the XT-3 and XT-4 systems. Over the past few years thermal design for cooling microprocessors has become increasingly challenging, as silicon technology continues to scale in accordance with Moore’s law. The industry has traditionally relied on gradual improvements in air cooled, solid metal heat sinks to keep pace with change in microprocessor design to provide cost effective solution for removing from microprocessors. In Cray’s XT-4 system, chip power density is expected to almost double that of it’s Red Storm/XT-3 predecessors, while maintaining the same cabinet size.