Oak Ridge National Laboratory Debuts Titan Supercomputer

Supercomputer combines gaming and traditional computing technologies to provide unprecedented power for research

OAK RIDGE, Tenn. – The U.S. Department of Energy’s (DOE) Oak Ridge National Laboratory (ORNL) launched a new era of scientific supercomputing today with Titan.  That means that Titan is positioned to take the title as the fastest supercomputer in the U.S. and the world’s Number 1 Open Science computer.  Titan will provide unprecedented computing power for research in energy, climate change, efficient engines, materials and many other fields while paving the way for new achievements in science and technology. 

This is more than just headlines. Titan’s impact will be a dramatic breakthough in scientific computing as it is a system capable of making more than 20,000 trillion calculations each second or over 20 petaflops – by using Nvidia Kepler graphics processing units (GPUs) first created for computer gaming.  Titan is ten times more powerful than ORNL’s last system, Jaguar, while fitting in the same physical space and using a similar power envelope as the old system.  As it is, Titan is expected to cost $38 million a year in electricity usage.

Basically ORNL is performing a supercomputer upgrade as their Jaguar supercomputer the size of a basketball court has been transformed into Titan in the same space. ORNL has upgraded from Cray’s XT5 compute blades to their XK6 compute blades, which use hybrid chipsets comprised of AMD Opteron CPUs and Nvidia Tesla K20 GPUs. Early application benchmarks demonstrated that the XK6 is yielding performance improvements ranging from 50 percent to 230 percent compared with the XT5.

In February 2012, each of Jaguar’s 18,688 Cray XT5 compute nodes was replaced with a new Cray XK6 compute node that consists of one AMD 16-core Opteron 6274 processor running at 2.2 GHz, 32GB of DDR3 memory, and Cray’s new high performance Gemini network. The upgraded Jaguar system has a total of 299,008 AMD Opteron CPU cores, more than 700 terabytes of memory, and is connected to the 240 GB/s Spider file system.

14,592 of Titan’s nodes, 78 percent of the total compute nodes, was just upgraded by adding Nvidia Tesla K20 “Kepler” GPU application accelerators each with 6 gigabytes of high speed memory.  These Kepler-accelerated nodes are in place well in advance of Nvidia’s commitment to do so by January 1, 2014. This leaves 4,096 XK6 compute nodes remaining without an accelerator and available to users throughout 2013.

Users of Titan will continue to have access to the Spider file system, with 240 GB/s data bandwidth and over 10 PB of storage capacity. Spider will be upgraded next year to increase both bandwidth and capacity.  Users will also have access to the HPSS data archive, LENS data analysis and visualization cluster, and the newly upgraded EVEREST high resolution visualization facility. All of these resources are available through high performance networks including ESnet’s recently upgraded 100 gigabit per second links.

Power Consumption Challenges

Amazingly, the new system not only occupies the same space as the old system, but it consumes only marginally more power for it’s ten times performance improvement.  One challenge in supercomputers today is power consumption and by combining GPUs and CPUs in a single system requires less power than using CPUs alone.  Discoveries that may take weeks on a system as powerful as Jaguar may now only take days on Titan.

Supercomputers have revolutionized science.  Before they existed, researchers came up with theories and then they carried out experiments.  Supercomputing gives researchers and scientists a third way – computer modelling.   Researchers still begin with a theory but we now can build a computer model to see if it works before even attempting physical experiments.

Because they handle hundreds of calculations simultaneously, GPUs can run many more calculations than CPUs in any given time. By using Titan’s 299,008 CPU cores while allowing its new Nvidia GPUs to do ninety percent of the work, Titan will enable researchers to run scientific calculations with much greater speed and accuracy than with Jaguar.

Just upgrading Jaguar to Titan – increasing the performance tenfold – by using CPUs only, would require ten times the electrical power – and Oakridge cannot afford a $380 million dollar annual electricity bill.  Using the GPU allows them to increase the performance tenfold while only increasing power consumption less than 15%!

An “Open” System

Titan will be made available to scientists in various fields. One program will tackle climate change and how increasing gas emissions might affect it. Another program will study the way fuel burns in engines to find ways of radically improving efficiency.  Yet another researcher has already had two million hours of Titan’s processing power awarded to focus on folding proteins.   This will enable researchers to design more effective medicines.

Tackling the world’s energy challenges is high on the list. Scientists are also looking forward to busing Titan for calculations that may bring sustainable fusion technology.  Improving nuclear energy reactors is another possibility.

Titan will be open to select projects while ORNL and Cray work through the process for final system acceptance. Most of the access to Titan will come from the Department of Energy’s Innovative and Novel Computational Impact on Theory and Experiment program, better known as INCITE.

Researchers have been preparing for Titan for over two years, with many ready to go on day one.   There is an insatiable demand for using Titan – the demand exceeds the time awarded by over three times.  Far more supercomputers are needed.

 The Future

All the charts and most of the pictures are courtesy of Nvidia.  They are quite proud of their accomplishment in changing the face of supercomputing with the GPU.  This editor was priviliged to attend and report on Nvidia’s GPU Technology Conference (GTC 2012) back in May.  This annual conference is devoted to GPU programming which includes supercomputing applications.  Look at the ABT report for Nvidia’s complete vision and their plans for 2013 and beyond.  You will also find a multi-page discussion of the Kepler architecture there and its uses for GPU computing and especially as it relates to scientific research.Nvidia uses part of the money they make from consumer GPU sales as video cards to leverage research and development of their professional brands, Tesla and Quadro.  In return, the knowledge gained from the professional market helps Nvidia to develop their gaming GPUs as the calculations for graphics and for scientific applications all benefit from the same massive parallel processing power of the GPU.  Basically, the same GPUs we game on are being used in a revolutionary way to benefit all of us in the world of supercomputing.

Happy Gaming!

Please join us in our Forums

Become a Fan on Facebook

Follow us on Twitter

For the latest updates from ABT, please join our RSS News Feed

Join our Distributed Computing teams

• Folding@Home – Team AlienBabelTech – 164304
• SETI@Home – Team AlienBabelTech – 138705
• World Community Grid – Team AlienBabelTech