Accelerating Physics and the "Big Bang" Theory with a Secure Network for Grid Computing

Located in Geneva, Switzerland and funded by 20 European member states, CERN is chartered with exploring the elementary constituents of matter and their interactions; the fundamentals of The Big Bang Theory.

CERN has 2,500 staff scientists and some 6,500 visiting scientists (half of the world's particle physicists) that come from 500 universities representing 80 nationalities.

Using a powerful new particle accelerator, CERN is looking to investigate fundamental questions such as the origin of mass, symmetry and super-symmetry models, matter and anti-matter asymmetry, and the basic structures of matter. Key to this research is a Grid network computer capable of processing and storing massive amounts of data produced from the accelerator and thousands of connected scientists.

The Challenge: Building a More Powerful, High-Speed Infrastructure The last decade has seen a substantial increase in commodity computing and network performance, mainly as a result of faster hardware and more sophisticated software. Nevertheless, there are still problems in the fields of science, engineering, and business, which cannot effectively be dealt with using the current generation of supercomputers.

In fact, due to the size and complexity of some projects, they require a variety of heterogeneous resources that are not available on a single machine. The cooperative use of distributed computing resources unified to act as a single powerful computer - is known as Grid Computing. Many applications can benefit from the Grid infrastructure, including collaborative engineering, data exploration, and distributed supercomputing.

In CERN's case, particle physics requires special tools to create and study new particles. These include Accelerators, which speed up particles to very high energies before colliding them with other particles, and Detectors, instruments that register the particles produced by the collision.

CERN is currently building the world's largest accelerator-the LHC or Large Hadron Collider-which will collide beams of protons at an astonishing energy level of 14 TeV. And with a 27 Km circumference, the accelerator will also be the largest superconducting installation in the world.

Particle collisions are independent events that can be analyzed on separate computers, lending itself to the use of computer clusters and is a perfect fit for Grid Computing. The physicist's goal is to count, trace and characterize all the particles produced and fully reconstruct the process. The LHC creates 40 million collisions per second. After filtering, 100 collisions of interest per second remain. A megabyte of digitized information for each collision results in a recording rate of 0.1 gigabytes/sec. The collisions recorded each year then result in 10 petabytes/year of data! This creates a significant challenge for storage, networking, and CPU capacity.

The Solution: 10 Gigabit Ethernet Secure Network Enables Next-Generation DataGrid Applications. Due to the rapid growth of the Internet and IP/Ethernet, there has been a rising interest in IP/Ethernet-based distributed computing and many projects have been started to exploit this as an infrastructure for running distributed and parallel applications. In this context, IP/Ethernet has the capability to be a platform for collaborative work as well as a key technology to create a pervasive and ubiquitous Grid-based infrastructure.

As a pioneer in its own right-developing high-speed, innovative Ethernet-based solutions for nearly 20 years- Enterasys was a logical choice for CERN.

The relationship between CERN and Enterasys Networks has existed since late 1998, when CERN's production network first implemented a Gigabit Ethernet routing solution from Enterasys.

So when CERN looked to leading IT companies to test future Grid technologies, it again turned to Enterasys. The CERN openlab is a collaboration between CERN and its partners to develop data-intensive Grid technologies to be used by the worldwide community of scientists. The LHC experiments will generate enormous amounts of data-several million gigabytes a year-and will require a network more powerful and functional than anything running over today's Internet.

Within the openlab, all participating companies such as Enterasys, IBM, Intel, Oracle, and HP provide the latest technology. In this context IBM is primarily responsible for the storage, Intel for the 64-bit CPUs and Gigabit/10Gigabit Ethernet NICs, Oracle for databases, and HP for the servers. Enterasys delivers the networking infrastructure: high-speed Gigabit and 10 Gigabit Ethernet switches and routers to support all of the open lab applications.

In September 2004 CERN's network for openlab was upgraded with 32 10 Gigabit Ethernet-enabled CPUs and 400 Gigabit Ethernet-enabled CPUs.

The Matrix N-Series is used to aggregate 10/100/1000 Ethernet connections into 10 Gigabit Ethernet trunks in the core. CERN is also evaluating Enterasys' next-generation core routing solution, featuring nonblocking, single-stream 10 Gigabit Ethernet capabilities for the openlab network.

Up to 128 10 Gigabit Ethernet ports in a single system will be supported by this breakthrough architecture. Bundled with its integrated secure, high-availability features-such as hitless software upgrades, no single point of failure design, and nondisruptive software recovery-this core router is ideally suited to support Grid applications.

"As a long-time customer of Enterasys, and one of the first adopters of its 10 Gigabit Ethernet technology, we look forward to continuing to build Gigabit capability for the data-intensive infrastructure that we are developing for the openlab," said Wolfgang von Ruden, Division Leader, Information technology at CERN. If all goes as according to plan-and the progress up to this point indicates CERN is well on schedule-the LHC accelerator should be fully up and running by 2007.