O’Reilly news

New Release Shows How to Cluster Linux Machines to Mimic the Performance of Supercomputers

July 28, 2000

Sebastopol, CA--O'Reilly's latest release, Building Linux Clusters by David HM Spector, defines a cluster in its simplest form as "a bunch of computers tied together with a network, working on some large problem that has been broken down into smaller pieces." The simplicity of the definition belies the importance of the subject. The list of the world's most powerful supercomputers now includes Beowulf machines: PCs running Linux, clustered together into a single machine.

From scientific applications to transaction processing, clustering technology provides an affordable, scalable computing solution. One of the hottest topics in Linux today is the ability to cluster Linux machines to mimic the performance of supercomputers (costing hundreds of thousands of dollars) for a fraction of the cost.

"Computing problems always seem to meet or exceed available computing resources. There is always a need for more processing power, more network bandwidth, and greater I/O capabilities," says Spector. "This need for speed has prompted a whole new class of machines: supercomputers. In fact, supercomputer class machines-the construction of which is the topic of my book-influence almost every aspect of modern life."

"The Linux operating system has given rise to a host of interesting and unique computing possibilities that would have been impossible, and perhaps even unthinkable, in a pre-Linux era," says Spector. Some examples that Spector gives, include:

  • A single cross-platform operating system

  • An open source operating system whose entire code base is freely available for research, commercial use, and improvement

  • A system that has the potential to displace both Unix variants and Windows NT as a server operating system

  • A scaleable parallel processing system that allows performance comparable to traditional supercomputers (IBM SP/2 or Cray), but at a fraction of the cost

"Since the ground-breaking work was done at NASA, Beowulf-style clusters have become their own well-defined genre of high-performance computing systems," explains Spector. "By October 1999, several of the systems in the top 200 supercomputers in the world were made up of Linux cluster systems."

Spector's first Linux cluster, and ultimately his new book Building Linux Clusters, arose from his need for an affordable home supercomputer for his research. Building it was a challenge. "I noticed that a lot of people talk about it, but the documentation is really terse. There's no way for someone to start who doesn't have all the knowledge already." Spector wrote up an outline and realized it would make a great O'Reilly book. And, as it turned out, O'Reilly was looking for just such a book.

Building Linux Clusters is a hands-on guide for people new to clusters. "Like most O'Reilly books," said Spector, "it's to get people going. You can build a cluster right out of the box: You pop the CD in, follow the instructions, and in ten or twenty minutes you have a fully functional cluster."

Spector likes to say that the book tells a story. And indeed, the first chapter is a brief, lively history of the evolution of supercomputers and, eventually, of clusters, including Don Becker's first Beowulf cluster. Built in 1994 to meet NASA's need for supercomputer performance on a shoestring budget, it used Linux and off-the-shelf hardware, saving NASA a couple of million bucks, according to Spector.

Spector thinks that the most exciting future applications of clusters are the least possible to predict. That's because learning to work with parallel computing changes the way people look at problems and opens new possibilities for applications. "People have to step out of the mindset of being serial and think about how to do everything at the same time. And that's a hard transition, but once you do that, suddenly new vistas open to you."

Online Resources:

Building Linux Clusters
Scaling Linux for Scientific and Enterprise Applications
By David HM Spector
1st Edition, August 2000
1-56592-625-0, 352 pages, $44.95, Includes CD-ROM

About O’Reilly

O’Reilly, the premier learning platform for technology professionals, offers the industry’s most extensive catalog of high-quality technical and professional skills development courses. From AI, programming, and cloud technologies to essential business skills such as leadership training and critical thinking, O’Reilly delivers highly trusted content from its network of renowned experts that meets a diverse array of learning needs, with over 5,000 role-based on-demand courses, nearly 200 live events each month, access to interactive sandboxes and labs, and more. For more information, visit www.oreilly.com.

Email a link to this press release