Google, IBM help teach massively parallel computing

By Stephen Withers
Tuesday, 09 October 2007 05:17

Home IT

Google and IBM have teamed to provide computer science students at a select group of universities with a testbed for large-scale distributed computing applications.

The initial cluster comprises "several hundred computers" - an unspecified mix of Google's 'homebrew' systems, IBM BladeCenters and IBM System x servers. The plan is to expand the cluster to 1600 processors.

The project is the brainchild of Google senior software engineer Christophe Bisciglia and Professor Ed Lazowska of the University of Washington.

"In this age of 'Internet-scale' computing, the new, evolving problems faced by computer science students and researchers require a new, evolving set of skills," said Bisciglia. "It's no longer enough to program one machine well; to tackle tomorrow's challenges, students need to be able to program thousands of machines to manage massive amounts of data in the blink of an eye."

A year after the program was initially developed, "we've seen how our students have mastered many of the techniques that are critical for large scale-internet computing, benefiting our department and students," said Lazowska.

Google and the University of Washington have released under a Creative Commons licence a set of curriculum materials on massively parallel computing techniques.

The cluster runs software including Linux, Xen and Hadoop. That last item is an open source implementation of Google's Google File System and MapReduce (software that distributes data across the nodes of a cluster and then processes the clusters where they reside).

IBM has also contributed software to simplify using the Eclipse development platform to create software for systems using Hadoop, plus Tivoli software to manage the cluster.

"We're aiming to train tomorrow's programmers to write software that can support a tidal wave of global Web growth and trillions of secure transactions every day," said Samuel Palmisano, IBM's chairman, president and CEO. 

Other universities involved in the pilot include Carnegie-Mellon University, Massachusetts Institute of Technology, Stanford University, the University of California at Berkeley and the University of Maryland.

"As the pilot progresses, we'll work with our technology partner IBM to shake the bugs out of the system so that we can expand the program to include more educators and academic researchers," said Bisciglia.