Industry group looks into virtualisation benchmarks

A technology industry standards body is considering setting benchmarks for how virtualisation should work in a computer network.

The Standard Performance Evaluation Corporation (SPEC) is forming a group to discuss whether virtualisation benchmarks are needed and, if so, then set those benchmarks over an estimated three-month process.

"It seems like a number of our member companies have simultaneously agreed they are ready to move on it and form a working group," said Walter Bays, president of SPEC "All of them must have been hearing the same thing from their customers."

SPEC is a non-profit organisation of top companies in the technology industry collaborating to establish impartial standards to measure the performance of various types of technology. SPEC has set benchmarks for servers, software applications, high performance computing systems, for example.

Virtualisation is a growing field for improving the performance of data centers without having to add more servers. For instance, virtualisation makes it possible to run several software applications, even based on different operating systems, on the same physical server.

Because many IT managers at companies are still learning what virtualisation is, benchmarks would help educate them, said Bays.

"It would actually increase the market for virtualisation because it would reduce uncertainty about it for the buyers," he said. "They would be better able to do comparison shopping."

"Based on the adoption rate, benchmarking is not first and foremost in customers' minds," said Brian Byun, vice president of global partners and solutions for VMware. Whether one virtualization software package runs 5 percent or 10 percent faster than another isn't as important as the increased performance of virtualisation versus non-virtualisation, Byun said.

The lack of benchmarks isn't preventing Advanced Micro Devices from touting the virtualisation capabilities of its microprocessors.

"Virtualisation is getting deployed. It's beyond the test beds and R&D labs," said Patrick Patla, director of server and workstation marketing for AMD. Still, he added, "We would love to have an industry standard benchmark."

Benchmarks are also needed to compare different types of virtualisation software. Although VMware is the leader, the benchmarking process would compare virtualisation software from multiple vendors.

"We do believe that industry standards are very important but they should be vendor neutral," said Lorie Wigle, marketing director for server, software and technology initiatives for chip maker Intel.

Intel already offers a virtualisation measurement tool, called VConsolidate, which hardware vendors and customers use to gauge how virtualisation will work in various set-ups.

A web server, an email server or a database program all have different performance benchmarks, but they could all operate simultaneously on one virtual server, so performance benchmarks have to be rethought. "Virtualisation requires a completely different usage model," Wigle said.

SPEC's Bays said the working group will include representatives from Intel, AMD, Fujitsu Siemens Computers, VMware and other major vendors, but that SPEC would also like to invite end users to participate in the process.