VMware benchmark may be too little, too late

Performance is just one component of virtualisation and tough to compare

VMware has begun offering its VMmark benchmark in a move that should go some way to resolving the debate over how to measure virtualisation performance for the sector leader. However, the ability to compare across virtualisation vendors and technology approaches remains some way off.

VMmark has taken about two years to come to market, with VMware spending that time in designing the tests and working with partners and customers. For makers of enterprise servers, VMmark scores are likely to be prized. Dell and Sun are already using VMmark numbers while others such as HP, Intel and AMD are sure to follow. For buyers, the benchmark should offer a guide to appropriate hardware configurations for different workloads.

Benchmarks may be published, providing they are accompanied by a full disclosure report.

“For a long time people have been asking about performance and traditional benchmarks were not applicable because they try to take a single instance of software and take it to saturation,” said Reza Malekzadeh, VMware product marketing director.

Ironically however, Malekzadeh believes that pure performance benchmarking might not be the issue it once was in virtualisation.

“Early on, there were a lot of customer requests for it but hardware performance has increased so much that performance is longer such a factor,” he said. “In the cases where we win or don’t win against other virtualisation vendors, for example, the differences are no longer that big.”

Benchmarking group Spec is also developing a cross-vendor virtualisation benchmark although many experts are sceptical that a suite can measure performance of competing technological approaches.

“VMmark is a step in the right direction but you need any benchmark tool to be vendor-agnostic and technology-neutral,” said Ben Rudolph, director of virtualisation rival SWsoft.

“Also, benchmarking is just one component of virtualisation. Manageability, functionality, partner programmes, ease of installation and ease of use are just as important or even more important.”

XenSource founder Simon Crosby said, "The development of a vendor-neutral framework for creating realistic mixed workloads of VMs that can be benchmarked together using characteristic traffic loads is a definite step up from the simple re-use of traditional benchmarks intended for a single OS instance per server. Intel and IBM led the way with the Consolidate benchmark suite, which has been used to kick off a Spec standardisation effort. VMmark is a similar approach and probably quite useful."