When you consider what IT veterans love most, benchmarks rank right up there. But in today's world of wildly-varying virtualized server setups and highly-customized application deployments, it's time to ask: Are server benchmarks an early love that you should let go? IT has been conditioned since the 1980's to measure all sorts of hardware and software buying decisions using performance benchmarks. But some of the current discussion around CPU, server and virtualization benchmarks makes me shake my head. And I'm not the only one, as I learned when I met this week with AMD's Kevin Knox, VP of commercial business.

"Even one or two years ago, customers focused on benchmarks," when comparing enterprise servers, he told me. "Now they focus more on power, cooling, density, environmental issues."

Some people will say "AMD is trying to de-emphasize benchmarks? No surprise." That's because a long time ago, in the early 90's, Intel was winning the benchmark competition for desktop PC microprocessors. Ancient history, if you ask me. I covered microprocessors in the late 90's, as Intel and AMD raced to the one-gigahertz mark for desktop processors. They both arrived at that finish line at roughly the same time, at about the same level of performance for desktop PCs. Since then, both companies' chips have become more than powerful enough for desktop PC applications, and PC buyers are much more concerned with choosing a PC vendor than a chip vendor. 

Servers, of course, are a different story. Enterprise IT buyers of servers have far more complex needs than desktop PC buyers. Today's servers, with multiple chips with multiple CPU cores, continue to be performance benchmarked heavily, by vendors, industry groups and technology publications.

But now, a growing number of AMD-based server customers, Knox says, are going beyond traditional server performance benchmarks. These enterprise IT leaders are creating their own equations for comparing servers, using factors such as (you guessed it) power, cooling, and other items that affect real life in a data center, he says. 

Knox calls it the "build your own equation" approach.

I find this fascinating, for a couple of reasons. First, technology journalists have spent man hours that would make NASA engineers cringe trying to come up with desktop and server buying-criteria equations similar to this, equations which reflect more than just performance scores on CPU tasks, application tasks, and the like. It's incredibly hard. 

Second, CIOs in some of the most demanding data center environments (such as financial services) are building these individualized buying decision equations, Knox says.

Maybe they've decided the same thing that I have after reading early accounts of vendors trying to out-benchmark each other on server virtualization and other early attempts to do virtualization benchmarking: It's not going to be meaningful to server buyers, anytime soon.  

As I have learned while interviewing many CIOs in the past year regarding virtualization, every virtualized server is different. When you talk about measuring the performance of a physical server full of VMs, there are a ton of variables in what that server looks like, even just comparing the servers inside one company. Many

You do not have flash or javascript support.

Average (1 vote)

Reply
Digg this
Slashdot
Reddit
LinkedIn