September 25, 2012
This week, the New York Times ran a series of articles about data center power use (and abuse) “Power, Pollution and the Internet” (http://nyti.ms/Ojd9BV) and “Data Barns in a Farm Town, Gobbling Power and Flexing Muscle” (http://nyti.ms/RQDb0a). Among the claims made in the articles were that data centers were “only using 6 to 12 % of the energy powering their servers to deliver useful computation. Like a lot of media broadsides, the reality is more complex than the dramatic claims made in these articles. Technically they are correct in claiming that of the electricity going to a server, only a very small fraction is used to perform useful work, but this dramatic claim is not a fair representation of the overall efficiency picture. The Times analysis fails to take into consideration that not all of the power in the data center goes to servers, so the claim of 6% efficiency of the servers is not representative of the real operational efficiency of the complete data center.
On the other hand, while I think the Times chooses drama over even-keeled reporting, the actual picture for even a well-run data center is not as good as its proponents would claim. Consider:
- A new data center with a PUE of 1.2 (very efficient), with 83% of the power going to IT workloads.
- Then assume that 60% of the remaining power goes to servers (storage and network get the rest), for a net of almost 50% of the power going into servers. If the servers are running at an average utilization of 10%, then only 10% of 50%, or 5% of the power is actually going to real IT processing. Of course, the real "IT number" is the server + plus storage + network, so depending on how you account for them, the IT usage could be as high as 38% (.83*.4 + .05).
Not good, but certainly better than the implication that more than 90% of the power is disappearing into thin air.
The other major flaw in the reasoning is the assumption that these efficiencies, which the Times claims are representative of a survey of “70 large data centers spanning the commercial gamut: drug companies, military contractors, banks, media companies and government agencies.”, are representative of the kinds of large Internet data centers taht the articles are focusing on. I’m certainly not going to argue that its data is bad, but a sample of this nature does not represent the kinds of large Internet/Cloud data ceners that the articles focus on. The sample almost certainly includes a number of corporate data centers with markedly less efficient power and cooling infrastructure, and running a very heterogeneous enterprise workload that probably does exhibit an overall low utilization for the servers. I would assert that the large Internet data centers cited in the article have some significant differences from the assortment that were included in the survey:
- These data centers run at much higher utilization than your average enterprise DC because the servers and the entire data centers are designed around a very few very large workloads whose characteristics are well understood in advance.
- They use more power-efficient servers (these people drive the development of ULP CPUS and stripped-down configurations. Facebook, for example, has separate configurations optimized for its core applications, for the memchache tier, and for file service. Some very large operators have even gone so far as to design their own custom servers with specialized power supplies and battery backup systems unlike the commercial mainstream.
- These data centers have among the world’s most efficient cooling and power management systems and are typically operated more efficiently than the average enterprise data center.
Other unnecessary drama in the articles includes wide-eyed astonishment at the fact that “Even running electricity at full throttle has not been enough to satisfy the industry. In addition to generators most large data centers contain banks of huge, spinning flywheels or thousands of lead-acid batteries.” Wow, that sounds … dangerous? Not sure, but sure sounds dramatic — maybe the Times has unearthed a deep conspiracy to monopolize all the Earth's power and ship it off to space aliens. This is just plain silly — it’s called backup power. The banks of huge spinning flywheels provide seconds of backup, and the batteries minutes. And they don’t pollute, and they only impose a very small penalty on overall power efficiency. Then we get to the diesel generators, the subject of much wailing and gnashing of teeth in the series.
After the authors point out a few incidents of data centers running their diesel generators more than they should have, and some violations of environmental and permitting regulations, readers are left with the impression that there is a vast conspiracy to somehow poison school children and innocent workers with diesel exhaust. Ignored is the practical reality of diesel generators — they are about the most expensive power available, and the data center operators will do anything (except allow themselves to go offline) to avoid using them. The vast number of generator permits cited in the articles are a result of the fact that if you have N megawatts of data center power consumption, you better be able to supply the same number of megawatts from your on-site generators, not a desire to run them willfully just to annoy the local citizenry.
Part of my irritation with these articles is that by exaggeration and an excess of “Sturm und Drung,” as dramaturges are wont to say, the authors are doing a disservice to a serious issue that deserves a higher standard of debate. Overall use of electricity by IT of all kinds is a major policy and environmental issue and cannot be ignored, but the debate should not be carried out via breathless and poorly informed articles in the mass media. Fortunately there are remedies, both from a policy and a market perspective, that are working to ameliorate the problem, including:
- Regulatory pressures on energy consumption and efficiency are increasing. This makes me nervous because I'm not sure that I want the same people who wrote our tax codes and other fair and simple laws intervening in a very competitive market process.
- Major improvements in power efficiency throughout the entire IT technology chain, from semiconductor technology through server design. What the articles fail to highlight is that energy efficiency has been a major competitive front in the multi-billion-dollar server and data center market for years, and will continue to be so. Even without the clumsy and heavy hand of government regulation, the efficiency of data centers has improved, and will continue to do so for the foreseeable future.
- A revolution over the last several years in overall data center engineering and in the holistic management of the data center.
Despite the NY Times’ sudden discovery of the problem, the IT industry has been working diligently on solving these problems for years and will continue to make progress long after the mainstream media has gone on to expose killer mimes and the hazards of the exploding population of Frisbee-playing bears. The simple fact is that if we want to live in an information society, we need the plumbing to support it. That is perhaps the most salient observation that the New York Times made — there really is no cloud, just more and more really big data centers.