August 19, 2009
I have blogged before about how much I enjoy mathematics and statistical analysis, and this is in fact another exciting, hard question!
The question was presented as the following hypothetical…
Given 10 quad-core servers in a cloud running 40 virtual machines. 20 virtual machines are simultaneously discarded. What power savings would you expect from PowerNap?
This is actually a rather complex combinatorics problem. I’ll restate it as such:
Given 10 buckets, each of which can hold 0 – 4 items, how many ways can 20 items be distributed among those 10 buckets? Furthermore, what is the statistical distribution of empty buckets?
It’s been 10 years since my university statistics classes, and my brain melted trying to derive the permutation formula. But a big thanks to my good friend Matt Dobson for solving that! If you’re interested, you can view his combinatorics formula here.
It took me about an hour to hack a Python script that could empirically solve this problem by brute force, generating a comprehensive list of all of the permutations. You can now download and run /usr/bin/powernap_calculator provided by the powernap package in Karmic.
Here are the results running on the parameters above:
$ time powernap_calculator –hosts 10 –guests-per-host 4 –guests 20
In a cloud with  hosts, which can handle  guests-per-host, currently running  guests,
you may expect the following:
[ 5.2%] likely that [0/10] of your hosts would powernap, for a [0%] power savings
[ 27.5%] likely that [1/10] of your hosts would powernap, for a [10%] power savings
[ 42.5%] likely that [2/10] of your hosts would powernap, for a [20%] power savings
[ 21.8%] likely that [3/10] of your hosts would powernap, for a [30%] power savings
[ 2.9%] likely that [4/10] of your hosts would powernap, for a [40%] power savings
[ <1%] likely that [5/10] of your hosts would powernap, for a [50%] power savings
The overall expected value is [19.0%] power savings.
So at this snapshot in time, when your cloud suddenly dropped to 50% utilization, the expected value is a 19% power savings. Note that expected value is a very specific statistics term. Wikipedia says:
In probability theory and statistics, the expected value (or expectation value, or mathematical expectation, or mean, or first moment) of a random variable is the integral of the random variable with respect to its probability measure. For discrete random variables this is equivalent to the probability-weighted sum of the possible values, and for continuous random variables with a density function it is the probability density-weighted integral of the possible values.
Right, so we might expect about a 19% power savings for this random moment in time, when we simultaneously reduce our utilization by 50%.
However, if we observe the cloud over time, and with perhaps more realistic usage patterns, the distribution should be much better than random. VMs will come and go at more staggered intervals than simultaneous destruction of half the instances.
And you will have another tremendous factor working in your favor — Eucalyptus’ greedy scheduling algorithm. This ensures that each time a new VM is launched, it will land on a system that’s already running. This is known as an annealing system — one that’s constantly correcting itself — since under-utilized systems will automatically powernap, and new VMs will fill in the gaps on running hardware. Pretty cool, I think.
So I’m curious…
- What does (or would) your cloud look like?
- How many -h|–hosts, -p|–guests-per-host, and -g|–guests do you usually have?
- What does powernap_calculator say about your parameters?
- Post your results!
Note that the powernap_calculator algorithm is exponential, O((p+1)h), so large values of (p, h) will take a very long time to compute! I’m totally open to code review of powernap_calculator and any performance enhancements 😉
In my cloud, I have 8 dual-core hosts. I typically limit my usage to 2 guests-per-host. And I rarely run more than 4 VMs at a time.
$ powernap_calculator -h 8 -p 2 -g 4
In a cloud with  hosts, which can handle  guests-per-host, currently running  guests,
you may expect the following power savings:
[ 26.3%] likely that [4/8] of your hosts would powernap, for a [50%] power savings
[ 63.2%] likely that [5/8] of your hosts would powernap, for a [62%] power savings
[ 10.5%] likely that [6/8] of your hosts would powernap, for a [75%] power savings
The overall expected value is [60.5%] power savings.
My servers run at about 80 Watts fully loaded. My electricity is about $0.10 per kilowatt-hour. So a year’s worth of electricity with all 8 servers running all of the time is 8 * .08 KW * 24 hr/day * 365 days/year * $0.10/KW-hr = $560/year.
I like the prospects of saving approximately 60.5% of that, or $339/year!
So how does the calculator work?
Let’s use a slightly smaller example: hosts,h=4, guests-per-host,p=2, guests,g=3
Since each system can hold 0, 1, or 2 virtual machines, we’re going to use a base-3 number (which is p+1). And we’re going to generate all possible 4-digit base-3 numbers (which is (p+1)h). In our case, that’s 34 or 81 scenarios to consider. For each of those scenarios, we convert the decimal integer to a list of each of the digits 0-2, and sum the list, throwing out any “invalid scenarios”, where the sum does not add up to the target number of guests, 3, g. Our valid scenarios is actually much smaller, the following 16:
[2, 1, 0, 0]
[1, 2, 0, 0]
[2, 0, 1, 0]
[1, 1, 1, 0]
[0, 2, 1, 0]
[1, 0, 2, 0]
[0, 1, 2, 0]
[2, 0, 0, 1]
[1, 1, 0, 1]
[0, 2, 0, 1]
[1, 0, 1, 1]
[0, 1, 1, 1]
[0, 0, 2, 1]
[1, 0, 0, 2]
[0, 1, 0, 2]
[0, 0, 1, 2]
Now, of these, we are interested in how many 0’s are in each row, as this indicates a host that has no guests, and can therefore be powernapped.
Our calculations yield: [0, 4, 12, 0, 0], or:
- 0 possible scenarios have no unused hosts
- 4 possible scenarios have 1 unused hosts
- 12 possible scenarios have 2 unused hosts
- 0 possible scenarios have 3 unused hosts
- 0 possible scenarios have 4 unused hosts
From this we can generate the following probabilities:
[ 25.0%] likely that [1/4] of your hosts would powernap, for a [25%] power savings
[ 75.0%] likely that [2/4] of your hosts would powernap, for a [50%] power savings
And weighting these probabilities, we can generate the expected value:
25%*.25 + 50%*.75 = 43.75%
The overall expected value is [43.8%] power savings.
July 29, 2009
If you don’t already follow the Ubuntu Podcast and the Ubuntu UK Podcast, you really should! Both are community driven multimedia presentations that cover the news in the Linux and Ubuntu communities. These guys provide a tremendous service to the world of Ubuntu.
And the Ubuntu UK Podcast guys (Ciemon Dunville, Alan Pope, Tony Whitmore, Dave Walker) interviewed your humbled author in S02E09, The Dimensions of Time. We talked about the Ubuntu Server, PowerNap, KVM, and eCryptfs.
July 23, 2009
I just read this article, referenced by Slashdot: Power Capping the Data Center.
And I thought I’d borrow this video (one of my favorites) from IBM:
I think that we in the Ubuntu Server Community are leading the charge in developing an energy-efficient server class Linux distribution.
It is my hope that by the 10.04 LTS release, the Ubuntu Server is widely recognized as the de facto Green Computing Server Platform.
I would like to highlight a few of the key ways we have improved the energy efficiency of the Ubuntu Server in 9.04.
- In Jaunty, we added pm-utils to the Server seed, and proved that Ubuntu Servers could suspend and hibernate, and be resumed remotely (requires support in the system’s BIOS).
- cpu frequency scaling
- We also enabled cpu frequency scaling on the Ubuntu Server, defaulting to the on-demand governor. This ensures that Ubuntu Servers run at minimum cpu frequency and power consumption when under-utilized.
- pwrkap is an open source project from the IBM Linux Technology Center which provides an energy use monitor and power capping enforcement tools (just what the Slashdot article is discussing!). We partnered with the LTC, working with Darrick Wong, to deliver this technology as a package in Ubuntu Universe.
- powerman is another new package in Jaunty. Here, we worked with Arnaud Quette of Eaton to provide in Ubuntu a tool for manipulating Power Distribution Units (PDUs) from a central location–useful for remote operation in data centers and cluster computing.
- Eucalyptus, EC2
- Finally, we embarked on a tremendous effort to make the Ubuntu Server a better host and client in virtual and cloud computing environments. Cloud, grid, and utility computing, in a generic sense, provide far more scalable resources at the data center level. And being able to move computing efforts between your data center and someone else’s (such as Amazon) also provides some interesting options on the power savings front.
These efforts continue in the development of Karmic 9.10.
9.10 (in progress)
- PowerNap is a new free software project from Canonical that acts as sort of a “screen saver” for servers. Ubuntu Servers running PowerNap can be configured to take a specific action (like suspending, or hibernating, or powering off) when deemed inactive (no keyboard mouse activity, and none of some list of processes running).
- PowerWake is part of the PowerNap project, and provides a command line interface for remotely resuming power napping systems.
- Ubuntu Enterprise Cloud powered by Eucalyptus
- Eucalyptus has been enhanced to leverage PowerNap and PowerWake, to maintain a far more energy efficient cloud. Eucalyptus uses PowerNap to suspend, hibernate, or power-off nodes that are not currently running any virtual machines. New VM requests are served from the available capacity of running systems. Eucalyptus will PowerWake sleeping systems only if load requires. With PowerNap, data centers will finally realize the energy savings promised by cloud computing.
- We are working on packaging Condor for Karmic. Condor is a system that “scavenges” otherwise unused computing cycles, leveraging them for a higher purpose. Think “protein folding” or “SETI@home”, except in your data center, for your grid-capable applications. As such, Condor is less about saving power, but more about increasing utilization and efficiency or your computing resources. You could perhaps choose to PowerNap your under utilized hardware and save energy, or instead Condor your systems and task them to other work.
- We have also synchronized the mpich2 package from Debian, thanks to some excellent work from a few Debian developers. mpich2 is an extremely important library for high performance, grid computing applications. Whereas Condor is intended for general purpose grid computing, mpich2 is used by developers and users of very specific applications. Like Condor, mpich2 is also about using available computing resources as efficiently as possible.
So what’s next? I certainly hope to continue working on energy efficiency in the Ubuntu Server. I have a few ideas about what we could do in 10.04.
- Low Power Architectures
- I have blogged a couple of times now (here, and here), about running the Ubuntu Server on Dell Mini’s. These systems have Intel Atom processors, and run the lpia architecture. I would like to see us work more on this, and perhaps partner with the vendors on an Ubuntu Server product for these architectures.
- Beyond that, ARM is a fascinating architecture, and will have a tremendous effect on the way we think about computing power. ARM based servers, with solid-state disks will soon run on fractions of a watt of power. Some people are excited about laptops that might have 24 hours of battery. I’m excited about Servers that have a 24 hour built-in battery backup, consume 1% of the power of their predecessors, and can fit in nooks and crannies in every room of your house.
July 13, 2009
Mark’s Karmic Koala announcement alluded to this work when he wrote:
A savvy Koala knows that the best way to conserve energy is to go
to sleep, and these days even servers can suspend and resume, so imagine
if we could make it possible to build a cloud computing facility that
drops its energy use virtually to zero by napping in the midday heat,
and waking up when there's work to be done. No need to drink at the
energy fountain when there's nothing going on. If we get all of this
right, our Koala will help take the edge off the bear market.
I have just uploaded PowerNap to Karmic, and we are well on our way to integrating the technology into the 9.10’s Ubuntu Enterprise Cloud.
Actually, I spent last week in sunny Santa Barbara, California working with Dan Nurmi, of Eucalyptus Systems. We shot some amateur digital videos of Ubuntu Karmic Servers, PowerNap/PowerWake, Eucalyptus, and a Watt meter in action. I’ll get those posted soon!
I’ll go into much deeper technical detail on the design and implementation of PowerNap over the next few weeks in subsequent posts, but I’ll give an overview here…
How Does It Work?
An administrator defines a list of regular expressions describing some critical MONITORED_PROCESSES that should be running. When powernapd notices that all of the MONITORED_PROCESSES have been absent from the process table for some configurable ABSENT_SECONDS, powernapd emits a warning to all users of the system that it will run powernap, unless canceled within the next GRACE_SECONDS.
In the Ubuntu Enterprise Cloud case, the configuration file, /etc/powernap/config, might look something like:
MONITORED_PROCESSES = [ "^/usr/bin/kvm " ]
ABSENT_SECONDS = 300
GRACE_SECONDS = 60
Thus, if no instance of kvm runs for 5 minutes, then the system will emit a warning, and powernap after a 1 minute grace period.
Alternatively, a system administrator can force the system to powernap immediately by either running /usr/sbin/powernap directly, or sending powernapd the “now” signal with: service powernap now. In fact, this is what the Eucalyptus Node Controller does, such that it can maintain state, and directly control its managed nodes.
What constitutes a powernap?
So, powernap will first check if /etc/powernap/action is executable, and if so, it will run that file. This will allow you, as an administrator, to run any arbitrary script or program of your design when powernapd determines that your server has become idle. Your script could send an email, for example.
echo "Inefficient server, wasting energy" | mail Al_Gore@example.com
But in the default case, powernap will check if your server supports suspend-to-ram, and if so, it will pm-suspend your system. Otherwise, it will suspend-to-disk, or power the system off, depending on the sleep states supported by your hardware.
Slick, huh? 😀
Beyond the Cloud
While PowerNap is bespoke for the Ubuntu Enterprise Cloud, I have implemented it in manner that I hope is generically useful.
In fact, I’m currently using PowerNap on my Mythbuntu front ends! My configuration currently looks something like:
MONITORED_PROCESSES = [ "mplayer", "vlc", "xine", "mythfrontend.real", "xmms", "gthumb" ]
ABSENT_SECONDS = 240
GRACE_SECONDS = 60
My hardware supports S3 suspend-to-ram, so this is great! If 4 minutes go by, where I’m not running any of my media players (mplayer, vlc, xine, mythfrontend.real, xmms), I’m given a 1 minute grace period, and then my system suspends. I have configured wake-on-usb and wake-on-lan in the BIOS, so I can resume the system in a couple of seconds either by tapping a key or sending a WoL magic packet.
But in the mean time, I’ve reduced the power consumption of 4 systems by 90%, for most of every day while I’m not directly using MythTV!
What about Waking Systems?
Which brings me to PowerNap‘s kid brother…PowerWake. /usr/bin/powerwake is another Python script. This script is designed to be a smarter, remote waking utility. Currently, it supports wake-on-lan, but it will eventually support other mechanisms, such as IPMI, and perhaps NUT.
With respect to wake-on-lan, it’s “smarter” than some other wake-on-lan utilities because it uses a hierarchy of cache files, configuration files, and the current arp table, such that you can wake a system by MAC address, or IP address, or hostname. I find this far more convenient than trying to remember or look up MAC addresses. powerwake respects static configuration in /etc/ethers and maintains a dynamically learning cache of known MAC addresses in /var/cache/powerwake/ethers.
I’m eager to hear what other uses you might have for PowerNap and/or PowerWake for your data centers, basements, and living rooms!
Saving a few Watts,