If you’re running Karmic and you use KVM, I’d appreciate your help testing the qemu-kvm package.

sudo apt-get install qemu-kvm

And then just use KVM as you normally do. Please file bugs at:

• https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+filebug

Thanks,
ustin

I was asked recently how much energy savings administrators might actually expect from an Ubuntu Enterprise Cloud, powered by Eucalyptus and using PowerNap.

I have blogged before about how much I enjoy mathematics and statistical analysis, and this is in fact another exciting, hard question!

The question was presented as the following hypothetical…

Given 10 quad-core servers in a cloud running 40 virtual machines. 20 virtual machines are simultaneously discarded. What power savings would you expect from PowerNap?

This is actually a rather complex combinatorics problem. I’ll restate it as such:

Given 10 buckets, each of which can hold 0 – 4 items, how many ways can 20 items be distributed among those 10 buckets? Furthermore, what is the statistical distribution of empty buckets?

It’s been 10 years since my university statistics classes, and my brain melted trying to derive the permutation formula. But a big thanks to my good friend Matt Dobson for solving that! If you’re interested, you can view his combinatorics formula here.

It took me about an hour to hack a Python script that could empirically solve this problem by brute force, generating a comprehensive list of all of the permutations. You can now download and run /usr/bin/powernap_calculator provided by the powernap package in Karmic.

Here are the results running on the parameters above:

$ time powernap_calculator –hosts 10 –guests-per-host 4 –guests 20
Calculating…99%

In a cloud with [10] hosts, which can handle [4] guests-per-host, currently running [20] guests,
you may expect the following:

[ 5.2%] likely that [0/10] of your hosts would powernap, for a [0%] power savings
[ 27.5%] likely that [1/10] of your hosts would powernap, for a [10%] power savings
[ 42.5%] likely that [2/10] of your hosts would powernap, for a [20%] power savings
[ 21.8%] likely that [3/10] of your hosts would powernap, for a [30%] power savings
[ 2.9%] likely that [4/10] of your hosts would powernap, for a [40%] power savings
[ <1%] likely that [5/10] of your hosts would powernap, for a [50%] power savings

The overall expected value is [19.0%] power savings.

real 0m46.919s
user 0m46.227s
sys 0m0.276s

So at this snapshot in time, when your cloud suddenly dropped to 50% utilization, the expected value is a 19% power savings. Note that expected value is a very specific statistics term. Wikipedia says:

In probability theory and statistics, the expected value (or expectation value, or mathematical expectation, or mean, or first moment) of a random variable is the integral of the random variable with respect to its probability measure. For discrete random variables this is equivalent to the probability-weighted sum of the possible values, and for continuous random variables with a density function it is the probability density-weighted integral of the possible values.

Right, so we might expect about a 19% power savings for this random moment in time, when we simultaneously reduce our utilization by 50%.

However, if we observe the cloud over time, and with perhaps more realistic usage patterns, the distribution should be much better than random. VMs will come and go at more staggered intervals than simultaneous destruction of half the instances.

And you will have another tremendous factor working in your favor — Eucalyptus’ greedy scheduling algorithm. This ensures that each time a new VM is launched, it will land on a system that’s already running. This is known as an annealing system — one that’s constantly correcting itself — since under-utilized systems will automatically powernap, and new VMs will fill in the gaps on running hardware. Pretty cool, I think.

So I’m curious…

• What does (or would) your cloud look like?
• How many -h|–hosts, -p|–guests-per-host, and -g|–guests do you usually have?
• What does powernap_calculator say about your parameters?
• Post your results!

Note that the powernap_calculator algorithm is exponential, O((p+1)^h), so large values of (p, h) will take a very long time to compute! I’m totally open to code review of powernap_calculator and any performance enhancements

In my cloud, I have 8 dual-core hosts. I typically limit my usage to 2 guests-per-host. And I rarely run more than 4 VMs at a time.

$ powernap_calculator -h 8 -p 2 -g 4
In a cloud with [8] hosts, which can handle [2] guests-per-host, currently running [4] guests,
you may expect the following power savings:
[ 26.3%] likely that [4/8] of your hosts would powernap, for a [50%] power savings
[ 63.2%] likely that [5/8] of your hosts would powernap, for a [62%] power savings
[ 10.5%] likely that [6/8] of your hosts would powernap, for a [75%] power savings
The overall expected value is [60.5%] power savings.

My servers run at about 80 Watts fully loaded. My electricity is about $0.10 per kilowatt-hour. So a year’s worth of electricity with all 8 servers running all of the time is 8 * .08 KW * 24 hr/day * 365 days/year * $0.10/KW-hr = $560/year.

I like the prospects of saving approximately 60.5% of that, or $339/year!

So how does the calculator work?

Let’s use a slightly smaller example: hosts,h=4, guests-per-host,p=2, guests,g=3

Since each system can hold 0, 1, or 2 virtual machines, we’re going to use a base-3 number (which is p+1). And we’re going to generate all possible 4-digit base-3 numbers (which is (p+1)^h). In our case, that’s 3⁴ or 81 scenarios to consider. For each of those scenarios, we convert the decimal integer to a list of each of the digits 0-2, and sum the list, throwing out any “invalid scenarios”, where the sum does not add up to the target number of guests, 3, g. Our valid scenarios is actually much smaller, the following 16:

[2, 1, 0, 0]
[1, 2, 0, 0]
[2, 0, 1, 0]
[1, 1, 1, 0]
[0, 2, 1, 0]
[1, 0, 2, 0]
[0, 1, 2, 0]
[2, 0, 0, 1]
[1, 1, 0, 1]
[0, 2, 0, 1]
[1, 0, 1, 1]
[0, 1, 1, 1]
[0, 0, 2, 1]
[1, 0, 0, 2]
[0, 1, 0, 2]
[0, 0, 1, 2]

Now, of these, we are interested in how many 0’s are in each row, as this indicates a host that has no guests, and can therefore be powernapped.

Our calculations yield: [0, 4, 12, 0, 0], or:

• 0 possible scenarios have no unused hosts
• 4 possible scenarios have 1 unused hosts
• 12 possible scenarios have 2 unused hosts
• 0 possible scenarios have 3 unused hosts
• 0 possible scenarios have 4 unused hosts

From this we can generate the following probabilities:

[ 25.0%] likely that [1/4] of your hosts would powernap, for a [25%] power savings
[ 75.0%] likely that [2/4] of your hosts would powernap, for a [50%] power savings

And weighting these probabilities, we can generate the expected value:

25%*.25 + 50%*.75 = 43.75%
The overall expected value is [43.8%] power savings.

Comments appreciated!

ustin

Do you want to be a rock star in your LoCo or LUG?

Give a presentation or demonstration on Byobu!

The hard work is already done, as I have created a set of slides for you, available under the Creative Commons Attribution-Share Alike 3.0 license.

There’s also a screencast available at YouTube:

• http://www.youtube.com/watch?v=4rBF8byfyvo

I know a couple of people have done something like this with their groups in the past few months, and I thought I’d help out by providing some screenshots and documentation.

Enjoy,
ustin

I just read this article, referenced by Slashdot: Power Capping the Data Center.

And I thought I’d borrow this video (one of my favorites) from IBM:

I think that we in the Ubuntu Server Community are leading the charge in developing an energy-efficient server class Linux distribution.

It is my hope that by the 10.04 LTS release, the Ubuntu Server is widely recognized as the de facto Green Computing Server Platform.

I would like to highlight a few of the key ways we have improved the energy efficiency of the Ubuntu Server in 9.04.

9.04

• suspend/hibernate/resume
  • In Jaunty, we added pm-utils to the Server seed, and proved that Ubuntu Servers could suspend and hibernate, and be resumed remotely (requires support in the system’s BIOS).
• cpu frequency scaling
  • We also enabled cpu frequency scaling on the Ubuntu Server, defaulting to the on-demand governor. This ensures that Ubuntu Servers run at minimum cpu frequency and power consumption when under-utilized.
• pwrkap
  • pwrkap is an open source project from the IBM Linux Technology Center which provides an energy use monitor and power capping enforcement tools (just what the Slashdot article is discussing!). We partnered with the LTC, working with Darrick Wong, to deliver this technology as a package in Ubuntu Universe.
• powerman
  • powerman is another new package in Jaunty. Here, we worked with Arnaud Quette of Eaton to provide in Ubuntu a tool for manipulating Power Distribution Units (PDUs) from a central location–useful for remote operation in data centers and cluster computing.
• Eucalyptus, EC2
  • Finally, we embarked on a tremendous effort to make the Ubuntu Server a better host and client in virtual and cloud computing environments. Cloud, grid, and utility computing, in a generic sense, provide far more scalable resources at the data center level. And being able to move computing efforts between your data center and someone else’s (such as Amazon) also provides some interesting options on the power savings front.

These efforts continue in the development of Karmic 9.10.

9.10 (in progress)

• powernap
  
  • PowerNap is a new free software project from Canonical that acts as sort of a “screen saver” for servers. Ubuntu Servers running PowerNap can be configured to take a specific action (like suspending, or hibernating, or powering off) when deemed inactive (no keyboard mouse activity, and none of some list of processes running).
• powerwake
  
  • PowerWake is part of the PowerNap project, and provides a command line interface for remotely resuming power napping systems.
• Ubuntu Enterprise Cloud powered by Eucalyptus
  
  • Eucalyptus has been enhanced to leverage PowerNap and PowerWake, to maintain a far more energy efficient cloud. Eucalyptus uses PowerNap to suspend, hibernate, or power-off nodes that are not currently running any virtual machines. New VM requests are served from the available capacity of running systems. Eucalyptus will PowerWake sleeping systems only if load requires. With PowerNap, data centers will finally realize the energy savings promised by cloud computing.
• Condor
  • We are working on packaging Condor for Karmic. Condor is a system that “scavenges” otherwise unused computing cycles, leveraging them for a higher purpose. Think “protein folding” or “SETI@home”, except in your data center, for your grid-capable applications. As such, Condor is less about saving power, but more about increasing utilization and efficiency or your computing resources. You could perhaps choose to PowerNap your under utilized hardware and save energy, or instead Condor your systems and task them to other work.
• mpich2
  
  • We have also synchronized the mpich2 package from Debian, thanks to some excellent work from a few Debian developers. mpich2 is an extremely important library for high performance, grid computing applications. Whereas Condor is intended for general purpose grid computing, mpich2 is used by developers and users of very specific applications. Like Condor, mpich2 is also about using available computing resources as efficiently as possible.

So what’s next? I certainly hope to continue working on energy efficiency in the Ubuntu Server. I have a few ideas about what we could do in 10.04.

10.04 (future)

• Low Power Architectures
  • I have blogged a couple of times now (here, and here), about running the Ubuntu Server on Dell Mini’s. These systems have Intel Atom processors, and run the lpia architecture. I would like to see us work more on this, and perhaps partner with the vendors on an Ubuntu Server product for these architectures.
  • Beyond that, ARM is a fascinating architecture, and will have a tremendous effect on the way we think about computing power. ARM based servers, with solid-state disks will soon run on fractions of a watt of power. Some people are excited about laptops that might have 24 hours of battery. I’m excited about Servers that have a 24 hour built-in battery backup, consume 1% of the power of their predecessors, and can fit in nooks and crannies in every room of your house.

ustin

A big thanks to everyone who has been testing the backport of Jaunty’s kvm-84 to Intrepid and Hardy.

I have backported a bit more of the virtualization stack from Jaunty to Intrepid and Hardy, namely:

• libvirt 0.6.1

Packages are available in the ~ubuntu-virt PPA:

• https://launchpad.net/~ubuntu-virt/+archive/ppa

I’m hoping some of you out there might be willing and interested to do some testing on these new packages. They fix a couple of bugs for me, including installation of Windows guests on Hardy hosts.

Please raise issues and discuss problems in Launchpad and IRC, rather than comments below.

Cheers,
ustin

Ladies and gentlemen, I’m pleased to announce an exciting, new open source project from Canonical for the Ubuntu Server — PowerNap!

Mark’s Karmic Koala announcement alluded to this work when he wrote:

A savvy Koala knows that the best way to conserve energy is to goto sleep, and these days even servers can suspend and resume, so imagineif we could make it possible to build a cloud computing facility thatdrops its energy use virtually to zero by napping in the midday heat,and waking up when there's work to be done. No need to drink at theenergy fountain when there's nothing going on. If we get all of thisright, our Koala will help take the edge off the bear market.

I have just uploaded PowerNap to Karmic, and we are well on our way to integrating the technology into the 9.10’s Ubuntu Enterprise Cloud.

Actually, I spent last week in sunny Santa Barbara, California working with Dan Nurmi, of Eucalyptus Systems. We shot some amateur digital videos of Ubuntu Karmic Servers, PowerNap/PowerWake, Eucalyptus, and a Watt meter in action. I’ll get those posted soon!

I’ll go into much deeper technical detail on the design and implementation of PowerNap over the next few weeks in subsequent posts, but I’ll give an overview here…

How Does It Work?

PowerNap operates sort of like a screen saver for servers. Besides watching the console and terminals for keyboard activity, it also watches the system’s process table for activity.

An administrator defines a list of regular expressions describing some critical MONITORED_PROCESSES that should be running. When powernapd notices that all of the MONITORED_PROCESSES have been absent from the process table for some configurable ABSENT_SECONDS, powernapd emits a warning to all users of the system that it will run powernap, unless canceled within the next GRACE_SECONDS.

Sample Configuration

In the Ubuntu Enterprise Cloud case, the configuration file, /etc/powernap/config, might look something like:

MONITORED_PROCESSES = [ "^/usr/bin/kvm " ]ABSENT_SECONDS = 300GRACE_SECONDS = 60

Thus, if no instance of kvm runs for 5 minutes, then the system will emit a warning, and powernap after a 1 minute grace period.

PowerNap Now!

Alternatively, a system administrator can force the system to powernap immediately by either running /usr/sbin/powernap directly, or sending powernapd the “now” signal with: service powernap now. In fact, this is what the Eucalyptus Node Controller does, such that it can maintain state, and directly control its managed nodes.

What constitutes a powernap?

So, powernap will first check if /etc/powernap/action is executable, and if so, it will run that file. This will allow you, as an administrator, to run any arbitrary script or program of your design when powernapd determines that your server has become idle. Your script could send an email, for example.

echo "Inefficient server, wasting energy" | mail Al_Gore@example.com

But in the default case, powernap will check if your server supports suspend-to-ram, and if so, it will pm-suspend your system. Otherwise, it will suspend-to-disk, or power the system off, depending on the sleep states supported by your hardware.

Slick, huh?

Beyond the Cloud

While PowerNap is bespoke for the Ubuntu Enterprise Cloud, I have implemented it in manner that I hope is generically useful.

In fact, I’m currently using PowerNap on my Mythbuntu front ends! My configuration currently looks something like:

MONITORED_PROCESSES = [ "mplayer", "vlc", "xine", "mythfrontend.real", "xmms", "gthumb" ]ABSENT_SECONDS = 240GRACE_SECONDS = 60

My hardware supports S3 suspend-to-ram, so this is great! If 4 minutes go by, where I’m not running any of my media players (mplayer, vlc, xine, mythfrontend.real, xmms), I’m given a 1 minute grace period, and then my system suspends. I have configured wake-on-usb and wake-on-lan in the BIOS, so I can resume the system in a couple of seconds either by tapping a key or sending a WoL magic packet.

But in the mean time, I’ve reduced the power consumption of 4 systems by 90%, for most of every day while I’m not directly using MythTV!

What about Waking Systems?

Which brings me to PowerNap’s kid brother…PowerWake. /usr/bin/powerwake is another Python script. This script is designed to be a smarter, remote waking utility. Currently, it supports wake-on-lan, but it will eventually support other mechanisms, such as IPMI, and perhaps NUT.

With respect to wake-on-lan, it’s “smarter” than some other wake-on-lan utilities because it uses a hierarchy of cache files, configuration files, and the current arp table, such that you can wake a system by MAC address, or IP address, or hostname. I find this far more convenient than trying to remember or look up MAC addresses. powerwake respects static configuration in /etc/ethers and maintains a dynamically learning cache of known MAC addresses in /var/cache/powerwake/ethers.

Interesting?

I’m eager to hear what other uses you might have for PowerNap and/or PowerWake for your data centers, basements, and living rooms!

Saving a few Watts,
ustin

So the backport of KVM-84 to Hardy and Intrepid has been in the works since March, and we’re now rounding 3rd base.

I’ve produced a couple of release candidates and fixed a few remaining issues. Thank you to everyone who has tested these packages and provided feedback.

The final step before releasing the backport is to ensure that these latest changes get uploaded to jaunty-updates, such that the package is in sync among Hardy, Intrepid, and Jaunty.

One more call for testing…

So I’ve been working hard on this, and I’m at a point where I require assistance from the community. I get emails on a weekly basis from people asking for advice on getting involved in Ubuntu. Here’s your shot

There is a package in jaunty-proposed that needs to be pushed to jaunty-updates before the Hardy and Intrepid backports can take place. In order to promote the package to jaunty-updates, I need users to verify that the new package fixes the four bugs that I think it fixes, and does not cause regressions.

Please, if you have a system running Jaunty + KVM, give the -proposed package a shot, and provide feedback in the following 4 bugs:

• 392295
• 359447
• 394953
• 382077

ustin

At UDS Karmic, we discussed providing daily builds of pristine upstream projects.

Specifically, I implemented this for the key virtualization packages, per https://blueprints.launchpad.net/ubuntu/+spec/server-karmic-pristine-daily-virt-builds:

• qemu
• qemu-kvm
• libvirt

Ubuntu users are often running some version of the project older than the current development branch, it’s more interesting to know if the user can reproduce the bug on the latest development sources.

I have constructed some automation that builds Debian/Ubuntu binaries for amd64, i386, and lpia platforms directly from your master git repositories every day at 11:00 UTC, and publishes them in a special, opt-in repository. These daily builds also drop any specific patches we may be carrying, in order to minimize the delta between the binaries built and your upstream sources.

Additionally, this tool also has the ability to expose build breakages, and it has revealed several during the last few weeks as I have been developing it.

Cheers,
ustin

QEMU provides the userspace and device emulation required by KVM. As such, it’s an extremely important project to Ubuntu’s virtualization and cloud computing efforts.

The upstream project has not had an actively maintained bug tracker for quite some time.

I recently helped QEMU’s maintainer, Anthony Liguori setup a Launchpad project for tracking QEMU’s bugs. This should be a good thing for the upstream QEMU project, as well as for tracking bugs in Ubuntu’s kvm and qemu packages.

See:
http://bugs.launchpad.net/qemu

I have also set up a bzr mirror of qemu’s git tree, for people who are more bzr-inclined.

You can now:

bzr branch lp:qemu

Also, you can use Loggerhead to browse QEMU’s source tree and revision history at:
http://bazaar.launchpad.net/~qemu/qemu/git/files

Cheers,
ustin

Here’s a neat trick that I find phenomenally useful…

I like to run KVM virtual machines inside of Byobu, using KVM’s -curses option. From QEMU’s manpage:

Normally, QEMU uses SDL to display the VGA output. With this option, QEMU can display the VGA output when in text mode using a curses/ncurses interface. Nothing is displayed in graphical mode.

So this only works with non-graphical virtual machines, such as the Ubuntu Server. But hey, that’s what I’m working on every day. Here’s a quick demo screen cast.

Commands used in this video:

1. start byobu
2. run kvm -curses karmic-server.img to launch one virtual machine
3. hit F8 to rename this window karmic
4. hit F2 to open a new window
5. start a second virtual machine, and rename that window
6. hit F3 or F4 to move back and forth between windows
7. hit F6 to detach
8. and byobu -x to re-attach

The detach/reattach is really cool, as these virtual machines will continue running in the background. Many people use this sort of method to background an irc client such as irssi, which allows it to serve as a persistently connected proxy.

Of course, virsh and virt-manager are the preferred methods to manage virtual machines in Ubuntu, but I find this useful for my development purposes.

Cheers,
ustin