First impressions of virtualization on Solaris
This article discusses some virtualization options in OpenSolaris. I was hoping to find a "silver bullet" solution for all my needs. I didn't, but it's not too far off.
We have quite a large support matrix for our software; 12 primary OS and architectures, with 4 major installation options. We test those as fresh installs, upgrades, upgrades from the previous major version and uninstalls.
To run all of these on real hardware takes a lot of metal, and we have a golden rule ("thou shalt not have the product installed in your build environment") that means that our engineers needs access to at least 2 copies of each of these during the release build--one to build, and one to test the results of the build.
We've been making use of VMWare server on beefy dual-2-way opteron workstations running Centos 4, but keep running into strangeness with the way that the clocks run in the vms. This leads to occasional stalls of the vm and makes it problematic to test code that is sensitive to timing. VMWare have a twenty-five page whitepaper on the topic of timing in VMs, with no good solution.
This has led us to evaluate some alternatives; Xen on Centos 5 and the large selection of virtualization options on Solaris. This article isn't a Centos vs Solaris comparison, so much as my impression of the state of virtualization on Solaris.
My workstation is running OpenSolaris 2008.05, which is the current distribution available from Sun. This environment installs the whole OS on a zfs filesystem which makes it very easy (and cheap) to manage snapshots of the filesystem (and thus virtual machine images).
The virtualization possibilties open to me on this OS include:
- Whole OS virtualization via VirtualBox
- Whole OS virtualization via qemu
- Whole OS virtualization via Xen
- Solaris Zones
- Linux "branded" Zones (BrandZ)
VirtualBox is an application that uses a kernel mode helper to implement virtualization of a complete operating environment. The VirtualBox kernel driver is incompatible with the Xen hypervisor, which means that you can either run VirtualBox or Xen, and you need to reboot to switch between them.
qemu is an application that can optionally use a kernel mode helper, but doesn't require one, to implement virtualization of a complete operating environment. Since it is entirely userspace, it is possible to emulate non-native CPU architectures (such as sparc).
Xen is a special kind of kernel that provides a "hypervisor" to manage machine resources. It can be used to implement hardware virtualization (HVM) or a co-operative virtualization called paravirtualization (PVM). HVM gives you more options for the emulated environment but needs hardware support from your CPU. Opterons tend to have patchy support for HVM (ours don't support it). PVM requires that the emulated environment run a PVM aware kernel, which restricts the guest environment possibilities. (Centos 4 and 5 are the only two platforms that we support that can be run under PVM without jumping through hoops. I hear that Solaris 10 update 6 will support running under PVM).
Each "Whole OS" implementation requires you to set aside a certain amount of RAM and disk for the emulated environment, which means that you can't have all of your emulated environments running at once (unless you have a lot of RAM in your box).
Zones are a special kind of chroot environment that can be configured to inherit various parts of the main OS filesystem and have their own IP address(es) and packages installed. It's very quick and easy to configure a Solaris zone. Since the zone technology is essentially "namespacing" kernel objects, the overhead for zones compared to the other virtualization technologies is extremely low, and the zones can share your machine resources more efficiently.
Linux branded zones are zones that have a system call translation shim enabled. This means that you can run linux binaries in such a zone and the linux syscalls they run get translated to the solaris equivalent. This technology isn't anything terribly new (I even dabbled with something like this for Windows a while back), but it is nicely integrated with the zones feature.
Linux branded zones can emulate the 2.4 linux kernel interface, and you can optionally enable an incomplete 2.6 kernel interface.
For my needs, I'd like to be able to build and test code for RHEL 3, 4 and 5, SuSE 9 and 10 and Solaris 10 amd64. Being able to emulate Solaris 10 sparc is a plus, but not essential (we have zones on a real sparc box that we tend to use for that).
VirtualBox strikes me as being similar to VMWare, which might lead to similar types of problems with the emulation. This put VirtualBox at the bottom of my list. I also really wanted to try Xen, and the thought of having to reboot to switch between the two wasn't very appealing. So I have yet to actually run VirtualBox for anything.
Xen PVM allows me to run our primary linux platforms (Centos 4 and 5) "natively". The hypervisor architecture eliminates the clock problems that we experienced under VMWare. Note that you will need to set your dom0 to store its time in UTC using the command line: 'rtc -z UTC'. You will then need to set your shell TZ variable to reflect your local time zone. You need to make the equivalent configuration in your PVM guest.
Here's the /etc/sysconfig/clock file from one of my centos5 xen vms:
[root@rh5 sysconfig]# cat clock
ZONE="America/New_York"
UTC=true
ARC=false
If you don't do this, you end up with clock skew between the dom0 and your domU which is important if you're using NFS to share a build tree.
I can't run Solaris 10 under Xen at this time, but I can run Solaris 11 as a Zone. Solaris 10 and 11 are not the same but they are pretty close, so it's not a bad solution. If need be, I can access a Solaris 10 zone on real hardware.
Linux branded zones allow me a lightweight approach to running the other platforms that we need to support--since I don't have to pre-allocate ram for branded zones it works out faster and less resource intensive to use a zone to build and run unit tests than a PVM. It's important to keep in mind that the branded zone is really solaris-that-smells-like-linux, so we can't rely solely on this environment for final testing.
Setting up a linux 2.4 branded zone works as advertised. The zone installer didn't grok my centos 3.8 install media, but you can download a 400MB centos 3.7 image to bootstrap your environment. I'd be wary of updating the image, because the zone installation modifies/disables some services that don't make sense to run in the zone.
Setting up a linux 2.6 branded zone isn't officially supported yet, so you need to import your 2.6 linux filesystem image into the zone by creating a tarball on an existing linux system. I ran into two gotchas; the first obvious gotcha was that it was not possible to boot a 64-bit linux image. Switching to a 32-bit linux image worked fine. The second was that the "tar" invocation suggested in the docs causes /usr/include/sys to be omitted from the tarball, so you need to re-install the glibc-headers rpm to fix this. Forewarned, you can craft a better "tar" invocation and avoid this.
Using a combination of Xen and Zones I can cover the main platforms that I'm interested in. ZFS allows me to snapshot and rollback virtual images for testing purposes. We've also invested some time in setting up kickstart files to help with prepping fresh images; combined with a local centos mirror and gigabit networking we can create fresh machines from scratch in a matter of minutes.
I'll be able to run Solaris 10 PVM when update 6 is released, and in the meantime I have a close approximation.
I haven't tried actually running it yet, but I also have the option of running Solaris 10 under qemu, and also the possiblity of running sparc Solaris 10 that way. I expect the performance of this option to be sub-optimal.
I've been running the Xen portions of this setup for about 6 weeks, and the linux branded zones for the past couple of days.
My feeling so far is that it is working out to my satisfaction. Solaris has poor network virtualization support (a solution to that is in the pipeline and not far off), which presented a couple of hurdles, but once you know the problem and the solution it's just a matter of putting the right bits in your configuration, which is mercifully short for both the xen and the zone based vms.