Skip to content

2008

First impressions of virtualization on Solaris

This article discusses some virtualization options in OpenSolaris. I was hoping to find a "silver bullet" solution for all my needs. I didn't, but it's not too far off.

We have quite a large support matrix for our software; 12 primary OS and architectures, with 4 major installation options. We test those as fresh installs, upgrades, upgrades from the previous major version and uninstalls.

To run all of these on real hardware takes a lot of metal, and we have a golden rule ("thou shalt not have the product installed in your build environment") that means that our engineers needs access to at least 2 copies of each of these during the release build--one to build, and one to test the results of the build.

We've been making use of VMWare server on beefy dual-2-way opteron workstations running Centos 4, but keep running into strangeness with the way that the clocks run in the vms. This leads to occasional stalls of the vm and makes it problematic to test code that is sensitive to timing. VMWare have a twenty-five page whitepaper on the topic of timing in VMs, with no good solution.

This has led us to evaluate some alternatives; Xen on Centos 5 and the large selection of virtualization options on Solaris. This article isn't a Centos vs Solaris comparison, so much as my impression of the state of virtualization on Solaris.

My workstation is running OpenSolaris 2008.05, which is the current distribution available from Sun. This environment installs the whole OS on a zfs filesystem which makes it very easy (and cheap) to manage snapshots of the filesystem (and thus virtual machine images).

The virtualization possibilties open to me on this OS include:

  • Whole OS virtualization via VirtualBox
  • Whole OS virtualization via qemu
  • Whole OS virtualization via Xen
  • Solaris Zones
  • Linux "branded" Zones (BrandZ)
That's quite a few different options, and they have their similarities.

VirtualBox is an application that uses a kernel mode helper to implement virtualization of a complete operating environment. The VirtualBox kernel driver is incompatible with the Xen hypervisor, which means that you can either run VirtualBox or Xen, and you need to reboot to switch between them.

qemu is an application that can optionally use a kernel mode helper, but doesn't require one, to implement virtualization of a complete operating environment. Since it is entirely userspace, it is possible to emulate non-native CPU architectures (such as sparc).

Xen is a special kind of kernel that provides a "hypervisor" to manage machine resources. It can be used to implement hardware virtualization (HVM) or a co-operative virtualization called paravirtualization (PVM). HVM gives you more options for the emulated environment but needs hardware support from your CPU. Opterons tend to have patchy support for HVM (ours don't support it). PVM requires that the emulated environment run a PVM aware kernel, which restricts the guest environment possibilities. (Centos 4 and 5 are the only two platforms that we support that can be run under PVM without jumping through hoops. I hear that Solaris 10 update 6 will support running under PVM).

Each "Whole OS" implementation requires you to set aside a certain amount of RAM and disk for the emulated environment, which means that you can't have all of your emulated environments running at once (unless you have a lot of RAM in your box).

Zones are a special kind of chroot environment that can be configured to inherit various parts of the main OS filesystem and have their own IP address(es) and packages installed. It's very quick and easy to configure a Solaris zone. Since the zone technology is essentially "namespacing" kernel objects, the overhead for zones compared to the other virtualization technologies is extremely low, and the zones can share your machine resources more efficiently.

Linux branded zones are zones that have a system call translation shim enabled. This means that you can run linux binaries in such a zone and the linux syscalls they run get translated to the solaris equivalent. This technology isn't anything terribly new (I even dabbled with something like this for Windows a while back), but it is nicely integrated with the zones feature.
Linux branded zones can emulate the 2.4 linux kernel interface, and you can optionally enable an incomplete 2.6 kernel interface.

For my needs, I'd like to be able to build and test code for RHEL 3, 4 and 5, SuSE 9 and 10 and Solaris 10 amd64. Being able to emulate Solaris 10 sparc is a plus, but not essential (we have zones on a real sparc box that we tend to use for that).

VirtualBox strikes me as being similar to VMWare, which might lead to similar types of problems with the emulation. This put VirtualBox at the bottom of my list. I also really wanted to try Xen, and the thought of having to reboot to switch between the two wasn't very appealing. So I have yet to actually run VirtualBox for anything.

Xen PVM allows me to run our primary linux platforms (Centos 4 and 5) "natively". The hypervisor architecture eliminates the clock problems that we experienced under VMWare. Note that you will need to set your dom0 to store its time in UTC using the command line: 'rtc -z UTC'. You will then need to set your shell TZ variable to reflect your local time zone. You need to make the equivalent configuration in your PVM guest.

Here's the /etc/sysconfig/clock file from one of my centos5 xen vms:

[root@rh5 sysconfig]# cat clock
ZONE="America/New_York"
UTC=true
ARC=false

If you don't do this, you end up with clock skew between the dom0 and your domU which is important if you're using NFS to share a build tree.

I can't run Solaris 10 under Xen at this time, but I can run Solaris 11 as a Zone. Solaris 10 and 11 are not the same but they are pretty close, so it's not a bad solution. If need be, I can access a Solaris 10 zone on real hardware.

Linux branded zones allow me a lightweight approach to running the other platforms that we need to support--since I don't have to pre-allocate ram for branded zones it works out faster and less resource intensive to use a zone to build and run unit tests than a PVM. It's important to keep in mind that the branded zone is really solaris-that-smells-like-linux, so we can't rely solely on this environment for final testing.

Setting up a linux 2.4 branded zone works as advertised. The zone installer didn't grok my centos 3.8 install media, but you can download a 400MB centos 3.7 image to bootstrap your environment. I'd be wary of updating the image, because the zone installation modifies/disables some services that don't make sense to run in the zone.

Setting up a linux 2.6 branded zone isn't officially supported yet, so you need to import your 2.6 linux filesystem image into the zone by creating a tarball on an existing linux system. I ran into two gotchas; the first obvious gotcha was that it was not possible to boot a 64-bit linux image. Switching to a 32-bit linux image worked fine. The second was that the "tar" invocation suggested in the docs causes /usr/include/sys to be omitted from the tarball, so you need to re-install the glibc-headers rpm to fix this. Forewarned, you can craft a better "tar" invocation and avoid this.

Using a combination of Xen and Zones I can cover the main platforms that I'm interested in. ZFS allows me to snapshot and rollback virtual images for testing purposes. We've also invested some time in setting up kickstart files to help with prepping fresh images; combined with a local centos mirror and gigabit networking we can create fresh machines from scratch in a matter of minutes.

I'll be able to run Solaris 10 PVM when update 6 is released, and in the meantime I have a close approximation.

I haven't tried actually running it yet, but I also have the option of running Solaris 10 under qemu, and also the possiblity of running sparc Solaris 10 that way. I expect the performance of this option to be sub-optimal.

I've been running the Xen portions of this setup for about 6 weeks, and the linux branded zones for the past couple of days.

My feeling so far is that it is working out to my satisfaction. Solaris has poor network virtualization support (a solution to that is in the pipeline and not far off), which presented a couple of hurdles, but once you know the problem and the solution it's just a matter of putting the right bits in your configuration, which is mercifully short for both the xen and the zone based vms.


OSCON 2008

OSCON 2008

I'm pleased to announce that I'll be speaking at OSCON again. I have the pleasure of co-presenting an Extending PHP tutorial session with Marcus Boerger, giving a new talk entitled Hot Chocolate: Creating Cocoa apps with PHP, and the tried and true PDO Talk. As always, I'm looking forward to catching up on what's going on outside of my usual stack of software, meeting up with friends and making a visit to my favourite restaurant. I hope to see you there :-)

C and Test Engineer Jobs @ Message Systems

I've got a couple of full-time positions open on my engineering team. We believe in a fun but focused development environment: open-plan, flexible hours, and great benefits. Our customers include Fortune-500 companies, hot startups and tier-1 telecommunications carriers. Our software helps those customers deliver billions of email messages per day.

Email Infrastructure Software Engineer (x2)

I'm looking for one mid-level and one mid-to-senior-level engineer with strong "C" programming skills (3+ years of professional experience). These roles involve design, implementation and testing of our flagship email server product. E-mail encompasses a very broad range of standards and specifications which in turn means that our code base touches on a little bit of everything; it's both interesting and challenging. [Full Job Description]

Gozer (The Destructor)

I'm also looking for someone with a knack for breaking things. This person would be dedicated to dreaming up ways to make the product stress out, panic and fall over, and distilling that abuse into test cases to run in our white box, smoke testing, stress and soak testing environments. This position requires strong Perl skills and 3-5 years industry experience. [Full Job Description]

If you're interested in working with us, please send your resume to me: jobs@messagesystems.com.

  • in php, PDO
  • 4 min read

PDO 2 and CLA

You may be aware that we're starting discussions on the future of PDO; despite being pretty good for many common uses, it isn't perfect, and we want to improve it. One of the items to be discussed is whether we can or should adopt a Contributor License Agreement (CLA), which will make it simpler for the database vendors to work together with us on PDO.

The discussion is aimed chiefly at the "core developer" community, that is, the people that are working on the internals code for PHP, because the CLA would impact how they contribute to PDO. The broader PHP user/developer community would not be affected by a CLA (if we were to go that route), as it would not affect their ability to use PDO in their applications.

If you are wondering what all the fuss is about, you might be interested in reading the transcript of a conversation I had on IRC this evening; you can find it below. It's between myself and a few members of the PHP Community IRC channel on FreeNode (used with their permission!), and I think it does a good job of explaining in fairly simple and somewhat unbiased terms a couple of the main arguments for and against the CLA. I'm not saying that this is all there is to it, just that these are likely to be the main points that the core developers need to discuss first.

It would be premature to say that you are for or against PDO 2 at this stage because we are yet to define what PDO 2 will actually be; that is the purpose of the discussion on the PDO mailing list.

If, after reading this, you have questions or comments of your own, then please read the email that Andi and myself put together, take a look at the FAQ, and if your question is still unanswered, join the discussion on the PDO mailing list (read the archives first!) Browse it via the web, over NNTP, or subscribe via email.

We'll be working to update the FAQ to reflect questions and answers that keep coming up. It's important to direct questions and comments to the mailing list, so that we have an archive of the discussion and a single thing to keep track of.

Here is the transcript from IRC; the initial question was in response to blog entries from Antony and Pierre, and some questions are based on content from our initial email to the PDO list. 

21:59 <vluther> Wez: so why would a CLA be bad?


22:00 <Wez> vluther: two arguments are 1) it actually increases the risk potential
            for some companies that have a patent portfolio (Rasmus and Adam
            brought this up on internals a little while back)


22:01 <vluther> but that puts the company at risk, not php right?


22:01 <Wez> vluther: 2) The contributor community might be segmented into people
            that can/will sign the CLA and people that won't, which could be bad
            for the community (but also might not matter)


22:02 <Wez> vluther: yeah, it puts the company at risk, not PHP, which means that
            it makes if difficult/impossible for people that work for those companies
            to contribute


22:03 <vluther> ahh, so if we force a cla.. then we risk losing contributors.. got it


22:03 <vortex`> does it work the other way, no CLA no commercial support?


22:04 <Wez> vluther: yeah, that seems to be the main argument against it really.
            Whether that has a tangible impact is something to discuss


22:04 <Wez> vortex`: without a CLA, the amount of support that a vendor can provide
            would be limited, because it is difficult for them to share their work
            with the other vendors (they're in competition)


22:04 <vluther> Wez: so the say no movement is based purely out of fear of what may
                happen?


22:04 <Wez> vluther: yep


22:08 <coldflame_> can i ask why it's been important for php to _not_ have a legal
                   entity?


22:09 <Wez> coldflame_: it's not "important to not have one" so much as impossible to
            retro-fit one.  To do so would require tracking down everyone that ever
            contributed and sign off on transferring that work to the legal entity


22:09 <coldflame_> Wez: ahh.. thanks


22:10 <Wez> so PHP as a whole could never (in any practical sense of the word) be put
            under CLA


22:10 <coldflame_> Wez: is there a list of main vendors?


22:11 <Wez> and that was another of the fears that people had about PDO being under a
            CLA; that it would "leak" out to affect PHP as a whole


22:11 <Wez> coldflame_: Andi, Ilia and myself have been talking to MySQL, PostgreSQL,
            IBM, Oracle and Microsoft


22:11 <Wez> all of which want to contribute to PDO


22:11 <vortex`> nice :D


22:12 <Wez> the problem is doing so in such a way that the companies feel happy about
            Working together, and in such a way that doesn't cause problems with the
            php developer community


22:12 <Wez> and that's what I want to get people discussing on the pdo mailing list


22:13 <coldflame_> Wez: oh cool, and the last of my n00b questions is that this CLA is
                   Meant  only for PDO yer? not to be confused with the one for PHP?


22:13 <Wez> correct, we don't want and are not suggesting that PHP go under CLA
            (it's impossible anyway), just that putting PDO under CLA will let us get
            the most out of the vendors


22:14 <coldflame_> Wez: thanks for your clear and concise answers :)


22:14 <Wez> coldflame_: you're welcome.


22:15 <vortex`> coldflame_: just judging how good pdo1 is, pdo2 should be a cracker :D


22:15 <coldflame_> Wez: good luck :) i look forward to hearing about it


22:15  * coldflame_ signs up to pdo list

Please don't comment on this blog entry, discuss on the pdo mailing list instead!

Message Systems, Inc.

At the start of this year, we spun off the email product side of OmniTI into its own entity, Message Systems, Inc. This marks another step on the road to dominating the world with our awesome software.

I've also changed roles; I'm now the Director of Engineering at Message Systems. I'm looking forward to see what challenges are in store for me, and will try hard to avoid adopting too much suit talk (I've already found myself using a few phrases that would have made me cringe last year).

What does this mean for me and PHP? Despite the increased responsibility, I think it should actually give me a bit more PHP time than I've had in the past (I'll have more control over my destiny). I should still be able to attend PHP related events, and I still deal with PHP (we use it for the management GUI in the product).

What about OmniTI? Well, we're still part of the family and share office space, jokes and good times at our HQ in Maryland.

PHP London 2008

I was scheduled to appear at PHP London 2008, but due to unforeseen circumstances, I've had to cancel my trip and back out from the conference. I don't like doing this, but unfortunately don't have much of a choice. Thankfully, the PHP London folks have managed to find replacement speakers for the two sessions that I was going to give.

If you're going to be in or around London on the leap day (February 29th), or are within commutable distance, then you might consider attending the conference; it's a one day conference with a number of expert speakers from the PHP Community. If you sign up now, the early bird rate is only GBP 90. Find out more at their web site.

I was really looking forward to this conference, and I'm sorry that I'm going to miss it; I hope you have fun!