Skip to content

Solaris

Two Factor SSH on Joyent SmartMachines

After reading this scary blog entry about domain hijacking I've been a bit concerned about brute forcing of credentials and have been turning on the two-factor authentication facilities that folks like Google provide for my gmail and personal domains.

I've just found out about Duo Security, a service that allows you to add two-factor authentication to your SSH server, Juniper VPN and even Wordpress blogs. Their service is free for up to 10 users and they start charging when you pass that threshold.

Read on to find out how to set it up.

First impressions of virtualization on Solaris

This article discusses some virtualization options in OpenSolaris. I was hoping to find a "silver bullet" solution for all my needs. I didn't, but it's not too far off.

We have quite a large support matrix for our software; 12 primary OS and architectures, with 4 major installation options. We test those as fresh installs, upgrades, upgrades from the previous major version and uninstalls.

To run all of these on real hardware takes a lot of metal, and we have a golden rule ("thou shalt not have the product installed in your build environment") that means that our engineers needs access to at least 2 copies of each of these during the release build--one to build, and one to test the results of the build.

We've been making use of VMWare server on beefy dual-2-way opteron workstations running Centos 4, but keep running into strangeness with the way that the clocks run in the vms. This leads to occasional stalls of the vm and makes it problematic to test code that is sensitive to timing. VMWare have a twenty-five page whitepaper on the topic of timing in VMs, with no good solution.

This has led us to evaluate some alternatives; Xen on Centos 5 and the large selection of virtualization options on Solaris. This article isn't a Centos vs Solaris comparison, so much as my impression of the state of virtualization on Solaris.

My workstation is running OpenSolaris 2008.05, which is the current distribution available from Sun. This environment installs the whole OS on a zfs filesystem which makes it very easy (and cheap) to manage snapshots of the filesystem (and thus virtual machine images).

The virtualization possibilties open to me on this OS include:

  • Whole OS virtualization via VirtualBox
  • Whole OS virtualization via qemu
  • Whole OS virtualization via Xen
  • Solaris Zones
  • Linux "branded" Zones (BrandZ)
That's quite a few different options, and they have their similarities.

VirtualBox is an application that uses a kernel mode helper to implement virtualization of a complete operating environment. The VirtualBox kernel driver is incompatible with the Xen hypervisor, which means that you can either run VirtualBox or Xen, and you need to reboot to switch between them.

qemu is an application that can optionally use a kernel mode helper, but doesn't require one, to implement virtualization of a complete operating environment. Since it is entirely userspace, it is possible to emulate non-native CPU architectures (such as sparc).

Xen is a special kind of kernel that provides a "hypervisor" to manage machine resources. It can be used to implement hardware virtualization (HVM) or a co-operative virtualization called paravirtualization (PVM). HVM gives you more options for the emulated environment but needs hardware support from your CPU. Opterons tend to have patchy support for HVM (ours don't support it). PVM requires that the emulated environment run a PVM aware kernel, which restricts the guest environment possibilities. (Centos 4 and 5 are the only two platforms that we support that can be run under PVM without jumping through hoops. I hear that Solaris 10 update 6 will support running under PVM).

Each "Whole OS" implementation requires you to set aside a certain amount of RAM and disk for the emulated environment, which means that you can't have all of your emulated environments running at once (unless you have a lot of RAM in your box).

Zones are a special kind of chroot environment that can be configured to inherit various parts of the main OS filesystem and have their own IP address(es) and packages installed. It's very quick and easy to configure a Solaris zone. Since the zone technology is essentially "namespacing" kernel objects, the overhead for zones compared to the other virtualization technologies is extremely low, and the zones can share your machine resources more efficiently.

Linux branded zones are zones that have a system call translation shim enabled. This means that you can run linux binaries in such a zone and the linux syscalls they run get translated to the solaris equivalent. This technology isn't anything terribly new (I even dabbled with something like this for Windows a while back), but it is nicely integrated with the zones feature.
Linux branded zones can emulate the 2.4 linux kernel interface, and you can optionally enable an incomplete 2.6 kernel interface.

For my needs, I'd like to be able to build and test code for RHEL 3, 4 and 5, SuSE 9 and 10 and Solaris 10 amd64. Being able to emulate Solaris 10 sparc is a plus, but not essential (we have zones on a real sparc box that we tend to use for that).

VirtualBox strikes me as being similar to VMWare, which might lead to similar types of problems with the emulation. This put VirtualBox at the bottom of my list. I also really wanted to try Xen, and the thought of having to reboot to switch between the two wasn't very appealing. So I have yet to actually run VirtualBox for anything.

Xen PVM allows me to run our primary linux platforms (Centos 4 and 5) "natively". The hypervisor architecture eliminates the clock problems that we experienced under VMWare. Note that you will need to set your dom0 to store its time in UTC using the command line: 'rtc -z UTC'. You will then need to set your shell TZ variable to reflect your local time zone. You need to make the equivalent configuration in your PVM guest.

Here's the /etc/sysconfig/clock file from one of my centos5 xen vms:

[root@rh5 sysconfig]# cat clock
ZONE="America/New_York"
UTC=true
ARC=false

If you don't do this, you end up with clock skew between the dom0 and your domU which is important if you're using NFS to share a build tree.

I can't run Solaris 10 under Xen at this time, but I can run Solaris 11 as a Zone. Solaris 10 and 11 are not the same but they are pretty close, so it's not a bad solution. If need be, I can access a Solaris 10 zone on real hardware.

Linux branded zones allow me a lightweight approach to running the other platforms that we need to support--since I don't have to pre-allocate ram for branded zones it works out faster and less resource intensive to use a zone to build and run unit tests than a PVM. It's important to keep in mind that the branded zone is really solaris-that-smells-like-linux, so we can't rely solely on this environment for final testing.

Setting up a linux 2.4 branded zone works as advertised. The zone installer didn't grok my centos 3.8 install media, but you can download a 400MB centos 3.7 image to bootstrap your environment. I'd be wary of updating the image, because the zone installation modifies/disables some services that don't make sense to run in the zone.

Setting up a linux 2.6 branded zone isn't officially supported yet, so you need to import your 2.6 linux filesystem image into the zone by creating a tarball on an existing linux system. I ran into two gotchas; the first obvious gotcha was that it was not possible to boot a 64-bit linux image. Switching to a 32-bit linux image worked fine. The second was that the "tar" invocation suggested in the docs causes /usr/include/sys to be omitted from the tarball, so you need to re-install the glibc-headers rpm to fix this. Forewarned, you can craft a better "tar" invocation and avoid this.

Using a combination of Xen and Zones I can cover the main platforms that I'm interested in. ZFS allows me to snapshot and rollback virtual images for testing purposes. We've also invested some time in setting up kickstart files to help with prepping fresh images; combined with a local centos mirror and gigabit networking we can create fresh machines from scratch in a matter of minutes.

I'll be able to run Solaris 10 PVM when update 6 is released, and in the meantime I have a close approximation.

I haven't tried actually running it yet, but I also have the option of running Solaris 10 under qemu, and also the possiblity of running sparc Solaris 10 that way. I expect the performance of this option to be sub-optimal.

I've been running the Xen portions of this setup for about 6 weeks, and the linux branded zones for the past couple of days.

My feeling so far is that it is working out to my satisfaction. Solaris has poor network virtualization support (a solution to that is in the pipeline and not far off), which presented a couple of hurdles, but once you know the problem and the solution it's just a matter of putting the right bits in your configuration, which is mercifully short for both the xen and the zone based vms.


Solaris libumem port on SourceForge

I've had an incredibly busy year so far, having spent the better part of half of it on-site with a customer/partner across the atlantic, and it's only March. In addition to working with them on a large scale deployment proof-of-concept project (I'll blog more about that when I'm sure it's ok to blog about it), we've been hard at work on our Ecelerity 2.1 release, which is just about out-the-door (just some final QA to go).

One of the internals features in our new release is the adoption of the Solaris slab memory allocator, libumem. We already had our own slab allocator, but there are some interesting innovations in libumem that reduce lock contention and cache invalidations that make it attractive for a very high performance multi-threaded application like Ecelerity.

Since Solaris is OpenSourced under the CDDL, we were able to incorporate the allocator into Ecelerity and port it to Linux and Windows and not be forced to open-up our entire source-code. The CDDL requires that we publish the code that we modified, so we set up a project on SourceForge. The code isn't out-of-the-box usable just yet, as it lacks its own configure script and makefile, but it won't take much effort to create those.

Thanks Sun for opening up such good quality code under a commercial software friendly license!

http://sourceforge.net/projects/umem/

  • in Solaris
  • 1 min read

Sun Studio 11 Compiler now available, and it's free

Sun's commercial grade compiler is now available for free on Linux (RHEL and SuSE) and Solaris platforms.

It will be interesting to see how the use of this takes off with the various OpenSource projects; the Sun compiler is pretty good, and should have some nice optimizations on AMD architectures (Sun's new favourite).

In other Solaris news, Intel Pro Wireless 2100 and 2200 chipsets are now supported on OpenSolaris, in addition to Atheros, which has been around for a little bit longer. I have the latter in my laptop and it works flawlessley.

In other-other Solaris news, Oracle announces that Solaris 10 is its preferred platform again.

php dtrace 1.0.3

I added two new probe parameters to the dtrace module; they don't do anything useful with PHP 4, but if you're running PHP 5, they add the classname (arg3) and a scoping operator (arg4) (either "::", "->" or "") that are filled in when making a call to an object or a static call to a class.

This means that you can use the concatentation of arg3, arg4 and arg0 (in that order) for the full name of the method or function being called.

I can probably fill in those things for PHP 4, but I didn't have the time to do that yet; I pushed this release because I'm likely to get too busy to do it otherwise :)

I also refined the build process very slightly; if you're running on 64-bit, you should have better luck.

One of these days (maybe at php|works) I'll write up some more docs on using dtrace. If you're on solaris and want to try it, let me know and I'll give you some pointers. If you're not on solaris, you can't use dtrace :)

  • in Solaris
  • 1 min read

toshiba hotkeys on solaris

Update: moved code to http://bitbucket.org/wez/toshutils/

Friday evening I sat down and wrote my first piece of solaris kernel code and an associated user-space application that activates the brightness up/down hotkeys for the LCD on my Toshiba Satellite M30.

I implemented a tosh_hci driver that can perform Toshiba Hardware Configuration Interface traps via an ioctl(2). This driver is really simple; the hardest part being the intel assembly needed to perform the trap (technically an inb instruction, not a trap).

The userspace code is a really slimmed down version of the code that I previously made available in my patch to the linux acpid. It currently only handles the LCD brightness keys because none of the features that the other hotkeys are supposed to invoke are currently supported by solaris/opensolaris.

Anyhoo, I've made the source available, under the CDDL, in a bundle here for any other toshibans that might like to get a bit more comfort factor back when running solaris.

dtracing PHP on Solaris

[Updated: just wanted to point out that Solaris 10 is free]

One of the things that Theo and myself have been salivating over at OmniTI recently is this really cool tool on Solaris (and OpenSolaris) called DTrace.

DTrace is one of those tools that makes you wonder how you did anything without it before you'd heard of it. What is it? You can think of it as being something like strace that's been exposed to ultimate steroid mix during its conception. Why is it better than strace and similar tools? It's non-invasive, fast, scriptable and extensible.

So, why am I posting about it here? I had the great pleasure of sitting down with Bryan Cantrill, (Solaris kernel developer and one of the guys behind dtrace--a very animated, funny, smart guy) at OSCON, and produced a dtrace provider for PHP.

This is implemented as a PECL extension and can be installed with this simple invocation, if you've built your own php with pear support (recommended): 

   # pear install dtrace

Once installed, you'll want to add a line to your php.ini file: 

   extension=dtrace.so

Once it's loaded, restart your apache server, and you're ready to dtrace. If you run: 

   # dtrace -l | grep php

You'll see a bunch of output like the following, for each apache child process: 

   34412    php9915         dtrace.so                php_dtrace_execute function-return
   34413    php9915         dtrace.so       php_dtrace_execute_internal function-return
   34415    php9915         dtrace.so                php_dtrace_execute function-entry
   34416    php9915         dtrace.so       php_dtrace_execute_internal function-entry

What this shows is that process ID 9915 is running and offers up 4 possible probe points. The probe points wrap around the Zend engine execution routines (php_dtrace_execute and php_dtrace_execute_internal) and provide two classes of probes; function-entry and function-return. What this means is that we can monitor PHP whenever a function is about to be called (function-entry) and when a function has finished being called (function-return).

These probes have 3 parameters; arg0 is the name of the function being called, arg1 is the name of the file from which the call is being made and arg2 is the line number of that file.

Now, let's say that you want to get an idea of which functions are being called in your application; the following dtrace line counts each call; it won't print out any information immediately, as it is sitting there gathering information; run it for a while and then hit CTRL-C and it will spit out the summary information. 

   # dtrace -n function-entry'{@[copyinstr(arg0)] = count()}'

I get this output if I try to set up media wiki: 

    dl                                                                1
    extension_loaded                                                  1
    version_compare                                                   1
    phpversion                                                        1
    ob_implicit_flush                                                 1
    install_version_checks                                            1
    is_writable                                                       1
    error_reporting                                                   1
    is_array                                                          1
    header                                                            1
    strpos                                                            1
    php_sapi_name                                                     1
    function_exists                                                   2
    dirname                                                           2
    ini_set                                                           2
    defined                                                           3
    file_exists                                                       4
    main                                                              9
    define                                                           74

Pretty cool huh? We can immediately see that media wiki is calling define() a LOT across the space of just 2 page loads. If you were looking for things to optimize (and if this wasn't the rarely used setup page), then you've very easily gotten an idea of what's going on. You can then refine your dtrace line to home in on the problem areas.

You can also get an idea of code coverage with this one-liner, which will summarize the calls made, grouping the information by filename, and pretty printing a histogram showing the relative number of calls made from the various lines in your app: 

   # dtrace -n function-entry'{@[copyinstr(arg1)] = lquantize(arg2, 0, 5000)}'

   /export/home/wez/public_html/mediawiki-1.4.7/config/index.php
           value  ------------- Distribution ------------- count
             114 |                                         0
             115 |@@@@@@                                   2
             116 |                                         0
             117 |@@@                                      1
             118 |                                         0
             119 |@@@                                      1
             120 |@@@                                      1
             121 |                                         0

This (abbreviated) output shows you the number of times a particular line of code was visited in config/index.php of media wiki, rendering the relative incidences as ascii-art bars (the Solaris gang are big ascii-art fanatics).

The really really cool thing about this is that it can aggregate the information across all the apache children running on your machine, transparently. The way that dtrace is implemented means that you can even have this module loaded into php permanently on production machines; it has no overhead when you're not running the dtrace command, and very very tiny overhead when you are.

DTrace is a powerful tool for sysadmins and developers alike; I'm looking forward to making heavy use of it in the near future and beyond. I've barely even scratched the surface of the surface here; if you want to learn more, check out Bryan's more detailed "dtrace and php" blog entry, where he shows how to view the complete call stack through php down into the kernel and back (neat!), and the DTrace Community.

DTrace is available on Solaris 10 and up (including OpenSolaris). I recommend experimenting with it, as it will revolutionize the way that you think about debugging and profiling.