Skip to content

Home

Wanted: Senior Backend Web Engineer

I'm looking for a senior-level engineer to help extend the web-based dashboards for our messaging infrastructure platform.

This position reports directly to me.

While we don't face the traditional scale-to-the-world types of problems in our web UI, we do face some difficult UI and data scalability challenges.

If you're interested, please contact me via jobs@messagesystems.com.

Job Description

You will enjoy in-depth exposure to all aspects of building scalable applications as part of a small, internationally-renowned team that helps our customers -- service providers and hot startups alike -- achieve their cutting-edge messaging goals.

Message Systems is a global leader in messaging, with a global customer base including tier-1 telecommunications carriers, email service providers and online enterprises. Our technologies help transit billions of messages per day for customers worldwide, while offering unmatched flexibility and architectural clarity. The company combines the best aspects of Open Source, startup, and established company in our business goals and corporate culture.

Message Systems believes in a fun but focused development environment. We have flexible hours, outstanding benefits, and a world-class team. Our engineering group works in an open plan and uses the Scrum development methodology with a focus on collaboration, automation, and testing.

Requirements

This position involves leading the backend design and architecture for our dashboards. Qualified candidates should have current expertise developing robust web applications that can scale to large datasets.

Candidates should have strong PHP skills, at least 5 years of professional programming experience, and experience and good understanding of:

  • UNIX (Linux, Solaris or FreeBSD) based PHP development
  • Apache
  • Database design and admin (PostgreSQL preferred)
  • The use of PDO for database access
  • Web application security and best practices
  • Strong HTML, CSS and JavaScript skills
  • AJAX, implementation and debugging
  • Knowledge of Firebug and similar tools
  • Experience writing unit and regression tests
  • Selenium experience a strong plus
  • Perl skills a strong plus
  • Good Statistical mathematics a strong plus

Responsibilities

Feature Development - Work with the product team to define feature specifications for the web-based components of our platform, and take a leading role in their implementation to production quality, complete with positive and negative test cases.

Testing - Testing is a primary focus in our development process, and every engineer is responsible for augmenting our automated test suite to guarantee test coverage on new and existing features.

Documentation - Ensuring documentation, code maintainability and product feature coherency.

This position is full-time in Columbia, MD.

We're only considering candidates with existing US work eligibility.

Looking for a Trainer

We at Message Systems are looking for someone to fill the position of "Technical Trainer / Curriculum Developer".

This position designs, develops and delivers online and in-person instructor led trainings on the Message Systems products and services for a variety of audiences including customers, partners and internal staff. You will design courses and instructional material for everything ranging from short informational quick-starts to multi-day workshops.

The audience for these training programs will primarily be mid-senior level Unix administrators, messaging anti-abuse operations staff, systems architects and other technical staff engaged in designing and maintaining corporate and carrier messaging infrastructure.

This is a full-time position based in Columbia, MD.

If you're interested, or know someone that might be, please read (or send them to read) the full job description.

I'll be at ZendCon 2008

Busy times here mean that I'm leaving it a bit late to say this, but I'll be at ZendCon this year too. I'll be giving the usual talk on PDO, but my main reason for attending this year is to sync up with other PHP folks and talk about where PHP is at and where it's going.

ZendCon has been consistently good, and I look forward to attending again this year... see you there?

  • in Blog
  • 3 min read

Virtualization on OSX

I'm about to go on the road again and I've been getting my laptop updated to make sure I can function without internet access. For me that means that I need a linux environment. I've been using Parallels for this because it was the only option when I first got my macbook, and I'm not terribly impressed with its ability to run linux virtual machines.

First I have to say that my preferred usage for vms is to disable as much graphical UI as possible and login using the terminal; I want to avoid any excess resource usage because I'm on a laptop and I want better battery life.

Here's my gripe list:

  • poweroff spins the cpus up to 100% or more utilization and doesn't actually power the machine off.
    The reason? ACPI is only supported for vista guests. I'm rather bemused by this statement, because the whole point of ACPI is to virtualize certain types of hardware access--it should not be targeted to a particular OS.
  • Parallels Tools requires X to run.
    You can manually run the daemon but it spins the CPU trying to open the display. This means that you can't get time synchronization with the host unless you want to load your CPU
  • Shared folder performance sucks
    Mounting the host filesystem over NFS is faster, but kernel panics OSX (the latter is probably an OSX bug)

Outside of these issues, it's not bad though. I'm rather disappointed about the level of Linux support from Parallels--I had all the same problems a year ago and nothing seems to have changed. It's clear that their priority is in making the Windows VM experience nice and integrated, and that's their perogative.I've also tried VMWare Fusion, both the current stable and the beta versions; here's my gripe list:




  • lvm and vmware fusion appear to not like each other.
    Sometimes on reboot the vm filesystem is corrupt, especially the rpm database, and the image needs to be tossed and reinstalled without lvm. This is problematic because the default install for centos is lvm based.
  • vmware fusion freezes and can't be killed, can't be restarted.
    The resolution is a reboot of the laptop, which isn't reliable--powercycling is required.

I like vmware (I've been using it for many years), but it's not a happy camper on my laptop; I've uninstalled it.I've also tried VitualBox, and it's pretty good, but not perfect; gripes:




  • Only supports NAT networking, with no locally visible IP/network.
    This means that I can't ssh into individual machines by IP and have to set up port forwarding to get into them from my terminal.
  • Setting up port fowarding requires 3 long and tedious command invocations for each port

Some positives for VirtualBox:


  • ACPI support appears to be very complete
    The GUI even allows you to distinguish between an ACPI power off request and yanking out the power cord
  • SATA controller emulation
    This is faster than IDE/ATA emulation, which is all that Parallels offers. VMWare offers SCSI as an option, but that's a non-starter for me currently.
  • You can run VirtualBox vms completely headless and optionally export the console display using Remote Desktop
  • It's free to download and run

I'm sticking with Parallels for the time being; I think that VirtualBox might become my favourite once they've beefed up the networking support on OSX.I'll leave you with a couple of performance tips that should apply to any virtualization software:




  • Use fixed size virtual disks in preference to dynamically expanding disks. This will improve filesystem performance
  • Linux kernels by default have a high timer frequency that can torture the emulation and cause it have higher CPU load.
    If it makes sense for your vm, you can rebuild the kernel to use a lower frequency.
    If you're using centos, grab one of these pre-compiled kernels and reboot.
    This resulted in a drop from 30% CPU utilization when idle to 7% for me in Parallels, and a less significant drop in VirtualBox.

Slides: PHP Streams

Here are the slides from my Streams talk; they cover a variety of bits and pieces of streams background and implementation that may or may not be useful to you.

SlideShare | View

Slides: PHP and COM

This slide deck is from php|works 2004. There's a lot of material in the speaker notes, which I've painstakingly pasted into the comment on the slideshare representation (wouldn't it be cool if it could automatically do that?).

SlideShare | View

Just in case slideshare vanishes, the PowerPoint version is also available: PHP and COM

Slides: PDO

[I've just noticed that the omniti.com re-design broke the various links from my blog to the slides I had been storing there. So I'm trying out slideshare; I'll be revisiting the slides I've given in the past and blogging one entry per presentation]

Here are the extended slides from my PDO talk. When I first put this talk together it was for a long hour slot, but conference sessions started to diminish in length and I had to pull out certain slides to avoid running over every time.

SlideShare | View

Just in case slideshare vanishes, the PDF version is also available: PDO

First impressions of virtualization on Solaris

This article discusses some virtualization options in OpenSolaris. I was hoping to find a "silver bullet" solution for all my needs. I didn't, but it's not too far off.

We have quite a large support matrix for our software; 12 primary OS and architectures, with 4 major installation options. We test those as fresh installs, upgrades, upgrades from the previous major version and uninstalls.

To run all of these on real hardware takes a lot of metal, and we have a golden rule ("thou shalt not have the product installed in your build environment") that means that our engineers needs access to at least 2 copies of each of these during the release build--one to build, and one to test the results of the build.

We've been making use of VMWare server on beefy dual-2-way opteron workstations running Centos 4, but keep running into strangeness with the way that the clocks run in the vms. This leads to occasional stalls of the vm and makes it problematic to test code that is sensitive to timing. VMWare have a twenty-five page whitepaper on the topic of timing in VMs, with no good solution.

This has led us to evaluate some alternatives; Xen on Centos 5 and the large selection of virtualization options on Solaris. This article isn't a Centos vs Solaris comparison, so much as my impression of the state of virtualization on Solaris.

My workstation is running OpenSolaris 2008.05, which is the current distribution available from Sun. This environment installs the whole OS on a zfs filesystem which makes it very easy (and cheap) to manage snapshots of the filesystem (and thus virtual machine images).

The virtualization possibilties open to me on this OS include:

That's quite a few different options, and they have their similarities.

VirtualBox is an application that uses a kernel mode helper to implement virtualization of a complete operating environment. The VirtualBox kernel driver is incompatible with the Xen hypervisor, which means that you can either run VirtualBox or Xen, and you need to reboot to switch between them.

qemu is an application that can optionally use a kernel mode helper, but doesn't require one, to implement virtualization of a complete operating environment. Since it is entirely userspace, it is possible to emulate non-native CPU architectures (such as sparc).

Xen is a special kind of kernel that provides a "hypervisor" to manage machine resources. It can be used to implement hardware virtualization (HVM) or a co-operative virtualization called paravirtualization (PVM). HVM gives you more options for the emulated environment but needs hardware support from your CPU. Opterons tend to have patchy support for HVM (ours don't support it). PVM requires that the emulated environment run a PVM aware kernel, which restricts the guest environment possibilities. (Centos 4 and 5 are the only two platforms that we support that can be run under PVM without jumping through hoops. I hear that Solaris 10 update 6 will support running under PVM).

Each "Whole OS" implementation requires you to set aside a certain amount of RAM and disk for the emulated environment, which means that you can't have all of your emulated environments running at once (unless you have a lot of RAM in your box).

Zones are a special kind of chroot environment that can be configured to inherit various parts of the main OS filesystem and have their own IP address(es) and packages installed. It's very quick and easy to configure a Solaris zone. Since the zone technology is essentially "namespacing" kernel objects, the overhead for zones compared to the other virtualization technologies is extremely low, and the zones can share your machine resources more efficiently.

Linux branded zones are zones that have a system call translation shim enabled. This means that you can run linux binaries in such a zone and the linux syscalls they run get translated to the solaris equivalent. This technology isn't anything terribly new (I even dabbled with something like this for Windows a while back), but it is nicely integrated with the zones feature.
Linux branded zones can emulate the 2.4 linux kernel interface, and you can optionally enable an incomplete 2.6 kernel interface.

For my needs, I'd like to be able to build and test code for RHEL 3, 4 and 5, SuSE 9 and 10 and Solaris 10 amd64. Being able to emulate Solaris 10 sparc is a plus, but not essential (we have zones on a real sparc box that we tend to use for that).

VirtualBox strikes me as being similar to VMWare, which might lead to similar types of problems with the emulation. This put VirtualBox at the bottom of my list. I also really wanted to try Xen, and the thought of having to reboot to switch between the two wasn't very appealing. So I have yet to actually run VirtualBox for anything.

Xen PVM allows me to run our primary linux platforms (Centos 4 and 5) "natively". The hypervisor architecture eliminates the clock problems that we experienced under VMWare. Note that you will need to set your dom0 to store its time in UTC using the command line: 'rtc -z UTC'. You will then need to set your shell TZ variable to reflect your local time zone. You need to make the equivalent configuration in your PVM guest.

Here's the /etc/sysconfig/clock file from one of my centos5 xen vms:

[root@rh5 sysconfig]# cat clock
ZONE="America/New_York"
UTC=true
ARC=false

If you don't do this, you end up with clock skew between the dom0 and your domU which is important if you're using NFS to share a build tree.

I can't run Solaris 10 under Xen at this time, but I can run Solaris 11 as a Zone. Solaris 10 and 11 are not the same but they are pretty close, so it's not a bad solution. If need be, I can access a Solaris 10 zone on real hardware.

Linux branded zones allow me a lightweight approach to running the other platforms that we need to support--since I don't have to pre-allocate ram for branded zones it works out faster and less resource intensive to use a zone to build and run unit tests than a PVM. It's important to keep in mind that the branded zone is really solaris-that-smells-like-linux, so we can't rely solely on this environment for final testing.

Setting up a linux 2.4 branded zone works as advertised. The zone installer didn't grok my centos 3.8 install media, but you can download a 400MB centos 3.7 image to bootstrap your environment. I'd be wary of updating the image, because the zone installation modifies/disables some services that don't make sense to run in the zone.

Setting up a linux 2.6 branded zone isn't officially supported yet, so you need to import your 2.6 linux filesystem image into the zone by creating a tarball on an existing linux system. I ran into two gotchas; the first obvious gotcha was that it was not possible to boot a 64-bit linux image. Switching to a 32-bit linux image worked fine. The second was that the "tar" invocation suggested in the docs causes /usr/include/sys to be omitted from the tarball, so you need to re-install the glibc-headers rpm to fix this. Forewarned, you can craft a better "tar" invocation and avoid this.

Using a combination of Xen and Zones I can cover the main platforms that I'm interested in. ZFS allows me to snapshot and rollback virtual images for testing purposes. We've also invested some time in setting up kickstart files to help with prepping fresh images; combined with a local centos mirror and gigabit networking we can create fresh machines from scratch in a matter of minutes.

I'll be able to run Solaris 10 PVM when update 6 is released, and in the meantime I have a close approximation.

I haven't tried actually running it yet, but I also have the option of running Solaris 10 under qemu, and also the possiblity of running sparc Solaris 10 that way. I expect the performance of this option to be sub-optimal.

I've been running the Xen portions of this setup for about 6 weeks, and the linux branded zones for the past couple of days.

My feeling so far is that it is working out to my satisfaction. Solaris has poor network virtualization support (a solution to that is in the pipeline and not far off), which presented a couple of hurdles, but once you know the problem and the solution it's just a matter of putting the right bits in your configuration, which is mercifully short for both the xen and the zone based vms.