Skip to content


mtrack: a software development tracker + wiki

[Updated to add IRC and Google Groups links]

I don't know if it's just me, or whether everyone in software development finds issue tracking software frustrating and/or broken in some way.  They're all either way too complicated to set up, configure or use (the Bugzilla's or the Jira's), or have annoying "features" (such as Trac's you-lose-your-edits-if-someone-else-changed-something).

We've been using Trac at Message Systems for several years now and have been enjoying its pragmatic approach of keeping the interface simple but expressive; just enough structure to be helpful but not too much that it intrudes.  We've added/modified a couple of plug-ins to it to help track time and draw some graphs, but it has otherwise served us well.

However, we've got a couple of projects that have started to converge and overlap and it's frustrating to visit the two different portals to interact and stay on top of things.  As we scale up our development teams even further (we continue to have bigger and bigger plans!) this will prove to be more widely frustrating.

Enter mtrack; on one hand it's a clone of many of Trac's features (possible due to their pragmatic BSD license), but on the other it has some refinements in terms of its workflow.  What's important to me is that it is built to work with multiple code repositories and allows breaking out information on a per project basis.  It also tries hard to avoid losing your wiki or ticket edits if someone else updates things while you're working.

I chose to implement mtrack in PHP rather than continuing to extend Trac in python.  My primary reason for this is that I've seen web apps written with Perl and Python, and while there are certainly guru developers out there that can build some awesome apps, those languages don't really lend themselves to web development and that limits the scope of potential contributors.  While I can probably persuade some of my colleagues to code python web bits, I'll find it much easier to persuade more of them to code PHP web bits.  I'm sure this property extends outside of our organization.

We've been running our engineering department on mtrack for a couple of weeks now (after a period of running it in parallel to Trac) with just under 20 or so people using it actively as part of their day-to-day activities, so I feel comfortable in making this blog post to solicit input from the broader community, both in terms of suggestions and bug reports as well as potential contributions.

While I am soliciting feedback, I'd like to emphasize that this is still relatively young and that there is no support and no warranty; if you are the cautious type, this is not the time to try mtrack.

And now for some Anticipated Questions to pre-empt FAQ:

Where can I get mtrack?

How is mtrack licensed?

What are some of the key features in mtrack?

  • Trac style wiki syntax (all of its core syntax is supported)
  • Subversion and Mercurial repository integration (API extensible to other VCS)
  • Email notifications batch related information together to reduce the amount of mail you need to read; if you make 5 changes relating to a ticket in a 5 minute period you get one email with all 5 changes in it, instead of 1 per change.  Code changes that reference a ticket generate a single email with the ticket changes and the diff, not a separate email for each.
  • Issue tracking with configurable components, priorities, severities, milestones.
  • Define "projects" and associate components and paths within repositories with them to control who gets notified of changes.
  • Flot based graphing of Scrum burn-down based on time estimates and recorded effort
  • Wiki edits have in-built conflict resolution and the Issue tracker tries hard to behave reasonably in the case where multiple people are working on the same item.
  • Uses the Gravatar service to add some personality throughout the application.
  • Includes an importer to migrate your Trac project(s) into mtrack.
  • Has a plugin/hook API to support both core and external enhancements.
  • Uses the Zend Framework Lucene implementation for full-text search capabilities. 

Why didn't you use <Name of Framework>?

  • Frameworks can increase development overhead and I wanted to get this done.  Also, if I had used a framework, I would have to defend my choice against the others ;-). I am not morally opposed to contributions that adopt a framework, but would want to carefully assess the impact on longer term maintenance before doing so.

How do I install mtrack?

How can I contribute to mtrack?

  • By trying it out; I'm looking forward to hearing suggestions, comments and bug reports.  Please file them in the issue tracker: and keep in mind that you may not get a response until the following weekend (or even longer!) due to my busy schedule
  • By fixing bugs; if you've found and resolved an issue and want to pass a patch back to me you can do so either by attaching the patch or details on how to get at the patch to the issue tracker.  I recommend doing this by keeping your own fork on bitbucket as it makes it much more convenient for both of us.
  • By building out and sharing new features; I'm very open to adding new features.  My preference is to enable those by making the core modular enough that new features don't have to be part of the main mtrack code to be used by it.  This will keep the the core light and make it easier to share modules.
  • By fleshing out docs.  I'm the first to admit that mtrack could do with more documentation.
  • Join in the discussion in the mtrack mailing list or in the #mtrack IRC channel on freenode.

You started this article claiming that all other issue trackers suck, but I think mtrack needs improvement

  • I'm sure it does; it is still early days and I welcome your feedback (see above!)

Wanted: Senior Backend Web Engineer

I'm looking for a senior-level engineer to help extend the web-based dashboards for our messaging infrastructure platform.

This position reports directly to me.

While we don't face the traditional scale-to-the-world types of problems in our web UI, we do face some difficult UI and data scalability challenges.

If you're interested, please contact me via

Job Description

You will enjoy in-depth exposure to all aspects of building scalable applications as part of a small, internationally-renowned team that helps our customers -- service providers and hot startups alike -- achieve their cutting-edge messaging goals.

Message Systems is a global leader in messaging, with a global customer base including tier-1 telecommunications carriers, email service providers and online enterprises. Our technologies help transit billions of messages per day for customers worldwide, while offering unmatched flexibility and architectural clarity. The company combines the best aspects of Open Source, startup, and established company in our business goals and corporate culture.

Message Systems believes in a fun but focused development environment. We have flexible hours, outstanding benefits, and a world-class team. Our engineering group works in an open plan and uses the Scrum development methodology with a focus on collaboration, automation, and testing.


This position involves leading the backend design and architecture for our dashboards. Qualified candidates should have current expertise developing robust web applications that can scale to large datasets.

Candidates should have strong PHP skills, at least 5 years of professional programming experience, and experience and good understanding of:

  • UNIX (Linux, Solaris or FreeBSD) based PHP development
  • Apache
  • Database design and admin (PostgreSQL preferred)
  • The use of PDO for database access
  • Web application security and best practices
  • Strong HTML, CSS and JavaScript skills
  • AJAX, implementation and debugging
  • Knowledge of Firebug and similar tools
  • Experience writing unit and regression tests
  • Selenium experience a strong plus
  • Perl skills a strong plus
  • Good Statistical mathematics a strong plus


Feature Development - Work with the product team to define feature specifications for the web-based components of our platform, and take a leading role in their implementation to production quality, complete with positive and negative test cases.

Testing - Testing is a primary focus in our development process, and every engineer is responsible for augmenting our automated test suite to guarantee test coverage on new and existing features.

Documentation - Ensuring documentation, code maintainability and product feature coherency.

This position is full-time in Columbia, MD.

We're only considering candidates with existing US work eligibility.

Looking for a Trainer

We at Message Systems are looking for someone to fill the position of "Technical Trainer / Curriculum Developer".

This position designs, develops and delivers online and in-person instructor led trainings on the Message Systems products and services for a variety of audiences including customers, partners and internal staff. You will design courses and instructional material for everything ranging from short informational quick-starts to multi-day workshops.

The audience for these training programs will primarily be mid-senior level Unix administrators, messaging anti-abuse operations staff, systems architects and other technical staff engaged in designing and maintaining corporate and carrier messaging infrastructure.

This is a full-time position based in Columbia, MD.

If you're interested, or know someone that might be, please read (or send them to read) the full job description.

First impressions of virtualization on Solaris

This article discusses some virtualization options in OpenSolaris. I was hoping to find a "silver bullet" solution for all my needs. I didn't, but it's not too far off.

We have quite a large support matrix for our software; 12 primary OS and architectures, with 4 major installation options. We test those as fresh installs, upgrades, upgrades from the previous major version and uninstalls.

To run all of these on real hardware takes a lot of metal, and we have a golden rule ("thou shalt not have the product installed in your build environment") that means that our engineers needs access to at least 2 copies of each of these during the release build--one to build, and one to test the results of the build.

We've been making use of VMWare server on beefy dual-2-way opteron workstations running Centos 4, but keep running into strangeness with the way that the clocks run in the vms. This leads to occasional stalls of the vm and makes it problematic to test code that is sensitive to timing. VMWare have a twenty-five page whitepaper on the topic of timing in VMs, with no good solution.

This has led us to evaluate some alternatives; Xen on Centos 5 and the large selection of virtualization options on Solaris. This article isn't a Centos vs Solaris comparison, so much as my impression of the state of virtualization on Solaris.

My workstation is running OpenSolaris 2008.05, which is the current distribution available from Sun. This environment installs the whole OS on a zfs filesystem which makes it very easy (and cheap) to manage snapshots of the filesystem (and thus virtual machine images).

The virtualization possibilties open to me on this OS include:

  • Whole OS virtualization via VirtualBox
  • Whole OS virtualization via qemu
  • Whole OS virtualization via Xen
  • Solaris Zones
  • Linux "branded" Zones (BrandZ)
That's quite a few different options, and they have their similarities.

VirtualBox is an application that uses a kernel mode helper to implement virtualization of a complete operating environment. The VirtualBox kernel driver is incompatible with the Xen hypervisor, which means that you can either run VirtualBox or Xen, and you need to reboot to switch between them.

qemu is an application that can optionally use a kernel mode helper, but doesn't require one, to implement virtualization of a complete operating environment. Since it is entirely userspace, it is possible to emulate non-native CPU architectures (such as sparc).

Xen is a special kind of kernel that provides a "hypervisor" to manage machine resources. It can be used to implement hardware virtualization (HVM) or a co-operative virtualization called paravirtualization (PVM). HVM gives you more options for the emulated environment but needs hardware support from your CPU. Opterons tend to have patchy support for HVM (ours don't support it). PVM requires that the emulated environment run a PVM aware kernel, which restricts the guest environment possibilities. (Centos 4 and 5 are the only two platforms that we support that can be run under PVM without jumping through hoops. I hear that Solaris 10 update 6 will support running under PVM).

Each "Whole OS" implementation requires you to set aside a certain amount of RAM and disk for the emulated environment, which means that you can't have all of your emulated environments running at once (unless you have a lot of RAM in your box).

Zones are a special kind of chroot environment that can be configured to inherit various parts of the main OS filesystem and have their own IP address(es) and packages installed. It's very quick and easy to configure a Solaris zone. Since the zone technology is essentially "namespacing" kernel objects, the overhead for zones compared to the other virtualization technologies is extremely low, and the zones can share your machine resources more efficiently.

Linux branded zones are zones that have a system call translation shim enabled. This means that you can run linux binaries in such a zone and the linux syscalls they run get translated to the solaris equivalent. This technology isn't anything terribly new (I even dabbled with something like this for Windows a while back), but it is nicely integrated with the zones feature.
Linux branded zones can emulate the 2.4 linux kernel interface, and you can optionally enable an incomplete 2.6 kernel interface.

For my needs, I'd like to be able to build and test code for RHEL 3, 4 and 5, SuSE 9 and 10 and Solaris 10 amd64. Being able to emulate Solaris 10 sparc is a plus, but not essential (we have zones on a real sparc box that we tend to use for that).

VirtualBox strikes me as being similar to VMWare, which might lead to similar types of problems with the emulation. This put VirtualBox at the bottom of my list. I also really wanted to try Xen, and the thought of having to reboot to switch between the two wasn't very appealing. So I have yet to actually run VirtualBox for anything.

Xen PVM allows me to run our primary linux platforms (Centos 4 and 5) "natively". The hypervisor architecture eliminates the clock problems that we experienced under VMWare. Note that you will need to set your dom0 to store its time in UTC using the command line: 'rtc -z UTC'. You will then need to set your shell TZ variable to reflect your local time zone. You need to make the equivalent configuration in your PVM guest.

Here's the /etc/sysconfig/clock file from one of my centos5 xen vms:

[root@rh5 sysconfig]# cat clock

If you don't do this, you end up with clock skew between the dom0 and your domU which is important if you're using NFS to share a build tree.

I can't run Solaris 10 under Xen at this time, but I can run Solaris 11 as a Zone. Solaris 10 and 11 are not the same but they are pretty close, so it's not a bad solution. If need be, I can access a Solaris 10 zone on real hardware.

Linux branded zones allow me a lightweight approach to running the other platforms that we need to support--since I don't have to pre-allocate ram for branded zones it works out faster and less resource intensive to use a zone to build and run unit tests than a PVM. It's important to keep in mind that the branded zone is really solaris-that-smells-like-linux, so we can't rely solely on this environment for final testing.

Setting up a linux 2.4 branded zone works as advertised. The zone installer didn't grok my centos 3.8 install media, but you can download a 400MB centos 3.7 image to bootstrap your environment. I'd be wary of updating the image, because the zone installation modifies/disables some services that don't make sense to run in the zone.

Setting up a linux 2.6 branded zone isn't officially supported yet, so you need to import your 2.6 linux filesystem image into the zone by creating a tarball on an existing linux system. I ran into two gotchas; the first obvious gotcha was that it was not possible to boot a 64-bit linux image. Switching to a 32-bit linux image worked fine. The second was that the "tar" invocation suggested in the docs causes /usr/include/sys to be omitted from the tarball, so you need to re-install the glibc-headers rpm to fix this. Forewarned, you can craft a better "tar" invocation and avoid this.

Using a combination of Xen and Zones I can cover the main platforms that I'm interested in. ZFS allows me to snapshot and rollback virtual images for testing purposes. We've also invested some time in setting up kickstart files to help with prepping fresh images; combined with a local centos mirror and gigabit networking we can create fresh machines from scratch in a matter of minutes.

I'll be able to run Solaris 10 PVM when update 6 is released, and in the meantime I have a close approximation.

I haven't tried actually running it yet, but I also have the option of running Solaris 10 under qemu, and also the possiblity of running sparc Solaris 10 that way. I expect the performance of this option to be sub-optimal.

I've been running the Xen portions of this setup for about 6 weeks, and the linux branded zones for the past couple of days.

My feeling so far is that it is working out to my satisfaction. Solaris has poor network virtualization support (a solution to that is in the pipeline and not far off), which presented a couple of hurdles, but once you know the problem and the solution it's just a matter of putting the right bits in your configuration, which is mercifully short for both the xen and the zone based vms.

C and Test Engineer Jobs @ Message Systems

I've got a couple of full-time positions open on my engineering team. We believe in a fun but focused development environment: open-plan, flexible hours, and great benefits. Our customers include Fortune-500 companies, hot startups and tier-1 telecommunications carriers. Our software helps those customers deliver billions of email messages per day.

Email Infrastructure Software Engineer (x2)

I'm looking for one mid-level and one mid-to-senior-level engineer with strong "C" programming skills (3+ years of professional experience). These roles involve design, implementation and testing of our flagship email server product. E-mail encompasses a very broad range of standards and specifications which in turn means that our code base touches on a little bit of everything; it's both interesting and challenging. [Full Job Description]

Gozer (The Destructor)

I'm also looking for someone with a knack for breaking things. This person would be dedicated to dreaming up ways to make the product stress out, panic and fall over, and distilling that abuse into test cases to run in our white box, smoke testing, stress and soak testing environments. This position requires strong Perl skills and 3-5 years industry experience. [Full Job Description]

If you're interested in working with us, please send your resume to me:

Message Systems, Inc.

At the start of this year, we spun off the email product side of OmniTI into its own entity, Message Systems, Inc. This marks another step on the road to dominating the world with our awesome software.

I've also changed roles; I'm now the Director of Engineering at Message Systems. I'm looking forward to see what challenges are in store for me, and will try hard to avoid adopting too much suit talk (I've already found myself using a few phrases that would have made me cringe last year).

What does this mean for me and PHP? Despite the increased responsibility, I think it should actually give me a bit more PHP time than I've had in the past (I'll have more control over my destiny). I should still be able to attend PHP related events, and I still deal with PHP (we use it for the management GUI in the product).

What about OmniTI? Well, we're still part of the family and share office space, jokes and good times at our HQ in Maryland.