Adam Leventhal mentioned on twitter that he’d found my umem slab allocator port on macports. I commented that he might miss some of the umem debugging features that are available on its native Solaris platform in mdb, and we dreamed briefly of a portable version of mdb.
This weekend I took a peek at the mdb sources (via the Illumos Gate) to see how hard this might be. As it happens, this isn’t the first time I’ve looked at porting mdb, so I was bit better prepared to go snooping around in there.
It didn’t look much easier to port this time around than last time… there are a number of fairly Solaris specific bits in there that would make it a challenge, and add in that my hobby-hacking platform these days is Darwin (OS/X) with MACH-O binaries rather than ELF meant that it would take a few hours just to get something to partially build.
I decided that what I really wanted wasn’t actually mdb, but a debugging tool that made things easier for me to actually debug with.
Some of my earlier angst in this area resulted in Gimli, which includes “glider”, a pstack-on-steroids that uses DWARF debugging information to provide a deep stack trace and has a C API for building tracer modules to extract more details from a crashing (or even a running) process. The primary purpose of Gimli is to give insight into faulting software on production systems where it may not be possible to run a full interactive debugger for various reasons.
So, what if I took the debugger core from Gimli and wrapped it up in a scripting language that exposed its DWARF assisted introspection capabilities?
I gave it a whirl; I added in Lua as the scripting language because it has a wonderful mixture of ease of embedding, dynamic object binding via metatables and a nice collection of powerful language features.
Here’s an example script that demonstrates some of the debugger capabilities:
Running the script gives output like this on a Cento 5.5 x86_64 system (just a snippet is shown):
The output and facilities are still very early stage, but they hint at the power available under the covers. One thing the main script above doesn’t show is how you can drill into vars; we could do something like this:
This shows how we can index into the structure fields by name; this is all made possible via the DWARF debugging information baked into the object file.
Because this is all wrapped up in a scripting language it’s possible to harness this to automate a debugging session (as demonstrated above),
It is also possible to make this into an interactive debugger–in fact, it already is; you can run ldb with no arguments and it will enter a REPL and let you type in lua expressions.
I think this is really quite a cool concept, and will be following through to flesh this out more fully. What I’m also interested in doing is adding in some more functionality to make it a halfway decent interactive debugger.
My personal debugger needs tend to be focused on inspecting a target process and peeking around in it, stopping and stepping. I don’t tend to use destructive debugger commands (pokes).
I asked Adam what his top features were in mdb, if only because my mdb-fu pales in comparison to his, and he gave me this list:
What I think is achievable in fairly short order is a reasonably flexible interactive shell and the ability to write your own extension functionality in lua.
On top of this framework it would be possible (and potentially very natural and easy) to write extensions that do things like reach into the VM of embedded languages and display augmented stack traces with both the C and the VM trace alongside each other, to replicate the mdb dcmds above, as well as others that work with umem to help diagnose memory issues.
What would you like to see in a debugger?
Bryan Cantrill adds:
Now lldb sounds interesting, but for me it is rather appealing to be able to script this stuff and not have to reach for a C (or in this case, C++) compiler to write a module to get at some guts.
You can find Gimli on BitBucket, complete with the work-in-progress lua branch.
Gimli is BSD licensed.
Comments and suggestions are welcome!