Skip to content

Home

Guru - Multiplexing

[The following mini-article is something that I wrote for the International PHP Magazine a while back, as part of the 'ask a guru' column; I'm re-publishing it here because it's useful and because people have asked me about the topic twice in the last two days]

Question:

Is there a way to do a form of threading in PHP?

Say for instance you write a PHP application to monitor a service on a number of servers, it would be nice to be able query a number of servers at the same time rather then query them one-by-one.

Can it be done?

Answer:

People often assume that you need to fork or spawn threads whenever you need to do several things at the same time - and when they realize that PHP doesn't support threading they move on to something less nice, like perl.

The good news is that in the majority of cases you don't need to fork or thread at all, and that you will often get much better performance for not forking/threading in the first place.

Say you need to check up on web servers running on a number of hosts to make sure that they are still responding to the outside world. You might write a script like this: 

<?php
$hosts = array("host1.sample.com", "host2.sample.com", "host3.sample.com");
$timeout = 15;
$status = array();
foreach ($hosts as $host) {
    $errno = 0;
    $errstr = "";
    $s = fsockopen($host, 80, $errno, $errstr, $timeout);
    if ($s) {
        $status[$host] = "Connected\\n";
        fwrite($s, "HEAD / HTTP/1.0\\r\\nHost: $host\\r\\n\\r\\n");
        do {
            $data = fread($s, 8192);
            if (strlen($data) == 0) {
                break;
            }
            $status[$host] .= $data;
        } while (true);
        fclose($s);
    } else {
        $status[$host] = "Connection failed: $errno $errstr\\n";
    }
}
print_r($status);
?>

This works fine, but since fsockopen() doesn't return until it has resolved the hostname and made a successful connection (or waited up to $timeout seconds), extending this script to monitor a larger number of hosts makes it slow to complete.

There is no reason why we have to do it sequentially; we can make asynchronous connections - that is, connections where we don't have to wait for fsockopen to return an opened connection. PHP will still need to resolve the hostname (so its better to use IP addresses), but will return as soon as it has started to open the connection, so that we can move on to the next host.

There are two ways to achieve this; in PHP 5, you can use the new stream_socket_client() function as a drop-in replacement for fsockopen(). In earlier versions of PHP, you need to get your hands dirty and use the sockets extension.

Here's how to do it in PHP 5: 

<?php
$hosts = array("host1.sample.com", "host2.sample.com", "host3.sample.com");
$timeout = 15;
$status = array();
$sockets = array();
/* Initiate connections to all the hosts simultaneously */
foreach ($hosts as $id => $host) {
    $s = stream_socket_client("$host:80", $errno, $errstr, $timeout, 
        STREAM_CLIENT_ASYNC_CONNECT|STREAM_CLIENT_CONNECT);
    if ($s) {
        $sockets[$id] = $s;
        $status[$id] = "in progress";
    } else {
        $status[$id] = "failed, $errno $errstr";
    }
}
/* Now, wait for the results to come back in */
while (count($sockets)) {
    $read = $write = $sockets;
    /* This is the magic function - explained below */
    $n = stream_select($read, $write, $e = null, $timeout);
    if ($n > 0) {
        /* readable sockets either have data for us, or are failed
         * connection attempts */
        foreach ($read as $r) {
            $id = array_search($r, $sockets);
            $data = fread($r, 8192);
            if (strlen($data) == 0) {
                if ($status[$id] == "in progress") {
                    $status[$id] = "failed to connect";
                }
                fclose($r);
                unset($sockets[$id]);
            } else {
                $status[$id] .= $data;
            }
        }
        /* writeable sockets can accept an HTTP request */
        foreach ($write as $w) {
            $id = array_search($w, $sockets);
            fwrite($w, "HEAD / HTTP/1.0\\r\\nHost: "
                . $hosts[$id] .  "\\r\\n\\r\\n");
            $status[$id] = "waiting for response";
        }
    } else {
        /* timed out waiting; assume that all hosts associated
         * with $sockets are faulty */
        foreach ($sockets as $id => $s) {
            $status[$id] = "timed out " . $status[$id];
        }
        break;
    }
}
foreach ($hosts as $id => $host) {
    echo "Host: $host\\n";
    echo "Status: " . $status[$id] . "\\n\\n";
}
?>

We are using stream_select() to wait for events on the sockets that we opened. stream_select() calls the system select(2) function and it works like this: The first three parameters are arrays of streams that you want to work with; you can wait for reading, writing and exceptional events (parameters one, two and three respectively). stream_select() will wait up to $timeout seconds for an event to occur - when it does, it will modify the arrays you passed in to contain the sockets that have met your criteria.

Now, using PHP 4.1.0 and later, if you have compiled in support for ext/sockets, you can use the same script as above, but you need to replace the regular streams/filesystem function calls with their equivalents from ext/sockets. The major difference though is in how we open the connection; instead of stream_socket_client(), you need to use this function: 

<?php
// This value is correct for Linux, other systems have other values
define('EINPROGRESS', 115);
function non_blocking_connect($host, $port, &$errno, &$errstr, $timeout) {
    $ip = gethostbyname($host);
    $s = socket_create(AF_INET, SOCK_STREAM, 0);
    if (socket_set_nonblock($s)) {
        $r = @socket_connect($s, $ip, $port);
        if ($r || socket_last_error() == EINPROGRESS) {
            $errno = EINPROGRESS;
            return $s;
        }
    }
    $errno = socket_last_error($s);
    $errstr = socket_strerror($errno);
    socket_close($s);
    return false;
}
?>

Now, replace stream_select() with socket_select(), fread() with socket_read(), fwrite() with socket_write() and fclose() with socket_close() and you are ready to run the script.

The advantage of the PHP 5 approach is that you can use stream_select() to wait on (almost!) any kind of stream - you can wait for keyboard input from the terminal by including STDIN in your read array for example, and you can also wait for data from pipes created by the proc_open() function.

If you want PHP 4.3.x and want to use the native streams approach, I have prepared a patch that allows fsockopen to work asynchronously. The patch is unsupported and won't be in an official PHP release, however, I've provided a wrapper that implements the stream_socket_client() function along with the patch, so that your code will be forwards compatible with PHP 5.

Resources:

documentation for stream_select()
documentation for socket_select()
patch for PHP 4.3.2 and script to emulate stream_socket_client(). (might work with later 4.3.x versions).

php|tropics coming up

tropics_logo I'm making some final preparations this weekend for my first journey to Mexico and the PHP|Tropics conference.

I'll be speaking on two topics: PDO (of course) and something of an odd-ball talk on PHP's Streams--basically a collection of various handy code snippets and best/worst practices for using streams.

I'm looking forward to the trip; I'll have my electrovaya fully charged for some plane hacking, and be accompanied by my friend, colleague and boss George Schlossnagle--it should be a lot of fun meeting up with the PHP gang again, as well as meeting with the attendees: it's always good meet people that are seriously using PHP.

The location is also going to be nice ;-)

Hope to see you there :)

  • in php
  • 1 min read

OSEvents - spammers?

It saddens (and annoys) me that someone promoting PHP conferences has subscribed me (and presumably other php core developers) to mailing lists containing more information than I ever wanted to know about their events--without my permission.

I don't mind the odd useful email here or there, but I've had enough of them recently to prompt me to take the time to unsubscribe and to blog about it. The chances of me associating myself with them or their conferences, in either attendee or speaker capacity are now pretty damned slim.

Am I just being touchy, or feeling overly self important? I don't think so. It's my inbox; don't sign me up for stuff that I didn't ask for. I consider that really bad behaviour. OS Events--consider your wrists slapped.

  • in Blog
  • 1 min read

rtl8180 under linux 2.6

You need to use ndiswrapper and the driver from the RealTek site; using the driver provided on the CD that came with my cheapo pcmcia card led to a panic on startup.

Update: According to Russel Phillips (some guy I don't know--his words :-) Realtek have recently released a Linux 2.6 driver for this chap. I've not tested this yet, as I'm not sure where I put the pcmcia card, and the ipw2200 driver for my internal wifi works like a champ.

  • in Blog
  • 2 min read

New toys for me

In anticipation of the imminent sale of our house back in England, we decided that some retail therapy was in order this weekend.

My top 3 toys are:

b2c_l_213t-black.jpg.t

Samsung SyncMaster 213T, a 21" flatscreen that looks really nice at 1600x1200. I have the silver version. You can rotate it into portrait mode and work at 1200x1600, which is an interesting new perspective for productivity. You'll need a chair with adjustable height control though, as the extra couple of inches of height are enough to require that you tilt your head back to read the top of the screen.

4e928caa-b20e-442e-a6d1-5dbef92442eb_large.jpg.t

Microsoft Wireless Optical Desktop, Comfort Edition, a not-so-severe take on the "Natual" keyboard idea--makes it much easier to transition between it and a laptop keyboard. If you've ever found yourself tapping the blank space between the two halves of a natural keyboard, you'll appreciate this one.

GCS632Usm.jpg.t

IOGEAR MiniView(tm) Micro USB PLUS KVM Switch an inexpensive USB KVM that allows me to use the keyboard and screen (and audio too, but I don't have that hooked up) with my machine and share it with Juliette's G4.

It all works very nicely, except that the KVM is a little too cheap to realize that the optical desktop needs only a single USB connector for both the mouse and keyboard. Depending on which port you plug into, you can either use only the keyboard (no mouse), or use the keyboard and mouse, but can't use any hotkeys to switch the KVM over to the other machine. My workaround for this was to use the latter, and plug in an older USB keyboard to the other socket and simply double-tap the scroll-lock key on that one when I need to make the switch. I have the second keyboard stowed under the desk but sitting on top of the G4--within easy tapping reach, and away from the desk top.

Why they can't just have a button on the top of the KVM itself eludes me, but at least it works and it didn't cost a fortune.

  • in php
  • 1 min read

FileIterator

<?php
   foreach (file('myfile.txt') as $line) {
       echo $line;
   }
?>

How sexy is that? ;-) (yes, I know, you can't aggregate it)

  • in Blog
  • 1 min read

gmail saga continues

I have 32 left, which means that I've managed to shed around 70 invites over the last few weeks. If you're still interested, let me know.

Update: Argh, up to 50 again.

  • in php
  • 3 min read

PDO on OSX?

A few people have reported problems with trying to run PDO on OSX; I believe them to be general dynamic loading issues in the PHP build system itself. We'll try to resolve them for PHP 5.1. In the meantime, if you're on OSX and want to try PDO, then you can try building it statically; here's how:

Now we need to integrate the PDO extensions into the PHP source tree. Open up a terminal window and cd to where you've downloaded the sources. 

   tar xjf php5-STABLE-latest.tar.bz2
   tar xzf PDO-0.2.2.tgz
   tar xzf PDO_SQLITE-0.2.2.tgz
   mv PDO-0.2.2 php5-STABLE-200502120730/ext/pdo
   mv PDO_SQLITE-0.2.2 php5-STABLE-200502120730/ext/pdo_sqlite
   cd php5-STABLE-200502120730
   rm configure
   ./buildconf --force

At this point, you're ready to build configure PHP. As I'm writing this, I'm testing things out on a G4 450MHz running OSX.2; it's not quite as fast as I'd like, so I'm going for a fairly minimal PHP install, skipping all the xml stuff and using only pdo and pdo sqlite. You should probably leave out the "--disable-all" option when you build it: 

   ./configure --disable-all --enable-debug \\
          --prefix=/usr/local/php-5.0.4 \\
          --enable-cli --enable-pdo \\
          --with-pdo-sqlite

Now, here's a tricky part. The build system in PHP 5.0.x doesn't know that PDO should be initialized before the PDO driver(s), so we need to edit two files to make that happen. For PHP 5.1, you won't need to do this step. Use your favourite editor to open up main/internal_functions.c. Find the part that looks like this: 

    zend_module_entry *php_builtin_extensions[] = {
        phpext_standard_ptr,
        phpext_pdo_sqlite_ptr,
        phpext_pdo_ptr,
    };

You need to change it so it looks like this instead: 

    zend_module_entry *php_builtin_extensions[] = {
        phpext_standard_ptr,
        phpext_pdo_ptr,
        phpext_pdo_sqlite_ptr,
    };

In other words, you need to make sure that the pdo_ptr line is listed before any other pdo_XXX_ptr line. Note that you will probably have a bunch of other extensions listed here; leave the order of those as they are; the important thing is that pdo comes before the other pdo lines.

Repeat this step for main/internal_functions_cli.c

Now you're ready to build: 

    make

And you're done. You can install this if you like, by running "make install"; it will land in the prefix you specified (if you copied me, that will be /usr/local/php-5.0.4), but for the sake of testing, you don't need to do that.

I quickly tested that my build worked by running: 

    ./sapi/cli/php -m

and I saw: 

    [PHP Modules]
    PDO
    pdo_sqlite
    standard

    [Zend Modules]

And a really quick test to make sure that it doesn't blow up straight-away: 

    ./sapi/cli/php -r '$d = new PDO("sqlite::memory:"); debug_zval_dump($d);'

Showed: 

    object(PDO)#1 (0) refcount(2){
    }

Done.

It took a long time to compose this post, mostly due to the processor speed on that box; please put the info to good use!

I'm not an OSX expert, so I can't tell you how to set up your build environment (because I don't remember!), and it's not convenient for me to check things out. If you have OSX specific problems please ask around on the pecl-dev@lists.php.net mailing list. If you run into a crash, use gdb to generate a backtrace and report the bug using the PECL bug tracker.

Good luck!