Doing Many Things, Like pings

Randal L. Schwartz

As a UNIX systems administrator, I'm often faced with those little mundane tasks that seem so trivial to me but so important to the community I'm supporting. Little things like "hey, is that host up and responding to pings?". Such tasks generally have a very repetitive nature to them, and scripting them seems to be the only way to have time to concentrate on the tasks that really need my attention.

Let's look at the specific task of pinging a number of hosts on a subnet. Now, there are tools to do this quickly (like nmap), and there are even Perl modules to perform the ping (as in Net::Ping), but I wanted to focus on something familar that can be launched from Perl as an external process, and the system C<ping> command seems mighty appropriate for that.

First, let's look at how to ping one host, on my BSD-ish system:

sub ping_a_host {
  my $host = shift;
  'ping -i 1 -c 1 $host 2>/dev/null' =~ /0 packets rec/ ? 0 : 1;
}

The output is scanned for the string 0 packets rec, which if absent means we got a good ping. So if the match is found, we return 0 (the ping was bad); otherwise, we'll return 1. The ping command spits out some diagnostics on standard error, which we'll toss using Bourne-shell syntax.

Note that the value of $host is not checked here for sanity. We certainly wouldn't want to accept a random command-line parameter or (gasp) a Web form value here without some serious validation. However, as we use this in our program, all of the values will be internally generated, so we've got some degree of safety.

To scan a particular subnet, looking for hosts that are alive, we would add to that subroutine something like:

print "ping $_ is ", ping_a_host($_), "\n"
  for map "10.0.1.$_", 1..254;

So, how can we speed that up? This is not a CPU-intensive loop: practically the entire time is spent waiting for some remote host to respond. We'll leave the ping_a_host subroutine alone, because that's not where we have a problem -- it's doing its job as fast as it can. So we need to do more of these at a time.

One approach is to fork a separate process for each host we want to ping. We'll then sit back in a wait loop. As each child process completes, we'll note its exit status, and when there are no more kids, we'll spit out a report.

First, we'll define the host list for the task:

my @hosts = map "10.0.1.$_", "001".."010";

Next, we'll want a hash to keep track of the kids:

my %pid_to_host;

for (@hosts) {
  if (my $pid = fork) {
  ## parent does...
  $pid_to_host{$pid} = $_;
  warn "$pid is processing $_\n";
  } else { # child does
  ## child does...
  exit !ping_a_host($_);
  }

The warn in the loop is merely for diagnostic purposes so that you can see what's happening. In a production program, I'd certainly remove that.

At the end of this loop, we'll have a number of processes -- far too many, in fact. For each host to check, we'll have two processes running: the shell forked by the backquotes, and the ping process itself. Perl has to fork a shell because we needed that child to have its standard error output redirected. If I could have gotten the redirection out of those backquotes somehow, we'd have only one child process per host, not two.

Launching 20 processes to check 10 hosts will start pushing us up against the typical per-user process limit. So you can see why I didn't do all 254 hosts at once!

Now it's time to wait for the results. A simple "wait" loop will reap the children as fast as they complete their task, and we'll declare a hash to hold the results:

my %host_result;

    while (keys %pid_to_host) {
      my $pid = wait;
      last if $pid < 0;
      my $host = delete $pid_to_host{$pid}
or warn("Why did I see $pid ($?)\n"), next;
      warn "reaping $pid for $host\n";
      $host_result{$host} = $? ? 0 : 1;
    }

We'll use the %pid_to_host hash to map the PID into the host for which it was processing. Again, we might have accidentally reaped a completed child that wasn't one of ours, so defensive programming requires checking for that. This won't happen unless other parts of this program are also forking children somehow, but I'm a cautious programmer most of the time.

Finally, we'll take the exit status in $?, and map it into the appropriate good/bad value for the result hash.

When this loop completes, we have no more kids performing tasks, and it's time to show the result:

for (sort keys %host_result) {
  print "$_ is ", ($host_result{$_} ? "good" : "bad"), "\n";
}

Putting this all together makes a nice little demo of forking 20 kids to check 10 hosts, but it won't scale to 254 hosts, because that would require more process slots than we typically have (or want to use, actually). We need to perform the forking gradually, so that we never have more than 20 kids at a time. One naive approach is to chunk the data into bite-size bits:

my @all_hosts = ...;
my %host_results;
while (my @hosts = splice @all_hosts, 0, 10) {
  ... process @hosts, adding into %host_results ...
}
... show results ...

A better way would be to fork until we hit the limit of active children, then wait for any one child to finish before we need to fork again. First, we'll need to factor out "waiting for a kid" into a subroutine so we can call it in two different places: while forking a new task, and at the end to reap all the remaining children:

    sub wait_for_a_kid {
      my $pid = wait;
      return 0 if $pid < 0;
      my $host = delete $pid_to_host{$pid}
or warn("Why did I see $pid ($?)\n"), next;
      warn "reaping $pid for $host\n";
      $host_result{$host} = $? ? 0 : 1;
      1;
    }

## final reap:
1 while wait_for_a_kid();

for (@hosts) {
  wait_for_a_kid() if keys %pid_to_host >= 10;
  ...

sub ping_a_host {
  my $host = shift;
  'ping -i 1 -c 1 $host 2>/dev/null' =~ /0 packets rec/ ? 0 : 1;
  }

  my %pid_to_host;
  my %host_result;

  sub wait_for_a_kid {
    my $pid = wait;
    return 0 if $pid < 0;
    my $host = delete $pid_to_host{$pid}
    or warn("Why did I see $pid ($?)\n"), next;
    warn "reaping $pid for $host\n";
    $host_result{$host} = $? ? 0 : 1;
    1;
  }

  my @hosts = map "10.0.1.$_", "001".."254";

  for (@hosts) {
    wait_for_a_kid() if keys %pid_to_host > 10;
    if (my $pid = fork) {
    ## parent does...
    $pid_to_host{$pid} = $_;
    warn "$pid is processing $_\n";
    } else { # child does
    ## child does...
    exit !ping_a_host($_);
    }
  }

  ## final reap:
  1 while wait_for_a_kid();

  for (sort keys %host_result) {
    print "$_ is ", ($host_result{$_} ? "good" : "bad"), "\n";
  }

One improvement to this program might be to pre-fork and re-use the children, using some sort of IPC (pipes or sockets) to communicate additional tasks to perform as each task completes, but I've run out of space to talk about that here. Until next time, enjoy!

Randal L. Schwartz is a two-decade veteran of the software industry -- skilled in software design, system administration, security, technical writing, and training. He has coauthored the "must-have" standards: Programming Perl, Learning Perl, Learning Perl for Win32 Systems, and Effective Perl Programming, as well as writing regular columns for WebTechniques and Unix Review magazines. He's also a frequent contributor to the Perl newsgroups, and has moderated comp.lang.perl.announce since its inception. Since 1985, Randal has owned and operated Stonehenge Consulting Services, Inc.