Cover V11, I10

Listing 1


Optimizing Time to Completion with runmany

Bill Davidsen

If you are using a computer with multiple CPUs, lots of memory, and other resources (like bandwidth), and you need to do many small, similar tasks, it makes sense to simplify the process. However, trying to do too much at once can really bog down your system. runmany is a tool I developed to help fit the load to the resources and complete jobs faster.

runmany is a Perl program that accepts lines of input from stdin, substitutes the string read into a command-line script, and runs some limited number of the commands in parallel. Typically, they don't all run simultaneously, so runmany keeps the load constant by starting another process as soon as one finishes, rather than starting a fixed number of processes and waiting for all to finish before starting more. See Listing 1. (All source code for this article can be downloaded from:


I originally wrote runmany to help me do some odd feeding of Usenet news articles that didn't fit the normal peer-to-peer model. For this example, assume I have identified a large number of articles that need to be fed (or fed again) to a remote site. If I feed them one at a time, I will take the sum time of all the transfers. If I have 100-k articles, I don't want to start a process and open a socket for each, so I run a limited number of sockets (experience tells me about six), and process a reasonable number of articles with each socket.

With this (assuming that I want batches of 500 articles), I want to run six streams at a time, and I have all the article information in one big file:

split -500 bigfile sfq.
ls sfq.* | runmany "innxmit -a -v server2 $PWD/%s" 6
The "%s" in the runmany argument is replaced with the value of a single line read from stdin -- in this case the filename of a small list of articles. I first break the input into files of 500 items each (using split), then use ls to generate a list of the small files, which I pipe into runmany.

The time to complete each of the individual small files depends on article size, bandwidth, and the server. The first processes are all started at once but the behavior quickly becomes unsynchronized. With some connections, large files will be slow due to limited bandwidth, and some servers will be slow to accept many small articles due to database performance adding the data. This way, I keep a reasonable load on the machine regardless of the performance of the individual connections.

Using runmany with Graphics

I sometimes have a large number of graphic images and want to perform a common operation on each image (e.g., making a scaled copy of each image). If I were doing this on a system with several CPUs, I would want to have multiple copies running to finish the job in minimum time. I use the netpbm package for image manipulation; I convert each image to standard format, scale it to fit in 640x480, and then save it as an optimized JPEG file with "_STD" appended to the name.

Here is one way to do this:

find images -name '*.jpg' |
runmany "djpeg %s | pnmscale -xys 640 480 | cjpeg -o >%s_STD" 2
Depending on your hardware and operating system, the number of processes might be between one less than the number of CPUs you want to use, and one more. This shows what someone actually did to address a problem like this:

find . -name '*.jpg' |
while read filename; do
  djpeg $filename | pnmscale -xys 640 480 | cjpeg -o >$filename_STD &
done; wait
Other than showing that AIX will stay up with a load average of 400+, it was definitely not the way to get the job done in minimum time. In practice, the number of jobs should be close to the number of CPUs you want to (or are allowed to) use, limited to the number of jobs that will fit comfortably in physical memory, or to the number of processes that will use the bandwidth of the network or disk. Don't forget disk! The actual clock time using runmany with two processes (shown previously) was 61% of the time doing one at a time. For 171 images, this was 59:51 vs. 36:31, a savings of more than 23 minutes.

Other Uses

While I wrote runmany to process fixed lists of things to do, I have since used it at the end of pipe to process lists generated under program control. Because the command strings are arbitrary, it is more flexible than xargs, and many versions of xargs will only run one process at a time.

Final Thoughts

runmany is primarily a tool for "one-time" problems. It is easy to use and understand and can greatly improve performance relative to the effort required to use it. It is not intended to be an optimal solution to any one problem, but rather an option to solve many problems. Perhaps the next time you have a bunch of small, identical tasks to perform, it will be as useful to you as it is to me.

Bill Davidsen has been doing systems programming and administration since 1968, and was one of the founders of TMR Associates in 1979. In addition to being a "part-time CTO" at TMR, he works as a project leader with a national ISP and writes an Internet column.