Automating Your Site with Expect
Mike Schwager Have you ever wanted a tool that would allow you to automate various command line tasks -- even those things you thought difficult or impossible to do? Such a tool has been available for years, and it's called Expect. If you become proficient with Expect, it can help you take care of the mundane, repetitive parts of your job. What is Expect? Expect is an extension built onto Tcl/Tk -- the Tool command language and the Tk toolkit. Expect allows you to manipulate any character-based application via a program that can look and act like a human being to your UNIX/Linux-based machines. Expect and Tcl/Tk also run on Windows NT. If you are already familiar with Tcl or Expect, you can jump down to the “Expect in Greater Depth” section below. Otherwise, I will touch on what you can do with Expect and the very basics of the Tcl language. Consider the following scenario -- you need to change the root password on a number of different machines. From where you are sitting at your local host, the interaction will look like this:
localhost$ telnet remote0.mydomain.com Welcome to remote0. login: myname Password: <password> Last login: Yesterday. remote0$ su Password: <root's password> remote0# passwd root New password: <new password> Re-enter new password: <new password> password changed. remote0# exit remote0$ exit Connection closed by foreign host. localhost$ telnet remote1.mydomain.com ... (repeat as necessary) Now, as a great programmer, your three virtues (“laziness, impatience, and hubris” -Larry Wall) all kick in at once. You write the following Expect script:
foreach host "remote0.mydomain.com ... remoteN.mydomain.com" {
spawn telnet $host
expect "login: "
send "myname\r"
expect "Password: "
send "mypassword\r"
expect "$ "
send "su\r"
expect "password: "
send "rootpassword\r"
expect "# "
send "passwd root\r"
# Here, we look for either of two strings, like an if/else construct
expect {
"password: " {
send "rootnewpassword\r"
exp_continue
}
"# " {
send "exit\r"
}
}
expect "$ "
send "exit\r"
}
Easy, no? As a human being, you spawn a job, then expect to see some feedback, and you send some text. Expect works the same way; hence the name. Notice in the seventh line that you do not need to match all text -- Expect can simply look for a substring. Once Expect has found the text, it executes the next statement. In line 14, we have a more elaborate Expect statement. This statement allows you to look for either "password:" or "#". The exp_continue essentially creates a while loop within the current Expect. Note that Tcl strips off the double quotes for Expect. A Look at Tcl To use Expect, you need to know Tcl. The entire Tcl/Tk language is available to you when you work with Expect. All of the usual programming features are available, like foreach, switch, if/then/else, while, etc. Obtaining and Installing Tcl/Tk/Expect The current release of Expect is 5.31, and it requires Tcl 8.2. Further information about the requirements is available in the Expect README file. Expect is available from http://expect.nist.gov/. You can download Tcl from there or go to http://www.scriptics.com. On Solaris and Linux, installing couldn't be easier. Instructions are in the INSTALL files with each distribution. How Tcl Processes Your Script Tcl is quite simple. There are two inner parts to it -- the parser and the executer. The parser performs substitutions as required. Tcl then assumes that the first word is a command name, and invokes the command procedure with the other strings that the parser found. It doesn't get any more complicated than that. The for command is a procedure that contains four arguments -- think of them as four strings. Even the body is just a big string -- it gets passed back to the interpreter by for. There is no “for loop” to Tcl, per se. Remember not to give the Tcl interpreter any more intelligence than it possesses. Here is a summary of some basics of Tcl:
• Statements are executed as they're encountered -- there is no optimization or byte-compilation of your script (à la Perl or Java). • There are no types -- all variables are strings. • Tcl uses command and variable substitution. Variable substitution is done with the dollar sign $. Command substitution is done using square brackets, such as:
set i [ set j “101” ]sets both $i and $j to 101. • Tcl contains variables, lists, and array constructs. Tcl arrays are associative arrays, and elements of the array are accessed like: $month(January). A list is basically a string with a space between each element. • To initialize a variable:
set var valueIf you try to say $var=value in Tcl, you will get an error. Remember that Tcl is very consistent. The first word of every line is assumed to be a command, except for comment lines, of course.
• Accessing files is done within the language, with “primitive” file statements like open, close, puts, stat, etc. • Creating a command (a.k.a. procedure) in Tcl is as simple as:
proc { arg1 ... argn } { body }
See the expect examples below.
• Procedures' arguments are called by value. It is possible to call by reference using the uplevel command. Variables within procedures have local scope. The global command will make their scope global. • In Tcl, the result of every command is a string. • You quote strings in double quotes or braces: {}. • Braces provide strong quoting -- that is, command and variable substitutions are deferred. Do not think that braces define blocks, like Perl. They are not really blocks. • Tcl is a complete programming environment. Your scripts won't be forking and execing lots of subshells or external programs. • The end of a line ends the Tcl command. You can also use a semicolon to end a Tcl command.
There is much more to learn about Tcl -- more than we have space for here. For more indepth study, get Tcl and the Tk Toolkit by John Ousterhout (Addison-Wesley). Expect in Greater Depth: Diving In I showed you a basic Expect script above. You can see that an Expect script goes like this:
1. Spawn a subprocess 2. Expect some output from the subprocess 3. Send some text to it in reply
I will show you more Expect by way of example. Expect Excellent Example 1 You are a responsible systems administrator. Your cleartext passwords are nowhere except in your brain, but now you need to move files from one machine to the next, and you need to extract them once they arrive. You'll be doing this time and time again. How? You create the following Expect script:
#!/usr/local/bin/expect -f
match_max 10000
set env(TERM) "dialup"
set user $env(LOGNAME)
stty -echo
send_user "Enter password for $user now: "
gets stdin password
send_user "\nEnter password for root on remotes now: "
gets stdin rootpw
stty echo
#
foreach machine $argv {
spawn ftp $machine
expect -re "Name .*: "
send "$user\r"
expect "word:"
send "$password\r"
expect "ftp> "; send "bin\r"
expect "ftp> "; send "cd /tmp\r"
expect "ftp> "; send "put localfile.tar\r"
expect "ftp> "; send "quit\r"
send_user "\r\nftp exited.\n"
sleep 1; spawn telnet $machine
expect "ogin: "; send "$user\r"
expect "word: "; send "$password\r"
expect -re "(\\$|>) "; send "su\r"
expect "word: "; send "$rootpw\r"
expect "# "; send "cd /tmp\r"
expect "# "; send "tar xvf localfile.tar\r"
expect "# "; send "exit\r"
expect -re "\\$|> "; send "exit\r"
}
What It Does In line five, I set up a simple TERM for my subprocesses; my fancy .*rc files on the remote machines are designed to recognize this terminal type and simplify my environment. All shell environment variables are available in the env() array. I assume in line six that my remote user name is the same as LOGNAME on this local host. In line nine, you can see how Tcl gets input from the user. You can also see how I shut off echoing of the passwords. Notice in line 16 how I am using a simple regular expression instead of a string, with the -re option. In line 28, I use another regular expression; it looks for a literal $ or >, followed by a space. The $ is meaningful to Tcl and Expect, and must be escaped. Regular expressions are similar to Perl regular expressions. You can elaborate on this basic script, perhaps interacting with ssh (you are very responsible), or running some install commands after the tar extraction. Expect Excellent Example 2 As a systems administrator, I am concerned that my systems are up and running adequately. Often I use ping, but that's hardly enough. ping merely sends ICMP requests to the remote system's hardware -- your CPU doesn't get involved. Your applications may have problems, and you'd never know it. Here's a better way to check on the status of a daemon, in this case innd:
#!/usr/local/bin/expect -f
set timeout 10
proc smart_expect { look send } {
expect {
-exact $look {
send $send
}
timeout {
send_user “Something is wrong with innd!\n”
exit 1
}
}
}
#
spawn telnet newshost.mydomain.com 119
match_max 10000
smart_expect "\r
200 " "group comp.risks\r"
smart_expect "\r
211 " "quit\r"
smart_expect "\r
205 " ""
smart_expect eof ""
What It Does Expect spawns a telnet into the nntp port (119) on the news host. I created a procedure for the Expect that takes two arguments -- what we are looking for, and what we should send if the item is found. The first smart_expect on line 21 is looking for a carriage return, linefeed, 2, 0, 0, and a space, in sequence. When it finds this sequence, Expect will send the string group comp.risks plus a return. I'm using a smarter Expect, so I don't have repeat the timeout case for each string that I'm looking for. Important Features • Line 3, the timeout: By default it's 30 seconds. You can adjust it to whatever is appropriate. This is an integral number of seconds. • Line 6, -exact: By default, certain characters have special meaning to the Expect command's input string. Expect matches strings using “glob patterns”, like the shell. In this case, we want to match all characters literally. Expect accepts a number of options to its string or pattern matching. We use the -exact option; pattern matching is turned off. However, normal Tcl substitutions will apply. Thus \r represents a carriage return (control-M) and not a two-character sequence. • Line 20, match_max: We don't want to have a huge input buffer -- eventually, old information is useless. At the same time, we want to make sure we are looking at enough output to grab the appropriate text. This should be adequate. • Line 21, smart_expect and the carriage return/linefeed: Notice how the end of line doesn't end the string on this line. This will not cause a syntax error; Expect correctly identifies the return/linefeed sequence in your script and will look for it as the output from telnet.
These examples are pretty simple. Fortunately, the Expect distribution comes with a directory full of more robust examples. Expect Excellent Example 3 Two programs that accompany this artcicle can be found on the Sys Admin Web site -- login.exp and run_general. Additionally, as I make changes, I will modify:
http://www.enteract.com/~schwager/expect.htm login.exp is a general login procedure that you can use in your own Expect scripts. As a side effect, it sets the $prompt variable. You call it by:
source /usr/local/lib/login.exp Use the real path to where you install it. The other script is run_general. There are many times when I just want to whip out a quick series of instructions and run them with automatic logins for my telnets and ftps. run_general allows me to do that in the general case, more easily than autoexpect (described next). Expect Tips and Tricks Instead of writing your own Expect scripts, get Expect to do it for you! Upon installing Expect, you will also install autoexpect. autoexpect will capture all the input and output from a session. You run autoinspect, do your work, and exit from the shell. Your work is saved for you to modify. Usually this entails whittling the file down significantly, but it's a great way to get started without having to know much about Tcl. It's excellent for automating curses-based programs, because you will be able to see the escape sequences that were sent by the program. Expect captures both the stdout and stderr of spawned processes and places them in stdout for you to see. This leaves Expect's stderr available for diagnostic output, such as:
print STDERR "Hit enter to continue.\n" Meanwhile, stdout contains all other output from your script. So you can interact with your script while redirecting stdout to a file. Use the command:
log_user 0 to shut off output from your spawned process. log_user 1 restarts it. You can intersperse them throughout your scripts. You can debug your Expect patterns with exp_internal 1. exp_internal 0 shuts debugging off again. Don't think of the Expect statement as a single-line, monolithic command. You can set up powerful if constructs and looping conditions. See my example at the beginning of this article and note how the exp_continue command is used. Part of any login procedure is to enter a password. You can turn off the display of the password when using Expect to automate logins. Do it with stty -echo. Turn character echoing back on with stty echo after you have entered the password. Expect has a rare capability -- it can sleep for fractions of a second. Try inserting a sleep .4 command in your Expect script. It works! Note that on most systems, the number given is a minimum. You'll sleep at least that long, but there is no guarantee that this number won't extend into multiple seconds. If you are a power user who likes to set up many things in your rc files, you may discover that your Expect script is having trouble handling cursor addressing escapes at login. I have found it useful to set the TERM environment variable to something innocuous (like dialup) in my Expect scripts:
set env(TERM) "dialup" Here's a scenario -- you start your Expect script, but change your mind and hit control-C. Later you discover that there are orphaned processes like ftpd hanging around, which were created by you. Here's a way to ensure a clean exit from Expect:
#! /usr/local/bin/expect -f
proc my_exit {} {
global pid
# This should do it, but we'll go down the line, to clean
# up any mess.
exec kill -TERM $pid
# Oh yeah? Take this!
exec kill -HUP $pid
# Oh double yeah? Take this!
exec kill -KILL $pid
send_user "Early exit.\n"
exp_exit
}
trap my_exit {SIGINT SIGTERM}
set pid [spawn -ignore SIGINT -ignore SIGTERM ftp remote
host]
expect -re "Name .*: "
...etc....
Now, when you hit control-C, my_exit is called, which tries in a number of ways to kill any recalcitrant child. Expect Caveats Note that in Tcl, you must set a variable before using it. Often it's difficult to get the pattern matching just right. Try to match on the smallest portion of text that you can guarantee to be unique, and be careful if you set timeout -1 (never timeout). Note also that if you place a closing brace (}) inside a comment, the comment ends and the rest of the line is read by the Tcl interpreter. Parentheses are used somewhat sparingly in Tcl/Expect. You can use them in expressions and regular expressions, but they are not used in while, for, or if statements. You use braces. Remember that braces defer substition of variables. If you write a Tcl statement like while { $i < 5 }, the while command actually sees the string $i < 5. This is because it must evaluate it every time. Control-flow commands do their own substitutions. Do not perform brace command substitutions; that is, don't do set j { [ set i 157 ] }. Put them in double quotes if you need to group them with other strings, like:
set j "Expression results: [ expr 1 > 0 ]" I use a shell prompt that shows part of the prompt string in boldface. Some of my colleagues use prompts that end in various characters. Matching prompts with Expect can be tricky. Here is something that works for me:
set prompt "(%|>|:|\#|\\$) ($|^[)" The last character before the closing parentheses at the end of line is not an up-arrow/bracket combination, but the “escape” character. The string is a regular expression matching literally % or > or : or # or $, followed by a space, followed by either the end of input or an escape character. When I match the prompt, I write an Expect command that looks like the following:
set $long_timeout 30
expect -re $prompt {
sleep .2 ; # timeout only takes integer seconds
set timeout 0
expect {
-notransfer -re "\[a-zA-Z0-9\]" {
; # not a prompt- fall through and continue the
# outer
}
timeout {
; # we got a prompt!
set timeout $long_timeout
continue ; # jump to an outer while or for loop
}
}
set timeout $long_timeout
exp_continue
}
The -notransfer option to Expect means that we will not slurp in the output from our spawned process -- we just want to take a look, without “touching” the text. The text is available to be scanned by a subsequent Expect. Why is there sleep in the above section of script? Because Expect's dollar sign does not match the end of line -- it matches end of input! What is end of input? It is whatever the operating system says it is. The OS reads characters into a buffer and hands that buffer to Expect. Sometimes strings you think are whole actually come in multiple chunks. I have found that a slight pause is enough to not slow my scripts down too much, yet ensure that we are looking at a prompt and not simply some slow buffers. Adjust the sleep value to taste. Where to Go from Here With this introduction, I hope you will be encouraged to go further with Expect/Tcl. You can write fancy X-based user interfaces quickly and easily with another Tcl extension, Tk. For more information, check out http://www.scriptics.com. My favorite books on Expect/Tcl/Tk are Tcl and the Tk Toolkit, by John Ousterhout (Addison-Wesley), and Exploring Expect, by Don Libes (O'Reilly and Associates). Armed with these tools, I Expect that you'll do great things!
About the AuthorMike Schwager is a contractor specializing in UNIX and the Internet. He has spent the past 15 years writing C and Perl code, shell scripts, and maintaining systems in the corporate and educational environments. Email him at Michael@Schwager.com or visit http://come.to/lanicservices. |