How I Recovered Data I Thought I'd Lost

Dave Plonka

A few years ago, when I worked in a commercial software development environment, we experienced a disk failure on one of the HP-UX application development machines. This disk held a file system that contained all of the original “hand-written” source code for a large customer software project. The work there-in was the product of a project team consisting of 3-5 software engineers over weeks of time. If the content of this file system could not be restored, it would be a major setback, but of course we had been performing daily full backups. Still, because of the amount of time estimated to restore the development environment, it was suggested that the engineers go home for the day, and the systems adminstrators began work on the restoration.

Not too long before this incident occurred, we had gone through the effort of formalizing our backup procedures. New scripts were prepared and tested independently by both our systems administrators and Oracle Database administrators. The SAs produced a backup script that would dump(1) each file system, one after the other, to a tape. Likewise, the DBAs prepared a backup script to preserve the Oracle database contents as a cpio(1) archive to a tape. The two scripts were subsequently integrated, so that the entire system backup would be performed nightly to a single high-density 4mm DDS tape, which was the state-of-the-art at the time -- mid 1996.

Unfortunately, after the integration of the entire backup procedure, no one had properly tested the resulting combined script by attempting to restore data from the resulting backup tapes. Although the file system and database portion performed flawlessly as independent backups, it was soon discovered that the database backup, which executed after the file system backups, either began by issuing an mt rew command, which caused the tape to be rewound (this was the method we sometimes used simply to determine whether a tape was loaded), or it did not access the tape drive by its “no rewind” device file.

Conventionally, a UNIX tape device file whose name contains the letter “n”, as in /dev/rmt/0hbn, leaves the tape read/write head at the current position after a tape operation completes. This feature allows the administrator to issue a sequence of backup commands that write multiple backups, one after another, to a single tape. Without this feature, every backup operation would be followed by the tape being rewound automatically, and mean that a subsequent backup would overwrite the previous. In other words, without the no-rewind device, a tape would hold the data from only one backup operation, such as a single file system dump.

As you can imagine, the consequences of this mistake were disastrous. When the SA attempted to “fast forward” the tape, using mt fsf n, to the position of the lost file system's backup, the tape drive encountered the “End of Data” marker and stopped. The relatively small database backup had “overwritten” one or more of the relatively large file system backups, and then the tape drive dutifully wrote a new “End of Data” marker just after the database backup, but before the position of file system backup that we needed to access. Essentially, the tape drive itself was keeping us from reaching our data, which unfortunately was located past the “End of Data” marker.

With some understanding of tape markers, and past experience with tapes of many sorts, the SAs laid out their options, each of which were pursued to some degree. At the same time, some nosy system programmers -- myself included -- who had in the past been responsible for some sys admin tasks, discussed the merits of each of these pursuits. We independently followed our own tack, in part, because of a division of responsibilities within the growing company and because some areas were off limits to us unless deemed absolutely necessary. I'll describe each of the various approaches.

Send for Help

An initial thought was that we could send the broken disk to a firm specializing in data recovery. If memory serves me, a phone call was made and it was determined that it would cost $5,000 minimum, regardless of whether anything could be retrieved. Also, the round-trip time for this process would be at least 2 to 4 days. Since we were game to follow our other leads, and perhaps make some progress in short order, this was not an interesting option to pursue immediately.

You Can't Get There From Here

A number of us thought we could investigate how one might be able to ask the tape drive to skip past the premature “End of Data” (EOD) marker. Also, the brute-force option of simply physically winding the tape past the marker, and then loading it into the tape drive, was considered.

Alas, these were short paths to nowhere. Apparently the DDS specification dictates the characteristics of the EOD marker and the behavior of the drive with regard to it. Neither the mt(1) command nor the driver itself seem to have a feature to skip past EOD. We surmised that it is either impossible for the tape drive hardware to read through or past EOD, or that it is only within the control of the drive firmware.

A brute force method was also tried. We manually wound the tape ahead by inserting the eraser-end of a pencil into one of the tape reel access holes. Unfortunately, upon loading the tape into the drive, the tape drive would automatically rewind the tape before any operations could be performed.

Measure Once, Cut Twice?

Another idea was that we could unwind the tape and splice out the premature “End of Data” marker -- certainly not elegant, but still a possibility.

Conveniently, the backup operation had been performed in this erroneous way for some time. As a side effect, we had accumulated boxes of similarly botched tapes. The older tapes provided lots of raw material with which to attempt to find a workable recovery procedure by trial-and-error.

For splicing, the tapes unfortunately weren't good ol' reel-to-reel 9-track tapes, but rather they were tiny 4mm Digital Data Storage “DDS” tapes, just like the Digital Audio Tapes “DAT” media. Physical manipulation of the media was further impeded by the tape enclosure, which is similar to that of a VHS video tape, only much smaller, with a spring-loaded “door” that snaps shut to protect the magnetic medium. The tape and tape-drive specifications were examined to determine the length of the tape and data density, and some rough values in meters were decided upon as to where the splice should occur. Before too long, the computer room floor was littered with snippets of tape, every one a reminder of another failed attempt. In the end, none of the spliced tapes was able to be read. A co-worker thought that this may be due to the “helical scanning” head that is used to write the data to the tape itself -- it is perhaps impossible to splice such a tape amidst the data and preserve its integrity.

Deja Voodoo

While fruitless experimentation continued in the computer room, I turned to my Web browser and visited http://www.deja.com, which was www.dejanews.com at the time, hoping to find the advice of others who had been in this predicament. Deja.com provides a wonderful archive of USENET news, a sort of “USENET Library of Congress”, but with commercials. Using the site's “Power Search” feature, I found a thread from the comp.sys.hp.hpux newsgroup with the subject “[Q] reading past End Of Tape with HP-UX 9.05”. To my surprise and delight, the poster described a method by which one can potentially recover data from a tape trashed in the way I've described. The method involved the following steps:

1. Overwriting the tape again, with just the right amount of data to clobber the premature EOD marker that was keeping us from reaching our backup data.

2. Powering off the tape drive, before it can write another EOD marker, as it is designed to do.

This procedure, while possibly specific to DDS tape technology, is neither HP nor HP-UX-specific, nor is it in principal even UNIX-specific.

The Operation

Before performing the steps below, which are my interpretation of the steps I discovered, you should practice this using old backup tapes that have been similarly trashed. If you don't have such tapes, I suggest taking the time to produce practice tapes that are similarly overwritten. Remember, this is potentially a destruction process, so you may only have one try with your most recent, best backup tape.

1. Load a clobbered tape into the drive.

2. Position the tape to EOD, using the “'n'o rewind” device file:

# mt -f /dev/rmt/0hbn eod
   

The following is a quote from Kevin Jones (kev@bri.hp.com), an HP employee who, in an “unofficial” capacity, authored and posted a step-by-step procedure from which I worked. Regarding this portion of the procedure he wrote:

On 354x0 drives, with a native transfer rate of 184 Kbytes/sec, the 17secs_of_tape file should be just over 3 Mbytes of compressed data. You can use whatever data you want, not just the HP-UX kernel. On HP C15x3 drives with a native transfer rate of 510 Kbytes/sec, you need about 8 Mbytes of data.

Kevin chose to use the HP-UX kernel itself as a source of data to write, since it was a known quantity to him, and contains sufficiently random content to produce a compressed data file of a given size. If you are not running HP-UX, or are running a version with a kernel very different in size from the 9.05 kernel, it may be best to choose some other data file. If it exists on your system, I suggest using /dev/random as a source of unpredictable content, which is unlikely to be able to be compressed much by the tape drive. In this example, I'll presume you want a file of 8 MB in size:

# ls -l /tmp/17secs_of_tape # determine whether or not it exists
# ...
# dd bs=1024 count=8192 if=/dev/random of=/tmp/17secs_of_tape
# ls -l /tmp/17secs_of_tape # should be 8MB in size

4. Write the sample data to the tape:

# dd if=/tmp/17secs_of_tape of=/dev/rmt/0hbn
   

6. Wait a bit, as you probably should whenever powering equipment off and back on.

7. Fast-forward to just past the first EOF marker (if you have multiple files, or dump(1)s on the tape):

# mt -f /dev/rmt/0hbn fsf 1

# mt -f /dev/rmt/0hbn fsr 100 # adjust \
  number of records accordingly

# restore if /dev/rmt/0hbn

One complication in our situation was that we were using a compression feature of the tape hardware. On many systems, this feature is used when one accesses the tape drive using a device file with the letter “h” in the name, as in /dev/rmt/0hbn. This compression means that it is hard to judge just how much sample data (in bytes) one must write to clobber the EOD marker, because the degree of compression achieved depends on the content of the data itself. To deal with this, it is best to use “unrepetitive” or already compressed data when making your “nsecs_of_tape” file, because it is repetition in a data stream that enables effective compression.

Another complication was that we did not have access to an external DDS tape drive -- they were all internally mounted within the cabinet of HP-UX machines. We decided to enlist the services of another similarly configured, but currently unused, HP-UX machine and, when the time came, throw its “big switch” -- thereby powering down the whole machine and the tape drive as a side-effect. While this may seem extremely risky, it was a machine that we could afford to “lose” and it has been noted that powering off an external drive, if we had one, may have elicited similarly undefined behavior, perhaps resulting in a kernel panic. If you consider doing this, it would be prudent to attempt this recovery operation from single-user mode if possible.

In Summary

In the end, we had great success with the method described above. The software engineers' work was restored and all returned to normal. This experience reminded us that successful execution of a backup script doesn't necessarily mean that one can recover data from the backup. As a result, backup recovery tests were added to the list of sys admins backup responsibilities. The creative method that was ultimately employed to do the “impossible” serves as an example of what can be accomplished when admins or engineers take the time to share their experiences with their co-workers and their peers elsewhere.

Thanks

I would like to thank Kevin Jones (kev@bri.hp.com) and Frank Slootweg (franks@neth.hp.com) for their participation in USENET news and, specifically, in assisting with this problem.

About the Author

Dave is a benevolent hacker and UNIX aficionado. He is a Senior Systems Programmer in the Network Engineering Technology group within the Division of Information Technology (DoIT), at the University of Wisconsin, Madison. He can be reached at: plonka@doit.wisc.edu or via: http://net.doit.wisc.edu/~plonka/.