Lessons Learned from PCASSO

Esme Cowles and Leonard K. Carson

Much of the effort that occurs in programming and systems development circles involves specific solutions to problems that are relatively narrow in scope. In many cases, however, the lessons learned in such projects can be applied more broadly, often with only minor adjustments to the code. The PCASSO (Patient-Centered Access to Secure Systems Online) project is a good example of such a project.

PCASSO in Abstract

The PCASSO project demonstrates that secure access to medical records can be provided over the Internet. Using a Web browser, a physician or patient starts a Java applet that allows the user to login, select patients, and display medical records. The Java applet communicates with a custom-written server that acts as an intermediary between the client and an Oracle database which contains both the medical records, and the information that determines access privileges.

The University of California, San Diego, (UCSD) has partnered with Science Applications International Corporation (SAIC) in a three-year research project funded by the National Library of Medicine. During the first two years, an SAIC development team designed and developed the PCASSO model system. During the third year, the model was implemented at UCSD and evaluated with a group of providers and patients. The primary goal of the PCASSO project was to build and evaluate a model system for providing safe Internet access to medical records. Because of the project's security focus, many decisions were made that had the effect of ensuring security at the cost of ease of use and administration. In the ongoing field test, we have encountered a number of implementation challenges and have resolved them as best we could within the confines of maintaining our rigorous security policies. This article explains the rationale behind the design decisions, the implementation challenges we faced, and our solutions.

Design Considerations

PCASSO was designed from the outset to be a secure system. Other people have developed Web-based access to content that needed to be secure (including medical records), but the security usually amounted to using Secure Socket Layer (SSL) to encrypt data as it passed over the Internet and using passwords for user authentication. Although this level of security may seem adequate at first, several limitations are found on closer inspection:

• Many commonly used operating systems and Web servers are vulnerable to penetration and denial of service attacks (such as the buffer overflow vulnerability recently discovered in IIS/Windows NT).

• Client machines running Windows 95 and Windows NT are vulnerable to Trojan Horses (such as Back Orifice and ExploreZip).

• Passwords can be guessed by a determined intruder.

• Vulnerable information can become corrupted.

In the electronic commerce setting, these limitations are acceptable. Denial of service vulnerabilities can often be avoided by vigilantly upgrading when security alerts are announced. It's an acceptable risk for a server to be unavailable for a short time while a problem is being fixed or when responding to a denial of service attack. Trojan Horses and other intrusions sometimes target credit card information. While it can be frustrating for users to have their credit card information stolen, the credit card companies have established systems to identify and contest fraudulent charges. When financial data becomes corrupted, these same systems can be used to restore the accuracy of the records. Furthermore, these types of attacks are infrequent enough that, in general, they do not prevent most customers from using a service.

In healthcare, on the other hand, the limitations listed above are much more serious. When people's lives are at stake, 24-7 availability is a must. Similarly, when decisions are made based on healthcare information, data corruption is the most insidious and threatening attack possible. When health information is stolen, the consequences of a security breach range from bad (embarrassment) to worse (loss of insurance, job, etc.).

There are also very motivated intruders after information such as celebrity health records. And, some other information is so sensitive that its disclosure can have serious consequences (e.g., teenage abortion information). Additionally, there is no concept of recovery from a disclosure. If highly sensitive health information, such as genetic or HIV/AIDS information, is disclosed, recovery is not possible. The disclosure can have a profound and irreversible effect on the individual's ability to get a job, insurance options, and overall lifestyle. These concerns led the PCASSO development team to make several design decisions to provide the maximum assurance that the system was not vulnerable to denial of service, disclosure, or corruption.

The PCASSO team chose to use a B2 operating system (Data General's DG/UX with B2 Extensions) to host the Web server, database, and related programs. Like most operating systems today, DG/UX provides a wide array of security features. However, unlike any other operating system available on the commercial market, DG/UX was developed to meet the Department of Defense's stringent architectural and evaluation requirements for a “B2” rating. This means that the DG/UX code is structured into independent modules that enforce the “least privilege principle” and that the code is subjected to thorough review and penetration testing by attackers with knowledge of that code. B2 (and B1) systems also provide label-based access control, which means that the operating system controls access based on labels stored in internal data structures. B2 DG/UX was designed to be able to withstand concerted attacks and to protect classified information. A battery of denial of service and penetration tests were run against our production server running B2 DG/UX, and it passed with flying colors. Even system performance wasn't noticeably degraded.

To build our database, we imported data from the hospital's legacy systems, applied security labels, and inserted them into a database that supports label-based access control (Trusted Oracle). Because of the complicated healthcare access rules, we used Trusted Oracle to implement label-based access control for the data imported from the legacy systems. This allows hierarchical levels of access, in addition to user-based constraints. A user's access permissions are the intersection of his or her hierarchical and user-based access permissions, which allows much finer distinctions to be made. For example, a patient and his or her physician would both have the same user-based access to the patient's account. The patient's hierarchical access would be narrower than the physician's. In addition to the information the patient could view, the physician would also be allowed to see information his or her provider had identified as “patient deniable” (such as private communications with a family member who wanted to remain anonymous), be able to make information “patient-deniable”, and be able to add or remove access for other providers.

For the client, the PCASSO development team used a Java applet accessed by a Web browser on a standard PC. Java was used more for its safety than for its portability. In fact, only Windows 95/98/NT were supported as client environments, and slightly different versions had to be developed for different browsers. The principle security feature of the Java environment is that the applet runs in a segregated virtual machine, which limits the ability of the applet to read files, corrupt memory, communicate with other servers and do any number of other things that malicious programs might want to do. Because the applet is stored on our server and downloaded on demand, there is less opportunity for corruption of the applet, and it is accessible to anyone who has a Web browser that supports Java.

As for standard electronic commerce applications, the development team used the SSL protocol to establish an encrypted session between the client and the server. We used public/private keys to authenticate both the client and the server rather than using anonymous SSL, where only the server is authenticated.

Instead of the traditional username/password (single-factor) authentication, we used a multiple-factor authentication process: username/password, public/private encryption key pairs, and challenge-response pairs token. This gives us significantly more assurance that only an authorized user can successfully login to use the system.

To reduce the ability of Trojan Horses to capture user input, we used a clickable keyboard image instead of the physical keyboard to enter such items as passwords, patient names, and medical record numbers. This was judged to be an acceptable burden for the small number of times it would be used, because PCASSO was not designed to be a data-entry system.

Implementation Challenges

We have faced a number of challenges while administering PCASSO during our ongoing field test. Here's how we dealt with them:

System instability disrupted other peripheral production systems, such as the pharmacy. We receive data in the form of HL7 messages sent from a central server (the Interface Engine, or IFE) to all peripheral systems. The IFE sends each message to every peripheral server and waits for confirmation of receipt from each system before dropping the message. If too many messages pile up, the IFE can run out of disk space, which causes it to stop sending messages or functioning in any meaningful way. This obviously disrupts the other peripheral systems.

To reduce disruption due to network and system outages, the PCASSO developers separated the program that inserts data into the database (HL7 Importer) from the program that receives data from the IFE (HL7 Server). The data is written to files on the local disk (HL7 queue files) to interface between the HL7 Importer and the HL7 Server. Because the most common problems are with the ability to insert records into the database, it is convenient to simply store the backlog in a local file, and catch up with the backlog when the problems are fixed. We can process data twice as fast as we receive it when things are functioning smoothly, so even an entire day's worth of downtime is not a serious problem.

Querying the audit trail data took too long. One of the key features of PCASSO is to allow patients to see a list of users who have accessed their accounts using PCASSO. Once the system is in general use, this would be a great way to have patients sniff out abuses by tracking who accesses their records, which is an effective way to prevent unauthorized access.

As part of a secure system with extensive auditing records, literally every system action generates audit events, and this made searching these audit records impractical. The time to return an audit query using the client began to increase once a few weeks had elapsed where the system was operational. The time quickly became intolerably long -- as long as 15 minutes. The development team modified the code to segregate the audit records. Queries would be performed only on the last 30 days, which reduces the query time to around ten seconds. However, a better solution is currently being implemented -- making a snapshot every night of the query results for each possible user, and displaying the snapshot instead of doing the query interactively.

System downtime due to backups needed to be reduced. Backing up the entire server takes approximately four hours, and we didn't want to take the system down that long every week. To minimize downtime, the HL7 queue files and audit trail files are backed up weekly, instead of the filesystems. This data allows the entire system to be recreated, with the exception of infrequent changes to the operating system. In the case of user account creation or software upgrades, the changes are well documented). The filesystems are backed up only monthly.

Many common programs will not compile or work on B2 DG/UX. For example, we were unable to get either Sendmail or lpd to work, even after extensive effort and working with DG/UX technical support. Even groff had to be modified before it would compile (man pages come pre-formatted). So, while the base operating system is very robust, the ability to add even common software to the system is limited. Most of the solutions we found for these problems were procedural, such as using groff to generate PCL and write it directly to the printer device. In other cases, we had to fix the programs ourselves.

A system built to provide a high level of security is very different from what most users are accustomed to. The login procedure is complex, and the clickable keyboard graphic is cumbersome. Also, providers are not accustomed to having their access restricted to only those patients with whom they have a caregiver relationship. To make PCASSO's security more acceptable to the user, we explain that it is critical to security.

Using public/private key pairs meant that sending the encryption keys over the Internet, even over an encrypted session, would reduce the level of assurance of our system. Additionally, because of the vulnerabilities associated with world-accessible CGI programs, we decided that we would not use CGI on our production server. These decisions meant that we were not able to have an online sign-up procedure. Instead, users must fill out a paper form and submit it by fax or mail. While this is secure, it also prevents immediate access for our users.

Many end users' browsers didn't support Java version 1.1. To enable the PCASSO client to read from a diskette containing the server's public key and the client's private key, the user must use a browser that supports Java version 1.1 (such as Netscape 4.06 or later, and Internet Explorer 4.0 or later). As the provider of technical support, I was somewhat surprised by the number of users using older browsers (such as Netscape 2.0, and Explorer 3.0). Also, many users were unable to download and install an updated browser, even when given explicit instructions on the phone, and some users had difficulty installing an updated browser from a CD-ROM. We have changed our user materials to include a browser CD-ROM in every user packet.

Conclusion

The central lesson learned from PCASSO is that safe access to healthcare information over the Internet is possible. The PCASSO model is a very robust system that provides a strong security infrastructure that is both functional and highly resistant to attack. But safe Internet use in healthcare comes at a cost. B2 DG/UX and multiple-factor authentication create a user and administrator environment that is more complex and cumbersome.

We have discovered that some extra effort on our part can improve the user's experience a good deal. Our backup procedure is a little more difficult than it could be, but it results in increased availability. Also, caching the audit trail results is more complicated than running the queries interactively, but results in better perceived performance for the user. On the other hand, some design decisions made by the development team had outcomes that we, as administrators, did not expect. This reminded us to consider our users more thoroughly in the design phase. Requiring the latest browser may allow greater design freedom, but results in more tech support and user confusion. The user reaction to the keyboard graphic reminded us that users have their limits.

Some of the development team's solutions, such as separating the HL7 Importer from the HL7 Server, reminded us that the basics of system design never go out of style: flexible, independent, modular designs are the most robust.

Code Samples

Listing 1 (available from the Sys Admin Web site or at ftp.mfi.com) is a Korn shell script that demonstrates our multi-level menu structure. The implementation can either be inserted into the script, or can be external scripts or programs, with the script acting as a front-end.

About the Author

Esme Cowles is the current sysadmin and tech support for the PCASSO project. When the PCASSO server and his Linux machine at home aren't acting up, he enjoys learning Java and Perl. He can be reached at: escowles@gort.ucsd.edu.

Leonard Carson was the sysadmin until March of this year. He now works as an infrastructure technologist for Sempra Energy and enjoys the San Diego coastline in his free time. He can be reached at: lcarson@ucsd.edu.