Case Study of User Assessment of a Corrections Electronic Health Record

by Madison L. Gates, PhD, and Phillip W. Roeder, PhD

Abstract

The federal government, through the Office of the National Coordinator for Health Information Technology, has moved vigorously to promote widespread and meaningful use of interoperable electronic health records (EHRs) by 2014. The Kentucky Department of Corrections implemented its EHR system in 2006 and in 2010 the department assessed user satisfaction and perception of usability based on criteria that reflect meaningful use.

Fifty percent of 345 users responded to an online survey with satisfaction averaging 3.0 out of 4.0 on a 14-item scale and usability averaging 2.8 out of 4.0 for 13 items. The two measures correlated strongly and positively but varied significantly by type of position. This study provides a positive but cautionary case study of how users assess components of an EHR in a relatively stable and controlled organizational setting.

Key words: health information technology, electronic health record, meaningful use, user evaluation, user satisfaction, corrections health system

Introduction

Healthcare remains one of the few industries that rely heavily on paper records. The federal government has moved vigorously to encourage and expand the transition from paper to health information technology (HIT), specifically electronic health records (EHRs). The Office of the National Coordinator for Health Information Technology (ONC) in the U.S. Department of Health and Human Services (HHS) is the principal federal agency charged with coordination of nationwide efforts to adopt and meaningfully use HIT.¹

The ONC was created in 2004 by executive order but was legislatively mandated in the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009. The HITECH Act seeks to improve American healthcare delivery and patient care through a substantial investment in HIT including transition assistance to providers, coordination within and among states, connectivity to the public health community in case of emergencies, and assurance that the workforce is trained and equipped to be meaningful users of EHRs.²

To advance the meaningful use of HIT and EHRs, the Certification Commission for Health Information Technology (CCHIT), a private nonprofit group, was funded in 2005 by the U.S. Department of Health and Human Services to develop a set of standards for EHRs that define meaningful use and interoperability among systems.³ “Standardized” EHRs have a core set of features defined by the concept of meaningful use and have to be interoperable across hospitals, physician practices, and service providers such as laboratories and pharmacies.⁴

While ONC has defined in some detail what meaningful use is and CCHIT has translated these criteria into features EHRs should have, few studies have explored how users might perceive meaningful use or how the federal government’s definition of meaningful use might impact user satisfaction and usability. This study assesses how features defined as essential for meaningful use impact user satisfaction and perceptions of usability of an operational EHR in the Kentucky Department of Corrections.

Kentucky Department of Corrections EHR

The Kentucky Department of Corrections (KyDOC) includes 13 state-operated prisons with medical facilities including a pre- and postoperative hospital unit and a psychiatric unit. All clinics are responsible for providing primary care, dental, and mental health services. Several years before the push by the federal government to adopt EHRs, KyDOC began planning a transition from a paper medical record to an EHR. KyDOC recognized that many healthcare experts view EHRs as a key management tool to improve medical services and patient health, while also helping control costs, and began in 2004 to assess benefits and costs of adopting a systemwide EHR.^5-8 After securing funds to adopt an electronic system, KyDOC administrators worked closely with medical staff in the facilities who would be using the EHR to plan and implement the system.

Most workers in large organizations who have been responsible for implementing new technologies or were required to learn and use them know that these innovations and adoptions are seldom free of problems and issues.^9-13 There usually is at least some resistance and dissatisfaction among users and implementers of new technologies, and in some cases, resistance and dissatisfaction contribute substantially to delays or even failure.¹⁴ However, in 2006, approximately one year after initial adoption, a basic system was in place across all 13 state facilities. This successful initial implementation can be attributed in part to the relationship established between management and staff, but also to the procurement process and the criteria used to select an EHR vendor.

EHR Requirements and Expectations

Vendors competing for the EHR contract were required to provide a system that was Web-based, was easy to use, and would not require an investment in local IT infrastructure. The system was also required to have electronic prescribing; receive lab results electronically; use structured data and nomenclature provided by SNOMED (Systematized Nomenclature of Medicine), NDC (National Drug Code), or other data dictionaries for documentation; and have the ability to generate clinical, administrative, and demographic reports. Table 1 lists and describes KyDOC’s EHR features as they relate to meaningful use.

Table 1: KyDOC EHR Features Related to Meaningful Use

Methods

Although not the only source of information for satisfaction and perception of usability as they relate to “meaningful use” features, the Kentucky Corrections Health Services Network (KCHSN) at the University of Kentucky believes healthcare professionals in the KyDOC system who use the EHR are in the best position to provide this information. To help determine how users evaluate features of the EHR and to assess user satisfaction systematically, the KCHSN conducted a survey of all users beginning in April 2010.

After University of Kentucky IRB approval of the research protocol was obtained on April 7, 2010 (10-0170-PH2), a draft survey instrument was developed and tested with a small sample of KyDOC users, who were then excluded from participating in the final survey. After piloting the instrument and making changes to some items and excluding others, the population was identified by log-on during the month of April 2010. The survey was made available to all 345 users via the online survey tool SurveyMonkey. The initial invitation to complete the survey was sent to all users by e-mail on June 3 with a reminder sent on June 8 and a final reminder sent on June 17.

Administration of a survey via online tools such as SurveyMonkey has shortcomings and limitations, especially around recruitment. Invitations via e-mail can be perceived as impersonal and easy to dismiss or ignore; however, for the corrections population, SurveyMonkey was a more tenable option to reach all EHR users compared to pencil and paper. In fact, the population is dispersed across 13 facilities and works three different shifts, and only primary care providers and nurse administrators have periodic meetings that might make paper administration more feasible. Nurses, who are the majority of users, would be very difficult to reach and encourage to respond to a paper survey. Furthermore, since security is the primary mission of corrections systems, an online survey eliminates the added burden on security to escort and protect researchers whom corrections officers consider a potential breach to their primary mission.

The instrument comprises three sections:

1. 14 items related to satisfaction with features,
2. 13 items related to perceptions of usability, and
3. demographics.

The instrument is provided in the Appendix. Participants responded to the satisfaction and usability items using a four-point Likert scale ranging from “very satisfied” to “very dissatisfied” for satisfaction and “strongly agree” to “strongly disagree” for usability. Negatively worded usability items were recoded so that all statements had the same direction for analysis. Averages for the 14-item satisfaction and 13-item usability measures, discussed below, provide overall scores for user satisfaction and perceptions of usability.

Results

A total of 173 out of 345 users participated, resulting in a 50.1 percent response rate and a sampling error of 5.27 percent at a confidence level of 95 percent. Four respondents did not indicate their position, leaving only 169 respondents grouped by type of position.

Respondent Characteristics

As expected, the largest groups of respondents were nurses (47.3 percent), followed by other staff (20.1 percent) and primary care providers (14.2 percent). The response rates for categories of users ranged from 44 percent for other staff to 75 percent for nurse administrators, shown in Table 2.

Table 2: Categories of Respondents and Response Rates

The survey items on length of employment in KyDOC and months of experience using the KyDOC EHR found a relatively experienced workforce, with 168 respondents averaging almost five years of KyDOC employment and 167 respondents averaging about 2.5 years of experience with the KyDOC EHR. As shown in Table 3, a large proportion of KyDOC users believe they are highly skilled or computer literate; however, there is no significant difference in satisfaction and usability among the levels of perceived computer literacy.

Table 3: Satisfaction and Usability Mean by Computer Literacy

Users with other EHR experience consider themselves to be very computer literate. Seventy-one users (42 percent) say they have experience with other EHRs, and the majority of these experienced users (86 percent) consider their computer literacy as advanced or expert compared to 69 percent of users without other EHR experience for an F-value of 3.5, significant at the .06 level.

Satisfaction and Usability

A factor analysis was conducted to measure internal variability among satisfaction and usability items. Three factors emerged for satisfaction and explained 78.9 percent of the variance, and four factors, explaining 64.9 percent of the variance, emerged for usability. The factor analysis was based on the principal components method with eigenvalues greater than or equal to 1.0. There was no theoretical basis upon which to base a rotated solution; therefore, none was selected. Cases were excluded using the listwise method, and components were suppressed for values less than or equal to .4. Reliability findings for the satisfaction (.94) and usability (.82) scales suggest that the items consistently measured how satisfied users were with features and the extent to which they found the system usable.

Table 4 shows that averaging 14 satisfaction items and 13 items measuring perceptions of usability produced relatively high mean scores of 3.0 for satisfaction and 2.8 for usability on a four-point scale. The summary measures of satisfaction with features and perception of usability were correlated moderately and positively (Pearson r = .56, p < .01).

Table 4: Mean Satisfaction and Usability Scores

Another way to describe the data is to categorize mean satisfaction and perceptions of usability. We identified three categories for both measures ranging between 1.00 and 4.00: unsatisfied or negative respondents (2.00 or less); neutrals (2.01–2.99); and satisfied or positive respondents (3.00 or greater). Using this measure found a majority of users who were satisfied with the EHR; however, most users were only neutral about the usability of the system, as shown in Figure 1.

Figure 1: Categorization of Satisfaction and Usability Scores

Table 5 and Table 6 summarize scores for each item of satisfaction and usability across all users in the sample. Table 5 shows that users were most satisfied with vital signs options, followed by the ability to document episodic visits, the ability to document chronic care visits, and the vital signs format. Users were least satisfied with the ability to track health maintenance, notification of lab alerts, the ability to track clinical notes by disease state, lab reports, and lab sign-offs.

Table 5: Satisfaction with EHR Features

Table 6: Perception of EHR Usability

Perhaps the strongest indicator of overall system usability, shown in Table 6, a large majority of users disagreed they preferred the paper record to the EHR, which was the most positively statement regarding usability. Users also indicated that the EHR improved quality of care, enhanced documentation, and enhanced productivity. However, users indicated:

1. the EHR had not decreased the use of paper
2. updates about new and enhanced features were not timely
3. the EHR had not decreased duplication, and
4. there was no need for additional training on the system.

In addition to different perceptions of satisfaction and usability among individual items, Table 7 shows average levels of satisfaction and usability among different types of position in the corrections system. For the satisfaction scale, nurse administrators and dentists rated the system lower than any other category of user. The largest group (nurses) indicated they were overall satisfied with the system, and psychologists were the most positive. However, dentists alone were the lowest scoring group for overall usability, and psychologists again found the system to be most useful.

Table 7: Mean Satisfaction and Usability Scores by Respondent Position

Using the one-way ANOVA test and excluding cases based on the listwise method, the differences in satisfaction and usability among staff positions have F-values of 2.96 (satisfaction) and 4.04 (usability), which are statistically significant at the .014 level for satisfaction and the .002 level for usability. Despite categories of users significantly perceiving satisfaction and usability differently, users in all types of positions rated satisfaction with the EHR higher than its usability, as depicted in Figure 2.

Figure 2: Comparison of Satisfaction and Usability Means

Discussion

Several variables thought to be related to satisfaction and usability had no statistically significant relationships. First, the degree to which users self-identify themselves as computer literate was not related to overall satisfaction with or perception of the system. Somewhat surprisingly, user self-identification of their computer literacy was not related to the item that there should be more training opportunities, that is, both advanced and expert users, as well as average users, responded more or less the same.

Also, we found no statistically significant difference in satisfaction and perceived usability between those with or without other EHR experience and the length of time they have used the KyDOC EHR. Researchers have noted a similar finding in which some users with previous EHR experience perceived a new system negatively while others viewed it positively.¹⁵

The absence of differences among users with varying computer literacy and experience with other EHRs may be explained by the KyDOC EHR system’s Web-based design, which many users may find easy to use, thereby minimizing the importance of computer expertise and previous EHR experience. These results also suggest that users tended to evaluate the KyDOC EHR as a stand-alone system; that is, their previous experience with other electronic health records had little or no relationship to how they perceive the KyDOC system.

Three of the five lowest mean scores for satisfaction, shown in Table 5, relate to the lab interface: alerts, reports, and sign-offs. Lab alerts and sign-offs are related features in which many providers rely upon the system to notify them that they have labs to sign. Dissatisfaction with the lab interface was known prior to this study, and changes to the lab interface whereby ordering providers will be notified of results and panic values in multiple places are pending and may decrease users’ frustration with this feature.

Once the changes are implemented, KCHSN will evaluate the impact on provider behavior, such as the extent to which providers are signing their lab results. KCHSN also will proactively seek and encourage feedback from users, in this case providers, about their experiences with this augmented feature.

The usability item “prefer paper to EHR,” which was recoded to indicate preference for the EHR, was the highest rated of all 27 survey items with an average score of 3.23. Although the item was thought initially to be a strong indicator of both usability and satisfaction, it was related significantly only to usability (Pearson r = .56, p < .01). While some users expressed dissatisfaction with some aspects of the EHR, the “prefer paper to EHR” item suggests that perceived usability weaknesses were not severe enough to make a return to paper charts desirable.

This finding is quite positive since KyDOC early in its adoption process sought to make as near as possible a break from paper with the exception of documents that patients must sign. A study examining the use of paper in electronic systems found users often document around or “work around” their EHR for reasons of convenience and efficiency.¹⁶ The use of paper for some users may ameliorate frustrations with EHRs as well as enable them to resist change.¹⁷ However, KyDOC addressed this issue by no longer funding supplies for the paper system and by using the reporting feature to monitor usage, such as the number and extent of log-ons, medication orders, and patients with electronic notes. Thus, the preference for the EHR over paper among most users is especially significant since paper documentation is difficult to complete, runs contrary to institutional policy, and can be easily discovered by administrators.

Findings from the few studies evaluating EHR satisfaction and usability indicate that users rate satisfaction favorably but do not always perceive productivity positively. Researchers have found that nurses from different settings believed the EHR had an overall positive effect on the quality of care they provide (67 percent), but only 31 percent believed that the system improved their ability to see more patients (an indicator of usability).¹⁸ Some studies have suggested that user satisfaction with a system can be positive while perception of usability can be low, which appears counterintuitive and may indicate that the study evaluated factors not specifically related to EHRs.¹⁹ Unlike other studies, as noted above, we found a significant relationship between satisfaction and usability.²⁰

We also found that categories of staff varied significantly in their satisfaction and perceived usability. This finding is similar to that of other studies, which found that physicians, nurses, and medical secretaries were satisfied with the content, accuracy, format, user friendliness, and speed of their EHR system; however, medical secretaries rated the system higher than nurses and physicians.²¹ Differences in user satisfaction and perception of usability may be explained by the different ways in which types of staff use EHRs.

For example, nurse administrators and dentists were least satisfied with the KyDOC EHR, and dentists found the system least usable. We know based on the nurse administrator job description that nurse administrators use the system in different ways than most other categories of users who primarily document encounters. Nurse administrators primarily manage clinic operations and need access to reports regarding diagnoses, staff usage, and computerized order entries. While these data are available, the reporting tool is accessible mostly to KyDOC administrators and requires expertise in database management and statistics, which most nurse administrators do not have. We also know the system does not include dental charting, which is available in some electronic charting systems specific to dentistry. This may explain why dentists rated satisfaction and usability low.

The findings of this study, particularly differences among staff, may be informative for other institutions transitioning to EHRs, as well as ones that already have adopted electronic systems; however, the following limitations contextualize how these findings should be interpreted. This study examined only one corrections system and had a 50.1 percent response rate. The response rate may have been greater had we identified corrections staff at all 13 facilities to promote the study; however, second- and third-shift employees would still be a challenge to recruit. To address the challenges of surveying this population electronically or via paper-and-pencil methods, we plan to contextualize and better understand our findings with in-depth interviews.

We also had to develop a survey instrument since our purpose was to learn about users’ satisfaction with and perceptions of usability of features specifically relating to the federal government’s definition of meaningful use. Existing instruments, such as the IBM Computer Usability Satisfaction Questionnaire and the EHR User Satisfaction Survey, are more broad and do not specifically address issues of meaningful use as defined here.^22,23

Conclusion

A 2010 survey of corrections medical staff found that, with few exceptions, users were satisfied with the features of the EHR. KyDOC did not set out in 2004 to implement an EHR that met ONC “meaningful use” guidelines, which at the time were still being developed. However, several years after the KyDOC implementation, the ONC has defined more clearly what meaningful use is and what features lend themselves to meaningful use.²⁴

Unlike the national effort to create EHRs that will be interoperable across all types of systems, KyDOC had to adopt one system for 13 state prisons that use one pharmacy and one lab vendor. Despite having only one pharmacy and one laboratory vendor, these interfaces required years to complete. Other organizations planning to adopt an EHR system likely will need to interface with many pharmacies and laboratories.

What lessons does this research hold for the ONC’s goal that healthcare providers and organizations adopt and meaningfully use HIT and “standardized” EHRs? It should be kept in mind that with new adoptions of EHRs and user evaluations of the systems, there usually is no clear or widely accepted frame of reference or comparison. For example, we found that users sometimes seemed to be criticizing or evaluating the new EHR system compared to some perceived ideal electronic system rather than the previous paper record system.

We have much anecdotal and unsystematic evidence of complaints about and shortcomings of the previous paper record system, but we did not conduct an empirical study of the previous system and have not found such studies in other organizations. Despite the lack of a systematic and empirical before/after comparison, users indicated they believed the EHR improved quality of care, and an overwhelming majority preferred the electronic system to paper.

Other lessons, we believe, may be instructive for healthcare providers and organizations transitioning to EHRs. While all certified EHR systems will include features required to meet the federal government’s definition of meaningful use, these systems will not all be similar and will influence how satisfied staff will be and how usable staff will find their electronic record. Thus, the ongoing adoption process, including selection of the system and implementation of new technologies as they are developed, is critical for success. The “prefer paper to EHR” item is our only direct comparison between the paper and electronic system, but we have no other benchmark to compare the two.

Our experience with adopting and using an EHR in a state corrections system provides a positive but cautionary story. We had no failures, and most implementation problems were resolved with minimal stress or conflict, but we emphasize that the organizational setting was relatively stable, controlled, and less complex than many other organizational settings. Also, staff who would be using the system were involved early in the implementation of the system. We also did not face the dozens or even hundreds of interfaces that would have to be developed in other settings.

Although corrections medical services are somewhat unique in the American healthcare industry, findings from our EHR experience might be useful for other organizations planning to adopt an effective EHR system. Healthcare professionals across different types of healthcare systems will share similar “meaningful use” expectations and needs. Our experience suggests that the ONC’s goal of expanded adoption of EHRs across the healthcare industry will be a slow, expensive, and sometimes painful process but if successful will likely improve healthcare quality and help control costs.

Madison L. Gates, PhD, is an assistant professor in the Department of Family and Community Medicine at the University of Kentucky in Lexington, KY.

Phillip W. Roeder, PhD, is a professor in the Department of Family and Community Medicine at the University of Kentucky in Lexington, KY.

Notes

¹ U.S. Department of Health and Human Services, Office of the National Coordinator for Health Information Technology. Health IT.  (accessed September 21, 2010).

² U.S. Department of Health and Human Services, Office of the National Coordinator for Health Information Technology. HITECH Programs.  (accessed September 21, 2010).

³ U.S. Department of Health and Human Services, Office of the National Coordinator for Health Information Technology. Being a Meaningful User of Electronic Health Records.  (accessed September 21, 2010).

⁴ Ibid.

⁵ Bates, D. W. “Getting in Step: Electronic Health Records and Their Role in Care Coordination.” Journal of General Internal Medicine 25, no. 3 (2010): 174–76.

⁶ Chaudhry, B., J. Wang, S. Wu, M. Maglione, W. Mojica, E. Roth, S. C. Morton, and P. G. Shekelle. “Systematic Review: Impact of Health Information Technology on Quality, Efficiency, and Costs of Medical Care.” Annals of Internal Medicine 144, no. 10 (2006): 742–52.

⁷ Gagnon, M. P., M. Desmartis, M. Labrecque, F. Legare, L. Lamothe, J. P. Fortin, J. F. Rancourt, and J. Duplantie. “Implementation of an Electronic Medical Record in Family Practice: A Case Study.” Informatics in Primary Care 18, no. 1 (2010): 31–40.

⁸ O’Malley, A. S., J. M. Grossman, G. R. Cohen, N. M. Kemper, and H. H. Pham. “Are Electronic Medical Records Helpful for Care Coordination? Experiences of Physician Practices.” Journal of General Internal Medicine 25, no. 3 (2010): 177–85.

⁹ Aranow, M. “Instructive Resistance.” Health Management Technology 24, no. 1 (2003): 76, 75.

¹⁰ Koo, C. C. “Getting Ready for an EMR: Physician Practices Should Consider Several Key Factors before Selecting an Electronic Medical Record System.” Health Management Technology 25, no. 8 (2004): 26, 28.

¹¹ Miller, R. H., and I. Sim. “Physicians’ Use of Electronic Medical Records: Barriers and Solutions.” Health Affairs (Millwood) 23, no. 2 (2004): 116–26.

¹² Timmons, S. “Nurses Resisting Information Technology.” Nursing Inquiry 10, no. 4 (2003): 257–69.

¹³ Valdes, I., D. C. Kibbe, G. Tolleson, M. E. Kunik, and L. A. Petersen. “Barriers to Proliferation of Electronic Medical Records.” Informatics in Primary Care 12, no. 1 (2004): 3–9.

¹⁴ Timmons, S. “Nurses Resisting Information Technology.”

¹⁵ Ludwick, D. A., and J. Doucette. “Adopting Electronic Medical Records in Primary Care: Lessons Learned from Health Information Systems Implementation Experience in Seven Countries.” International Journal of Medical Informatics 78, no. 1 (2009): 22–31.

¹⁶ Saleem, J. J., A. L. Russ, C. F. Justice, H. Hagg, P. R. Ebright, P. A. Woodbridge, and B. N. Doebbeling. “Exploring the Persistence of Paper with the Electronic Health Record.” International Journal of Medical Informatics 78, no. 9 (2009): 618–28.

¹⁷ Ibid.

¹⁸ de Veer, A. J., and A. L. Francke. “Attitudes of Nursing Staff towards Electronic Patient Records: A Questionnaire Survey.” International Journal of Nursing Studies 47, no. 7 (2010): 846–54.

¹⁹ Ibid.

²⁰ Ibid.

²¹ Laerum, H., T. H. Karlsen, and A. Faxvaag. “Use of and Attitudes to a Hospital Information System by Medical Secretaries, Nurses and Physicians Deprived of the Paper-based Medical Record: A Case Report.” BMC Medical Informatics Decision Making 4 (2004): 18.

²² Lewis, James R. “IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use.” International Journal of Human-Computer Interaction 7, no. 1 (1995): 57–79.

²³ Edsall, R. L., and K. G. Adler. “The 2009 EHR User Satisfaction Survey: Responses from 2,012 Family Physicians.” Family Practice Management 16, no. 6 (2009): 10–16.

²⁴ U.S. Department of Health and Human Services, Office of the National Coordinator for Health Information Technology. Being a Meaningful User of Electronic Health Records.