by Anjum Khurshid, PhD, MPAff, MBBS; Mark L. Diana, PhD; and Susan D. Luce, MPH
The purpose of this study was to find out how many health information exchanges (HIEs) are using metrics to gauge their impact and to gather examples of metrics being used by HIEs.
We administered a web-based survey to a list of functioning HIEs. Valid responses were received from 18 HIEs. Most respondents were nonprofits, most had sustainable business models, and half used metrics to gauge their impact. Reduction of duplicative testing, quality improvement efforts, care coordination, and improved readmission rates were cited as having the greatest potential to show return on investment. Most respondents selected patient-centered care as having the greatest potential to show quality improvement. The major limitation of this study is the low response rate. This study suggests that there are no standard metrics used by HIEs to evaluate their impact. The HIE community needs to take the lead in developing metrics to evaluate the benefits of information exchange.
Key words: heath information exchange, quality, return on investment
A recent assessment of the progress toward use of health information technology (HIT) and health information exchanges (HIEs) found a persistent lack of clarity on the value of electronic health records (EHRs) and HIEs.1 The authors of the assessment recommended advancing policies that clearly articulate and measure the value of HIEs to consumers and patients so that system-level transformations will find the support of different stakeholders to align incentives for exchange of health information. This recommendation is significant in the context of the federal government’s allocation of billions of dollars under the 2009 Health Information Technology for Economic and Clinical Health (HITECH) Act, aimed at promoting meaningful use of health information technology, including adoption of EHRs and care coordination through HIEs.2
The last decade has seen significant progress in HIE technologies and substantial investments in HIT adoption, yet the lack of evidence on the value delivered by such efforts remains a major hurdle in making a strong case for both adoption and investment at the local level.3 The low rates of adoption and investment in HIEs may be attributed to the relatively nascent stage of evolution of the industry, the resistance to change of the healthcare delivery system, or the lack of incentives for efficient exchange of information, quality improvement, and safety built into the existing reimbursement and regulatory environment. Hence, the question remains whether the additional influx of dollars through current government programs can solve the underlying problem in implementing HIE.
While most HITECH dollars will be paid as Medicaid and Medicare incentives, approximately $2 billion will be distributed to advance the creation and expansion of HIE infrastructure and services.4 HIEs are also a cornerstone of the Nationwide Health Information Network (NHIN) being pursued by the Office of the National Coordinator for Health Information Technology (ONC).5 However, successful implementation of the NHIN relies on state, regional, and local HIEs developing sustainable and effective models of sharing electronic information that can demonstrate improvements in efficiency and effectiveness of health service delivery and a return on investment (ROI). This means that to convert the stimulus funding into a transformational long-term regional strategy, a case has to be made to private-sector, local, and regional stakeholders to invest in HIE initiatives. While most experts agree with the transformational potential of HIEs to support a system that delivers coordinated, affordable, and quality healthcare services, we lack robust empirical evidence of HIEs improving quality of care and providing value to patients and to participating entities.6,7
The number of communities establishing HIEs has increased following the large federal investments.8 Several major efforts have been undertaken to develop an inventory of all such programs in the country. An annual survey of HIEs by the eHealth Initiative (eHI) is probably the most recognized source for such information.9 Researchers at Harvard University have also conducted a survey of HIEs.10,11 The Healthcare Information and Management Systems Society (HIMSS) also conducted a study of operational HIEs.12 The combined results of these efforts suggest that the number of entities planning or conducting HIEs may be in the low hundreds. However, a recent update showed that while the numbers of HIEs are increasing, many are limited in scope with uncertain sustainability.13
The evidence of the transformational role of HIEs in healthcare delivery systems is limited.14 As with the introduction of most technological innovations, the promise of revolutions and system change abound.15,16 However, such promises assume that human behavior will change as easily as information systems are changed, and that system incentives will be altered to bring about a reduction in healthcare spending.17 Nevertheless, there have been efforts to estimate the potential cost savings achieved through HIE. Walker et al. estimated the benefits of HIEs for the national healthcare system to be close to $77 million each year, based on reduced duplicate testing and increased care coordination.18 Frisse and Holmes estimated more than $8 million savings per year from using HIEs in urban emergency settings in Tennessee.19 Hook et al. calculated that the potential benefits of an HIE in New York state, when fully implemented, would be around $4.54 billion per year.20 Kho et al. estimated that three healthcare systems in Indianapolis could save $2.3 to $4.6 million per year by sharing information.21
The purpose of our study was to find out how many fully functional HIEs are using metrics to gauge the impact of HIEs on quality improvement and ROI. The study was also aimed at gathering some examples of metrics that may be helpful to newly forming and already established HIEs. Such evidence, we believe, will be key in pursuing continued investment in HIE initiatives by communities, healthcare systems, government, and others.
Identification of Eligible HIEs
We used the eHI directory to identify HIEs.22 We also performed Google searches using each state’s name plus the term health information exchange. We identified 230 initiatives existing as of February 2010. We used the seven stages of HIE development as described by the eHI to classify these 230 initiatives. We focused on the HIEs identified as being at stage 5 or above in the eHI directory and others that could possibly be at stage 5 based on the description presented on their website.
Survey Development and Content
Our survey consisted of 21 questions. The first half of the survey focused on organizational demographics, and the second half focused on the organizations’ use of metrics related to ROI and quality of care. Quality of care was defined using the Institute of Medicine’s quality domains: safety, timeliness, effectiveness, efficiency, equitability, and patient-centeredness.23 Only the organizations exchanging data as of January 1, 2010, were asked to complete the survey. Respondents were given the opportunity to have the name of the HIE withheld from the final survey results.
We used the eHI’s stages 5 to 7 of HIE development as the inclusion criteria since these stages include fully operational HIEs that are in a position to measure the impact of their operations on the delivery of healthcare services. Stage 5 is defined as “fully operational health information organization; transmitting data that is being used by healthcare stakeholders.” Stages 6 and 7 relate to operational HIEs having a sustainable business model and demonstrating expansion of the coalition from the initial operation model, respectively. We sent the survey to a wider list of organizations and asked them to respond only if they fulfilled our criteria of being actively engaged in the exchange of health information (at least stage 5). In this way, we were less likely to leave a functional HIE out of the study. We started with a list of 149 potential participants that had a website or contact information available. Following the initial e-mail request to complete the survey, at least two phone calls were made to each HIE with a valid phone number, and two additional attempts were made to obtain responses from nonrespondents.
We sent the survey to 149 organizations but asked them to respond only if they fulfilled the eHI’s stage 5 criteria. Twenty-three e-mails either were returned as undeliverable or received a representative’s response saying the organization was not functioning as a HIE. Thirty organizations had no valid phone number and could not be reached to confirm an e-mail address. Our final sample consisted of 96 HIEs that we identified as potentially meeting stage 5 criteria. Responses were received from 21 organizations (a response rate of approximately 22 percent), with 18 of those actually exchanging data as of January 1, 2010, and being at a stage of development between 5 and 7. We analyzed the survey results of the 18 organizations that fulfilled the inclusion criteria. Characteristics of the respondent organizations are listed in Table 1.
Summary Characteristics of HIE Respondents (n = 18)
|Type of organization
|Number of patients served annually
|Annual operating budget
Our results show that most HIE respondents operate as nonprofit organizations, serve fairly large patient populations, and have annual budgets in excess of $1 million. Hospitals, ambulatory care clinics, and laboratories are the most frequent participants in electronic HIEs. The top three categories of data being exchanged were test results, medication summaries, and ambulatory care patient information, in that order.
The remainder of the survey focused on metrics for ROI and quality of care. Ten respondents (56 percent) said that based on the performance of their own HIE, they believed that HIEs show positive ROI, while 8 respondents (44 percent) felt more evidence was needed to make such a determination. Two respondents who believed HIEs show positive ROI stated that they have not used metrics to calculate ROI but are in the process of developing ROI metrics. Seventeen respondents (94 percent) believed that HIEs improve quality of care.
Reduction of duplicative testing, quality improvement efforts, care coordination, and improved readmission rates were cited as having the greatest potential to show ROI. Sixty-one percent selected patient-centered care as having the greatest potential to show quality improvement, with specific examples related to information being available at the point of care. Efficiency from reductions in the length of stay and duplicative testing was cited by 56 percent of respondents. Reduced errors and medication reconciliation were listed as examples of safety improvements that improve quality (see Table 2).
Areas That HIE Representatives Believe Have the Greatest Potential to Show ROI and Quality Improvement (n = 18)
| Reduction in duplicative testing
| Quality improvement efforts
| Care coordination
| Improving readmission rates
| Simplifying administrative services
| Managing chronic diseases
| Population health management
| Improved timeliness of data sharing
| Patient-centered care
More than half of the organizations have used or plan to use a reduction in duplicative tests or procedures, improved communication among providers, and improved health outcomes to measure the impact on ROI. Thirty-nine percent of the respondents listed specific metrics currently being used to determine ROI. Forty percent of the organizations that were using some ROI metrics also believed more evidence was needed to show a positive impact of HIEs.
Fifty percent of the HIEs reported using metrics for quality improvement. Of these, most organizations tracked quality improvement in a number of different areas related to clinical outcomes and some preventive measures, including readmissions, vaccination rates, diabetes management, and cancer screening.
Our study suggests that only 50 percent of the HIEs in our survey use or plan to use metrics to measure the impact of the HIE. Most respondents agree that HIEs add value in quality improvement and offer returns on investment. Reduction of duplicate tests, better coordination of care for patients with chronic conditions, and decreased readmissions seem to be the metrics quoted most often in making a business case for HIEs. Patient-centered approaches, efficiency, and safety were the most often cited areas of quality improvement.
Our results provide a list of metrics that HIEs have used or intend to use for measuring quality improvements and ROI. The development of specific measures under the meaningful use definition has greatly facilitated the process of developing a consensus list of measures that can be used to show improvements in quality.24 The development and reporting of these measures would not be easy given that our results show that even the most established HIEs are not using metrics to measure their impact.
According to the eHI survey, only 27 HIEs had a sustainable business model in 2009.25 Hence, the eHI recommended that HIEs “measure and document improvements in patient care, savings, and value.”26 An unprecedented one-time infusion of funds from the federal government is unlikely to improve the long-term sustainability for most HIEs. Budgetary constraints at the state level and the economic slowdown will only enhance the sustainability challenge for HIEs. State-designated entities (those organizations designated to receive HITECH funding) and other stakeholders will have to decide who is benefitting from HIE and who should fund the ongoing operation of HIEs that are emerging as a result of the federal stimulus. Whether the sustainability models are developed around private-sector funding, public-sector support, or a hybrid approach, HIEs will have to demonstrate ROI through well-chosen, well-executed quality initiatives based on standardized metrics.
The major limitations of this study are the response rate and the sensitivity of the information organizations were asked to share. We recognize that the number of operational HIEs responding to our survey is small. However, the 2009 eHI survey identified 57 operational HIEs,27 Harvard University’s survey found 55 fully operational HIEs,28 and the HIMSS HIE Common Practices Survey found only 21 HIEs actively exchanging information.29 Using these numbers for the known operational HIEs would generate a response rate ranging from 32 to 86 percent for our survey. Further, most HIEs face the challenge of showing ROI to their stakeholders, and details on such metrics may have been perceived by some to be too sensitive to share publicly.
A major concern is the lack of preparedness of established HIEs to measure and document improvements in quality and to show the value of the exchange of health information. In our view, the most likely reason for the lack of such metrics is the absence of incentives to develop and use them. HIEs need to incorporate quality improvement and ROI metrics into their business models. In the absence of such planning, many HIEs may fail to meet the federal government’s criteria for meaningful use and face penalties. The federal funds have provided an impetus to the HIE movement to promote meaningful exchange of health information.30 It is now up to both the newly formed and well-established HIE entities, which rushed to secure public funds with a promise to demonstrate improvements and successful exchange in the next few years, to show the same enthusiasm to incorporate metrics into their model and share their successes with each other.
Our experience of more than a decade of using HIE data for quality reporting and ROI calculations in a fully operational HIE in central Texas has shown that a sustained effort with adequate resources and a commitment to collaborate across organizational boundaries are required to develop meaningful metrics. The transformational changes that HIEs promise come much later and require trust, evidence-based approaches, perseverance, and collaboration among all healthcare stakeholders. It would be disappointing if the already high rate of HIE failures increased in the next few years as federal funds are exhausted. Adler-Milstein, Bates, and Jha reported a failure rate of about 20 percent over 17 months.31 HIEs need to come together and move toward self-regulation by establishing standards of performance that can be used to convince their stakeholders and the patients they serve of the value of their existence.
Anjum Khurshid, PhD, MPAff, MBBS, is the director of the health systems division at the Louisiana Public Health Institute in New Orleans, LA.
Mark L. Diana, PhD, is an assistant professor at Tulane University in New Orleans, LA.
Susan D. Luce, MPH, was a MPH student at the University of Texas School of Public Health in Houston, TX, at the time of the survey.
We would like to thank Julia Adler-Milstein and Ann Kitchen for their review of the initial draft of the manuscript.
1 eHealth Initiative. National Progress Report on eHealth 2010. 2010. (accessed July 6, 2010).
2 Healthcare Information and Management Systems Society (HIMSS). ARRA HITECH FAQs related to HIE. 2010. Available at http://www.himss.org/content/files/12_04_09_ARRAHITECHHIE_FactSheet.pdf (accessed July 26, 2010).
3 Vest, J. R., and L. D. Gamm. “Health Information Exchange: Persistent Challenges and New Strategies.” Journal of the American Medical Informatics Association 17, no. 3 (2010): 288–94.
4 Healthcare Information and Management Systems Society (HIMSS). ARRA HITECH FAQs related to HIE.
5 Department of Health and Human Services Office of the National Coordinator for Health Information Technology. Summary of the NHIN Prototype Architecture Contracts. May 31, 2007. (accessed March 7, 2010).
6 Vest, J. R. “Health Information Exchange and Healthcare Utilization.” Journal of Medical Systems 33, no. 3 (2009): 223–31.
7 Vest, J. R., and L. D. Gamm. “Health Information Exchange: Persistent Challenges and New Strategies.”
8 eHealth Initiative. HIE Directory. 2010. http://www.ehealthinitiative.org/directory-health-information-exchange-initiatives.html (accessed March 4, 2010).
9 eHealth Initiative. 2009 Survey: Migrating Toward Meaningful Use: The State of Health Information Exchange. 2009. (accessed March 7, 2010).
10 Adler-Milstein, Julia, David W. Bates, and Ashish K. Jha. “U.S. Regional Health Information Organizations: Progress and Challenges.” Healh Affairs 28, no. 2 (2009): 483–92.
11 Adler-Milstein, J., A. P. McAfee, D. W. Bates, and A. K. Jha. “The State of Regional Health Information Organizations: Current Activities and Financing.” Health Affairs 27, no. 1 (2008): w60–w69.
12 Healthcare Information Management and Systems Society (HIMSS) Health Information Exchange Best Practices Task Force. Health Information Exchanges: Similarities and Differences: HIMSS HIE Common Practices Survey Results White Paper. March 2009. Available at http://www.himss.org/content/files/RHIO/HIE_CommonPracticesWhitePaper20090330.pdf (accessed May 23, 2010).
13 Adler-Milstein, Julia, David W. Bates, and Ashish K. Jha. “U.S. Regional Health Information Organizations: Progress and Challenges.”
14 Vest, J. R., and L. D. Gamm. “Health Information Exchange: Persistent Challenges and New Strategies.”
15 Shapiro, J. S. “Evaluating Public Health Uses of Health Information Exchange.” Journal of Biomedical Informatics 40, no. 6 (suppl.) (2007): S46–S49.
16 Diamond, C. C., and C. Shirky. “Health Information Technology: A Few Years of Magical Thinking?” Health Affairs 27, no. 5 (2008): w383–w390.
17 Congressional Budget Office. Evidence on the Costs and Benefits of Health Information Technology. 2008. Available at http://www.cbo.gov/ftpdocs/91xx/doc9168/HealthITTOC.2.1.htm (accessed June 7, 2010).
18 Walker, Jan, Eric Pan, Douglas Johnston, Julia Adler-Milstein, David W. Bates, and Blackford Middleton. “The Value of Health Care Information Exchange and Interoperability.” Health Affairs (suppl.) (2005): W5–W18.
19 Frisse, M., and R. Holmes. “Estimated Financial Savings Associated with Health Information Exchange and Ambulatory Care Referral.” Journal of Biomedical Informatics 40, no. 6 (2007): S27–S32.
20 Hook, Julie M., Eric Pan, Douglas Johnston, J. Adler-Milstein, D. W. Bates, and Blackford Middleton. “The Value of Healthcare Information Exchange and Interoperability in New York State.” Paper presented at the AMIA Annual Symposium, Washington, DC, November, 2006.
21 Kho, A. N., L. Lemmon, M. Commiskey, S. J. Wilson, and C. J. McDonald. “Use of a Regional Health Information Exchange to Detect Crossover of Patients with MRSA between Urban Hospitals.” Journal of the American Medical Informatics Association 15, no. 2 (2008): 212–16.
22 eHealth Initiative. HIE Directory. 2010. http://www.ehealthinitiative.org/directory-health-information-exchange-initiatives.html (accessed March 4, 2010).
23 Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press, 2001.
24 Department of Health and Human Services. “Medicare and Medicaid Programs; Electronic Health Record Incentive Program; Final Rule.” 42 CFR Parts 412, 413, 422, et al. Federal Register 75, no. 144 (July 28, 2010): 44313–588. Available at http://edocket.access.gpo.gov/2010/pdf/2010-17207.pdf.
25 eHealth Initiative. 2009 Survey: Migrating Toward Meaningful Use: The State of Health Information Exchange.
28 Adler-Milstein, Julia, David W. Bates, and Ashish K. Jha. “U.S. Regional Health Information Organizations: Progress and Challenges.”
29 Healthcare Information Management and Systems Society (HIMSS) Health Information Exchange Best Practices Task Force. Health Information Exchanges: Similarities and Differences: HIMSS HIE Common Practices Survey Results White Paper.
30 Blumenthal, David, and Marilyn Tavenner. “The ‘Meaningful Use’ Regulation for Electronic Health Records.” New England Journal of Medicine 363, no. 6 (2010): 501–4.
31 Adler-Milstein, Julia, David W. Bates, and Ashish K. Jha. “U.S. Regional Health Information Organizations: Progress and Challenges.”
Khurshid, Anjum; Diana, Mark L; Luce, Susan D.
"Health Information Exchange: Metrics to Address Quality of Care and Return on Investment"
Perspectives in Health Information Management
(Summer, July 2012).