Development and Testing of a Survey Instrument to Measure Benefits of a Nursing Information System

by Amany A. Abdrbo, BSN, MSN, PhD; Jaclene A. Zauszniewski, PhD, RN-BC, FAAN; Christine A. Hudak, RN, PhD, CPHIMS; and Mary K. Anthony, PhD, RN, CS

Abstract

Information systems (IS) benefits for nurses are outcomes related to the tangible products or improvements that nurses realize from using IS. This study examined the development and psychometric testing of a measure of nurses’ benefits from IS.

A random sample of 570 nurses working in hospitals, providing direct patient care, and using IS completed the study questionnaire. The internal consistency reliability of the results was .97. Exploratory factor analysis, using principal components extraction and varimax rotation, revealed items loaded on four factors (saving time and efficiency, quality of care, charting, and professional practice) that were confirmed by confirmatory factor analysis. Continued refinement of the instrument is needed with more diverse samples of nurses.

Key words: nursing information systems, benefits, instruments, psychometrics

Introduction

Information systems (IS) benefits for nurses are defined as the outcomes related to the tangible products or improvements that nurses realize from using IS in nursing. There are few studies that have investigated IS benefits in nursing. Among those studies, some have become outdated because of the fast development of IS in healthcare.^1–3 Some studies have assessed IS benefits using existing instruments that were developed to evaluate specific types of IS, such as physician order entry (POE) systems, electronic documentation systems, and management information systems.^4-6 More recent studies have examined modified forms of previously developed questionnaires to evaluate nurses’ beliefs about specific types of IS, such as POE.⁷

However, using system-specific instruments limits the applicability of results and does not facilitate understanding of IS benefits in general. Moreover, there is no instrument that measures all components of IS benefits. Therefore, in this study, a comprehensive list of items derived from instruments measuring IS benefits was examined. The list encompassed IS benefits that are relevant to nurses and their practice. This study examined the development and psychometric testing of a measure of perceived IS benefits for nurses.

Background of the IS Benefits Instrument

A comprehensive list of IS benefits (56 items) was obtained from previous studies. These items reflected improvements related to quality of care; communication and documentation; saving time and efficiency; and professional practice. An explanation of each component of the instrument is provided.

Benefits Related to Quality of Care

Benefits related to quality of care from using IS are improvements related to the accessibility, promptness, and completeness of patient information that enhance the effectiveness of nursing care. Some studies in nursing informatics included quality of care as a benefit of using IS.^8-10 The way that quality of care is enhanced from IS use can be assessed by looking at aspects that affect patient care such as improvements related to accessing patient information; obtaining more prompt and complete patient information; obtaining more uniform information about patients; and processing patient admissions more efficiently.¹¹ Nauright and Simpson¹² reported high reliability (Cronbach’s alpha = .94) for the quality-of-care items that were included in the questionnaire they used in their study of 697 nurses and general hospital staff. In addition, previous studies supported the use of an item measuring “improved quality and administration of patient care overall.”^13-17 An item addressing the need to help nurses to set priorities better and faster was added from Axford and Carter’s study.¹⁸

Benefits Related to Communication and Documentation

Communication and documentation are means for exchanging data and information. IS can facilitate communication between and among nurses, physicians, and other health team members and improve patient outcomes. In addition, use of IS will assure completeness of patient care documentation, facilitate evaluation of patient care outcomes, and improve patient safety.

Improvements in communication and documentation have been reported in nursing studies as benefits both for electronic documentation systems and for IS in general.^19-23 Benefits related to communication and documentation have been identified mainly from the “bedside computer impact questionnaire,” which was used to evaluate a bedside computer system for nursing documentation among 28 nurses working in a medical unit.²⁴ Dennis et al.²⁵ developed the questionnaire based on experience with IS in critical care units and previous studies. In addition, these authors reported acceptable internal consistency (Cronbach’s alpha = .86) and content validity, determined by four clinical nurse specialists. Nurses’ average congruency percentage was 85 percent, and the validity index averaged .90 across all combinations of rater pairs.

Items selected and adapted for IS use include those dealing with improvements related to compliance with nursing documentation standards, charting consistency with the care plan, chart availability, and improved communication among nursing staff, between nurses and patients, and with other nurses and other healthcare team members.^26,27 Additionally, other items related to improvement in communication and documentation were added.^28-30

Benefits Related to Saving Time and Efficiency

Saving time and efficiency is the production of a desired outcome with a minimal waste of time, effort, and resources. Some studies have identified items related to saving time and efficiency.^31-37 Chin and Haughton³⁸ reported a reliability coefficient of .84 for the efficiency subscale that was developed to measure IS benefits among nursing directors.

Benefits Related to Professional Practice

Professional practice comprises the activities and qualifications that are distinctive to a specific profession. Using IS has been reported to be beneficial to nurses’ professional practice. Use of IS has increased nurses’ autonomy,³⁹ sense of professionalism,⁴⁰ and accountability.⁴¹ In addition, benefits that are indirectly related to professional practice have been reported in the literature, such as improved decision-making and patient safety.^42-46 Still other benefits have been identified, including increases in nurses’ sense of responsibility and job excitement.⁴⁷ Items measuring IS benefits were identified in Weir et al.’s study⁴⁸ of the impact of POE on nursing practice; high reliability (Cronbach’s alpha > .72) was reported for their questionnaire.

The Nursing Information System Benefits Instrument developed in the current study begins with the general statement “use of information systems in your work” to maintain consistency. This statement is followed by the comprehensive list of potential benefits of IS use. To avoid response bias, that is, the tendency to respond in the same way to all items on a questionnaire, some negatively phrased items were included on the scale;⁴⁹ these items were phrased negatively in the original instrument.⁵⁰ As the purpose of the IS benefits questionnaire is to assess the nurses’ extent of agreement about benefits they experience from using IS, the nurses were asked to respond by using a five-point Likert scale that ranged from strongly disagree (1) to strongly agree (5) to indicate the extent of their agreement with each item.⁵¹ The summative score of the items was used; higher scores indicated greater benefit from using IS in nursing practice.

Methods

Sampling

Random sampling was used to select nurses (N = 570) who were members of the Ohio Nurses Association (ONA). Inclusion criteria were that the nurses had to be hired as staff nurses, working in hospitals at least eight hours per week, spending at least 50 percent of their time providing direct patient care, and using at least one form of IS. The ONA was selected because its database included nurses who are staff nurses and work in hospitals. The majority of the nurses (n = 531; 93.1 percent) were women and were white (n = 525; 92.1 percent), while only 5.4 percent (n = 31) were African American. The average age of the nurses was 49.75 years (SD = 24.92), their years of nursing experience averaged 23.41 years (SD = 9.40), and their years of experience working in a hospital averaged 17.99 years (SD = 9.61). About 40 percent of the nurses had a baccalaureate degree in nursing (n = 229; 40.2 percent), while 26.3 percent (n = 150) had a nursing diploma and 24.6 percent (n = 140) had an associate degree. Most of the nurses worked an average of 34.82 hours per week.

Of the nurses who participated in the study, 40.0 percent worked in critical care units (n = 228), 14.7 percent (n = 83) worked in medical surgical units, and 12.8 percent (n = 73) worked in obstetrics, while 25.8 percent (n = 147) reported that they worked in other inpatient subspecialty units, such as oncology, dialysis, and rehabilitation, or that they floated throughout units, or that they worked in preoperative and ambulatory surgery units.

Procedure

The study was reviewed and approved by the internal review board (IRB) of Case Western Reserve University in Cleveland, Ohio. ONA cooperation approval was obtained for the IRB. The ONA was contacted for a list of names and mailing addresses of staff nurses who work in hospitals. A mail survey guided by Dillman’s Tailored Design Method (TDM)⁵² was used. The study questionnaire was mailed to nurses with a cover letter and a stamped return envelope. A reminder card was mailed a week after the questionnaire to ask nurses who had not yet completed the questionnaire to please do so. For nurses who responded, a thank-you card with the promised incentive ($5) was sent to them. Of the 1,379 surveys that were mailed, 570 were returned, resulting in a response rate of 41.33 percent.

Preliminary Analysis

The total sample of 570 surveys obtained from nurses was randomly split into two groups using SPSS 15.0 so that the instrument could be developed on the first half (n = 285) and tested on the second half (n = 285). Preliminary analyses were conducted to ensure that the two groups are similar in age (t(1, 564) = –0.12, p = .90), gender (X²(1, 567) = .49, p = .48), race (X²(5, 566) = 5.62, p = .35), highest level of education (X²(4, 566) = 6.23, p = .18), years of experience in nursing (t(1, 563) = .34, p = .74), years of experience working in the hospital (t(1, 556) = .27, p = .79), hours worked per week (t(1, 558) = .78, p = .44), and computer experience at work (t(1, 567) = .96, p = .34).

Reliability and Validity Assessments

The reliability of the IS benefits instrument was determined by Cronbach’s alpha coefficient and item analysis. Reliability is the proportion of variance in the scale scores that is attributable to the true score.⁵³ Internal consistency reliability, measured by Cronbach’s alpha coefficient, reflects the homogeneity of the items within a scale.⁵⁴ The first step was to examine the internal consistency in the randomly selected first half of the sample. Corrected item-to-total scale correlations, in which an item is correlated with the total scale score excluding itself, were examined for each item. Items with corrected item-to-total scale correlations that were below .40 were removed.⁵⁵ Next, the Cronbach’s alpha if the item was deleted was examined for each item.

Exploratory factor analysis (EFA) using the principal components method of extraction and varimax rotation was performed on the 56 items constituting the IS benefits instrument for the randomly selected first half of the sample (n = 285). A level of .32 was set as the criterion for determining whether an item loaded sufficiently on a factor.⁵⁶ Items not meeting that criterion were deleted from the scale. Reliability analysis was repeated to examine the internal consistency of the remaining items. Then, a confirmatory factor analysis was performed for the randomly selected second half of the data (n = 285).

Results

Reliability and Item Analysis

A summary of the item analysis of the 56 items of the IS benefits instrument revealed an acceptable Cronbach’s alpha of .97, which exceeded the recommended criterion level of .70 for new instruments.^57,58 Initially, when examining the corrected item-to-total scale correlations, the items with correlations below .30 were removed, resulting in the removal of four items, all of which were negatively phrased (item XX, “does not facilitate using information which influences patient care”; item XX, “makes patient outcomes worse because nurses rely on standards rather than their own judgments”; item XX, “increases costs overall”; and item XX, “increases nurses’ responsibility”). For these, alphas if the item was deleted were less than the alphas on all other items (Table 1).

Table 1: Reliability of the IS Benefits Instrument

Reliability analysis was repeated for the remaining 52 items of the IS benefits instrument, and one additional item (item XX, “decreases nursing professional status because nurses focus on plans rather than care”) fell below the .40 criterion; it was deleted. The reliability analysis was repeated again to find that the remaining 51 items were internally consistent; however, one more item, which was the only remaining negatively phrased item (“waste time overall”), was also deleted at that time. The reliability analysis was repeated with the remaining 50 items; one more item (“facilitates obtaining client records from other healthcare agencies”) was found to have a corrected item-to-total scale correlation below .40 and was removed.

Factor Analysis

Exploratory factor analysis using the principal components method of extraction and varimax rotation was performed on the 49 items of the IS benefits instrument on the randomly selected first half of the data (n = 244 valid cases). Exploratory factor analysis is a statistical technique that is used to look for patterns or relationships among items on a measuring instrument.^59,60 The Kaiser-Meyer-Olkin measure of sampling adequacy (.96) confirmed that the data and sample size were adequate for factor analysis.⁶¹ The Kaiser-Meyer-Olkin measure of sampling adequacy is a statistical value that is used as an index for deciding whether or not the sample is sufficient for performing factor analysis.⁶² Bartlett’s test of sphericity is a second measure of sampling adequacy; it tests for the overall significance of all correlations among all items on the measuring instrument.^63,64 Bartlett’s test of sphericity was 10011.13 (p < .001), which supported the hypothesis that all correlations, tested simultaneously, were statistically different from zero. Rejection of the null hypothesis showed that the data were appropriate for factor analysis.⁶⁵ Taken together, these two statistical values (Kaiser-Meyer-Olkin measure and Bartlett’s test of sphericity) provide minimal standards that should be met before conducting a factor analysis.^66,67 These standards were met in the analysis reported here.

In addition, to test for multicollinearity, the value of the determinant for the correlation matrix was computed. A value of 5.60 was obtained. Because this value exceeded .00001, multicollinearity was judged not to be a problem⁶⁸ and the correlation matrix was deemed suitable for factor analysis.⁶⁹ In general, multicollinearity means that two or more variables are very highly correlated (i.e., r = .90 or above). This is not a desirable situation because it means the variables are so similar that they may be redundant in measuring the same construct.^70,71 The eigenvalue is the sum of the squared loadings for each factor that emerges from the factor analysis. Conceptually, this value represents the amount of variance that is accounted for by each respective factor.^72-74 The eigenvalues were greater than 1.00 for the six factors obtained, which explained 66.20 percent of the variance in IS benefits. However, the scree plot suggested a four-factor solution since the eigenvalues showed a linear decline⁷⁵ commencing with the fourth factor.

Factor analysis using the principal components method of extraction and varimax rotation was performed on the 49 items of the IS benefits instrument on the same sample, forcing to a four-factor solution. The Kaiser-Meyer-Olkin measure was .96, Bartlett’s test of sphericity was 10011.13 (p < .001), and the determinant was 5.60; these four factors explained 61.12 percent of the variance in IS benefits. Sixteen items were deleted because they loaded on two factors (secondary factor loading = .2) or had weak loadings (<.32).⁷⁶ One more item was deleted in a subsequent factor analysis of the 33 remaining items of the IS benefits instrument, resulting in a final 32-item scale (Table 2). The Kaiser-Meyer-Olkin measure was .95, Bartlett’s test of sphericity was 6701.11 (p < .001), and the determinant was 6.79; these four factors explained 65.26 percent of the variance in IS benefits.

Table 2: Reliability and Validity (Exploratory Factor Analysis) Estimates for 32 Items of the IS Benefit Instrument (n = 252, Cronbach's alpha =.97)

Factor analysis was done for the final 32 items (see Table 2). The Kaiser-Meyer-Olkin measure was .95, Bartlett’s test of sphericity was 6525.82 (p < .001), and the determinant was 1.47; these four factors explained 65.89 percent of the variance in IS benefits. Reliability analysis was repeated with the remaining 32 items. The Cronbach’s alpha was .97, the scale mean was 104.84 with a variance of 496.56, and the average inter-item correlation was .46.

Items were clustered as proposed theoretically. However, some items loaded on different dimensions, making it necessary to rename the factors. Factor 1 reflected the theme of saving time and efficiency. Factor 2 reflected themes related to quality of and access to patient information, which theoretically had been named quality of care. Factor 3 reflected the theme of communication and documentation; however, all the items that were related to communication were previously deleted during the exploratory factor analysis, so Factor 3 was renamed to reflect charting. Factor 4 reflected professional practice.

Confirmatory Factor Analysis

Using the 32 items obtained from the development of the IS benefits instrument on the randomly selected first half of the sample, a confirmatory factor analysis (Table 3) using the principal components method of extraction and varimax rotation was done on the random selected second half of the sample (n = 254). The Kaiser-Meyer-Olkin measure of sampling adequacy (.95) confirmed that the data and sample size were adequate for this factor analysis. Bartlett’s test of sphericity was 6261.05 (p < .001), and the determinant was 5.50 (more than .00001), which indicated that the correlation matrix was suitable for factoring. The eigenvalues were greater than 1.00 for four factors, which explained 64.06 percent of the variance in IS benefits.

Table 3: Reliability and Validity (Confirmatory Factor Analysis) Estimates for 32 Items of the IS Benefits Instrument (n = 254, Cronbach's alpha =.97)

The confirmatory factor analysis (CFA) revealed similar results as the exploratory factor analysis (EFA) except that one item (“reduces administrative tasks”), which loaded on Factor 1 in the EFA, now loaded on Factor 3 in the CFA. Items were loaded strongly; however, some items were double loaded across factors.

Discussion

This study developed and tested a measure of IS benefits among nurses. The IS benefits instrument was developed to provide nurse-specific measures about IS in general. Because today’s healthcare environment increasingly relies on IS, it is important to identify how nurses perceive benefits related to their use of these systems.

The IS benefits instrument performed well in this initial psychometric testing, and the reliability results indicated that the benefits scale’s internal consistency, reflected by Cronbach’s alpha, was high. Internal consistency is important for IS benefits because it is based on the correlations between different items on the same instrument. It measures whether several items that propose to measure the same general construct are doing so. Internal consistency is critical due to the multiple meanings assigned to typical IS benefits. Indeed, literature on IS benefits is frequently interchanged with literature on user satisfaction. Interpretation of what is a benefit of IS ranges from actual, objective parameters (for example, save time) to subjective parameters (for example, the system is easy to use). In any study of IS benefits, it is necessary to clearly define and objectively measure the factors. Internal consistency assures the objectivity of the analysis.

Items that were deleted had a corrected item-to-total correlation less than .40. A low correlation means the item is not really measuring the same construct that other scale items are measuring. One item from each theoretical subscale was eliminated. The content of these items was theoretically proposed to measure quality of care (“does not facilitate using information which influences patient care”), communication and documentation (“makes patient outcomes worse because nurses rely on standards rather than their own judgments” and “facilitates obtaining client records from other healthcare agencies”), saving time (“increases costs overall” and “wastes time overall”), and professional practice (“decreases nursing professional status because nurses focus on plans rather than care” and “increases nurses’ responsibility”). Except for one item, these items were negatively worded and recoded. One possible explanation for the low corrected item-to-total correlation is that it is possible that nurses may have answered the items incorrectly due to their misunderstanding the negative wording. Carmines and Zeller⁷⁷ and Hinkin⁷⁸ reported that factor analysis is frequently complicated by negatively worded items.

It was expected that the confirmatory factor analysis (CFA) would replicate the exploratory factor analysis (EFA). However, except for one item (“reduces administrative tasks”) that loaded on the factor of saving time in the EFA, the items were loaded strongly on the same factors. Another issue noticed in the CFA was that there were items with double loadings on factors in the CFA (Table 3), especially 6 of the 15 items for Factor 1 (saving time and efficiency) and 2 of the 6 items for Factor 3 (charting). However, the primary loadings were consistent with the EFA. Factor 2 (quality of care) and Factor 4 (professional practice) had strong, clean item loadings. Differences between the EFA and the CFA could have resulted from ambiguity within the items. Differences in interpretation of items by multiple, diverse users may have led to item cross-loadings. Individual perceptions of item content may have resulted in different interpretations.

Limitations of the study include that the IS benefits instrument is a self-report measurement, which may induce bias when study participants were asked to describe the perceived benefits. In addition, the current study used a descriptive cross-sectional design. This design limits the understanding of change over time related to perception of benefits, which may change according to the stage of system implementation and how long nurses use the system. Another limitation was that there is no theoretically related, existing instrument that could be used to examine convergent validity. Validation of this measure with other outcomes measures, including patient or hospital outcomes, is recommended. This study involved multiple IS systems for nurses (for the study of IS benefits in general); however, the instruments from which items were drawn focused on a single specific system and some nonnurse users.

Conclusion

The findings provide acceptable estimates for the initial reliability and validity of the measure and indicate that it can be potentially useful for determining IS benefits. Replicating this psychometric testing among nurses representing other specialty areas and performing separate analyses for full-time versus part-time nurses as well as across different specialty areas and across geographic regions are recommended. In addition, obtaining more specific information about past education and experience with IS would be informative.

Information systems are being progressively integrated more and more into nursing work. Nursing care work areas such as nursing documentation, nursing care planning, reporting, and decision making could be improved by using IS. Researchers in nursing informatics posit that using IS in nursing practice is important for improving clinical practice and the quality of patient care. A measure of IS benefits that is clinically relevant and useful for qualitative evaluation of nursing perceptions about the effectiveness of hospital IS is essential.

Acknowledgments

This study was funded by the American Nurses Foundation and Sigma Theta Tau International.

Authors’ Note

The survey may be obtained by contacting the authors.

Amany A. Abdrbo, BSN, MSN, PhD, is a lecturer in the Nursing Administration Department at Cairo University in Cairo, Egypt and Assistant Professor, College of Nursing—Al Ahsa, King Saud Bin Abdul Aziz University for Health Sciences, SA.

Jaclene A. Zauszniewski, PhD, RN-BC, FAAN, is the Kate Hanna Harvey Professor in Community Health Nursing, associate dean for doctoral education, and PhD program director in the Frances Payne Bolton School of Nursing at Case Western Reserve University in Cleveland, OH.

Christine A. Hudak, RN, PhD, CPHIMS, is an associate professor in the Frances Payne Bolton School of Nursing at Case Western Reserve University in Cleveland, OH.

Mary K. Anthony, PhD, RN, CS, is a professor and associate dean for research in the College of Nursing at Kent State University in Kent, OH.

Notes

¹ Dennis, K., P. Sweeney, L. Macdonald, and N. Morse. “Point of Care Technology: Impact on People and Paperwork.” Nursing Economic$ 11, no. 4 (1993): 229–37, 248.

² Simpson, G., and M. Kenrick. “Nurses’ Attitudes Toward Computerization in Clinical Practice in a British General Hospital.” Computers in Nursing 15, no. 1 (1997): 37–42.

³ Ngin, P. M. “Organizational Analysis of Computer User Acceptance Among Nurses.” Doctoral dissertation, University of Michigan, 1993.

⁴ Pierpont, G. L., and D. Thilgen. “Effect of Computerized Charting on Nursing Activity in Intensive Care.” Critical Care Medicine 23, no. 6 (1995): 1067–73.

⁵ Walker, K., and C. Prophet. “Nursing Documentation in the Computer-based Patient Record.” Studies in Health Technology and Informatics 46 (1997): 313–17.

⁶ Weir, C., V. Johnsen, D. Roscoe, and A. Cribbs. “The Impact of Physician Order Entry on Nursing Roles.” Proceedings of the AMIA Annual Fall Symposium 1996: 714–17.

⁷ Weiner, M., T. Gress, D. Thiemann, et al. “Contrasting Views of Physicians and Nurses about an Inpatient Computer-based Provider Order-Entry System.” Journal of the American Medical Informatics Association 6, no. 3 (1999): 234–44.

⁸ Darr, A., M. Harrison, L. Shakked, and N. Shalom. “Physicians’ and Nurses’ Reactions to Electronic Medical Records: Managerial and Occupational Implications.” Journal of Health, Organisation and Management 17, no. 5 (2003): 349–59.

⁹ Nauright, L., and R. Simpson. “Benefits of Hospital Information Systems as Seen by Front-Line Nurses and General Hospital Staff.” Journal of Nursing Administration 24, no. 4 (suppl.) (1994): 26–32.

¹⁰ Ngin, P. M. “Organizational Analysis of Computer User Acceptance Among Nurses.”

¹¹ Nauright, L., and R. Simpson. “Benefits of Hospital Information Systems as Seen by Front-Line Nurses and General Hospital Staff.”

¹² Ibid.

¹³ Dennis, K., P. Sweeney, L. Macdonald, and N. Morse. “Point of Care Technology: Impact on People and Paperwork.”

¹⁴ Ngin, P. M. “Organizational Analysis of Computer User Acceptance Among Nurses.”

¹⁵ Weir, C., V. Johnsen, D. Roscoe, and A. Cribbs. “The Impact of Physician Order Entry on Nursing Roles.”

¹⁶ Darr, A., M. Harrison, L. Shakked, and N. Shalom. “Physicians’ and Nurses’ Reactions to Electronic Medical Records: Managerial and Occupational Implications.”

¹⁷ Nauright, L., and R. Simpson. “Benefits of Hospital Information Systems as Seen by Front-Line Nurses and General Hospital Staff.”

¹⁸ Axford, R., and B. Carter. “Impact of Clinical Information Systems on Nursing Practice: Nurses’ Perspectives.” Computers in Nursing 14, no. 3 (1996): 156–63.

¹⁹ Dennis, K., P. Sweeney, L. Macdonald, and N. Morse. “Point of Care Technology: Impact on People and Paperwork.”

²⁰ Walker, K., and C. Prophet. “Nursing Documentation in the Computer-based Patient Record.”

²¹ Chin, L., and D. Haughton. “A Study of Users’ Perceptions on the Benefits of Information Systems in Hospitals.” Healthcare Information Management 9, no. 1 (1995): 37–46.

²² Ruland, C. M., and I. H. Ravn. “Usefulness and Effects on Costs and Staff Management of a Nursing Resource Management Information System.” Journal of Nursing Management 11 (2003): 208–15.

²³ United States General Accounting Office. Information Technology: Benefits Realized for Selected Health Care Functions (Publication No. GAO-04-224). October 2003. Available at http://www.gao.gov/new.items/d04224.pdf. Accessed September 1, 2004.

²⁴ Dennis, K., P. Sweeney, L. Macdonald, and N. Morse. “Point of Care Technology: Impact on People and Paperwork.”

²⁵ Ibid.

²⁶ Ibid.

²⁷ Weir, C., V. Johnsen, D. Roscoe, and A. Cribbs. “The Impact of Physician Order Entry on Nursing Roles.”

²⁸ Nauright, L., and R. Simpson. “Benefits of Hospital Information Systems as Seen by Front-Line Nurses and General Hospital Staff.”

²⁹ United States General Accounting Office. Information Technology: Benefits Realized for Selected Health Care Functions.

³⁰ Dumas, J. A., E. O. Dietz, and P. M. Connolly. “Nurse Practitioner Use of Computer Technologies in Practice.” Computers in Nursing 19, no. 1 (2001): 34–40.

³¹ Pierpont, G. L., and D. Thilgen. “Effect of Computerized Charting on Nursing Activity in Intensive Care.”

³² Walker, K., and C. Prophet. “Nursing Documentation in the Computer-based Patient Record.”

³³ Axford, R., and B. Carter. “Impact of Clinical Information Systems on Nursing Practice: Nurses’ Perspectives.”

³⁴ Chin, L., and D. Haughton. “A Study of Users’ Perceptions on the Benefits of Information Systems in Hospitals.”

³⁵ Ruland, C. M., and I. H. Ravn. “Usefulness and Effects on Costs and Staff Management of a Nursing Resource Management Information System.”

³⁶ Nauright, L., and R. Simpson. “Front-Line Hospital Staff from Six Hospitals Rate Benefits of Hospital Information Systems.” Journal of the American Medical Informatics Association 8, pt. 1 (1995): 594–97.

³⁷ Welychka, P. “Monitoring and Evaluating Benefits of an Automated Patient Care.” Studies in Health and Technology Informatics 46 (1997): 343–47.

³⁸ Chin, L., and D. Haughton. “A Study of Users’ Perceptions on the Benefits of Information Systems in Hospitals.”

³⁹ Ibid.

⁴⁰ Welychka, P. “Monitoring and Evaluating Benefits of an Automated Patient Care.”

⁴¹ Axford, R., and B. Carter. “Impact of Clinical Information Systems on Nursing Practice: Nurses’ Perspectives.”

⁴² Dennis, K., P. Sweeney, L. Macdonald, and N. Morse. “Point of Care Technology: Impact on People and Paperwork.”

⁴³ Axford, R., and B. Carter. “Impact of Clinical Information Systems on Nursing Practice: Nurses’ Perspectives.”

⁴⁴ Chin, L., and D. Haughton. “A Study of Users’ Perceptions on the Benefits of Information Systems in Hospitals.”

⁴⁵ Ruland, C. M., and I. H. Ravn. “Usefulness and Effects on Costs and Staff Management of a Nursing Resource Management Information System.”

⁴⁶ United States General Accounting Office. Information Technology: Benefits Realized for Selected Health Care Functions.

⁴⁷ Weir, C., V. Johnsen, D. Roscoe, and A. Cribbs. “The Impact of Physician Order Entry on Nursing Roles.”

⁴⁸ Ibid.

⁴⁹ DeVellis, R. F. Scale Development Theory and Applications. Vol. 26. 2nd ed. Thousand Oaks, CA: Sage, 2003.

⁵⁰ Axford, R., and B. Carter. “Impact of Clinical Information Systems on Nursing Practice: Nurses’ Perspectives.”

⁵¹ DeVellis, R. F. Scale Development Theory and Applications.

⁵² Dillman, D. Mail and Internet Surveys: The Tailored Design Method. 2nd ed. New York: Wiley, 2000.

⁵³ DeVellis, R. F. Scale Development Theory and Applications.

⁵⁴ Ibid.

⁵⁵ Nunnally, J. C., and I. H. Bernstein. Psychometric Theory. 3rd ed. New York: McGraw-Hill, 1994.

⁵⁶ Tabachnick, B. G., and L. S. Fidell. Using Multivariate Statistics. 4th ed. Boston: Allyn and Bacon, 2001.

⁵⁷ Nunnally, J. C., and I. H. Bernstein. Psychometric Theory.

⁵⁸ Knapp, T., and J. Brown. “Ten Measurements That Often Should Be Taken.” Research in Nursing & Health 18 (1995): 465–69.

⁵⁹ DeVellis, R. F. Scale Development Theory and Applications.

⁶⁰ Tabachnick, B. G., and L. S. Fidell. Using Multivariate Statistics. 5th ed. Boston: Allyn and Bacon, 2007.

⁶¹ Kaiser, H. F. “An Index of Factorial Simplicity.” Psychometrika 39, no. 1 (1974): 31–36.

⁶² Tabachnick, B. G., and L. S. Fidell. Using Multivariate Statistics. 5th ed.

⁶³ DeVellis, R. F. Scale Development Theory and Applications.

⁶⁴ Tabachnick, B. G., and L. S. Fidell. Using Multivariate Statistics. 5th ed.

⁶⁵ Tabachnick, B. G., and L. S. Fidell. Using Multivariate Statistics. 4th ed.

⁶⁶ DeVellis, R. F. Scale Development Theory and Applications.

⁶⁷ Tabachnick, B. G., and L. S. Fidell. Using Multivariate Statistics. 5th ed.

⁶⁸ Field, A. Discovering Statistics Using SPSS for Windows: Advanced Techniques for the Beginner. Thousand Oaks, CA: Sage, 2000.

⁶⁹ Morgan, G., and O. Griego. Easy Use and Interpretation of SPSS for Windows: Answering Research Questions with Statistics. Mahwah, NJ: Lawrence Erlbaum Associates, 1998.

⁷⁰ Tabachnick, B. G., and L. S. Fidell. Using Multivariate Statistics. 5th ed.

⁷¹ Mertler, C. A., and R. A. Vannatta. Advanced and Multivariate Statistical Methods. 4th ed. Glendale, CA: Pyrczak, 2010.

⁷² DeVellis, R. F. Scale Development Theory and Applications.

⁷³ Tabachnick, B. G., and L. S. Fidell. Using Multivariate Statistics. 5th ed.

⁷⁴ Mertler, C. A., and R. A. Vannatta. Advanced and Multivariate Statistical Methods.

⁷⁵ Brown, T. Confirmatory Factor Analysis for Applied Research. New York: Guilford, 2006.

⁷⁶ Tabachnick, B. G., and L. S. Fidell. Using Multivariate Statistics. 4th ed.

⁷⁷ Carmines, E., and R. Zeller. Reliability and Validity Assessment. Beverly Hills, CA: Sage, 1979.

⁷⁸ Hinkin, T. “A Review of Scale Development Practices in the Study of Organizations.” Journal of Management 21, no. 5 (1995): 967.