ATS
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
QUICK SEARCH:   [advanced]


     

This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to Personal Folders
Right arrow Download to citation manager
Right arrow Author home page(s):
Syma L. Prince
Mitchell J. Magee
Michael J. Mack
Right arrow Permission Requests
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Herbert, M. A.
Right arrow Articles by Mack, M. J.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Herbert, M. A.
Right arrow Articles by Mack, M. J.
Related Collections
Right arrow Professional affairs

Ann Thorac Surg 2004;77:1960-1964
© 2004 The Society of Thoracic Surgeons

Original article: cardiovascular

Are unaudited records from an outcomes registry database accurate?

Morley A. Herbert, PhDa*, Syma L. Prince, RN, BSNb, Janet L. Williams, BAb, Mitchell J. Magee, MDb, Michael J. Mack, MDb

a Department of Research, Medical City Dallas Hospital, USA
b Cardiopulmonary Research Science and Technology Institute, Dallas, Texas, USA

Accepted for publication December 12, 2003.

* Address reprint requests to Dr Herbert, Department of Research, Medical City Dallas Hospital, 7777 Forest Ln, Suite C-740, Dallas, TX 75230, USA
e-mail: morley.herbert{at}lonestarhealth.com


   Abstract
Top
 Abstract
Introduction
 Material and methods
 Results
 Comment
 References
 
BACKGROUND: Data from outcomes registry databases are being increasingly used for peer review and public reporting. However, administrative and clinical databases are mostly unaudited; thus, their accuracy has not been verified.

METHODS: Outcomes data from all coronary artery bypass operations from a single cardiac surgery practice were entered into The Society of Thoracic Surgeons (STS) National Cardiac Database. From our practice of 18 surgeons, we audited 247 (10%) of the clinical records of patients undergoing surgery in 2001 and correlated them with all 315 elements of the STS National Cardiac Database for verification of accuracy. Inaccuracies were defined as a disagreement with a nominal or categorical variable or, for continuous variables, as the value not being within a predetermined window. When discrepancies existed, the hospital clinical record was assumed to be accurate. Outcomes discrepancies were then analyzed by four major categories: components of the preoperative risk algorithm, operative mortality, major complications, and other outcomes.

RESULTS: Discrepancies were noted in 5% (16) or fewer of the audited fields for 98.8% of the records. Of the 32 variables in the mortality risk algorithms, discrepancies were present in fewer than 10% of the audits on 30 of the 32 variables. More than 95% of the audited charts had zero or one discrepancy in the seven most important variables in the mortality risk models. Operative mortality was determined to be completely accurate with no discrepancies between the database and the audited clinical record. Among major complications, the error rate was less than 1% for all complications except prolonged ventilation (4.0%). A higher rate of discrepancies did exist in some of the other variables, including discharge medications (14.1%) and ventilator time (36.4%).

CONCLUSIONS: A detailed audit of a clinical outcomes registry database demonstrated that the major fields within this specific database including operative mortality, major complications, and the significant factors in the risk algorithm were highly accurate. Process improvement factors were identified to further increase the accuracy of data collection.


   Introduction
Top
 Abstract
 Introduction
Material and methods
 Results
 Comment
 References
 
Outcomes reporting of medical procedures are being increasingly used for peer review, contracting purposes, determination of "centers of excellence," and public reporting [1–5]. Outcomes for various procedures are reported from a variety of clinical and administrative databases. Although determinations of quality are made on the basis of reported data, the accuracy of these data has not been clearly substantiated [6, 7]. As a participant in The Society of Thoracic Surgeons (STS) National Cardiac Database, the Cardiopulmonary Research Science and Technology Institute (CRSTI) collects data locally and then "harvests" data to the central data warehouse at defined intervals.

To determine the accuracy of data gathered into our clinical database, each year we randomly audit 10% of the clinical records of patients whose procedural and outcomes data had been entered into our database during a 1-year period for verification of accuracy. This report covers the results of the 2001 audit.


   Material and methods
Top
 Abstract
 Introduction
 Material and methods
Results
 Comment
 References
 
Data source
Using specialized data collection forms, surgical data were collected on all patients from a single, multicenter cardiac surgical practice; collected data were stored locally in a STS-approved database (Armus, Burlingame, CA). Since 1985, the practice has consisted of 22 cardiac surgeons practicing in 18 hospitals and the database now contains more than 33,000 patient entries.

The data were gathered by an independent not-for-profit research organization, CRSTI, which supervises the data collection and provides data entry and local warehousing. The collected data were exported to the STS for participation in the national registry and also analyzed locally. Prepared reports were used for peer review within the surgical practice. Outcomes data for individual surgeons were provided to that surgeon along with benchmarking within the practice, within the hospital in which that surgeon practiced, within the region, and compared with national standards. Data were also provided to the hospitals for performance and quality improvement initiatives.

Data collection
Data were gathered prospectively for each patient at the various stages of the clinical course. The preoperative portion of the database was collected at the time of the surgical procedure on a paper entry form by a clinical person, either a nurse practitioner, physician assistant, or perfusionist familiar with clinical data. After completion of the surgical procedure, operative data were then usually completed by the same staff member after transfer of the patient to the intensive care unit. The data elements regarding the postoperative clinical course were then generally completed by a physician assistant, clinical nurse specialist, or nurse practitioner at the time of discharge from the hospital. The individual patient data collection was sent to the clinician's office and the rest of the postoperative data was completed at the 30-day follow-up visit of the patient. Data collection and entry was under supervision of the operating physician, who generally reviewed the data record for accuracy and completeness before the record was entered into the local database. If the patient had a prolonged operative stay or did not return for follow-up, retrospective collection of missing data elements was performed.

Once the record was presented for electronic data entry, the data were subjected to a review process during which missing elements or data inconsistencies were identified. These forms were then returned to the clinicians for review, completion, or correction. After the completed forms were entered into the computerized database, weekly audits of a minimum of 10 records helped ensure integrity of the data entry process. This procedure enabled correction of data entry errors and identified procedural problems.

Database audit
On a semiannual basis an administrative person not involved with data collection or entry randomly selected 10% of the data records for audit from a list of entries generated from the previous 6 months. An outcomes database supervisory person with a background in clinical medicine then correlated the computerized data with the patient's clinical record for accuracy, using the STS National Cardiac Database definitions of each of the 315 data elements. Determination of whether a discrepancy exists depended on the variables being checked. All nominal and categorical variables (eg, yes/no) were scored as a discrepancy if there was any disagreement. For continuous variables, a definition of discrepancy was determined before the audit process was initiated. A "reasonable" window of acceptable range was determined in consultation with clinicians to determine discrepancies.

The data set included 102 variables for demographic and preoperative risk data, 113 variables containing operative procedural data, and 100 variables for postoperative data, complications, and discharge information. All discrepancies were noted and any pertinent clinical data recorded on an audit form.

The 10% sample yielded 247 data records for audit from the practices of 18 of the surgeons. The results of the audit were then divided into four major categories: preoperative risk algorithm variables, operative mortality, major postoperative complications, and other selected outcome variables.


   Results
Top
 Abstract
 Introduction
 Material and methods
 Results
Comment
 References
 
Complete correlation of the clinical record and the database occurred for 5 (2.0%) of the patients. An additional 190 records (82.2%) had one to ten discrepancies, and 39 (16.8%) had more than ten fields of disagreement. Overall, discrepancies occurred in fewer than 16 (5%) of the data fields in 98.8% of all charts audited. No discrepancy in the data element was noted for operative mortality, with complete correlation between the database variable and the audited clinical records.

Discrepancies were found in the entries for the preoperative risk variables used in the operative mortality risk algorithm. Table 1 lists the 34 different variables used in the calculation of risk of operative mortality for the coronary artery bypass grafting, valve, and coronary artery bypass grafting + valve models, and the overall discrepancy rates measured for each variable. Procedure status and New York Heart Association class had the largest discrepancy rates (13.4% and 28.3%, respectively). Disagreement occurred in more than 5% of the records for height, weight, diabetes control method, presence of chronic obstructive pulmonary disease, timing of preoperative myocardial infarction, ejection fraction, number of diseased vessels, presence of left main disease, and mitral insufficiency.


View this table:
[in this window]
[in a new window]
  		Table 1. Operative Mortality Risk Algorithm Discrepancy Rates

 
Measuring the distribution of the discrepancies in the risk algorithm variables, 43 (17.4%) of the audited charts had no discrepancies in these variables, whereas 149 (60.3%) had only one or two discrepancies. The remaining 55 charts had as many as seven discrepancies in these variables.

Because variables appearing in a risk algorithm are not weighted equally, it would be helpful to know how many of these discrepancies occurred among the more important variables. Examining variables in the coronary artery bypass grafting risk model for operative mortality, we used the seven variables with the highest weighting and largest odds ratio. These included age, previous surgery, cardiogenic shock, immunosuppressive therapy, diabetes, renal failure, and operative status [8]. One hundred seventy-two (69.6%) of the charts had no discrepancies in these variables, 64 (25.9%) had one discrepancy, and 11 (4.5%) had two or three discrepancies each. Of the total of 434 discrepancies found among all the risk variables of the algorithm, only 97 (22.4%) occurred in the variables with the highest weighting or largest odds ratio. More than 95% of the charts had either zero or one discrepancy in the most important variables used in mortality risk modeling.

Table 2 lists the discrepancy rates measured for six major complications. Operative mortality and renal failure requiring dialysis was accurately reported on all patients, whereas minor discrepancies were noted for permanent stroke (0.4%), deep sternal infection (0.8%), and cardiac surgery reoperation (0.8%). Prolonged ventilation was recorded the most inaccurately, with discrepancies occurring in 4% of the records.


View this table:
[in this window]
[in a new window]
  		Table 2. Discrepancy Rates for Major Complications

 
The discrepancy rate for other selected outcome variables (Table 3) showed much higher rates, ranging from a low value of 0.4% for leg infection to a high of 36.4% in reporting of the number of postoperative ventilation hours.


View this table:
[in this window]
[in a new window]
  		Table 3. Discrepancy Rates for Other Selected Outcome Variables

 
Depending on the hospital and practice, the data collection forms were completed by surgeons, nurse practitioners, nurses, physician assistants, perfusionists, or clinical nurse specialists. At seven hospitals, we analyzed the data based on the recorded average number of discrepancies per data record with values ranging from a low of 3.7 to a high of nearly 10. These results were then correlated with the method of data collection defined by the combination of individuals involved (Fig 1).



View larger version (36K):
[in this window]
[in a new window]
  		Fig 1. Modes of data collection used. (NP = nurse practitioner; PA = physician's assistant; RN = registered nurse.)

 
The modes of collection shown were physician's assistant and RN (at one hospital), RN alone (at one hospital), nurse practitioner and clinical nurse specialist (two hospitals), and perfusionist with nurse practitioner and physician's assistant (three hospitals). The graph shows no correlation between the professional training of the team members doing the data collection and the reported number of discrepancies per data record.


   Comment
Top
 Abstract
 Introduction
 Material and methods
 Results
 Comment
References
 
The use of outcomes data as a measure of the quality of medical care is becoming increasingly important. Quality outcomes data are used for internal peer review within institutions, determination of quality compared with regional and national benchmarks, and for determination of centers of excellence—both by payers and government agencies. Yet public "report cards" on hospitals and surgeons and continuous quality improvement measures are generally based on unaudited data and outcomes [9, 10]. Frequently, outcomes reporting is based on administrative databases with an unproven correlation with actual clinical outcomes [11]. Moreover, the accuracy of clinical outcomes databases has generally not been verified. The accuracy of outcomes reporting and calculations of risk adjustment applied to the raw data are only as accurate as the data entered. Most data collection and entry are performed by individuals with varying degrees of clinical expertise and access to data sources. Yet auditing of these data for accuracy is seldom performed.

The STS National Cardiac Database is a voluntary, unaudited outcomes registry of cardiac surgical procedures performed within the United States. Since 1993, data on 2 million operative procedures performed at more than 600 sites have been entered, analyzed, and reported [12]. The data warehouse has performed ongoing internal data checks to assure data consistency and general accuracy of outcomes reporting [6, 7]. The national data warehouse cannot, however, audit the specific clinical records within the database, instead relying on the reporting centers for data accuracy.

As a whole, we found generally excellent correlation between the data our center reported to the STS National Cardiac Database and that found in the patient's clinical record. No discrepancy was noted in operative mortality, the most frequently cited outcomes measurement of "quality." Data however, are frequently reported as "risk adjusted" in order to account for varying patient preoperative risk factors. The audit of the variables contained in the risk algorithm showed generally very good correlation. The risk algorithm calculation, however, is proprietary and all variables do not count equally. Variables such as New York Heart Association classification, which showed a high discrepancy rate, will have a lesser effect on the risk calculation. However, the discrepancy rate of procedural status will have a more significant impact on the accuracy of the calculation of risk algorithm.

In a similar manner, correlation of reporting of major complications was generally good. This finding would be another endorsement of the accuracy of registry outcomes reporting, at least with the methodology used in our practice. Discrepancy rates from other selected outcomes, however, showed more variability when audited.

Table 4 summarizes our approach to maximize the accuracy of the data collection process. As a result of this audit and as part of a process improvement initiative, we have broken down the factors in the data collection process as follows: data definitions, data sources, data entry personnel, background and training, and a commitment to completeness and accuracy of data collection. In our experience, a lack of clear understanding of the definition of some variables is a problem. The variables for which the definitions were most often misunderstood included type of angina and procedural status. Specific remedies to this issue may include adding the STS definitions to the data collection forms. (See the Appendix for an example of our form including definitions.)


View this table:
[in this window]
[in a new window]
  		Table 4. Steps to Improve Accuracy of Data Collection Process

 
Another aspect of our process improvement initiative is ongoing education about the origin of common discrepancies. As part of this initiative, personnel are alerted to areas in which the accuracy of the data has been problematic. As shown previously in Figure 1, the educational background of the staff collecting the data has no effect on the discrepancy rate. Nevertheless, for institutions and practices in which physician leadership and advocacy stress the importance of the data, both the accuracy and promptness with which the data are collected has improved. Having a physician champion as an advocate clearly improves outcomes reporting.

Another common cause for error is an inability to access the data source for determining accurate information. With same-day admissions for cardiac surgery and transfer of patients from other institutions, medical records are frequently incomplete during data collection. The greater ubiquity of electronic medical records should probably improve this situation. In the interim, however, extra effort must be expended by the staff member responsible for data collection to determine the entry information from the source data. We have entertained implementing financial incentives for data accuracy and promptness of submission to enhance the accuracy of data collection. To date, however, we have not initiated this process.

To assist in providing the data collection personnel with complete data, surgeons are being provided with "cheat sheets" containing a list of the risk variables. Using these sheets as guides, the expectation is that the surgeons' dictations will address all the necessary elements, thereby facilitating accurate data collection.

Accuracy of data entry is also enhanced by data monitoring. Training individuals involved with data collection to increase awareness of potential areas for inaccuracy and early rectification of the high areas of inaccuracy has been helpful. Although we have determined by regular auditing that data integrity is generally good, we have identified a number of areas for process improvement. Continued monitoring of the data and of the processes involved in data collection and accuracy should improve the integrity of the data. With outcomes reporting being used increasingly as a determination of quality, it behooves every individual involved in the process to strive for excellence in data collection promptness and accuracy.


   Appendix
 
Preoperative Risk Factors
Weight (kg) Height (cm)

Circle All That Apply History of Smoking: No Yes

Family History of Coronary Artery Disease (defined as any direct blood relatives with angina, myocardial infarction, or sudden cardiac death before the age of 55): No Yes Diabetes (defined as a history of diabetes, regardless of duration or need for medication): No Yes

Hypercholesterolemia (diagnosed and or treated by a physician; total cholesterol more than 200 mg/dL; low-density lipoprotein level of 130 mg/dL or higher; high-density lipoprotein level of less than 30 mg/dL): No Yes

Renal Failure (defined as a creatinine level more than 2.0 mg/dL): No Yes

• Dialysis: No Yes

• Last Preoperative Creatinine Level:

Hypertension (diagnosed with or treated for hypertension with medication, diet, or exercise; blood pressure more than 140 mm Hg systolic or more than 90 mm Hg diastolic on at least two occasions): No Yes

Cerebrovascular Accident (defined as central neurologic deficit lasting more than 24 hours): No Yes

When:

Infectious Endocarditis: No Yes

Type:

Chronic Lung Disease:


   References
Top
 Abstract
 Introduction
 Material and methods
 Results
 Comment
 References
 

  1. Birkmeyer J.D., Stukel T.A., Siewers A.E., Goodney P.P., Wennberg D.E., Lucas F.L. Surgeon volume and operative mortality in the United States. N Engl J Med 2003;349:2117-2127.[Abstract/Free Full Text]
  2. Kizer K.W. The volume-outcome conundrum. N Engl J Med 2003;349:2159-2161.[Free Full Text]
  3. New York State Department of Health. Coronary artery bypass surgery in New York State, 1997–1999. 2002 September. Available from: URL: http://www.health.state.ny.us/nysdoh/heart/1997-99cabg.pdf. Accessed January 30, 2004
  4. Texas Healthcare Information Council. Available from: URL: http://www.thcic.state.tx.us/default.htm. Accessed January 30, 2004
  5. HealthGrades, the Healthcare Quality Experts. Available from: URL: http://www.healthgrades.com/. Accessed January 30, 2004
  6. Grover F.L., Shroyer A.L., Edwards F.H., et al. Data quality review program: the Society of Thoracic Surgeons Adult Cardiac National Database. Ann Thorac Surg 1996;62:1229-1231.[Free Full Text]
  7. Shroyer A.L., Edwards F.H., Grover F.L. Updates to the Data Quality Review Program: the Society of Thoracic Surgeons Adult Cardiac National Database. Ann Thorac Surg 1998;65:1494-1497.[Abstract/Free Full Text]
  8. Shroyer A.L., Coombs L.P., Peterson E.D., et al. The Society of Thoracic Surgeons: 30-day operative mortality and morbidity risk models. Ann Thorac Surg 2003;75:1856-1865.[Abstract/Free Full Text]
  9. Shahian D.M., Normand S.L., Torchiana D.F., et al. Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg 2001;72:2155-2168.[Abstract/Free Full Text]
  10. Grunkemeier G.L., Zerr K.J., Jin R. Cardiac surgery report cards: making the grade. Ann Thorac Surg 2001;72:1845-1848.[Free Full Text]
  11. Gardner E. UB-82 forms offer wealth of information, misinformation. Mod Healthc 1990;20(38):18–9, 22–9
  12. Society of Thoracic Surgeons National Cardiac Database Available from: http://www.ctsnet.org/file/STSNationalDatabaseFall2003ExecutiveSummary_Adult_Revised.pdf. Accessed April 21, 2004.



This article has been cited by other articles:


Home page
 CirculationHome page
S. M. O'Brien, E. R. DeLong, R. S. Dokholyan, F. H. Edwards, and E. D. Peterson
Exploring the Behavior of Hospital Composite Performance Measures: An Example From Coronary Artery Bypass Surgery
Circulation, December 18, 2007; 116(25): 2969 - 2975.
[Abstract] [Full Text] [PDF]

Home page
 Ann. Thorac. Surg.Home page
K. F. Welke, E. D. Peterson, M. S. Vaughan-Sarrazin, S. M. O'Brien, G. E. Rosenthal, G. J. Shook, R. S. Dokholyan, C. K. Haan, and T. B. Ferguson Jr
Comparison of Cardiac Surgery Volumes and Mortality Rates Between The Society of Thoracic Surgeons and Medicare Databases From 1993 Through 2001
Ann. Thorac. Surg., November 1, 2007; 84(5): 1538 - 1546.
[Abstract] [Full Text] [PDF]

Home page
 Ann. Thorac. Surg.Home page
M. J. Magee, M. A. Herbert, T. M. Dewey, J. R. Edgerton, W. H. Ryan, S. Prince, and M. J. Mack
Atrial Fibrillation After Coronary Artery Bypass Grafting Surgery: Development of a Predictive Risk Algorithm
Ann. Thorac. Surg., May 1, 2007; 83(5): 1707 - 1712.
[Abstract] [Full Text] [PDF]

Home page
 J. Thorac. Cardiovasc. Surg.Home page
P. E. Falcoz, M. Conti, L. Brouchet, S. Chocron, M. Puyraveau, M. Mercier, J. P. Etievent, and M. Dahan
The Thoracic Surgery Scoring System (Thoracoscore): Risk model for in-hospital death in 15,183 patients requiring thoracic surgery
J. Thorac. Cardiovasc. Surg., February 1, 2007; 133(2): 325 - 332.
[Abstract] [Full Text] [PDF]

Home page
 Ann. Thorac. Surg.Home page
R. J.F. Baskett, F. H. Cafferty, S. J. Powell, R. Kinsman, B. E. Keogh, and S. A.M. Nashef
Total Arterial Revascularization is Safe: Multicenter Ten-Year Analysis of 71,470 Coronary Procedures
Ann. Thorac. Surg., April 1, 2006; 81(4): 1243 - 1248.
[Abstract] [Full Text] [PDF]

Home page
 J. Thorac. Cardiovasc. Surg.Home page
M. J. Mack, M. Herbert, S. Prince, T. M. Dewey, M. J. Magee, and J. R. Edgerton
Does reporting of coronary artery bypass grafting from administrative databases accurately reflect actual clinical outcomes?
J. Thorac. Cardiovasc. Surg., June 1, 2005; 129(6): 1309 - 1317.
[Abstract] [Full Text] [PDF]

Home page
 J Am Coll CardiolHome page
R. H. Jones
The Year in Cardiovascular Surgery
J. Am. Coll. Cardiol., May 3, 2005; 45(9): 1517 - 1528.
[Full Text] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Right arrow Citation Map
Services
Right arrow Email this article to a friend
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Add to Personal Folders
Right arrow Download to citation manager
Right arrow Author home page(s):
Syma L. Prince
Mitchell J. Magee
Michael J. Mack
Right arrow Permission Requests
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Herbert, M. A.
Right arrow Articles by Mack, M. J.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Herbert, M. A.
Right arrow Articles by Mack, M. J.
Related Collections
Right arrow Professional affairs

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
ANN THORAC SURG ASIAN CARDIOVASC THORAC ANN EUR J CARDIOTHORAC SURG
J THORAC CARDIOVASC SURG ICVTS ALL CTSNet JOURNALS