HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Gary L. Grunkemeier Permission Requests Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Grunkemeier, G. L. Search for Related Content PubMed Articles by Grunkemeier, G. L.

Ann Thorac Surg 2004;77:1964-1965
© 2004 The Society of Thoracic Surgeons

Invited commentary

Medical Data Research Center Providence Health System 44785 NW Elk Mountain Rd Banks, OR 97106, USA

e-mail: gary.grunkemeier{at}providence.org

Many cardiac surgery programs are undertaking internally driven initiatives for quality improvement, and most are experiencing external pressure for public reporting to document accountability. The common pathway to these related goals is data collection and risk modeling. Raw data is collected for each patient, and becomes converted by statistical procedures into "risk- adjusted" results for the provider, by incorporating the patient characteristics particular to that provider.

Some centers collect and produce risk models themselves, but most rely on models produced by others, usually by regional or national organizations which coordinate multi-institutional databases. This process is becoming a widespread and serious undertaking, and must be done very carefully and very well. The results influence the ranking and reputation of providers, and ultimately the welfare of patients. This issue of the Annals contains papers that critique the steps of data collection [1] and model building [2].

Data collection

Herbert and colleagues [1] investigate the accuracy and completeness of cardiac surgery data routinely collected for submission to the Society of Thoracic Surgery (STS) national database. They use an audit of patient records containing 315 STS variables from 18 surgeons to describe the discrepancies that occurred, and to suggest some solutions for improving the process.

Accurate data collection relies on comprehensive definitions of the variables involved. With the available guidelines and standardized definitions this would seem to be a foolproof exercise. But collecting data completely and accurately is difficult in the context of providing care to sick patients. And, in practice there are always perplexing patients who challenge even the best and most well thought out definitions of preoperative conditions and postoperative events.

Risk modeling

Omar and colleagues [2] review many issues involved in developing risk models. Based on operative mortality models presented in 21 published studies, they describe the problems and pitfalls of risk modeling and present recommendations for improvement.

Risk modeling is a blend of science and art, and has more subjective elements than most realize. Unlike data collection, where competent data extractors should reproduce the exactly the same results for the same patient, there are many models that could be produced for the same endpoint using the same data set, depending on the many technical decisions that have to be made [3].

Common threads

An important area of intersection between these two stages of knowledge production, and mentioned in both reports, is the problem of missing data. Data collectors must try to minimize missing data, and data analysts must decide upon an appropriate method to deal with it.

Another common area, not mentioned in either report, is the treatment of continuous variables (age, ejection fraction, body surface area, creatinine level, cross clamp time, intubation time, length of stay, etc). Such variables are often collapsed into dichotomies (high/low, normal/abnormal), thus essentially discarding information. The ability to model differences is increased by respecting the intrinsic continuity associated with such variables. But, in order for this to happen, the data abstracter must record the value as continuous, which takes no more time or space. And, the analyst must treat it as such, at least initially; he/she should be the one to collapse it into groups, if warranted.

All models are wrong

Most clinicians are aware of the Type 1 (false positive) and Type 2 (false negative) statistical errors of hypothesis testing. Analogously, risk factor effects can be erroneously included or excluded in a risk model. Clinical importance and statistical significance are completely different metrics. Even after applying all the good practices for data collection and analysis recommended in these two papers [1, 2], we should be cautious in applying the resulting risk models, keeping in mind the famous observation that "All models are wrong—but some models are useful" [4].

References

1. Herbert MA, Prince SL, Williams JL, Magee MJ, Mack MJ. Are unaudited records from an outcomes registry database accurate? Ann Thorac Surg 2004;77:1960–5.
2. Omar RZ, Ambler G, Royston P, Eliahoo J, Taylor KM. Cardiac surgery risk modeling for mortality: a review of current practice and suggestions for improvement. Ann Thorac Surg 2004;77:2232–7.
3. Naftel DC. Do different investigators sometimes produce different multivariable equations from the same data? J Thorac Cardiovasc Surg 1994;107:1528–9.
4. Box GEP. Robustness in the strategy of scientific model building. In: R. L. Launer and G. N. Wilkinson, editors. Robustness in statistics. New York: Academic Press; 1979:201–36.

This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Gary L. Grunkemeier Permission Requests Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Grunkemeier, G. L. Search for Related Content PubMed Articles by Grunkemeier, G. L.