| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Ann Thorac Surg 1999;68:1195-1200
© 1999 The Society of Thoracic Surgeons
a Division of Cardiothoracic Surgery, State University of New York—Health Science Center at Brooklyn, Brooklyn, New York, USA
b Scientific Academic Computing Center, State University of New York, Health Science Center at Brooklyn, Brooklyn, New York, USA
Address reprint requests to Dr Burack, Division of Cardiothoracic Surgery, Department of Surgery, Box 40, State University of New York, Health Science Center at Brooklyn, 450 Clarkson Ave, Brooklyn, NY 11203
e-mail: ejsd{at}erols.com
Presented at the Thirty-fifth Annual Meeting of The Society of Thoracic Surgeons, San Antonio, TX, Jan 25–27, 1999.
 |
Abstract |
|---|
 Top Abstract IntroductionMaterial and methodsResultsCommentAppendixReferences |
|---|
Methods. All 150 New York State cardiac surgeons were sent an anonymous mail survey in 1997. Data was analyzed to determine the dominant opinion regarding the CSRS.
Results. One hundred and four surgeons (69.3%) responded. The majority (70%) did not experience a change in practice. Data reporting was performed by the surgeon or an employee (58%). Many picked the incorrect definition of chronic obstructive pulmonary disease (COPD) (45%) or statistical method (60%). The aspect of CSRS most in need of improvement was gaming with risk factors (40%). Most surgeons (62%) refused to operate on at least one high-risk CAB patient over the prior year, primarily because of public reporting. Refusal was more common in surgeons in practice less than 10 years, those with less than 100 cases per year, and those with a mixed cardiothoracic practice (p < 0.05, Pearson’s 
2 test). A significantly higher percentage of high-risk CAB patients were treated non-operatively, when compared with ascending aortic dissection patients (not disclosed) (p < 0.001, Wilcoxon signed ranks test).
Conclusions. The public disclosure of surgical results may be based on imperfect data and appears to have resulted in denial of surgical treatment to high-risk patients.
|
Introduction |
|---|
|
TopAbstractIntroductionMaterial and methodsResultsCommentAppendixReferences |
|---|
Many controversies have been precipitated by public disclosure: the accuracy of the statistical model, the relationship between clinical volume and outcome, the denial of treatment to high-risk patients, and the effect on the careers of the cardiac surgeons [2–8]. Additional controversy has created a new vocabulary: the manipulation of patient risk factors to exaggerate risk (gaming), the selective management of patients before operation (outmigration), and after operation (decanting) [2, 9].
The New York State Department of Health (NYSDOH) Cardiac Surgery Reporting System (CSRS) is a prototype regulatory system [10]. The NYSDOH, with the assistance of the 21-member appointed Cardiac Advisory Committee, regulates the 32 cardiac surgical programs statewide. Participation in the CSRS is mandatory to maintain a valid certificate of need (CON). A member of the surgical program compiles a cardiac surgery report for each patient. The report includes demographic data, preoperative risk factors, intraoperative data, and postoperative in-hospital morbidity and mortality. Data are submitted electronically, each quarter, to the NYSDOH.
The observed mortality rate for isolated CABG patients is recorded. The preoperative risk factor data are subject to a multivariate statistical analysis to determine an expected mortality rate. The conventional ratio between observed and expected mortality is calculated, which in turn is multiplied by the statewide mortality rate to determine a risk-adjusted mortality rate (RAM). "The risk-adjusted mortality rate represents the best estimate, based on the associated statistical model, of what the providers’ mortality rate would have been if the provider had a mix of patients identical to the statewide mix" [11]. The data are subject to periodic audit and statistical revision and released to the public approximately 2
years later. RAM is calculated yearly, and if a hospital or an individual surgeon’s RAM is above the confidence limits of the statewide mean, a NYSDOH audit is initiated and various regulatory penalties can occur [10].
The effect of public reporting on the behavior of patients and practitioners in Pennsylvania has been recorded [12, 13]. New York State cardiologists have been polled by the NYSDOH [14]. The direct effect on the New York cardiac surgeons remains unknown. A simple questionnaire was developed to evaluate the process.
|
Material and methods |
|---|
|
TopAbstractIntroductionMaterial and methodsResultsCommentAppendixReferences |
|---|
The questionnaires were analyzed for frequency of response for each question. The data was entered into a statistical software package (SPSS for Windows, V8.0, SPSS Inc, Chicago, IL) to determine if a correlation existed between certain subgroups of surgeons and their resultant opinion and behavior (Pearson’s 2 test). A direct comparison of surgical practice, based primarily on public disclosure, was performed. Categorical responses to repeated measures (questions 16 and 18) were analyzed with the Wilcoxon signed ranks test.
|
Results |
|---|
|
TopAbstractIntroductionMaterial and methodsResultsCommentAppendixReferences |
|---|
Approximately half of the surgeons were in practice more than 10 years (52%) and most surgeons (53%) devote at least 90% of their time to adult cardiac procedures. Typically, most surgeons (69%) perform between 100 and 300 major cardiothoracic cases yearly, and practically all (85%) are enrolled in additional databases, including institutional types (57%) and the Society of Thoracic Surgeons (STS) (38%). Half of the surgeons are involved in protocol supervised clinical research with minimally invasive techniques, CABG and valve replacement procedures being the most popular (Table 1).
|
Data entry was performed by the surgeon (38%), his or her employee (14%), or by hospital financed staff (48%). Almost half of the surgeons (45%) selected the incorrect CSRS definition of COPD and the majority (60%) selected the incorrect type of statistical model used by the CSRS (Appendix). Most surgeons (76%) accurately picked cardiogenic shock as having the highest odds ratio for mortality in New York State, as compared to acute myocardial infarction (MI), renal dialysis, age, and EF 20–29% [11]. Risk factor gaming (40%) followed by method of publication and statistical analysis, were cited as the aspects of the CSRS system needing the most improvement (Table 2).
|
2 test) (Table 3).
|
|
|
|
Comment |
|---|
|
TopAbstractIntroductionMaterial and methodsResultsCommentAppendixReferences |
|---|
For a surgeon, a major benefit of the CSRS is the ability of a large database to provide a "birds-eye view" perspective on surgical results. Far more powerful than a local quality assurance process, a surgeon can compare and compete with the results of his or her colleagues throughout the region. One New York State surgical department substantially improved operative results after it became apparent that the unstable patients from the cardiac catheterization laboratory were being rushed into operation without a preoperative intraaortic balloon pump (IABP). Comparison to statewide mortality rate, and the incidence of preoperative IABP use, along with a hospital-wide collaborative effort, permitted identification of a problem and a rational solution [15]. However, most New York State surgeons participate in additional voluntary databases, which can facilitate quality review and improvement. The national STS database has documented a nationwide reduction in CABG mortality over the past 8 years, without the need for public disclosure or regulatory pressure [16].
From the patients’ perspective, the accurate clinical data detailing surgeon and hospital mortality rates fulfills a basic right to information [17]. In New York, legal interpretation of the freedom of information law (FOIL) favored the broad intent of freedom of information compared with the narrow protection of personal (physician) privacy [18]. Additionally, all the other members of the healthcare process—hospital administration, insurers, and public health planners—can benefit from precise information.
Based on the current survey results, public dissemination of information in New York appears to have initiated a dialogue between surgeons, cardiologists, and their patients. New York surgeons frequently discuss outcome data with both colleagues (44%) and patients (29%). In a survey of New York cardiologists, 22% routinely discuss the data with their patients [14]. In Pennsylvania, the publicly reported data receives less attention, with less than 10% of cardiologists and surgeons discussing the information with their patients [13]. Only 12% of patients were aware of the report and fewer than 1% knew the correct rating of their surgeon or hospital [12].
The accuracy of the CSRS statistical model and the method of data entry remain a significant dilemma. Since its inception in 1989, the actual CSRS statistical model has never been subject to independent peer review. The nuances and limitations of a complex multivariate logistic regression analysis model are not easily appreciated by a physician or layperson [3, 5, 18]. Many surgeons (60%) are ignorant as to the type of statistical method used in the CSRS. An independent, statistical analysis of the NYSDOH database reveals serious flaws with the predictive power of the risk-adjusted model and inconsistencies with the risk factor data [2]. The CSRS method was evaluated from an alternative statistical perspective and inconsistencies were revealed in calibration, resolution, and the overall predictive performance of the model [2].
Predictive value is perhaps the sine qua non of any mathematical construct and can be measured by the C-index, which increases proportionately with the predictive accuracy. A value of 0.5 corresponds to pure chance and 1.0 indicates a test with 100% sensitivity and specificity. A weather prediction statistical model has C-index typically between 0.71 and 0.89, and the current CABG mortality predictive models range between 0.70 and 0.81, with CSRS being between 0.79 and 0.81 [3, 10, 11]. Multivariate logistic regression analysis and newer artificial intelligence techniques remain imperfect and primitive in their ability to provide accurate and useful risk stratified data. In New York State, the vast majority of surgeons rarely or never (91%) use the CSRS program to predict operative mortality. Similarly, most New York State cardiologists (62%) were not affected by the CSRS data when it was time for a surgical referral [14].
Since the inception of patient risk factor schemes, it has been recognized that trained, impartial observers should conduct the data submission [19]. Particularly, when the certificate of need and public reporting hang in the balance, it appears to be a direct conflict of interest to have data reporting remain the responsibility of the practitioners. At present, more than half (52%) of the surgeons, or his or her employee, are responsible for the CSRS data forms. Furthermore, many (45%) surgeons chose the incorrect CSRS definition of COPD.
Gaming is a technique commonly in use to find loopholes to reduce mortality rates. Gaming can reduce published mortality rates by improved data collection, or by intentional deception. The initial CSRS definition of chronic obstructive pulmonary disease (COPD) is relatively soft, and has subsequently been redefined. At one hospital, the incidence of COPD rose from 1.8% to 52% and the overall incidence of COPD varied from 1.4% to 60.6% statewide [2].
Gaming can occur in the operating room: A prophylactic IABP can be inserted before induction, a CABG can be made into a CABG/ventricular aneurysmorrhaphy with one or two additional sutures. Gaming can also occur in the postoperative phase: Comatose patients, who have sustained massive perioperative neurologic injury, can be expeditiously transferred to a chronic care facility. Postcardiotomy shock patients can be transferred on an assist device to a transplant center. Difficult to track, the "decanting" of moribund patients is commonplace and can favorably affect hospital mortality rates.
Recognition of gaming has prompted the creation of a stringent and time-consuming audit mechanism. The NYSDOH audits hospital discharge data from the administrative sources and compares it with the CSRS data [4, 5]. In the current survey, gaming, publication method, and statistical accuracy were repeatedly mentioned as the aspects of the CSRS needing the most improvement.
The CSRS program has had important direct effects on patients and physicians within New York State. The surgeon alone is held responsible for the hospital mortality of CABG patients. In the CSRS, surgical mortality reflects overall hospital mortality and not the conventional 30-day surgical mortality [16]. In New York, 23% of cardiac surgical patients die more than 30 days after admission, and some survive more than 1 year [2]. Clearly, survival during a long-term hospitalization may reflect other mortal disease. Several clinical series have documented a reduction in CABG mortality if genuine teamwork is present. Multidisciplinary collaborative efforts, which focus on the entire process of CABG, can significantly reduce operative mortality [15, 20]. In one instance, an identical team of cardiac surgeons and anesthesiologists, reported significantly different risk-adjusted operative mortality rates in a public and private hospital in the same city during the same time interval [21].
A significant effect of the CSRS on the individual surgeon’s professional practice was reported in 30% of individuals and 37% of their peers, primarily as change in patient selection, or relocation or retirement. In the first 3 years of the CSRS program, at least 21 New York cardiac surgeons retired or relocated [7].
The CSRS program has had substantial effects on New York patients. On the whole, the observed and RAM New York State CABG mortality rates have steadily declined to 2.44% and 2.42%, respectively, well below the nationwide rates of 2.76% and 3.08% [11, 16]. However, New Yorkers must contend with the phenomenon of outmigration and the denial of surgical treatment, a potentially lethal phenomena, particularly for uneducated, disadvantaged, or acutely ill patients.
Outmigration is a term coined by a large cardiac surgical program, in a neighboring state, to document the exodus of high-risk patients from New York to Ohio [9]. Between 1989 and 1992, migrating New York patients were compared with all CABG patients at the Cleveland Clinic. Significantly more New York patients were New York Heart Association class IV, over the age of 65 years, and requiring repeat CABG. New York patients in Ohio had a significantly higher operative mortality rate (5.2%), as compared with all other American patients (3.1%) [9]. Contrary to the reported regional outmigration effect, is administrative data recovered from the national Medicare program. Between 1987 and 1992, the percentage of New Yorkers receiving out-of-state CABG declined significantly from 12.5% to 11.3%, and the likelihood of an elderly patient (over the age of 65 years) receiving CABG in New York State increased significantly over the time interval [22]. At present, the controversy exists because there is no cohesive means of tracking New Yorkers afflicted with coronary artery disease who are potential surgical candidates. All three major therapeutic modalities—surgery, interventional cardiology, and medical therapy—share mortality. Intentional, and elusive, transfer of mortality may occur between the different specialties.
As was first publicized in The New York Times, New York surgeons are not enthusiastic about operating on excessively high-risk patients [8]. In Pennsylvania, cardiac surgical performance reports caused 63% of surgeons to be "much less willing," or "less willing" to operate on severely ill patients [13]. In the current survey, "primarily because of NYS reporting" most surgeons (62%) refused treatment to at least one high-risk patient in the previous year. Comparison of a high-risk CABG, a publicly disclosed surgical procedure, with surgical repair of an ascending aortic dissection, an as yet undisclosed procedure, reveals a significant difference in behavior. In New York, surgeons are significantly more likely to deny treatment to a CABG patient. Despite claims of a fair risk-adjusted method for the high-risk subset of patients, surgeons are subject to substantial pressure to avoid potential mortality [23].
The relationship between surgeons’ volume and operative outcome remains controversial. In New York State, high surgical volume has been associated with significantly better outcome [7]. Nationwide, analyses of the STS and the Veterans Affairs Hospital databases have failed to reveal a significant relationship between hospital volume or practice-group volume and surgical results. However, there was a significant inverse relationship between volume and quality in those programs with less than 100 cases annually, where a significantly poorer outcome was confirmed [6, 24]. The pressure to deny patients treatment appears to be intensified among the young and low volume surgeons, where a small denominator of patients cannot easily absorb a mortality. In New York State, the subset of surgeons in practice less than 10 years, those with less than 100 major cases annually, and those with a mixed cardiothoracic practice all had a significant tendency toward refusal of treatment.
In summary, after 7 years of exposure, the surgeons’ perspective on a public reporting system was solicited in an anonymous survey. The New York CSRS has documented a low statewide CABG mortality rate. However, public disclosure of surgical results has precipitated additional controversy over the methodology and has resulted in the denial of treatment to high-risk patients.
|
Appendix |
|---|
|
TopAbstractIntroductionMaterial and methodsResultsCommentAppendixReferences |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
a Pearson’s 2 test.
CABG = coronary artery bypass grafting.
* Correct answer
|
References |
|---|
|
TopAbstractIntroductionMaterial and methodsResultsCommentAppendixReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  R. Hamblin Regulation, measurements and incentives. The experience in the US and UK: does context matter? The Journal of the Royal Society for the Promotion of Health, November 1, 2008; 128(6): 291 - 298. [Abstract] [PDF]
|
|
|
|
 |
|
 S W Grant, A D Grayson, M Jackson, J Au, B M Fabri, G Grotte, M Jones, and B Bridgewater Does the choice of risk-adjustment model influence the outcome of surgeon-specific mortality analysis? A retrospective analysis of 14 637 patients under 31 surgeons Heart, August 1, 2008; 94(8): 1044 - 1049. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
B Bridgewater and B Keogh Surgical "league tables": Ischaemic heart disease Heart, July 1, 2008; 94(7): 936 - 942. [Full Text] [PDF]
|
|
|
|
 |
|
 H. Miyata, N. Motomura, Y. Ueda, H. Matsuda, and S. Takamoto Effect of procedural volume on outcome of coronary artery bypass graft surgery in Japan: Implication toward public reporting and minimal volume standards. J. Thorac. Cardiovasc. Surg., June 1, 2008; 135(6): 1306 - 1312. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. Goldman, S. Kinnear, J. Chung, and D. J. Rothman New York City's Initiatives on Diabetes and HIV/AIDS: Implications for Patient Care, Public Health, and Medical Professionalism Am J Public Health, May 1, 2008; 98(5): 807 - 813. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. L. Segev, C. E. Simpkins, R. E. Thompson, J. E. Locke, D. S. Warren, and R. A. Montgomery Obesity Impacts Access to Kidney Transplantation J. Am. Soc. Nephrol., February 1, 2008; 19(2): 349 - 355. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. H. Fung, Y.-W. Lim, S. Mattke, C. Damberg, and P. G. Shekelle Systematic Review: The Evidence That Publishing Patient Care Performance Data Improves Quality of Care Ann Intern Med, January 15, 2008; 148(2): 111 - 123. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. Kolling, B. R. Simmen, G. Labek, and J. Goldhahn Key factors for a successful National Arthroplasty Register J Bone Joint Surg Br, December 1, 2007; 89-B(12): 1567 - 1573. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. Westaby, N. Archer, N. Manning, S. Adwani, C. Grebenik, O. Ormerod, R. Pillai, and N. Wilson Comparison of hospital episode statistics and central cardiac audit database in public reporting of congenital heart surgery mortality BMJ, October 13, 2007; 335(7623): 759 - 759. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
B. Bridgewater, A. D Grayson, N. Brooks, G. Grotte, B. M Fabri, J. Au, T. Hooper, M. Jones, B. Keogh, and on behalf of the North West Quality Improvement Pr Has the publication of cardiac surgery outcome data been associated with changes in practice in northwest England: an analysis of 25 730 patients undergoing CABG surgery under 30 surgeons over eight years Heart, June 1, 2007; 93(6): 744 - 748. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. M. Shahian, F. H. Edwards, V. A. Ferraris, C. K. Haan, J. B. Rich, S.-L. T. Normand, E. R. DeLong, S. M. O'Brien, C. M. Shewan, R. S. Dokholyan, et al. Quality Measurement in Adult Cardiac Surgery: Part 1--Conceptual Framework and Measure Selection Ann. Thorac. Surg., April 1, 2007; 83(4_Supplement): S3 - S12. [Full Text] [PDF]
|
|
|
|
 |
|
 L. P. Casalino, G. C. Alexander, L. Jin, and R. T. Konetzka General Internists' Views On Pay-For-Performance And Public Reporting Of Quality Scores: A National Survey Health Aff., March 1, 2007; 26(2): 492 - 499. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. M. Kahn, A. A. Kramer, and G. D. Rubenfeld Transferring Critically Ill Patients Out of Hospital Improves the Standardized Mortality Ratio: A Simulation Study Chest, January 1, 2007; 131(1): 68 - 75. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. J. Epstein Do cardiac surgery report cards reduce mortality? Assessing the evidence. Med Care Res Rev, August 1, 2006; 63(4): 403 - 426. [Abstract] [PDF]
|
|
|
|
 |
|
 C R Canning The power of integrated data. Br. J. Ophthalmol., August 1, 2006; 90(8): 938 - 939. [Full Text] [PDF]
|
|
|
|
 |
|
 Z. G. Turi Reply J. Am. Coll. Cardiol., April 18, 2006; 47(8): 1737 - 1738. [Full Text] [PDF]
|
|
|
|
|
|
M. B. Rosenthal and R. G. Frank What is the empirical basis for paying for quality in health care? Med Care Res Rev, April 1, 2006; 63(2): 135 - 157. [Abstract] [PDF]
|
|
|
|
|
|
Z. G. Turi The Big Chill: The Deleterious Effects of Public Reporting on Access to Health Care for the Sickest Patients J. Am. Coll. Cardiol., June 7, 2005; 45(11): 1766 - 1768. [Full Text] [PDF]
|
|
|
|
 |
|
 C. J. Hilton, J.R. L. Hamilton, N. Vitale, and R. Haaverstad Effects of 'Bristol' on surgical practice in the United Kingdom Interactive CardioVascular and Thoracic Surgery, June 1, 2005; 4(3): 197 - 199. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R. M. Werner and D. A. Asch The Unintended Consequences of Publicly Reporting Quality Information JAMA, March 9, 2005; 293(10): 1239 - 1244. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
B. Bridgewater and on behalf of the adult cardiac surgeons of north w Mortality data in adult cardiac surgery for named surgeons: retrospective examination of prospectively collected data on coronary artery surgery and aortic valve replacement BMJ, March 5, 2005; 330(7490): 506 - 510. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. R. Narins, A. M. Dozier, F. S. Ling, and W. Zareba The Influence of Public Reporting of Outcome Data on Medical Decision Making by Physicians Arch Intern Med, January 10, 2005; 165(1): 83 - 87. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
D. M. Shahian Improving Cardiac Surgery Quality--Volume, Outcome, Process? JAMA, January 14, 2004; 291(2): 246 - 248. [Full Text] [PDF]
|
|
|
|
 |
|
 M. J. Swart and G. Joubert The EuroSCORE does well for a single surgeon outside Europe Eur. J. Cardiothorac. Surg., January 1, 2004; 25(1): 145 - 145. [Full Text] [PDF]
|
|
|
|
|
|
D. Pitches, A. Burls, and A. Fry-Smith How to make a silk purse from a sow's ear--a comprehensive review of strategies to optimise data for corrupt managers and incompetent clinicians BMJ, December 20, 2003; 327(7429): 1436 - 1439. [Full Text] [PDF]
|
|
|
|
|
|
P. P. Goodney, G. T. O'Connor, D. E. Wennberg, and J. D. Birkmeyer Do hospitals with low mortality rates in coronary artery bypass also perform well in valve replacement? Ann. Thorac. Surg., October 1, 2003; 76(4): 1131 - 1137. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. H. Liu, D. A. Etzioni, J. B. O'Connell, M. A. Maggard, D. T. Hiyama, and C. Y. Ko Inpatient Surgery in California: 1990-2000 Arch Surg, October 1, 2003; 138(10): 1106 - 1112. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
R. S. Hartz, J. A Swain, and L. Mickleborough Sixty-year perspective on coronary artery bypass grafting in women J. Thorac. Cardiovasc. Surg., September 1, 2003; 126(3): 620 - 622. [Full Text] |
