| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Ann Thorac Surg 2005;80:802-810
© 2005 The Society of Thoracic Surgeons
a Department of Cardiothoracic Surgery, The Albert Einstein College of Medicine, New York, New York
b Thoracic Surgery Directors Association, Arlington, Virginia
c Division of Cardiothoracic Surgery, University of Colorado HSC, Denver, Colorado
d Division of Thoracic Surgery, Massachusetts General Hospital, Boston, Massachusetts
e Department of Surgery, University of Michigan Medical Center, Ann Arbor, Michigan
f Division of Cardiothoracic Surgery, University of Washington, Seattle, Washington
Accepted for publication March 7, 2005.
* Address reprint requests to Dr Gold, The Medical University of Ohio at Toledo, College of Medicine, Raymon H. Mulford Library, 3045 Arlington Ave, Toledo, OH 43614-5805 (Email: jgold{at}mco.edu).
Presented at the Forty-first Annual Meeting of The Society of Thoracic Surgeons, Tampa, FL, Jan 24–26, 2005.
 |
Abstract |
|---|
 Top Abstract IntroductionMaterial and MethodsResultsCommentDiscussionAcknowledgmentsReferences |
|---|
METHODS: Over 3 years the utilization and impact of this curriculum was evaluated by 8 different methods including resident surveys, faculty surveys, web tracking, examination scores, and individual online learning module critiques. Each completed critique evaluated the resident’s perception of the relevance, content quality and presentation quality of the module on a scale corresponding to low, average and high.
RESULTS: A total of 11,117 learning modules were used through the end of 2004 with complete critiques available on 1458 (13%). Utilization and measured quality continue to increase over the 3-year interval. The average rating scores for relevance, content and presentation tracked together and were overall rated as good or excellent (92%). There were large variations in utilization and perceived quality by section subject. As the average module rating score improved, the utilization increased and the variability of the rating decreased substantially.
CONCLUSIONS: Module specific learner evaluation of educational content is one of several important feedback tools for monitoring ongoing curricular development and refinement. These studies, based upon web tracking, demonstrate increasing resident use and strong resident satisfaction. This educational format will hopefully lend itself to many important improvements in web-based Thoracic Surgery education.
|
Introduction |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionAcknowledgmentsReferences |
|---|
The ongoing changes in the work hour time available for resident education in the clinical setting and the distribution of those hours during the typical residency work week has further increased the challenges of transmitting increasingly large quantities of important cognitive information [6, 7]. The ongoing shift to ambulatory and same day procedures and other changes are further separating residents from clinical and didactic opportunities. While focusing here on thoracic surgery residency learning processes, these same challenges are faced in the life-long learning and maintenance of certification aspects of clinical practice and academia across the spectrum of Thoracic Surgery and most other medical specialties.
The rigorous exploration of curriculum development and Web-based implementation modalities using Thoracic Surgery resident education as a model is the primary goal of this Thoracic Surgery Directors Association (TSDA) program. As a preliminary step, the Prerequisite Curriculum was established to create an inclusive outline and a robust catalog of factual knowledge that would be "optimally required" for residents to master prior to initiation of their Thoracic Surgery residency. As a secondary goal, the TSDA focused on the development of an innovative high technology methodology to deliver the curriculum in such a way that it would be highly efficient, impart enduring knowledge and become easily integrated into the didactic spectrum of our residency education programs during subsequent training and practice. This curriculum content would be immediately useful and later serve thoracic surgeons as a permanent reference before, during, and after the completion of their formal thoracic surgery residency [8].
It was necessary to analyze the current structure of thoracic surgery education and determine a useful division of that didactic material which would be transmitted before as opposed to during the formal residency program [9–16]. In 1994 the TSDA provided a requisite core curriculum outline for the technical, clinical and didactic material that is to be taught during the entire span of the thoracic surgery residency [17, 18]. It is the intent of the curriculum project to focus on the didactic aspects of that information that we believe residents should master before beginning a thoracic surgery residency.
|
Material and Methods |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionAcknowledgmentsReferences |
|---|
|
The 75 segments were all built around a short audiovisual presentation narrated by the individual segment author. Relevant media including video, photography, radiographic scans, images and other material were included. Each segment also contains a 2- to 6-page summary document as well as a set of frequently asked questions (and answers), a bibliography, a set of relevant Internet linkages, and a dynamic linked self-assessment examination for each resident to successfully complete before being able to consider any learning module as "done" [19]. Each modular segment also has three evaluation questions relevant to the learning experience just completed.
The CD-ROM Internet hybrid product itself represents a high technology marriage between the internet and CD-ROM-based technology. This was done to minimize utilization loading type delays and associated streaming video types of user frustration. A unique user security system was developed, tested, and implemented to securely identify the user and prevent "sharing" of content or user identities. Over time, most of the high bandwidth content was transferred to the secured central servers as high-speed internet access became less of an issue. The unique serialized CD-ROM was preserved as a high security access logon key.
There are four different ways to navigate through the curriculum content. The textbook-based navigation system is similar to a table of contents with numerous sections and subsections related to the breath and depth of the PRC. A second means of navigation is the case-based system in which 12 clinical surgical cases together make up the 75 segments. By managing the 12 individual patients, the residents would be able to acquire the same amount of factual material as if they had worked through the textbook-based section in a sequential fashion. Any one of the segments and segment sub-components can be located and learned or relearned in any sequence, in any number of sessions and at any time. Once a segment is completed, the users receive "credit" for the segment and are informed as such should they revisit the segments at a later time.
Implementation
Following the 4 years of development and testing, a letter describing the Prerequisite Curriculum Project was sent to all of the 142 Thoracic Surgery residents matched in June 2001. The letter included a description of the project and requested the residents to sign an informed consent allowing them to be prospectively randomized into 1 of 2 groups, those receiving the full CD-ROM Internet hybrid curriculum and those receiving only an outline of the content of the curriculum, appropriate references, but no educational content. Great care was taken to inform all of the thoracic surgery and general surgery program directors of their residents’ participation in the program but not of their randomization status at any point. Each resident also agreed not to share his/her randomization status with his/her faculty, current program director, and future program director or with his/her co-residents. The TSDA reassured all participating residents that their randomization status and performance/utilization parameters would not be identified with them in any way, and as such, never shared with their faculty, program directors, or used in the credentialing, promotion or certification process.
The 138 residents agreeing to participate were randomized in 2001. They were instructed to proceed at a pace of approximately 1 section (approximately 7 segments) per month, allowing all of the sections to be completed in the course of the upcoming prematriculation year. They were informed that their time investment in each of the sections as well as their critiques of the individual segments would be tracked on the Internet. The eight intended evaluation and follow-up modalities were described along with the associated projected time lines for the implementation and subsequent evaluation.
During the ensuing 36 months after the curriculum was distributed there have been follow-up phone calls and written survey questionnaires during their preresidency year and several times during their residency years for all of the randomized residents requesting specific information regarding the ease of use, overall evaluation and in particular, any technical problems or suggestions they might have during the time period from their randomization until their residency program graduation in July 2004.
The evaluation of the two resident groups has been ongoing and extensive (Fig 1). All of the residents matriculating in July 2002 were asked to take the American Board of Thoracic Surgery In-Training Examination (ITE) in August 2002 reflecting their didactic base of knowledge at a time 4 weeks after they began their thoracic surgery residency. They also took the (different) ensuing in-training examination given by the American Board of Thoracic Surgery in April 2003 and again in April 2004. Additional follow-up information concerning the transition from prerequisite to the TSDA Core Curriculum, as well as performance on subsequent in-training examinations and ultimately the American Board of Thoracic Surgery qualifying and certifying examinations were also arranged. Techniques were developed and implemented during the residency to assess other areas of impact of the prerequisite curriculum on the thoracic surgery residency and thereafter. In particular, psychological and performance parameters, comfort levels and the overall satisfaction of the residents were periodically assessed.
|
|
Results |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionAcknowledgmentsReferences |
|---|
Prematriculation Web Tracking
Internet tracking was reviewed for all residents utilizing the curriculum (Table 2). The utilization patterns varied considerably in average and on a resident-by-resident basis from those reported by the residents in the survey tools they completed. A total of 47 of the 64 residents receiving the CD-ROM set actually used the prerequisite curriculum more than once and only 31 of 69 residents receiving the curriculum used the product for 20 or more sessions. These 47 residents had a total of 3161 educational sessions averaging 19.6 to 23.3 minutes each, with an average of 148 sessions per resident over the prerequisite year. This was measured as an average of 1.44 hours per week for each resident who actually used the curriculum. Thus, there were substantial differences between information provided by resident survey completion and analysis by objective internet tracking.
|
|
Learning Module Critiques
All completed learning module critiques for 2002 were reviewed (865 of 3202 completed segments; Table 1). Use of these modules varied greatly with the subject matter, with the highest use in the normal thoracic anatomy, adult cardiac diagnostic studies, normal thoracic physiology and cardiopulmonary bypass areas. The lowest use was seen in the research methodology, professional information, academics, professionalism, and ethics sections (Table 4). Ninety-two percent of the segment critiques for all rated modules were rated to be in the useful/high quality or very useful/very high quality range. There was a high correlation for each section with regard to the three questions of relevance, quality of foundation and presentation quality was noted for all 13 sections (Fig 2). There was a relationship noted between the number of instances each of the sections were used and the average critique score in the 3 (Fig 3). The higher rated modules seemed to be used more frequently. As the number of learning module critiques completed annually rose through the end of 2004, so did the number of total module critiques completed (Table 3). During this period of time the average rating in each of the three categories (relevance, foundation, and quality) also rose from a mean of 1.38 to 1.53 (p < 0.05).
|
|
|
|
On the postmatriculation surveys, the faculty and program directors reported smaller but significant differences in overall fund of knowledge and application of acquired knowledge favoring those residents using the PRC. No significant differences were seen in the teaching categories and none were measured in the comfort/satisfaction parameters when the two groups were compared. Interestingly, the program director scores evaluated on an identical scale were statistically higher on average in each of the 51 categories of knowledge and performance than the resident’s self-assessment of the same category of knowledge and particularly for those residents not receiving the PRC.
Study Limitations
The evaluation of the first 3 years of experience is limited in that it is predominantly descriptive due to the relatively small total number of randomized residents involved and the even smaller number of residents actually using the curriculum materials in what was believed to be a meaningful way. While it was not possible to scientifically define meaningful utilization, the benchmark of twenty total sessions, approximately one quarter of the total modules or an average of one twenty-minute session every other week prior to matriculation was used to facilitate the analysis. Had a larger number of sessions been required to be deemed significant, such as half of the sessions, the total number of residents available for analysis in this group would have been significantly smaller yet. The lack of statistical power was anticipated given these small numbers and the imprecise parameters available to measure impact of this type of curricular implementation.
The educational outcome assessment tools that were utilized do not mirror the content that was delivered, nor do they separate the outcomes of a highly motivated learner from a less motivated learner. In particular, the American Board of Thoracic Surgery ITE has a scope far broader than those topics covered in the PRC, as it is designed to cover the core curriculum as well as all topics within the prerequisite curriculum. Attempts to look at responses by randomized residents to a specific ITE question set specifically relevant to those topics covered within the Prerequisite Curriculum and not the entire examination was attempted and rapidly became logistically impossible.
Extending the randomized study to additional generations of residents matriculating in subsequent years was considered but was felt to deprive the matching residents of a potentially highly effective learning tool. Further evaluation is of little potential for better comparative data, as the teaching tool has become widely available and will likely be "shared" among residents and program directors. It has subsequently been provided in full format to all subsequent matriculating residents and to the program directors of all training programs. It was not possible to accurately measure the amount of "sharing" that occurred prior to and following the matriculation date described in this prospective randomized study, thus contaminating the two groups. There was a true element of individual and group competition between the two groups of residents, thus skewing the performance of the control group away from historical baselines. In addition, those residents in the control group were promised the full curriculum upon matriculation as a condition for participation, which they did indeed receive.
Significant differences between the resident survey data and the web-based tracking of utilization measured objectively on the electronic system underscores the benefits of objective measurements and the associated inherent challenges in self-reported surveys. Analysis of these differences revealed substantially less utilization (sessions and duration) than initially anticipated and as self-reported on many of the self-reported survey tools. The widely known TSDA expectation that the curriculum was to be completed prior to residency matriculation may have influenced the survey responses.
|
Comment |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionAcknowledgmentsReferences |
|---|
Given the recent changes that have occurred regarding resident work hours, the ability to deliver high quality educational didactic material, clinical material, and hopefully technical materials through a distance learning system and to track utilization and performance becomes even more appealing. This form of technology has broad-based applicability as a highly specialized learning management system focused on distance learning in a cohort of residents not directly under the watchful eye and gentle hand of their thoracic surgery program director. The extensive and ongoing user feed back on relevance and quality is an important aspect of this learning system. It is clear that those learning modules deemed to be of the highest relevance and quality were used the most. In addition, it appears that the traditional areas of anatomy and physiology have transitioned best to a web-based format, while our greatest challenges of resident utilization lie in the areas of professionalism, ethics and other newer Accreditation Council for Graduate Medical Education competencies. Only by continually tracking module use, learner assessed quality/relevance and other appropriate outcome parameters (exam performance, patient interaction, clinical performance, etc.) can we continue to mature these educational tools.
Although Web-based education has been available for many years, a prospective randomized study comparing it with traditional textbook-based learning in thoracic surgery is novel. Indeed, prospective and retrospective educational trials, "educational epidemiology", have been relatively rare [20–26]. Multiple attempts have been made to implement Web-based, CD-ROM-based, and integrated educational tools in many specialties with variable assessment systems and with widely varying results [27–40]. It is our anticipation that successful completion of this project will not only allow for the use of an innovative, highly effective means of education for our residents, but may in turn become broadly attractive to other types of educational projects within thoracic surgery education particularly in the venues of life-long learning and maintenance of certification. This may also be suitable for other types of educational projects in the postgraduate education universe and other venues of non-medical education as well. The conduct of quality prospective randomized scientific studies monitoring the impact of curricular projects, as well as the measurement of acceptance of the curricular materials by the residents and program directors, is a highly desirable implementation model and will hopefully be continually scrutinized and improved by educators over time.
|
Discussion |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionAcknowledgmentsReferences |
|---|
In thoracic surgery, however, we have been blessed with more than our fair share of distinguished medical educators. Not only were surgeons like Drs Lillehei, DeBakey and Cooley pioneers in the clinical realm, but they have also cast a large shadow over all of medical education. It would be even fair to say that thoracic surgeons have traditionally been national leaders in the way we educate medical students and residents.
With the establishment of this important prerequisite curriculum project by Dr Gold and other members of the Thoracic Surgery Directors Association, I believe the bar has been raised to the next level for the way we educate current and future residents. In essence, what we have is "resident education" version 2.0.
In putting together this ambitious project, the authors indirectly had to deal with a number of fundamental questions, namely, what do we mean by resident education and how exactly should residents acquire new skills and knowledge? Admittedly these are challenging issues but Dr Gold and colleagues have not only addressed these questions, they have devised new and innovative solutions. By designing a web-based prerequisite curriculum which is comprehensive, accessible and relevant, incoming thoracic surgery residents will have the unparalleled opportunity to assimilate important didactic information prior to matriculating into thoracic surgery. I think the significance of this e-learning initiative has implications well beyond just the education of thoracic surgery residents.
As you know, the practice of thoracic surgery has crossover into several other disciplines, including cardiology, radiology, oncology, vascular surgery, and pulmonary medicine. By breaking the entire thoracic surgery prerequisite curriculum into its component parts, it would be possible for incoming residents in other specialties to access those parts of the thoracic surgery web-based curriculum which would be pertinent for their own residency education curriculum. This would also unite and bring together an entire network of resident education.
Additionally, specialties with only an indirect connection to thoracic surgery, for example, family medicine, but with a direct connection to one of our crossover specialties, could benefit by incorporating relevant components of that specialty’s prerequisite curriculum into its own curriculum. By plucking a single string, educators in thoracic surgery have produced beautiful music which will be heard throughout the educational universe.
Dr Gold, I have a few questions that I have shown on this slide. In your manuscript you mentioned that 12 of the 75 learning modules were clinical surgical cases. This problem centered-approach can be an effective method of instruction and is often well received by students and residents. Did you break out the resident response and utilization data regarding those specific modules? Were these particular modules perceived to be more effective and enjoyable for the residents in the study? Do you intend on including more problem-centered modules into the prerequisite curriculum?
Second, although the response and feedback by residents was favorable for each of the e-learning modules in this initiative, you were unable to demonstrate any difference between the two study groups with respect to long-term in-service training exam scores. There was trend among residents who completed more than 20 e-modules, but this difference diminished over time. What other alternatives have you considered to better assess educational outcomes in your study groups?
Finally, what efforts will be made to update these learning modules and incorporate feedback provided by residents and program directors into future versions of this product?
I would like to thank the Society again for the opportunity to discuss this paper, and I want to thank you, Dr Gold, and your colleagues and the Thoracic Surgery Directors Association for bringing this important addition to resident training forward and for your continued efforts on behalf of resident education. Thank you.
DR GOLD: Thank you, Dr Wheatley, for your comments. It is truly an honor to have you discuss this presentation not only as a discerning colleague but from the perspective of a resident with extensive ‘hands on’ experience with this curriculum tool. Thank you for your excellent questions.
The clinical case-based navigation system was only minimally used, and therefore it is impossible to reach conclusions about its efficacy. The TSDA invested much in resources to produce the case-based system, but the utilization has lagged behind the textbook-based system.
Many problem-based learning activities are under development at this time by the TSDA and other organizations, but are not currently a part of the prerequisite curriculum. The prerequisite curriculum was developed to be a core of basic knowledge and serve as a foundation upon which to build a myriad of interactive dynamic educational techniques. The Prerequisite Curriculum Editorial Board is actively reviewing the entire curriculum from this perspective. This editorial board meets frequently and is looking at quality, format and presentation and is looking forward to participation of many of our members to continue to add and update the curriculum.
The American Board of Thoracic Surgery in-training examination scores were not statistically different between the two randomized groups of residents. This is most likely because the numbers of residents are small and that the exam does not specifically target the subject content that we particularly try to deliver in a prerequisite curriculum. There was a trend toward improved ITE performance in those residents receiving the prerequisite curriculum, which showed increasingly larger differences as resident use increased in number of sessions and total hours of utilization.
The TSDA and I thank you again for your kind words and insightful questions. In addition, the TSDA is very grateful to the Society of Thoracic Surgeons for allowing the opportunity to present this maturing educational project. It has been a great honor to stand before you today.
|
Acknowledgments |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionAcknowledgmentsReferences |
|---|
|
References |
|---|
|
TopAbstractIntroductionMaterial and MethodsResultsCommentDiscussionAcknowledgmentsReferences |
|---|
This article has been cited by other articles:
|
|
 |
|
  A. N. Hoofnagle, D. Chou, and M. L. Astion Online Database for Documenting Clinical Pathology Resident Education Clin. Chem., January 1, 2007; 53(1): 134 - 137. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 R. Friedl, H. Hoppler, K. Ecard, W. Scholz, A. Hannekum, W. Oechsner, and S. Stracke Comparative Evaluation of Multimedia Driven, Interactive, and Case-Based Teaching in Heart Surgery Ann. Thorac. Surg., November 1, 2006; 82(5): 1790 - 1795. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. Gold Invited commentary. Ann. Thorac. Surg., May 1, 2006; 81(5): 1766 - 1766. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
= relevance; 
= foundation; 
= quality.