BACKGROUND
Residency education occurs in the context of a learning and working environment.1 The Accreditation Council for Graduate Medical Education Common Program Requirements2 include residency, faculty, and program evaluation. Program review of the learning environment for trainees is necessary to identify strengths, weaknesses, opportunities, and threats related to a program’s mission and aims. End-of-rotation evaluations are administered to residents to assess the educational value of individual clinical experiences.3
Program threats can be identified by evaluating trends in resident assessment of training rotations. Program threats may include factors affecting medical knowledge acquisition, achievement of training milestones, supervisory oversight until clinical competency, duty hours, and the balance of service versus education.4,5 Rotations in graduate medical education are defined as training periods in a medical discipline or clinical care setting which residents practice medicine under the supervision of established physicians.6
Lower rotation scores necessitate intentional changes be made by program leadership, core faculty, and other stakeholders.7 Our academic program is a large internal medicine residency in an urban, multi-hospital system. Each of the distinct program sites of training include general medicine wards as required, core rotations for physician trainees. In this study, we sought to incorporate generalizable solutions to improve the overall quality of education for internal medicine residents rotating on inpatient medicine at one of our sites of training.
METHODS
A collective of faculty members assembled to identify strategies to augment the educational rotation experience for residents at the university-affiliated Veterans Affairs Medical Center. Post-rotation evaluations were known to be below program average. This group included hospitalists board-certified in internal medicine who spend a portion of their clinical duties as supervising attending physicians to healthcare professional trainees on this low-scoring teaching service.
The collective critically examined quarterly end-of-rotation evaluations by residents using thematic analysis. Qualitative data in the form of comments was codified and organized by themes. Critical analysis identified three common themes representing areas of improvement in the learning environment: culture, communication, and workload. Correlation was made between the recurrent themes and teaching sub-metrics measured quantitatively. A group problem solving process was employed to design solutions for each domain.
First, workplace culture was addressed by initiating a site-specific orientation. The session objectives were to provide education on the role of this hospital in the community, understand the unique needs of the patient population served by this hospital, and familiarize the residents with the learning environment by facilitating tours of the facility. Trainees were also introduced to key staff who provide instruction on the core values of the health care system (i.e., Veterans Health Administration).
Second, communication was improved using a secure messaging platform that was enhanced to provide real-time notification of patient admissions via electronic text messages with closed loop communication. This facilitated open channels for the admitting providers, hospitalists, resident teams, and utilization management to cooperate in the admission process of patient care.
Third, an automated, digital rounding tool was integrated into the organization’s electronic medical record to bolster efficiency of the medical team and reduce individual provider workload. This tool allowed the direct capture of pertinent patient information (e.g., location, vital signs, and laboratory tests) from the comprehensive information contained in the electronic health record. Extracted information was configured to auto-populate into a single document made available to all members of the health care team for use during clinical rounds for patient care and medical education.
SMART (Specific, Measurable, Achievable, Relevant, and Time-Bound) goals were established for each intervention to ensure individual effectiveness. Each solution had a process owner, and all members of the collective agreed to participate in the ongoing implementation of the combined intervention. After a period of development for the solutions using available rotation data in the form of resident evaluations, a roll-out plan for the combined intervention occurring over the beginning of the new academic year (2023-2024) was launched. The collective met monthly to review new evaluation data, discuss implementation progress, and adjust solutions based on feedback.
Outcome measures of education were obtained via end-of-rotation survey evaluations (Figure 1). Surveys assessed 5 domains of education quality using a 5-point Likert scale (1-poor, 5-excellent). Residents completed rotation surveys following their rotation; results were aggregated by quarter and spanned the pre- and post-intervention periods. Surveys were anonymous and blinded to the reviewing collective. Thus, demographic data including post-graduate year of training was unavailable for analysis. Statistical analysis was performed using a two-sample t-test.
The collective of internal medicine faculty responsible for leading this performance improvement project in education began meeting in February of 2023 upon program release of the 2022-2023 academic year second quarter (October-December) rotation evaluations. Monthly meetings occurred thereafter with subgroup meetings occurring for the development of the individual interventions as needed. Three consecutive quarters of rotation data were collected during the development phase of the project. The roll-out phase for the combined intervention occurred during the first quarter (July-September) of the 2023-2024 academic year. Resident rotation evaluation data was collected during the roll-out phase and final comparative quarter (October-December) of the 2023-2024 academic year.
This performance improvement project was conducted in accordance with our institution’s stated principles for program evaluation and was exempt from institutional review board evaluation or ethics approval given use of anonymized evaluation survey data in a non-interventional study.
RESULTS
The pre-intervention cohort of trainees included 198 evaluations representing four consecutive quarters spanning from the second quarter (October-December) of the 2022-2023 academic year through the end of first quarter (July-September) of the 2023-2024 academic year. The post-intervention cohort consisted of 58 evaluations representing the final comparative quarter (October-December) of the 2023-2024 academic year. An end-of-rotation evaluation was assigned to a resident for each consecutive period spent assigned at this clinical practice site during both the pre-intervention and post-intervention portions of this study. Rotation evaluations by residents were mandatory as part of ACGME program requirements, and residents were required to complete evaluations according to program policy.
The roll-out phase for study interventions was started during the first quarter of the 2023-2024 academic year and completed before the start of the second quarter of that academic year. After full implementation of all three solutions, significant improvements in the educational experience were achieved for this previously low-scoring rotation. Trainees’ perceived quality of education improved from good to very good (3.26 to 4.02, +15%, p < 0.001) overall on a 5-point Likert scale compared to one year prior (Figure 2). All four subcategories of education demonstrated increases in scoring (Figure 3): diversity/quality of pathology seen (0.33; +7%), supervision (0.50; +10%), workload (0.42; +8%), and learning value (0.51; +10%).
To account for the roll-out phase for of these interventions occurring during the first quarter of the 2023-2024 academic year, a separate subgroup analysis was performed on quarterly evaluations collected immediately prior and after the roll-out phase. When the five categories were compared between the fourth quarter of the 2022-2023 academic year and the second quarter of the 2023-2024 academic year using a paired t-test, the improvement remained statistically significant (p = 0.007).
DISCUSSION
This study demonstrates a multifaceted intervention by immediate stakeholders to reinvigorate the learning climate for an internal medicine core rotation at a Veterans Affairs training site within a multi-hospital academic institution. This educational improvement work underscores the importance of understanding the needs and challenges faced by learners at an individual site to successfully steer the rotation to the shared program goal of education excellence.8 The remedy for team-based underperformance in hospital care rarely consists of a single solution; instead, combinations of simple solutions may be successful.9,10 This project utilized multiple interventions in developing a cure for a rotation with unhealthy ratings. The objective for each solution was to address modifiable aspects of each common theme identified as impacting the educational experience of the rotation.
The solutions used in this study to improve the educational experience are based on similar results elsewhere,11 but to our knowledge have not been previously combined in this specific manner. Programs are responsible for creating a learning-oriented culture accountable for developing competent residents.12 Effective communication between healthcare professionals facilitates teamwork, clinical effectiveness, and working relationships.13 Literature supports the perception that heavy workload can lead learners to adopt superficial clinical shortcuts rather than deeper, higher-order processing and critical thinking.14 This process can be mitigated through intentional planning by faculty and programs to allow for more efficient teaching and learning.
Immediate next steps to further validate our healthcare professional trainee educational experience improvement collective involve the continuous monitoring of evaluation trends and learner feedback. This will be achieved through regular review of quarterly rotation evaluation reports provided anonymously by the training program. Resident feedback sessions and class representation meetings with program leadership occur as part of the continuous educational curriculum review by our residency. Finally, our institution conducts a yearly overall program review of all data available including individual rotation scores by training site, resident clinical performance in competency milestones, in-training exam scores, program exit surveys, and pass rates for the American Board of Internal Medicine initial certification exam.
The orientation developed as part of our three-pronged change approach introduced trainees to site-specific educational benefits that are distinct from other hospitals in our program. Training enhancements within this orientation included descriptions of the healthcare system (i.e., Veteran’s Health Administration), the patient population in our region, local wellness programs available to providers, and basic training tips for operational success. These efforts to address institutional knowledge gaps in culture inherent to new employees within a healthcare organization were borrowed from frameworks for talent management15 and generally well-received by trainees in attendance at orientation.
Based on our success with the offering of an orientation tailored for our teaching hospital within an overall program welcome for incoming internal medicine trainees, this new onboarding session has been incorporated into the enduring training agenda for future incoming residents. In fact, our organization has mirrored core aspects of the introductory medicine seminar to model onboarding sessions offered to other healthcare professional trainees including residents and fellows from other subspecialities starting on rotation at our hospital. The organizational principles described here are transferable to other medical centers with a similar multihospital profile, offering best practices that target workplace culture and bolster staff engagement in a specific hospital setting.
Our intervention involving real-time notification of new patient encounters for on-call providers, including residents, has optimized admission throughput and provider communication. An unintended positive consequence of the use of this platform has been increased workload transparency across our inpatient medicine services, including active tracking of medical provider and team patient census, admission caps and thresholds, and patient care assignments to teaching versus direct care services. Communication and transparency across services have been shown to strengthen workplace culture during times of increased workload and stress.16
A multifunctional rounding tool was made available to the healthcare trainees as an embedded tool within the current electronic health record system. Accessible as a web-based application, the main functionality of this tool was to collect pertinent information from an entire panel of patients for use on clinical rounds quickly and effectively. Use of this tool improved the efficiency of pre-rounding productivity and allowed for prioritization of patient management decisions based on the ubiquitous availability of these data sets. Adoption of the assistive rounding tool by residents on the inpatient medicine rotation has been successful. This was achieved through a publicity campaign, feasibility study with survey, and role modeling of use by attending physicians across medical specialty services. Based on preliminary data regarding its use and benefits in rounding efficiency, our hospital has approved funding to secure continued use of this product for our electronic health record. Similar positive trends in resident experience gained through efficiency while preserving education and patient safety have been demonstrated by others using electronic health record-integrated tools and other novel rounding strategies.17,18
One subdomain (quality and diversity of pathology seen) remains a target for growth. Although this site’s patient population is static, opportunities to channel admitted patients to different types of providers may exist. A future solution under consideration is to create a service staffed by non-trainees dedicated to the care of patients deemed low impact for teaching purposes. Another strategy is to improve admission screening through vigilant utilization management. Finally, stakeholders in this study have begun piloting an embedded hospitalist service in the emergency department to improve admission throughput and patient care experience.
This study reports successful improvements for an internal medicine rotation with low resident evaluation scores. Limitations of this study include its design as an observational study, which hinders interpretations of causality. Although observational studies are more susceptible to bias and confounding, this study was prospective and thus less prone to recall bias. Second, the cohort of residents sampled via end-of-rotation evaluations during the pre- and post-intervention stages included only those rotating during the study timeframe. While there was some overlap in participants between the two cohorts, the two groups were not matched to be entirely the same. Regardless, the evaluation survey participants in this study were anonymized by the program and data was blinded to the project investigators. Thus, cumulative scores used for comparison were derived from quarterly data involving a high number of respondents including new evaluators to offset this discrepancy and minimize selection bias. Finally, this educational improvement project was conducted in a US-based hospital and internal medicine residency training program, which may limit the applicability of our findings to a wider, global audience of hospital medicine and program directors.
CONCLUSIONS
In conclusion, the educational experience for a core rotation in internal medicine training may be improved by identifying common barriers perceived to threaten the learning climate, then developing and implementing interconnected solutions in culture, communication, and workload. Future directions for this core inpatient medicine rotation include enhancing the past solutions and incorporating new ones. Following this framework, graduate medical educators may use this model for system interventions to achieve a high-quality learning environment in resident clinical rotations.
Acknowledgement
The authors wish to acknowledge Mary Ann Kirkconnell Hall, MPH, for providing medical writing support.
Author Contributions
All authors have reviewed the final manuscript prior to submission. All the authors have contributed significantly to the manuscript, per the International Committee of Medical Journal Editors criteria for Authorship.
-
Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND
-
Drafting the work or revising it critically for important intellectual content; AND
-
Final approval of the version to be published; AND
-
Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Disclosures/Conflicts of Interest
The authors declare no conflicts of interests.
Corresponding Author:
Dustin Smith, MD
Associate Professor of Medicine,
Emory University School of Medicine,
100 Woodruff Circle, Atlanta 30322, Georgia, USA,
Telephone: 404-321-6111
Email: dtsmit2@emory.edu