Background
After studies demonstrated that research fosters intellectual curiosity, promotes career development, increases resident satisfaction, and contributes to the scholarly literature, the Accreditation Council for Graduate Medical Education (ACGME) incorporated research as a mandatory component of internal medicine training in the United States in 1994.1–3 However, internal medicine clinical training requirements are time-intensive,1 leaving residents with limited time to learn and master non-clinical skills. Despite widespread interest in promoting research and improving resident research skills, residency program directors still struggle to achieve this with every trainee in their program.4,5
Over the past several years, programs have implemented various interventions to increase resident research productivity.3,6–9 For example, at the University of California San Francisco, program directors restructured the PRIME (Primary Care Medicine Education) cohort into an individualized learning track focused on clinical-outcomes research with dedicated journal clubs, mentorship pairing, and protected research time.6 A cross-sectional analysis showed more research publications among PRIME graduates than among non-PRIME graduates.6 Similarly, Baystate Medical Center addressed research barriers by providing dedicated research time, administrative support during Institutional Review Board (IRB) applications, access to research assistants for data entry and biostatisticians for analysis, and a resident research director for mentorship.3 Afterward, resident publications increased substantially with subsequent improvements in fellowship match rates.
Several barriers to research during residency have been described – most notably time constraints – but the extent to which these barriers influence research productivity is less clear.2,4,5 Few studies investigate interventions aimed at enhancing resident research output.10–12 With these challenges in mind, program directors and researchers at Brown University implemented a mandatory research training program for all internal medicine interns to 1) assess the impact of the implementation of an intern research retreat on subsequent research productivity and 2) evaluate the factors that increase or decrease research productivity in residency.
Methods
Setting, intervention, and survey administration
The study included internal medicine residents at Brown University from 2015-2020, including those in the categorical, primary care/general internal medicine (GIM), and medicine/pediatrics programs. During their final year of residency, all internal medicine residents were required to submit an abstract of their choosing to present at an internal annual research day. As part of the abstract submission process, residents were asked to complete a research program evaluation to support ongoing quality improvement in the research program. The program provided no financial or direct incentive, although the research director strongly encouraged completion. This survey consisted of questions developed internally by the program leadership. Questions pertained to the type of research performed, the resident’s overall level of research interest, the magnitude of selected barriers to research in residency (lack of time, lack of interest, finding a mentor, the IRB process, lack of experience, and lack of statistical support), the number of prior publications and conference presentations, and the number of publications and conference presentations during residency. We used Likert scales to quantify responses where applicable.
We collected survey data for each graduating cohort from 2015-2020. In 2015, we recruited a new research director, who introduced a one-hour intern orientation on research basics and expectations. The study intervention began in 2016, when the orientation was expanded into an ‘intern training retreat.’ Every intern was excused from clinical responsibilities for half a day in order to attend. Expert biostatisticians and researchers taught the four-hour curriculum, which included a review of research methodology, an introduction to the IRB, information on accessing biostatistics support, an overview of a data collection and storage system (REDCap), and a lecture on professionalism and ethics. At the same time, all interns completed a Collaborative Institution Training Initiative (CITI) course on protecting human subjects in research studies, a prerequisite for conducting research at our institution. We evaluated survey responses from six senior resident classes graduating from 2015-2020.
Statistics
We divided the study population into two cohorts: cohort 1, senior residents from 2015-2017, who did not experience the study intervention, and cohort 2, senior residents from 2018-2020, who did undergo the intervention. Descriptive statistics were calculated for study participant characteristics overall and by cohort, including: program type (categorical, GIM, medicine/pediatrics, other), overall interest in conducting research during residency (0-4; very low, low, moderate, high, very high), self-reported research barriers (categorized as binary, moderate or serious vs minor or none), prior research experience (case report, review article, original clinical research, basic research), peer-reviewed publication record (prior vs residency, for total publications and publications with the resident in the first- or last-author position), and conference presentations (prior vs residency, for local and national conferences). Continuous variables were presented as means and standard deviations and compared between cohorts using t-tests. We presented categorical variables as counts and percentages and compared them between cohorts using chi-squared tests.
To evaluate which resident characteristics were associated with research productivity, we categorized total peer-reviewed publications and national conference presentations produced during residency into binary variables (at least one residency publication vs none; at least one national presentation vs none), as most counts were 0 or 1. We then used unadjusted logistic regression to estimate the odds of each binary outcome for each resident characteristic described above. To account for the large number of comparisons, we set a more conservative alpha threshold of 0.01. Data were collected using REDCap13 and analyzed using SAS Statistical Software 9.4 (SAS Institute Inc., Cary, North Carolina, U.S.). The Brown University Health IRB reviewed and deemed this study exempt.
Results
Resident baseline characteristics
A total of 260 residents in their final year participated during the study period from 2015-2020, with 130 in Cohort 1 (2015-2017) and 130 in Cohort 2 (2018-2020) (Table 1). 100% of eligible participants responded to the survey. We did not observe significant differences between cohorts in the types of research experiences residents had participated in before residency or in residents’ level of research interest. We observed no significant differences between cohorts in the mean number of pre-residency publications, either overall or with the resident in the first- or last-author position, or in the mean number of pre-residency presentations at local conferences. However, we did see a significant difference in the mean number (per resident) of pre-residency national conference presentations (0.5 in Cohort 1 vs 1.2 in Cohort 2, p=0.007).
Barriers to research
Forty-seven percent of residents expressed ‘moderate’ interest in research during residency, and 28% reported ‘high’ or ‘very high’ levels of interest. The largest proportion of residents (83%) identified lack of time as a ‘moderate’ or ‘serious’ barrier to research, followed by lack of statistical assistance (53%), difficulty finding a mentor (50%), and the IRB process (50%). A smaller proportion of residents cited lack of experience (41%) and disinterest (27%). The proportion of residents reporting each barrier did not differ between the two cohorts except for lack of experience (48% in Cohort 1 vs 34% in Cohort 2, p=0.02) and lack of statistical assistance (60% in Cohort 1 vs 46% in Cohort 2, p=0.02), although these differences did not meet our significance threshold.
Residency productivity
We found significant differences between cohorts in the mean number of total publications (0.4 in Cohort 1 vs 1.5 in Cohort 2, p<0.001), first/last author publications (0.3 in Cohort 1 vs 0.7 in Cohort 2, p=0.0016), and national conference presentations (0.3 in Cohort 1 vs 1.1 in Cohort 2, p<0.001) during residency, but not for local conference presentations (1.0 in Cohort 1 vs 1.1 in Cohort 2, p=0.85) (Table 1). In unadjusted logistic regression analysis, several resident characteristics were associated with increased odds of a residency publication or national presentation (Table 2). Training in the categorical vs GIM program track, increasing levels of research interest, and having published or presented a prior case report or review were all associated with higher odds of producing peer-reviewed publications and national conference presentations. Prior clinical research experience was the strongest predictor of both a publication (OR 5.3, CI 2.9-9.5, p<0.001) and a national presentation (OR 4.6, CI 2.5-8.4, p<0.001) during residency. The number of pre-residency publications was not significantly associated with publications during residency (OR 1.1, CI 1.0-1.2, p=0.017) or with national presentations (OR 1.0, CI 0.9-1.1, p=0.46). Among the barriers, only lack of interest in research (OR 0.3, CI 0.1-0.5, p=0.0002) and lack of experience (OR 0.5, CI 0.3-0.8, p=0.008) were significantly associated with decreased odds of a publication during residency. No barriers to research were associated with decreased odds of a national presentation. Figure 1 presents the total number of publications and national presentations during residency by year. After implementing the intern research retreat in 2016, peer-reviewed publications during residency increased from 57 in Cohort 1 to 199 in Cohort 2; national conference presentations increased from 40 to 140.
Discussion
We found that a cohort of residents who participated in an intern research retreat intervention had greater research productivity during residency compared to the cohort that trained prior to implementation of the research retreat. In addition, this study is among the first to examine the association of individual research barriers with research output during internal medicine residency. Lack of time was the most prevalent barrier, followed by lack of statistical support, difficulty finding a research mentor, and challenges with the IRB process. Lack of interest and lack of experience were associated with decreased odds of publication (but not national presentations). Prior participation in clinical research and interest in research during residency were both more meaningful than total number of prior publications in predicting publications and national presentations during residency.
In our cohort, categorical residents were more likely to publish than GIM residents. This may reflect the research productivity of residents applying for competitive fellowships, as GIM residents may be less likely to pursue fellowship training. However, the GIM track does not preclude residents from engagement in research, and this may be institution-specific. Overall research interest was a stronger predictor of future productivity than the number of prior publications or presentations, highlighting motivation as a key factor. In interpreting the significance of prior research, while the total number of prior publicatiosns was not significantly associated with resident research productivity in our study, our data suggest that programs seeking to recruit residents likely to be productive during training might identify applicants with at least one clinically oriented research experience. This information may help individualize research training initiatives more likely to be effective in certain subgroups. For example, those with at least one clinically-oriented research experience and high interest in research might be better suited to an advanced research training session with information about accessing support services and networking with high-output researchers to initiate projects that may span the entirety of the resident’s length of training. Alternatively, those without prior clinically-oriented research or without significant research interest may benefit from a more basic introductory course to research with less emphasis on long-term mentorship (perhaps even with a senior resident) to ensure they quickly acquire the skills to produce a clinically-oriented presentation early in their training so as to not lose much time should they realize they would like to participate in more time-intensive projects.
We found that basic science research experience was not associated with either publications or presentations during residency. This finding might be related to the differing skillsets of basic science research and the clinical research typically done in residency. However, this distinction may not predict which residents will pursue research-oriented careers, as most formalized pathways for physicians to pursue research careers focus on training basic or translational scientists.14
The proportions of residents reporting common barriers to research productivity were similar to those in a prior study among presenters at an American College of Physicians (ACP) meeting.2 However, finding research mentors was a greater barrier among participants in our study (50% vs 25%), possibly due to differences in sampling, as ACP presenters would more likely have already identified a mentor.15 Most of the barriers to research that we identified were negatively associated with residency publications, but not with national presentations. This may be because producing a product competitive enough for publication requires greater commitment and research expertise than presenting at national conferences, which often support smaller projects with preliminary data. Additionally, our program typically provides financial support to first-author presenters at conferences to encourage attendance at national conferences. Interestingly, lack of statistical support was a commonly cited barrier that improved between the first and second cohorts, but it was not negatively associated with research productivity. The availability and promotion of statistical assistance near the beginning of the study period may have influenced this outcome. Statistical competence is generally low among resident physicians, and adding statistical support may encourage residents to design better studies from the start or help analyze raw data.16
The main limitation of the study was the observational design, which prevents causal attribution of the observed increase in research productivity to the intern research retreat. Smaller changes in the residency program over the study period may have confounded the observed relationship with the intern research day, including social networking events, resident-led workshops, and the appointment of a new research director. In addition, cohort 2 entered residency with greater prior national conference presentation experience, which was itself associated with subsequent productivity and therefore represents a potential confounder in the unadjusted cohort comparison. Several additional differences between cohorts did not meet the prespecified p<0.01 threshold but may still be relevant. The later cohort had greater prior publication experience and less frequently reported lack of experience, lack of statistical support, and difficulty identifying a mentor. Such factors may also contribute to differences in research productivity. Broader trends in graduate medical education and scholarly activity, or potential cohort-specific variations in residents over time, may have affected the findings. Finally, the intervention was implemented at a single academic institution, which limits generalizability given that residency programs may differ substantially in resident characteristics, research culture, mentorship availability, and institutional resources.
Implementing a research day is not particularly resource- or time-intensive and could apply to other programs. Nevertheless, residents’ baseline characteristics and the availability of research mentors may differ between programs, and some programs may benefit more from such an intervention than others. This study’s strengths were its ability to capture residents’ research productivity longitudinally over six years and its 100% response rate. This allowed us to evaluate the program’s impact over time while minimizing the nonresponse bias common in survey studies; however, the survey administration context and strong encouragement from program leadership may have influenced responses. Although residents changed yearly, baseline characteristics across cohorts were similar. Additionally, the relatively large sample for this type of study allowed for an adequately powered analysis of specific predictors of resident productivity.
Conclusion
Implementing an early research-training program may improve resident research productivity, although the observational design and concurrent appointment of a research director limit attribution of these changes specifically to the research retreat. These findings support the idea that relatively simple research-focused initiatives may be feasible for residency programs. Additionally, identifying common barriers to resident research may help program directors target areas for support, improve research productivity, and guide program development.
Funding
Matthew Murphy: NIH Grant K23DA054003
Disclosures/Conflicts of Interest
None
Corresponding Author
Michael A. Liu, MD, MPH,
Assistant Professor of Medicine
Northwestern University Feinberg School of Medicine
Robert H. Lurie Comprehensive Cancer Center
675 N St Clair St, Ste 20-150 | Chicago, IL 60611
Email: michael.liu1@northwestern.edu