Competency-based medical education (CBME) can positively impact and improve patient care through better procedural skills training, noted speakers presenting at the Lois Margaret Nora Endowed Lecture – Competency-Based Medical Education: The Future for Specialty Training – at ABMS Conference 2022.
“Being present for a case is not the same thing as being competent for that case,” stated Brenessa Lindeman, MD, MEHP, FACS, Associate Designated Institutional Official for the Clinical Learning Environment and Program Director of the Endocrine Surgery Fellowship at the University of Alabama-Birmingham. Becoming a surgeon currently requires a five-year pathway of clinical training, which has served as a good educational framework with one exception. “It does not recognize that individuals develop knowledge, skills, and attitudes at different rates,” said Dr. Lindeman, who is also Assistant Dean of Graduate Medical Education. This fixed-time system has led to variable outcomes in training, but CBME can serve as the remedy.
Dr. Lindeman provided a brief history of CBME, reviewing the integration of the Accreditation Council for Graduate Medical Education (ACGME)/American Board of Medical Specialties (ABMS) six core competencies and more recently, the ACGME milestones into residency programs. The milestones are defined as observable markers of a trainee’s ability along a developmental continuum, and by their very nature, provide a retrospective look back. “But as a faculty member in a residency program, that’s not what I’m thinking about when I’m interacting with a trainee day to day,” she said. Dr. Lindeman is thinking about how much she can trust a resident to perform surgery if she isn’t in the operating room (OR). The challenge is on what to base entrustment decisions. Are they based purely on information Dr. Lindeman knows about the resident, the resident’s experience, or something she has been told by other faculty?
That’s where the entrustable professional activities (EPAs) come into play. EPAs are essential tasks that a trainee in a specific discipline can be trusted to perform independently in a given context, said Dr. Lindeman, who is also an ABMS Visiting Scholar alumna. They must be both observable and measurable. More importantly, they shift the assessment focus away from competencies alone. “We talk about the milestones as very granular descriptive behaviors of how someone develops a competency or skill that they possess, whereas EPAs are the larger view of how a person is able to put those things together to deliver care for a patient,” she explained.
In 2015, pediatrics became the first specialty to publish EPAs. Shortly thereafter, the American Board of Surgery (ABS) identified five EPAs for general surgery that served as the foundation for a pilot study conducted between 2018 and 2020.
The purpose of the pilot was to determine the relationship of EPA assessments to summative entrustment and identify the successes and barriers to EPA implementation across general surgery programs. Twenty-eight residency programs participated; mostly academic medical centers associated with medical schools. Each program was assigned two EPAs to assess and were asked to innovate with the implementation. Programs were supported with toolkits and ongoing virtual seminars.
More than 800 residents were assessed at any given time, about 10 percent of the population of surgical residents. Participation was highly variable, she said. A couple programs never collected micro-assessments; some started and then stopped, possibly due to the COVID-19 pandemic; and some programs maintained their momentum throughout the entire pilot. Still, numerous micro-assessment data points were collected. Most were in the intra-operative phase of care likely because it’s a natural opportunity for direct observation, Dr. Lindeman said. The data demonstrated that trainees were given increasing entrustment as they progressed to the next level of training.
The pilot also showed that Clinical Competency Committees (CCCs) required between seven and 10 micro-assessments per resident for each EPA to provide a summative entrustment rating. “That was an early signal about how much data would be needed to make these decisions,” she said.
Meanwhile, ABS has contracted with a vendor to provide a secure, interactive platform and data repository to facilitate the implementation of EPAs at the more than 350 U.S. general surgical residencies certified by ACGME as the board transitions to EPAs for general surgery resident evaluation, beginning in July 2023.
Implementing ABS EPA Pilot
Jacob A. Greenberg, MD, EdM, Associate Professor of Surgery and Program Director at the Duke Community General Surgery Residency, and Director of the Duke Surgery Hernia Center, discussed his experience implementing the ABS EPA pilot when he served as the residency program director at the University of Wisconsin.
The primary outcome of the pilot was the entrustment decisions made by the CCC across two EPAs for all surgical residents. The secondary outcomes were to determine the number of micro-assessments used to make those decisions and how the performance of EPAs correlated to that of the milestones. Being allowed to innovate when implementing the EPAs is what Dr. Greenberg called “the secret sauce of this pilot.”
The implementation strategy was three-fold. First, an assessment tool that was easy to use and access had to be created, he said. The Information Technology department staff developed a smartphone-based app, which automatically puts in the date and time of the case. The faculty member then selected the resident to be assessed, the EPA being used, and the level of entrustment. Each level included the descriptors of the behaviors witnessed, so faculty could review the list, Dr. Greenberg noted. The app allowed either the faculty or resident to initiate the assessment. Over time, nightly notifications and then monthly reminders were added to help promote engagement. Despite best efforts, there was a steep drop-off in March 2020 due to the COVID-19 pandemic.
Second, faculty and resident development was essential. Prior to its launch, the pilot took center stage at the annual EPA educational retreat. Attendees were given the EPAs along with vignettes and clinical behaviors, which served as a mental model and helped create expectations for all parties. For the first two weeks of the pilot years, the program held educational sessions for the residents. “This was a great way for the residents to understand the progression of the EPAs and also get to know other residents,” he said. The same was done with faculty. “I think it helped create camaraderie and a shared sense of what we were trying to accomplish,” Dr. Greenberg added. He reached out to faculty in emergency medicine and hospital medicine, who also had numerous opportunities to observe the surgical residents, to ask if they would assess the residents, as well.
Lastly, the app made it easy to integrate the EPAs into the CCC process because it automatically funneled all the assessments into a database, allowing faculty to track a resident’s performance over time for each EPA. Faculty now had numerous micro-assessments on which to base an entrustment decision, Dr. Greenberg said, rather than having observed the resident in similar circumstances or relying on what other faculty have said. To test how accurate these entrustment decisions were, faculty made “blinded” decisions. Every six months, Dr. Greenberg compiled a list of the entrustment decisions made for each resident that he emailed to all faculty to help them better gauge the residents’ abilities. He also added the list to the app, enabling residents to show the attendings in the OR their ability scores.
Lessons learned? Change takes time. It’s unfair to expect people to immediately embrace a new process and get it right, Dr. Greenberg said. “But with time and reminders, people got better. Overall, the faculty response rate improved, and the number of micro-assessments increased from Year 1 to Year 2,” he added. These assessments were meaningful for residents because they helped them to think reflectively and do accurate self-assessments. When comparing resident assessments to faculty assessments, he found that the residents tended to entrust themselves at a lower level than did faculty. “And that’s exactly what I want to see,” Dr. Greenberg said, adding, “I want residents who are competent but not overly confident.” While the surgery faculty tended to evaluate the residents’ decision-making, judgment, and technical skills in the OR, the emergency medicine physicians and hospitalists focused more on the residents’ communication skills, timeliness, and overall professional behavior. “Capturing these different areas of performance provided a better picture of the residents’ overall performance,” he said. Most importantly, the pilot demonstrated increased autonomy for the residents.
Implementing CBME in a general surgery residency is not necessarily easy, but it is feasible and offers a wide range of benefits for faculty and trainees, Dr. Greenberg said. “I do think it will lead to earlier autonomy and residents who are better prepared when they go out into practice,” he concluded.
Applying CBME Principles Across the Continuum
Patricia L. Turner, MD, MBA, FACS, Executive Director and Chief Executive Officer of the American College of Surgeons (ACS), spoke about the importance of understanding how CBME impacts practicing surgeons despite its primary focus on trainees. “We have to make significant efforts to engage faculty, attendings, and practicing surgeons in CBME because it would be unfortunate to do this in residency and not have it continue in the lifelong process of learning,” Dr. Turner said. Additionally, programs and systems that promote CBME must work for all stakeholders not just the highly resourced institutions and academic programs with large resident classes.
While ACS is focused on determining how the principles of CBME can be applied across a surgeon’s career, it has partnered with all the surgical specialty boards, including ABS, to move it forward in residency programs. Providing the appropriate education and training will help surgery remain a self-regulated specialty, she said, acknowledging that there are many entities that would gladly step into that role.
Dr. Turner highlighted the role of technology for implementing CBME across the continuum. It must be relevant and accurate, as well as provide rapid, if not real-time, assessment. It’s essential to have self-directed learning that is accessible and easy to use. The reality is there are essentially five generations that are used to five different cadences of feedback working together in hospitals and training programs, she explained. “For the most junior among us, immediate feedback like the app that Dr. Greenberg described is the only meaningful feedback,” she added.
It’s important to recognize that some indicators are measurable but not meaningful, Dr. Turner noted. Remaining focused on meaningful metrics that impact surgical patient outcomes is critical. ACS has historically used competency-based verification models that move from verifying the surgical knowledge and skills of individuals who attend its courses through satisfactory patient outcomes at the highest level, mirroring the principles of EPAs. Many of these programs, which are used at the undergraduate and graduate medical levels, have been in place for years. All of them are associated with measurable outcomes.
One of the newer programs is the ACS Objective Assessment of Skills in Surgery (OASIS), which is similar to EPAs in that it focuses on both residents and faculty members with an element of self-assessment, she said. This simulation-based, multi-station format program is aimed at rigorously assessing junior residents’ technical skills. Preliminary pilot results were recently presented at the Association of Program Directors in Surgery meeting. OASIS is scheduled to launch in 2023 around the same time the EPAs will be introduced.
At the ACS Clinical Congress in 2019, the society launched the Surgical Metrics Project, which is designed to assess senior-level physicians’ ability to execute specific surgical tasks. During the 25-minute assessment, participants completed a demographic survey and were fitted with EEG and motion tracking sensors and had audio recorders placed in their lab coats. In a simulated OR environment, participants were asked to “run the bowel” and perform open repair of any small bowel enterotomies identified in a segment of the porcine intestines. A total of 255 surgeons participated, most of which were practicing surgeons. The data found that surgeons who repaired the two injuries together, repaired them in a double layer, and placed a corner stitch had significantly lower leak rates than those who repaired the injuries separately, used a single layer, and did not place a corner stitch. The Surgical Metrics Project returned to the ACS Clinical Congress this year, featuring a laparoscopic ventral hernia repair, which takes between one-and-a-half and two hours to complete. Each of the 120 surgeon participants received a snapshot report card that showed how their results compared with other surgeons’ results, including the quality of the repair.
Although these assessments involve one procedure and a relatively small number of surgeons, Dr. Turner concluded, the purpose in sharing the data is to determine what decisions surgeons make when faced with a surgical task and how do those decisions and technical approaches impact outcomes and quality. The data could potentially be used as a benchmarking resource for residents, or best practices or an expert strategy for practicing surgeons.
ABMS President and Chief Executive Officer Richard E. Hawkins, MD, stated that ABMS is committed to partnering with other stakeholders to implement CBME. “We believe it is the best way, if not the only way, to assure the public of the competence of those who own our certificate,” said Dr. Hawkins, who served as moderator.
