New AO-funded study lays foundation for standardization of intraoperative teaching and assessment

Responding to a paradigm shift from an apprenticeship-based surgical training model to proficiency-based progression in training, the AO—long the gold standard in surgical education—is again breaking new ground by developing and validating discrete procedural metrics that establish a foundation for standardization of intraoperative teaching and assessment. The proficiency-based approach requires trainees to demonstrate an objective benchmark established through a comprehensive assessment process based on performance metrics.

In an AO-funded study published in June 2022 in Injury, a research team including surgeons and scientists from AO Trauma, the AO Education Institute (AO EI), and the AO Innovation Translation Center (AO ITC) sought to determine whether the performance metrics could distinguish between low-performing and high-performing surgeons (eg, novices and expert surgeons).

The study is significant for several reasons, according to coauthor and AO Trauma Milestone Taskforce member Matt Graves.

“By developing and validating discrete procedural metrics, intraoperative teaching and assessment can be standardized. This should ensure that more objective criteria can be used for proficiency evaluations,” Graves said.

The study is the first phase of the IMMPACT (Improved Morbidity and Mortality for surgical fixation with intramedullary nailing of unstable pertrochanteric fractures resulting from Procedure charACterization and proficiency-based progression simulation Training) project initiated by the AO EI in collaboration with the AO ITC's Clinical Evidence team. It also is the first AO study in which real procedures from five different hospitals were filmed and assessed.

The key role of metrics
Metrics usually are developed by breaking down the key steps of a surgical procedure and defining the units of performance that constitute the skill. These metrics also delineate deviations for optimization performance (ie, errors) and are used to objectively assess surgeon performance.

“We first developed metrics to objectively assess the performance of an unstable pertrochanteric AO/OTA 31A2 intramedullary nailing procedure,” said coauthor and past AO Trauma International Board Chairperson Kodi E Kojima. Those metrics, which were approved by an international Delphi panel of 32 surgeons from 18 countries, are described in Kojima K et al., Injury, 2018 and provide unambiguous operational definitions of 15 phases with 75 steps, 88 errors, and 28 sentinel errors.

Coauthor and chair-elect AO Trauma International Board Chairperson Wa'el Taha described the current study, which was carried out in hospital operating rooms in Belgium, Germany, Hungary, Ireland, Israel and South Korea. “We validated the previously developed metrics by establishing that these metrics are capable of distinguishing between low-performing and high-performing surgeons and can be reliably scored by different assessors,” he said, noting that this was done by filming several real surgical procedures in the operating rooms of five hospitals. Taha said the main challenges in implementing the study protocol and the researchers’ solutions are described in Kojima K et al., 2021 to inform and guide future metrics validation studies.

Key contributors
Kojima underscored that a large, multidisciplinary team made key contributions to the current study.

“We thank the patients, principal investigators, surgeons, operating room personnel, and study coordinators of the five participating study sites, as well as AO Trauma International Board Chairperson Michael Baumgaertner, the 32 AO surgeons who were part of the Delphi study, the AO EI video team, and the AO Clinical Operations and AO Clinical Science teams for their key contributions to the study,” Kojima said.

In addition to Kojima, Graves, and Taha, coauthors of the current study included AO EI Continuing Medical Education Accreditation (CME) & Educational Research Senior Project Manager Monica Ghidinelli, AO Trauma faculty members Bernard Struelens and Jorge Alberto Amaya Aliaga, AO EI Curriculum Development Manager Mike Cunningham, AO ITC Clinical Science Head Alexander Joeris, and ORSI Academy Research & Skills Development Director Anthony G Gallagher.