All GME-accredited residency programs, not just vascular surgery, use milestones to measure progress and provide that data to the ACGME. Also included in the ACGME data are residents’ case logs, reflecting their operative experience during training, and program features such as faculty-to-resident ratio. Linking data from two organizations that normally do not interface with one another can provide interesting insights, Smith said.
“In prior research, we found that patients of surgeons who, as residents, were rated lower on their milestones have an increased risk of post-operative complications,” Smith said. “By linking milestones data from when surgeons were in training and patient outcomes data now that they’re in practice, we’re able to investigate which educational experiences are associated with patient outcomes.”
In the second phase of the project, Smith’s team will provide programs with a report that shows correlations between the performance assessments of the programs’ trainees, and those who have completed the program and started their careers. Once they see these links, including how their graduates perform relative to the graduates of other programs, Smith explained, programs can take steps to tweak the training, and assist trainees to ensure they are ready for independent practice.
“Most programs consider the ACGME milestones a relatively low-stakes performance measure, not really predictive of how someone is going to perform as a doctor,” Smith said. “But now we have evidence that milestones actually matter a lot. Programs using this predictive, data-based model may be more likely to encourage a learner to spend more time developing their competence in certain areas, because the model shows the learner is more likely to have patient care complications if they don’t.”
One area that Smith and her team are focusing on is medical optimization, or the process of ensuring a patient is in the best possible condition before surgery. For example, as best practice, vascular surgeons should give their pre-operative patients aspirin and, if they plan to operate on the carotid artery, a statin. Smith and her team wanted to look at variations in how and whether programs taught this step.
“The data showed that many programs’ graduates are consistently prescribing the correct medications for their patients before surgery, but some are not. There is enough variation in practice, based on where a surgeon trained, to justify a report that encourages programs to improve their training in the future.”
The goal, she said, is to train physicians who are better prepared to handle the rigors of a challenging and demanding specialty.
The grant is one of 11 precision education grants awarded by the AMA this year. Precision education, prioritized through the AMA ChangeMedED initiative, leverages data and technology to increase personalization, efficiency and agency for learners. Out of the 11, Smith’s project is one of two that are focused on improvement on a programmatic level, rather than the level of the individual learner.
Vascular surgeons leading the project at UW have access to the resources needed to create a proof-of-concept predictive model, including data processing frameworks, called data engines, and the analytics expertise provided by the biomedical informatics team at the Institute for Clinical and Translational Research, led by Jomol Mathew, PhD, associate dean for informatics and information technology and associate professor in the Department of Population Health Sciences. Collaborating partners include Johns Hopkins University School of Medicine, University of Florida, and the ACGME.
While the first model will be focused on vascular surgery, Smith and her team are designing it to operate at scale.
“We want everyone, from the largest urban medical center to the smallest rural program, to be able to receive reports about how their training assessment tools are working and how their graduates are performing,” she said.
