Vascular surgeons are known as “the surgeon’s surgeon.” They lead or assist with health care’s most complex surgical cases, helping to manage bleeding and repair major blood vessels. They also treat a wide range of life and limb-threatening chronic diseases, including arterial atherosclerosis affecting the brain and legs, as well as aortic aneurysms.
To conduct this delicate work, a vascular surgeon needs excellent training. Across the country, however, training programs vary, as does the performance of their graduates. This affects patient outcomes, said Dr. Brigitte Smith, associate professor of surgery and the vice chair of education for the Department of Surgery in the University of Wisconsin School of Medicine and Public Health.
In January, Smith received a $1.1 million precision education grant from the American Medical Association to build a platform that will help vascular surgery training programs — and eventually, any graduate medical education program, nationwide — analyze residents’ performance data alongside performance data of recent graduates now working in the field. The tool will help programs identify areas of weakness and strength when it comes to preparing residents for practice.
Better training means fewer complications and more lives saved. But until now, programs haven’t had a reliable method for evaluating the effectiveness of their training.
“In the past, medical training programs have tried to improve by collecting feedback from their learners and faculty, but what they get includes everything from ‘I really hated this rotation,’ to ‘I wish there were more snacks in the call room,’” Smith said. “Evaluations have been satisfaction-driven, as opposed to being driven by program results. The platform we’re creating draws from robust, objective data to help programs identify the factors that impact graduates’ performance and modify those factors to enhance graduates’ readiness.”
To achieve this predictive analytic model, Smith and her team have linked two datasets: patient outcomes data from the Vascular Quality Initiative (VQI) registry, and training and learner data from the Accreditation Council for Graduate Medical Education (ACGME). The patient outcomes data includes the surgeons who performed each procedure. The ACGME data contains residents’ milestones ratings, which show how well they are doing at critical phases of their training.
All GME-accredited residency programs, not just vascular surgery, use milestones to measure progress and provide that data to the ACGME. Also included in the ACGME data are residents’ case logs, reflecting their operative experience during training, and program features such as faculty-to-resident ratio. Linking data from two organizations that normally do not interface with one another can provide interesting insights, Smith said.
“In prior research, we found that patients of surgeons who, as residents, were rated lower on their milestones have an increased risk of post-operative complications,” Smith said. “By linking milestones data from when surgeons were in training and patient outcomes data now that they’re in practice, we’re able to investigate which educational experiences are associated with patient outcomes.”
In the second phase of the project, Smith’s team will provide programs with a report that shows correlations between the performance assessments of the programs’ trainees, and those who have completed the program and started their careers. Once they see these links, including how their graduates perform relative to the graduates of other programs, Smith explained, programs can take steps to tweak the training, and assist trainees to ensure they are ready for independent practice.
“Most programs consider the ACGME milestones a relatively low-stakes performance measure, not really predictive of how someone is going to perform as a doctor,” Smith said. “But now we have evidence that milestones actually matter a lot. Programs using this predictive, data-based model may be more likely to encourage a learner to spend more time developing their competence in certain areas, because the model shows the learner is more likely to have patient care complications if they don’t.”
One area that Smith and her team are focusing on is medical optimization, or the process of ensuring a patient is in the best possible condition before surgery. For example, as best practice, vascular surgeons should give their pre-operative patients aspirin and, if they plan to operate on the carotid artery, a statin. Smith and her team wanted to look at variations in how and whether programs taught this step.
“The data showed that many programs’ graduates are consistently prescribing the correct medications for their patients before surgery, but some are not. There is enough variation in practice, based on where a surgeon trained, to justify a report that encourages programs to improve their training in the future.”
The goal, she said, is to train physicians who are better prepared to handle the rigors of a challenging and demanding specialty.
The grant is one of 11 precision education grants awarded by the AMA this year. Precision education, prioritized through the AMA ChangeMedED initiative, leverages data and technology to increase personalization, efficiency and agency for learners. Out of the 11, Smith’s project is one of two that are focused on improvement on a programmatic level, rather than the level of the individual learner.
Vascular surgeons leading the project at UW have access to the resources needed to create a proof-of-concept predictive model, including data processing frameworks, called data engines, and the analytics expertise provided by the biomedical informatics team at the Institute for Clinical and Translational Research, led by Jomol Mathew, PhD, associate dean for informatics and information technology and associate professor in the Department of Population Health Sciences. Collaborating partners include Johns Hopkins University School of Medicine, University of Florida, and the ACGME.
While the first model will be focused on vascular surgery, Smith and her team are designing it to operate at scale.
“We want everyone, from the largest urban medical center to the smallest rural program, to be able to receive reports about how their training assessment tools are working and how their graduates are performing,” she said.
