Northwestern Medicine Delivers Timely and Quality Care with W&B
"The systems tab gives us great insight into how to efficiently allocate resources while also keeping costs low—if we don’t see above 95% GPU utilization, we know there is room to optimize, making our experiments run faster and helping control our costs."
Mozzi Etemadi
Anesthesiologist and Director of Advanced Technologies
Innovation in Healthcare
When it comes to implementing new technology, the healthcare industry is known to be slow in its progress. It’s a sector that traditionally favors legacy systems, and replacing them is often met with the ‘do more harm than good’ mindset.
Yet, the contrary is happening at Northwestern Medicine—an academic medical center that is the shared strategic vision of Northwestern Memorial HealthCare and Northwestern University. Among the 25,000 plus staff members, a small, multidisciplinary team led by Mozzi Etemadi, an anesthesiologist and Northwestern Medicine’s Director of Advanced Technologies, is exploring the use of natural language processing (NLP) to streamline follow-up processes and improve patient outcomes.
Delayed and Missed Care
Physicians often order dozens of imaging studies like computed tomography (CT) scans and magnetic resonance imaging (MRI) scans for their patients daily as follow-up recommendations. But with the busy nature of medical work, sometimes those recommendations fall through the cracks, especially if they require no immediate action. And to make matters more complicated, patient findings and recommendations are noted in long, extensive reports like this one:
This begs the question: how often are findings followed up on? In the literature, Mozzi and his team found that only about two-thirds of follow-up recommendations tend to be addressed, and unfortunately, that is usually a best-case scenario. Flaws like this can lead to dire consequences for a patient.
Here’s just one example: a physician recommends a patient to get a chest CT scan in twelve months and notes it in their report. If that recommendation goes unnoticed and it turns out the patient has cancer, the delayed diagnosis makes treatment less likely to succeed and reduces their chances of survival—a worst possible outcome.
To ensure follow-up recommendations are addressed, the team set out to create an NLP system to automatically identify radiographic findings requiring follow-up. Delivering efficient, timely, and quality patient care was top of mind for the team.
NM Results Management AI
While Mozzi and his team are a group of seasoned engineers, none of them had much experience with deep learning (DL) before. To build their first end-to-end DL project, they had to familiarize themselves with DL, and W&B became a crucial part of expediting their learning.
“When we started this project, none of us had ever built a DL model before. Using W&B greatly accelerated our education and helped keep our knowledge and experiments organized. We were able to track specific and granular information about our experiments and quickly validate if we were on the right track,” said Mozzi.
The intuitive visualizations of W&B and the ease of use became even more important as the team entered the model development phase. For most practitioners, their experience can usually help them answer questions like “will my model converge?” or “do we need to change the batch size?” But for engineers just getting started with DL, that is often not the case.
As the team decided on building a transformer-based model to better parallelize the training process and enable efficient training on large datasets, they leaned on W&B to develop the intuition of seasoned practitioners. “We’ve designed our workflows using W&B, and in turn, it’s helped us build the intuitions needed for understanding the influence of different parameters on the system,” explains Mozzi.
Epic Integration
Developing the model is just one piece of the puzzle, the next is integrating it in real-life. Epic is an electronic health records system commonly used in hospitals. The team’s following task was to find a way to integrate their model with Epic and to do that, they built a custom interface.
To see just how simple and efficient the new system is, here’s a quick walkthrough of how it works.
1) Within three minutes of completing a report, a physician gets a notification through Epic and can click “Place Follow-up Orders” to open the Results Management system.
2) Next, a chest CT is recommended for the patient in six to twelve months and the physician can either click “Accept” to automatically order the scan or they can select an acknowledgement reason to close the workflow loop.
3) In the six to twelve months time frame, the patient will get an electronic message and a letter in the mail reminding them they’re due for a chest CT.
During development of their AI system, W&B served as a single system of record to centralize all the information related to the project, while also instantly tracking the entire model development process—from raw data through the final model.
“W&B helps us stay organized and keep track of our models,” Mozzi said. “Our team isn’t going to use Excel or Post-it notes to experiment track when there’s a much more efficient way to do it.”
Another aspect of W&B that is particularly helpful for the team is tracking and visualizing system metrics. Having the ability to keep an eye on the entire system utilization helps identify training bottlenecks and avoids wasting expensive resources. “The systems tab gives us great insight into how to efficiently allocate resources while also keeping costs low—if we don’t see above 95% GPU utilization, we know there is room to optimize, making our experiments run faster and helping control our costs,” said Mozzi.
Since rolling out the Results Management workflow, it has generated nearly 5,000 interactions with ordering physicians and has tracked more than 2,400 follow-ups to completion. The system screens hundreds of radiology reports per day and identifies dozens of follow-up recommendations daily. It’s not an exaggeration to say this system really and truly saves lives.
“As an anesthesiologist, I often take care of patients undergoing cancer surgery—not a month goes by where I don’t have the privilege of helping take care of someone who had an abnormal finding flagged by our tool,” said Mozzi.
The implementation of such initiatives to use AI and resolve a problem in the clinical space highlights the biggest promise of AI in healthcare—changing clinical workflows and operational processes. In short, the system added value in two ways: automating the way work is done and augmenting the way work is done.
Transforming Healthcare
While healthcare may lag other industries in AI adoption, it may well be the ultimate proving ground for AI to truly demonstrate its worth across many dimensions—boosting productivity and responsiveness in a sector facing tremendous financial challenges, paperwork and regulations, and simultaneously improving outcomes in life-or-death situations.
As Northwestern Medicine continues to lead the way in realizing the full potential of AI in healthcare, having a flexible and intuitive MLOps platform is crucial. Leveraging W&B means the team can focus on experimentation and innovation while having the peace of mind that all their data, runs, trained models, and notes on model performance are preserved. Past that, W&B provides full visibility into their model’s resource utilization to significantly improve performance and allow for greater efficiency.
In a time where health systems are struggling with delivering care and facing financial burdens, Northwestern Medicine is driving the innovative wave with AI across the industry, and W&B is thrilled to support them to be at the forefront of this change.