The AAMI eXchange 2026 brings together biomedical engineers, clinical technologists, and health IT leaders to work through the real challenges of medical technology in practice.
Our team had the chance to present and connect with peers at this year’s event in Denver, Colorado, diving into one of the more underexamined corners of the AI-in-healthcare conversation: what happens when the data feeding your AI algorithms doesn’t actually reflect what’s happening with the patient.
Here are the biggest takeaways from our session and the conversations that followed.
Medical Devices Were Built for a Different World
Most medical devices, such as infusion pumps, physiologic monitors, and ventilators, were designed with the foundational assumption that a caregiver is in the room, looking at the patient first, then consulting the device data for context.
That assumption made sense. A nurse at the bedside sees the blood pressure cuff wedged under the wheel of the bed and immediately discounts the reading on the monitor. The device didn’t fail. The human just had context the device didn’t.
The problem is that the devices haven’t changed, but the workflow has. Today, that same device data is being routed off to EHR backends, AI data lakes, and clinical decision support algorithms — systems that have no visibility into what’s actually happening in the room.
Takeaway:
That gap between device data and clinical reality has real consequences. Closing it before deployment is exactly the kind of work that protects patients and makes AI investments worthwhile.
Good Data Can Still Give You Bad Advice
Not all bad AI outputs come from bad data. Sometimes the data is technically correct and still leads you in the wrong direction.
There are a few ways this plays out in practice:
- Transient signals: A patient with obstructive sleep apnea shows a pulse and SpO2 dip right before a big snore. The data is real. It’s also not actionable.
- Irrelevant alarms: Respiratory rate alarms firing on a ventilated patient, where the device simply isn’t measuring what matters.
- Mismatches with reality: An infusion pump logs a completed delivery, but fluid is still in the bag. Or a secondary infusion is documented as delivered when the bag wasn’t hung correctly and a mix of medications has gone in instead.
A clinician at the patient’s bedside would catch any of these as they would be able to see the bag, read the room, and apply judgement built from years of experience. That’s context no AI algorithm is capable of inheriting automatically.
Takeaway:
Device data that’s technically accurate can still be clinically misleading. Building AI that understands that difference is precisely where real value gets unlocked.
Alarm Fatigue Has Already Taught Us This Lesson
Alarm fatigue taught the industry a hard lesson. A system that generates constant alerts, accurate or not, trains clinicians to tune them out. The same risk applies to AI outputs that don’t hold up against clinical reality.
The good news is that the lessons are transferable, even if the specific implementations aren’t. What worked in one sepsis algorithm won’t port directly to another institution, but the practice of building it by involving clinicians, validating against real workflows, and continuously evaluating outputs absolutely does.
We also know that retraining alone isn’t the answer. How a nurse interacts with data at 3 a.m., on hour eleven of a shift, is different from how she interacts with it at the start of the next shift. Systems that only work when people are at their best aren’t safe systems.
Takeaway:
The industry learned hard lessons from alarm fatigue. AI adoption in clinical settings needs to inherit those lessons — not rediscover them.
The Fix Starts Before the Deploy
If you’re implementing an AI platform that incorporates medical device data, the work of making it trustworthy happens before it goes live.
That means:
- Involving practicing clinicians early to map bedside workflows and identify where device data and clinical reality diverge
- Finding the edge cases where automated data collection doesn’t reflect what’s actually happening with the patient
- Continuously evaluating model outputs to make sure the AI is producing results that make clinical sense instead of just statistically plausible ones
- Treating AI as one layer of protection, not a replacement for the human judgment that catches what algorithms miss
The goal isn’t to slow down AI adoption. There’s a real and growing need to reduce the burden on nurses and clinical staff, and technology has a meaningful role to play in that. It’s on us to make sure the systems we deploy are built on data we can actually trust.
Takeaway:
Validate device data against clinical workflow before you build on it. What the algorithm doesn’t know can hurt the patient.
Bridging the Gap
After years at the bedside and years in this industry, one thing is consistent: technology and clinical reality have to align and move together. That’s not a reason to slow down but a reason to look harder and build smarter.
The HTM community is uniquely positioned to do exactly that. Close enough to the devices to understand the data, but connected enough to clinical and IT teams to find where it breaks down and fix it. We’re grateful to AAMI for the platform to have that conversation, and to our co-presenters for bringing their clinical and research perspectives alongside ours.
At SEI, we help healthcare organizations ask those harder questions before deployment, not after, turning clinical insight into implementation strategy that actually holds up at the bedside. If that’s where you are, we’d love to be part of that work.
