Out of nearly 300 AI-enabled medical devices approved by the United States Food and Drug Administration (FDA), only a few have been evaluated through prospective randomized controlled trials (RCTs) This gap in real-world testing highlights concerns about AI's reliability and effectiveness, with issues like alert fatigue from faulty AI predictions as demonstrated by a sepsis model.