As England develops new rules for using AI in healthcare, Lucy Seymour, Understanding Patient Data's Policy and Communications Manager, spoke with researchers and AI specialists in the United States to understand what happens when these tools are introduced quickly, and what England can learn about safety, transparency and public trust.
Five lessons for getting AI-supported clinical tools right
England can still avoid healthcare AI’s ‘Wild West’. The US experience shows what happens when adoption outpaces oversight: rules fragment, responsibility blurs and gaps emerge in safety and accountability. England needs a joined-up system before deployment accelerates further.
Approval is only the start. AI can behave differently when it is updated or used in new hospitals, populations and care settings. Continuous monitoring, local validation and reassessment are essential to catch failures before they cause harm.
A technically impressive tool can still make care worse. Healthcare AI should be judged by whether it improves decisions and outcomes, works fairly for different patient groups and avoids generating tests or activity without clear benefit.
Someone must be accountable when AI causes harm. Patients and clinicians need to know who is responsible for a tool’s decisions, who can challenge them and what happens when something goes wrong. Shared involvement must not become shared evasion.
Public trust is an asset England could quickly lose. Greater confidence in UK regulation creates an opportunity, but trust will depend on visible safeguards, meaningful public involvement and honest explanations of when AI is used, how it affects care and who is accountable.
Introduction
AI is already transforming healthcare, helping to interpret scans, identify patients at risk, summarise records and support clinical decision-making. While these tools could improve care and reduce administrative burden, they also raise important questions about safety, effectiveness, accountability and transparency.
Not all AI tools present the same level of risk. A system that schedules appointments requires different oversight from one that influences diagnosis or treatment. Equally, some AI systems evolve over time through updates or use in new settings, meaning regulation cannot end at product approval but must also consider how tools are implemented, monitored and evaluated in practice.
The United States offers England a cautionary example. AI-supported clinical tools have been adopted there at greater scale, showing both the potential of rapid innovation and the risks of fragmented oversight. Against the backdrop of the MHRA’s National Commission into the Regulation of AI in Healthcare, England’s task is not simply to regulate individual products, but to create a coherent system for implementation, monitoring, accountability, transparency and public involvement across the NHS.
Five lessons run through the US experience: oversight must be joined up; approval must lead to continuous monitoring; technical performance must translate into fair patient benefit; accountability must be explicit; and public trust must be earned through meaningful transparency and involvement.
A cautionary tale from the United States
Researchers interviewed for this piece described parts of the US healthcare AI landscape as a ‘Wild West’. Their concern was not that regulation is entirely absent, but that existing rules leave gaps and oversight is fragmented. Responsibility is spread across federal agencies, state legislatures and local healthcare systems, making it unclear which tools are assessed before deployment, who monitors them afterwards, and who is accountable when problems arise. State-level action has become particularly important as federal policy shifts, but this can create a patchwork of rules, with some states pursuing broad AI laws and others adapting existing health, data or consumer-protection legislation.
Several organisations play important but distinct roles. The Food and Drug Administration (FDA) regulates certain AI-enabled medical devices and has begun adapting its approach for tools that change over time, including guidance on planned software updates and risk management. The Office of the National Coordinator for Health Information Technology (ONC) focuses on health information systems, while the National Institute of Standards and Technology (NIST) develops voluntary risk-management frameworks. No single body oversees the entire ecosystem of AI-supported clinical tools, and even within federal structures, responsibility can be split across different departments and categories of product.
This creates problems where tools fall between regulatory categories. AI used in insurance decisions, population-risk stratification, ambient documentation or patient communication may affect care but will not always be treated as a clinical medical device. And yet tools developed for administrative or efficiency purposes can still shape access to care, clinical workload and patient understanding.
This fragmentation becomes more challenging as adoption accelerates. Interviewees highlighted pressure to deploy tools before sufficient evidence of safety, effectiveness or real-world readiness is available. While the FDA has authorised more than 1,500 AI-enabled medical devices, regulators are still determining how best to oversee newer autonomous systems capable of tasks such as diagnosis, prescribing and care management.
This fragmented system offers England a clear warning. When responsibility is distributed across regulators, states and healthcare organisations, tools can fall between categories and accountability becomes harder to trace. England therefore needs a joined-up framework that covers all AI capable of influencing care, not only products formally classified as medical devices.
Approval should be the beginning, not the end
Approval is only the first step. AI-supported clinical tools need to be assessed after deployment, as performance can change when systems are updated or introduced into new clinical settings. This is sometimes described as model drift, where accuracy deteriorates over time.
The risks of inadequate monitoring are already becoming apparent. A recent study of ChatGPT Health’s triage recommendations found that, although overall performance was strong, the system sometimes recommended less urgent care when emergency assessment was needed. Such errors may be uncommon, but they carry disproportionately serious risks because they can delay critical treatment. The findings show why AI systems must be continually monitored and reassessed after deployment rather than judged solely on pre-market testing.
The need for ongoing oversight extends beyond tools formally deployed in healthcare settings. The boundary between clinical and consumer use is becoming less clear, with OpenAI reporting that more than 230 million people ask its chatbot questions about their health and wellbeing each week. Inappropriate guidance from general-purpose tools can still influence decisions about care, even when those tools fall outside conventional medical-device regulation.
Regulators are beginning to respond to the wider challenge of overseeing AI as it changes and is used in different contexts. The US FDA, Health Canada and the UK MHRA have begun developing joint approaches to support ongoing oversight after deployment. The MHRA Commission similarly argues that approval should be the start, rather than the end, of evaluation, with clear arrangements for monitoring performance, identifying emerging risks and responding when problems arise.
Local validation also matters. US interviewees noted that AI tools developed, tested or deployed in one setting may not perform as well in another because of differences in patient populations, care pathways, data quality and health-system infrastructure. For England, this means national approval alone cannot guarantee safe and effective performance across every NHS organisation.
This concern is not confined to the country where a tool is first developed. US-origin software and risk models can be sold or adapted internationally before there is a shared standard for assurance, local validation or post-deployment monitoring. For England, this underlines the importance of being explicit about what evidence is needed before tools are used in NHS settings, especially where they were developed for different health systems.
England therefore needs a common national approach to post-deployment oversight: clear monitoring requirements, local validation before and after implementation, routes for sharing evidence across the NHS and defined triggers for reassessment or withdrawal. Without this, approval risks becoming a one-off gateway rather than the start of continuous assurance.
AI should improve outcomes, not simply increase activity
A strong prediction does not automatically improve care. Healthcare AI should therefore be assessed not only on technical accuracy, but also on whether it improves clinical decisions, patient pathways and health outcomes in real-world settings.
The US healthcare system adds a further test: who benefits when AI changes clinical activity? Providers may earn more from additional tests or scans, creating incentives to do more. AI tools that recommend or enable further testing could reinforce those incentives even where the benefit to patients is uncertain.
The NHS does not share the same profit incentives, but it can still reward activity, throughput or rapid adoption without establishing whether patients benefit. The US experience therefore reinforces the need to judge AI by whether it improves clinical decisions and care, not simply whether it increases activity. The MHRA Commission reflects this principle by emphasising human oversight, clinical context and clear evidence of benefit rather than technical performance alone.
AI frameworks also need to test who benefits and who bears the risk. Systems trained on incomplete data or tested in narrow settings can reproduce or worsen existing inequalities, particularly for groups underrepresented in research or already facing barriers to care. Evidence of improved average performance is therefore not enough: developers and providers should show that benefits are shared fairly and that poorer outcomes for particular groups are identified and addressed.
The ONC’s framework for predictive decision-support interventions illustrates this broader test. It considers not only technical performance, but also whether risks are identified and managed and whether tools are fair, appropriate, effective and safe. This reflects the point made by US interviewees: healthcare AI should be judged by whether it improves care in practice and works fairly for different patient groups.
Who is accountable when AI influences care?
US interviewees highlighted uncertainty over accountability when AI is used in care. Responsibility is shared across developers, suppliers, healthcare organisations, clinicians and regulators, raising a fundamental question for patients: who is really making the decision, the clinician or the AI? Simply keeping a human "in the loop" is not enough if clinicians lack the information, time or authority to challenge an AI-generated recommendation.
Questions of responsibility become particularly acute when harms occur. For example, there have been at least 10 reported injuries linked to the AI-enabled TruDi Navigation System, including allegations of strokes and other serious surgical complications. While the role of AI remains disputed and adverse event reports do not establish causation, the resulting litigation highlights how difficult it can be to determine responsibility when software developers, healthcare providers and clinicians all contribute to care delivery.
Emerging models of care raise further challenges. Some US pilots involve AI counsellors providing therapy for depression and anxiety under the supervision of a single clinician overseeing large numbers of interactions. These approaches test traditional assumptions about professional accountability and where responsibility lies if a patient receives inappropriate advice or experiences harm.
Accountability is also shaped by the wider structure of the US healthcare system. Without a single national healthcare body, learning from problems may be uneven, and organisations can be hesitant to share evidence of success or failure in a competitive market. That makes it harder to build a shared understanding of what safe, effective and fair deployment looks like.
Together, these examples point to a practical requirement for England: every AI-supported clinical tool should have a named organisational owner, clear responsibility for monitoring and responding to harm, an independent route for escalating concerns and an accessible process through which patients and clinicians can challenge decisions. Shared governance must not allow responsibility to disappear between developers, providers, professionals and regulators.
Patients need meaningful transparency
Perhaps most pertinent to UPD, interviewees repeatedly argued that public trust cannot be secured through technical regulation alone. Patients often have limited awareness of when AI is being used, whether it is supporting administrative tasks, informing clinical decisions, or generating information they receive directly. Recent Pew Research Center polling reflects this challenge, as 46% of Americans were unsure whether AI had ever been used in their healthcare, while only 16% believed it had.
Although some organisations have introduced disclosure agreements, apps and educational materials, approaches remain inconsistent. The survey suggests patients want greater transparency even for relatively low-risk uses, as 72% said they should be informed when AI is used to take notes during appointments, while 64% supported disclosure for prescription refills and 56% for appointment scheduling.
The United States has begun to introduce transparency requirements. For example, ONC’s HTI-1 rule requires certain developers to provide information about how tools were developed, tested and intended to be used. Similarly, the US Department of Health and Human Services' (HHS) AI Strategic Plan emphasises transparency, public engagement and trust-building.
However, interviewees suggested a gap remains between policy and practice because transparency requirements often serve organisations and professionals rather than patients. Meaningful transparency should answer the questions people are most likely to ask: Is AI being used in my care? What is it doing and what information does it use? Can I question or refuse its use? Who is accountable if it gets something wrong?
Two recent US cases show how gaps in transparency can affect patients directly. A lawsuit against Sharp HealthCare alleges that an ambient voice technology (AVT) tool recorded patient-clinician conversations without appropriate consent and shared information with a third-party vendor. Separately, a Washington court ruled that recordings created by Nuance’s Dragon Ambient eXperience (DAX) were administrative tools rather than part of the medical record, meaning patients could not access them. Together, these cases highlight the importance of clear expectations around consent, access, retention and use of AI-generated recordings.
What England is doing differently
England is not starting from scratch. Pew Research Center polling found that 57% of people in the UK trusted their country to regulate AI effectively, compared with 44% in the United States. This is a useful foundation, but it is also fragile: confidence will depend on whether people see clear safeguards, meaningful public involvement and honest explanations of how AI affects their care. The MHRA Commission’s continued engagement with patients, the public and healthcare leaders is therefore a necessary part of regulation, not an optional addition.
That engagement is important because public confidence cannot be assumed. UPD’s Patient Data Sentiment Tracker consistently finds that awareness and support for using patient data to develop AI and digital tools remains relatively low. Support is lower for AI and digital tools than for research or service planning, and people are less comfortable with NHS partnerships involving technology companies than those involving pharmaceutical companies or government organisations.
Recent UPD media analysis points to a similar challenge. Across 1,329 UK media articles on health data published between March 2025 and June 2026, AI dominated coverage, with narratives divided between its potential benefits and concerns about oversight, surveillance and the role of major technology companies. Online discussions increasingly focused on sovereignty, resilience and control of data, suggesting that trust in AI-enabled healthcare is closely linked to wider questions of accountability and public benefit.
Research from Healthwatch England on AVT also highlights how finely balanced public opinion can be. Four in five respondents wanted to be told and asked for permission before an AI scribe was used, while comfort fell when appointments involved sensitive issues. Most said they would feel more comfortable if a healthcare professional reviewed AI-generated notes.
The Health Foundation has similarly argued that successful AI adoption depends not only on technology, but also on strong infrastructure, high-quality data, staff confidence and meaningful engagement with patients and communities.
Evidence from a recent UK-wide UPD study shows why engagement must influence decisions rather than simply demonstrate that people were consulted. Public involvement in health data is widespread, but its impact is often unclear, particularly outside research settings. Sustained engagement will build confidence in healthcare AI only if people can see how their views have shaped safeguards, deployment decisions and routes for accountability.
Where next?
As AI-supported clinical tools become more common across the NHS, the United States offers a clear warning: rapid innovation can bring benefits, but fragmented oversight leaves gaps in safety, accountability and public understanding. England should respond with a joined-up approach that combines proportionate national regulation with local validation and monitoring, clear responsibility for harms, meaningful public involvement and explicit evidence requirements for tools developed in other health systems.
Putting these elements in place early would give the NHS a firmer basis for deciding which tools to adopt, where further evidence is needed and when deployment should pause. It would also help ensure that innovation is shaped by clear public expectations, rather than asking patients to accept safeguards designed after the event.
Acknowledgements
With sincere thanks to colleagues in the United States who gave their time and expertise to inform UPD’s discussions on the adoption, governance and regulation of AI-supported clinical tools. Their insights, spanning academic medicine, clinical informatics, healthcare delivery and digital ethics, helped shape this article. Contributors have remained anonymous to support open discussion and to reflect that views were shared in a personal rather than organisational capacity.