A central safeguard and common principle in the use of artificial intelligence in healthcare is that whilst the artificial intelligence systems may assist in assessment, note keeping and general patient overview, the responsibility remains with the practitioner to make final decisions on diagnosis and treatment.
A recent study however suggests that this approach may be less reliable than it appears. Researchers writing in PLOS Digital Health conducted two experiments in which physicians treated fictitious patients classified by an AI system as either highly or minimally responsive to a particular treatment. The classifications were deliberately incorrect.
The doctors received information about patient outcomes as the experiments progressed, allowing them to see that the AI classifications were unreliable. Despite this, they generally continued to follow the system’s guidance. In one experiment, they also failed to recognise that the treatment was ineffective for all patients.
The experiments involved simplified scenarios and cannot establish how doctors will behave in every clinical setting. They nevertheless expose an interesting weakness in one of the key principal safeguards governing medical AI, being human oversight.
Human oversight only protects the patient when the practitioner is genuinely exercising independent judgement. These studies suggest unfortunately that independent judgment can be fairly readily swayed by false information.
It is therefore an eye opener, and reminder, to practitioners to ensure diligent and critical assessment of the information that is provided to them.
HPCSA - Responsibility remains with the Practitioner
The Health Professions Council of South Africa’s guidelines on the ethical use of AI recognise that the technology may assist with functions such as triage, image analysis and clinical decision support. It may not replace professional judgement, and responsibility for the eventual decision remains with the practitioner.
From a legal perspective, the issue is not simply who approved the diagnosis or treatment. The question is whether the practitioner applied the skill, care and independent reasoning expected in the circumstances.
An AI generated recommendation would not ordinarily relieve a practitioner of responsibility where a patient suffers harm. The practitioner would still need to explain how the patient was assessed, what clinical information was considered, whether any evidence contradicted the system’s recommendation and why the eventual course of treatment was chosen.
Continued reliance on an AI output becomes more difficult to justify as the available evidence begins to undermine it. A practitioner may remain the decision maker in name while having done little more than accept the conclusion presented by the technology.
The study therefore raises a question that goes beyond whether AI systems are accurate. It asks whether practitioners can recognise when their judgement has already been shaped by the system they are expected to supervise.
Training the platform
Healthcare providers are frequently advised to use properly validated platforms, ensure that systems are correctly configured and train practitioners to use them responsibly. These remain necessary precautions, but they do not address the separate risk that a practitioner may be influenced by the apparent authority of an algorithmic classification.
Once an AI system has identified a patient as more or less likely to respond to treatment, the practitioner may interpret later information through that initial conclusion. Evidence supporting the classification may receive greater weight, while contradictory evidence is overlooked or explained away.
Clinical governance should therefore extend beyond technical training and general warnings that AI can make mistakes. Healthcare providers should consider whether their processes require practitioners to test the system’s conclusions, identify contradictory clinical evidence and explain the reasoning behind the final decision.
The clinical record may become particularly important. A note recording only the diagnosis or treatment chosen will reveal very little about whether the practitioner independently assessed the AI recommendation or simply accepted it. AI generated notes can also have different focus or emphasis by training and this also highlights the necessity to ensure that true doctor to patient critical assessment clinical oversight is indeed taking place.
AI may improve the speed and effectiveness of healthcare decisions, but only where practitioners remain willing and able to disagree with it.
The legal safeguard is not the presence of a human at the end of the process. It is the independent clinical judgement exercised before the final decision is made.
For advice concerning the use of AI, POPI, healthcare regulation or related disputes, contact Julia Penn (juliap@thomsonwilks.co.za) and Justine Paries (justine@thomsonwilks.co.za)
