‘Little oversight’: Officials air AI scribe worries

Newly aired Government documents acknowledge official unease over a lack of oversight of the current use of artificial intelligence (AI) scribes in healthcare.
 
The heavily redacted papers, obtained by The Guardian under freedom of information laws, relate to a briefing ahead of Senate Estimates hearing in February, and the agenda for a meeting of the Government’s AI expert advisory group in April.
 
‘Many AI products, such as AI medical scribes, fall outside this framework and have little oversight, raising concerns about the use of these tools within healthcare settings,’ the briefing note states.  
 
‘Anecdotally, some are marketed as outside regulatory levers (eg, not a medical device) or as privacy compliant, often with limited transparency.
 
‘Some suppliers may be unaware their cloud platforms send data outside Australia, raising risks for patient data security.’
 
The documents reflect an intensifying Government focus on the growing use of AI in healthcare, which has seen many GPs adopt the technology in their everyday practice.
 
Last November, 40% of newsGP readers said they regularly used AI scribes, a number that is likely to have increased substantially since, with Heidi Health and Lyrebird among the most commonly used.
 
Reforms put in place in 2021 mean certain types of software are excluded from therapeutic goods regulation if they do not meet the definition of a medical device.
 
Under existing Therapeutic Goods Administration (TGA) guidance, AI scribes are not defined as such if they ‘only generate clinical notes, summaries or letters which may be added to patient records’. 
 
However, a TGA report into medical device software regulation last year found ‘users report digital scribes frequently propose diagnosis or treatment options beyond the stated diagnosis or treatment a clinician has identified during consultations’.
 
It also noted developers provide ‘limited information’ on datasets used to train the models, and said a ‘review of digital scribes is needed to determine whether they are medical devices, and, if so, whether they comply with existing regulatory requirements’.
 
Dr Sean Stevens, RACGP Specific Interests Digital Health and Innovation Chair, believes the current medical devices definition makes sense.
 
‘AI scribes should not be counted as software as a medical device unless they’re helping make or making diagnosis and recommending treatments,’ he told newsGP. 
 
‘That is the line in the sand.
 
‘At the end of the day, what’s most important is that the GP or other doctor retains clinical oversight.’
 
In the Government’s briefing document, officials report that some AI scribe suppliers market the potential for ‘30% revenue increase with no additional hours or patient consultations’, a move it says has implications for MBS costs.
 
In the same release, an agenda item for the Government’s AI advisory group notes the potential for AI scribes to enhance productivity and efficiency, while also highlighting ‘significant variation’ in the way patient consent is sought for using AI scribes.
 
‘Our position is that informed consent requires consumers to understand the benefits and limitations of the technology to which they are consenting,’ the Department of Health, Disability and Ageing (DoHDA) advice states.
 
It says a strong existing medical device framework is among the regulatory options it has at its disposal, as well as ‘clinician and consumer capability and literacy uplift’ for the use of software that does not meet the medical device threshold.
 
On Wednesday, the DoHDA confirmed the TGA’s work on AI scribes is still underway.
 
‘The review is currently ongoing, prioritising those products more widely used in Australia,’ a spokesperson told newsGP.
 
‘The TGA is working with industry associations and individual companies to ensure regulatory requirements are understood, and we plan to publish a summary of the outcomes from the review in coming months.
 
‘The TGA does not comment on the review of individual products.’
 
Dr Stevens says the current dynamic situation is common when new technology is rapidly adopted.
 
‘The college will be looking closely at the final report,’ he said.
 
‘It’s important that we have safeguards, but it’s also important we don’t stifle innovation.
 
‘It is a rapidly evolving space and it’s always going to be the case that regulators are going to be behind where technology is happening.
 
‘It needs to be looked at, but it’s very important that the regulators do consult with the college and other professional bodies.’
 
Even with the rapid uptake of AI scribes, Dr Stevens emphasises the fundamentals of clinical oversight remain the same.
 
‘It’s important you read through all of the output of the scribe, focusing on the critical areas like medications, doses of medications, recommended treatments, left and right,’ he said.  
 
‘There are a few key things you really need to make very, very sure of with every transcript, every output from the scribe.
 
‘It really is down to the clinical judgement of the doctor.
 
‘It’s like any tool – the doctor is still responsible for what goes into the notes.’
 
The RACGP has published guidance on the use of AI scribes, as have the Australian Commission on Safety and Quality in Health Care and the Australian Health Practitioners Regulatory Authority.
 
Log in below to join the conversation.

AI AI Scribes artificial intelligence Heidi Lyrebird