Are ‘vision-enabled’ AI scribes on the horizon?

The use of AI scribes in general practice is booming, with almost 40% of newsGP readers using them by November last year – a figure that has no doubt increased since.
 
So far, only audio is used to aid notetaking and consultation summaries, but could video also play a useful part?
 
The answer is a resounding yes if the results of a proof-of-concept trial conducted by researchers at Flinders University are backed up.
 
Using Ray-Ban Meta smart glasses and Google’s Gemini model to develop a ‘vision-enabled’ AI scribe, they tracked 10 pharmacists documenting 110 simulated medication history interviews.
 
The vision-enabled scribe achieved 98% overall accuracy, correctly recording 2114 out of 2160 data points, and ‘significantly outperformed’ audio-only processing, which stood at 81% overall, according to the study authors.
 
The results were mainly due to fewer omissions in the video-enabled consultations, with 10 errors compared to 358.
 
‘A key finding of the study was that video content substantially improved the accuracy of the AI scribe,’ researchers concluded.
 
Lead author Bradley Menz, a pharmacist and PhD researcher at Flinders University, said there could be a range of opportunities from these visual scribes that ‘we’re only starting to recognise now’.
 
‘A task that we share with doctors and with nurses – this could be in general practice, or it could also be in an emergency department – is taking a medication history,’ he told newsGP.
 
‘What we’ve known with medication histories is quite often there’s obviously a lot of spoken words, but also visual elements as well.
 
‘People might bring in a Webster-pak, or they might bring in physical medications, so our argument was … there’s probably a whole heap of visual information that’s potentially lost if [the AI scribe] can’t see.’
 
While clinical notes did not differ much, the study highlighted a stark gap between vision-enabled and audio-only results on the strength and form of medication, with video capturing the right details 97% of the time, compared to 28% accurate from the audio alone.
 
‘This makes sense because people are inclined to say, “I take one Panadol in the morning”, and they forget to mention the strength,’ Mr Menz said.
 
‘With some drugs, maybe that’s okay, something over the counter – but with a high-risk medication, that’s really important.’
 
For RACGP Specific Interests Digital Health and Innovation Chair Dr Sean Stevens, the trial seems like ‘a natural progression’ for AI scribes, highlighting some of the limitations of audio-only scribes.
 
‘You can get into habit of the way you conduct your consult to include important clinical details – the classic is the patient says “Oh, it hurts here” and they point with their finger to their left elbow,’ he told newsGP.  
 
‘And so you need to get into the habit of saying, “Oh, so it’s sore in your left elbow,” and the AI scribe will catch that.
 
‘We’re all human, you forget sometimes, so you miss out on a critical piece of information.
 
‘There is all sorts of information you get visually that may not be in an audio transcription, so I am not at all surprised that when you combine using a vision-enabled AI scribe with a classic audio scribe that you get an improved output.’
Examples of video and audio input being auto scribed by the AI scribe.

Of the 46 errors registered by the vision-enabled scribe, 10 were of omission while 36 were so-called ‘commission errors’, in which the AI scribe documented incorrect information.
 
Mr Menz said these errors are the most harmful.
 
‘The most important thing is identifying how can we stop these errors from happening,’ he said.
 
‘If it describes the wrong drug or the wrong dose and someone believes it, that can have really serious consequences.
 
‘You want to be really careful before you deploy something like this into routine practice even though there is someone checking over the document for accuracy at the end of it.’
 
However, he hopes the successful proof-of-concept demonstration may inspire others working in healthcare.
 
‘Maybe doctors, nurses and other healthcare professionals could think of tasks where they might be able to optimise their current workflow with AI,’ he said.
 
‘Our current scribe demonstrated in this study is quite a way off that, but we’d certainly like to expand on how we might be able to optimise this and contribute to the broader research field.’
 
Dr David Adam, a member of the RACGP Expert Committee – Practice Technology and Management, finds the publication interesting although also noted the concept is at a very early stage and that US data was used in the simulations.
 
‘The errors noted are potentially clinically significant, including the wrong formulation being detected,’ he told newsGP.
 
 He also said that other smart-glasses trials have shown they struggle in real-world environments.
 
‘It would be useful to see how well this works in a real clinical setting,’ he said.
 
Dr Stevens, meanwhile, predicts a greater role for smart glasses ahead.
 
‘In future, once we start getting towards software as a medical device, I can see smart glasses providing some useful input,’ he said.
 
‘And when you start to get some hints on diagnosis, which is where AI is heading, I consider they will provide a lot of very useful information.
 
‘We’re going to see lots more of these sorts of studies coming out and we’re going to see more and more use of different technologies.
 
‘Some of them we can predict, a lot if it we can’t, we’ll be taken by surprise – and I think it’s a very exciting time.’
 
Log in below to join the conversation.

AI AI scribes artificial intelligence