Is AI A-OK? Medicolegal considerations for general practitioners using AI scribes

This article is part of a longitudinal series on artificial intelligence (AI).

Medical records are essential to the safe delivery of high-quality healthcare. They facilitate communication and continuity of care between practitioners, health services and patients, thereby reducing clinical errors.¹ In addition to serving important research, quality improvement and disease-outbreak monitoring functions, they also provide valuable evidence in the assessment, prosecution and defence of medicolegal claims against general practitioners (GPs). There is a well-known axiom that ‘good records are a good defence, poor records are a poor defence, and no records are no defence’. Where there is a dispute between a GP and patient about what transpired during a consultation, courts will often prefer the evidence of the GP where it is supported by, and consistent with, what is documented in the medical record.^2,3

Despite the benefits of good medical records, excessive time required to create and review them can adversely impact clinician and health service productivity.⁴ Clinicians often say that they spend more time documenting clinical encounters than they spend with patients.⁵ This ‘pain point’ in daily medical practice can amplify stress and burnout.⁶ In Australia, over the last 18 months, we have witnessed the advent of a potential solution with the rapid development and release of artificial intelligence transcription tools (hereafter described as ‘AI scribes’). AI scribes work by audio recording a consultation, converting speech to text, then adopting predictive models to generate a summarised output, such as a progress note, referral letter or report.⁷ Many AI scribes can be integrated into existing electronic medical records systems, making them easy to implement into standard clinical practice.

AI scribes have been labelled a ‘game-changer’,⁸ reducing stress levels⁹ and the time taken to document consultations by up to 40%.¹⁰ A survey of 1000 Australian GPs showed that AI scribe use has increased from <3% in May 2024 to 8.24% by October 2024,¹¹ and they are being used in over 10,000 consultations per day.¹² In the UK, at least one in five GPs are using an AI scribe on a daily basis.¹³ The combination of novel technology, evolving laws and regulations and increasing uptake makes it critical that GPs understand the medicolegal implications before implementing AI scribes in their practice. Even after implementation, AI scribes need to be continuously monitored to ensure that they operate as intended, and that they remain compliant with Australian laws and regulations. This article presents some of the critical medicolegal considerations for GPs to consider when using AI scribes, with reference to legislation, recent cases and guidelines.

Due diligence

In Australia, AI scribes do not meet the definition of ‘medical devices’ because they do not support diagnostic decision making.¹⁴ This currently makes them unregulated goods that are exempt from registration with the Therapeutic Goods Administration (TGA).¹⁵ Therefore, before incorporating AI scribes into clinical practice, GPs should conduct due diligence to ensure that the software has been tested and is suitable for its intended use, meets GPs’ clinical needs and is not also designed to support clinical decision making (in which case, the software may need to be TGA registered).

To better understand the product and its functionality, GPs can request a trial period, demonstration and/or training in the use of the AI scribe and should carefully review (and obtain legal advice on) any user or service agreements before purchasing a licence. In particular, GPs should be alert to indemnity clauses that seek to transfer liability away from the software vendor in the event of product failure or error. GPs should also consider the size and reputation of the software vendor and its capacity to protect the security of information. If a product licence is purchased, GPs should continue to monitor the performance of the AI scribe to ensure that it remains appropriate and compliant with the privacy obligations in the section on ‘Privacy laws’. The TGA is reviewing current frameworks regulating the use of AI in healthcare,¹⁶ and it is possible that regulatory settings will evolve as AI scribes become mainstream.

Privacy laws

A complex web of overlapping Commonwealth, state and territory privacy laws regulate the collection, use, disclosure, storage, retention and cross-border transfer of health and/or personal information in the Australian public and private sectors (see Table 1). These are relevant to the use of AI scribes.

First, when AI scribes listen to and transcribe clinical encounters, health and/or personal information is collected. Personal information is information or an opinion about an individual who is identifiable or reasonably identifiable, while health information is personal information relating to an individual’s health or information collected during the provision of healthcare.^17–21 We hereafter refer to health and/or personal information as ‘personally identifiable information’ (PII). PII can be in written or audio form.^22,23

Second, some AI scribes anonymise the content of audio and transcription files by redacting identifying information such as name and date of birth. If this results in the information being de-identified, then it would not be considered PII and privacy laws may not apply to its collection, use, disclosure, storage, retention or cross-border transfer. However, GPs need to be aware that merely removing the patient’s name or date of birth may be insufficient to de-identify PII if it is still reasonably re-identifiable. It is critical that GPs understand and check whether transcripts are de-identified, as this would impact on whether the further privacy obligations outlined below must be observed. GPs may need to obtain legal advice or speak with their medical defence organisation to fully understand these obligations.

Third, PII may only be used for the primary purpose for which it was collected, unless patients have consented to (or would reasonably expect) an unrelated secondary use. For example, the primary purpose of collecting PII using an AI scribe is to facilitate the provision of healthcare. However, some AI scribes are open source, meaning that they may use PII to train and improve the AI model, and make this training data publicly available. Privacy laws would not permit this unrelated secondary use or disclosure unless patients specifically consent or unless it was explicitly outlined in a privacy policy brought to the patient’s attention prior to collection or unless the information was sufficiently de-identified. If PII is shared to an open-source AI scribe, it becomes difficult to track, control or remove. Without fully understanding what happens to their PII, it may be challenging for patients to make informed decisions about the use of an AI scribe. Therefore, before purchasing an AI scribe licence, GPs should ask software vendors whether any secondary uses of PII are intended.

Fourth, some overseas-based AI scribes may store or process PII outside Australia. The Privacy Act 1988 (Cth) requires GPs using overseas-based AI scribes to either: 1) ensure that the overseas jurisdiction in which the information is stored or processed is subject to privacy laws that are substantially similar to Australian privacy laws (these countries will be prescribed by the Attorney-General through regulation); or 2) obtain consent from patients to the overseas transfer of their PII.²⁴ Thus, while it is not unlawful to use overseas-based AI scribes, it increases the regulatory compliance burden for GPs and practices.

Finally, many available AI scribes do not store recordings of the consultation and delete the text transcripts either as soon as the output has been generated or within seven to 30 days. This is consistent with the Privacy Act 1988 (Cth), which requires that PII must be destroyed or de-identified when no longer required for its original purpose and there is no other legal requirement to retain it.²⁵ However, in Victoria (Vic), New South Wales (NSW) and the Australian Capital Territory (ACT), there is a legal requirement to retain health information for at least seven years from when a patient was last seen or until a child reaches the age of 25 years.²⁶ In those jurisdictions, if audio files or text transcripts contain health information that has not been incorporated into the AI-generated output, it would be unlawful in that jurisdiction to delete that health information prior to the expiration of these statutory timeframes. If a software vendor claims that only non-PII is deleted, GPs in Vic, NSW and ACT may need to check that transcripts are non-identifiable. On the other hand, if PII is deleted, GPs may need to check that all clinically relevant PII has been incorporated into the final output and, if not, to manually amend the output before finalising it. This is a complex issue and largely untested by the courts.

In August 2024, the Office of the Australian Information Commissioner (OAIC) released timely guidance on privacy and the use of commercially available AI products.²⁷ The OAIC expects organisations adopt a cautious approach to using AI. It recommends that organisations update their privacy policies with clear and transparent information about their use of AI. It also recommends that PII not be entered into publicly available AI tools, as individuals may lose control of their PII and it may increase the risk of data breaches from unauthorised access or cyberattacks. The OAIC also warns users about the risk of algorithmic bias that can lead to systematic output errors that may disproportionately impact vulnerable groups, such as children or Aboriginal and Torres Strait Islander peoples. This may be especially relevant in a healthcare context.

In September 2024, the Deputy Commissioner of the Victorian Office of the Information Commissioner investigated the Victorian Department of Families, Fairness and Housing (DFFH) after a child protection worker (CPW) used ChatGPT to draft a child protection report that was submitted to the Children’s Court in a case about whether a child required state protection.²⁸ In that case, the CPW entered personal and sensitive information about the child and the child’s family into ChatGPT, including names and risk assessments. The Deputy Commissioner found that the AI-generated report contained significant errors that had not been checked and this had the potential to seriously mislead the Court. Furthermore, the Deputy Commissioner found that, in using ChatGPT, the CPW had disclosed without consent personal and sensitive information to an overseas company (Open AI) that could further use and disclose that information for unrelated secondary purposes. This amounted to a serious breach of Victoria’s Information Privacy Principles (IPPs). The Deputy Commissioner issued a compliance notice banning the use of ChatGPT and similar tools by CPWs.

Surveillance devices laws

To comply with Australian privacy laws, PII must only be collected by lawful means. Legislation in each Australian state and territory defines ‘listening devices’ and protects individuals from invasions of privacy through the use of listening devices by criminalising their unlawful use. These laws are summarised in Table 2. As AI scribes record private conversations, they are likely to meet the definition of a listening device, and their use without patient consent could breach surveillance devices laws. For example, in Toth v DPP,²⁹ a patient was found guilty of an offence under the Surveillance Devices Act 2007 (NSW) for secretly recording a clinical consultation without the GP’s knowledge or consent. GPs could be at risk of similar criminal sanctions for recording consultations using AI scribes without patient consent, or sending clinical information to a colleague based on a consultation conducted using an AI scribe without patient consent.

The use of AI scribes in relation to telehealth consultations is also complex, as GPs physically located in an Australian jurisdiction should be aware that the privacy and/or surveillance devices laws of another Australian or overseas jurisdiction may also apply where the patient is physically located interstate or overseas. In addition, the use of an AI scribe during a telehealth consultation without patient consent could amount to the interception of a telecommunication in breach of Section 7 of the Telecommunications (Interception and Access) Act 1979 (Cth). To ensure that GPs do not breach these laws, consent must be obtained.

Patient consent

As described above, GPs must obtain patient consent before using an AI scribe. This not only protects GPs from breaching surveillance devices and privacy laws, but is also an important ethical obligation to ensure transparency and to engender patient trust in the technology and the therapeutic alliance.³⁰ Box 1 sets out some of the key matters that patients must consent to when using an AI scribe. Some software vendors helpfully provide prompts that require clinicians to confirm that they have sought consent before each consultation can be transcribed.³¹ Some also provide detailed patient information and written consent forms.³² While written consent forms are helpful in demonstrating that consent has been obtained, a verbal discussion should also occur in which patients are offered an opportunity to ask questions.

If consent is obtained, GPs should document the details of the information provided to the patient and the patient’s consent (or refusal of consent) in the medical record. This also serves as a record that an AI scribe was or was not used in the generation of the record. Patients should be given the opportunity to withdraw their consent at any time and re-consenting should occur prior to each consultation using an AI scribe, as the recording of each consultation without consent may be a separate criminal offence in some Australian jurisdictions. While the matters in Box 1 should be canvassed with patients prior to the first use of an AI scribe, the discussion and consent process prior to subsequent consultations is likely to be limited to reminding patients about the use of the AI scribe and providing an opportunity for the patient to ask questions or opt out. Where patients withhold consent to the use of an AI scribe, the consultation should be documented in the usual manner.

To support the consent process, patients should also be made aware of the proposed use of AI scribes ahead of the consultation, such as through the clinic’s website, booking engine, privacy policy or notice, new patient registration forms or information pamphlets. Where patients lack the capacity to consent due to age, illness or disability, consent should be sought from the patient’s substitute decision maker, parent or guardian, in accordance with legal requirements in each Australian state or territory.

Accuracy

Although AI scribes continue to improve, one of their persisting limitations is the risk of false or misleading outputs, known as ‘hallucinations’.³³ Hallucination rates across popular AI tools vary from 0.8% to 3.9%.³⁴ Errors may arise when AI scribes mishear and/or incorrectly transcribe spoken words, and when they fail to detect the clinically relevant aspects of a history. These errors can compromise patient safety (when other practitioners caring for a patient rely on inaccurate health information) or data breaches (when PII is disclosed to the incorrect recipient when patient or practitioner contact details are wrong).³⁵ In all, over-reliance on the accuracy of tools that ordinarily perform well is a danger, especially when GPs are under significant time pressure.

AI scribes generate outputs based on what is heard and transcribed during a consultation. Hence, outputs may also be inaccurate or incomplete when parts of the consultation are unspoken or rely on information from extraneous sources (such as discharge summaries or test results). GPs need to consider this limitation by either verbalising additional aspects of the consultation (such as physical examination findings and test results) or by manually adding further clinical information to the output before it is finalised. Although verbalisation of the consultation may assist patients to understand GPs’ clinical reasoning, it substantially changes the manner in which consultations are conducted and may not always be appropriate (such as verbalising the patient’s general appearance, weight, behaviours or sensitive issues, such as mental health or suspected child abuse). GPs may need to adjust their consulting style to the circumstances of the consultation and exercise discretion.

Ultimately, it is crucial that GPs remain vigilant and check the accuracy of all AI-generated outputs after every recorded consultation. If the output generated does not accurately reflect the content of the consultation, the GP must edit, amend or correct the output before it is finalised. This is consistent with GPs’ legal and professional obligations under various Australian privacy laws,³⁶ and the Medical Board of Australia’s Code of Conduct (see Box 2).³⁷ Although there are no published cases involving medical practitioners, there have been two recent and widely reported cases involving lawyers in Vic³⁸ and NSW³⁹ who used AI to generate court documents that contained serious errors, including fictitious cases. Both lawyers were criticised by the courts for failing to check the accuracy of those documents, and were referred to the relevant state legal complaints commissions. This has prompted legal regulators to send a clear warning to the legal profession about the importance of checking all AI-generated outputs for accuracy.⁴⁰

 

Professional guidance

In response to the rapid integration of AI into healthcare, regulators, professional bodies and courts are creating guidance for clinicians.

The Australian Health Practitioner Regulation Agency (Ahpra) has created a guideline that explains how the Code of Conduct applies to practitioners using AI scribes.⁴¹ It states that practitioners must: understand and explain to patients the privacy implications of using AI scribes; obtain informed consent before using AI; ensure outputs are culturally safe; and always apply human judgment to AI outputs by checking for accuracy and relevance. Ahpra also warns practitioners against entering confidential patient information into open-source AI tools (such as ChatGPT) where health information can be stored overseas indefinitely for later public use.

Similarly, The Royal Australian College of General Practitioners has released a fact sheet for GPs.⁴² It advocates a cautious approach, given the limited data on the safety, utility, validity, efficacy or applicability of AI scribes in an Australian GP context. It recommends that GP clinics develop their own policies and procedures to decide if, when and how AI scribes will be deployed. This includes the need for training and additional IT security to prevent data breaches. It also recommends that, before purchasing a product, GPs talk with software vendors to ensure they understand how it works and any potential impact that altered clinical workflows may have on daily practice.

Finally, recent directions from the Chief Justice of the NSW Supreme Court prevent expert witnesses from using AI to draft or prepare an expert report (in whole or in part) without prior court permission.⁴³ If the court grants leave, the report must state which parts of the report were generated using AI, how the AI scribe was used and what codes (if any) regulate the expert’s use of the AI tool. Lawyers commissioning expert opinions have an obligation to bring these directions to the expert’s attention. In Victoria, witnesses are encouraged to disclose to courts if AI is used in the preparation of a court document.⁴⁴

Conclusion

The rise of AI scribes over the last 18 months has been unprecedented. By expediting the creation of medical records, AI scribes can reduce the administrative burden for busy GPs, potentially enhancing patient safety and GP wellbeing and productivity. However, to harness these powerful new tools, GPs need to understand the medicolegal issues that may arise before adopting them into their practice. For those who are already using these tools, this paper serves to highlight the importance of GPs regularly reviewing the performance of these tools and their compliance with Australian laws and regulations.

Before implementing an AI scribe into clinical practice, GPs need to ensure that they are prepared: new technologies may impact workflows; potential products should be trialled; user agreements should be carefully reviewed; staff require training; and privacy policies need updating. GPs also need to understand how patient information will be handled by the AI scribe (whether, what, where and for how long data is stored) so that they can explain this to patients, who must consent. Additional components of consultations may need to be verbalised and the accuracy of all AI-generated outputs must be checked for accuracy and appropriateness.

Even after AI scribes have been embedded into practice, their use and performance must be regularly reviewed to ensure ongoing suitability and legal compliance. As experience with AI scribes increases, our understanding of the benefits and risks will deepen. Simultaneously, there will be continued legal and regulatory developments, as more cases, guidelines and laws seek to further regulate this burgeoning area of healthcare practice. GPs need to keep abreast.

Key points

• AI scribes can save time, increase productivity and reduce GP stress.
• Before using an AI scribe, GPs should test the product, ensure that it is designed for a healthcare context and check that information is not shared publicly.
• Patient consent must be obtained before each consultation using an AI scribe.
• GPs must understand what happens to patient information after collection so this can be effectively explained to patients.
• Every AI output must be checked for accuracy before being finalised.

Disclaimer

The information contained in this article is intended to be general information only and should not be relied upon as legal advice. You should seek legal or other professional advice before relying on any content and exercise proper clinical decision-making with regard to your individual circumstances. Information is current at the date of acceptance of this article.