May 2024 correspondence

Artificial intelligence takes on the Australian Medical Council’s trial examination: Comment

We would like to comment on the published article entitled ‘Universal precautions required: Artificial intelligence takes on the Australian Medical Council’s trial examination’, which was recently published in AJGP.¹ The study looked at the ability of OpenAI’s ChatGPT to answer medical multiple-choice questions (MCQs) in an Australian setting. The researchers graded the chatbot’s responses using MCQs from the Australian Medical Council’s medical licensing practice examination. The experiment was carried out twice. ChatGPT received a score of 29 out of 50 for being moderately accurate in answering the questions. It was able to generate explanations for most questions (45 out of 50). The chatbot also showed moderate consistency, with the same overall answer to 40 of the 50 questions across trial runs.

One study flaw is that the sample size of questions might not be representative of the entire range of medical MCQs. Furthermore, because the study did not include a comparison to human performance, it is unclear how the responses generated by ChatGPT compared to those of human experts.

In terms of future directions, additional research could investigate ways to improve ChatGPT’s accuracy and consistency in answering medical MCQs. This could include providing the chatbot with broader and more diverse medical information, as well as incorporating feedback from human experts to improve its performance. Furthermore, studies could be conducted to compare the performance of ChatGPT to that of human experts to assess its potential application in medical practice. More research and development are needed to improve and optimise ChatGPT’s capabilities in clinical pathology. Finally, the appropriate use code² should be used for all AI applications.

Authors

Hinpetch Daungsupawong PhD, Private Academic Consultant, Phonhong, Lao People’s Democratic Republic

Viroj Wiwanitkit MD, Adjunct Professor, Chandigarh University, Punjab, India

Competing interests: None.

References

1. Kleinig O, Kovoor JG, Gupta AK, Bacchi S. Universal precautions required: Artificial intelligence takes on the Australian Medical Council’s trial examination. Aust J Gen Pract 2023;52(12):863–65. doi: 10.31128/AJGP-02-23-6708.
2. Kleebayoon A, Wiwanitkit V. ChatGPT, critical thing and ethical practice. Clin Chem Lab Med 2023;61(11):e221. doi: 10.1515/cclm-2023-0495.

Reply

We appreciate the informed comments regarding our study.¹ It is agreed that further research to understand ChatGPT’s performance in medical applications is beneficial and will inform the potential integration of large language models in medicine. Further research on prompting patterns will help generate more accurate and useful outputs. Additionally, training medicine-specific large language models might help further enhance their performance.²

Sampling the entire range of medical multiple-choice questions (MCQs) is beyond the scope of a single examination and this study. Multiple previous studies have evaluated the performance of ChatGPT in a variety of medical specialty and overseas examinations.³ We believe that the Australian Medical Council’s practice medical licencing examination is relevant to the Australian context. The range of questions in this examination includes content regarding pathophysiology, diagnosis, management and Indigenous Health.⁴

Direct comparisons between humans and ChatGPT is not a favoured methodology,³ as the inherent variability in human performance would lead to unreliable comparisons. Instead, our paper uses questions that are standard – set to the level of a graduating Australian medical student, which provides an objective human benchmark.¹ Furthermore, the capabilities of ChatGPT are updated regularly so exact comparisons between ‘person and machine’ are quickly outdated. For example, our piece discusses a limitation of ChatGPT (GPT-3 version) not accepting image inputs. However the latest iteration, ChatGPT-4, can now interpret images with moderate-to-low accuracy.⁵ Despite such updates in ChatGPT, the findings in the presented study provide illustrations of selected large language model limitations in an Australian context, which might be helpful when discussing these technologies with both Australian trainees and patients.

Authors

Oliver Kleinig MBBS III, Faculty of Health and Medical Sciences, The University of Adelaide, Adelaide, SA; Royal Adelaide Hospital, Adelaide, SA

Joshua Kovoor MS, Surgical Resident Medical Officer, Department of General Surgery, Queen Elizabeth Hospital, Woodville South, SA

Aashray K Gupta MBBS, MS, Cardiothoracic Registrar, Department of Cardiothoracic Surgery, Gold Coast University Hospital, Gold Coast, Qld

Stephen Bacchi MBBS, PhD, Neurology Registrar, Department of Neurology, Royal Adelaide Hospital, Adelaide, SA; Neurology Registrar, Department of Neurology, Flinders University, Bedford Park, SA

Competing interests: None.

References

1. Kleinig O, Kovoor JG, Gupta AK, Bacchi S. Universal precautions required: Artificial intelligence takes on the Australian Medical Council’s trial examination. Aust J Gen Pract 2023;52(12):863–65. doi: 10.31128/AJGP-02-23-6708.
2. Kleinig O, Gao C, Kovoor JG, Gupta AK, Bacchi S, Chan WO. How to use large language models in ophthalmology: From prompt engineering to protecting confidentiality. Eye (Lond) 2023. doi: 10.1038/s41433-023-02772-w. Epub ahead of print.
3. Levin G, Horesh N, Brezinov Y, Meyer R. Performance of ChatGPT in medical examinations: A systematic review and a meta-analysis. BJOG 2023. doi: 10.1111/1471-0528.17641. Epub ahead of print.
4. Australian Medical Council Limited. AMC Computer Adaptive Test (CAT) Multiple Choice Question (MCQ) Examination. Australian Medical Council, 2023. Available at www.amc.org.au/pathways/standard-pathway/amc-assessments/mcq-examination/ [Accessed 17 December 2023].
5. Waisberg E, Ong J, Masalkhi M, et al. GPT-4: A new era of artificial intelligence in medicine. Ir J Med Sci 2023;192(6):3197–200. doi: 10.1007/s11845-023-03377-8.

Hydrochlorothiazide use and risk of non-melanoma skin cancer

Hydrochlorothiazide is one of the most frequently prescribed antihypertensives in the world and it has been associated with increased risk of non-melanoma skin cancers (NMSC) due to its photosensitising property.^1,2 The findings for the association between hydrochlorothiazide and risk of NMSC continue to be heterogenous and conflicting.^1,3,4 Multiple studies have reported the use of hydrochlorothiazide was associated with an increased risk of squamous cell carcinoma (SCC) with a duration and dose-response relationship.^1,4,5 In particular, high cumulative use of hydrochlorothiazide in Caucasians is associated with increased risk of SCC.^1,4 Use of other diuretics and other antihypertensives are not associated with increased risk of NMSC.^1,4 A recent cohort study based in the United States (US) did not establish a clear difference between hydrochlorothiazide and angiotensin-converting enzyme (ACE) inhibitor in terms of skin cancer risk after adjusting for race and ethnicity.² However, the cohort study had only evaluated for a 12-month duration and might not have been able to evaluate the high cumulative dose as described in other studies.^1,3–5

Australia has one of the highest rates of skin cancer in the world with a large Caucasian population susceptible for skin cancer.⁶ General practitioners (GPs) are often the first in line to commence antihypertensive treatment and to continue to monitor and adjust antihypertensive medications for patients. Given the potential association between hydrochlorothiazide and NMSC, it is important to assess the benefits of hydrochlorothiazide use and potentially consider other available antihypertensive medications, particularly in patients with risk factors for NMSC such as previous skin cancers, sun exposure and family history of skin cancers.³ Finally, it is important to continue to promote preventive strategies against sun exposure and encourage regular skin examinations as part of the skin cancer prevention strategy.³

Author

Chon-Wai Jeremy Chan MD, BSc, Resident Medical Officer, The Maitland Hospital, Metford, NSW

Competing interests: None.

References

1. Haisma MS, Greven N, Logendran M, et al. Chronic use of hydrochlorothiazide and risk of skin cancer in Caucasian adults: A PharmLines Initiative Inception cohort study. Acta Derm Venereol 2023;103:adv3933. doi: 10.2340/actadv.v103.3933.
2. Birck MG, Moura CS, Machado MAA, et al. Skin cancer and hydrochlorothiazide: Novel population-based analyses considering personal risk factors including race/ethnicity. Hypertension 2023;80(10):2218–25. doi: 10.1161/HYPERTENSIONAHA.123.21274.
3. Garrido PM, Borges-Costa J. Hydrochlorothiazide treatment and risk of non-melanoma skin cancer: Review of the literature. Rev Port Cardiol (Engl Ed) 2020;39(3):163–70. doi: 10.1016/j.repc.2019.07.008.
4. Shao SC, Lai CC, Chen YH, Lai EC, Hung MJ, Chi CC. Associations of thiazide use with skin cancers: A systematic review and meta-analysis. BMC Med 2022;20(1):228. doi: 10.1186/s12916-022-02419-9.
5. Rouette J, Yin H, Pottegård A, Nirantharakumar K, Azoulay L. Use of hydrochlorothiazide and risk of melanoma and nonmelanoma skin cancer. Drug Saf 2021;44(2):245–54. doi: 10.1007/s40264-020-01015-1.
6. Czarnecki D. Mortality from nonmelanoma skin cancer in Australia from 1971 to 2021. Cancers (Basel) 2024;16(5):867. doi: 10.3390/cancers16050867.

Research Letter

Better Osteogenesis Needs Exercise - Australian Indigenous Dance (BONE-AID): Lessons learned from a pilot study of postmenopausal women

Exercises with moderate-to-high impact loads, more than twice bodyweight, which are progressive and multidirectional, are likely to be osteogenic for postmenopausal women.¹ Studies show that sports and physical activities that involve high- or unusual-impact weight-bearing, often with rapid rates of loading (eg gymnastics, basketball, power lifting, ballet dancing), provide superior bone mass at specific loaded skeletal sites, compared to lower-impact or non-weight-bearing activities.^2–5 It was therefore hypothesised that Australian Indigenous dance, which includes considerable foot stomping, might be beneficial for bone health. A pilot study was funded by The Royal Australian College of General Practitioners’ (RACGP) Foundation and an Australian Association of Musculoskeletal Medicine (AAMM) grant to examine the effect of once-weekly 12-month Australian Indigenous dance on bone density, wellbeing, functional strength, mobility, balance and safety in postmenopausal women.

This exploratory uncontrolled pilot intervention recruited non-Indigenous healthy community-dwelling postmenopausal women and employed dance teachers of the Bundjalung and Mununjali nations from a regional Australian town. Outcome measures included hip and spine bone mineral density (BMD), wellbeing (SF-12), hand grip strength, Berg Balance Test, Timed Up and Go, Sit to Stand (30s and 60s), program feedback and adherence, and were measured pre- and postintervention. Data were analysed using paired t-tests.

The study was approved by the Southern Cross University Human Research Ethics Committee (ECN-17-216) and the Aboriginal Health and Medical Research Council (reference: 1409/18). All participants provided written informed consent.

Overall, 25 women (59±5 years; body mass index 26.5±5.7) were recruited, of whom 13 dropped out. Dance class adherence was 69.9% and no adverse events occurred. No differences were observed in any outcome in the remaining 12 participants.

The 50% drop out occurred early in the program when the original dance teacher had an illness, and the research leader had a catastrophic injury. Outcomes suggest once-weekly Indigenous dance is insufficient to improve bone density and functional strength despite being an enjoyable activity for participants.

The study highlights challenges with implementing an exercise-based intervention researched in a rural primary care setting with a small team and limited funding.

Authors

Rosemary Craig MBBS, FRACGP, Adjunct Professional Fellow, Faculty of Health, Southern Cross University, Lismore, NSW

Suzanne Broadbent PhD (ExSc), BExSc (Hon), BEd, AEP ESSA, Associate Professor, Clinical Exercise Physiology, School of Health, University of the Sunshine Coast, Sippy Downs, Qld

Belinda Beck FACSM, FASMF, FASBMR, FESSA, PhD, MS, BMHS (Ed), Professor, Exercise Science, School of Health Sciences and Social Work, Griffith University, Gold Coast, Qld

Competing interests: BB is the Director of The Bone Clinic Pty Ltd.

References

1. Beck BR, Daly RM, Singh MA, Taaffe DR. Exercise and Sports Science Australia (ESSA) position statement on exercise prescription for the prevention and management of osteoporosis. J Sci Med Sport 2017;20(5):438–45. doi: 10.1016/j.jsams.2016.10.001
2. Watson SL, Weeks BK, Weis LJ, Harding AT, Horan SA, Beck BR. High-intensity resistance and impact training improves bone mineral density and physical function in postmenopausal women with osteopenia and osteoporosis: The LIFTMOR Randomized Controlled Trial. J Bone Miner Res 2018;33(2):211–20. doi: 10.1002/jbmr.3284
3. Young CM, Weeks BK, Beck BR. Simple, novel physical activity maintains proximal femur bone mineral density, and improves muscle strength and balance in sedentary, postmenopausal Caucasian women. Osteoporos Int 2007;18(10):1379–87. doi: 10.1007/s00198-007-0400-6
4. Zitzmann AL, Shojaa M, Kast S, et al. The effect of different training frequency on bone mineral density in older adults. A comparative systematic review and meta-analysis. Bone 2022;154:116230. doi: 10.1016/j.bone.2021.116230
5. Kemmler W, Shojaa M, Kohl M, von Stengel S. Effects of different types of exercise on bone mineral density in postmenopausal women: A systematic review and meta-analysis. Calcif Tissue Int 2020;107(5):409–39. doi: 10.1007/s00223-020-00744-w

Research Letter

Reflecting general practice trainees’ experiences of teaching senior medical students during general practice placement: A pilot study

Teaching is one of the important components of general practice training for trainees, with evidence-proven benefit of enhanced knowledge retention compared with non-teaching peers,¹ and deliberate adoption of good clinical practice.² However, observed barriers include lack of training and limited opportunities, with research showing around 60% of general practice trainee involvement with some teaching and 90% having a desire for further teaching training.³ General practice trainees are limited to teaching informally and occasionally, through ad-hoc opportunities without prior notice.⁴ Further research has shown medical students’ preference for trainee-delivered teaching due to having a similar age with learners and role modeling.⁵ To foster a consistent teaching model for general practice trainees, this pilot study reflects on experiences of general practice trainees teaching senior medical students during general practice placements.

Two general practice trainees in their final year of training and 15 medical students were included from a general practice clinic during four weeks of clinical placements. The 15 medical students allocated to the general practice clinic included five Chinese exchange students from 2017 to 2019 and 10 Australian students from 2020 to 2022. A regular weekly training session was booked during the four-week general practice placement. Prior to each training session, the general practice trainees randomly pre-booked six to eight patients for the student to consult with during the 3.5-hour session. The booked patients provided consent for the students to conduct consultations with them in a general practitioner role under the supervision of the general practice trainee (in an observer role). The general practice trainee provided structured in-person feedback during the student-led consultation by using a standardised, validated consultation feedback tool to assess components of a consultation, including introduction, history, physical exam, investigation, diagnosis, management and closing summary.

Students rated general practice trainees as excellent teachers and confirmed the feedback tool’s importance in clinical reasoning learning, Objective Structured Clinical Exam (OSCE) preparation and clinical competency development. The identified barriers included a limited cohort of patients suitable for teaching, limited resources to establish a student-led teaching clinic, time management and teaching training education for the general practice trainees.

General practice trainees can deliver general practice placement teaching using a student-led consultation teaching model for clinical reasoning learning, clinical competency building and OSCE exam preparation. The identified barriers of this study can be overcome by providing teaching training education, sharing time-management skills and establishing dedicated teaching clinics for the general practice trainees.

Authors

Shaoting Feng^A MD, Deputy Director, Department of Medical Education; Director, Undergraduate Education Office, First Affiliated Hospital, Sun Yat-Sen University, Guangzhou, China

Daya Yang^A MD, PhD, Deputy Director, Medical Faculty Development Center, First Affiliated Hospital, Sun Yat-Sen University, Guangzhou, China

Kunsong Zhang^A MD, PhD, Director, Medical Faculty Development Center, First Affiliated Hospital, Sun Yat-Sen University, Guangzhou, China

Dennis Findlay MD, Year 4 Academic Coordinator, Curtin Medical School, Faculty of Health Sciences, Curtin University, Perth, WA

Ming Kuang^B MD, PhD, Vice President and Executive Dean, First Affiliated Hospital, Sun Yat-Sen University, Guangzhou, China

Haipeng Xiao MD, PhD, President and Deputy Vice Chancellor, First Affiliated Hospital, Sun Yat-Sen University, Guangzhou, China

Dan Xu^B MD, PhD, FRACGP, General Practice and International Collaboration Academic Coordinator, First Affiliated Hospital, Sun Yat-Sen University, Guangzhou, China; Visiting Professor, Curtin Medical School, Faculty of Health Sciences, Curtin University, Perth, WA

^AEqual first authors.

^BCorresponding authors.

Competing interests: None.

References

1. de Villiers MR, Cilliers FJ, Coetzee F, Herman N, van Heusden M, von Pressentin KB. Equipping family physician trainees as teachers: A qualitative evaluation of a twelve-week module on teaching and learning. BMC Med Educ 2014;14:228. doi: 10.1186/1472-6920-14-228.
2. Kleinitz A, Campbell D, Walters L. General practice registrar perceptions on training medical students. Aust Fam Physician 2014;43(1):64–67.
3. Halestrap P, Leeder D. GP registrars as teachers: A survey of their level of involvement and training. Educ Prim Care 2011;22(5):310–13. doi: 10.1080/14739879.2011.11494024.
4. Jones M, Kirtchuk L, Rosenthal J. GP registrars teaching medical students - an untapped resource? Educ Prim Care 2020;31(4):224–30. doi: 10.1080/14739879.2020.1749531.
5. Sternszus R, Cruess S, Cruess R, Young M, Steinert Y. Residents as role models: Impact on undergraduate trainees. Acad Med 2012;87(9):1282–87. doi: 10.1097/ACM.0b013e3182624c53.