Largest remote clinical exam in the southern hemisphere goes off without a hitch

Four days of exams spaced out over 13 days.
 
More than 1100 candidates undertaking over 14,100 individual examination sessions.
 
An assessment team of around 450 examiners and 30-plus quality assurance examiners.
 
All backed by more than 130 RACGP staff members across 27 teams.
 
That was what it took to deliver the RACGP’s inaugural Remote Clinical Exam (RCE), said to be the largest such exam in the southern hemisphere. And it all went almost without a hitch.
 
RACGP General Manager of Education Services Dr Genevieve Yates told newsGP she is ‘absolutely delighted’ with the success of the exam, which was introduced as a COVID-safe replacement for the face-to-face Objective Structured Clinical Examination (OSCE). 
 
‘It really has been a mammoth team effort,’ she said.
 
‘Of literally thousands of exam sessions, only 12 had delays. We managed the four days without any major incidents, and the minor ones were managed smoothly and professionally. We’ve had very good feedback from RTOs [Regional Training Organisations], examiners and candidates.
 
‘We will take the lesson from this exam and apply them to the next.
 
‘We are also really happy with the preliminary statistical metrics of the exam and how reliable it is. We are quite confident it will have done its job in working out the competencies of our candidates.
 
‘Educationally, our exam seems to have performed very well on statistical measures, noting that much further analysis is yet to be done.’
 
RCE Project Manager Karen Connaughton told newsGP she felt privileged to be chosen to lead the way, together with RCE Clinical Lead Dr Karin Jodlowski-Tan.
 
‘Karin and I came together early in the piece to think about how we could develop an exam with the same academic rigour but by a remote method,’ Ms Connaughton said.
 
Dr Jodlowski-Tan said the team consulted externally and sought advice from international experts to find robust ways of conducting this type of examination. Eventually, it settled on using the video meeting platform Zoom, given how well it had dealt with the spike in demand across the globe at the beginning of the pandemic.
 
The RCE team examined many possible issues that could affect the exam, ranging from security to candidate experience to accessibility to internet speeds to reliability for rural candidates.
 
‘We were testing skills such as structured and systematic ways of thinking, problem-solving and clinical reasoning. That was a major focus,’ Dr Jodlowski-Tan told newsGP. 
 
As the exam unfolded, Dr Jodlowski-Tan and the rest of the team could watch events unfold in real-time.
 
‘One advantage of this technology was that we could jump from hub to hub across the country,’ Dr Jodlowski-Tan said.
 
‘Previously, we wouldn’t have had direct eyes on what was happening from a project manager or clinical lead perspective, as it was state-team based.’
 
The RCE saw the use of a model of sequential testing, where data analysis after the first three days determined which cohort had already demonstrated competence, which had not, and which cohort was borderline. The borderline cohort ­– around 20% of participants – did the last exam on day four, while those who had either already passed or already failed did not need to.
 
‘This model worked exceptionally well,’ Ms Connaughton said.
 
While it is often assumed that remote exams are less labour intensive, Ms Connaughton said the opposite was the case with the RCE.
 
‘It was extremely resource-intensive, significantly higher than face-to-face,’ she said. ‘You had a huge cohort of staff involved, who had a lot of buy in, and we were expecting examiners to do something completely different.
 
‘They were just incredible, how they took on that new role and ran with it. They learned new technology and managed that, letting candidates scroll down their screen while assessing them and marking on a separate device.
 
‘It was a big ask, and they took on that task very well.’ 
 
Despite this, Dr Jodlowski-Tan said the new model has shown error rates among examiners to be very low.
 
‘It was all very new for everyone and examiners had to learn to assess in a very different way to the OSCE. But the rates of error were so low as to be almost negligible,’ she said.
 
‘I’m very much relieved and proud, because other medical colleges who have put their exams on hold will be looking to us.
 
‘We’ve shown it can be done.’
 
Log in below to join the conversation.

exams GPs in training RCE remote clinical exam