Largest remote clinical exam in the southern hemisphere goes off without a hitch

Doug Hendrie

6/11/2020 3:48:17 PM

The COVID pandemic posed real challenges to the RACGP’s traditional clinical exams – but the new digital RCE has performed well, organisers say.

Young woman on video call
The RCE saw more than 1100 candidates undertake over 14,100 individual examination sessions.

Four days of exams spaced out over 13 days.
More than 1100 candidates undertaking over 14,100 individual examination sessions.
An assessment team of around 450 examiners and 30-plus quality assurance examiners.
All backed by more than 130 RACGP staff members across 27 teams.
That was what it took to deliver the RACGP’s inaugural Remote Clinical Exam (RCE), said to be the largest such exam in the southern hemisphere. And it all went almost without a hitch.
RACGP General Manager of Education Services Dr Genevieve Yates told newsGP she is ‘absolutely delighted’ with the success of the exam, which was introduced as a COVID-safe replacement for the face-to-face Objective Structured Clinical Examination (OSCE). 
‘It really has been a mammoth team effort,’ she said.
‘Of literally thousands of exam sessions, only 12 had delays. We managed the four days without any major incidents, and the minor ones were managed smoothly and professionally. We’ve had very good feedback from RTOs [Regional Training Organisations], examiners and candidates.
‘We will take the lesson from this exam and apply them to the next.
‘We are also really happy with the preliminary statistical metrics of the exam and how reliable it is. We are quite confident it will have done its job in working out the competencies of our candidates.
‘Educationally, our exam seems to have performed very well on statistical measures, noting that much further analysis is yet to be done.’
RCE Project Manager Karen Connaughton told newsGP she felt privileged to be chosen to lead the way, together with RCE Clinical Lead Dr Karin Jodlowski-Tan.
‘Karin and I came together early in the piece to think about how we could develop an exam with the same academic rigour but by a remote method,’ Ms Connaughton said.
Dr Jodlowski-Tan said the team consulted externally and sought advice from international experts to find robust ways of conducting this type of examination. Eventually, it settled on using the video meeting platform Zoom, given how well it had dealt with the spike in demand across the globe at the beginning of the pandemic.
The RCE team examined many possible issues that could affect the exam, ranging from security to candidate experience to accessibility to internet speeds to reliability for rural candidates.
‘We were testing skills such as structured and systematic ways of thinking, problem-solving and clinical reasoning. That was a major focus,’ Dr Jodlowski-Tan told newsGP
As the exam unfolded, Dr Jodlowski-Tan and the rest of the team could watch events unfold in real-time.
‘One advantage of this technology was that we could jump from hub to hub across the country,’ Dr Jodlowski-Tan said.
‘Previously, we wouldn’t have had direct eyes on what was happening from a project manager or clinical lead perspective, as it was state-team based.’
The RCE saw the use of a model of sequential testing, where data analysis after the first three days determined which cohort had already demonstrated competence, which had not, and which cohort was borderline. The borderline cohort ­– around 20% of participants – did the last exam on day four, while those who had either already passed or already failed did not need to.
‘This model worked exceptionally well,’ Ms Connaughton said.
While it is often assumed that remote exams are less labour intensive, Ms Connaughton said the opposite was the case with the RCE.
‘It was extremely resource-intensive, significantly higher than face-to-face,’ she said. ‘You had a huge cohort of staff involved, who had a lot of buy in, and we were expecting examiners to do something completely different.
‘They were just incredible, how they took on that new role and ran with it. They learned new technology and managed that, letting candidates scroll down their screen while assessing them and marking on a separate device.
‘It was a big ask, and they took on that task very well.’ 
Despite this, Dr Jodlowski-Tan said the new model has shown error rates among examiners to be very low.
‘It was all very new for everyone and examiners had to learn to assess in a very different way to the OSCE. But the rates of error were so low as to be almost negligible,’ she said.
‘I’m very much relieved and proud, because other medical colleges who have put their exams on hold will be looking to us.
‘We’ve shown it can be done.’
Log in below to join the conversation.

exams GPs in training RCE remote clinical exam

newsGP weekly poll What is your chief concern with role substitution?

newsGP weekly poll What is your chief concern with role substitution?



Login to comment

Dr Moe Swe   7/11/2020 7:53:15 AM

Was RCE counted as a “clinical exam” while candidates were commenting some of the exam questions were factual recalls and some were not case based? I gathered many candidates said RCE was just an oral KFP. The quality and purpose of previous OSCE were lost. Other medical colleges should not follow the models of exam questions of RCE by RACGP. Other weaknesses was about the integrity of the exam as the candidates who sat the exam at very different times of the exam were asked exactly the same questions. Yes, there could be some candidates who live in the same household and did the exams at different times.

Dr Jasmin Smyth   7/11/2020 8:29:07 AM

'We were testing skills such as structured and systematic ways of thinking, problem-solving and clinical reasoning,' which is what the KFP already assesses. Many participants commented that this exam was like 'an oral KFP' and the study they had undertaken which would have served them well in the previous clinical exam model (the now defunct OSCE) was essentially useless. With no examination of clinical skills, how is this a clinical exam?

Dr Ruo Zhong Xie   7/11/2020 9:30:19 AM

Not really the clinical exam. Some of the Questions were poorly formed, computer screen scrolling significantly affects reading the case. Most concern is no quarantine btw candidates at different time points so very easily ends up with cheating!!! It’s a joke to use this exam to assess clinical competence and other colleges should not learn from it!

Dr Yee Pyone   7/11/2020 10:49:01 AM

Congratulations to the whole team of 2020.1 RCE candidates, examiners and the key staff members for the successful online clinical exams! This unprecedented challenging 2020.1 RACGP clinical exam was accomplished well before the end of 2020 which is one of the great news amongst full of uncertainties of Covid-19 dramas all over the world. I believe the RACGP has been fully equipped with the best technology back up and adequate contingency plans for unforseeable technology glitches. The reputation lost by the RACGP during the 2020.1 KFP/ AKT online proctered exams on 9th & 10th October ( taken by a larger number of candidates-over 1400 KFP participants & over 1100 AKT participants ) has been regained. Thank you to the RACGP team for all your relentless hardwork supporting the GPs to move on to the next higher level of standards to serve the Australians.

rural gp   7/11/2020 1:47:48 PM

Assuming all the data is in, why does it take so long for a result? Is it another month?
This is a genuine question, there may be a valid reason.
But, if it just “we need to have meetings and more meetings to approve” , I think that would be unacceptable. GP’s have all learnt to be service orientated, to undertake teleconferencing, and to be nimble. Inertia suggests arrogance once again.