External clinical teaching visits (ECTVs) are a key workplace-based assessment (WBA) and teaching method in general practice training in Australia.1 ECTVs involve an experienced, independent general practitioner (GP) educator (teaching visitor) visiting the practice for a half-day session of direct observation of general practice/rural generalist registrar consultations with patients. Teaching visitors can be medical educators, GP supervisors and/or other GPs. Visitors undertake formal induction and regular professional development.
ECTVs employ an adapted version of the mini-clinical evaluation exercise (mini-CEX) as a formative assessment tool and mechanism for feedback.2,3 More recently, ECTVs have also incorporated random case analysis (RCA),4 with formal case-based discussion likely to be added in the near future. By providing an independent assessment and feedback on direct observation of registrar performance, ECTVs offer both a powerful learning opportunity for registrars and, potentially, guidance on future supervision activities.5 Additionally, ECTVs can allow an independent review of the supervisor– registrar alliance, the quality of the learning environment and level of access to supervisory assistance.6
Most registrars training in the Australian General Practice Training (AGPT) program receive five ECTVs throughout their core training, up to two of which may be conducted remotely.7
The Royal Australian College of General Practitioners (RACGP) AGPT training site and supervisor handbook discusses the ‘opportunity for the supervisor and the ECT (external clinical teacher) visitor to discuss the registrar and their progress after the visit’.8 It follows that their educational utility should be maximised by high levels of supervisor and visitor engagement, including an effective discussion, as well as documentation and follow up of identified learning needs.
While ECTVs are perceived to have very high educational value by registrars and teaching visitors,9 the engagement with, and value for, supervisors has not been explored. The aim of this study was to describe the engagement with and perceived utility of ECTVs by GP supervisors. We also sought to identify the views of supervisors who undertook external visits themselves (visitor- supervisors).
Methods
This was a prospective cross-sectional study design with data drawn from the General Practice Supervision Australia (GPSA) online annual national survey.10 Invitations to participate were advertised on social media, in member communiques and on GPSA’s online community platform. Survey items were developed via an iterative review and refinement process involving subject and psychometric experts to ensure face, content and construct validity and reliability of measurement. Sociodemographic questions were primarily forced-choice, with open-text options. The use and perception of ECTVs was explored using 12 quantitative items to which participants responded using Likert-type scales (eg 1 = ‘not at all’ to 5 = ‘extremely’) and one open-text question. All responses were anonymous, and consent was implied by survey completion.
Data were collected in April–May 2024, downloaded from SurveyMonkey (SurveyMonkey Inc.) and imported into IBM SPSS Statistics V.30 (IBM). A total of 369 GP/rural generalist (RG) supervisors began the survey. Cases with 100% missing ECTV data were removed from the sample (n = 26). Little’s Missing Completely at Random (MCAR) test showed that missing data were not random (χ2 (172) = 32.8.5, P <0.05) and, subsequently, were not replaced. Data were explored using descriptive analyses (retaining all available cases), and group comparisons were performed using t-tests and Chi-squared tests (complete case analysis).
A high-level sentiment analysis was performed to categorise open-text qualitative data into two broad themes (ie positive and negative perspectives about ECTVs) following which an inductive content analysis was performed to categorise the data into subthemes (eg educational value).
Ethics approval was granted in April 2024 (Monash University #42111).
Results
Participants
Of the 4939 supervisors invited to participate via email, a total of 343 GP supervisors completed at least one of the 12 ECTV items in the survey (6.9% response rate). Sociodemographic and professional factors are presented in Table 1. Most respondents had more than 2 years supervision experience and had completed their primary medical degree in Australia. Most respondents resided in major cities in Victoria, New South Wales or Queensland; however, there was some representation across all Australia states/territories and non-urban locations.
| Table 1. Sociodemographic and professional characteristics of participants |
| Factor |
Category |
n |
% |
| Age (years) |
<45 |
51 |
14.9 |
| 45–54 |
91 |
26.5 |
| 55–64 |
124 |
36.2 |
| ≥65 |
71 |
20.7 |
| Not given |
6 |
1.7 |
| Gender identity |
Women |
162 |
47.2 |
| Men |
176 |
51.3 |
| Non-binary / Gender diverse / Prefer to self-describe |
3 |
0.9 |
| Prefer not to say |
2 |
0.6 |
| State/territory location of main training practice |
New South Wales |
76 |
22.2 |
| Victoria |
100 |
29.2 |
| Queensland |
81 |
23.6 |
| South Australia |
33 |
9.6 |
| Western Australia |
23 |
6.7 |
| Tasmania |
12 |
3.5 |
| Northern Territory |
10 |
2.9 |
| Australian Capital Territory |
8 |
2.3 |
| Region (Modified Monash category) |
MM1 (Metropolitan area: major city) |
157 |
45.8 |
| MM2 (Regional centre: population >50,000) |
61 |
17.8 |
| MM3 (Large rural town: population between 15,000 and 50,000) |
36 |
10.5 |
| MM4 (Medium rural town: population between 5000 and 15,000) |
36 |
10.5 |
| MM5 (Small rural town: population between 1000 and 5000) |
36 |
10.5 |
| MM6/MM7 (Remote/very remote community: population <1000) |
15 |
4.4 |
| Role (all that apply) |
General practitioner (GP) |
310 |
90.4 |
| Rural generalist |
52 |
15.2 |
| Medical educator |
51 |
14.9 |
| External clinical teaching visitor |
41 |
12 |
| GP academic |
26 |
7.6 |
| Practice manager |
3 |
0.9 |
| I have another role within general practice training |
9 |
2.6 |
| Supervision/medical educator experience |
<2 years |
15 |
4.6 |
| 2–5 years |
67 |
20.5 |
| 6–10 years |
95 |
29.1 |
| 11–20 years |
77 |
23.5 |
| ≥21 years |
73 |
22.3 |
Engagement with ECTVs
GP supervisors’ engagement with ECTVs is shown in Figure 1. As shown, most supervisors often/always discuss the ECTV with the visitor (80% of 330 respondents), often/always read the ECTV report (83% of 332 respondents) and often/always discuss the report with the registrar (87% of 330 respondents).
Figure 1. General practitioner supervisors’ use of and engagement with external clinical teaching visits (ECTVs).
GP supervisors who also perform ECTVs themselves (visitor-supervisors, n = 41) discussed the ECTV report with their registrars significantly more frequently than did non-visitor supervisors, t(87.7) = 3.98, P <0.001. There were no other differences between visitor and non-visitor GP supervisors in the engagement with ECTVs.
Most GP supervisors (58% of 322 respondents) reported that they spend less than 10 minutes with the teaching visitor during the ECTV. There were no differences between visitor-supervisors and non-visitor supervisors in the amount of time spent with teaching visitors, χ2(5, 326) = 9.31, P >0.05.
Perceived utility of ECTVs
GP supervisors were asked to indicate the perceived utility of ECTVs as an independent method of teaching and assessment. Most supervisors (88% of 322 respondents) reported that the ECTV feedback and report often/always aligned with their own assessment of registrar performance. Most supervisors (83% of 321 respondents) sometimes/often/always modify their teaching in response to ECTVs (refer to Figure 2).
Figure 2. General practitioner supervisors’ perceptions of teaching and assessment utility of external clinical teaching visits (ECTVs).
ECTVs were perceived as generally useful by GP supervisors across a range of practice domains (Figure 3, n = 331–333). The most useful domains were consultation and communication skills, with the least useful being clinical knowledge. Overall, 75% of supervisors (n = 248) felt the visits were very/extremely useful to assess the overall competence of the registrar.
Visitor-supervisors perceived greater utility in ECTVs in relation to consultation skills, t(330) = 3.10, P <0.01; communication skills, t(331) = 3.28, P <0.01; managing uncertainty, t(54.24) = 2.82, P <0.05; and professionalism, t(330) = 2.35, P <0.05, in comparison to non-visitor supervisors.
Figure 3. General practitioner supervisors’ perceptions of teaching and learning utility of external clinical teaching visits (ECTVs) across different domains of practice.
Qualitative responses
GP supervisors had the option to provide free-text responses about ECTVs as a teaching and assessment method. A total of 124 participants provided comments. Excluding miscellaneous comments (n = 36), 63% of comments reflected positive aspects of ECTVs, with most of these pertaining to overall educational value and the benefits of independent assessment. The remaining 37% of comments reflected negative aspects of ECTVs, with variability in the quality of the visit and limited visitor discussion with supervisors emerging as the strongest concerns. Themes, subthemes and illustrative quotes are shown in Table 2.
| Table 2. General practitioner (GP) supervisor perceptions of external clinical teaching visits (ECTVs) |
| Theme |
n |
Quote |
| Positive perceptions of ECTVs |
| Overall educational value |
19 |
‘Fantastic teaching method.’
‘It is an excellent method.’ |
| Specific educational benefits |
4 |
‘Catalyst for registrar goal setting and self-directed learning …’
‘Useful to build professional networks …’ |
| Independence of assessment |
22 |
‘ECTV are useful as they provide an independent assessment of the registrar’s performance.’
‘ECTV is very important to give a registrar a different viewpoint and chance to discuss different experiences and styles.’
‘Good way to sometimes raise issues that may be sensitive to deal with inhouse, allowing supervisor/registrar to remain positive ...’ |
| Value for poorly performing registrars |
4 |
‘I find these visits most useful when I have a registrar who is performing poorly. It gives the registrar better insight into their overall clinical and management skills and reinforces what I am trying to teach my registrars.’
‘I think it’s good to have an external experienced GP come in to provide feedback but in reality the value is only for the registrar who are lacking in skills …’ |
| Collaborative approach |
2 |
‘I think it is valuable for both the registrar and myself to have ECTVs, they allow the strengths and the needs of the registrar to be discussed in a collaborative manner.’
‘… shedding light to the performance of the registrar and a platform to discuss with another peer as to the progress and future direction of learning plans for the registrars …’ |
| Validation of supervisor performance |
4 |
‘… external validation of GP supervisor/teaching and training environment …’
‘ECTV validates our own assessments and to some extent provides a comparison with other supervisors …’ |
| Negative perceptions of ECTVs |
| Discussion between supervisor and visitor |
11 |
‘I often find that the visiting clinician devotes v(ery) little time to discussion. I don’t get positive or negative feedback so it is hard to gauge how or where to improve ...’
‘I rarely see the ECT(V) visitor, they haven’t engaged with me per se, come in and leave without discussion.’ |
| Variability in visitor quality and experience |
15 |
‘… (It) depend(s) entirely on the competence and experience of the ECTV visitor – so they can range from extremely useful to not at all useful (even counterproductive) depending on the skill and expertise of the visitor.’
‘They can be very variable, each ECTV supervisor has their own agenda, and not necessarily evidence based. Many also seem to feel they need to critique all aspects the registrar, when the time is so limited, and the setting is so artificial.’
‘… I have certainly had Registrars feel that this was a terrible and traumatic experience primarily due to the assessor …’
‘Occasionally can be destructive if ECT(V) is poorly behaved’ |
| Limited and variability of patient encounters |
7 |
‘… The drawback of ECTVs is the small sample size …’
‘… the information gleaned can be very skewed by the patients seen.’ |
| Miscellaneous |
| Other |
14 |
‘They are woefully underpaid ...’
‘They are onerous and impact on patient relationship and flow ...’ |
| No comment |
22 |
‘No’
‘Not applicable’ |
Discussion
The findings from this study demonstrate a high level of engagement with and perceived utility of ECTVs by GP supervisors in Australian general practice training. While the views of registrars and teaching visitors have previously been explored,6,9 this is the first study to describe supervisor perceptions of the ECTV.
Engagement
While GP supervisor engagement was good overall, we identified several areas where this could be enhanced. One in five supervisors reported that they sometimes, rarely or never had a discussion with the visitor, and over half spend less than 10 minutes doing so. In open-text responses, several supervisors raised the concern that teaching visitors do not seek them out for discussion prior to leaving the practice. While there are many practical barriers to an effective discussion between supervisor and visitor always occurring (time, availability, clinical demand), this is likely to diminish the educational utility of the visit.
Furthermore, we found that approximately one in six supervisors do not routinely read the visit report, and approximately one in eight do not routinely discuss the visit with the registrar afterwards. Similarly, this is likely to reduce the educational impact of these visits.
Utility
In general, supervisors perceived ECTVs as a very useful method of teaching and assessment. Overall, approximately three in four supervisors felt that ECTVs were very/extremely useful to assess overall competence of the registrar, and almost 90% thought that the visitor’s and their own assessment of registrar performance was strongly aligned. This sense of ‘validation’ was supported by free-text responses, showing that supervisors valued the opportunity to ‘gauge’ their assessment of registrars against an external assessor.
Most supervisors reported that the visit led to a change in aspects of their supervision practice. Free-text responses highlighted the value of the independence of the visitor, allowing for an ‘unbiased’ and ‘independent assessment’, with the provision of ‘fresh eyes’ and ‘another perspective’ assisting with identification of ‘unknown learning needs’. Supervisors also mentioned the benefit of the visitor raising potentially sensitive issues as an educator with no previous relationship with the registrar, helping to preserve the supervisor–registrar alliance. The benefits of external, independent educators as credible observers who ‘cross boundaries’ between different communities of practice have been previously described.11 These are also described merits of the ECTV in RACGP WBA documentation.12
The domains of practice in which supervisors felt the visit was most valuable (consultation and communication skills) were not a surprising finding. Visitors typically only observe 3–4 patients and conduct one RCA during the visit. While this limits the scope of clinical knowledge assessment, communication and consultation skills can readily be observed. This is consistent with registrar perceptions of the ECTV, with registrars identifying consultation and communication skill development as particular strengths of direct observation.5
Many supervisors commented on the ‘variability’ and ‘quality’ of teaching visitors and the impact of this on the educational utility of the ECTV. Concerns were raised about a range of issues, including the visitor being ‘biased’, ‘non-evidence based’ or ‘over-inclusive’ regarding feedback. Some supervisors flagged specific concerns about visits being ‘destructive’ or ‘traumatic’ for the registrar. It is well described that negative feedback can elicit feelings of humiliation and shame in medical learners.13,14 However, to our knowledge, this is the first time this has been reported in the context of the ECTV.
Visitor-supervisors
We found that visitor-supervisors had greater engagement and perceived utility across several areas when compared with non-visitor supervisors. This finding suggests a benefit in further recruitment and support of supervisors to undertake the teaching visitor role.
Limitations of the study and future research directions
It is possible that the relatively small sample size of GP supervisors (constituting less than 10% of the GP supervision workforce in Australia) may not be representative of all GP supervisors. Given that missing data were not missing at random, the responses may represent a selection bias towards those holding more extreme views – either positive or negative – towards ECTVs. Subgroup analyses indicated that ECTVs were generally viewed more positively by visitor-supervisors (n = 41) than by supervisors; sensitivity analyses indicated a modest impact on whole sample descriptives, with the overall means largely representative.
Nonetheless, future research is needed to confirm the generalisability of the findings to the broader GP supervision workforce. This includes exploration of the potential barriers to engagement (post-visit discussions) and the frequency and nature of ‘negative’ ECTV experiences for registrars. Such research may determine the need for and value of a range of possible interventions, including targeted training and resource development for GP supervisors and teaching visitors.
Conclusions
The findings from this study provide novel insights into the perceived utility and engagement of GP supervisors with ECTVs. While there is a high level of engagement and perceived utility of the ECTV by GP supervisors in Australian general practice training, we have identified several areas where the educational utility could be enhanced.