The RCoA examinations: The weakest link in anaesthetic training?
The purpose of examinations has changed over time. Yuval Noah Harrari writes in his book,
Homo Deus:
“Originally, schools were supposed to focus on enlightening and educating students, and
marks were merely a means of measuring success. But naturally enough schools soon began
focusing on achieving high marks. As every child, teacher and inspector knows, the skills
required to get high marks in an exam are not the same as a true understanding of literature,
biology or mathematics. Every child, teacher and inspector also knows that when forced to
choose between the two, most schools will go for the marks” [1].
For the RCoA exams, the balance between ‘enlightening and
educating’ and accumulation of marks seems to tip firmly
towards the latter. Although this is common for many schools
and colleges, the RCoA relies particularly heavily on testing
knowledge of minutiae in order to separate candidates.
Obsolete respirometers, a variety of hygrometers, physical
principles and other basic science that permeate the exam may
be a useful test for assigning marks to differentiate between a
pass or fail. It is for the examinee, however, extremely tedious
studying material that is of limited or no relevance to practice. Furthermore, it does not differentiate between those who are
clinically competent and safe and those who are not. Many very
competent doctors have been lost to training, or the speciality
entirely, because of the RCoA exam hurdle.
The GMC state that “The FRCA examinations are high-stake
summative assessments that have the potential to impact on
trainee careers and patient safety.” A life consuming effort is
required by most to pass the exam. Moreover, it presents a
significant opportunity to develop and improve practice. It is
then, of course, incumbent on the college to write an exam that
pertains to modern anaesthetic practice.
Box 1, a question from the current RCoA Primary guide, is a
well-written question requiring some knowledge of membrane
potentials and deductive thinking [2]. However, prioritising
knowledge of membrane potentials in the context of the large
amount of clinically relevant knowledge required for safe and
high-quality care is questionable. It is important to remember
that this question displaces a question on, for example, difficult
airway management, trauma and resuscitation, TIVA or regional
anaesthesia. The latter two, increasingly cornerstones of modern
practice, were entirely absent from both my Primary and Final
exams. Box 2 is a question from the current RCoA Final guide [3].
RA-UK’s guide to the regional anaesthesia part of the RCoA
curriculum contains a collection of papers and other resources
with the ultimate goal of “providing safe and effective regional
anaesthesia”. This goal-orientated practical material is much
more engaging for the trainee and more likely to be committed
to memory than material from either of the above questions.
To the credit of the College, the exam enhances the perception
of the specialty, forms an exacting test requiring perserverance
and commitment, and has certainly improved over time
with regard to its content. Furthermore, improvements have
undoubtedly been made regarding the quality of the exam,
notably a gradual move away from a bank of low-quality MTFs.
Nonetheless, last year my Final exam mark was based entirely on
the MCQ part of the exam in consequence of a significant error
in the provision of the CRQ exam.
Certainly, we can all take pride in passing an exam that is
relatively difficult. However, anaesthetists’ development seems to
be suffering at the expense of indulging the College with regard
to exam content. The reality is that once the exam is passed, most
anaesthetists jettison much of this knowledge from memory in
order to make way for that which is useful for them and the care
of their patients.
Ben McCartney
ST6 Anaesthetic trainee
Altnagelvin Area Hospital, Derry
References
- Harari YN. Homo Deus, a brief history of tomorrow. New
York: Vintage, 2017.
- Wilkinson M. Guide to the FRCA examination. The Primary,
4th edition. Oxford: Blackwell’s, 2013.
- Guide to the FRCA examination – The Final, 4th Edition.
London: Royal College of Anaesthetists, 2020.
A response
Thank you for the opportunity to respond to this letter, which
raises some interesting points and challenges. The College fully
recognises and acknowledges the personal impact and immense
stress to anaesthetists in training, and others, when studying for
the FRCA examinations. This impact was echoed in the recent
external review that we commissioned of the FRCA as well as
our Faculty examinations (FFPMRCA and FFICM) [1]. In response
we have published a programme for the development of these
exams, with an Examinations Development and Assurance Group
(EDAG) being constituted to implement the necessary changes
over the coming months and years. Significantly, this review
includes a consideration of the impact of the assessment burden
on anaesthetists in training. The College is also committed to
giving anaesthetists in training a greater role in the development
of the exams; for example, there will be trainee representation on
the EDAG for all the exams.
It is important to emphasise that the exam items and questions
are under constant review to ensure their validity and reliability.
There is inevitably a degree of inertia in the system of question
writing, reviewing, and editing as it takes a large amount of
time and work to develop new question items that are valid and
reliable. Currently this burden largely falls on examiners, but
we are looking at how we can open this up to a larger group of
question writers. The Examinations Committee is considering how interested anaesthetists, including anaesthetists in
training, can be involved in the development and writing
of exam materials, with the examiner body editing and
refining to ensure quality assurance and alignment with the
curriculum.
Many anaesthetists will be aware of doctors who are
competent in the clinical environment but have struggled to
pass exams. This is an area requiring balance and judgment. If
the standard of the exam is too high, too few doctors will pass
through the system having an impact on both individuals
and healthcare provision; but at the same time it is a matter
of great importance that high standards are maintained.
In particular, an understanding of the basic science that
underpins clinical practice is a core element of anaesthetic
knowledge and should remain so. However, testing in all
areas of the curriculum must be proportionate and balanced.
There is limited evidence to compare postgraduate medical
examination performance with clinical outcomes [2], and the
external review has recommended that further research in
this area should be considered. To address this, the exams
team and senior examiners will be working with anaesthetists
in training, on secondment as part-time education fellows, to
conduct research projects looking at exam validity and the
attainment gap, as suggested in the review.
Finally, it is important to remember that examinations are
only one part of the College’s assessment strategy. High stakes
examinations are deemed by most professions to be
a necessary part of such a strategy, but organisations such
as ours must remain open to feedback to drive changes. The
exam in its current format is fit for purpose and approved by
the GMC, but the recent exam reviews provide the College
with an opportunity to make some significant changes to
align our exams better with contemporary assessment theory
and practice, and make sure that the overall assessment
burden is proportionate.
Roger Sharpe
Chair, FRCA Examinations
Royal College of Anaesthetists
References
- Royal College of Anaesthetists. Independent review
of the assessment processes of the Royal College of
Anaesthetists, 2023. https://rcoa.ac.uk/sites/default/files/documents/2023-02/Independent-review-assessment-processes-RCoA-FINAL.pdf (accessed 17/5/2023).
- Wakeford R, Ludka K, Woolf K, McManus IC. Fitness to
practise sanctions in UK doctors are predicted by poor
performance at MRCGP and MRCP(UK) assessments:
data linkage study. BMC Medicine 2018; 16: 1-16.