The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations

dc.creator	Dewhurst Neil G
dc.creator	Chis Liliana
dc.creator	McManus IC
dc.creator	Tighe Jane
dc.creator	Mucklow John
dc.date	2010-06-01T00:00:00Z
dc.date.accessioned	2015-07-20T22:24:59Z
dc.date.available	2015-07-20T22:24:59Z
dc.identifier	10.1186/1472-6920-10-40
dc.identifier	1472-6920
dc.identifier	https://doaj.org/article/b4324b8d00e746c386205b793889601e
dc.identifier.uri	http://evidence.thinkportal.org/handle/123456789/21305
dc.description	<p>Abstract</p> <p>Background</p> <p>Cronbach's alpha is widely used as the preferred index of reliability for medical postgraduate examinations. A value of 0.8-0.9 is seen by providers and regulators alike as an adequate demonstration of acceptable reliability for any assessment. Of the other statistical parameters, Standard Error of Measurement (SEM) is mainly seen as useful only in determining the accuracy of a pass mark. However the alpha coefficient depends both on SEM and on the ability range (standard deviation, SD) of candidates taking an exam. This study investigated the extent to which the necessarily narrower ability range in candidates taking the second of the three part MRCP(UK) diploma examinations, biases assessment of reliability and SEM.</p> <p>Methods</p> <p>a) The interrelationships of standard deviation (SD), SEM and reliability were investigated in a Monte Carlo simulation of 10,000 candidates taking a postgraduate examination. b) Reliability and SEM were studied in the MRCP(UK) Part 1 and Part 2 Written Examinations from 2002 to 2008. c) Reliability and SEM were studied in eight Specialty Certificate Examinations introduced in 2008-9.</p> <p>Results</p> <p>The Monte Carlo simulation showed, as expected, that restricting the range of an assessment only to those who had already passed it, dramatically reduced the reliability but did not affect the SEM of a simulated assessment. The analysis of the MRCP(UK) Part 1 and Part 2 written examinations showed that the MRCP(UK) Part 2 written examination had a lower reliability than the Part 1 examination, but, despite that lower reliability, the Part 2 examination also had a <it>smaller </it>SEM (indicating a more accurate assessment). The Specialty Certificate Examinations had small Ns, and as a result, wide variability in their reliabilities, but SEMs were comparable with MRCP(UK) Part 2.</p> <p>Conclusions</p> <p>An emphasis upon assessing the quality of assessments primarily in terms of reliability alone can produce a paradoxical and distorted picture, particularly in the situation where a narrower range of candidate ability is an inevitable consequence of being able to take a second part examination only after passing the first part examination. Reliability also shows problems when numbers of candidates in examinations are low and sampling error affects the range of candidate ability. SEM is not subject to such problems; it is therefore a better measure of the quality of an assessment and is recommended for routine use.</p>
dc.language	English
dc.publisher	BioMed Central
dc.relation	http://www.biomedcentral.com/1472-6920/10/40
dc.relation	https://doaj.org/toc/1472-6920
dc.rights	CC BY
dc.source	BMC Medical Education, Vol 10, Iss 1, p 40 (2010)
dc.subject	Medicine (General)
dc.subject	R5-920
dc.subject	Medicine
dc.subject	R
dc.subject	DOAJ:Medicine (General)
dc.subject	DOAJ:Health Sciences
dc.subject	Special aspects of education
dc.subject	LC8-6691
dc.subject	Education
dc.subject	L
dc.subject	DOAJ:Education
dc.subject	DOAJ:Social Sciences
dc.subject	Medicine (General)
dc.subject	R5-920
dc.subject	Medicine
dc.subject	R
dc.subject	DOAJ:Medicine (General)
dc.subject	DOAJ:Health Sciences
dc.subject	Special aspects of education
dc.subject	LC8-6691
dc.subject	Education
dc.subject	L
dc.subject	DOAJ:Education
dc.subject	DOAJ:Social Sciences
dc.subject	Medicine (General)
dc.subject	R5-920
dc.subject	Medicine
dc.subject	R
dc.subject	Special aspects of education
dc.subject	LC8-6691
dc.subject	Education
dc.subject	L
dc.subject	Medicine (General)
dc.subject	R5-920
dc.subject	Medicine
dc.subject	R
dc.subject	Special aspects of education
dc.subject	LC8-6691
dc.subject	Education
dc.subject	L
dc.subject	Medicine (General)
dc.subject	R5-920
dc.subject	Medicine
dc.subject	R
dc.subject	Special aspects of education
dc.subject	LC8-6691
dc.subject	Education
dc.subject	L
dc.title	The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations
dc.type	article