Irrelevance of Reliability Coefficients to Accountability Systems: Statistical Disconnect in Kane-Staiger

dc.creator	David Rogosa
dc.date	2005-04-01T00:00:00Z
dc.date.accessioned	2015-07-20T22:16:45Z
dc.date.available	2015-07-20T22:16:45Z
dc.identifier	2150-6477
dc.identifier	https://doaj.org/article/c57e61bb36ca48c39f1a3c0564a3bace
dc.identifier.uri	http://evidence.thinkportal.org/handle/123456789/18631
dc.description	The body of this report consists of a fairly thorough effort to discredit the empirical assertions and methodological prescriptions of Kane and Staiger (KS). The four main sections of content that follow this (lengthy) Preamble are:Section 1 Accuracy Of Group SummariesExact results are obtained for the accuracy of grade-level scores (forn=68) which are then compared with the reliability-style calculationsreported in KS for North Carolina data. Also, accuracy properties ofCalifornia API school-level scores are presented, and to compare with KS assertions, the reliability coefficients for these scores are calculated. KS find high volatility even when accuracy is very good, and KS find extreme absence of volatility even when accuracy is moderate to poor.Section 2 Accuracy of ImprovementPrecision of improvement is contrasted with KS-style reliability ofimprovement. Analytic and empirical examples for accuracy of improvement reinforce the basic message: reliability is not precision. Most importantly, precision, which is what matters, can be low, and reliability still be high. And vice versa. Also, school-level California API data display no relation between amount of improvement and uncertainty in the scores (Figures 2.1-2.3), refuting a key KS assertion about school size.Section 3 Persistence of Change.The KS correlation of consecutive changes--and thus the KS estimate of"proportion of variance in changes due to nonpersistent factors"--isshown to be a function of the reliability of the difference score. KSdeterminations of persistence of change are shown to be without valuein accountability systems. Common-sense definitions of consistency ofimprovement and empirical demonstrations using artificial data arepresented.Section 4 California Academic Performance Index Award ProgramsDiscussion of appropriate methods for describing the properties of Award Programs (e.g., determinations of false positive and false negatives) are contrasted with the incorrect empirical assertions and methodologies in KS. Counterexamples to each of the KS "Lessons" are presented in detail. The focus is on the effect of school size, to link with the accuracy results of previous sections.
dc.language	English
dc.publisher	Nonpartisan Education Review
dc.relation	http://www.nonpartisaneducation.org/Review/Resources/StatisticalDisconnect.htm
dc.relation	https://doaj.org/toc/2150-6477
dc.source	Nonpartisan Education Review , Vol 1, Iss 4, Pp 1-78 (2005)
dc.subject	education
dc.subject	policy
dc.subject	Kane
dc.subject	Staiger
dc.subject	Rogosa
dc.subject	API
dc.subject	STAR
dc.subject	California state testing
dc.subject	test score volatility
dc.subject	reliability coefficients
dc.subject	accountability
dc.subject	assessment
dc.subject	education
dc.subject	school
dc.subject	AYP
dc.subject	accountability index
dc.subject	Education (General)
dc.subject	L7-991
dc.subject	Education
dc.subject	L
dc.subject	DOAJ:Education
dc.subject	DOAJ:Social Sciences
dc.subject	Education (General)
dc.subject	L7-991
dc.subject	Education
dc.subject	L
dc.subject	DOAJ:Education
dc.subject	DOAJ:Social Sciences
dc.subject	Education (General)
dc.subject	L7-991
dc.subject	Education
dc.subject	L
dc.subject	Education (General)
dc.subject	L7-991
dc.subject	Education
dc.subject	L
dc.subject	Education (General)
dc.subject	L7-991
dc.subject	Education
dc.subject	L
dc.title	Irrelevance of Reliability Coefficients to Accountability Systems: Statistical Disconnect in Kane-Staiger
dc.type	article