Seeing the whole picture: evaluating automated assessment systems

dc.creator	Debra Trusso Haley
dc.creator	Pete Thomas
dc.creator	Anne De Roeck
dc.creator	Marian Petre
dc.date	2007-10-01T00:00:00Z
dc.date.accessioned	2015-07-20T22:38:19Z
dc.date.available	2015-07-20T22:38:19Z
dc.identifier	1473-7507
dc.identifier	https://doaj.org/article/a88c9b73084048469dadb2c57f12d8fd
dc.identifier.uri	http://evidence.thinkportal.org/handle/123456789/23136
dc.description	This paper argues that automated assessment systems can be useful for both students and educators provided that its results correspond well with human markers. Thus, evaluating such a system is crucial. We present an evaluation framework and show why it can be useful for both producers and consumers of automated assessment. The framework builds on previous work to analyse Latent Semantic Analysis- (LSA) based systems, a particular type of automated assessment, that produced a research taxonomy that could help developers publish their results in a format that is comprehensive, relatively compact, and useful to other researchers. The paper contends that, in order to see a complete picture of an automated assessment system, certain pieces must be emphasised. It presents the framework as a jigsaw puzzle whose pieces join together to form the whole picture and provides an example of the utility of the framework by presenting some empirical results from our assessment system that marks questions about html. Finally, the paper suggests that the framework is not limited to LSA-based systems. With slight modifications, it can be applied to any automated assessment system.
dc.language	English
dc.publisher	Higher Education Academy Subject Network for Information & Computer Sciences
dc.relation	http://www.ics.heacademy.ac.uk/italics/vol6iss4/Haleyetal.pdf
dc.relation	https://doaj.org/toc/1473-7507
dc.source	ITALICS, Vol 6, Iss 4, Pp 203-224 (2007)
dc.subject	automated assessment systems
dc.subject	computer aided assessment
dc.subject	CAA
dc.subject	Latent Semantic Systems
dc.subject	LSA systems
dc.subject	teaching programming
dc.subject	Special aspects of education
dc.subject	LC8-6691
dc.subject	Education
dc.subject	L
dc.subject	DOAJ:Education
dc.subject	DOAJ:Social Sciences
dc.subject	Special aspects of education
dc.subject	LC8-6691
dc.subject	Education
dc.subject	L
dc.subject	DOAJ:Education
dc.subject	DOAJ:Social Sciences
dc.subject	Special aspects of education
dc.subject	LC8-6691
dc.subject	Education
dc.subject	L
dc.subject	Special aspects of education
dc.subject	LC8-6691
dc.subject	Education
dc.subject	L
dc.subject	Special aspects of education
dc.subject	LC8-6691
dc.subject	Education
dc.subject	L
dc.title	Seeing the whole picture: evaluating automated assessment systems
dc.type	article