Description:
In an effort designed to guide and improve the assessment of a newly developed writing environment, the reliability and developmental and concurrent validity of a previously validated rubric developed for pen-and-paper-created narratives, Writing What You Read (WWYR), was determined when applied to hypermedia-authored narratives of children in second and third grade. Children (n = 60) from four intact classrooms produced hypermedia narratives (text, audio, graphic, and video elements) over a 4-month period in a school-based computer laboratory. Raters (n = 5) with knowledge of the teaching of process writing and use of hypermedia software judged the hypermedia narrative productions. Raters judged all students' (n = 60) hypermedia narrative productions individually without resolving differences through discussion. Two analyses were used to determine reliability: percentages of agreement and Pearson correlations. Percentages of agreement for the WWYR rubric averaged across 10 pairs of raters found high percentages of agreement among raters (.70 for ±0 and .99 for ±1). Pearson correlations averaged across 10 pairs of raters found acceptable interrater reliability for four of the five subscales. For the five subscales (Theme, Character, Setting, Plot, and Communication), the r values were .59, .55, .49, .50, and .50, respectively. Developmental validity of the WWYR scores was examined with one-way MANOVA to evaluate the WWYR scores of children grouped as low, medium, or high ability based on their Iowa Test of Basic Skills (ITBS) National Percentile Rank for Literacy Skill. Evidence for the developmental validity of the WWYR scores was supported across the three ability groups, F(2, 36) = 2.59, p <. 01. Concurrent validity was examined through correlational analysis between students' mean WWYR score and ITBS score. Scores from the two measures were positively correlated, r = .83, p < .01, providing evidence of the sensitivity of the WWYR assessment to measure the developmental literacy competency of the third-grade students. Results support teachers' use of a validated rubric developed for pen-and-paper-created narratives applied to hypermedia narratives, despite additional visual and audio narrative elements inherent to hypermedia. Implications for literacy teaching and learning with hypermedia address core questions about the similarities and differences between written textual expression and visual and verbally recorded expression. A revised and expanded WWYR rubric is proposed to begin to address these core questions generated by teachers' use of the hypermedia writing environment.