A direct functional measure of text quality: Did the reader understand?

authored by: Joachim Grabowski, Moti Mathiebe
Abstract: Assessing text quality as an indication of underlying skills still remains challenging; irrespective of the approach, many studies struggle with reliability or validity problems. If writing is considered problem-solving, a report must make the reader understand the described situation and call for its mental reconstruction. Therefore, text quality may not only comprise linguistic aspects but also the cognitive-functional power of a text. The presented study aims at exploring the functionality of students’ reporting texts in relation to general text-quality measures, using a corpus of accident reports written by German fifth- and ninth-graders (n = 277) prompted by a pictorial stimulus of a bike accident scenario. An online tool was developed in which 277 university students graphically reenacted the situation from one respective text according to the existence, position, and color of the involved elements. Thereafter, the match of the resulting spatial reconstructions with the original situation was assessed by two raters. While most subscales showed sufficiently high interrater reliabilities, the aggregated functionality score (α =.74) had medium-high correlations with other text-quality ratings and was comparably dependent on grade, education level, and linguistic family background. However, the correlational pattern, regression analysis, and factor analysis showed that the functionality score also contributed unique portions of variance to the assessment of writing skill that were not represented by rating measures. Moreover, the direct indication of whether a text allows for the reader’s adequate cognitive representation is evident. Altogether, the approach of indicating text functionality through practical understanding offers a sound, though empirically laborious, alternative for text-quality measurement. Results are discussed with regard to the didactical strategy according to which students can improve their writing when they observe whether others can make use of their texts.
Organisation(s): Institute of Psychology
Type: Article
Journal: Written Communication
Volume: 41
Pages: 203-229
No. of pages: 27
ISSN: 0741-0883
Publication date: 04.2024
Publication status: Published
Peer reviewed: Yes
ASJC Scopus subject areas: Communication, Literature and Literary Theory
Electronic version(s): https://doi.org/10.1177/07410883231222952 (Access: Open)

BibTeX

@article{72463f35a5164ed9981c56cc406807cf,
title = "A direct functional measure of text quality: Did the reader understand?",
abstract = "Assessing text quality as an indication of underlying skills still remains challenging; irrespective of the approach, many studies struggle with reliability or validity problems. If writing is considered problem-solving, a report must make the reader understand the described situation and call for its mental reconstruction. Therefore, text quality may not only comprise linguistic aspects but also the cognitive-functional power of a text. The presented study aims at exploring the functionality of students{\textquoteright} reporting texts in relation to general text-quality measures, using a corpus of accident reports written by German fifth- and ninth-graders (n = 277) prompted by a pictorial stimulus of a bike accident scenario. An online tool was developed in which 277 university students graphically reenacted the situation from one respective text according to the existence, position, and color of the involved elements. Thereafter, the match of the resulting spatial reconstructions with the original situation was assessed by two raters. While most subscales showed sufficiently high interrater reliabilities, the aggregated functionality score (α =.74) had medium-high correlations with other text-quality ratings and was comparably dependent on grade, education level, and linguistic family background. However, the correlational pattern, regression analysis, and factor analysis showed that the functionality score also contributed unique portions of variance to the assessment of writing skill that were not represented by rating measures. Moreover, the direct indication of whether a text allows for the reader{\textquoteright}s adequate cognitive representation is evident. Altogether, the approach of indicating text functionality through practical understanding offers a sound, though empirically laborious, alternative for text-quality measurement. Results are discussed with regard to the didactical strategy according to which students can improve their writing when they observe whether others can make use of their texts.",
keywords = "rating, reports, web-based experiment, writing assessment, writing skill",
author = "Joachim Grabowski and Moti Mathiebe",
note = "The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a grant from the German Ministry of Education and Research (BMBF) to the first author for a research project on subcomponents of writing literacy (grant number 01GJ1208A).",
year = "2024",
month = apr,
doi = "10.1177/07410883231222952",
language = "English",
volume = "41",
pages = "203--229",
journal = "Written Communication",
issn = "0741-0883",
publisher = "SAGE Publications Inc.",
number = "2",
}