2004
Volume 41, Issue 2
  • ISSN: 1573-9775
  • E-ISSN: 2352-1236

Abstract

Abstract

Manually annotated corpora of writing products may greatly contribute to writing research: they offer detailed insights in the quality of these texts, in the text features actually attended to by human text raters, in possibilities and difficulties for the use of automatic writing analytics and writing tools, and in the relations between different text quality dimensions. This paper presents the Utrecht System for Annotation of Learner text (USALT), that covers both general features (orthography, punctuation, wording, coherence) and genre-specific elements (such as openings, endings, structuring devices and politeness). The annotations contain up to three items (annotation unit; problem type; part-of-speech tag). USALT reflects various text quality dimensions, notably correctness, comprehensibility and appropriateness (both stylistically and in terms of genre conventions).

We present an USALT analysis of 371 texts produced by Dutch students from grades 7-9 (aged 12-15 years), taken from the so-called -corpus. The assignment concerned a letter about ‘typically Dutch things’ to a Swedish girl about to emigrate to The Netherlands. USALT reliabilities were adequate. In terms of problem frequency, we were struck by the pervasiveness of punctuation problems. Furthermore, the orthography and punctuation problems together present considerable difficulties for automatic analysis of original learner texts at this level. A remarkable result regarding relations between various text quality dimensions is that the frequency of orthography problems correlates higher with genre convention problems than with lexico-grammatical problems. We also used the annotations as predictors of the holistic scores assigned to the texts by human raters. Standardized annotation frequencies by themselves may account for 45% of the score variance, with a prominent role for annotations regarding genre elements; text length by itself explains 52%. The best model includes both text length and annotations (65% explained variance). In ongoing work, USALT is being extended to handle argumentative writing assignments.

Loading

Article metrics loading...

/content/journals/10.5117/TVT2019.2.001.PAND
2019-10-01
2024-12-26
Loading full text...

Full text loading...

/deliver/fulltext/15739775/41/2/01_TVT2019.2_PAND.html?itemId=/content/journals/10.5117/TVT2019.2.001.PAND&mimeType=html&fmt=ahah

References

  1. Bazerman, C.(2015). What do sociocultural studies of writing tell us about learning to write. In C.MacArthur, S.Graham & J.Fitzgerald (Eds.), Handbook of writing research (Vol. 2, pp. 11-23). New York: Guilford.
    [Google Scholar]
  2. Bochove, A. van, & Vermeer, A.(2015). Schrijvers in spe? Stilistische kenmerken in verhalen van groep 5 tot en met 8. Tijdschrift voor Taalbeheersing38(3), 305-336.
    [Google Scholar]
  3. Boekelder, A., Glopper, K. de, & Wijk. C. van(1991). Tekstanalytisch onderzoek van een functionele schrijfopdracht: de inhoud, opbouw en beoordeling van excuusbrieven. Toegepaste Taalwetenschap in Artikelen, 40, 23-34.
    [Google Scholar]
  4. Bouwer, R., Koster, M.Van der Bergh, H.(2016). Benchmark rating procedure, best of both worlds? Comparing procedures to rate text quality in a reliable and valid manner. Manuscript submitted for publication.
    [Google Scholar]
  5. Chenoweth, N.A., & Hayes, J.R.(2001). Fluency in writing: Generating text in L1 and L2. Written communication, 18(1), 80-98.
    [Google Scholar]
  6. Chodorow, M., & Burstein, J.(2004). Beyond essay length: evaluating e‐rater®’s performance on toefl® essays. ETS Research Report Series, 2004(1).
    [Google Scholar]
  7. Christie, F.(2010). The ontogenesis of writing in childhood and adolescence. In D.Wyse, R.Andrews and J.Hoffman (Eds.) The Routledge international handbook of English, language and literacy teaching (pp. 146-58). Oxford: Routledge.
    [Google Scholar]
  8. Deane, P., & Quinlan, T.(2010). What automated analyses of corpora can tell us about students’ writing skills. Journal of Writing Research2, 151-177.
    [Google Scholar]
  9. Díaz-Negrillo, A., & Domínguez, J.F.(2006). Error tagging systems for learner corpora. Revista española de lingüística aplicada, (19), 83-102.
    [Google Scholar]
  10. Fayol, M.(2016). From language to text: The development and learning of translation. In C. A.MacArthur, Graham, S., & Fitzgerald, J. (Eds.), Handbook of writing research (pp. 130-143). New York, NY: Guilford.
    [Google Scholar]
  11. Glopper, K. de.(1984). Opstelkenmerken en opstelbeoordelingen. Tijdschrift voor Taalbeheersing, 6, 176-189.
    [Google Scholar]
  12. Glopper, K. de, & Prenger, J.(2013). Schrijfmeters maken. In A.Mottart & S.Vanhooren (Red.), Zevenentwintigste conferentie Onderwijs Nederlands (pp. 86-89). Gent: Academia Press.
    [Google Scholar]
  13. Hawkins, J.A., & Buttery, P.(2010). Criterial features in learner corpora: Theory and illustrations. English Profile Journal, 1(1), e5.
    [Google Scholar]
  14. Jisa, H., Reilly, J.S., Verhoeven, L., Baruch, E., & Rosado, E.(2002). Passive voice constructions in written texts: A cross-linguistic developmental study. Written Language & Literacy, 5(2), 163-181.
    [Google Scholar]
  15. MacArthur, C.A., & Graham, S.(2016). Writing research from a cognitive perspective. In C. A.MacArthur, S.Graham & J.Fitzgerald (Eds.), Handbook of writing research (pp. 24-40). New York, NY: Guilford.
    [Google Scholar]
  16. Myhill, D.(2008). Towards a linguistic model of sentence development in writing. Language and Education, 22(5), 271-288.
    [Google Scholar]
  17. Nicholls, D.(2003). The Cambridge Learner Corpus: Error coding and analysis for lexicography and ELT. In Proceedings of the Corpus Linguistics 2003 conference (Vol. 16, pp. 572-581).
    [Google Scholar]
  18. Pander Maat, H., Raaijmakers, K., Veerbeek, J., & Vermeulen, D.(2017). Handleiding Utrechts systeem voor annotatie van leerlingstekst. Interne publicatie UniversiteitUtrecht.
    [Google Scholar]
  19. Pollmann, E., Prenger, J., & Glopper, K. D.(2012). Het beoordelen van leerlingteksten met behulp van een schaalmodel. Levende Talen Tijdschrift, 13(3), 15-24.
    [Google Scholar]
  20. Prior, P.(2006). A sociocultural theory of writing. In C. A.MacArthur, S.Graham & J.Fitzgerald (Eds.), Handbook of writing research (pp. 54-66). New York, NY: Guilford.
    [Google Scholar]
  21. Rijlaarsdam, G., & Van den Bergh, H.(2006). Writing process theory. In C. A.MacArthur, S.Graham & J.Fitzgerald (Eds.), Handbook of writing research (pp. 41-53). New York, NY: Guilford.
    [Google Scholar]
  22. Sanders, T., & Schilperoord, J.(2006). Text structure as a window on the cognition of writing. In C. A.MacArthur, S.Graham & J.Fitzgerald (Eds.), Handbook of writing research (pp. 386-402). New York, NY: Guilford.
    [Google Scholar]
  23. Tolchinsky, L.(2016). From text to language and back: The emergence of written language. In C.A.MacArthur, S.Graham & J.Fitzgerald (Eds.), Handbook of writing research (pp. 144-159). New York, NY: Guilford.
    [Google Scholar]
  24. Torrance, M., & Galbraith, D.(2006). The processing demands of writing. In C.A.MacArthur, S.Graham & J.Fitzgerald (Eds.), Handbook of writing research (pp. 67-80). New York, NY: Guilford.
    [Google Scholar]
  25. Upton, T.A., & Cohen, M.A.(2009). An approach to corpus-based discourse analysis: The move analysis as example. Discourse Studies, 11(5), 585-605.
    [Google Scholar]
  26. VandeKerckhove, R., & Sandra, D.(2015). De potentiële impact van informele online communicatie op de spellingpraktijk van Vlaamse tieners in schoolcontext. Tijdschrift voor Taalbeheersing38(3), 201-234.
    [Google Scholar]
  27. Verheijen, L.(2013). The effects of text messaging and instant messaging on literacy. English studies, 94(5), 582-602.
    [Google Scholar]
  28. Verhoeven, L., Aparici, M., Cahana-Amitay, D., Hell, J. V., Kriz, S., & Viguié-Simon, A.(2002). Clause packaging in writing and speech: A cross-linguistic developmental analysis. Written Language & Literacy, 5(2), 135-161.
    [Google Scholar]
/content/journals/10.5117/TVT2019.2.001.PAND
Loading
/content/journals/10.5117/TVT2019.2.001.PAND
Loading

Data & Media loading...

  • Article Type: Research Article
Keyword(s): annotation; learner text; text quality ratings; writing quality
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error