9.1 Background
Assessment is an integral part of good writing instruction. Its primary goal should be to help students learn to write better and to produce more effective/powerful texts (Huot & Perry, 2010; IRA & NCTE, 2010). Writing assessment involves, among other things, an evaluation of the quality of students' writing. Such evaluation is expected to help teachers not only identify students' levels of performance but, more importantly, provide insights into students' strengths and needs for the purpose of planning subsequent instruction or remediation. Currently available evaluation tools fall short of these goals, however.
For example, one of the tools recommended for language teachers (and researchers alike) is called L2 Syntactic Complexity Analyzer, or L2SCA (Lu, 2011; 陆小飞、许琪, 2016). An automated computational tool, L2SCA uses 14 different indices to measure syntactic complexity of second language writing, covering length of production units (as measured by mean length of sentence, mean length of T-unit, and mean length of clause), amounts of coordination (as measured by clause per sentence, verb phrase per T-unit, and clause per T-unit), amounts of subordination (as measured by dependent clause per clause, dependent clause per T-unit, and T-unit per sentence), degree of phrasal sophistication (as measured by coordinate phrase per T-unit, coordinate phrase per clause, complex nominal per T-unit, and complex nominal per clause), and overall sentence complexity (as measured by complex T-unit ratio). L2SCA has been found to be a valuable tool for corpus-based evaluation of student writing. Its classroom applications appear to have been more limited, however, due to concerns about its relevance, utility, and effectiveness for instructional purposes.
One issue with trying to make connections between syntactic complexity measures and writing quality is that judgments of quality should not be based on syntactic complexity alone, but on how aspects of syntax intersect with other meaning-making systems (e.g., lexical choices, Theme choices, visuals) in particular contexts (Gebhard & Martin, 2011). A related issue with L2SCA is that it does not account for function. Texts with high syntactic complexity scores may lack appropriateness, effectiveness, or power. As Martin (1996) pointed out, making a text syntactically complex may enhance the status of the text but does little to improve its discursive quality and functionality. A third criticism of the assessment tool was levied by Biber and Gray (2011), who questioned the validity of using T-units and clause subordination, which they found to be more characteristic of conversational texts, as measures of second language writing development. It is in part for these reasons that Lu (2019) has recently called for a functional turn in L2 writing syntactic complexity research. Specifically, he acknowledged that the quality of writing should not be determined solely by the presence and frequency of specific linguistic features. He calls for a shift away from a focus on form (i.e., syntactic complexity features) to a functional approach that systematically examines the genre appropriateness and functional effectiveness of syntactically complex structures. He suggests the urgent need to align syntactic complexity features with the rhetorical functions they are deployed to realize when judging the quality of writing.
Another popular tool for evaluating student writing is the rubric. Widely used in both high-stakes and classroom writing assessments, the rubric is regarded by many as an exact, efficient, and objective tool for evaluating student writing. In rubric-based writing evaluation, teachers develop several score levels, such as 1 to 6, with a description of what is expected at each level in terms of mechanics, content, organization, grammar, vocabulary, and voice. A popular rubric for evaluating student writing is the six-traits writing rubric. Figure 9-1 describes the expectations for the highest level (Level 6) of writing performance in a six-traits writing rubric.
| Ideas and Content: | * Exceptionally clear, focused, engaging with relevant, strong supporting details | |
| Organization: | * Effectively organized in logical and creative manner | |
| * Creative and engaging introduction and conclusion | ||
| Voice: | * Expressive, engaging, sincere | |
| * Strong sense of audience | ||
| * Shows emotion, humor, honesty, suspense or life | ||
| Word Choice: | * Precise, carefully chosen | |
| * Strong, fresh, vivid images | ||
| Sentence Fluency: | * High degree of craftsmanship | |
| * Effective variation in sentence patterns | ||
| Conventions: | * Exceptionally strong control of standard conventions of writing |
Figure 9-1 Six-Traits Writing Rubric: Level 6 (Fang & Wang, 2011, p. 148)
One problem with this rubric is that it is neither exact nor objective. It does not, for example, give teachers any hint of what it is that makes a piece of writing "exceptionally clear, focused, engaging" and "effectively organized". Nor does it specify what "high degree of craftsmanship" entails. Moreover, the rubric does not elaborate on what it means to be "creative", "fresh", "engaging", "expressive", "strong", and "effective". Teachers are left wondering as to what textual evidence to look for when evaluating a text's content, organization, and style. Often, they have to rely on their own intuition and discursive knowledge in making judgment calls. This can be especially troubling because English teachers are rarely trained to understand the discursive features of writing (Schleppegrell, 2004) and they often feel ill equipped to evaluate student writing in linguistic terms (Harper & Rennie, 2009; Hammond & Macken-Horarik, 2001). In fact, many teachers usually do no more than point to errors in the more obvious aspects of writing, such as spelling, capitalization, punctuation, subject-verb agreement, tense, and idiomatic expressions (Christie & Derewianka, 2008; Martin, 1996).
Another problem with the rubric is that it focuses on grammatical forms with little attention to their functions. For example, the six-traits writing rubric calls for variation in sentence patterns, but fails to specify which kind of variation is appropriate for which type of text, task, or discipline. While syntactic variety and complexity for its own sake may enhance a text's status, it does little to improve its quality and effectiveness (Myhill, 2008). Scholars (e.g., Fang & Schleppegrell, 2008; Martin, 1989; Schleppegrell, 2004) have shown that different disciplines, genres, and registers draw on different constellations of lexicogrammatical features that enable a text to mean what it does in a particular context. For example, factual genres in academic contexts, such as reports and explanations, tend to use sentences that are grammatically simpler but lexically dense and that contain abstract nouns and expanded noun phrases, whereas personal genres such as recounts and narratives often use sentences that are grammatically more intricate, stringing together coordinate and subordinate clauses. Moreover, the same constructs (e.g., voice) are often instantiated differently across different academic disciplines. For example, Schleppegrell (2019) demonstrated that interpersonal meaning (i.e., authorial voice) is enacted in different ways across different disciplines, with literature, for example, foregrounding the feelings and attitudes of characters through showing (e.g., George slammed the door.) rather than telling (e.g., George was mad.), science arguments exhibiting caution about knowledge claims by calibrating degree of commitment and confidence in presenting findings, and historical explanations showing authorial bias by including or excluding the thoughts and opinions of particular historical actors. These differences are a reflection of the fundamental differences in the ways experts in different disciplines read, write, and think.
The rubric's indifference to discipline-, genre- and register-specific requirements is also found in its undue emphasis on personal involvement with the topic (e.g., expressive, emotion, humor, honesty, suspense, vivid images, creative and engaging). Scholars (e.g., Biber et al., 1999; Schleppegrell, 2004) have pointed out that unlike narrative genres where personal involvement is valued, academic texts, particularly those of factual and analytical genres, often feature a more objective, abstract, and authoritative style of writing that encourages personal detachment. Martin (1996, 2002) further demonstrated that academic texts are typically organized in "waves of abstraction" that make the introduction and conclusion paragraphs, as well as the beginning and ending sentences of each paragraph, highly nominalized and abstract. This pattern of text organization, Martin suggested, facilitates flow of information and development of argument. Therefore, blind adherence to certain grammatical forms or styles without regard for their functions or the context of their use can result in texts that have little rhetorical power and unexpected communicative effects. Unfortunately, assessment rubrics and curriculum standards too often conceptualize voice or style as a unitary construct, failing to acknowledge that texts in different disciplines or genres have different voices or styles that are realized in different linguistic patterns (O'Halloran & Schleppegrell, 2016).
It is clear that currently popular writing rubrics have their limitations and problems (see also Wilson, 2006 for a different critique of rubrics). On one hand, rubrics encourage the teacher to pigeonhole individual students into a certain proficiency level; on the other hand, they fail to provide specific, concrete, discipline/genre/register-sensitive criteria that enable the teacher to render a more objective and valid judgment. As such, rubrics give teachers few insights into what exactly makes a text more or less effective/valued and are marginally useful for informing writing instruction. For these reasons, O'Halloran and Schleppegrell (2016) have called for increased specificity in the descriptions of rubric categories to support teachers and students in identifying and deploying the language resources that address the specific criteria for particular registers, genres, and tasks in particular disciplines. Making textual expectations for academic writing tasks concrete and explicit is, according to Schleppegrell (2004), the only means of providing access and ensuring fairness in assessment.