Entering the Dialogue
Writing Assessment Pilot
Rubric
Reading
Response
Recommendations
Bringing it Back Home
Filling a Gap
Mid-Level Checkpoints
Integrating Research
Interdisciplinary Program
Continuing the Dialogue
Contact Information
|
ASSESSING WRITING
AT WESTERN WASHINGTON UNIVERSITY
Carmen
Werder, Donna Qualley, and Gary R. McKinney
November 1998
Imagine that you enter a parlor. You come late. When you arrive, others have
long preceded you, and they are engaged in a heated discussion, a discussion too
heated for them to pause and tell you exactly what it is about. In fact the discussion
had already begun long before any of them got there . . . You listen for awhile,
until you decide you have caught the tenor of the argument: then you put in your
oar. . . -Kenneth Burke We are engaged in a national conversation
that has been going on for some time. It is a conversation focused on making universities
accountable for what they do. This national dialogue has compelled four-year schools
to rethink their missions, especially in terms of a commitment to teaching. Out
of the talk has emerged a strong sense that the burden of proof lies with universities:
we need to articulate what we do to the local community and wider public, and
we need to supply evidence of that performance. Assumed is the tenet that in an
open society, the people providing the funds can expect its public institutions
to be held accountable for how those funds are spent. But if what we do is teach
students to think, assessing the success of that enterprise becomes a tricky proposition.
What measures do we use in determining how well we are leading that complex and
intricate intellectual journey? Situated in this larger national discussion
of accountability, the state of Washington is actively pursuing its own answers.
The K-12 system has been developing a series of essential learning outcomes and
performance-based models of assessment, designed to demonstrate the effectiveness
of their instruction. In the state's higher education system, response centers
around certain mandated accountability measures, such as retention rates and the
graduate efficiency index, which aim to hold institutions to a certain level of
performance. Yet while these measures may relate to an institution's level of
efficiency, they say very little, if anything, about what our instruction has
accomplished in the way of true education. At the same time as these accountability
terms have been imposed on the state's four-year schools, we also have opportunities
to create our own instruments. Last spring, the public, four-year institutions'
assessment coordinators and a few accountability members met in an effort to find
better accountability measures. One idea that surfaced in that meeting was a proposal
to evaluate the best writing of college seniors. This plan was based on the assumption
that writing offers a significant, if not the best, window into student reasoning
ability. Because of this melding of abilities, assessing student writing could
get us closer to assessing learning. And yet what could be more daunting than
finding such a measure? After considerable discussion by some of the state's college
composition teachers, the decision was made to pilot the proposal. Below is a
report on this study representing the writing instructors' perspective, followed
by a closer look at Western's plans for writing asessment. Top
of Document In
late spring, several writing faculty and administrators representing all but one
of the state four-year public schools gathered to discuss a proposal by assessment
coordinators to assess senior-level writing. Most of us went to the meeting concerned-if
not downright grouchy. Three concerns were immediately evident. How could we assess
papers across disciplines without common assignment parameters? How could we apply
one set of criteria to judge all papers regardless of discipline? And how could
one writing sample possibly be used as an accountability measure linked to university
funding? In spite of these serious concerns, we surprised ourselves by agreeing
to participate in a pilot. We had not been converted; in fact, some of us thought
it judicious to participate in the pilot simply to confirm our suspicions that
it wouldn't work. Neither did we want writing faculty to look like close-minded
pedagogues. After all, we did (and still do) believe that writing can be assessed
and that colleges are accountable for their instruction. Finally, it was at minimum
an important political move at least to try the idea, so we agreed to give it
a go and hastily worked to gather papers and to identify discipline people from
our campuses to participate in scoring sessions. Top
of Document At the first session in
July, each of the five disciplinary groups (biology, business, engineering, English,
and sociology) drafted a rubric, or set of criteria, that seemed to mark successful
papers. From these small group drafts, we worked together as a whole group to
generate a master rubric, then worked again in small groups. What emerged were
six broad categories: Content (what ideas were included); Organization
(how they were ordered and connected); Reasoning (how they were developed);
Rhetoric (whether they were expressed appropriately for the situation); Disciplinarity
(whether they were expressed appropriately for the discipline); and Conventions
(whether they were presented conventionally in terms of standard written English).
Under each category, we listed various sub-criteria that might be considered
in assessing each category. In our discussions, we also noted the value of connecting
the language of our rubric to that used in K-12 where possible, and while our
six categories reflect the shift to discipline-based writing at the college level,
there is some shared language with the six-trait scheme used at the lower levels.
After leaving the July sessions, we revised the rubric over e-mail, and then used
it for the August rating sessions when business and community members joined the
groups. While further revisions have been suggested regarding the rubric, participants
seemed generally satisfied with the categories and the comprehensiveness of the
sub-criteria. Top of Document The
August sessions began and ended with surveys polling the attitudes of participants
towards assessing writing. After completing the pre-session surveys, we began
the reading process by doing an exercise in "Deep Reading," a method devised by
Margaret Himley and recommended by Lynn Briggs (EWU). The approach involves reading
aloud, paraphrasing, and describing a piece of writing before judging its worth.
While there was some initial question about the need for such an approach, the
process turned out to be extremely useful in the way it set a context of careful
reading for the remaining three days. After trying out this deliberate, deep reading
process in the whole group, we formed disciplined-based groups in business, engineering,
biology, sociology, and English. During the first round, each group included a
writing specialist, a disciplinary faculty, and sometimes an assessment coordinator.
For the second cycle of reading, these groups also included business and community
members. Carmen worked primarily with the business group, and was struck by
the absolutely central role of the disciplinary faculty. Several times, she rated
a paper higher than the business teachers. Often they would point out inaccuracies
in the information and gaps in the reasoning that only people from the discipline
could do. At the same time, as a writing teacher, she was able to suggest language
for talking about common ideas as well as recommend ways for rethinking and reframing
the criteria. And the assessment people were very helpful in urging the group
to keep their focus on textual features-to make empirically-based judgments. All
three perspectives proved valuable in reaching assessment decisions on the sample
papers. Top of Document Overall,
the pilot affirmed the value of getting teachers together in the same room around
a set of papers, clarified our understanding of the complexity of writing assessment,
and confirmed our fear that the absence of common specifications for the papers
would be problematic. Indeed, a recurring theme was that judgments on individual
papers kept veering back to the assignment-which, unfortunately, was often missing
or vague. It was especially difficult to determine the effectiveness of the writing
without an assignment, and when the assignment was vague or mostly form-driven,
the writing tended to present a superficial/unreasoned discussion of a topic.
The consensus was that the more well-founded the assignment, the stronger the
writing. The group's reluctance to reduce writing to one overall score was
reflected in the intricacies of the rubric as well as in the post surveys, where
the strongest area of agreement was a firm belief in the complexity of writing
assessment-a sentiment deepened by our experience. Moreover, participants indicated
even greater concern with assessment decisions based on just one writing sample
Clearly, the most impressive value of the pilot was faculty development. Faculty
were willing to modify their thinking when they had the opportunity to work with
actual student papers and to discuss them with colleagues. Pre- and post-session
survey results indicated that a third of the participants changed their attitudes
from the first day to the last. By articulating our beliefs in the face of challenging
questions, we all came to a fuller understanding of what constitutes effective
writing at the senior level in various disciplines. Top
of Document Given
continuing concerns as well as an appreciation for the benefits of such a cross-disciplinary
enterprise, several recommendations emerged, primarily to conduct another pilot-with
the following changes: Revisit the rubric to see how it might be refined and
perhaps streamlined. Establish some common submission specifications (e.g.
page range and kinds of tasks) Ask for assignments to be submitted with all
writing samples. Ask for a writer's reflective commentary attached to each
paper stating writer's consent to use the paper, along with understanding of assignment
expectations, and intentions for the piece. Open a campus-wide dialogue on
each of our campuses exploring ways to duplicate the kind of deep reading-assessment
process experienced in the pilot. Another idea that has surfaced is the suggestion
to invite all senior capstone faculty whose students would be participating in
another pilot to attend a campus workshop on designing writing assignments and
creating evaluation criteria. That way, the writing samples would spring from
writing assignments that were more likely to prompt the best student writing possible.
As writing teachers, we can't help but wonder now. How do we sustain the perception
of writing assessment as a complex activity and still move forward to use it as
a measure of undergraduate learning? And, most importantly, we ask: How can we
translate our assessment observations into real changes in writing instruction
on our own campus? Top of Document A Comprehensive and Ongoing Program of Writing Assessment,
Research, Faculty Development, and Classroom InstructionRecent events
on our own campus echo the national and state focus on assessment as both top-down
and bottom-up initiatives have emerged. After the 1998 Accreditation Report was
delivered, Western received a call from President Morse for the further development
and articulation of a university program of assessment, and each department has
been asked to develop learning outcomes and assessment criteria. Emerging as a
cornerstone of this institutional assessment plan is our campus writing assessment
effort. Many of us in the campus community have come to realize the integral
relationships among assessment, research, instruction, and faculty development.
We can't forge ahead in one area without continual attention to our efforts in
the other areas. A number of plans, pilots, and initiatives currently underway
aim to establish a comprehensive and ongoing program of writing assessment, research,
and faculty development-with the overall goal being to improve instruction across
subjects at all levels. Of course, underlying all these efforts are some fundamental
understandings about the nature of writing itself: Writing is a situated activity:
When we compose, we write for a particular audience and for a particular purpose.
By its very nature, writing effectiveness depends on context. Therefore, writing
assessment and instruction must be discipline-based. While broad categories of
performance may be agreed on, specific elements play out differently across disciplines.
Successful writing in English is distinct from successful writing in biology.
Thus, any valid measure of writing effectiveness must account for those disciplinary
distinctions. Writing is an acquired ability: Unlike some skills that can
be mastered once and for all, writing ability involves a complex set of skills
gained over time. While a student may master the language of a word processing
program in a fairly limited and set amount of time, the challenges of composing
with the English language demand a much longer time (some would say a lifetime).
Therefore, writing instruction must be ongoing and diffused throughout the curriculum.
Writing integrates reasoning with communicating: When we write, we must both
think and present that thinking in a way that readers will understand. These skills
are inextricable. When we talk about college-level writing, we're not talking
about "how to say nothing correctly in 500 words," we mean saying something meaningful
and saying it clearly enough for others to make sense of it. Teaching writing
necessarily means teaching ways of thinking. Therefore, the responsibility to
teach and to assess writing is a shared responsibility and not one that should
be confined to the English Department (though the English Department has a significant
role to play). Top of Document The above assumptions about
the nature of writing underlie recent discussions in our writing requirements
at both the lower and upper levels. The approval of a second writing course as
part of the GUR reflects an effort to make both writing instruction and assessment
part of our students' education throughout their careers at Western, not just
in one class during their first year. The first-year course is designed to introduce
students to some key rhetorical concepts and critical and reflective habits of
mind that will help them learn the specific requirements of other communities.
The second-level writing course builds on the first by providing more situated
writing instruction and by offering the opportunity for students to practice using
their newly acquired ways of thinking and communicating. These new sophomore level
writing courses are scheduled to go into effect in the 1999-2000 academic year.
This year, the English Department, in conjunction with Writing Center faculty,
are piloting fourteen sections of two possible variations of courses that might
satisfy the new writing requirement. Seven sections are stand-alone English 201
composition courses that are linked to other GUR subjects (geology, psychology,
anthropology, and East Asian studies) and a one credit library research methods
course. Seven sections are English 297 "Writing in the Context" courses, where
instructors will balance course content with writing instruction. If deemed successful,
the English 297 course could be used as a model for developing "Writing in the
Context" courses in a variety of disciplines. Carmen Werder and Steven Vanderstaay
are coordinating these pilot courses, including efforts to: assess student
satisfaction and course effectiveness; identify faculty responses to teaching
these courses; and propose student learning outcome models. Top
of Document Recent revisions to our writing proficiency
requirement, resulting in elimination of the Junior Writing Exam, have led to
a proposal for integrating assessment into courses that carry "writing units."
An idea, first suggested by Diana Weymark (Economics) and then given voice by
Thor Hansen (Geology), the writing unit plan would recognize specific courses
that currently (or might with some revision) contain writing components. Depending
on the nature and extent of the writing instruction, these courses would be given
a designation of 1-3 writing units. By requiring students to have so many writing
units to graduate, the plan would allow us to better integrate writing assessment
with instruction-where it belongs-and would provide an incentive to include more
writing instruction in existing courses. Further examination and discussion of
the writing unit plan will continue this year. The writing units plan at
the upper level, together with the second GUR writing course at the lower level,
has the potential to provide multiple assessment opportunities. More instructors
and more courses will be involved in writing instruction; thus, more chances for
diagnosis, referral, and further instruction will arise. By expecting students
to have writing experiences at every level, more opportunities to assess their
progress will be ensured. Top of Document So
that we may identify the patterns in the development and deepening of students'
learning during their time at Western, we are beginning to gather descriptive
information on students' writing. This year, a writing prompt is being administered
at the beginning and end of selected 100, 200 and 300 level writing courses. In
addition to providing an ongoing demonstration of students' writing at different
points in their education, the prompt will also offer an indication of their self-assessment
abilities and their developing awareness of effective writing. This information
will allow us to hypothesize about the effects of instruction on students' understanding
in the short term (at the beginning and end of a course) and in the long term
(over a period of several years and many courses). Once some patterns begin to
emerge in this data, teachers will be better able to evaluate which students may
not be prepared to handle the writing demands of their particular courses. In
this way the prompt (see below) will also serve as a diagnostic assessment tool
that will allow faculty to gauge the progress and needs of students at numerous
points in their education. It is envisaged that this information will better enable
us to advise students and channel them into appropriate courses. Writing
Prompt "In a few pages, talk about a paper you wrote for a class during
the past year. Explain what you were supposed to do, how you went about doing
it, and how effective your efforts were. Then, on the basis of this writing experience
(the process you went through and your teacher or peers' comments/advice), discuss
what you think are the characteristics of good writing. Finally, explain why you
think most teachers would agree/disagree with you." (At the end of the course,
the prompt will be adjusted to read "talk about a paper you wrote for this class.")
It is hoped that in the future, the use of this prompt may be extended to upper
level writing units and senior capstone courses. In these courses, the prompt
would be recast from "characteristics of good writing" to "characteristics of
good writing in (name of the discipline)." Top
of Document Given the new directions and
needs in writing assessment, research, and instruction across the curriculum,
a proposal has emerged for an interdisciplinary program that would serve as a
consortium, an alliance for cross-disciplinary writing instruction and assessment
at Western, as well as a coordinating body for campus-wide writing initiatives,
such as the ones just discussed. Last June, the Provost's Writing Committee
recommended the establishment of such a program, which would be responsible for
university-wide efforts to: coordinate and support faculty development and
writing instruction in the disciplines; develop comprehensive assessment models
for writing; and contribute
to an ongoing program of research on student writing at Western. The
Interdisciplinary Writing Program (IWP) is envisaged as one way to increase the
possibility that writing will be integrated into instruction at Western. As envisaged,
the IWP would consist of permanent tenure and non-tenure track composition faculty
as well as disciplinary (writing and non-writing) faculty from departments on
an annual and part-time basis. All faculty-composition and disciplinary, tenure
and non-tenure, permanent and part-time-would have the opportunity to participate
in ongoing professional development and to be part of a dynamic teaching community
while they teach writing in their disciplines. Currently, the Provost is forming
an interim group to study the proposal further; meanwhile, discussion for developing
such a coordinating body is underway. Top
of Document Given
these ongoing conversations at the national, state, and local levels, we could
gain as a university community by talking with each other about what we do.
What ideas do you
have for ways we can work in a joint venture to use writing assessment to forge
the connections between how we teach and what students learn? Please feel free to jot
down your thoughts and mail them to us at MS: 9010, attn: Gary McKinney.
Carmen
Werder, Ph.D., is the Associate Director of the University Writing Center Program.
Donna Qualley, Ph.D., is Director of Composition, English Department.
Gary R. McKinney, MA, is a Planning Analyst for The Office of Institutional Assessmsent
and Testing. published
by
Office of Institutional Assessment and Testing
Dr. Joseph E. Trimble,
Director; Gary R. McKinney, General Editor
technical assistance by Center
for Instructional Innovation
Dr. Kris Bulcroft, Director; Web Design by Karen
Casto Dialogue
Home
Center
for Instructional Innovation Home | Western Home .gif)
|
