New Assessment System: Evidence of Teacher and Student Learning (ETSL)

Defining Successful ETSL Template Units
By Robert A. Southworth, Jr., Ed.D.
The SchoolWorks Lab, Inc.
Outside Evaluator

Pioneering Performance Assessments

The SchoolWorks Lab, Inc. recommends from previous evaluations (Southworth 2008)  that there is a need for a performance assessment system to more accurately and equitably measure the learning outcomes described in the ETSL Templates. There is a vacuum associated with the accurate measurement of complex student performance in education. The arts have a long history of pioneering and were one of the first subject areas to adopt the pursuit of national standards (Consortium of National Arts Education Associations 1994). Researchers (Moss 1996) have argued that assessment is trapped in the psychometric side of understanding, in the standardization of the process across individuals, classrooms, districts and that the creative use of performance assessment might lead to more accurate measurement of student achievement.

Strategy of Common Template

The use of a common template that is individually constructed allows us to form an assessment system that achieves reliability without sacrificing the validity of individual contributions.

Once a community begins to engage in dialogue about the values that inform its assessment-based interpretations, inevitable differences in value perspectives will arise. I propose two principles to guide the negotiation of these differences.  First, differences can be a resource that strengthens validation practices. Second, some value differences may be irreconcilable. Thus, while consensus may be an ideal to strive for, the reality is that it is not always possible to achieve (Shay 2004, p. 326).

Working towards consensus in the system supports a variety of approaches to meeting or exceeding standards and promotes diversity while specifying what needs to be standardized. Students should realize gains in a performance assessment system because a wide variety of approaches are acceptable in order for them to demonstrate understanding of the integration of the arts.

Recommendations to the Field

The purpose of this report is to make recommendations about what we have learned from the first year of piloting the Evidence of Teacher and Student Learning (ETSL) template. The ETSL Template is a powerpoint representation of arts integration produced by arts-in-education partnerships in K-12 schools in New York State. Almost 100 of these partnerships are funded by the New York State Council on the Arts (NYSCA) every year. The goal of the ETSL template is to show evidence of teacher and student learning through data collection, assessment and peer-to-peer documentation and evaluation. The first year of piloting, 2007-2008 follows many years of hard work by the field of funded partnerships to improve their work. The SchoolWorks Lab, Inc., as the evaluator of this work since 2003, recommends that through the introduction and use of an authentic achievement performance assessment system, individual partnership ideas of quality can be honored across the entire system of partnerships while meeting or exceeding state standards for quality.

History of Evaluation

This new system for evaluating NYSCA’s funded partnerships is different from an older system developed many years ago where partnership evaluation was conducted by contracting with outside evaluators who produce a report that is submitted to NYSCA. The new system uses an “assessment data collection as professional development” strategy (Southworth 2003). In 2003 an evaluation of the NYSCA Arts-In-Education Department recommended the adoption of new forms of assessment to drive stronger accountability for learning by placing some of the burden for evaluation on the partnerships (Southworth 2003). In the following year, each partnership’s accountability was modified to include a performance assessment that asked each partnership to collect evidence of student learning (Southworth 2004). In 2005, a web-based solution to hosting individual partnership evidence was successful with only a few partnerships. In 2006 the search for a more scaleable approach that would engage more partnerships was developed through the use of individually configured power-point presentations. In 2007 a template for presenting this evidence was adapted from the CAPE (Chicago Arts Partnership 2008). In 2008, the first 25 partnerships completed the piloting of the Evidence of Teacher and Student Learning (ETSL) template.

Increase Arts Quality

In order to transform the “data collection as a professional development” strategy into increased quality in arts learning, a peer-to-peer approach needs to be supported by standards for quality embedded in an assessment system. The template is the strategy for formalizing the story of partnerships, providing contextual framing of the learning environment, and marshaling evidence of the extent to which students learn directly through the partnerships. The assessment system is the strategy for standardizing the quality across the partnerships. The act of gathering assessment data as a professional development strategy is the peer-to-peer approach most likely to improve the quality of the work in arts integration.

GENERAL RECOMMENDATIONS

It is therefore recommended that through the introduction and use of an authentic achievement performance assessment system, individual partnership ideas of quality can be honored across the entire system of partnerships while meeting or exceeding system standards for quality. The parts of a good assessment system are:

1. Authentic Student Achievement
2. Performance Assessment
3. Tasks and Rubrics
4. Clarity of Goals and Questions
5. Types of Evidence
6. Essential Questions
7. Developmental Stages of Template Use

Authentic Student Achievement

At the heart of good assessment systems is the need for evidence of student learning that represents authentic student achievement (Wiggins 1989a).

Newmann et al. (1996) recommend these conditions for authentic student achievement:

1. Primary concern for the intellectual quality of student learning
2. Commitment to maintain high expectations for all students, regardless of individual differences.
3. Support for innovation, debate, inquiry, and seeking new professional knowledge.
4. Ethos of caring, sharing, and mutual help among staff, and between staff and students, based on respect, trust, and shared power relations among staff.
5. Sustained time for instruction, planning, staff development, and student advising.
6. Interdependent work structures for staff, especially teaching teams and committees for schoolwide decision making.
7. School autonomy from regulatory constraints.
8. Small size for school and instructional units (Newmann, 1996, p. 289).

Performance Assessment

The ETSL Template is a record of a performance assessment. Unlike a multiple-choice or true-false test in which a student is asked to choose one of the responses provided, a Performance Assessment requires a student to perform a task or generate his or her own response.  For example, a performance assessment in writing would require a student to actually write something, rather than simply answering some multiple-choice questions on grammar or punctuation.

A Task and a Rubric

A performance assessment consists of two parts, a task and a set of scoring criteria or “rubric.”  The task may be a product, performance or extended written response to a question that requires the student to apply Critical Thinking skills. Some examples of performance assessment tasks include written compositions, speeches, works of art, science fair projects, research projects, musical performances, open-ended math problems, and analysis and interpretation of a story the student has read. Existing classroom instructional activities may often be transformed into a performance assessment with the addition of suitable scoring criteria.

(Chicago Public Schools Inranet: http://intranet.cps.k12.il.us/Assessments/Ideas_and_

Rubrics/Intro_Scoring/Definition_of_P1/definition_of_p1.html).

Tasks — Working Backwards from Actual Practice

Performance tasks are, by definition, about assessing learning through actual practice. The key here is to articulate what the final student learning outcome(s) will be and work backwards from there to design classroom activities. The various skills that students need to know and be able to apply in order to achieve the unit goal are the assessable components of the performance task. In creating a performance assessment it is helpful to bear three things in mind:

1. Which skills are most important to assess? (Prioritize the assessment. Not everything needs to be assessed or assessed in the same way.)
2. Which tools can best capture the different levels of learning occurring in the unit of study?
3. Performance tasks can draw upon a variety of assessment tools, such as surveys, portfolio sharing, checklists, rubrics, observation protocols, debates, oral defenses, monologues, etc.

Clarity of Goals and Questions

When starting to work on the ETSL template, keep in mind the backwards design that keeps the end result in mind at the beginning (Wiggins 1998). This means, for example, that the clarity of the goals, standards, outcomes and unit objectives be clear and manageable. Include the teacher’s standards for professional development learning. Classroom subject area and arts learning standards should play a central role in the story of arts integration. Extraneous goals, standards and outcomes should be kept out. Try to show how essential questions were explored and how teaching choices affect the quality and sequence of student experiences. At the beginning, emphasize the role assessment will play and how it will support student demonstration of learning. Develop the template around assessment activities that provide real tangible evidence of student learning. At the end of the template, show through assessment and reflection how teachers and students met standards for integration and what teachers and students learned from this experience.

Types of Evidence

The best type of evidence of student learning usually comes in the form of an outcome, that is to say, as a result of this learning experience what types of evidence show kids demonstrating the outcome of their learning? Is there evidence of kids making connections between and among the art forms and the other disciplines? Doee the template’s work samples and student responses link back to the guiding questions and Learning Goals set out at the beginning of the unit? Do the Teacher, Artist and/or students make statements about their learning in relationship to these. Who participated in creating the template? How useful or interesting were the discussions while putting it together? Does the template show what YOU have learned in the process of teaching and assessing this unit?

Recommended Essential ETSL TEMPLATE Questions

Authentic Achievement	How well did the template show students constructing knowledge, engaging in disciplined inquiry, and going beyond the classroom?
Authentic Instruction	How well did the template show teachers using higher order thinking skills, deep knowledge, substantive conversation and connecting to the world beyond the classroom?
Authentic Tasks	How well did the template show students organizing information, considering alternatives, using disciplinary content and processes, writing about their understandings and connecting to the world beyond the classroom?
Authentic Student Performance	How well did the template show students using analysis, elaborated written communication and demonstrated performance beyond the classroom?
Quality of the Artwork Displayed	How well did the template show student artwork that was clearly executed, directly related to standards and integrated into the classroom subject or discipline?
Power Point Representation	How well did the partnership use the medium of the powerpoint to convey the work of the partnership, the context of the collaboration and the story of the integrated curriculum?
Reflections	How well did the template display the reflections of the students and the teachers on their work, their next steps and what they have learned?

Recommended Developmental Stages of ETSL Template UseNew York State Council on the ArtsEvidence of Teacher and Student Learning (ETSL)
Stages of Development in ETSL Template Use	1Beginning	2Intermediate	3Successful	4Above Expectations
Student	TriesTemplate shows students doing art	EngagesTemplate shows students engaged and learning in art	AccomplishesTemplate shows student accomplishment of arts integration	Goes beyondTemplate shows student demonstration of understanding of arts integration
Teacher	TeachesTemplate shows teacher goals, standards and directions	SupportsTemplate shows teacher support for student learning	GuidesTemplate shows teacher guidance and student accomplishment	LaunchesTemplate shows how teachers launch student performance to go beyond expectations
Standards	ApproachesStandards are mentioned and art is shown	DevelopsClear use of standards in Arts and in Subject matter although Art is still a resource to the classroom	MeetsEvidence shows students meeting standards for art and subject discipline and teachers show integration through sharing and planning	ExceedsStudent success in achieving Standards for Arts Integration is demonstrated through assessment evidence supported by teachers

Next Steps

The evaluators have tried to show that in the pilot year of this work, the field is well engaged in building a learning community through the ETSL Template. It is also clear that the work is heavy lifting and that it is beginning to change entire partnerships as for example, DREAM YARD and WORKING PLAYGROUND. In each of these cases the work of developing a quality template has resulted in “assessment data collection as professional development” (Southworth 2003). We strongly recommend the continued use of the template as the cornerstone of a performance assessment system.

• A Performance Assessment System should be built around the ETSL
• The ETSL TEMPLATE should be put online
• The field should be given access to an online support system
• A performance assessment system should be enhanced through Rubrics
• RUBRICS should be built with authenticity to guide their applicability
• Rubrics should include authenticity and arts integration
• Inter-rater reliability should be developed
• The community should be strengthened through the rigorous assessment of the Template

Rubric — Scoring Guidelines

Because a Performance Assessment does not have an answer key in the sense that a multiple choice test does, scoring a performance assessment necessarily involves making some subjective judgments about the quality of a student’s work.  Many people feel uncomfortable with making and using subjective judgments and find that a good set of scoring guidelines or “rubric” provides a way to make those judgments fair and sound.  It does so by setting forth a uniform set of precisely defined criteria or guidelines that will be used to judge student work.

Generic Rubric Examples
	Levels of Performance
Types of Generic Criteria	1Beginning	2Intermediate	3Successful	4Above Expectations
Standards	Uneven	Approaches	Meets	Exceeds
Arts Integration	Doing Art	Art as resource	Art and Subject Integration	Art and Subject Integration through Assessment
Teaching	Teaches	Supports	Guides	Launches
Tasks	Tries	Engages	Accomplishes	Goes beyond
Performance	Shows	Reveals	Demonstrates	Insightful