Technology mediated feedback processes in formative assessment with beginning teachers
King’s College London , UK,
Paper presented at IFIP Workshop "New developments in ICT and Informatics education", June 28-30, 2010, Université de Picardie Jules Verne, Amiens, France,
Abstract
By drawing on findings from two distinct strands of research this paper addresses the questions: What perceptions do pupils and teachers have of the nature of feedback processes in formative assessment and their value for learning? How can technology support the interactions and feedback processes of formative assessment? What models of pedagogy involving formative assessment are emerging? The first strand focused on case studies of classes where experienced teachers had embedded formative assessment practices into their teaching. The second strand was a study with pre-service teachers that set out to investigate how a range of technologies could be integrated into pedagogical practices that emphasised formative assessment and how the technologies supported and enabled these approaches. Methods included structured lesson observations, structured teacher interviews, student focus group interviews, informal discussions, questionnaires and scrutiny of lesson plans and students’ work. Findings were analysed using a framework derived from activity theory. The findings revealed complex pedagogical sequences incorporating feedback processes. Furthermore technologies provided a range of tools that can be deployed by teachers as they engineer situations for students to learn and to develop as autonomous learners. The tools mediate the feedback processes and can extend and support teacher and peer feedback. Planning and engineering these sequences of activities and deployment of these mediating tools requires sophisticated pedagogical reasoning by the teachers using their developing pedagogical knowledge. These are significant challenges for beginning teachers but are central to their development as teachers.
Keywords
pedagogy, pre-service teachers, formative assessment, feedback, classroom teaching, primary education, teaching methods, teacher education
Introduction
This paper aims to address the questions: What perceptions do pupils and teachers have of the nature of feedback processes in formative assessment and their value for learning? How can technology support the interactions and feedback processes of formative assessment? What models of pedagogy involving formative assessment are emerging?
These questions are addressed by drawing on findings from two distinct strands of research. The first strand focused on primary teachers who had embedded formative assessment practices into their teaching. The research was conducted in the State of Jersey that undertook a major professional development programme, referred to as Jersey Action for Formative Assessment (JAFA), to develop formative assessment. The JAFA professional development programme followed similar principles and built on methods used previously (Black, Harrison, Lee, Marshall, & Wiliam, 2003). Research, over four years between 2004 and 2008 aimed, to understand classroom practices in formative assessment and how these developed, the nature of interaction and feedback and how formative practices were related to pedagogy (Jones & Webb, 2010 (in preparation); Webb & Jones, 2009). Although this work had no particular focus on technology some of the participating teachers were experienced users of technology and integrated ICT use into their teaching. The second strand was a study funded by the UK Training and Development Agency for Schools (TDA) and King’s College Teaching Fund that set out to investigate: How can technology facilitate interaction and collaborative learning between beginning teachers within the classroom and beyond in an integrated and coherent way? In order to encourage beginning teachers to reflect, the pedagogical approach emphasised collaboration between students and was designed to model the kinds of formative assessment approaches that were emerging from our research in schools. The aim was to enable beginning teachers to adopt formative practices, including peer and self assessment, as well as collaborative learning in their own teaching. The development was successful to the extent of enabling collaborative learning in beginning teachers’ own development (Webb, 2010 (in press)). Furthermore it engendered a positive approach to incorporating collaborative learning with a focus on peer feedback within a repertoire of pedagogical approaches in beginning teachers’ own teaching practices. Some technologies integrated easily into these approaches and made them more efficient. This paper examines the feedback processes and use and perceptions of these processes by experienced teachers, their students and beginning teachers as they develop their practices. Thus the paper examines feedback processes across school learning, learning in higher education and initial teacher education.
Formative assessment and learning
From a meta-analysis of research into classroom assessment practices Black and Wiliam (Black & Wiliam, 1998b) found that, when teachers implemented formative assessment strategies, the learning gains of the students in these teachers' classes were significantly greater than those of control groups (Black & Wiliam, 1998a, 1998b). For Black et al., the term ‘formative’ applies not to the assessments themselves, but to the functions they serve in supporting students’ learning and providing evidence that is used to adapt the teaching to meet learning needs (Black, Harrison, Lee, Marshall, & Wiliam, 2003). Research has demonstrated that working at improving the teacher-student interaction through formative assessment and particularly focusing on feedback can catalyze changes in both the teacher’s role and those adopted by that teacher’s students (Black & Wiliam, 2006).
In UK higher education there is evidence of dissatisfaction with feedback but also lack of clarity or consensus about the nature of feedback. For example, in the National Student Satisfaction Survey in the UK in 2008 as only 54% of undergraduate students, agreed that feedback had been prompt and had helped to clarify points they did not understand (Tahir, 2008). Furthermore research based on interviews with thirty five students and staff from three institutions of higher education, suggested that students and teachers lacked clear understanding of the nature of what constitutes feedback and had no shared understanding of its purpose (Price & O’Donovan, 2008).
Evidence that for feedback to be effective students need to receive information both about the task and about how to perform it more effectively comes from Hattie and Temperley’s (2007) meta-analysis of feedback studies. They obtained high effect sizes where students received information about the task and about how to perform it more effectively, lower effect sizes for interventions which focused on target-setting, and much lower effect sizes where only praise, or rewards were given. Their subsequent categorisation of types of feedback also took account of factors affecting students’ responses to teachers’ feedback including the commitment and confidence of the students. However their categorisation focused on teacher-student interactions rather than peer, group and whole class interactions where feedback processes are also in operation. These more complex settings are examined in this paper.
Methods and Data Sources
In order to examine changes in classes, teachers’ pedagogical decision making, the challenges they experienced in developing their own practice and changes in the students’ learning approaches and attitudes, on the JAFA project, a combination of structured lesson observations, structured teacher interviews, student focus group interviews, informal discussions and scrutiny of lesson plans and students’ work was used. Findings were analysed using QSR NVIVO and a framework derived from activity theory and are reported more extensively elsewhere (Webb & Jones, 2009). In this paper the focus is on findings about feedback strategies and perceptions of teachers and students in classes where formative assessment was embedded.
On the PGCE research strand reported here, three successive cohorts of approximately 24 secondary ICT students were studied over three years. Students participated in discussions with teaching staff about which technologies to use and how to deploy them in their learning. Various pedagogical approaches with different combinations of hardware and software including mobile devices such as Tablet PCs and UMPCs together with wireless technology and Web 2.0 facilities such as wikis and application sharing were explored in a variety of taught sessions as well as being available to students for use in study time and group work. In order to investigate the students’ learning approaches and how technologies could support this learning, a combination of peer observations by lecturers, structured group interviews, focus groups, informal discussions, scrutiny of students’ assignments and online questionnaires was used. The author worked on the JAFA Project as a researcher and on the second strand as director of the secondary ICT PGCE course, tutor and as a participant researcher.
Results and Discussion
The Nature and categorisation of feedback
Identifying feedback is not as straightforward as it might first appear for several reasons. First a number of different meanings of feedback are in regular use in every day language as well as in specialised fields and the term can be used both as a noun and a verb. Secondly the most common use in education is probably as a noun describing feedback as a piece of information rather than a process, for example:
“feedback is conceptualized as information provided by an agent (e.g., teacher, peer, book, parent, self, experience) regarding aspects of one’s performance or understanding” (Hattie & Temperley, 2007 p. 81).
Thus even though the PGCE beginning teachers had already been involved in discussions about the nature of feedback, when they were asked in a focus group about the assessment and feedback that they had engaged with as a learner their first responses were about written feedback they had received on assignments. Some prompting was needed to enable them to think about feedback processes involving oral interaction in teaching sessions.
In the context of formative assessment as defined by Black and Wiliam (Black & Wiliam, 2009), for feedback to move learning forward, students must engage with the feedback. In this connection a more useful definition of feedback as a process with origins in systems theory is:
“the return of part of the output of an electronic circuit, device or mechanical system to its input so modifying its characteristics” (Collins English Dictionary).
A third reason for difficulties in identifying feedback is that much feedback is deeply entwined in the ongoing interactions in classrooms. Indeed feedback is a crucial element of good teaching.
A fourth factor that contributes to the complexity of identifying and characterising feedback is that it can be focused at different levels of learning e.g. task, process, self regulation (Hattie & Temperley, 2007). Thus feedback may simply focus on the outcome of the task and what the learner has achieved. A higher level might be a focus on process: how well the learner has carried out a task. An even higher level according to Hattie and Temperley (2007) is on self-regulation i.e. the way the learner manages their own learning and is engaged in metacognitive reflection. Another level of feedback that emerged as important in both the work with beginning teachers and with the primary classes is feedback on feedback. This is about a process that is needed to support learners in developing their own abilities to engage in peer feedback. Making a comment on another person’s work or performance is challenging and requires: understanding the success criteria; being able to evaluate the product or process using those criteria and formulating a comment that is accurate, supportive and likely to lead to improvement. In this connection it is also important to emphasise that feedback is a two-way process in that while students are learning how to learn teachers are learning how to teach them.
Previous work (Black et al., 2003) revealed that engaging in peer assessment led to students being able to self assess. Hattie and Temperley (2007) view self assessment as one element of self regulation. Thus there may be complex interrelationships between these “levels of feedback”. Furthermore in any one lesson there may be feedback operating at all these levels. For example, , in a Year 6 lesson taught by an expert teacher who had embedded formative assessment into her teaching the whole one hour lesson focused on feedback on an extended piece of work (Webb & Jones, 2009 p. 171) (see Figure 1). This lesson was highly focused and structured and incorporated all four levels of feedback” through: teacher feedback, as written comments to individuals, orally to the whole class, pairs and individuals; peer feedback and self assessment.
Year 6 South American Challenge
Introduction
This was a feedback lesson following a “critical skills challenge1” that focused on writing a report about South America. The learning intention was to “write a report” and the success criteria focused on the genre of report writing together with accuracy and presentation.
Lesson structure
Episode 1 (5 minutes)
The class were settling down and reading through their South American reports.
Episode 2 (4 minutes)
Whole class discussion in which the teacher asked the students to recap the focus of their challenge from last week. The teacher acknowledged the answers with positive responses summarised and asked “Why did you find it difficult?” The students found this difficult to answer. The teacher collected responses from several students and it was clear that the students understood that she expected them to explain their answers. The students were prepared to try even when they were unsure of the answer. When students struggled to answer the teacher prompted them to think and often said she would come back to them when they had had time to think e.g. :
P “I found it quite hard because (pause) I don’t know”
T “Would it be your natural style of writing?”
P “No”
T “OK think about your natural style”
Some students did put their hands up to answer questions but the teacher often chose those who did not raise their hands
Episode 3 (8 minutes)
The class recapped the success criteria through a whole class question and answer session. They then shared their self assessments of their reports and targets for future work that were based on “2 stars and a wish”2
Episode 4 (14 minutes)
The students had five minutes to peer assess the challenges in pairs by filling in a sheet and then another few minutes to explain the assessment to their partners. The teacher told them that she would want them to feedback to the class about how their partners had helped them. The sheets prompted the students to peer assess based on the success criteria and to summarise the assessment in the form of 2 stars and a wish. The teacher circulated around encouraging the students to explain their assessments and the evidence they had used. About half of the pairs were able to explain their assessments without help from the teacher and their summaries were focussed on the success criteria e.g. “clearly set out” but others wrote general comments that were less helpful, e.g. “nice report” or focussed on aspects that were not directly relevant to the success criteria e.g. the use of colour. The teacher worked hard to focus these groups on the success criteria by prompting them to think and explain. Most of the students were able to explain their reasoning when probed by the teacher.
Episode 5 (10 minutes)
Whole class discussion in which students were asked to feedback to the class about how their partners’ comments had helped them. The teacher probed them to explain and encouraged them to support and challenge each other. Thus the talk became a shared exploration involving several members of the class e.g. Heather is explaining how Ann’s feedback has helped her
Heather: She likes the border because it makes the work clearer
T Ann what was the one thing that stood out?
Ann It was interesting
T How did that help?
(Ann looks unsure )
T ( holding up the report) What point is Ann making here about layout, presentation and use of colour? Can anyone help here?
James (gives a fairly clear and lengthy explanation) This culminates in- you want to read it because it looks nice.
T Can I challenge here?
The teacher points out the learning intentions and there follows a discussion on colour and whether it is necessary for achieving the learning intention.
Episode 6 (10 minutes) Whole class discussion in which students were asked how they felt about their work and what and how they had learned.
Note 1 Challenges are a key component of the Critical Skills Programme (see http://www.criticalskills.co.uk ). Typically students would be presented with the challenge on a sheet which would explain the product(s) expected, list the learning intentions and success criteria and “chunk” the work into a set of activities and questions. Students would be expected to complete the work within a set time using a set of techniques and strategies learned previously.
Note 1 2 stars and a wish refers to feedback with two positive points and one issue for improvement
Figure 1 a lesson focussed on feedback (Webb & Jones, 2009 p. 171)
Focus group sessions with beginning teachers also revealed a wide range of types of feedback in use although selecting the appropriate method for a particular situation was seen as challenging and there was a sense of a struggle to become proficient in making these choices, e.g. :
S1 When I have thought about what level of assessment I would like to be able to do (pause) Really in an ideal world I world like to video tape the lesson and go through it with a fine tooth comb. That's not really practical. It does feel like what we have got are still relatively
blunt tools. Compared to what I think may be possible.
S2 Yes we are novices still (Beginning teacher Focus group February 2010)
Thus feedback is complex to understand and to undertake. It can be categorised by: the method of delivery (e.g. written comments on paper, written responses on a wiki page, oral comments via a podcast); the originator (teacher, peer or computer software); the level of focus (task, process, self regulation, feedback on feedback); the immediacy (e.g. straightaway usually through an oral comment, after a lesson observation, a few days later via homework marking); the recipients (whole class, group, individual); the degree of specificity; the opportunities for discussion and clarification. This categorisation places the student as learner as the focus of feedback where the purpose is to improve their learning but it also allows for an ongoing complex process with a series of loops. In this way feedback supports learners in understanding their learning so that they can improve their learning and supports teachers in understanding their students so that they can improve their teaching. Thus a learning conversation or dialogue develops. This process is similar to Laurillard’s conversational framework (Laurillard, 2002) or Alexander’s dialogic teaching (Alexander, 2006). However an important difference is in the emphasis on formative assessment. The nature of formative assessment as purpose is characterised in a framework proposed by Black and Wiliam (see Figure 2) that integrates key elements of formative assessment and focuses on understanding: where the learner is going; where the learner is right now and how to get there (Black & Wiliam, 2009).
|
|
Where the learner is going |
Where the learner is right now |
How to get there |
|
Teacher |
1 Clarifying learning intentions and criteria for success |
2 Engineering effective class-room discussions and other learning tasks that elicit evidence of student understanding |
3 Providing feedback that moves learners forward |
|
Peer |
Understanding and sharing learning intentions and criteria for success
|
4 Activating students as instructional resources for one another
|
|
|
Learner |
Understanding learning intentions and criteria for success
|
5 Activating students as the owners of their own learning
|
|
Figure 2: Aspects of formative assessment (Black & Wiliam, 2009).
The role of technologies
The main technologies that were used successfully for feedback processes in College-based learning with beginning teachers and were felt to add value are summarised in Table 1. In their own teaching some of the beginning teachers experimented with these technologies and their use increased over the three years of the study. Their view was that all of these technologies could add value for their feedback processes. In addition some experimentation in College sessions was conducted with an electronic voting system. Although myself as tutor and the beginning teachers felt that this was entertaining, the consensus was that it added little in this context beyond what could be achieved through a show of hands or use of the thumb tool. The “thumb tool” was used to indicate clear understanding (thumbs up), some confusion (thumbs horizontal) and very confused (thumbs down). Those beginning teachers who experimented with electronic voting systems in schools did feel that they added value because their students were confident to answer whereas they had not yet achieved the culture of openness and honesty which empowered their students to admit their difficulties in public. Furthermore these systems gave the beginning teachers instant feedback about their students’ understanding which enabled them to target their questions and support.
Table 1 Facilities provided by the Technology that enabled feedback processes (adapted from (Webb, 2010 (in press))
Perceived value of types of feedback
In focus group discussions beginning teachers, after an initial struggle to identify the formative assessment and feedback reached a consensus that there was value in all the various methods of delivery of feedback e.g.
They are all useful for different reasons. So someone giving you oral feedback is great because if you don't understand you can follow up on it. Written comments are good because you can go back and reflect on them later. (Beginning teacher Focus group March 2010)
They also valued both feedback through electronic means and other means.
The value of peer feedback compared with teachers’ comments was more contentious. Students in the JAFA project in a Year 6 class where formative assessment was embedded were well aware of the value of peer feedback, e.g.:
S2 in year five we didn’t do things like peer marking much and in year six we have done quite a lot of that, and I just think it helps me improve.
S3 Yeah, like in year five we didn’t really know what to get better on but now that we have done all this peer marking we know where we have to improve.
Ryefields Year Six Students Group Interview
S1 Because we can learn from our partners. It helped me to think
S2 By looking at lots of comments you can work harder next time
S3 You’re learning from what other people might say
Heatherbank Year Six Students class discussion
Beginning teachers, on the other hand, had spent many years in a system where teacher judgements were valued and outcomes of external examinations were key to success. By February beginning teachers had begun to see the value of peer assessment e.g.:
When I started I wanted to know what the teacher/lecturer has to say but now it ( peer assessment) is important but I still want to hear what my lecturer has to say because they've been out there in the world and they know. (Beginning teacher Focus group February 2010)
Some were doubtful of the accuracy of peer feedback compared to that of teachers because they were aware themselves of “shying away from negative aspects” (Beginning teacher Focus group February 2010).
Thus teacher feedback was seen as important for giving a clear accurate view of how students were getting on. Peer feedback was also valued but for different reasons: new perspectives, ideas to think about etc., for example:
I think peer feedback is useful but you need to take it in a slightly different way from feedback from tutors. You need to think why do they think that? What experience have they had? (Beginning teacher Focus group February 2010).
An obvious advantage of peer feedback, but one which was not volunteered in focus groups with beginning teachers is the increase in teaching resource available which is summarised well by this Year 6 student:
Like Miss Elliot says we are all little teachers and we can help each other, and then we learn.
Feedback as part of wider pedagogical processes
The previous section including Table 1 summarised the feedback processes in which technologies were deployed but it is important for these processes to be part of a series of pedagogical steps. These uses of technologies were always incorporated in a sequence of activities that may include group discussions with hand written notes or diagrams, annotating printouts etc. Sequences of pedagogical processes often comprised of complex sequences of activities alternating between whole class and group work both in the PGCE work and in the primary classes as illustrated in Figure 1. In order to analyse this complexity consideration was given to the temporal aspects of talk in learning (Mercer, 2007) and the interrelationships between sequences of activities using frameworks derived from Activity Theory (Webb, 2010 (in press); Webb & Jones, 2009; G. Wells, 2007; Gordon Wells & Mejia Arauz, 2006). In any one pedagogical sequence the technologies can perform a variety of different functions as listed in Table 1.
Sequences of activities in both the PGCE College sessions and the primary classrooms were generally highly planned and engineered. Furthermore, as with the PGCE course some activities were designed to enable students to work together using peer feedback with only the minimal teacher input required to achieve the learning intentions. Typically this meant that the teacher was still kept completely occupied supporting students but was able to devote more time to those who were finding difficulties. For example Figure 2 shows a typical generalised activity sequence for many of the group planning tasks that beginning teachers were set.
