Well, most peple seem to get into a bit of a mess when it comes to assessmnt for 21st century learning. I've been battling away, making quite useful prgress I think, to stop assessment being the barrier that it has so often been. A host of projects - from the International Certificate of Digital Creativity (that very much informed Edexcel's DIDA) through to the remarkable QCA funded eVIVA project where children ended their year of learning with a full viva via their mobile phones.
Why the QCA, after their KS3 ICT tests have collapsed, didn't pick up the extraordinary success of eVIVA (they were vocal enough about its successes at the time) is completely beyond me. But three steps forward and two back I gess. We'll get there... and there is exciting news to come on assessment projects - watch this space...
Anyway, this was a turn-of-the-century piece about assessment. Still holds good today (and probably tomorrow, sadly).
----------------------------------
300 word summary if you are in a hurry:
Computers are everyday tools for us all, seen or unseen, but their value in learning is as tools for creativity and learning rather than as machines to “deliver” the curriculum. These tools, in our children’s hands, are forever pushing the envelope of expertise that previous technologies excluded them from: they compose, quantise and perform music before acquiring any ability to play an instrument, they shoot, edit and stream digital video before any support from media courses, they produce architectural fly-throughs of incredible buildings without any drafting or 2D skills, they make stop frame animations with their plasticine models, they edit and finesse their poetry, they explore surfaces on their visual calculators, swap ideas with scientists on-line about volcanic activity, follow webcam images of Ospreys hatching, track weather by live satellite images, control the robots they have built and generally push rapidly at the boundaries of what might be possible, indeed what was formerly possible, at any age. Little of this was easily achieved in the school classroom ten years ago although the many projects emanating from Ultralab over that decade offered clear enough indicators of what might be possible. The challenge here is to criterion referencing. So often the cry of the teacher “that work is better than my degree exhibition piece!” reflects a substantial step change in both the age at which a creative act can be enjoyed and the quality of the tools supporting that creativity.
----------------------------------------
Before we get too excercised about the revolutionary impact that ICT might have on assessment, it is quite instructive to reflect on the extent to which our current assessment techniques have been distorted by the rather poor technologies of a past century. For example, we are seeing the second generation of children who have enjoyed mastery of the word processor, with all its agility and opportunities to revise or refine, yet in their assessments this “wired” generation are very rarely offered a chance to demonstrate their new literacy and creativity. Yet the ubiquitous word processor, in various guises and brands, has been found in classrooms since the beginning of the 1980s. The consequence is, as many moderators have found, that reduced to a pen, children must rely on a patchwork of arrows, insertions, and crossings out as they discover just what a poor vehicle for their new creativity and literacy the pen is.
Another example is the way that weak past technologies drove oracy out of our schools; every parent, every employer, every teacher and certainly all children will testify to the importance of the spoken word in communicating, critiquing, collaborating and demonstrating understanding. Yet, until we reach the giddy heights of a PhD with its interrogative viva, an oracy element in our assessment experience is almost entirely absent. Why is this? Simply because our poor technology offered no solution to the need for rapid moderation of children’s work. We had no tools for rapid indexing or for skimming speech, no solutions for taxonomising spoken contributions, and because the tools were non existent, we conveniently forgot that the thing we were finding impossible to moderate was nevertheless fundamentally important. Probably, we should have trusted teachers more and accredited their judgement; countless report slips had already reflected : "if only her written work was as good as her spoken contributions she would be top of the class".
It is clear that there is a mismatch between the assessments we want to do and what old technology, pens, OMR and the like, allow us to do. Mercifully, we do now have great tools for dealing with spoken contributions - indeed for very many years our various national security agencies have been turning our phone calls into text with considerable accuracy, which begs the question about why we have been so slow to harness the same well documented, commercially available technologies to assessment. Good enough for GCHQ should be good enough for GCSE too!
Part of the answer to that conundrum lies in the need of policy makers and politicians to demonstrate that they have made a difference, trapping us in the un-ambitious quicksand of criterion referencing. “We spent more and look, things are better” is the clear message that re-elects parties. Sadly, given the pace of change of new technology and the speed with which the children’s sense of entitlement accelerates alongside it, the best we know that we will be able to offer is “We spent more and look, things are different but here is our evidence to show that different is better” which will win less hearts and minds at the ballot box unless the evidence that different is better is clearly presented, is persuasive and is in the public domain. One of many sorrows about the impact of much current assessment of children is that their very best work, the work that results from focus and intense application, is produced for the assessment process yet is rarely seen or celebrated by any audience other than examiners and moderators. In a recent exploration of creativity (part of an Ultralab national education policy consultancy for a Pacific rim nation) children were asked about the piece of work they were most proud of having produced. Over three quarters reported that the work was produced as a direct assessment output. The supplementary questions “where is that work now?” and “did your parents see that work?” we largely, met with “don’t know” and “no”. This is individually tragic, but nationally a substantial missed opportunity; if we are to win hearts and minds in moving education forward ambitiously this evidence of excellence has a key role in persuading nations that children’s work, even where it is very different from what went before, is showing a real improvement.
And different is inevitable. New technology advances apace and this, of course, is well documented; computers are designed with computers, the better they are the faster that even better ones can be designed. Computers are everyday tools for us all, seen or unseen, but their value in learning is as tools for creativity and learning rather than as machines to “deliver” the curriculum. These tools, in our children’s hands, are forever pushing the envelope of expertise that previous technologies excluded them from: they compose, quantise and perform music before acquiring any ability to play an instrument, they shoot, edit and stream digital video before any support from media courses, they produce architectural fly-throughs of incredible buildings without any drafting or 2D skills, they make stop frame animations with their plasticine models, they edit and finesse their poetry, they explore surfaces on their visual calculators, swap ideas with scientists on-line about volcanic activity, follow webcam images of Ospreys hatching, track weather by live satellite images, control the robots they have built and generally push rapidly at the boundaries of what might be possible, indeed what was formerly possible, at any age. Little of this was easily achieved in the school classroom ten years ago although the many projects emanating from Ultralab over that decade offered clear enough indicators of what might be possible. The challenge here is to criterion referencing. So often the cry of the teacher “that work is better than my degree exhibition piece!” reflects a substantial step change in both the age at which a creative act can be enjoyed and the quality of the tools supporting that creativity.
Unfortunately this extraordinary potential for progress comes at a time when we are wedded to an assessment model that satisfies us if children attempt the same activities as they predecessors, but do so a little better. In an age of rapid progress is this fatally masks rapidly falling standards and stultifies ambition. On the one hand new technology supports children’s abi;lity to make new leaps of imagination and creativity, yet a reliance on criterion referencing denies the value of that imagination and creativity by excluding it under the feeble pretext that it wasn’t how we did it before. The result is that schools habitually confiscate or deny new technology removing everything from ball-point pens (“it will spoil your handwriting”) to mobile phones (“disruptive”) and teachers report that the best creativity they observe is in the non curriculum space of lunchtime clubs or out of school activities. We have failed to respond within the curriculum and its assessment process to these new opportunities for creativity. On the other hand the new tools that children are able to harness for their learning also deliver real productivity, but we have failed to respond to this productivity either by setting rapidly rising standards. Surely we know that children can write better with a word processor?; it offers them the opportunity to refine, revisit, draft, finesse and error check their work. They can take countless risks with their work without the penalty of longhand copying. Word processors bring them whole new strategies for creative writing (again, strategies that were largely once the domain of authors, not schoolchildren); they produce more and better work and this has been well documented since the 1980s. So, thirty years after the first word processors appeared in our classrooms, have we responded by setting much more ambitious targets for children's writing performance? Of course not, we simply exclude the word processor from the examination room under the pretext that it won’t allow us to compare their ambitious work with the pen written output of a previous age. Thus we rob our children of both opportunity and ambition. Or worse, we capture the productivity of the word processor for our own assessment administration convenience. Wouldn’t it be good if the computer could mark our assessments for us ? Bluntly no, it would be such a wasted opportunity, but it would be good if the new things that children do with computers might be acknowledged by the ambition and creativity of the targets we set them.
To attempt to face down the constancy of change, the certainty of uncertainty, with the leaden yardstick of past history is palpably silly, but whilst we can take the phone or the word processor out of children’s hands we cannot take the accompanying sense of entitlement out of their heads. A major impact of new technology has been the rapidity with which our attitudes change. Ultralab drove the national rebranding of IT (the sterile “information technology” label) into ICT where the added “C” stood for Communication. New technologies are essentially communication tools and whether that was the visual spreadsheets of the 70s, the desktop publishing of the 80s, the world wide web of the 90s or the pocketability of the new millennium these have been communication revolutions. Incidentally, it is interesting to reflect on how these four massive revolutions have impacted on the assessment process so far: very little. But his revolution in communication has brought with it a sense of entitlement that we might all have a voice in the communication age. We might communicate with others. The effects of this entitlement to communicate can be seen in many sectors: for example, in the way that the under 35s are falling out of representative democracy around the world (“why won’t our government listen to us!”). A sense of entitlement to communicate carries with it a sense of audience and of being an audience for others. Even those for whom the symmetry of communication (essentially a two way act) has been difficult concept to embrace have begun to realise that without this two way conduit we lock people out, deny their new sense that technology will give them a voice and role and generate dissatisfaction, alienation and anomie. A major Ultralab project with the BBC started in 2002 and finally sees children empowered to make, broadcast and stream their own programmes, by children for children, using a variety of broadcasting opportunities including digital terrestrial channels and large screens at football stadiums. In another research project at Ultralab Stan Owers surveyed over 3,000 “A” level students, in 1996 to build a picture of their views of the curriculum. its creativity, relevance, interest and more. Some of the data produced was unsurprising: children interested in a subject tended to be studying it for example. However some deeply disturbing trends emerged too for which much blame must be laid at the door of a assessment processes that values product over process and that had failed to recognise the changing tide of children’s expectations of this ICT age. For example, the question “how creative does the curriculum all you to be in Mathematics?” brought a predictably dismal response from those who had rejected Mathematics as an “A” level specialisms, but it also brought a near identical overlaid graph of responses from those who had opted for mathematics as an “A” level option. We should be dismayed by this. What had changed of course was not the curriculum, indeed that may be a substantial part of the problem, but the student’s expectations that their learning world might be a creative world too. Repeating the survey in 2002, again with over 3,000 students Owers and Constable are already confident part way through their analysis that in some ways the situation has declined further.
Ultralab runs an annual summer school. Each year a group of around 100 children are given tough tasks that recent changes in technology have made cheaply accessible to school age children. Their tasks are phrased in an open ended way, and they spend a relatively small amount of time in the lab being introduced to both task and technology. In the summer of 2001, for example, students were confronted with a challenge to produce a “framed” artwork to be shown at the Victoria and Albert museum, if good enough. The artwork was to be executed not in water-colour or acrylic, but in video and sound. The simple rules include a requirement to incorporate the “artists” names as some form of “signature” and a limit of two minutes for what was to be a “looping” collage of images and sound. Each of the one hundred children was given a unique word, for example: Dazed, Smoothly or Tragedy. Each group of four children, rejected three words and adopted one as their theme for this tough collaborative challenge. Only an hour and a half was set aside to introduce the task, the software (iMovie), the computers (iMacs), the cameras (hand held Canon DVs) and to introduce the intended display area in the V&A. Staff and parents were under strict guidance to support the summer schoolers by driving them around and helping with other logistic support, but not to intervene in the production or creative processes. The children we left in no doubt that their work needed to be good enough to be shown at the V&A and of course had the ambitious outcomes of previous, very different, summer schools to help set a goal for their ambitions. After the summer, and after spending varied amounts of time, typically spread across parts of a week, the children came together to show the fruits of the labour and imagination. Each year the children demonstrate an ambition beyond the expectations of school and curriculum. The teachers are a key part of the process that sets the annual tasks, but in the first year one commented “this has been like an epiphany to me; I had literally no idea of the quality of work that these children might be able to produce. It has changed my whole view of what we teach and what we should teach”. The 2001 “video collage” group were no exception and produced work that stood up to the quality of the V&A and was previewed to an ecstatic audience. Interestingly a parent confided at the end of the preview that his film and media degree final piece had been eclipsed by the work of these mostly 12 year old children that he had just witnessed. This annual creativity feast confirms several key lessons for assessment:
Firstly, it is clear that the entirely new tasks set annually do not map easily, indeed at all, onto any pretence of criterion referencing. There is not even an existing genre to pastiche; this is designedly entirely new work yet technology has allowed the summer school children into the domain of experts very early in their learning lives. It is also clear however that the teachers, who are effectively action researchers on the ground during the exercise, can make clear and valid judgements about quality to the extent of being clear about what is “good enough”, but beyond that the key feedback from children reporting their processes (for example in the formal interview and “crit” that is a part of their V&A show) is critical to injecting any granularity of judgement beyond that “good enough” hurdle. The product in every case was engaging, but the reported processes offered a very fresh perspective, often stunning, but always moving our understanding forwards . Describing the construction of the final pieces enabled the deconstruction and critiquing to be a much more subtle affair. Finally, the clear sense of impending audience served not only as a prime motivator, but was a useful adjunct to the learning process.
In the light of all this it is not unreasonable to ask “Whither Assessment?”. A ground breaking QCA / Ultralab project seeks to explore how some of these lessons might inform the process of assessment. Approaching the task of Key Stage 3 ICT assessment the project proposes three stages:
The first stage is a mapping, by he student, of what their starting points are. At KS 3 students have brought a wide and welcome diversity of experience into their secondary schools. Standards are not about standardisation and helping each learner to map the “beginning of the KS3 journey” is a necessary first step. There is a second task to this first stage, explained below.
The second stage sees the student posting milestones onto a common website to mark what they consider to be pivotal or significant moments in their work. These milestones might be images, video, sounds, speech or text; each is “annotated” initially by the student but subsequently by peers, teachers, even moderators using a device which effectively allows “margin notes” on the web page. A maximum of six milestones is anticipated at this early stage but the project is highly iterative and doubtless feedback from students and their teachers will inform that decision.
The final stage sees students, and their teachers concluding that a sufficient “distance” from the initial starting point has been travelled to be worthy of credit and at that point the student nominates a time and place and prepares themselves for a telephone call. That call, using text to speech technology poses a series of questions for the student to defend, perhaps in the manner of a “Crit” or “Viva”, indeed we are calling this process the eViva. As with any answerphone the student can re-record their responses to each question at any time. Crucially the questions posed are themselves selected from a list by the student at the initial stage, with feedback advice about the portfolio of questions selected. In this way the student is already encouraged towards meta-level reflection about their learning through an awareness of the questions to be defended against at the end. The telephone makes an interesting conduit because sophisticated voice to text technology can turn the student’s defence into text for moderation. A fundamental design intention of this new assessment strategy however is to celebrate in a public space, a website, the children’s work, with some of their milestones (they choose which) and their spoken comments.
Is this the future of assessment? Certainly it is part of the debate that will build such a future, but we can be confident that we need many more such ambitious explorations of the new opportunities presented by both technology and children’s new confidence with it before we can be certain that the express train of children’s capability is not running full tilt at the buffers of the assessment system.
_______________________________________________
© Prof Stephen Heppell 2001
