Tuesday, 20 February 2001

Technology? It's a people thing.

In New Zealand I was interviewed on the excellent Kim Hill Show, then a radio programme. Kim wanted a few "think pieces" to read before the interview and, true to her word, had read everything I posted for her. She was witty and incisive. One of the bits i mailed over to her was a short article I was asked to write for CGI magazine and I've posted it below. Re-reading, it is always interesting to find words like: "Looking ahead it is starkly clear just what the future holds." when they were written in 2001 (!) but I'm pleased with what i wrote then and, of course, it has all come true... hasn't it? Here is what I wrote:

As the cool tools of the creative industries get paradoxically hotter and hotter a key question is whether public sector organisations can keep up. Such a question presumes some direction to the headlong rush of technology and further assumes that the direction is forwards(!). This may not necessarily be the case. It's a curious thing that as technology moves faster and faster the failed ideas and misapprehensions of previous technology eras seem to come round quicker and quicker, whilst great ideas from previous eras don't seem to re-emerge with anything like the same tenacity. Why should this be? History may have some lessons for us, to explain why.

One enduring misapprehension throughout technological evolution is that what people want from their equipment will only ever be the delivery of "professional" content, quality assured from the centre, to consume passively at leisure; content, we are led to believe, is king. Of course any fool can see that what people actually do with technology is find new ways to communicate, collaborate, commune, annotate, participate and narrate with others. Good content matters but is only an increasingly small part of the jigsaw. Content will always be deposed by communication and community.

When telephones first appeared, London owners were able to "enjoy" broadcasts down their phone lines. The early pioneers saw no prospect that individuals would have much useful to say into their own phones which were not seen as symmetrical devices; you listened, but you didn't contribute. Of course to a telephone owner a 'phone was quickly perceived to be an opportunity to participate, something that you spoke into because you had plenty to say and thus a world of telecommunications was born. It's good to talk. The same misapprehension, that everyday people had no useful contribution to make, underpins "Let nation speak unto nation". The reality of course is less grand; it is that Kirsten in Canvey shall speak unto the rest of her county as she warns about trouble on the A12. People participate. In the 60's radio waves were "subverted" as pirate radio stations, anchored in the North Sea, broadcast content that was neither quality controlled nor centrally originated. A less likely subversive than Tony Blackburn would be hard to find, yet at the time he and his colleagues were seen, implausibly, as a threat to national stability; in practice the pirate radio stations had simply confirmed that the newly accessible technology of radio was less about broadcasting and more about contribution or annotation as zany "DJs" risked all by airing their views across the ether.

Still later the videodisc sought to bring another kind of revolution, this time to the screens in our learning environments. These huge silver platters offered yet more high quality content, but this time with the added feature of better navigation through their linear contents. As universities and schools struggled to afford these revolutionary players it was already clear that they would bomb. People wanted to have some control over what was on the discs, to be able to add value and to be able to author this new media. The cost of doing so was prohibitively high and the technology failed. People wanted more than to be passive viewers with a bit of interaction. The National Archive of Educational Computing at Ultralab houses hundreds of these discs, many in their original unopened cellophane; these unbroken wrappers tell all there is to know about videodisc technology. Thinking they should "keep up" public institutions tried to buy the players but by the time they had saved enough (with a few exceptions, Florida State for example) the technology was dead. The few successful Videodisc projects, like the BBC's Domesday discs or Apple's Visual Almanac gave users a role and an opportunity to contribute, but these markers for a better future were rare. At Ultralab (then the Learning Technology Research Centre) we were connecting computers to videodisc players to allow people to author their own work and being told that we misunderstood the technology. We didn't.

After Videodiscs came an explosion of media storage. Of half a dozen CD "standards" CDi claimed to lead the pack and was touted as the next "next great thing". Again the personal authoring costs were prohibitive, again it bombed. This time public institutions were a little more circumspect and waited before saving up, let alone buying, but conferences around the world worried that schools and universities were missing the important new direction. In truth the really important new direction lay elsewhere; it lay in people becoming increasingly hungry to participate and it was delivery technology that had, quite literally, lost the plot. At Ultralab our first CD-ROM in 1989 was "Tools for Multimedia" to help people make their own CD's cheaply. Publishers phoned us to say that such democracy and cheapness would kill the market. It didn't.

In the early '90s all this changed with the World Wide Web as a new collaborative, symmetrical, technology, designed for exchanging papers and research, exploded into our lives. The initial seduction of the web lay in the ease with which many could make, create and do on-line. As many "experts" including, famously, Bill Gates undervalued the democracy of this new "web" technology real people were having fun and building websites. At last a technology gave them an active role and the sites grew at exponential rates as many exercised that opportunity. Public sector organisations woke up to a real revolution and this time spent money as the National Grid for Learning became a sound bite to represent a huge real investment into schools. Some though were less easily persuaded. At Ultralab in these early web days we were running big database driven sites like our DTI Schools OnLine project with personalised pages for participants and later our biggest contributory projects, like Tesco SchoolNet 2000, were entering the Guinness Book of Records, but we were still criticised because "that isn't how the Internet should be" as one agency put it. But of course it was.

As the web revolution gathered pace telecommunications companies, after a century of allowing people to make their own personal contributions one-to-one missed the whole one-to-many opportunity that the web had confirmed to be seductive. Instead they sought to use their empowering phone lines for video-on-demand. But, new technology or not, there had been no ripple in the gene pool and as before people still wanted simply to participate; it was never likely that broadcasting would replace conversation. After some spirited trials video-on-demand technology predictably bombed, but those enthusiastic for it scampered away to suggest instead that the future of broadcasting might be a choice of viewing angles via an interactive box. It won't be. As Video on Demand was bombing the real phone revolution, mobiles, was giving people more and more opportunities to make their contribution together with some interesting new ways (like the asynchronous SMS messaging) to do it. The worst technology ostriches by this time were wondering why Big Brother was so successful whilst drama seemed to be slipping away and dreaming about a Big Brother CD perhaps?. Someone should tell them, it's only kind to do so. In the lab we are currently working with annotated digital video that offers media redundancy and a non linear route to narrative. Siren voices (and they sound curiously familiar) are telling us we should be more into ADSL. "Is ADSL symmetrical?" we ask, impishly. "No" they say, "but people will be able to choose what they receive...". Oh dear.

Looking ahead it is starkly clear just what the future holds. A string of new content delivery technologies ("wouldn't movies on a mobile phone be a great idea?". No!) will bomb as the same people keep returning, convinced that past failures were only due to lack of bandwidth, poor IPR legislation, immature standards (hah!), or a lack of "proper channel". In the 1980s when the desktop publishing revolution took off pundits dismissed it ("there is no substitute for a good literary editor"), the MP3 revolution took off as pundits dismissed it too ("there is nothing to match the skill of a good agent") but each resulted in more writing or more live music. Television will succumb. People will use the revolutionary tools like iMovie or Final Cut Pro and beyond to become video contributors and just as no child ever watches a DVD disc right though (they prefer to navigate to the best bits) so the ownership of narrative will pass to the viewer from the broadcaster. As this all happens pundits will be explaining that there is no substitute for the commissioning process and looking puzzled.

At Ultralab we are using an eclectic and evolving mix of technologies to run vast media rich communities of practice for the UK's headteachers, for children excluded from school, for health professionals and for countless other groups. These successful projects use cool technology, but the shared constant is a simple understanding that people like to participate, enjoy a sense of audience, want to make a difference, need some facilitation, value being social. It's not about "keeping up", it's a people thing.

© Stephen Heppell 2001

Monday, 29 January 2001

Assessment and new technology: new straightjackets or new opportunities?

Written for the book “Assessment Futures” published by the UK's Qualification and Curriculum Authority 2002.

Well, most peple seem to get into a bit of a mess when it comes to assessmnt for 21st century learning. I've been battling away, making quite useful prgress I think, to stop assessment being the barrier that it has so often been. A host of projects - from the International Certificate of Digital Creativity (that very much informed Edexcel's DIDA) through to the remarkable QCA funded eVIVA project where children ended their year of learning with a full viva via their mobile phones.

Why the QCA, after their KS3 ICT tests have collapsed, didn't pick up the extraordinary success of eVIVA (they were vocal enough about its successes at the time) is completely beyond me. But three steps forward and two back I gess. We'll get there... and there is exciting news to come on assessment projects - watch this space...

Anyway, this was a turn-of-the-century piece about assessment. Still holds good today (and probably tomorrow, sadly).

300 word summary if you are in a hurry:

Computers are everyday tools for us all, seen or unseen, but their value in learning is as tools for creativity and learning rather than as machines to “deliver” the curriculum. These tools, in our children’s hands, are forever pushing the envelope of expertise that previous technologies excluded them from: they compose, quantise and perform music before acquiring any ability to play an instrument, they shoot, edit and stream digital video before any support from media courses, they produce architectural fly-throughs of incredible buildings without any drafting or 2D skills, they make stop frame animations with their plasticine models, they edit and finesse their poetry, they explore surfaces on their visual calculators, swap ideas with scientists on-line about volcanic activity, follow webcam images of Ospreys hatching, track weather by live satellite images, control the robots they have built and generally push rapidly at the boundaries of what might be possible, indeed what was formerly possible, at any age. Little of this was easily achieved in the school classroom ten years ago although the many projects emanating from Ultralab over that decade offered clear enough indicators of what might be possible. The challenge here is to criterion referencing. So often the cry of the teacher “that work is better than my degree exhibition piece!” reflects a substantial step change in both the age at which a creative act can be enjoyed and the quality of the tools supporting that creativity.


Before we get too excercised about the revolutionary impact that ICT might have on assessment, it is quite instructive to reflect on the extent to which our current assessment techniques have been distorted by the rather poor technologies of a past century. For example, we are seeing the second generation of children who have enjoyed mastery of the word processor, with all its agility and opportunities to revise or refine, yet in their assessments this “wired” generation are very rarely offered a chance to demonstrate their new literacy and creativity. Yet the ubiquitous word processor, in various guises and brands, has been found in classrooms since the beginning of the 1980s. The consequence is, as many moderators have found, that reduced to a pen, children must rely on a patchwork of arrows, insertions, and crossings out as they discover just what a poor vehicle for their new creativity and literacy the pen is.

Another example is the way that weak past technologies drove oracy out of our schools; every parent, every employer, every teacher and certainly all children will testify to the importance of the spoken word in communicating, critiquing, collaborating and demonstrating understanding. Yet, until we reach the giddy heights of a PhD with its interrogative viva, an oracy element in our assessment experience is almost entirely absent. Why is this? Simply because our poor technology offered no solution to the need for rapid moderation of children’s work. We had no tools for rapid indexing or for skimming speech, no solutions for taxonomising spoken contributions, and because the tools were non existent, we conveniently forgot that the thing we were finding impossible to moderate was nevertheless fundamentally important. Probably, we should have trusted teachers more and accredited their judgement; countless report slips had already reflected : "if only her written work was as good as her spoken contributions she would be top of the class".

It is clear that there is a mismatch between the assessments we want to do and what old technology, pens, OMR and the like, allow us to do. Mercifully, we do now have great tools for dealing with spoken contributions - indeed for very many years our various national security agencies have been turning our phone calls into text with considerable accuracy, which begs the question about why we have been so slow to harness the same well documented, commercially available technologies to assessment. Good enough for GCHQ should be good enough for GCSE too!

Part of the answer to that conundrum lies in the need of policy makers and politicians to demonstrate that they have made a difference, trapping us in the un-ambitious quicksand of criterion referencing. “We spent more and look, things are better” is the clear message that re-elects parties. Sadly, given the pace of change of new technology and the speed with which the children’s sense of entitlement accelerates alongside it, the best we know that we will be able to offer is “We spent more and look, things are different but here is our evidence to show that different is better” which will win less hearts and minds at the ballot box unless the evidence that different is better is clearly presented, is persuasive and is in the public domain. One of many sorrows about the impact of much current assessment of children is that their very best work, the work that results from focus and intense application, is produced for the assessment process yet is rarely seen or celebrated by any audience other than examiners and moderators. In a recent exploration of creativity (part of an Ultralab national education policy consultancy for a Pacific rim nation) children were asked about the piece of work they were most proud of having produced. Over three quarters reported that the work was produced as a direct assessment output. The supplementary questions “where is that work now?” and “did your parents see that work?” we largely, met with “don’t know” and “no”. This is individually tragic, but nationally a substantial missed opportunity; if we are to win hearts and minds in moving education forward ambitiously this evidence of excellence has a key role in persuading nations that children’s work, even where it is very different from what went before, is showing a real improvement.

And different is inevitable. New technology advances apace and this, of course, is well documented; computers are designed with computers, the better they are the faster that even better ones can be designed. Computers are everyday tools for us all, seen or unseen, but their value in learning is as tools for creativity and learning rather than as machines to “deliver” the curriculum. These tools, in our children’s hands, are forever pushing the envelope of expertise that previous technologies excluded them from: they compose, quantise and perform music before acquiring any ability to play an instrument, they shoot, edit and stream digital video before any support from media courses, they produce architectural fly-throughs of incredible buildings without any drafting or 2D skills, they make stop frame animations with their plasticine models, they edit and finesse their poetry, they explore surfaces on their visual calculators, swap ideas with scientists on-line about volcanic activity, follow webcam images of Ospreys hatching, track weather by live satellite images, control the robots they have built and generally push rapidly at the boundaries of what might be possible, indeed what was formerly possible, at any age. Little of this was easily achieved in the school classroom ten years ago although the many projects emanating from Ultralab over that decade offered clear enough indicators of what might be possible. The challenge here is to criterion referencing. So often the cry of the teacher “that work is better than my degree exhibition piece!” reflects a substantial step change in both the age at which a creative act can be enjoyed and the quality of the tools supporting that creativity.

Unfortunately this extraordinary potential for progress comes at a time when we are wedded to an assessment model that satisfies us if children attempt the same activities as they predecessors, but do so a little better. In an age of rapid progress is this fatally masks rapidly falling standards and stultifies ambition. On the one hand new technology supports children’s abi;lity to make new leaps of imagination and creativity, yet a reliance on criterion referencing denies the value of that imagination and creativity by excluding it under the feeble pretext that it wasn’t how we did it before. The result is that schools habitually confiscate or deny new technology removing everything from ball-point pens (“it will spoil your handwriting”) to mobile phones (“disruptive”) and teachers report that the best creativity they observe is in the non curriculum space of lunchtime clubs or out of school activities. We have failed to respond within the curriculum and its assessment process to these new opportunities for creativity. On the other hand the new tools that children are able to harness for their learning also deliver real productivity, but we have failed to respond to this productivity either by setting rapidly rising standards. Surely we know that children can write better with a word processor?; it offers them the opportunity to refine, revisit, draft, finesse and error check their work. They can take countless risks with their work without the penalty of longhand copying. Word processors bring them whole new strategies for creative writing (again, strategies that were largely once the domain of authors, not schoolchildren); they produce more and better work and this has been well documented since the 1980s. So, thirty years after the first word processors appeared in our classrooms, have we responded by setting much more ambitious targets for children's writing performance? Of course not, we simply exclude the word processor from the examination room under the pretext that it won’t allow us to compare their ambitious work with the pen written output of a previous age. Thus we rob our children of both opportunity and ambition. Or worse, we capture the productivity of the word processor for our own assessment administration convenience. Wouldn’t it be good if the computer could mark our assessments for us ? Bluntly no, it would be such a wasted opportunity, but it would be good if the new things that children do with computers might be acknowledged by the ambition and creativity of the targets we set them.

To attempt to face down the constancy of change, the certainty of uncertainty, with the leaden yardstick of past history is palpably silly, but whilst we can take the phone or the word processor out of children’s hands we cannot take the accompanying sense of entitlement out of their heads. A major impact of new technology has been the rapidity with which our attitudes change. Ultralab drove the national rebranding of IT (the sterile “information technology” label) into ICT where the added “C” stood for Communication. New technologies are essentially communication tools and whether that was the visual spreadsheets of the 70s, the desktop publishing of the 80s, the world wide web of the 90s or the pocketability of the new millennium these have been communication revolutions. Incidentally, it is interesting to reflect on how these four massive revolutions have impacted on the assessment process so far: very little. But his revolution in communication has brought with it a sense of entitlement that we might all have a voice in the communication age. We might communicate with others. The effects of this entitlement to communicate can be seen in many sectors: for example, in the way that the under 35s are falling out of representative democracy around the world (“why won’t our government listen to us!”). A sense of entitlement to communicate carries with it a sense of audience and of being an audience for others. Even those for whom the symmetry of communication (essentially a two way act) has been difficult concept to embrace have begun to realise that without this two way conduit we lock people out, deny their new sense that technology will give them a voice and role and generate dissatisfaction, alienation and anomie. A major Ultralab project with the BBC started in 2002 and finally sees children empowered to make, broadcast and stream their own programmes, by children for children, using a variety of broadcasting opportunities including digital terrestrial channels and large screens at football stadiums. In another research project at Ultralab Stan Owers surveyed over 3,000 “A” level students, in 1996 to build a picture of their views of the curriculum. its creativity, relevance, interest and more. Some of the data produced was unsurprising: children interested in a subject tended to be studying it for example. However some deeply disturbing trends emerged too for which much blame must be laid at the door of a assessment processes that values product over process and that had failed to recognise the changing tide of children’s expectations of this ICT age. For example, the question “how creative does the curriculum all you to be in Mathematics?” brought a predictably dismal response from those who had rejected Mathematics as an “A” level specialisms, but it also brought a near identical overlaid graph of responses from those who had opted for mathematics as an “A” level option. We should be dismayed by this. What had changed of course was not the curriculum, indeed that may be a substantial part of the problem, but the student’s expectations that their learning world might be a creative world too. Repeating the survey in 2002, again with over 3,000 students Owers and Constable are already confident part way through their analysis that in some ways the situation has declined further.

Ultralab runs an annual summer school. Each year a group of around 100 children are given tough tasks that recent changes in technology have made cheaply accessible to school age children. Their tasks are phrased in an open ended way, and they spend a relatively small amount of time in the lab being introduced to both task and technology. In the summer of 2001, for example, students were confronted with a challenge to produce a “framed” artwork to be shown at the Victoria and Albert museum, if good enough. The artwork was to be executed not in water-colour or acrylic, but in video and sound. The simple rules include a requirement to incorporate the “artists” names as some form of “signature” and a limit of two minutes for what was to be a “looping” collage of images and sound. Each of the one hundred children was given a unique word, for example: Dazed, Smoothly or Tragedy. Each group of four children, rejected three words and adopted one as their theme for this tough collaborative challenge. Only an hour and a half was set aside to introduce the task, the software (iMovie), the computers (iMacs), the cameras (hand held Canon DVs) and to introduce the intended display area in the V&A. Staff and parents were under strict guidance to support the summer schoolers by driving them around and helping with other logistic support, but not to intervene in the production or creative processes. The children we left in no doubt that their work needed to be good enough to be shown at the V&A and of course had the ambitious outcomes of previous, very different, summer schools to help set a goal for their ambitions. After the summer, and after spending varied amounts of time, typically spread across parts of a week, the children came together to show the fruits of the labour and imagination. Each year the children demonstrate an ambition beyond the expectations of school and curriculum. The teachers are a key part of the process that sets the annual tasks, but in the first year one commented “this has been like an epiphany to me; I had literally no idea of the quality of work that these children might be able to produce. It has changed my whole view of what we teach and what we should teach”. The 2001 “video collage” group were no exception and produced work that stood up to the quality of the V&A and was previewed to an ecstatic audience. Interestingly a parent confided at the end of the preview that his film and media degree final piece had been eclipsed by the work of these mostly 12 year old children that he had just witnessed. This annual creativity feast confirms several key lessons for assessment:

Firstly, it is clear that the entirely new tasks set annually do not map easily, indeed at all, onto any pretence of criterion referencing. There is not even an existing genre to pastiche; this is designedly entirely new work yet technology has allowed the summer school children into the domain of experts very early in their learning lives. It is also clear however that the teachers, who are effectively action researchers on the ground during the exercise, can make clear and valid judgements about quality to the extent of being clear about what is “good enough”, but beyond that the key feedback from children reporting their processes (for example in the formal interview and “crit” that is a part of their V&A show) is critical to injecting any granularity of judgement beyond that “good enough” hurdle. The product in every case was engaging, but the reported processes offered a very fresh perspective, often stunning, but always moving our understanding forwards . Describing the construction of the final pieces enabled the deconstruction and critiquing to be a much more subtle affair. Finally, the clear sense of impending audience served not only as a prime motivator, but was a useful adjunct to the learning process.

In the light of all this it is not unreasonable to ask “Whither Assessment?”. A ground breaking QCA / Ultralab project seeks to explore how some of these lessons might inform the process of assessment. Approaching the task of Key Stage 3 ICT assessment the project proposes three stages:

The first stage is a mapping, by he student, of what their starting points are. At KS 3 students have brought a wide and welcome diversity of experience into their secondary schools. Standards are not about standardisation and helping each learner to map the “beginning of the KS3 journey” is a necessary first step. There is a second task to this first stage, explained below.

The second stage sees the student posting milestones onto a common website to mark what they consider to be pivotal or significant moments in their work. These milestones might be images, video, sounds, speech or text; each is “annotated” initially by the student but subsequently by peers, teachers, even moderators using a device which effectively allows “margin notes” on the web page. A maximum of six milestones is anticipated at this early stage but the project is highly iterative and doubtless feedback from students and their teachers will inform that decision.

The final stage sees students, and their teachers concluding that a sufficient “distance” from the initial starting point has been travelled to be worthy of credit and at that point the student nominates a time and place and prepares themselves for a telephone call. That call, using text to speech technology poses a series of questions for the student to defend, perhaps in the manner of a “Crit” or “Viva”, indeed we are calling this process the eViva. As with any answerphone the student can re-record their responses to each question at any time. Crucially the questions posed are themselves selected from a list by the student at the initial stage, with feedback advice about the portfolio of questions selected. In this way the student is already encouraged towards meta-level reflection about their learning through an awareness of the questions to be defended against at the end. The telephone makes an interesting conduit because sophisticated voice to text technology can turn the student’s defence into text for moderation. A fundamental design intention of this new assessment strategy however is to celebrate in a public space, a website, the children’s work, with some of their milestones (they choose which) and their spoken comments.

Is this the future of assessment? Certainly it is part of the debate that will build such a future, but we can be confident that we need many more such ambitious explorations of the new opportunities presented by both technology and children’s new confidence with it before we can be certain that the express train of children’s capability is not running full tilt at the buffers of the assessment system.

© Prof Stephen Heppell 2001