Improving online language assessment: Using Pecha Kucha to assess spoken production in English

Mejorando la evaluaci贸n ling眉铆stica en l铆nea: usando Pecha Kucha para evaluar la producci贸n oral en ingl茅s

Lorna Carson (Trinity College Dublin)

Art铆culo recibido: 11-10-2017 | Art铆culo aceptado: 06-11-2017

RESUMEN: En este reto E-LENGUA se busca mejorar la evaluaci贸n en l铆nea de la lengua inglesa usando entornos de aprendizaje virtual dentro del sistema de programas de lenguas de Trinity College de Dubl铆n. Nos centramos en usos espec铆ficos de la lengua, orientados a la acci贸n, y procedimientos integrados de evaluaci贸n por el profesor, entre iguales y autoevaluaci贸n. Este art铆culo propone una actividad de presentaci贸n dise帽ada para evaluar la producci贸n oral de ingl茅s en niveles B1 y B2 que puede emplearse en entornos tradicionales, en l铆nea o mixtos. El equipo del Trinity College Dublin explora los motivos y el uso de un formato de presentaciones innovador, Pecha Kucha, que se complementa con Blackboard Learn. El art铆culo explora el contexto de entornos de aprendizaje virtual como Blackboard Learn o Moodle y estudia el reto de la evaluaci贸n de lenguas en l铆nea. La tarea propuesta como evaluaci贸n busca potenciar la vinculaci贸n del aprendiz y su autorreflexi贸n a trav茅s de actividades compartidas de evaluaci贸n y asegurar la respuesta positiva frente a la negativa en la clase de lenguas.
ABSTRACT: In this E-LENGUA challenge to improve online English language assessment using virtual learning environments, undertaken within the institution-wide language programme at Trinity College Dublin, we focus on specific, action-oriented descriptions of language use, joined-up learning and assessment materials, and embedded self-, peer- and teacher-assessment procedures. This article presents a presentation activity designed to assess spoken production in English at B1 and B2 proficiency levels which can be used in a traditional, blended or online environment. The team at Trinity College Dublin explore the rationale and use of an innovative slide presentation format, Pecha Kucha, to be completed within Blackboard Learn. The article explores the context of virtual learning environments such as Blackboard Learn or Moodle and discusses the challenge of online language assessment in such an environment. The proposed assessment task aims to foster learner engagement and reflection through shared assessment activities, and to ensure positive rather than negative washback in the language classroom.

PALABRAS CLAVE: Evaluaci贸n, evaluaci贸n formativa, evaluaci贸n sumativa, entornos de aprendizaje virtual
KEY WORDS: Assessment, formative assessment, summative assessment, virtual learning environments (VLE)

The E-LENGUA project is financed by the KA2013 Strategic Partnerships Actions for Higher Education.

1. Introduction

This paper presents the E-LENGUA challenge of improving online English language assessment within virtual learning environments in higher education. We report on an innovative slide presentation task, designed and trialled within the institution-wide language programme at Trinity College Dublin. The language programme deploys specific, action-oriented descriptions of language use, joined-up learning and assessment materials, and embedded self-, peer- and teacher-assessment procedures (Carson, 2010; Carson, 2016) based on the taxonomic 鈥淐an do鈥 descriptor approach of the CEFR (Council of Europe, 2001). This contribution focuses on the formative assessment of Spoken Production in English at B1 and B2 proficiency methods using an innovative presentation format called Pecha Kucha, completed within Blackboard Learn. Below, we explore the context and affordances of virtual learning environments, and consider the challenges of online assessment of language proficiency. The remainder of the paper presents the design and presentation of the Pecha Kucha task, including instructions and assessment scheme.

2. The Context: Virtual Learning Environments

Multiple and sometimes interchangeable definitions are deployed in the discussion of e-learning and online educational systems or platforms, including Learning Management Systems (LMS), Virtual Learning Environments (VLE), Course Management Systems or Content Management Systems. Whilst distinctions can be drawn between the names given to some of the commercial applications of these systems, for the purposes of this paper, the term Virtual Learning Environment will be used to refer to types of web-based technology that allow educational courses to be organised, delivered, tracked and assessed through embedded content, interactive applications and online communication and collaboration between learners and instructors. Watson and Watson (2007: 28) describe this kind of online platform as a framework or an infrastructure that 鈥渄elivers and manages instructional content, identifies and assesses individual [鈥 goals, and collects and presents data for supervising the learning process鈥.

The most frequently employed Learning Management Systems in the European Higher Education context are Moodle and Blackboard Learn, with a regional market share at third-level of 57% in 2017 for Moodle, and 18% for Blackboard Learn (Hill, 2017), followed by an increasing number of small educational technology start-ups often eventually acquired by one of the larger companies. Watson and Watson (2007: 30) describe the 鈥減owerful potential鈥 of Learning Management Systems, due to four key characteristics of 鈥渞eusability across multiple contexts鈥, 鈥済enerativity鈥 (generating new instruction), 鈥渁daptability to meet the needs of individual learners鈥, and 鈥渟caleability to meet the needs of both larger and smaller audiences without significant changes in cost鈥. These four characteristics mean that LMS represent an important technology in language learning and in Higher Education. Both Moodle and Blackboard Learn include the following key features:

  • Attendance management
  • Lesson planning
  • Collaboration management
  • Discussion forums
  • File exchange
  • Internal messaging, live chat, blogs, wikis and other learning objects
  • Assessment functions and grade management

However, as Coates et al. (2005: 26) point out in their discussion of the influence of VLEs on teaching and learning, 鈥渋t is not the provision of features but their uptake and use that really determines their educational value鈥.

The language assessment activity described below was implemented at Trinity College Dublin through Blackboard Learn, one of the most widely used VLEs in higher education. All Trinity College staff, students and modules are automatically registered in Blackboard Learn, although its use by academic staff across the university 鈥 and, most likely, many other universities 鈥 varies enormously in terms of purpose (e.g. whether simply using the VLE as a means of storing lecture notes), function (e.g. teaching, assessment, discussion) and regularity. In the university鈥檚 institution-wide language programmes, coordinated within the School of Linguistic, Speech and Communication Sciences, a shared template is employed which provides standard information across all modules and suggests possible means of organizing module content, for instance thematically or chronologically. Whilst staff training, including on how to assess in Blackboard Learn and how customise modules, is regularly available, competing pressures on staff time appear to translate into fairly modest use of the VLE. The university is working hard to ensure that the positive affordances of Blackboard Learn are maximised, through, for instance, support for the creation of interactive lesson design. Thus, the goals of the E-LENGUA project provided us with a timely opportunity to explore online language assessment in the university鈥檚 VLE.

3. The Challenge: Online language assessment

Whilst, as described above, there have been significant leaps in educational technology in the last decade, much of our educational and assessment practice has remained in older pedagogical practices, including placing the onus on teachers to ensure that student learning takes place, tending to encourage passivity in learners, and mistrusting learners鈥 ability to self-assess or peer-assess. However, progress in models of e-learning have allowed curriculum designers and instructors to create customised and learner-centred environments which foster engagement, allowing learners to take control of their pace, sequence and direction of their learning.

Language assessment comprises both formative and summative phases of decision-making regarding skills, progress and achievements in language learning proficiency. Innovation in language assessment shifts what is often the final stage of the curriculum design cycle (summative assessment) to become an integrated part of teaching and learning activities in the classroom (formative assessment), not just assessing language learning, but also assessing for learning. In other words, assessment should be viewed from a learner-centred perspective as much as any other part of the curriculum. However, it can be very difficult for many language learners who have been accustomed to a teacher-led classroom or test-focussed regime to adapt to such a shift. Providing scaffolding (Wood, Bruner & Ross, 1976) allows learners to take control of their own learning through learner involvement in the process of learning, learner reflection and target language use (Little, 2007). The online assessment task and tools outlined below attempt to respond to this teaching and learning context, and to provide scaffolding throughout the assessment phase of language curriculum design.

For the researcher and test designer, language assessment has the aim of providing reliable and valid information regarding specific language constructs such as speaking, writing and so forth. However, typical language test-takers generally take scores for granted, and mostly place trust in the validity and reliability of the testing exercise. This is particularly the case for large-scale English language proficiency tests which tend to be deployed across the globe for a variety of gatekeeping purposes (such as IELTS, TOEIC and TOEFL), as well as national English language proficiency examination schemes.

However, the impact from these tests often ripples back to language classrooms, where much time is spent in test preparation, and forward to the school, university or workplace, where decisions are based around English language test scores, often without discussion or understanding of what can be extrapolated from the results. Test designers are aware of such effects, and much research has been conducted on the impact and 鈥榳ashback鈥 of language assessment and tests (Wall, 2005; Spolsky, 1997; Alderson and Wall 1993). Washback can be defined as how assessment influences the activities which go on in a language classroom before testing, or what Alderson and Wall describe as something that 鈥渃ompels teachers and learners to do things they would not necessarily otherwise do because of the test鈥 (Alderson and Wall, 1993: 115). Washback can be positive and negative in terms of the types of changes in a curriculum that a test can induce. For example, incorporating spoken interaction into a test is likely to lead to washback in the classroom, and encourage teachers and learners to place more emphasis on the skills involved in dialogue, listening comprehension and creating appropriate spontaneous utterances. By creating assessment that is authentic, closely related to the target language use domain, and delivered in ways that are clear, fair and transparent, language assessment can create positive washback in the classroom. Our assessment activity described below aims to create such positive washback.

Whilst the aims and tools of language assessment have changed significantly over time (see Spolsky, 2008, for an historical overview), often the shift to a digital environment has simply been a migration of approaches traditionally used in so-called 鈥榩aper and pencil鈥 tests to online formats, without any substantial changes (such as a reliance on multiple choice questions). Online language assessment allows new tools and methods which can strengthen assessment and also inform pedagogical activities in online and blended classes. These include ways of capturing learners鈥 language use in different contexts and their progress over time. Online language assessment can be motivating for learners, as many of the examples of 鈥榞amification鈥 in language learning applications demonstrate. Online language assessment allows us to re-conceptualize assessment, to shift away from the teacher-led model, and to focus on lifelong language learning as a core aim. Assessing productive language skills online is still a significant challenge, despite advances in technological tools. For instance, automatic computer assessment of the outputs from receptive skills of listening and reading is something most learners are familiar with. The algorithms that run in proficiency tests such as TOEFL and web applications such as Duolingo, Busuu and Memrise replace human assessors. Online language assessment methods have provided us with the means of creating large-scale tests of receptive skills through closed format and multiple choice questions. Coates et al (2005: 27) warn:

One of the most obvious limitations of LMS is their reliance on forms of assessment which can be automatically 鈥渃orrected鈥, such as multiple choice and short response tests [鈥 The danger is that, if this is the most prominent aspect of the assessment function in LMS, it will drive pedagogy towards a simplistic form of the vitally important assessment and feedback loop.

We are still far from having access to the type of software that allows online marking of subjective and extensive free-writing or voice recognition, but online formative assessment of productive skills still offers many positive affordances, especially through the tools offered within VLEs such as integrating self-, peer- and teacher-assessment. With right guide of guidance and scaffolding, formative assessment of language skills based on a sound pedagogical foundation is feasible for learners and teachers.

4. The Tool: Assessing Spoken Production in Blackboard Learn through 鈥楶echa Kucha鈥 presentations

One of the online assessment activities designed at Trinity College as part of the E-LENGUA project includes a Pecha Kucha presentation to be uploaded by learners to Blackboard Learn. This activity assesses spoken production skills at B1 and B2 common European proficiency levels. Pecha Kucha is a slide presentation style invented in Japan in 2003, created by architects Mark Klein and Astrid Dytham (Klentzin et al., 2010), initially used for managing multiple presentations by young designers. Pecha Kucha is a Japanese word (銉氥儊銉c偗銉併儯) which means chit chat or chatter, and the approach has increasingly been used in Europe for crowdsourcing and pitches in Pecha Kucha themed nights.

Very much in the same family as presentation styles such as lightning talks or the 鈥榯hesis in three鈥 format, Pecha Kucha is a rigid and fast-moving oral presentation format for individual learners which employs 20, mostly visual, slides which are shown for strictly 20 seconds each (6 minutes and 40 seconds in total).聽 This format keeps presentations concise, fun and fast-moving.聽It has been described as a kind of visual haiku, a way to deliver a more striking or memorable talk rather than using more conventional presentation methods.聽Students are requested to present on any topic of their choice, using this visual slide format as their prompts. Pecha Kucha presentations can be delivered synchronously, live or via video calling applications, or asynchronously as a pre-recorded and timed slide/audio presentation which is then uploaded to the VLE for assessment.

Image 1. Blackboard Learn assessment overview
Image 1. Blackboard Learn assessment overview

Pecha Kucha has been the subject of academic discussion and investigation as a presentation format conducive to fostering student learning and engagement (Beyer, 2011; Beyer et al., 2012; Dunlap, 2011). Klentzin et al. (2010) provide a review of scholarly literature on the use of Pecha Kucha presentations in university instruction. They write:

Potential advantages for the use of Pecha Kucha as a pedagogical technique in the college classroom are evident. Similar to traditional PowerPoint presentations, Pecha Kucha provides students with both auditory and visual modes of instruction in thereby reaching students with two different learning styles simultaneously. Pecha Kucha also forces an instructor to focus on the most relevant information during the strictly timed presentation. (2010: 160).

In an empirical study among 67 US college students, Beyer et al. (2012) compared student recall between Pecha Kucha and PowerPoint presentations. They found more favourable student preferences for Pecha Kucha versus traditional PowerPoint peer presentations, and found similar levels of recall and retention of material presented across both formats.

In the assessment unit in Blackboard Learn, students are provided with an overview of the assessment exercise, and are then invited to watch an embedded video clip which presents an example of Pecha Kucha, selected from the Pecha Kucha website鈥檚 Presentation of the Day ( Next, a structured Learning Widget within Blackboard (see Image 2 below) then guides learners through the assessment activity, with (1) instructions, (2) tips on the three key areas which involve planning and preparation (selecting the images, writing the script, and timing) and finally (3) information on the assessment of the presentation (self-assessment, peer-assessment and teacher assessment). This widget provides a table of contents for the learners, and can be easily adapted by teacher-assessors:

Image 2: Pecha Kucha Widget in Blackboard Learn
Image 2: Pecha Kucha Widget in Blackboard Learn

Klentzin et al. (2010) address the impact the constraints inherent in this format, including the rigidity of the timing, the need to make swift transitions between many slides whilst maintaining narrative integrity, and the restrictions of a short time frame which may render some topics too complex to address (161). Dunlap (2011: 254), in her consideration of using Pecha Kucha in an online education context, notes that it is a case of carefully planned creativity:

Just because the Pecha Kucha format is intentionally informal, this does not mean that presenters do not have to prepare fully for their six minute and forty seconds. To avoid uttering too many 鈥渦mmms鈥 and 鈥渁nds鈥 and running out of time, it is necessary to prepare and to practice. This becomes even more evident when recording a Pecha Kucha for online delivery. (254)

In the preparation of this task, learners are presented with tips in the Blackboard Learning Widget on selecting images (including issues of copyright, visibility etc.), how to craft a script or story that maintains their listeners鈥 attention, and how to manage the technical details of a timed and pre-recorded slide presentation. These are valuable transferable skills which are gained during the assessment preparation.

The assessment scheme for the Pecha Kucha project is based on a combination of scored self- and teacher-assessment. Self-assessment accounts for 10% of students鈥 overall presentation score, with teacher assessment accounting for 90%. Peer-assessment is conducted through making access to the teacher rubric open to all the class, but this is not formally scored at present. Rather than simply listening as audience members, all learners are encouraged to act as peer-assessors for their classmates鈥 presentations. These steps are implemented through the Blackboard Learn Assignment toolkit. The self- and teacher-assessment materials for the Pecha Kucha presentation involve a checklist as well as an analytic rating scale. The three-point checklist ensures that presentations are appropriately timed, that the slides are visual in nature, and that the narration continues for the duration of the slide show (no 鈥榚mpty鈥 time). Learners who do not meet these criteria are deducted 10 points per item from their final overall score. In conjunction, an analytic rating scale is used as a basis for scoring spoken production. This rating scale is embedded in Blackboard Learn. Learners achieve a maximum of 20 points, converted to a final percentage score.

Image 3: Teacher assessment rubric
Image 3: Teacher assessment rubric

Online language learning and assessment in virtual learning environments allows an opportunity for sustained self-assessment which can be scaffolded by the curriculum designers. It has been argued that self-assessment embedded in learning, or immediately following, and related to, a particular task (on-task rather than off-task) leads learners to self-assess more accurately (Rea-Dickens, 2006; Goto Butler & Lee, 2006). Detailed and informative descriptions of language competency in assessment rubrics, rather than more abstract concepts of overall progress and effort, seem to help identify progress and to diagnose areas of weakness (Carson, 2010). In the same way that we learn a language by using the language, we learn most about assessment when we assess, or in the case of the learner, when they are co-assessors or peer-assessors as well as self-assessors. Self-assessment of each Pecha Kucha presentation is based on the same rubric and checklist as used by teachers. Students are provided with the following information:

Your Pecha Kucha presentation is assessed by your teacher using five-point scale, (one 鈥 very poor to five 鈥 very good). You can access a copy of the rating scale used for this project in the Assessment folder in Blackboard Learn. Now, we ask you to assess your own work using the same five-point scale. Please complete the following assessment, and be ready to discuss your self-assessment with your teacher.

Self-assessment of spoken production is presented in Blackboard Learn assessment rubric as follows (Image 4):

Image 4: Self-assessment rubric
Image 4: Self-assessment rubric

The assessment checklist is presented as follows:

  • Was your Pecha Kucha presentation on time? (20 seconds per slide, 6 minutes and 40 seconds in total)
  • Did you use visual slides? (Slides without writing)
  • Did your narration continues for the duration of the slide show? (No 鈥榚mpty鈥 time)
  • Please note that 10 points will be deducted per item.

In summary, the Pecha Kucha assessment activity is presented within a unit of a Blackboard Learn language module. Using the Learning Widget function in the VLE, an overview of the assessment activity is presented with clear instructions for successful completion. Students pre-record their slide presentation and narrative, and upload these to Blackboard Learn for assessment. Peer-assessment functions in Blackboard mean that all class members are able to view these files as well as the teacher-assessor. Online assessment rubrics are provided as well as checklists.

5. Conclusion

The Pecha Kucha format is an innovative approach to assessing spoken production, stimulating 鈥渃reativity, enthusiasm, high energy and engagement鈥 (Dunlap, 2011: 257), favoured by students over other traditional forms of slide presentation (Beyer 2011; Beyer et al., 2012), pointing towards its potential for positive washback in language education in higher education. Its suitability as an online formative assessment activity responds to the need to provide flexible course delivery, to cater to large numbers of language learners, and to incorporate innovative assessment methods using technology rather than relying on summative, paper and pencil testing whilst also providing diagnostic feedback to students. The integration of Pecha Kucha slides and script within a strictly timed format, integrating content and language, is a powerful means of engaging student interest and channelling their own interests and specialisms and of countering the disengagement that occurs during peer presentations (鈥榙eath by PowerPoint鈥). In this assessment exercise, learners also acquire valuable transferable skills in the preparation and delivery of the task 鈥 addressing issues of image copyright, learning how to create timed and recorded presentations, maintaining the floor, narrating transitions between slides, conveying a concise message to the audience, and so forth. The tools within Blackboard Learn also allow the VLE鈥檚 interactive functions to be used in conjunction with the requirements of the formative assessment, supplementing the files that are uploaded through a Pecha Kucha discussion board or live chat. Positive washback from assessment is derived from the use of specific and concrete items in the rating scales and the prompt online accessibility of scores from each part of the assessment rubric. The challenge of delivering online assessment of productive language skills in a formative context is a significant one, but this proposed tool as presented above offers a worthwhile avenue for university language teachers and assessors to explore.

6. Works cited

Alderson, J. Charles & Diane Wall (1993).鈥 Does washback exist?鈥. Applied Linguistics 14 (2): pp. 115-129.

Beyer, Alisa M. (2011). 鈥淚mproving student presentations Pecha Kucha and just plain PowerPoint鈥. Teaching of Psychology 38 (2): pp. 122-126.

Beyer, Alisa M., Catherine Gaze & Julia Lazicki (2012). 鈥淐omparing students鈥 evaluations and recall for Student Pecha Kucha and PowerPoint Presentations鈥. Journal of Teaching and Learning with Technology 1 (2): pp. 26-42.

Carson, Lorna (2010). 鈥淚nnovation and Autonomy in an Institution-wide Language Programme鈥. Eds. Breffni O’Rourke & Lorna Carson. Language Learner Autonomy: Policy, Curriculum, Classroom. Oxford: Peter Lang. pp. 151-170.

Carson, Lorna (2016). 鈥淔ostering engagement with the Common European Framework of Reference for Languages and the European Language Portfolio: Learning from good practice in university language centres鈥. Language Learning in Higher Education 6 (2): pp. 275鈥 282.

Coates, Hamish, Richard James & Gabrielle Baldwin (2005). 鈥淎 critical examination of the effects of Learning Management Systems on university teaching and learning鈥. Tertiary Education and Management 11: pp. 19-36.

Council of Europe (2001). The Common European Framework of Reference for Languages: Teaching, Learning and Assessment. Strasbourg: Council of Europe.

Dunlap, Joanna (2011). 鈥淧echa Kucha for Learning鈥. Patti Shank (ed.). The Online Learning Idea Book:聽Proven Ways to Enhance Technology-Based and Blended Learning. New Jersey: Wiley. pp. 253-257.

Klentzin,聽Jacqueline Courtney, Emily Bounds Paladino,聽Bruce Johnston & Christopher Devine (2010). 鈥淧echa Kucha: Using 鈥榣ightning talk鈥 in university instruction鈥.聽Reference Services Review 38 (1): pp. 158-167.

Hill, Phil (2017). 鈥淎cademic LMS Market Share: A View Across Four Global Regions. E-Literate: What We Are Learning About Online Learning鈥. <>. (20-9-2017).

Little, David, G. (2007). 鈥淟anguage learner autonomy: Some fundamental considerations revisited鈥. International Journal of Innovation in Language Learning and Teaching 1 (1): pp. 14-29.

Spolsky, Bernard (1997). 鈥淭he ethics of gatekeeping tests: what have we learned in a hundred years?鈥. Language Testing 14 (3): pp. 242-247.

Spolsky, Bernard (2008). 鈥淟anguage Assessment in Historical and Future Perspective鈥. Nancy Hornberger (ed.). Encyclopedia of Language and Education. London: Springer. pp: 2570-2579.

Wall, Diane (2005). The Impact of High-Stakes Examinations on Classroom Teaching: A Case Study Using Insights from Testing and Innovation Theory. Cambridge: Cambridge University Press.

Watson, William R. & Sunnie Lee Watson (2007). 鈥淎n Argument for Clarity: What are Learning Management Systems, What are They Not, and What Should They Become?鈥. TechTrends 51 (2): pp. 28-34.

Wood, David, Jerome S. Bruner & Gail Ross (1976). 鈥淭he role of tutoring in problem solving鈥. Journal of Child Psychology and Psychiatry 17(2): pp. 89-100.

Caracteres vol.6 n2

路 Descargar el vol.6 n潞2 de Caracteres como PDF.

路 Descargar este texto como PDF.

路 Regresar al 铆ndice de la edici贸n web.

Caracteres. Estudios culturales y cr铆ticos de la esfera digital | ISSN: 2254-4496 | Salamanca