Mejorando la evaluación lingüística en línea: usando Pecha Kucha para evaluar la producción oral en inglés

Lorna Carson (Trinity College Dublin)

Artículo recibido: 11-10-2017 | Artículo aceptado: 06-11-2017

RESUMEN: En este reto E-LENGUA se busca mejorar la evaluación en línea de la lengua inglesa usando entornos de aprendizaje virtual dentro del sistema de programas de lenguas de Trinity College de Dublín. Nos centramos en usos específicos de la lengua, orientados a la acción, y procedimientos integrados de evaluación por el profesor, entre iguales y autoevaluación. Este artículo propone una actividad de presentación diseñada para evaluar la producción oral de inglés en niveles B1 y B2 que puede emplearse en entornos tradicionales, en línea o mixtos. El equipo del Trinity College Dublin explora los motivos y el uso de un formato de presentaciones innovador, Pecha Kucha, que se complementa con Blackboard Learn. El artículo explora el contexto de entornos de aprendizaje virtual como Blackboard Learn o Moodle y estudia el reto de la evaluación de lenguas en línea. La tarea propuesta como evaluación busca potenciar la vinculación del aprendiz y su autorreflexión a través de actividades compartidas de evaluación y asegurar la respuesta positiva frente a la negativa en la clase de lenguas.
ABSTRACT: In this E-LENGUA challenge to improve online English language assessment using virtual learning environments, undertaken within the institution-wide language programme at Trinity College Dublin, we focus on specific, action-oriented descriptions of language use, joined-up learning and assessment materials, and embedded self-, peer- and teacher-assessment procedures. This article presents a presentation activity designed to assess spoken production in English at B1 and B2 proficiency levels which can be used in a traditional, blended or online environment. The team at Trinity College Dublin explore the rationale and use of an innovative slide presentation format, Pecha Kucha, to be completed within Blackboard Learn. The article explores the context of virtual learning environments such as Blackboard Learn or Moodle and discusses the challenge of online language assessment in such an environment. The proposed assessment task aims to foster learner engagement and reflection through shared assessment activities, and to ensure positive rather than negative washback in the language classroom.

PALABRAS CLAVE: Evaluación, evaluación formativa, evaluación sumativa, entornos de aprendizaje virtual
KEY WORDS: Assessment, formative assessment, summative assessment, virtual learning environments (VLE)

---

The E-LENGUA project is financed by the KA2013 Strategic Partnerships Actions for Higher Education.

---

1. Introduction

This paper presents the E-LENGUA challenge of improving online English language assessment within virtual learning environments in higher education. We report on an innovative slide presentation task, designed and trialled within the institution-wide language programme at Trinity College Dublin. The language programme deploys specific, action-oriented descriptions of language use, joined-up learning and assessment materials, and embedded self-, peer- and teacher-assessment procedures (Carson, 2010; Carson, 2016) based on the taxonomic “Can do” descriptor approach of the CEFR (Council of Europe, 2001). This contribution focuses on the formative assessment of Spoken Production in English at B1 and B2 proficiency methods using an innovative presentation format called Pecha Kucha, completed within Blackboard Learn. Below, we explore the context and affordances of virtual learning environments, and consider the challenges of online assessment of language proficiency. The remainder of the paper presents the design and presentation of the Pecha Kucha task, including instructions and assessment scheme.

2. The Context: Virtual Learning Environments

Multiple and sometimes interchangeable definitions are deployed in the discussion of e-learning and online educational systems or platforms, including Learning Management Systems (LMS), Virtual Learning Environments (VLE), Course Management Systems or Content Management Systems. Whilst distinctions can be drawn between the names given to some of the commercial applications of these systems, for the purposes of this paper, the term Virtual Learning Environment will be used to refer to types of web-based technology that allow educational courses to be organised, delivered, tracked and assessed through embedded content, interactive applications and online communication and collaboration between learners and instructors. Watson and Watson (2007: 28) describe this kind of online platform as a framework or an infrastructure that “delivers and manages instructional content, identifies and assesses individual […] goals, and collects and presents data for supervising the learning process”.

The most frequently employed Learning Management Systems in the European Higher Education context are Moodle and Blackboard Learn, with a regional market share at third-level of 57% in 2017 for Moodle, and 18% for Blackboard Learn (Hill, 2017), followed by an increasing number of small educational technology start-ups often eventually acquired by one of the larger companies. Watson and Watson (2007: 30) describe the “powerful potential” of Learning Management Systems, due to four key characteristics of “reusability across multiple contexts”, “generativity” (generating new instruction), “adaptability to meet the needs of individual learners”, and “scaleability to meet the needs of both larger and smaller audiences without significant changes in cost”. These four characteristics mean that LMS represent an important technology in language learning and in Higher Education. Both Moodle and Blackboard Learn include the following key features:

• Attendance management
• Lesson planning
• Collaboration management
• Discussion forums
• File exchange
• Internal messaging, live chat, blogs, wikis and other learning objects
• Assessment functions and grade management

However, as Coates et al. (2005: 26) point out in their discussion of the influence of VLEs on teaching and learning, “it is not the provision of features but their uptake and use that really determines their educational value”.

The language assessment activity described below was implemented at Trinity College Dublin through Blackboard Learn, one of the most widely used VLEs in higher education. All Trinity College staff, students and modules are automatically registered in Blackboard Learn, although its use by academic staff across the university – and, most likely, many other universities – varies enormously in terms of purpose (e.g. whether simply using the VLE as a means of storing lecture notes), function (e.g. teaching, assessment, discussion) and regularity. In the university’s institution-wide language programmes, coordinated within the School of Linguistic, Speech and Communication Sciences, a shared template is employed which provides standard information across all modules and suggests possible means of organizing module content, for instance thematically or chronologically. Whilst staff training, including on how to assess in Blackboard Learn and how customise modules, is regularly available, competing pressures on staff time appear to translate into fairly modest use of the VLE. The university is working hard to ensure that the positive affordances of Blackboard Learn are maximised, through, for instance, support for the creation of interactive lesson design. Thus, the goals of the E-LENGUA project provided us with a timely opportunity to explore online language assessment in the university’s VLE.

3. The Challenge: Online language assessment

Whilst, as described above, there have been significant leaps in educational technology in the last decade, much of our educational and assessment practice has remained in older pedagogical practices, including placing the onus on teachers to ensure that student learning takes place, tending to encourage passivity in learners, and mistrusting learners’ ability to self-assess or peer-assess. However, progress in models of e-learning have allowed curriculum designers and instructors to create customised and learner-centred environments which foster engagement, allowing learners to take control of their pace, sequence and direction of their learning.

Language assessment comprises both formative and summative phases of decision-making regarding skills, progress and achievements in language learning proficiency. Innovation in language assessment shifts what is often the final stage of the curriculum design cycle (summative assessment) to become an integrated part of teaching and learning activities in the classroom (formative assessment), not just assessing language learning, but also assessing for learning. In other words, assessment should be viewed from a learner-centred perspective as much as any other part of the curriculum. However, it can be very difficult for many language learners who have been accustomed to a teacher-led classroom or test-focussed regime to adapt to such a shift. Providing scaffolding (Wood, Bruner & Ross, 1976) allows learners to take control of their own learning through learner involvement in the process of learning, learner reflection and target language use (Little, 2007). The online assessment task and tools outlined below attempt to respond to this teaching and learning context, and to provide scaffolding throughout the assessment phase of language curriculum design.

For the researcher and test designer, language assessment has the aim of providing reliable and valid information regarding specific language constructs such as speaking, writing and so forth. However, typical language test-takers generally take scores for granted, and mostly place trust in the validity and reliability of the testing exercise. This is particularly the case for large-scale English language proficiency tests which tend to be deployed across the globe for a variety of gatekeeping purposes (such as IELTS, TOEIC and TOEFL), as well as national English language proficiency examination schemes.

However, the impact from these tests often ripples back to language classrooms, where much time is spent in test preparation, and forward to the school, university or workplace, where decisions are based around English language test scores, often without discussion or understanding of what can be extrapolated from the results. Test designers are aware of such effects, and much research has been conducted on the impact and ‘washback’ of language assessment and tests (Wall, 2005; Spolsky, 1997; Alderson and Wall 1993). Washback can be defined as how assessment influences the activities which go on in a language classroom before testing, or what Alderson and Wall describe as something that “compels teachers and learners to do things they would not necessarily otherwise do because of the test” (Alderson and Wall, 1993: 115). Washback can be positive and negative in terms of the types of changes in a curriculum that a test can induce. For example, incorporating spoken interaction into a test is likely to lead to washback in the classroom, and encourage teachers and learners to place more emphasis on the skills involved in dialogue, listening comprehension and creating appropriate spontaneous utterances. By creating assessment that is authentic, closely related to the target language use domain, and delivered in ways that are clear, fair and transparent, language assessment can create positive washback in the classroom. Our assessment activity described below aims to create such positive washback.

Whilst the aims and tools of language assessment have changed significantly over time (see Spolsky, 2008, for an historical overview), often the shift to a digital environment has simply been a migration of approaches traditionally used in so-called ‘paper and pencil’ tests to online formats, without any substantial changes (such as a reliance on multiple choice questions). Online language assessment allows new tools and methods which can strengthen assessment and also inform pedagogical activities in online and blended classes. These include ways of capturing learners’ language use in different contexts and their progress over time. Online language assessment can be motivating for learners, as many of the examples of ‘gamification’ in language learning applications demonstrate. Online language assessment allows us to re-conceptualize assessment, to shift away from the teacher-led model, and to focus on lifelong language learning as a core aim. Assessing productive language skills online is still a significant challenge, despite advances in technological tools. For instance, automatic computer assessment of the outputs from receptive skills of listening and reading is something most learners are familiar with. The algorithms that run in proficiency tests such as TOEFL and web applications such as Duolingo, Busuu and Memrise replace human assessors. Online language assessment methods have provided us with the means of creating large-scale tests of receptive skills through closed format and multiple choice questions. Coates et al (2005: 27) warn:

One of the most obvious limitations of LMS is their reliance on forms of assessment which can be automatically “corrected”, such as multiple choice and short response tests […] The danger is that, if this is the most prominent aspect of the assessment function in LMS, it will drive pedagogy towards a simplistic form of the vitally important assessment and feedback loop.

We are still far from having access to the type of software that allows online marking of subjective and extensive free-writing or voice recognition, but online formative assessment of productive skills still offers many positive affordances, especially through the tools offered within VLEs such as integrating self-, peer- and teacher-assessment. With right guide of guidance and scaffolding, formative assessment of language skills based on a sound pedagogical foundation is feasible for learners and teachers.

4. The Tool: Assessing Spoken Production in Blackboard Learn through ‘Pecha Kucha’ presentations

One of the online assessment activities designed at Trinity College as part of the E-LENGUA project includes a Pecha Kucha presentation to be uploaded by learners to Blackboard Learn. This activity assesses spoken production skills at B1 and B2 common European proficiency levels. Pecha Kucha is a slide presentation style invented in Japan in 2003, created by architects Mark Klein and Astrid Dytham (Klentzin et al., 2010), initially used for managing multiple presentations by young designers. Pecha Kucha is a Japanese word (ペチャクチャ) which means chit chat or chatter, and the approach has increasingly been used in Europe for crowdsourcing and pitches in Pecha Kucha themed nights.

Very much in the same family as presentation styles such as lightning talks or the ‘thesis in three’ format, Pecha Kucha is a rigid and fast-moving oral presentation format for individual learners which employs 20, mostly visual, slides which are shown for strictly 20 seconds each (6 minutes and 40 seconds in total).  This format keeps presentations concise, fun and fast-moving. It has been described as a kind of visual haiku, a way to deliver a more striking or memorable talk rather than using more conventional presentation methods. Students are requested to present on any topic of their choice, using this visual slide format as their prompts. Pecha Kucha presentations can be delivered synchronously, live or via video calling applications, or asynchronously as a pre-recorded and timed slide/audio presentation which is then uploaded to the VLE for assessment.

Pecha Kucha has been the subject of academic discussion and investigation as a presentation format conducive to fostering student learning and engagement (Beyer, 2011; Beyer et al., 2012; Dunlap, 2011). Klentzin et al. (2010) provide a review of scholarly literature on the use of Pecha Kucha presentations in university instruction. They write:

Potential advantages for the use of Pecha Kucha as a pedagogical technique in the college classroom are evident. Similar to traditional PowerPoint presentations, Pecha Kucha provides students with both auditory and visual modes of instruction in thereby reaching students with two different learning styles simultaneously. Pecha Kucha also forces an instructor to focus on the most relevant information during the strictly timed presentation. (2010: 160).

In an empirical study among 67 US college students, Beyer et al. (2012) compared student recall between Pecha Kucha and PowerPoint presentations. They found more favourable student preferences for Pecha Kucha versus traditional PowerPoint peer presentations, and found similar levels of recall and retention of material presented across both formats.

In the assessment unit in Blackboard Learn, students are provided with an overview of the assessment exercise, and are then invited to watch an embedded video clip which presents an example of Pecha Kucha, selected from the Pecha Kucha website’s Presentation of the Day (www.pechakucha.org). Next, a structured Learning Widget within Blackboard (see Image 2 below) then guides learners through the assessment activity, with (1) instructions, (2) tips on the three key areas which involve planning and preparation (selecting the images, writing the script, and timing) and finally (3) information on the assessment of the presentation (self-assessment, peer-assessment and teacher assessment). This widget provides a table of contents for the learners, and can be easily adapted by teacher-assessors:

Klentzin et al. (2010) address the impact the constraints inherent in this format, including the rigidity of the timing, the need to make swift transitions between many slides whilst maintaining narrative integrity, and the restrictions of a short time frame which may render some topics too complex to address (161). Dunlap (2011: 254), in her consideration of using Pecha Kucha in an online education context, notes that it is a case of carefully planned creativity:

Just because the Pecha Kucha format is intentionally informal, this does not mean that presenters do not have to prepare fully for their six minute and forty seconds. To avoid uttering too many “ummms” and “ands” and running out of time, it is necessary to prepare and to practice. This becomes even more evident when recording a Pecha Kucha for online delivery. (254)

In the preparation of this task, learners are presented with tips in the Blackboard Learning Widget on selecting images (including issues of copyright, visibility etc.), how to craft a script or story that maintains their listeners’ attention, and how to manage the technical details of a timed and pre-recorded slide presentation. These are valuable transferable skills which are gained during the assessment preparation.

The assessment scheme for the Pecha Kucha project is based on a combination of scored self- and teacher-assessment. Self-assessment accounts for 10% of students’ overall presentation score, with teacher assessment accounting for 90%. Peer-assessment is conducted through making access to the teacher rubric open to all the class, but this is not formally scored at present. Rather than simply listening as audience members, all learners are encouraged to act as peer-assessors for their classmates’ presentations. These steps are implemented through the Blackboard Learn Assignment toolkit. The self- and teacher-assessment materials for the Pecha Kucha presentation involve a checklist as well as an analytic rating scale. The three-point checklist ensures that presentations are appropriately timed, that the slides are visual in nature, and that the narration continues for the duration of the slide show (no ‘empty’ time). Learners who do not meet these criteria are deducted 10 points per item from their final overall score. In conjunction, an analytic rating scale is used as a basis for scoring spoken production. This rating scale is embedded in Blackboard Learn. Learners achieve a maximum of 20 points, converted to a final percentage score.

Online language learning and assessment in virtual learning environments allows an opportunity for sustained self-assessment which can be scaffolded by the curriculum designers. It has been argued that self-assessment embedded in learning, or immediately following, and related to, a particular task (on-task rather than off-task) leads learners to self-assess more accurately (Rea-Dickens, 2006; Goto Butler & Lee, 2006). Detailed and informative descriptions of language competency in assessment rubrics, rather than more abstract concepts of overall progress and effort, seem to help identify progress and to diagnose areas of weakness (Carson, 2010). In the same way that we learn a language by using the language, we learn most about assessment when we assess, or in the case of the learner, when they are co-assessors or peer-assessors as well as self-assessors. Self-assessment of each Pecha Kucha presentation is based on the same rubric and checklist as used by teachers. Students are provided with the following information:

Your Pecha Kucha presentation is assessed by your teacher using five-point scale, (one – very poor to five – very good). You can access a copy of the rating scale used for this project in the Assessment folder in Blackboard Learn. Now, we ask you to assess your own work using the same five-point scale. Please complete the following assessment, and be ready to discuss your self-assessment with your teacher.

Self-assessment of spoken production is presented in Blackboard Learn assessment rubric as follows (Image 4):

The assessment checklist is presented as follows:

• Was your Pecha Kucha presentation on time? (20 seconds per slide, 6 minutes and 40 seconds in total)
• Did you use visual slides? (Slides without writing)
• Did your narration continues for the duration of the slide show? (No ‘empty’ time)
• Please note that 10 points will be deducted per item.

In summary, the Pecha Kucha assessment activity is presented within a unit of a Blackboard Learn language module. Using the Learning Widget function in the VLE, an overview of the assessment activity is presented with clear instructions for successful completion. Students pre-record their slide presentation and narrative, and upload these to Blackboard Learn for assessment. Peer-assessment functions in Blackboard mean that all class members are able to view these files as well as the teacher-assessor. Online assessment rubrics are provided as well as checklists.

5. Conclusion

The Pecha Kucha format is an innovative approach to assessing spoken production, stimulating “creativity, enthusiasm, high energy and engagement” (Dunlap, 2011: 257), favoured by students over other traditional forms of slide presentation (Beyer 2011; Beyer et al., 2012), pointing towards its potential for positive washback in language education in higher education. Its suitability as an online formative assessment activity responds to the need to provide flexible course delivery, to cater to large numbers of language learners, and to incorporate innovative assessment methods using technology rather than relying on summative, paper and pencil testing whilst also providing diagnostic feedback to students. The integration of Pecha Kucha slides and script within a strictly timed format, integrating content and language, is a powerful means of engaging student interest and channelling their own interests and specialisms and of countering the disengagement that occurs during peer presentations (‘death by PowerPoint’). In this assessment exercise, learners also acquire valuable transferable skills in the preparation and delivery of the task – addressing issues of image copyright, learning how to create timed and recorded presentations, maintaining the floor, narrating transitions between slides, conveying a concise message to the audience, and so forth. The tools within Blackboard Learn also allow the VLE’s interactive functions to be used in conjunction with the requirements of the formative assessment, supplementing the files that are uploaded through a Pecha Kucha discussion board or live chat. Positive washback from assessment is derived from the use of specific and concrete items in the rating scales and the prompt online accessibility of scores from each part of the assessment rubric. The challenge of delivering online assessment of productive language skills in a formative context is a significant one, but this proposed tool as presented above offers a worthwhile avenue for university language teachers and assessors to explore.

6. Works cited

Alderson, J. Charles & Diane Wall (1993).” Does washback exist?”. Applied Linguistics 14 (2): pp. 115-129.

Beyer, Alisa M. (2011). “Improving student presentations Pecha Kucha and just plain PowerPoint”. Teaching of Psychology 38 (2): pp. 122-126.

Beyer, Alisa M., Catherine Gaze & Julia Lazicki (2012). “Comparing students’ evaluations and recall for Student Pecha Kucha and PowerPoint Presentations”. Journal of Teaching and Learning with Technology 1 (2): pp. 26-42.

Carson, Lorna (2010). “Innovation and Autonomy in an Institution-wide Language Programme”. Eds. Breffni O’Rourke & Lorna Carson. Language Learner Autonomy: Policy, Curriculum, Classroom. Oxford: Peter Lang. pp. 151-170.

Carson, Lorna (2016). “Fostering engagement with the Common European Framework of Reference for Languages and the European Language Portfolio: Learning from good practice in university language centres”. Language Learning in Higher Education 6 (2): pp. 275– 282.

Coates, Hamish, Richard James & Gabrielle Baldwin (2005). “A critical examination of the effects of Learning Management Systems on university teaching and learning”. Tertiary Education and Management 11: pp. 19-36.

Council of Europe (2001). The Common European Framework of Reference for Languages: Teaching, Learning and Assessment. Strasbourg: Council of Europe.

Dunlap, Joanna (2011). “Pecha Kucha for Learning”. Patti Shank (ed.). The Online Learning Idea Book: Proven Ways to Enhance Technology-Based and Blended Learning. New Jersey: Wiley. pp. 253-257.

Klentzin, Jacqueline Courtney, Emily Bounds Paladino, Bruce Johnston & Christopher Devine (2010). “Pecha Kucha: Using ‘lightning talk’ in university instruction”. Reference Services Review 38 (1): pp. 158-167.

Hill, Phil (2017). “Academic LMS Market Share: A View Across Four Global Regions. E-Literate: What We Are Learning About Online Learning”. MFeldstein.com. <http://mfeldstein.com/academic-lms-market-share-view-across-four-global-regions>. (20-9-2017).

Little, David, G. (2007). “Language learner autonomy: Some fundamental considerations revisited”. International Journal of Innovation in Language Learning and Teaching 1 (1): pp. 14-29.

Spolsky, Bernard (1997). “The ethics of gatekeeping tests: what have we learned in a hundred years?”. Language Testing 14 (3): pp. 242-247.

Spolsky, Bernard (2008). “Language Assessment in Historical and Future Perspective”. Nancy Hornberger (ed.). Encyclopedia of Language and Education. London: Springer. pp: 2570-2579.

Wall, Diane (2005). The Impact of High-Stakes Examinations on Classroom Teaching: A Case Study Using Insights from Testing and Innovation Theory. Cambridge: Cambridge University Press.

Watson, William R. & Sunnie Lee Watson (2007). “An Argument for Clarity: What are Learning Management Systems, What are They Not, and What Should They Become?”. TechTrends 51 (2): pp. 28-34.

Wood, David, Jerome S. Bruner & Gail Ross (1976). “The role of tutoring in problem solving”. Journal of Child Psychology and Psychiatry 17(2): pp. 89-100.

· Descargar el vol.6 nº2 de Caracteres como PDF. · Descargar este texto como PDF. · Regresar al índice de la edición web.