Orchestrating Evaluation of Complex Educational Technologies: A Case Study of a CSCL System
Keywords:
Downloads
Abstract
As digital technologies permeate every aspect of our lives, the complexity of the educational settings, and of the technological support we use within them, unceasingly rises. This increased complexity, along with the need for educational practitioners to apply such technologies within multi-constraint authentic settings, has given rise to the notion of technology-enhanced learning practice as “orchestration of learning”. However, at the same time, the complexity involved in evaluating the benefits of such educational technologies has also increased, prompting questions about the way evaluators can cope with the different places, technologies, informants and issues involved in their evaluation activity. By proposing the notion of “orchestrating evaluation”, this paper tries to reconcile the often disparate “front office accounts” of research publications and the “shop floor practice” of evaluation of educational technology, through the case study of evaluating a system to help teachers in coordinating computer-supported collaborative learning scenarios (GLUE!-PS). We reuse an internationally-evaluated conceptual framework of “orchestration aspects” (design, management, adaptation, pragmatism, etc.) to structure the case’s narrative, showing how the original evaluation questions and methods were modulated in the face of the multiple (authentic) evaluation setting constraints.
Downloads
References
Bote-Lorenzo, M. L., Gómez-Sánchez, E., Vega-Gorgojo, G., Dimitriadis, Y., Asensio-Pérez, J. I., & Jorrín-Abellán, I. M. (2008). Gridcole: a tailorable grid service based system that supports scripted collaborative learning. Computers & Education, 51(1), 155–172.
Google Scholar CrossrefBruce, B. C. (2008). Ubiquitous Learning, Ubiquitous Computing, and life experience. In Proceedings of the 6th International Conference on Networked Learning (NLC2008) (pp. 583–590).
Google Scholar CrossrefCreswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches. Sage.
Google Scholar CrossrefDillenbourg, P. (2013). Design for classroom orchestration. Computers & Education, 69, 485–492. doi:10.1016/j.compedu.2013.04.013
Google Scholar CrossrefDillenbourg, P., Järvelä, S., & Fischer, F. (2009). The Evolution of Research in Computer-Supported Collaborative Learning: from design to orchestration. In N. Balacheff, S. Ludvigsen, T. de Jong, A. Lazonder, & S. Barnes (Eds.), Technology-Enhanced Learning: Principles and Products (pp. 3–19). Springer.
Google Scholar CrossrefDraper, S. W. (1997). Prospects for summative evaluation of CAL in higher education. Research in Learning Technology, 5(1).
Google Scholar CrossrefEconomides, A. A. (2005). Collaborative Learning Evaluation (CLE) framework. WSEAS Transactions on Advances in Engineering Education, 2(4), 339–346.
Google Scholar CrossrefEwing, J., & Miller, D. (2002). A framework for evaluating computer supported collaborative learning. Educational Technology & Society, 5(1), 112–118.
Google Scholar CrossrefFischer, F., & Dillenbourg, P. (2006). Challenges of orchestrating computer-supported collaborative learning. In Paper presented at the 87th Annual Meeting of the American Educational Research Association (AERA).
Google Scholar CrossrefGarfinkel, H. (2002). Ethnomethodology’s program: Working out Durkheim's aphorism (Vol. 6). Rowman & Littlefield.
Google Scholar CrossrefGlass, R. L. (1995). A structure-based critique of contemporary computing research. Journal of Systems and Software, 28(1), 3–7.
Google Scholar CrossrefGómez-Sánchez, E., Bote-Lorenzo, M. L., Jorrín-Abellán, I. M., Vega-Gorgojo, G., Asensio-Pérez, J. I., & Dimitriadis, Y. (2009). Conceptual framework for design, technological support and evaluation of collaborative learning. International Journal of Engineering Education, 25(3), 557–568.
Google Scholar CrossrefGuba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries. A Journal of Theory, Research, and Development on Educational Communication and Technology, 29(2), 75–91.
Google Scholar CrossrefGuba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Sage Publications, Inc.
Google Scholar CrossrefHernández-Leo, D., Asensio-Pérez, J. I., & Dimitriadis, Y. (2005). Computational Representation of Collaborative Learning Flow Patterns Using IMS Learning Desing. Educational Technology & Society, 8(4), 75–89.
Google Scholar CrossrefJorrín-Abellán, I. M., & Stake, R. E. (2009). Does Ubiquitous Learning Call for Ubiquitous Forms of Formal Evaluation?: An Evaluand oriented Responsive Evaluation Model. Ubiquitous Learning: An International Journal, 1(3), 71–82.
Google Scholar CrossrefJorrín-Abellán, I. M., Stake, R. E., & Martínez-Monés, A. (2009). The Needlework in evaluating a CSCL system: The Evaluand-oriented Responsive Evaluation Model. In Proceedings of the 9th International Conference on Computer Supported Collaborative Learning (pp. 68–72).
Google Scholar CrossrefKovalainen, M., Kumpulainen, K., & Satu, V. (2001). Orchestrating classroom interaction in a community of inquiry: Modes of teacher participation. Journal of Classroom Interaction, 36(2), 17–28.
Google Scholar CrossrefLuckin, R. (2008). The Learner Centric Ecology of Resources: a Framework for using Technology to Scaffold Learning. Computers & Education, 50, 449–462.
Google Scholar CrossrefMacNeill, S., & Kraan, W. (2010). Distributed Learning Environments: A Briefing Paper. Retrieved from http://publications.cetis.ac.uk/wp-content/uploads/2011/02/Distributed_Learning.pdf
Google Scholar CrossrefMartı́nez, A., Dimitriadis, Y., Rubia, B., Gómez, E., & de la Fuente, P. (2003). Combining qualitative evaluation and social network analysis for the study of classroom social interactions. Computers & Education, 41(4), 353–368. Retrieved from http://www.sciencedirect.com/science/article/pii/S0360131503000824
Google Scholar CrossrefMartínez-Monés, A., Gómez-Sánchez, E., Dimitriadis, Y., Jorrín-Abellán, I. M., Rubia-Avi, B., & Vega-Gorgojo, G. (2005). Multiple Case Studies to Enhance Project-Based Learning in a Computer Architecture Course. IEEE Transactions on Education, 48(3), 482–489.
Google Scholar CrossrefMcKenney, S. (2013). Designing and researching technology-enhanced learning for the zone of proximal implementation. Research in Learning Technology, 21. doi:http://dx.doi.org/10.3402/rlt.v21i0.17374
Google Scholar CrossrefMiles, M. B., & Huberman, M. (1994). Qualitative Data Analysis: An Expanded Sourcebook. Newbury Park, CA, USA: Sage Publications.
Google Scholar CrossrefOliver, M. (2000). An introduction to the Evaluation of Learning Technology. Educational Technology & Society, 3(4), 20–30.
Google Scholar CrossrefOliver, M., & Conole, G. (2004). Evaluating information and communications technology: a tool kit for practitioners. In R. Holliman & E. Scanlon (Eds.), Mediating Science Learning through Information and Communications Technology (pp. 139–150). Routledge.
Google Scholar CrossrefOrlikowski, W. J., & Baroudi, J. J. (1991). Studying information technology in organizations: Research approaches and assumptions. Information Systems Research, 2(1), 1–28.
Google Scholar CrossrefPozzi, F., Manca, S., Persico, D., & Sarti, L. (2007). A general framework for tracking and analysing learning processes in computer-supported collaborative learning environments. Innovations in Education and Teaching International, 44(2), 169–179.
Google Scholar CrossrefPrieto, L. P. (2012). Supporting orchestration of blended CSCL scenarios in Distributed Learning Environments. School of Telecommunications Engineering, University of Valladolid, Spain.
Google Scholar CrossrefPrieto, L. P., Asensio-Pérez, J. I., Dimitriadis, Y., Gómez-Sánchez, E., & Muñoz-Cristóbal, J. A. (2011). GLUE!-PS: A multi-language architecture and data model to deploy TEL designs to multiple learning environments. In Proceedings of the European Conference on Technology-Enhanced Learning (EC-TEL 2011) (pp. 285–298).
Google Scholar CrossrefPrieto, L. P., Asensio-Pérez, J. I., Muñoz-Cristóbal, J. A., Dimitriadis, Y., Jorrín-Abellán, I. M., & Gómez-Sánchez, E. (2013). Enabling Teachers to Deploy CSCL Designs Across Distributed Learning Environments. IEEE Transactions on Learning Technologies, 6(4), 324–336. doi:http://doi.ieeecomputersociety.org/10.1109/TLT.2013.22
Google Scholar CrossrefPrieto, L. P., Asensio-Pérez, J. I., Muñoz-Cristóbal, J. A., Jorrín-Abellán, I. M., Dimitriadis, Y., & Gómez-Sánchez, E. (2014). Supporting orchestration of CSCL scenarios in web-based Distributed Learning Environments. Computers & Education, 73, 9–25. Retrieved from http://www.sciencedirect.com/science/article/pii/S0360131513003321
Google Scholar CrossrefPrieto, L. P., Holenko-Dlab, M., Abdulwahed, M., Gutiérrez, I., & Balid, W. (2011). Orchestrating Technology Enhanced Learning: a literature review and a conceptual framework. International Journal of Technology-Enhanced Learning (IJTEL), 3(6), 583–598.
Google Scholar CrossrefPrieto, L. P., Villagrá-Sobrino, S., Dimitriadis, Y., Schank, P., Penuel, W., & H., D. A. (2011). Mind the Gaps: Using Patterns to Change Everyday Classroom Practice Towards Contingent CSCL Teaching. In Proceedings of the International Conference of Computer-Supported Collaborative Learning (CSCL2011) (Vol. 1, pp. 518–525).
Google Scholar CrossrefRoschelle, J., Dimitriadis, Y., & Hoppe, U. (2013). Classroom orchestration: Synthesis. Computers & Education, 69, 523–526. doi:http://dx.doi.org/10.1016/j.compedu.2013.04.010
Google Scholar CrossrefSchön, D. A. (1983). The Reflective Practitioner - How Professionals Think in Action. Basic Books.
Google Scholar CrossrefStahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning: An historical perspective. In R. K. Sawyer (Ed.), Cambridge Handbook of the Learning Sciences. Cambridge University Press.
Google Scholar CrossrefStake, R. (2010). Qualitative Research: Studying How Things Work. The Guilford Press.
Google Scholar CrossrefStake, R. E. (1983). Program Evaluation, Particularly Responsive Evaluation. In Evaluation Models (Vol. 6, pp. 287–310). Springer Netherlands. doi:10.1007/978-94-009-6669-7_17
Google Scholar CrossrefStake, R. E. (2004). Standards-based and responsive evaluation. Sage.
Google Scholar CrossrefStrijbos, J. W., & Fischer, F. (2007). Methodological challenges for collaborative learning research. Learning and Instruction, 17(4), 389–393.
Google Scholar CrossrefSutherland, R., & Joubert, M. (2009). D1.1: The STELLAR vision and strategy statement. Retrieved from http://www.stellarnet.eu/kmi/deliverables/20090929_d1-1___vision-and-strategy.pdf
Google Scholar CrossrefTreleaven, L. (2004). A new taxonomy for evaluation studies of online collaborative learning. In Online collaborative learning: Theory and practice (pp. 160–179). Idea Group Inc. (IGI).
Google Scholar CrossrefTsiatsos, T., Andreas, K., & Pomportsis, A. (2010). Evaluation Framework for Collaborative Educational Virtual Environments. Educational Technology & Society, 13(2), 65–77.
Google Scholar CrossrefVatrapu, R., Suthers, D. D., & Medina, R. (2008). Usability, sociability, and learnability: A CSCL design evaluation framework. In Proceedings of the 16th International Conference on Computers in Education. Taipei, Taiwan.
Google Scholar CrossrefVillasclaras-Fernández, E., Hernández-Leo, D., Asensio-Pérez, J. I., & Dimitriadis, Y. (2013). Web Collage: An implementation of support for assessment design in CSCL macro-scripts. Computers & Education, 67, 79–97. doi:10.1016/j.compedu.2013.03.002
Google Scholar CrossrefDownloads
Published
Almetric
Dimensions
How to Cite
Issue
Section
License
All articles are published under Creative Commons copyright (CC BY). Authors hold the copyright and retain publishing rights without restrictions, but authors allow anyone to download, reuse, reprint, modify, distribute, and/or copy articles as the original source is cited.
