Orchestrating Evaluation of Complex Educational Technologies: A Case Study of a CSCL System

Authors

  • Luis P. Prieto École Polytechnique Fédérale de Lausanne
  • Yannis Dimitriadis Universidad de Valladolid
  • Juan I. Asensio-Pérez Universidad de Valladolid

https://doi.org/10.4471/qre.2014.44

Keywords:


Downloads

Abstract

As digital technologies permeate every aspect of our lives, the complexity of the educational settings, and of the technological support we use within them, unceasingly rises. This increased complexity, along with the need for educational practitioners to apply such technologies within multi-constraint authentic settings, has given rise to the notion of technology-enhanced learning practice as “orchestration of learning”. However, at the same time, the complexity involved in evaluating the benefits of such educational technologies has also increased, prompting questions about the way evaluators can cope with the different places, technologies, informants and issues involved in their evaluation activity. By proposing the notion of “orchestrating evaluation”, this paper tries to reconcile the often disparate “front office accounts” of research publications and the “shop floor practice” of evaluation of educational technology, through the case study of evaluating a system to help teachers in coordinating computer-supported collaborative learning scenarios (GLUE!-PS). We reuse an internationally-evaluated conceptual framework of “orchestration aspects” (design, management, adaptation, pragmatism, etc.) to structure the case’s narrative, showing how the original evaluation questions and methods were modulated in the face of the multiple (authentic) evaluation setting constraints.

Downloads

Download data is not yet available.

Metrics

Metrics Loading ...

References

Bote-Lorenzo, M. L., Gómez-Sánchez, E., Vega-Gorgojo, G., Dimitriadis, Y., Asensio-Pérez, J. I., & Jorrín-Abellán, I. M. (2008). Gridcole: a tailorable grid service based system that supports scripted collaborative learning. Computers & Education, 51(1), 155–172.

Google Scholar Crossref

Bruce, B. C. (2008). Ubiquitous Learning, Ubiquitous Computing, and life experience. In Proceedings of the 6th International Conference on Networked Learning (NLC2008) (pp. 583–590).

Google Scholar Crossref

Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches. Sage.

Google Scholar Crossref

Dillenbourg, P. (2013). Design for classroom orchestration. Computers & Education, 69, 485–492. doi:10.1016/j.compedu.2013.04.013

Google Scholar Crossref

Dillenbourg, P., Järvelä, S., & Fischer, F. (2009). The Evolution of Research in Computer-Supported Collaborative Learning: from design to orchestration. In N. Balacheff, S. Ludvigsen, T. de Jong, A. Lazonder, & S. Barnes (Eds.), Technology-Enhanced Learning: Principles and Products (pp. 3–19). Springer.

Google Scholar Crossref

Draper, S. W. (1997). Prospects for summative evaluation of CAL in higher education. Research in Learning Technology, 5(1).

Google Scholar Crossref

Economides, A. A. (2005). Collaborative Learning Evaluation (CLE) framework. WSEAS Transactions on Advances in Engineering Education, 2(4), 339–346.

Google Scholar Crossref

Ewing, J., & Miller, D. (2002). A framework for evaluating computer supported collaborative learning. Educational Technology & Society, 5(1), 112–118.

Google Scholar Crossref

Fischer, F., & Dillenbourg, P. (2006). Challenges of orchestrating computer-supported collaborative learning. In Paper presented at the 87th Annual Meeting of the American Educational Research Association (AERA).

Google Scholar Crossref

Garfinkel, H. (2002). Ethnomethodology’s program: Working out Durkheim's aphorism (Vol. 6). Rowman & Littlefield.

Google Scholar Crossref

Glass, R. L. (1995). A structure-based critique of contemporary computing research. Journal of Systems and Software, 28(1), 3–7.

Google Scholar Crossref

Gómez-Sánchez, E., Bote-Lorenzo, M. L., Jorrín-Abellán, I. M., Vega-Gorgojo, G., Asensio-Pérez, J. I., & Dimitriadis, Y. (2009). Conceptual framework for design, technological support and evaluation of collaborative learning. International Journal of Engineering Education, 25(3), 557–568.

Google Scholar Crossref

Guba, E. G. (1981). Criteria for assessing the trustworthiness of naturalistic inquiries. A Journal of Theory, Research, and Development on Educational Communication and Technology, 29(2), 75–91.

Google Scholar Crossref

Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Sage Publications, Inc.

Google Scholar Crossref

Hernández-Leo, D., Asensio-Pérez, J. I., & Dimitriadis, Y. (2005). Computational Representation of Collaborative Learning Flow Patterns Using IMS Learning Desing. Educational Technology & Society, 8(4), 75–89.

Google Scholar Crossref

Jorrín-Abellán, I. M., & Stake, R. E. (2009). Does Ubiquitous Learning Call for Ubiquitous Forms of Formal Evaluation?: An Evaluand oriented Responsive Evaluation Model. Ubiquitous Learning: An International Journal, 1(3), 71–82.

Google Scholar Crossref

Jorrín-Abellán, I. M., Stake, R. E., & Martínez-Monés, A. (2009). The Needlework in evaluating a CSCL system: The Evaluand-oriented Responsive Evaluation Model. In Proceedings of the 9th International Conference on Computer Supported Collaborative Learning (pp. 68–72).

Google Scholar Crossref

Kovalainen, M., Kumpulainen, K., & Satu, V. (2001). Orchestrating classroom interaction in a community of inquiry: Modes of teacher participation. Journal of Classroom Interaction, 36(2), 17–28.

Google Scholar Crossref

Luckin, R. (2008). The Learner Centric Ecology of Resources: a Framework for using Technology to Scaffold Learning. Computers & Education, 50, 449–462.

Google Scholar Crossref

MacNeill, S., & Kraan, W. (2010). Distributed Learning Environments: A Briefing Paper. Retrieved from http://publications.cetis.ac.uk/wp-content/uploads/2011/02/Distributed_Learning.pdf

Google Scholar Crossref

Martı́nez, A., Dimitriadis, Y., Rubia, B., Gómez, E., & de la Fuente, P. (2003). Combining qualitative evaluation and social network analysis for the study of classroom social interactions. Computers & Education, 41(4), 353–368. Retrieved from http://www.sciencedirect.com/science/article/pii/S0360131503000824

Google Scholar Crossref

Martínez-Monés, A., Gómez-Sánchez, E., Dimitriadis, Y., Jorrín-Abellán, I. M., Rubia-Avi, B., & Vega-Gorgojo, G. (2005). Multiple Case Studies to Enhance Project-Based Learning in a Computer Architecture Course. IEEE Transactions on Education, 48(3), 482–489.

Google Scholar Crossref

McKenney, S. (2013). Designing and researching technology-enhanced learning for the zone of proximal implementation. Research in Learning Technology, 21. doi:http://dx.doi.org/10.3402/rlt.v21i0.17374

Google Scholar Crossref

Miles, M. B., & Huberman, M. (1994). Qualitative Data Analysis: An Expanded Sourcebook. Newbury Park, CA, USA: Sage Publications.

Google Scholar Crossref

Oliver, M. (2000). An introduction to the Evaluation of Learning Technology. Educational Technology & Society, 3(4), 20–30.

Google Scholar Crossref

Oliver, M., & Conole, G. (2004). Evaluating information and communications technology: a tool kit for practitioners. In R. Holliman & E. Scanlon (Eds.), Mediating Science Learning through Information and Communications Technology (pp. 139–150). Routledge.

Google Scholar Crossref

Orlikowski, W. J., & Baroudi, J. J. (1991). Studying information technology in organizations: Research approaches and assumptions. Information Systems Research, 2(1), 1–28.

Google Scholar Crossref

Pozzi, F., Manca, S., Persico, D., & Sarti, L. (2007). A general framework for tracking and analysing learning processes in computer-supported collaborative learning environments. Innovations in Education and Teaching International, 44(2), 169–179.

Google Scholar Crossref

Prieto, L. P. (2012). Supporting orchestration of blended CSCL scenarios in Distributed Learning Environments. School of Telecommunications Engineering, University of Valladolid, Spain.

Google Scholar Crossref

Prieto, L. P., Asensio-Pérez, J. I., Dimitriadis, Y., Gómez-Sánchez, E., & Muñoz-Cristóbal, J. A. (2011). GLUE!-PS: A multi-language architecture and data model to deploy TEL designs to multiple learning environments. In Proceedings of the European Conference on Technology-Enhanced Learning (EC-TEL 2011) (pp. 285–298).

Google Scholar Crossref

Prieto, L. P., Asensio-Pérez, J. I., Muñoz-Cristóbal, J. A., Dimitriadis, Y., Jorrín-Abellán, I. M., & Gómez-Sánchez, E. (2013). Enabling Teachers to Deploy CSCL Designs Across Distributed Learning Environments. IEEE Transactions on Learning Technologies, 6(4), 324–336. doi:http://doi.ieeecomputersociety.org/10.1109/TLT.2013.22

Google Scholar Crossref

Prieto, L. P., Asensio-Pérez, J. I., Muñoz-Cristóbal, J. A., Jorrín-Abellán, I. M., Dimitriadis, Y., & Gómez-Sánchez, E. (2014). Supporting orchestration of CSCL scenarios in web-based Distributed Learning Environments. Computers & Education, 73, 9–25. Retrieved from http://www.sciencedirect.com/science/article/pii/S0360131513003321

Google Scholar Crossref

Prieto, L. P., Holenko-Dlab, M., Abdulwahed, M., Gutiérrez, I., & Balid, W. (2011). Orchestrating Technology Enhanced Learning: a literature review and a conceptual framework. International Journal of Technology-Enhanced Learning (IJTEL), 3(6), 583–598.

Google Scholar Crossref

Prieto, L. P., Villagrá-Sobrino, S., Dimitriadis, Y., Schank, P., Penuel, W., & H., D. A. (2011). Mind the Gaps: Using Patterns to Change Everyday Classroom Practice Towards Contingent CSCL Teaching. In Proceedings of the International Conference of Computer-Supported Collaborative Learning (CSCL2011) (Vol. 1, pp. 518–525).

Google Scholar Crossref

Roschelle, J., Dimitriadis, Y., & Hoppe, U. (2013). Classroom orchestration: Synthesis. Computers & Education, 69, 523–526. doi:http://dx.doi.org/10.1016/j.compedu.2013.04.010

Google Scholar Crossref

Schön, D. A. (1983). The Reflective Practitioner - How Professionals Think in Action. Basic Books.

Google Scholar Crossref

Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning: An historical perspective. In R. K. Sawyer (Ed.), Cambridge Handbook of the Learning Sciences. Cambridge University Press.

Google Scholar Crossref

Stake, R. (2010). Qualitative Research: Studying How Things Work. The Guilford Press.

Google Scholar Crossref

Stake, R. E. (1983). Program Evaluation, Particularly Responsive Evaluation. In Evaluation Models (Vol. 6, pp. 287–310). Springer Netherlands. doi:10.1007/978-94-009-6669-7_17

Google Scholar Crossref

Stake, R. E. (2004). Standards-based and responsive evaluation. Sage.

Google Scholar Crossref

Strijbos, J. W., & Fischer, F. (2007). Methodological challenges for collaborative learning research. Learning and Instruction, 17(4), 389–393.

Google Scholar Crossref

Sutherland, R., & Joubert, M. (2009). D1.1: The STELLAR vision and strategy statement. Retrieved from http://www.stellarnet.eu/kmi/deliverables/20090929_d1-1___vision-and-strategy.pdf

Google Scholar Crossref

Treleaven, L. (2004). A new taxonomy for evaluation studies of online collaborative learning. In Online collaborative learning: Theory and practice (pp. 160–179). Idea Group Inc. (IGI).

Google Scholar Crossref

Tsiatsos, T., Andreas, K., & Pomportsis, A. (2010). Evaluation Framework for Collaborative Educational Virtual Environments. Educational Technology & Society, 13(2), 65–77.

Google Scholar Crossref

Vatrapu, R., Suthers, D. D., & Medina, R. (2008). Usability, sociability, and learnability: A CSCL design evaluation framework. In Proceedings of the 16th International Conference on Computers in Education. Taipei, Taiwan.

Google Scholar Crossref

Villasclaras-Fernández, E., Hernández-Leo, D., Asensio-Pérez, J. I., & Dimitriadis, Y. (2013). Web Collage: An implementation of support for assessment design in CSCL macro-scripts. Computers & Education, 67, 79–97. doi:10.1016/j.compedu.2013.03.002

Google Scholar Crossref

Downloads

Published

2014-06-28

Almetric

Dimensions

How to Cite

Prieto, L. P., Dimitriadis, Y., & Asensio-Pérez, J. I. (2014). Orchestrating Evaluation of Complex Educational Technologies: A Case Study of a CSCL System. Qualitative Research in Education, 3(2), 175–205. https://doi.org/10.4471/qre.2014.44