Between Analysis and Transformation: Technology, Methodology and Evaluation on the SPLICE project

This is an abstract for a paper I'm presenting at ALT-C in Manchester this year. Methodology has been my obsession this year. I'm particularly interesting in the way that technology is changing the way we 'know' things, and how it can contribute to collegial decision-making. The paper documents some of the tools we developed on the SPLICE project.

This paper concerns the ways in which technological change entails methodological development in e-learning research. The focus of our argument centres on the subject of evaluation in e-learning and how technology can contribute to consensus-building on the value of project outcomes, and the identification of mechanisms behind those outcomes. We argue that a critical approach to evaluation which harnesses technology in this way is vital to agile and effective policy and strategy making in institutions as they grapple with the challenges of transformation in a rapidly changing educational and technological environment.

With its focus on mechanisms, we identify Pawson and Tilley’s ‘Realistic Evaluation’ as an appropriate methodological approach, and we report on its use within a JISC-funded project on social software, SPLICE (Social Practices, Learning and Interoperability in Connected Environments). The project created new tools to assist the identification of mechanisms responsible for change to personal and institutional technological practice. These tools included collaborative mind-mapping and focused questioning, and tools for the animated modelling of complex mechanisms. By using these tools, large numbers of project stakeholders could engage in a process where they were encouraged to articulate and share their theories and ideas as to why project outcomes occurred. Using the technology, this process led towards the identification and agreement of common mechanisms which had explanatory power for all stakeholders.

We argue that the opportunities for mass collaboration and communication afforded by technology have methodological implications. Given this technology, an approach based on the mass cumulation of stakeholder theories and ideas about mechanisms feasible. A technology-driven realistic evaluation can lead to summative outcomes which are rich in explanatory and predictive power, and useful to sector. Finally, we argue that as well as generating better explanations for phenomena, the evaluation process can itself become transformative for stakeholders.