Chapter four - Design-Based Research



Origins of the OU Learning Design Initiative

The OU Learning Design Initiative emerged from previous work on the development of a learning design toolkit, DialogPlus (Fill and Conole, 2008). Like the Phoebe and the LPP tools, DialogPlus was intended to act as a step-by-step guide to enable teachers to create learning designs. The tool was based on an underlying taxonomy which defined the components of a learning activity (Conole, 2008), which was derived through a series of interviews with teachers about their design practices. However, evaluation of the actual use of such design planner tools indicated that they did not match actual design practice closely enough. Their relatively linear and prescriptive structure did not match the creative, iterative and messy nature of actual teacher design practice.

 

The OU Learning Design Initiative was initiated in 2007, supported through strategic funding from the OU. The intention was to derive a more practice-focussed approach to learning design, identified from empirical evidence of actual practice. This included gathering 43 case studies of the ways in which the then new Learning Management System (LMS) (Moodle) was being used (Wilson, 2007) and a series of interviews with teachers to articulate their actual teaching practice (Clark and Cross, 2010). The key focus of the teacher interviews was to better understand existing practice. The authors note in their introduction that ‘Even experienced academics who have participated in a range of course production tasks find it difficult to articulate how they go about developing a “learning design” that will be transformed into effective learning materials’ (Clark and Cross, 2010). The interviews focussed on five main questions: i) process: how do teachers go about designing a course?, ii) support: how do they generate ideas?, iii) representation: how do they represent their designs?, iv) barriers: what barriers do they encounter?, v) evaluation: how do they evaluate the effectiveness of the design?

 

A range of approaches to design were evident, including gathering of resources, brainstorming, listing concepts and skills, creating week-by-week plans, etc. On the whole these were paper-based and primarily text-based. There was little evidence of use of alternative, more visual representations or visual software tools. Interviewees wanted help with understanding how to integrate ICT-based activities into courses. Face-to-face workshops and meetings were favoured over online support as they were felt to be the most effective way of thinking about, and absorbing, new ideas and ways of working. Case studies interestingly were considered to be too demanding in time and effort, interviewees wanted just-in-time support to specific queries. The most effective form of support was considered to be sharing of experience with peers. A variety of representations were mentioned from simple textual representations or lists through to more complex and connected mindmaps. The interviewees listed a variety of purposes for the representations, including communicating personal vision, capturing or sharing ideas, comparing with others, viewing the course at different levels and mapping content to learning outcomes. Barriers included concerns about a lack of experience of creating online activities and a lack of successful examples and an OU-specific issue in terms of the difficulty of melding together the innovative (and often idiosyncratic) ideas of course creators with the needs of a production system delivering the OU’s size and range of learning materials and services. A range of mechanisms were cited in terms of evaluation approaches. These included feedback from students and tutors, comments from critical readers, peer course team critiques and comments from external examiners.

This empirical work provided a sound basis for the development of our approach. Our initial focus centered on the following questions:

 

·      How can we gather and represent practice (and in particular innovative practice) (capture and represent practice)?

·      How can we provide ‘scaffolds’ or support for staff in creating learning activities that draws on good practice, making effective use of tools and pedagogies (support learning design)? (Conole, 2009).

 

We have identified six reasons why adopting a learning design approach might be beneficial:

1.     It can act as a means of eliciting designs from academics in a format that can be tested and reviewed with developers, i.e. a common vocabulary and understanding of learning activities.

2.     It provides a means by which designs can be reused, as opposed to just sharing content.

3.     It can guide individuals through the process of creating learning interventions.

4.     It creates an audit trail of academic design decisions.

5.     It can highlight policy implications for staff development, resource allocation, quality, etc.

6.     It aids learners in complex activities by guiding them through the activity sequence.

 

These map closely with the benefits of adopting a design-based approach outlined by Gibbons and Brewer (2005). They argue that the benefits include: improving the rate of progress (in the creation of designs), influencing the designer conceptions through making the design process explicit, helping to improve design processes, improvements in design and development tools, and bringing design and production closed together. Fundamentally, I would agree with their assertion that it opens up new ways of thinking about designs and designing.

 

We were interested in a number of research questions in particular. Can we develop a range of tools and support mechanisms to help teachers design learning activities more effectively? Can we agree a shared language/vocabulary for learning design, which is consistent and rigorous, but not too time consuming to use? How can we provide support and guidance on the creation of learning interventions? What is the right balance of providing detailed, real, case studies, which specify the detail of the design, compared with more abstract design representations that simply highlight the main features of the design? How can we develop a sustainable, community of reflective practitioners who share and discuss their learning and teaching ideas and designs?

An overview of Design-Based research

This section draws in particular on Barab (2006) and Kelly et al. (2008). Barab provides a useful overview of Design-Based Research (Barab, 2006, p. 155). He argues that the value of Design-Based Research (DBR) is that it offers a methodology for dealing with the complexity of real learning contexts by ‘iteratively changing the learning environment over time – collecting evidence of the effect of these variations and feeling it recursively into future designs’ (citing Brown, 1992; Collins, 1992). He argues that cognition, ‘rather than being a disembodied process occurring in the confines of the mind, is a distributed process spread out across the knower, the environment, and even then meaning of the activity’ (citing Salomon, 1993). Barab suggest that DBR can yield rich insights into the complex dynamics whereby theories become contextualised. He lists the following as mechanisms for making DBR effective:

1.     Make assumptions and theoretical bases that underlie the work explicit

2.     Collect multiple types of theoretically relevant data

3.     Conduct ongoing data analysis in relation to theory

4.     Invite multiple voices to critique theory and design

5.     Have multiple accountability structures

6.     Engage in dialectic among theory, design and extant literature.

He argues that DBR has the following characteristics: design, theory, and problem in the context of a naturalistic setting, involving multiple iterations or progressive refinement (Figure 1).

ch4_fig1.jpg

Figure 1: The interactive nature of Design-Based Research

 

Kelly et al. (2008, p. 5) suggests that DBR foregrounds ‘the fluid, empathetic, dynamic, environment-responsive, future-orientated and solution-focused nature of design’.

The OULDI learning design methodology

We are adopting a design-based research (DBR) approach; starting with a stated problem we were trying to address, a proposed solution and then an iterative cycle of developments and evaluation. Design-based research has emerged in recent years as an approach for studying learning in context through systematic design and study of instructional strategies and tools (Brown, 1992; Collins, 1992 cited in Design-Based Research Collective, 2003). Wang and Hannafin (2005:5-6) define it as ‘a systematic, but flexible methodology aimed to improve educational practice through iterative analysis design, development and implementation, based on collaboration between researchers and practitioners in real-world settings, and leading to contextually-sensitive design principles and theories’. Reigeluth and An (2009:378-379) articulate the following set of characteristics of DBR:

1.      It is driven by theory and prior research. In our work, as described above we are building on the substantive body of prior research on instructional design, learning sciences, learning objects/Open Educational Resources and more recently learning design. The approach we adopt is socio-cultural in nature, with a focus on the design and use of a range of mediating artefacts involved in teaching-learning processes (See Conole, 2008 for a more detail account of this).

2.      It is pragmatic. Our aim is to develop tools and resources which are useful in actual practice, by practitioners to address real educational challenges. Our intention is to be theory-driven, but pragmatic, recognising the complex, messy and often craft-based nature of teaching practice.

3.      It is collaborative. We see working in close connection with end users as a vital part of our approach. Our initial interviews with teachers confirmed our view that teaching practice is complex and situated. Changing practice will only occur through close working with and understanding of practitioners’ needs.

4.      It is contextual. Our vision is to change actual practice, to achieve this it is important that the development activities occur in real, authentic contexts.

5.      It is integrative. Wang and Hannifin (2005: 10) state that ‘DBR uses a variety of research methods that vary as new needs and issues emerge and the focus of the research evolves’. We have adopted a mixed-method approach to evaluating our developments, matching the methods we use to the specific sub-research questions and the context that we are focusing on.

6.      It is iterative. Our approach consists of an interactive cycle of identification of problems to be addressed, suggestion of proposed solutions, development, use, evaluation and refinement.

7.      It is adaptive and flexible. Because our work is closely tied to actual practice, we need to ensure that the approach we are adopting is agile in nature, so that we can adapt based on evidence from changing practice.

8.      It seeks generalisation. In addition to the practical, pragmatic nature of our work, we are also attempting to develop a coherent underlying learning design framework of concepts and approaches.

 

The main components of the OU Learning Design methodology

In essence we are focusing on three aspects of design: i) the development of a range of conceptual tools to guide the design process and provide a means of representing (and hence sharing) designs, ii) the development of visual tools to render some of the conceptual tools and enable practitioners to manipulate their designs and share them digitally with others, iii) the development of collaborative tools – both in terms of structures for face-to-face events such as workshops and use of digital tools to foster communication and sharing. For each aspect we have now developed a set of tools, resources and activities and over the last two years we have been trialling these in a range of settings, both with the OU and also externally with a number of partner institutions and through demonstrations and workshops at conferences. It would be impossible in the scope of this paper to describe all the tools, resources and activities in detail; hence a selection will be described to give an overall view of the work to date. An evolving online learning design toolkit is being developed which includes our current set of tools, resources and activities (http://cloudworks.ac.uk/cloudscape/view/1882). In addition a learning activity taxonomy has been developed (Conole, 2008) and more recently a Learning Design taxonomy which provides a map of the domain, the key concepts and where individual tools, resources and activities fit (Conole, 2010a).

 

OULDI aims to bridge the gap between the potential and actual use of technologies outlined in the introduction, through the development of a set of tools, methods and approaches to learning design, which enables teachers to making better use of technologies that are pedagogically informed. Conole (2009) provides a reflection on the origins of OULDI and the benefits of adopting this approach. The aim is to provide a design-based approach to the creation and support of learning and teaching, and to encourage a shift away from the traditional implicit, belief-based approaches to design-based, explicit approaches. This will encourage sharing and reflection. The tools and resources are designed to help guide decision-making. The work is underpinned by an ongoing programme of empirical evidence which aims to gain a better understanding of the design process and associated barriers and enablers, as well as an ongoing evaluation of the tools, methods and approaches we are developing and using and in particular to what extent they are effective. There are three main aspects to the work we are doing:

1.      Conceptualisation – the development of a range of conceptual tools to help guide the design decision-making process and to provide a shared language to enable comparisons to be made between different designs.

2.      Representation – identification of different types of design representation and use of a range of tools to help visualise and represent designs.

3.      Collaboration – mechanisms to encourage the sharing and discussing of learning and teaching ideas.

4.       

Barab, S. (2006). Design-Based Research - A methodological toolkit for the Learning Scientist. In R. K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (pp. 153-169). Cambridge Cambridge University Press.

Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. The journal of the learning sciences, 2(2), 141-178.

Collins, A. (1992). Towards a design science of education. In E. Scanlon & T. O’Shea (Eds.), New directions in educational technology (pp. 15-22). New York: Springer-Verlag.

Kelly, A. E., Baek, J. Y., Lesh, R. A., & Banna-Ritland, B. (2008). Enabling innovations in education and systematizing their impact. In A. E. Kelly, R. A. Lesh & J. Y. Baek (Eds.), Handbook of design research methods in education: innovations in science, technology, engineering and mathematics learning and teaching (pp. 3-18). New York and London: Routledge.

Kelly, A. E., Lesh, R. A., & Baek, J. Y. (2008). Handbook of design research methods in education: innovations in science, technology, engineerin and mathematics learning and teaching. New York and London: Routledge.

Salomon, G. (Ed.). (1993). Distributed cognitions - pyschological and educational considerations. Cambridge: Cambridge University Press.

 

5.       

Leave a Reply