The METIS evaluation

metis.png

The METIS project has three main objectives:

  1. To develop an Integrated Learning Design Environment (ILDE). The ILDE will integrate existing free and open source solutions that include: co-design support for communities of practitioners; learning design authoring tools following different pedagogical approaches and authoring experiences; interface for deployment of learning designs on mainstream Virtual Learning Environments (VLEs).
  2. To run a series of workshops for teachers at partner institutions using ILDE. The workshops will be aimed at fostering the adoption of learning design methods among teachers and advancing their skills in the orchestration of ICT-based learning environments according to innovative pedagogical approaches. The ILDE will play a central role in the workshops, malady because one of workshop goals will be to support teachers’ familiarization with the ILDE and to promote the usage of the authoring tools integrated in it.
  3. To disseminate the project’s outcomes and maintain a community of teachers engaged with learning design and its tools.

Work Package 5 of the METIS project is concerned with evaluation. This work package is led by Istitulo Tecnologie Didattiche (ITD), cheap in collaboration with other partners. METIS is adopting a user-centred design approach: the development of both the ILDE and the workshops will be cyclic, nurse with two evaluation phases informed by practice. These two evaluation phases, nested within the four cycles of the METIS project development, will incrementally incorporate the needs expressed by end users for both the ILDE and the workshops. The workshops are the basis for the formative evaluation of the different versions of the ILDE and of the workshop packs themselves. Thus evaluation will play a crucial role in the project, will occur in an iterative way and will be formative, i.e. aimed at informing the following stages of re-design and development.

Section 1 provides an overview of the role of the evaluation. Section 2 provides a clear definition of what is meant by evaluation, with arguments being made backed up by relevant research literature. The approach adopted draws on Guskey (2002) definition; i.e. that ‘evaluation is “the systematic investigation of merit or worth” (Joint Committee on Standards for Educational Evaluation, 1994, p. 3). It goes on to state that:

… evaluation in METIS will be mainly oriented to pinpoint strengths and weaknesses of both the ILDE and the workshop package and to find useful indications to further tune and improve them, so we can state that our evaluation will primarily have formative aims. 

Section 3 concentrates on the ILDE. Theoretically the focus is on user’s acceptance of technology, based around the follow factors:

  • Attitude Toward Behavior
  • Subjective Norm
  • Perceived Usefulness
  • Perceived Ease of Use
  • Extrinsic Motivation
  • Intrinsic Motivation
  • Job-fit
  • Long-term Consequences
  • Social Factors
  • Relative Advantage
  • Voluntariness of Use
  • Image
  • Visibility
  • Performance Expectancy
  • Effort Expectancy
  • Social Influence

The section then outlines six theoretical models in relation to technology acceptance:

  • Theory of Reasoned Action (TRA)
  • Technology Acceptance Model (TAM)
  • Motivational Model (MM)
  • Model of PC Utilization (MPCU)
  • Innovation Diffusion Theory (IDT)
  • Unified Theory of Acceptance and Use of Technology (UTAUT)

Having reviewed these the Technology Acceptance Model was chosen; Section 3.2 provides a clear rationale for the choice.  Section 4 describes the approach adopted to the evaluation of the workshops. It states that there are two approaches to this: objective/goal-based models and systems-based models, and lists the follow models:

  • Kirkpatrick’s 4 levels
  • Hamblin’s 5 levels
  • Guskey’s 5 levels
  • Tyler’s model
  • Hammond’s model
  • CIPP (Context, Input, Process, Product)
  • IPO (Input, Process, Output)
  • TVS (Training Valuation System)

Having examined the models, the team selected the Guskey’s model as the main source of inspiration for the evaluation of the METIS workshop packages, as this model seems to offer the closest fit with the project requirements. Using this the following aspects are listed: participants’ reactions, participants’ learning, organization support and change, participants’ use of new knowledge and skills, and student learning outcomes.

Section 5 states that:

In METIS the evaluation of both the ILDE and the workshops will be conceived and designed by a team of researchers of ITD-CNR, and then carried out with the support of all the METIS partners and particularly those partners who will design and run the workshops (UKOU, KEK and Agorà + ULeicester as WP4 responsible), as well as the partner who is developing the ILDE (UBa and all WP2 partners). ITD-CNR is not involved in the development of the system and the workshops, except for what attained to its role of formative evaluator.

Section 6 provides a discussion of the critical issues associated with the evaluation and in particular the fact that the evaluation will occur in real contexts. Section 7 provides a conclusion for the report and next steps.

The document provides a clear and methodologically robust outline of the evaluation plan. It makes a clear rationale for the approach adopted for the evaluation of both the ILDE and the workshops. The report draws on a range of theoretical models associated with users’ acceptance of technology and uses this to develop a robust and thorough evaluation plan. Clear timescales and milestones are indicated, along with a critical discussion of any anticipated issues associated with the evaluation.

References

Guskey, T. R. (2000). Evaluating professional development. Corwin Press, Inc. Thousand Oaks, California.

Guskey, T. R. (2002). Professional development and teacher change. Teachers and Teaching: theory and practice, 8(3), 381-391.

Venkatesh, V., and Davis, F. D. (2000). A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies. Management Science (45:2), pp. 186-204.

Venkatesh, V., Morris, M. G., Davis, G. B., and Davis, F. D. (2003). User Acceptance of Information Technology: Toward a Unified View. MIS Quarterly (27:3), pp. 425-478.

Venkatesh, V., Thong, J. Y., Xin, X. (2012). Consumer acceptance and use of information technology: extending the Unified Theory of Acceptance and Use of Technology. MIS Quarterly (36:1), pp. 157-178.

 

 

 

Leave a Reply