at 1st February 2001

TILT Group E - Evaluation

This group aimed to develop and apply evaluation methods for improvements in teaching & learning by using IT. They also trained and assisted other groups to carry out evaluations.

The TILT Evaluation Group, based mainly in Psychology, made important advances in adapting evaluation methods so that the application of IT to real classes can be well informed.

References

1. Draper S W, Brown M I,Edgerton E,Henderson,F P,McAteer E,Smith E D,Watt H D,Observing And Measuring The Performance Of Educational Technology: ISBN 085261 521 3, , TILT University of Glasgow (March 1994).
2. Doughty G,Arnold S,Barr N,Brown M,Creanor L,Donnelly P,Draper S,Duffy C,Durndell H,Harrison M,Henderson F,Jessop A,McAteer E,Milner M,Neil D,Pflicke T,Pollock M,Primrose C,Richard S,Sclater N,Shaw R,Tickner S,Turner I,van der Zwan R,Watt H; Using Learning Technologies: Interim Conclusions from the TILT Project , TILT University of Glasgow (1994).
3. Stephen W. Draper Observing, Measuring, or Evaluating Courseware
4. Barr N S F ,McAteer E,Neil D M,Integrating computer-based learning with conventional laboratory teaching, TLTP Newsletter 2 (1994).
5. Draper S W,McAteer E,Tolmie A K,Anderson A, Bringing a constructivist approach to bear on evaluation, Foot et al (Eds), Group and Interactive Learning,Computational Mechanics Inc (1994).
6. Barr N S F,McAteer E,Neil D M, CBL in the Laboratory, Life Sciences Educational Computing, 5 (1995).
7. Neil D M,McAteer E,Barr N S F,Heitler W J, Evaluation of a neurophysiological simulation in a real teaching context,Soc Exp Biology Annual Meeting (1995).
8. Duffy C,Arnold S,Henderson F, NetSem - electrifying undergraduate seminars , Active Learning, 2 (1995)
9. Creanor L,Durndell H,Henderson F P,Primrose C,Brown M I,S Draper S W,McAteer E, A Hypertext Approach to Information Skills: Development and Evaluation, TILT University of Glasgow (1995).
10. McAteer,E,Harland,M,and Sclater N, De Tudo Um Pouco - a little bit of everything, Active Learning 3 (1996).
11. Draper S,Brown M I, Henderson F P,McAteer E, Integrative Evaluation: an emerging role for classroom studies, Comp & Educ CAL 95 special edition (1996).
12. McAteer E,Neil D,Barr N,Brown M,Draper S,Henderson F, Simulation software in a Life Sciences practical course, Comp & Educ CAL 95 special edition (1996).
13. Pollock M,McAteer E,Doughty G,Turner I,Rapid conversion of a mathematics course to CAL: a case study of a large-scale rapid change of resources and organisation, ALT-J,4 (1996)
14. Brown M,Doughty G F,Draper S W,Henderson F P,McAteer E, Measuring Learning Resource Use,to be published in Comp & Educ (1996)

Summary of Evaluation Practice

We performed about 20 evaluation studies of teaching software in Glasgow University across a very wide range of subject disciplines. Some exercises have been single episode field trials, others have looked at CAL material in use within a full degree course. Our approach has been empirical: trying to observe effects on students using both quantitative and qualitative methods. In the process, we have learnt as much about evaluation as about the courseware. We employ a variety of instruments, three of the most important in our recent practice being student confidence logs, short quizzes and learning resource questionnaires.

These studies, necessarily as well as from principle, have been done with the collaboration of the teaching staff, and depended upon their cooperation, their domain expertise, and usually substantial work on their part, for instance in devising and marking tests. One or more meetings take place between evaluators with teachers and developers to:

•  
• Establish the teachers' goals for the evaluation
• Elicit the learning aims and objectives to be studied
• Discuss the creation of a learning quiz
• Elicit the classroom provision, course integration, and other features of the teaching situation and support surrounding the courseware itself.
• Work out what opportunity there is for classroom observation and administering the instruments (e.g. questionnaires) to the students.
• Evaluator and teacher finalise a design for the study.

Every study is different, but a prototypical design might have:

1. A pre-task questionnaire to discover aspects of what each student brings to the session e.g. prior experience, personal motivation
2. Confidence logs after each kind of activity.
3. A learning test ("quiz"). Ideally a version of this would be administered at the start of the session, at the end of the session, and after a delay of some weeks.
4. Access to subsequent exam performance on one or more relevant questions.
5. Post-task questionnaire to elicit personal reactions to the experience, and to ask about the relative value each individual put on various resources.
6. Interviews of a sample of students
7. Observation and/or video-taping of one or more individuals.

Classroom study occurs.

A preliminary report from the evaluators is presented to the teacher, and interpretations of the findings sought and discussed.

A final report is produced, written mainly by the evaluators, but incorporating the teacher's interpretations and conclusions.

Outcomes and Insights

Some of the conclusions from our work are:

•  
• Teachers often find our reports useful, not as summative measures of the quality of the software, but as formative evaluations of the teaching situation in which it is embedded.
• For much courseware, the only prior computer experience that seems to be relevant is experience of using a mouse. On the other hand, needing to use a spreadsheet or database within the CAL exercise not only requires (relevant) training on software operation, but also depends on whether the student has prior experience of that kind of application interface.
• Simulation software seems to be a valuable addition to practical classes, but there are indications that it is best used in conjunction with physical labs rather than as a substitute, and with human support from tutors and demonstrators.
• We have also worked on diagnostic evaluation during courseware development. One approach is to administer a test, where each question corresponds to a learning objective: low average scores on a particular question points to the objective that needs better treatment. Another is to divide the courseware into sections of a few screens each, and ask test subjects to comment immediately after each section; and to award scores separately for presentation and content.
• Educational interventions have a small effect on learning outcomes compared with other factors - eg whether the students believe they will be examined upon the material.
• If measurements made on one occasion are to have any predictive power, we need a theory of the overall teaching and learning process that specifies what the main causal factors determining learning outcomes might be, develop an instrument for each factor and record the value of each factor in the situation studied.
• Situational factors are critical - the role of the courseware within the course, its integration with other teaching components the availability and use (and usefulness) of other learning resources: the influence of individual and social variables.

ELICT

The TILT Evaluation group's work, and subsequent other activities, has led to an Evaluation service: Evaluation of Learning with Information & Communication Technology

provided by the Robert Clark Centre for Technological Education

 ELICT,
 Robert Clark Centre for Technological Education
 66 Oakfield Avenue
 University of Glasgow        Telephone: 0141 330 4976
 Glasgow G12 8LS              E-mail: rcc@elec.gla.ac.uk

For further information

Contact:

Dr Steve Draper
Department of Psychology
University of Glasgow
Glasgow  G12 8QQ  UK 
email: steve@psy.gla.ac.uk

See Steve Draper's Home Page.

Back to the main TILT topics:

TILT home page

Overview

Learning software packages

What have we learned?

Acronyms

Copyright © University of Glasgow TILT Project