Alice Szwelnik, 2005 Oxford Brookes University

Methodology

This research is underpinned by a review of the previous work and visits to selected universities and colleges with low and high QAA rating for ‘Quality management and enhancement’. Twenty senior academic staff and managers involved in quality assurance were interviewed between March and July 2005

Literature review

The literature review identified a number of interrelated purposes for evaluation:

Hounsell & Tait & Day, 1997

• To ascertain how well a course or programme of study is doing, and ;thus to evaluate the extent to which it meets its stated aims and objectives ;or learning outcomes. ;
• To identify both its chief strengths and its main weaknesses so that strengths can be appropriately acknowledged and ;capitalised upon, while weaknesses can be remedied or at least ;ameliorated wherever possible.

Harvey, 2001

• Internal information to guide improvement.
• External information for potential students and other stakeholders.

CNAA Report, 1992

• Accountability, management data.
• Enhancement of students’ ability to evaluate and reflect.

FDTL Briefing Paper 1

• To improve teaching quality and students’ learning experiences.
• To inform decisions on course content and delivery.
• To feed into central decision-making.
• To increase student involvement in their education process.

Why do we evaluate modules?

Several reasons:

1. To satisfy institutional requirements (faculty, QAA)
2. 
   
3. To improve my teaching
4. To improve the students’ learning experience
5. To identify staff development needs
6. Other.

In most cases respondents evaluate modules in order ‘to improve students’ learning experience’. This shows that a student-centred approach is widely accepted.

How do we evaluate modules? Summative versus formative evaluation

In all institutions the summative evaluation was the most common approach. Summative assessment collected at the end of the module was seen as more appropriate because it:

• gives students the opportunity to express views after the whole experience of the module; sometimes students realise the value of the module only at the end;
• is not possible, half-way through the module, to suddenly change the teaching formula or assessment
• gives module leaders a chance to take into consideration not only students’ feedback, but also input from external examiners, the teaching team and the school’s committee.

The majority of respondents agreed that formative evaluation in the middle of the semester/term is useful an important, but not feasible considering the time pressure.

How do we evaluate modules? Standardised and diversified systems

With standardised systems, a standard questionnaire which can be applied to all modules is designed by the management team responsible for quality assurance. They collect, process and analyse all the data. In diversified systems, different methods of evaluation may be used for different modules, at the discretion of the module leader or teacher. This person has ownership of the data and is generally responsible for its collection and analysis, and any action to be taken as a result of findings.

;

Advantages of the standardised system

• Objectivity/credibility of feedback.
• Comparability across the modules.
• Lecturers do not spend time designing, collecting and processing data.
• Ownership by management team increases credibility of the data.
• More effective monitoring system.
• Data can be used for marketing purposes.
• Data can be used as a benchmark against other institutions (in the case of an external body conducting the process).
• Students feel more comfortable giving feedback to administrators then to module leaders (more anonymity).
• Lecturers are rewarded for ‘good teaching’.
• Effective system for quick identification of serious problems with modules.
• Disadvantages of the standardised system
• The process is too remote from a module leader, who is the key stakeholder in the process. If lecturers feel less ownership of the process, it is more difficult to make changes and improve quality.
• One standardised questionnaire is not appropriate for a wide variety of modules.
• Mechanistic attitude towards the process.
• Negative impact on staff morale. Many respondents said ‘We are employed as professionals, so we should be treated as professionals’.
• Student fatigue – students can become bored answering the same questions.
• Not possible to ask specific questions, which may be especially relevant for new material.

Who is the evaluation for?

• Professional bodies
• Lecturer
• External examiners
• Students
• Course managers/faculty members
• Employers

• QAA
• Other

The majority of respondents perceive lecturers and students to be the most important stakeholders in the process.

What would be the most effective way of communicating with students on actions planned and taken?

The majority of interviewees identified lack of effective communication with students as a weakness of the evaluation process. There was no consensus as to the most effective way of communication with students. The most popular option was the face-to-face communication with new cohort of students.

Conclusion

The introduction of standardised questionnaires, administered centrally, should improve the quality of modules because:

• the management team will be able to identify poor lecturers.
• an immediate action would be taken in agreement between lecturer and the line manager.
• the progress will be easily monitored by the line manager.
• students would appreciate external monitoring by management team.
• there will be more transparency and comparability of the process.

As the quality of modules improves the management team should consider transferring part of the responsibility and ownership of the module evaluation to module leaders. This could be achieved by introducing a standardised questionnaire with three questions designed by module leader to address the specific aspects of a particular module. This should encourage a module leader to greater reflection and involvement in the process.

Over time, when the management team is generally satisfied with the process and the quality of the modules, module leaders can be given more responsibility for the quality assurance and more ownership of the process. All respondents agreed that the more ownership of the process by the module leader, the better involvement and responsiveness. If the quality is high there is no need for rigorous control systems by the management team. However, retaining some generic questions for all modules would be useful to maintain a degree of control and comparability across modules.

The full research paper is available as a PDF.