Introduction

This paper is a modified version of a presentation made at the first AsiaTEFL international conference in Busan, South Korea in November 2003. It is a paper of two parts. The first section is intended to provide an insight into the nature and purpose of evaluation in educational (and other) settings. This section looks at the literature and outlines the various, and perhaps conflicting purposes, behind evaluation. There is also a discussion on who (insiders or outsiders) should carry out such a study. It is hoped that this will be useful to anyone from the large AsiaTEFL constituency who is thinking of carrying out an evaluation study into any aspect of their own specific educational context. The second part of the paper presents the methodology, rationale, context and results of a small-scale evaluation study carried out at the University of the West of England, Bristol, UK. This evaluation focused on a single aspect (namely, feedback) of an initial teacher training course in EFL. This evaluation study is designed to shed light on the appropriateness and effectiveness of the feedback process which takes place following each teaching practice session on this intensive teacher training programme. It is an attempt to bring about evidence-based betterment and will serve as an example of how small-scale studies of this sort can create meaningful insights which can result in appropriate, evidence-based change.

Part One

Background: Evaluation as Construct

It may be tempting to say, since the popularity and prevalence of evaluation is relatively recent, that evaluation exists as a specialised form of testing or assessment. In fact, it is probably more useful to consider evaluation as a largely generic term which we can use to include conventional testing and other forms of assessment such as appraisal, quality assurance, audit and so on. Like testing however, we should always be able to say that evaluation should be seen as the basis for some sort of decision. We produce test results and on the basis of these we (for example) promote, place, rank, diagnose or review in relation to factors such as the testees, the teachers, the materials or other more specific aspects of the learning/teaching context. In the same way when we undertake some form of evaluation we should be in a good position to, for example, recommend the continuation of a project or an approach, make important changes to a set of materials or make suggestions as to how an innovation might become (even) more effective.

Within the educational context we should regard evaluation as a tool which transcends intuition - useful as that may be - to provide a much more "scientific" and rigorous basis for decision-making. Stufflebeam et al (1971: 43) suggest that evaluation, in an educational setting, is the "process of delineating, obtaining and providing useful information for judging decision alternatives." A more pragmatic approach towards a definition of evaluation is taken by Worthen and Sanders (1973) who simply state that evaluation is "the process of deciding on the worth of something". This definition has the virtue of quite succinctly highlighting the importance in evaluation of the ideas of value and judgement. Weir and Roberts (1994) express the view that evaluation is the process of collecting "information systematically in order to indicate the worth or merit of a programme or project (from certain aspects or as a whole) and to inform decision making". All the above definitions indicate quite clearly that, much as one might warm to the simplicity of the idea, evaluation is more than simply trying to find out how things are going and involves - within each unique context - the crucial, and sequential, elements of

•data collection;
•information on value or worth;
•decision-making.

At one level, as Rea-Dickins and Germaine (1992) have pointed out, evaluation is a part of our everyday existence, occurring naturally in a wide variety of domestic and professional situations: the golfer makes a judgement as to the strength of the wind before attempting a stroke, the parent does what he or she can to gauge the temperature of the bath water, and the canny politician comes to a view as to the appropriateness of releasing an especially gloomy set of unemployment figures on a particular day. All these situations involve a process of evaluation. Some writers such as Leviton & Hughes (1981) have pointed to the presence of "conceptual utility" in the evaluation process: this idea describes the opportunity presented by evaluation for greater clarity in relation to stakeholder views or perceptions. In other words the evaluation can offer stakeholders the opportunity to look beyond the anecdotal or the comfort of the accepted wisdom to see more accurately in relation to some aspect of the educational process. And clearly this will, in turn, make for decision-making which is simply better informed.

At another level however, the rise in popularity of evaluation in professional settings such as education, health, transport and so on is, in a sense, a response to the fact that society's systems or cultures do not exist as separate entities but as interconnecting, interdependent and complex structures. At a more micro level however, it is probably true to say that evaluation is at its most successful when it operates within a recognition of evolving systems rather than assuming that there is one static target. (In an interesting way, this "evolutionary" approach mirrors the more recent view (e.g. Dornyei: 2001) that learner motivation is a process-orientated concept in that it can shift over time and is not forever fossilised as instrumental or integrative, but can involve shifting elements of both.)

Those involved in evaluation research in areas such as education hope that the results which they produce can have some level of generalisability "to guide action in other contexts." (Sanderson: 2000). Some level of generalisability is very much in keeping with older, more traditional research approaches but this does not invalidate the evaluation study which takes place in the context of a situation which is not generally replicated elsewhere.

Just as a great deal has been written about the many reasons for administering tests in an educational context, so we can list a number of important reasons for the existence of evaluation. While it is possible to draw up a fairly long list that would include validating changes or innovations, confirming existing practice, substantiating decisions and so on, it may be more useful to make a relatively clear distinction in terms of evaluation that has accountability as its purpose and evaluation that is essentially developmental in nature. This is the distinction made by Weir and Roberts (1994) where they make the point that accountability is essentially concerned with levels of answerability to a broad range of stake-holders such as parents, employers, or even society at large. (By stakeholders I mean those who are either involved in the making of a decision and/or those who are affected by that decision.) In a sense this approach or justification is related increasingly to powerful and relatively modern notions of value for money and is certainly a crucial element in the evaluation of large one-off and publicly funded projects. Sanderson and others have pointed to the idea that in the search for what is popularly termed joined-up government, there is increasing emphasis on "evidence-based policy making" (Sanderson: 2000). In a sense this is no more than an expression of the need to respect some fairly basic principles in relation to value for money and is perhaps a necessary result of the way in which patients or students, for example, now see themselves as clients or customers or consumers.

Evaluation which is developmental - and also probably formative - is designed to bring about well-informed changes to a programme of learning or training. Equally however, an evaluation which is developmental should look to bring about important qualitative changes in learner/teacher/trainer behaviour or knowledge. In this way, an evaluation study may lead to the acquisition of new skills or to the development of existing skills in a new and appropriate way. Developmental evaluation is likely to be supported by "insider" teachers largely because it appears to allow them the opportunity to acquire skills or knowledge that will enable them to do their jobs more efficiently, effectively or appropriately. Put another way, stakeholders such as these may cooperate more fully with evaluation data collection on the grounds that the fundamental rationale is not to criticise or to find fault but to develop or enhance them as responsible professionals.

This latter point invites the question as to who should carry out an evaluation. In acknowledgement of a predictable them/us dichotomy evaluators have been divided into insiders and outsiders. It is not difficult to imagine that outsiders may carry with them the idea of impartiality, but may also have to accept criticism on the grounds of a lack of familiarity with local conditions. They may find it difficult to overcome the surprisingly pejorative term "expert" which often carries an implication of unsuitability and is often modified by the term "so-called". Insiders may be said to have a vested interest but may have a detailed knowledge of history, developments, sequences, chronology, systems and above all, personalities. (In passing it may be important to note that outsiders may also have a vested interest: it is however, likely to be different in nature. Both outsiders and insiders will find themselves working in a complex environment involving power relationships with a wide range of stakeholders.) At the very least there may be a feeling among "insiders" that the expertise of "outsiders" has been acquired from contexts that are to a greater or lesser extent alien or inappropriate. The issue of who is to evaluate cannot be separated from the relevant political situation and a range of other factors that might be difficult to predict and will be as varied as the number of stakeholders operating around a particular context. As Rea-Dickins (1994) has written, "The contributions of different stakeholders (those who either make a decision or who are affected by that decision) raise a range of interesting but complex issues that have to do with notions of "objectivity", "subjectivity", "power-relationships", the relative "value" of outsider and insider information, the "use" of evaluation data and so forth." (p75)

To revisit the previous discussion on definitions of evaluation it may be useful to end this section by looking at Sanderson's view of evaluation which is that it has an "essentially 'instrumental' function in answering the question: how effective are the chosen means in achieving specified ends?" (Sanderson: 2000) While Sanderson tends to apply this definition in the very large contexts of public policy, this is precisely the question which needed to be asked and answered in this particular small-scale study. The course management has taken the view that feedback sessions, conducted in a certain way, are the best means of improving trainee performance in teaching practice. Has the course management team put in place a system which actually meets certain aims, and, if not, what can be done to put in place systems which will meet these desired objectives?

Part Two

The Evaluation Study: Rationale, Aims and Approaches, Participants and Findings

The evaluation context which is at the heart of this particular study concerns the feedback sessions that feature as an on-going and integral part of the Trinity Certificate in TESOL courses at the University of the West of England (UWE). These courses advertise themselves as "initial teacher training courses in EFL" and the vast majority of trainees on these courses have no experience as teachers. Since the most important part of the assessment on intensive courses of this sort is the 6 hours' teaching practice (TP), UWE, like other institutions offering courses at this level, has put in place a system of feedback to trainees following their teaching practice sessions. At one level tutors shared an intuitive view of the value of these sessions and there was some anecdotal and other evidence that they were valued by trainees and did actually improve performance in subsequent TP sessions. The study set out to look more closely and systematically at these feedback sessions in an attempt to understand more about them in terms of location, length, participation, atmosphere and so on and to determine the extent to which they did bring about improvements in performance or the acquisition of new skills and awarenesses. The course as a whole offered trainees the important opportunity to observe "skills in action and to undertake practice with feedback" (Eraut: 1994, 93). Central to this superficially neat and appropriate arrangement is the idea that feedback sessions, chaired by an observing tutor but involving other trainees, are the context in which strengths and weaknesses can be discussed and through which improvements in terms of focus, aims, timing, manner, delivery, presentation and so on can come about. As Imogen Taylor has noted however, "feedback is an area which causes students considerable anxiety" (Taylor: 1997, 65) and there is a need for great care to be taken in relation to issues such as timing, location, participation, attitudes, outcomes and so on.

The study sought to evaluate these feedback sessions in two ways. Firstly, trainees were given a questionnaire in the expectation that more quantitative data would emerge. Secondly, feedback tutors were interviewed and encouraged to provide more qualitative responses to a very similar range of questions to those which appeared in the trainee questionnaire. Ten trainees and seven observing tutors took part in this study, which was carried out in October/November 2001.

Both trainees and tutors were asked to provide information under the following headings:

• feedback location;
•length of sessions;
•participants and contributions;
•time gap between lesson and feedback;
•atmosphere;
•dealing with problems;
•extent of influence on future TP sessions;
•content;
•points for next time;
•attitude of participants;
•improvements.

These areas/topics were selected because they had proved to be the most prominent among those issues and causes for concern or satisfaction as expressed by trainees in course evaluation questionnaires over a period of 5 years.

Clearly location is an important issue in that given the sensitivity of the occasion some level of privacy is desirable. Most trainees seemed to believe that the correct location for feedback is the room in which the observed lesson took place. This location was believed by trainees to be suitable because it was generally a place in which immediate feedback could be given, it was helpful in establishing a sense of focus and because it was generally preferable to the lack of privacy offered by the refectory or the staff room or some more public (or semi-public) place.

Tutors largely agreed with this in that they felt that using the room in which the lesson had taken place provided a good opportunity to deal with issues such as layout and made pertinent issues much easier to recall. Some tutors felt that a change of scene would be beneficial to the feedback process and "detach them (the trainees) from the emotion" although there was also a feeling that this might cause some break in continuity. There was a feeling among tutors that staff offices were unsuitable as they might emphasise an unhealthy staff/student divide and might encourage tutors to wield an inappropriate authority over their trainees.

In relation to the length of these sessions trainees indicated that they had experienced session lengths ranging from 5 minutes to well over an hour. While trainees seemed to indicate that 30 minutes would be ideal, there was a feeling that a significantly shorter time might suffice as long as all the relevant points were dealt with. Trainees felt that the length of the feedback session should be dependent on the number and range of issues to be dealt with.

Tutors also felt that the length of these sessions was dependent on the nature of the issues raised and on a range of personal factors relating to both trainee and tutor preference. Equally, tutors felt that the overall length of these sessions would be dependent on the number of participants and on the extent of peer participation.

When asked about the length of the gap between the TP session and the feedback session, trainees indicated that they had experienced a number of situations ranging from immediate feedback to sessions which took place the following day. Tutors in most cases felt that next day feedback was not a good thing, although some could see the benefits of sensible reflection and of the need for a sizeable gap in order to allow for a more positive or realistic view to emerge. Immediate, or near-immediate, feedback was supported by most tutors. If there had to be a short gap - some tutors felt - this could be filled by the completion of a self-evaluation form which might formalise the trainee's view of the session. There was a feeling among some tutors that training in self-reflection would be necessary and a view that adding extra "gap-filling" tasks would simply lengthen the feedback process for no obvious gain. While some recovery time was felt to be useful, there was little support for delaying the feedback for too long: fairly quick feedback was felt to be important in meeting the trainee development aspects of the process. And equally, trainees were anxious to receive some judgement on their performance. Some tutors emphasised the personal nature of this issue, explaining that instant feedback suited some while some sort of delayed feedback would suit others. One tutor mentioned the possibility of a second optional feedback session at which more mature and reflective considerations could surface. This might be especially important where grades or marks were involved.

In relation to who should attend these sessions, it was the experience of most trainees that feedback could be given with only a tutor and one trainee present or with a tutor and all those who had participated in a particular 2- or 3- hour TP session, that is all the members of a particular teaching practice sub-group. In relation to this variable trainees were not asked for their preference but to describe the situations they had experienced on the course.

The tutors felt that all the trainees in a particular sub-group should be present. There was a feeling that participation in group feedback was all part of the group learning experience. Some tutors felt that a focused observation task would be useful and a discussion of this could feature as a way of presenting different perspectives on the TP experience. Tutors articulated the idea that group feedback could be unnerving for some trainees, especially on those occasions when there were difficult or sensitive issues to resolve. One tutor felt that while group feedback was preferable, it was actually quite rare for trainees to offer useful commentary on the work of other course participants. Most tutors seemed to feel that trainees actually learned a lot from watching others in TP and from being subsequently given the opportunity to comment during the feedback sessions. A 1-1 approach may be better on the occasions where TP has not gone well. Perhaps, one tutor felt, the configuration should be optional and determined by either the trainee or the tutor.

Both trainees and tutors were asked about the desirability of the language learners themselves taking part in the feedback sessions. The overwhelming view of the trainees was that they should not, as it could be stressful and uncomfortable. Some felt that the learners might not take these occasions seriously or that they might find it difficult to be appropriately critical. One trainee stressed that it was important to receive feedback from them but not with them! Another felt that steps could be taken through a questionnaire to gather information from the learners as to how effective the trainees' performance had been.

The tutors felt that learners might lack genuinely useful insights and that their comments might have a damaging effect on trainee morale and might undermine delicate self-confidence. Learners could be asked for their views in some other forum or through some other means such as through a questionnaire. There was a feeling that the role of the language learners could grow as the confidence of the trainees grew. One tutor felt that it was not obvious as to how the learners could make a valuable contribution, but it may be useful to determine whether there was a meaningful match between tutor views of performance and those of the learners.

The questionnaire and the interviews also sought to generate views on the desirability of encouraging contributions from the other trainees being assessed and attending the feedback sessions as a part of a TP sub-group. Trainees seemed to have a very positive view of this idea, saying that such participation encouraged openness and supportiveness, and offered an opportunity to discuss things in much more detail and from a different perspective. Some trainees warned of the need for comments from their peers to be useful and constructive and of the advisability of not forcing trainees into comments they might not wish to make. This seemed to point to the idea that contributions - their level and nature - need to be handled in a very sensitive way. While all tutors reported on the value of trainee contributions, some felt that the exact configuration in relation to attendance - and therefore participation - should be flexible and should respond to factors such as trainee preference, stage of the course, how successful the TP session had been and so on. One trainee went so far as to suggest that the best feedback that he/she had received on the course had come from a fellow trainee.

In responding to a question in relation to the atmosphere in these feedback sessions, the trainees overwhelmingly felt that the atmosphere had been positive. They expressed this in a number of ways using words such as good, honest, forthright, encouraging, supportive, constructive, focussed and so on. There was a feeling that these sessions could be emotional or that some level of mixed feelings was present.

Most tutors expressed the view that at most times the atmosphere was positive and productive. Tutors felt that there was a need for them to work at creating this sort of climate by stressing the positive elements in the lesson and by inviting comments from the trainee at an early stage. Some tutors felt that the atmosphere could be influenced by the unequal relationship between the trainee and the tutor and that there was a need to hand down some of the elements of power on these occasions. Many tutors felt that these sessions needed to be constructive, more developmental and less judgmental. One tutor felt that trainee nervousness was not really a factor on these occasions as the trainees really knew how they had performed. There was a feeling that the atmosphere was created by the dynamics of the sub-group and by overall levels of confidence. Atmosphere often depended on the trainee's ability to accept criticism and some trainees may not fully understand the nature of the occasion/process.

A positive atmosphere was more likely to be created where tutors took the time to explain the rationale behind these occasions and when all participants understood the nature of their roles and responsibilities. With strong groups the atmosphere tended to be positive and influential. One tutor felt that it might be useful to record (video or audio) these feedback sessions so that trainees could replay the tutor (and other) comments at some later stage. One tutor felt that it was important to tackle problems directly although there was something to be said for dealing with the positive first followed by the more problematic. There was a feeling among tutors that poor TP performance and therefore a low grade or mark, often led to a negative atmosphere.

The trainees were specifically asked to say something about how problems were dealt with in these sessions. There was an overwhelming feeling that problems had been dealt with in a fair, sensitive, supportive and tactful way, all of which points to the problems which will result if the tutor is not able to manage these occasions appropriately.

On the important topic of the extent to which these feedback sessions led to improved TP performance, the trainees' responses were somewhat more mixed. This is in fact a crucial question since creating enhanced TP performance is the principle rationale behind these sessions. Most felt that some improvement had taken place although some trainees expressed this in rather reserved terms. While there was a feeling that the feedback sessions resulted in improved focus and confidence, there was also a feeling that the feedback merely reiterated what the trainee already knew. One trainee observed that any form of feedback would be helpful.

The tutors felt that where the trainees were sensitive and responsive to tutor comments it was more likely to result in improved performance. This pointed to the important element of trainee attitude. Where trainees understood the nature of the feedback process, the impact was likely to be at its greatest. There was a feeling among some tutors that some trainees simply did not act on their comments and simply did whatever was required to secure a pass. Many trainees do however, genuinely want to improve and this is likely to occur where the trainees see feedback as an integral part of the course as a whole and therefore an important part of the learning process.

Crucial in this area of improvement is the points for next time: this is an attempt by the system (through the observing tutors) to formalise a short series of points and suggestions which should be addressed for the next teaching practice session. These may be expressed in terms of the need for more correction, or more effective monitoring during group work, or more attention requiring to be paid to learner responses, and so on. Without exception the trainees felt that this was an extremely important part of the feedback process. One trainee saw this as "vital to improvement" and an "essential summary of progress". While in general the tutors also saw the value of these specific and recorded comments, some felt that certain trainees may not be able to place these comments in context and may therefore not be able to act on them, especially when under pressure of time.

One important element of these sessions is to do with the allocation of a grade or mark, which will in time contribute towards a final teaching practice mark for the course. On the UWE course the teaching practice mark is worth 50% of the total of marks awarded for the course as a whole. Both trainees and tutors were asked if they felt that the feedback session should confine itself to issues concerning pedagogic practice and thus avoid dealing with a mark for the session. The majority of trainees felt that both the pedagogical issues and the mark should be dealt with since it is useful to have a clear idea of the standard of performance as the course progressed. The view among trainees seemed to be that since both marks and classroom issues are important, both should be dealt with. In group feedback sessions some trainees felt that it was very beneficial to see how different lessons were treated in assessment terms. Some trainees pointed to the need for sensitivity and to the need, occasionally, to give grades privately and at a later stage.

Tutors also acknowledged that this was a difficult area and that certain trainees became "grade-obsessed". They also pointed to the fact that there was a lack of understanding of the level of achievement that was implied by the award of certain grades. It was important, tutors felt, to have a transparent and simple grading system: it was also felt that trainees who were doing well, and making strong progress towards a personal target, began to be more relaxed in terms of grades. Issues to do with grading, some tutors felt, were particularly important for borderline cases. There was a feeling among some tutors that grading had the effect of complicating their task during the feedback sessions. They understood, of course, the relationship between these two elements, but felt that some delay in the giving of grades might simplify things and help with issues to do with focus. Some tutors felt that to some extent the way in which pedagogical issues and grades were dealt with might depend on the particular stage of the course at which the feedback took place.

In describing the attitude adopted by the tutor at these sessions the trainees overwhelmingly used positive expressions such as supportive, friendly, encouraging and so on. Their description of their own attitude during these sessions is a little "mixed", however where words such as positive, open-minded, receptive were balanced by ideas such as addled, nervous, tired, and humble.

In the crucial area of how these sessions could be improved the trainees felt that improvement might be created by:

• having a specific location for the feedback to take place in;
•waiting an hour before giving feedback;
•doing feedback on a 1-1 basis, so as to avoid the temptation to spare the trainee's feelings;
•basing feedback on personal written evaluations completed by the trainee.

As far as the tutors were concerned these sessions could be improved by:

•videoing the feedback sessions;
•giving more time to feedback;
•involving the trainees much more in the process;
•ensuring that all members of the particular sub-group were present;
•encouraging self-evaluation as much as possible;
•holding whole-of-cohort feedback sessions to deal with general issues arising from TP;
•having a more standardised approach from tutors.

Summary of Findings

In general there seemed to be a high level of satisfaction on the part of trainees with these feedback sessions. Their suggestions for improvement are not especially radical or threatening and this may be in part due to the means through which the data was obtained, namely a questionnaire. Had they been given more opportunity to talk more open-endedly about their experiences and views, they might have provided more by way of recommendations. The tutors were given this opportunity in their interviews and their responses are in many ways indicative of their greater experience in this context and of their awareness of the breadth of areas in which changes could be created.

In terms of the extent to which the data that has been collected in this study can become involved in the decision-making process, there is no doubt that the positive tutor/trainee attitudes to these feedback sessions represent a good foundation upon which to create meaningful structural changes. Running through much of the comments and responses of both tutors and trainees is the recurring idea that feedback of this sort requires careful handling, especially if it exists in a group context where trainees are required to both give and receive feedback. To end on another quotation from Taylor, she points to the fact that peer feedback on short, professional courses can cause great concern, a concern that is "in part related to fears…about the expression of difference. Students are unsure what feedback to give, and how to give it in a way that is acceptable." (Taylor: 1997) Tutors may be more skilled in handling the idea of "difference" and it is clear from this modest study that they are, in this particular context, willing to create conditions in which that sense of celebrated difference can flourish within a feedback context which is fair and responsive and which does in fact create meaningful improvements in performance.

Bibliography

Dornyei, Z. (2001) Motivational Strategies in the Language Classroom. Cambridge: CUP
Eraut, M. (1994) Developing Professional Knowledge and Competence. London: Falmer Press.
Leviton, L.C. & E. F. X. Hughes (1981) Research on the Utilisation of Evaluations: A Review and Synthethis. Evaluation Review, 5: 525 - 48.
Rea-Dickins, P. (1992) Programme evaluation as a management tool for both accountability and development. ELT Management, Newsletter of the Management Special Interest Group of IATEFL, 12, 2-6.
Rea-Dickins, P. & K. Germaine (1992) Evaluation. Oxford: OUP.
Rea-Dickins, P. (1994) Evaluation and English Language Teaching. Language Teaching, Vol.27/2:71-91
Sanderson, I. (2000) Evaluation in Complex Policy Systems. Evaluation, October 2000.
Stufflebeam, D.L. et al (1971) Educational Evaluation and decision-making. Ithaca, Illinois: Peacock.
Taylor, I. (1997) Developing Learning in Professional Education. Buckingham: SRHE.
Weir, C & J. Roberts (1994) Evaluation in ELT. Oxford: Blackwell.
Worthen. B.R. & J.R. Sanders (1973) Educational evaluation: Theory and practice. Belmont CA: Wadsworth.