I. Introduction
This paper aims to introduce an evaluation measure called, The Annual Activities Survey for Language Teachers (AASLT). The AASLT is a survey designed for administrators to deal with a language teacher's weaknesses in the area of professional development. In order to provide the relevant background knowledge and an explanation as to why the AASLT was created, some current concerns found in the area of professional development research will be explored. This will be followed by a description of the AASLT. Finally, an examination of the cognitive pre-testing the AASLT was subjected to will be discussed to reveal how its use and validity have been ameliorated.

II. Concerns with professional development
Professional development in language teaching is often misperceived as a one-time activity that is not continuous, and not subject to re-evaluation. And yet, while a teacher may see professional development and non-teaching related activities as either just an addition to one's work load, an initiative from above (Stir, 2006) or feel there is little support for it from administration, there does appear to be an awareness among teachers of their lack of professional development undertaken and weaknesses in it. However, in the language teaching profession today, there is little in the sense of motivation and action taken to improve this overall situation. The reasons for this are often inter-related and, in many cases, compounded by one another. Consequently, in order to fully understand the current concerns with professional development in language teaching today, it is important to look at what some of the research has shown.

To begin with, it is important to recognize that concerns with professional development first surface during one's initial teacher training period. Based on a study done by Stir (2006), it was reported that 90% of student teachers during their training indicated that they want to become more aware of their practice. Yet, comparatively, the same percentage of these student teachers takes no action. Moreover, as such teachers progress past the training stage and into the teaching profession, the lack of effort put into professional development continues and is often inevitably 'justified' by positive student opinions (Marsh, 1984: Kulik, 2001). This, along with the failure of teachers to consider the social contexts they work in, is often not accounted for (Stir, 2006). What is more, the unwillingness of tertiary institutions and universities to respond to the challenges of sustainability and the rate at which professional development needs to occur, suggests that they themselves have only succeeded in compounding this ongoing dilemma (Cortese, 1998).
Accountability for professional development in the workplace also appears as a concern. Confusion over who is actually responsible for doing it, the teachers or the administration, can often lead to its failure to materialize. To add, when there is no teacher control in professional development, a feeling of stress and dissatisfaction can settle in, which can lead to overall disinterest in it.

Yet, in contrast, when professional development is perceived as optional, there is little to no follow up on it. In some related cases, teachers can even develop a fake sense of competence completion, believing they know enough and have reached their pinnacle of expertise. (Williams & Berry, 1999) This, in itself, can have a profound effect on teachers, as over time, they can mistakenly come to see their colleagues as professionally equal in every aspect when, in truth, they are not. What this suggests is most concerning. Teachers are potentially prone to manifest an imagined level of professionalism that can lead to a marked plateau. They come to believe they have reached ‘expertise’ and consequently, become content with that.

This confusion between professionalism (i.e. current expertise) and professional development (i.e. the ongoing development of expertise) (Gleeson, Davies, & Wheeler, 2005) is inevitably responsible for the misnomer that a community of professional practice exists (Lave & Wenger, 1991). It is hard to refute this when professional development is often not included in aspects of hiring practices (Gleeson, Davies, & Wheeler, 2005). Current trends would indicate the presence of flexible and casual practices in professional recruitment, which can, in the long run, hinder opportunities for professional development (Gleeson, Davies, & Wheeler, 2005). This would seem to only negate the process of developing professionally in the minds of teachers. Other similar limitations, like those relating to contracts and employment duration, can also adversely affect teachers since there are little long-term benefits for them. Consequently, with such increasing internal instability in the workplace, there appears an even greater need to continuously re-evaluate professional development in language teaching.

III. The AASLT: What is it?
The AASLT is a measure specifically designed to resolve inadequacies in professional development evaluation. It is not a measure used to evaluate classroom teaching. Instead, the AASLT functions to evaluate the non-teaching related activities of an individual teacher, particularly those related to professional development. What is more, it serves to specifically diagnose, address and raise awareness of a language teacher’s known and unknown weaknesses in professional development. In so doing, it is believed to provide the motivation necessary for a teacher to improve upon these.

IV. The cognitive pre-testing of the AASLT
The value of a survey depends on whether the respondent interprets the meaning of the items as the survey designers intended. This requires the survey designers to agree upon a standardized referential and connotative meaning for each of the items. The standardization of items assumes that respondents are able to understand the items being asked, are understood in the same way by all respondents, and that the respondents are willing and able to answer them (Collins, 2003). However, standardization of data measurement tools does not remove all errors. While piloting a survey will sometimes reveal overt problems it will not provide evidence of causes (Collins, 2003). Pre-testing items enables the survey designers to ensure that the meaning the respondents attribute to the item is consistent with the one that the designers intended.

Various cognitive methods have been applied to the testing of surveys. These include cognitive interviewing, behaviour coding, respondent debriefing, and expert review, among others. Several papers have investigated how these methods compared to determine if they varied in anyway in terms of reliability and validity (Presser & Blair, 1994;Willis, Schechter, & Whitaker, 1999). The study by Willis, Schechter, and Whitaker (1999) found that different pre-testing techniques appeared to show a reasonable degree of consistency and that cognitive interviews conducted by different interviewers with varying methods and levels of experience, still revealed similar results. Cognitive interviewing is one diagnostic tool for pre-testing surveys, which has been used for identifying problems at an early stage of development (Hughes, 2003). Two main cognitive interviewing techniques are think-aloud and probing. In the think-aloud approach, the respondent is asked to verbalize his/her thoughts as one proceeds through the 4 stages of item comprehension, retrieval, judgment, and response. Cognitive interviews can be either concurrent or retrospective.

In concurrent interviews, respondents describe their thoughts while answering the item. In retrospective interviews, after the respondent completes the interview, he/she is asked about the process used to generate his/her answers. The concurrent think aloud method has the advantage of capturing the information while it is fresh in the respondent’s mind. Probes are used to explore comprehension, retrieval, judgment and response processes (Willis, 1994). Probing involves the interviewer asking specific questions to elicit how the respondent went about answering the item. In addition to being either concurrent or retrospective, probes can be pre-scripted before hand and agreed upon by the interviewers, or unscripted where the individual interviewer creates them during the interview as needed. The Think aloud method tends to work better for self-completion surveys than it does for face-to-face interviews, however, both methods can be combined effectively (Collins, 2003; Willis, 1994). In the cognitive interviews conducted for this research, the concurrent think-aloud method was used with unscripted concurrent probes.

The analysis of interviews has been criticized for being too subjective (Drennan, 2003). Therefore, in order to address the issue of the validity of data obtained from these interviews, a taxonomy of problem classification can be used (Drennan, 2003). This taxonomy allows for the creation of a problem-coding scheme to standardize the process of interview analysis so to increase its objectivity. The item-and-answer model derived from cognitive psychology is a commonly cited representation of how respondents answer survey items. To answer items, they must comprehend, retrieve the necessary information from long-term memory, make a judgment about the information needed to answer the item, and respond to the item (Tourangeau, 1984). Comprehension is conclusive to the respondent understanding each item in the same way the researcher intended. After this, the respondent must retrieve relevant information from long-term memory. Judgment is the process by which respondents formulate their answers to a survey item. There are two parts to the response: formatting and editing it. Formatting is fitting the response into one of the answers offered and editing is the act of revising the answer before it is communicated because the respondent may want to protect him/herself. A coding scheme was used based on this item-and-answer model, which was adapted by Hughes (2003) from the original developed by Presser and Blair (1994).

There were a total of five one-on-one interviews completed in a work setting. The respondents were all full time language teachers at a university in Japan. Audio recordings were made of the interviews, for review later on. Upon completion of the cognitive interviews, the coding results for each survey item were summarized in a chart. Briefly, we found that out of a total of 23 items, there were 12 P1, 0 P2, 5 P3, and 3 P4 problems. A review of each problem took place in the form of a discussion between the authors. Following the discussion a decision was made to determine the actions required for each of the items. We used the following list of possible actions taken from Bowden et al. (2002).  

1. acceptance of the original question and meaning
2. acceptance of the original question with change in meaning
3. change question (slightly) but keep the meaning the same
4. drop the question and write a new question (and new intended meaning)

V. cognitive pre-testing results & example changes
The AASLT was reduced from a 23 item survey to a 19 item one through the use of cognitive pre-testing (n = 5). Following the pre-testing, P1 was found to be the most common problem with 12 occurrences. This was followed by P3 with 5, and P4 with 3. No P2 problems were encountered during this round of pre-testing.

An example of a P1 problem occurred with such items as18 and 19:

18. I made specific positive changes to the general running of the department outside committee responsibilities.
19. I made a direct contribution to the general running of the department outside of my committee responsibilities.

Some respondents could not comprehend what was meant by “specific positive changes” and “a direct contribution”, whereas others saw them as having the same meaning. As a result, these were both dropped and replaced with:

16. I belonged to the________________________ committee at my institution this year. (Name of Committee)        Committee Position: ______________________

An example of a P3 problem occurred with such items as 22:

22. I am currently an academic advisor to ______ students. Please list the student name, year, and major.

The term academic advisor was interpreted differently. Some respondents interpreted it as meaning supervising graduate students (intended meaning). However, others interpreted it as only giving general educational advise to any student that requests it (unintended meaning). Consequently, the question was rephrased and simplified, but without a change in meaning:
18. I was an official academic advisor to _______ graduate/undergraduate student(s) this year.
An example of a P4 problem occurred with such items as 11.

11. I am a member of ______ academic society or societies. Please provide the society name(s) and date(s) you joined.   

Name: _______________ Date joined:__________________

Most respondents could not remember the date joined, especially if it was more than 1 or 2 years ago. Therefore, the question was slightly altered without a change in meaning:

10. I became a member of _____ academic society or societies this year.
 
Please provide the society name(s) you joined: __________________________________

VI. Concluding remarks
In this presentation, an evaluation measure, The Annual Activities Survey for Language Teachers (AASLT), was initially introduced to explain how it could be used by administrators to diagnose and address problematic circumstances in professional development through the assessment of specific professional development activities of one’s teaching staff. This was necessary in order to bring the fundamental focus; the cognitive pre-testing that the AASLT, into perspective and to reveal how its use and validity have been improved upon. As the next step in the development of the AASLT, it will now be necessary to engage in a second round of pre-testing, but with a larger number of respondents. Upon completion of this, the results will be scrutinized against the results of the first round of pre-testing to validate the changes that were made. This will be done and paralleled with the continued collection of feedback from presentations and workshops as the means to possible future revisions.

References
Bowden, A., Fox-Rushby, J. A., Nyandieka, L., & Wanjau, J. (2002). Methods for pre-testing and piloting survey questions: Illustrations from the KENQOL survey of health-related quality of life. Health policy and planning, 17(3), 322-330.

Collins, C. (2003). Pretesting survey instruments: An overview of cognitive methods. Quality of Life Research, 12, 229-238.

Cortese, A. (1998). Afterword, academic planning in college and university environmental program. The Proceedings of the 1998 Sanibel Symposium. Florida: North American Association for Environmental Education.

Drennan, J. (2003). Cognitive interviewing: Verbal data in the design and pretesting of surveys. Journal of Advance Nursing, 42(1), 57-63.

Gleeson, D., Davies, J., & Wheeler, E. (2005). On the making and taking of professionalism in the further education workplace. British Journal of Sociology of Education, 26(4), 445-460.

Hughes, K. A. (2003). Comparing pre-testing methods: Cognitive interviews, respondent debriefing, and behavior coding. Paper presented at the meeting of the Annual Meeting of the Federal Committee on Statistical Methodology. Arlington, VA.

Kulik, J. A. (2001). Student ratings: Validity, utility and controversy, New Directions for Institutional Research, 27(4), 9-25.

Lave, J. & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press.

Marsh, H.W. (1984). Students, evaluations of university teaching: Dimensionality, reliability, validity, potential biases and utility, Journal of Educational Psychology, 76(5), 707-754.

Presser, S., & Blair, J. (1994). Survey pretesting: Do different methods produce different results? In P.V. Marsden (Ed.), Sociological methodology (pp. 73-104). Beverly Hills, CA: Sage.

Stir, J. (2006). Restructuring teacher education for sustainability: Student involvement through a “strengths model”, Journal of Cleaner Production, 14, 830-836.

Tourangeau, R. (1984). Cognitive sciences and survey methods. In T. Jabine & M. Straf & J. Tanur & R. Tourangeau (Eds.), Cognitive aspects of survey methodology: Building a bridge between the disciplines (pp. 73-100). Washington, DC: National Academy Press.

Williams P.L. & Berry, J.E. (1999). What is competence? A new model for diagnostic radiographers: Part 1, Radiography 5, 221-235.

Willis, G. B. (1994). Cognitive interviewing and survey design: A training manual. Hyattsville, MD: National Centre for Health Statistics.

Willis, G., Schechter, S., & Whitaker, K. (1999). A comparison of cognitive interviewing, expert review, and behavior coding: What do they tell us? Proceedings of the American Statistical Association (pp. 28-37). Survey Research Methods Section.