Many programs develop their own instruments. This can be problematic for several reasons:

• Validity and reliability have not been established.
• Much effort has gone into the construction, development, and validation of a wide range of standardized measures used to evaluate prevention programs. It is likely that your own efforts would be duplicating these efforts and you'd be reinventing the wheel.
• An "expert" has probably done a better job of developing a measure than you as a director or evaluator for a prevention project can do in the time available.
• If you decide to construct your own scale, you may find it difficult to write a set of questions that are clear to respondents, tap different aspects of important attitudes, beliefs, and behaviors, and "hang together" well enough to make a good, reliable, scale. You will probably make some errors in your first attempts and will need to rewrite or drop some items.
• Even after you think you've developed a good measure, you should try to conduct studies of the measure's reliability and validity. You'll need to look at whether it seems to have enough room for change in the groups of participants you propose to use it with and whether it is culturally sensitive in the settings where you plan to use it.
• Developing and assessing your new scale may be very time consuming and you may not have the resources to support the effort.
• If you use existing measures, it is easier to compare your results to other evaluations that have used the same measure.
• Critics will be able to point to the ad hoc nature of your measures and question their reliability and validity--unless you are able to publish studies of the measures' properties before you need to present the results of program evaluations that use the measures.

Remember that a pilot test is the stage at which you use your instruments (surveys, record review forms, personal interview forms, etc.) in as "real" a setting as possible.

The pilot test can tell you:

• If consent forms or letters about the evaluation can be easily delivered
• How long the instruments take to administer
• If the subjects understand the questions
• If any records you need are readily available
• If you can collect needed information in the established timeframe
• If the instruments give you the information you want

Tips for Conducting a Pilot Test

1. Always ask for feedback from the subjects in your pilot test. They will tell you what worked for them and what didn't. You should instruct them to take notes and make comments on the process of using the instruments. These notes and comments can be used to determine any needed changes. Jack's lucky. Middle school kids are especially willing to inform the adults where they "messed up"!
2. Record the time it takes to complete each item. It may take only 10 minutes to complete a survey, but if half the time is spent on one question, it needs to be rewritten.
3. Talk through the instrument. Some respondents feel comfortable talking out loud while answering. When they do, note their comments.
4. Record questions asked and clarifications made. When the interviewee asks a question, record key words or verbatim text as well as your own response next to the relevant item. These comments will help in rewriting items.
5. Note nonverbal behavior. Record any nonverbal behavior and body language that coincide with particular questions. Hesitance in responding, foot tapping, fidgeting, and similar behaviors may indicate item-design faults, question difficulty, or lack of relevance.
6. Note whether instructions and format were easy to follow. Question instructions and format vary from item to item. Notice how smoothly and quickly the interviewee reads directions and moves from one item to another.
7. Note erasures, incomplete items, errors, and inconsistencies. These types of responses may indicate questionnaire design flaws.