Pathways Courses - Evaluation for the Unevaluated:

Evaluation for the Unevaluated:
Program Evaluation 102

Module 1: Evaluation and the Program Planning Process - Page 6 of 13

Custom Data Collection

Custom data collection allows you to gather exactly the data you need. You can design your own instrument or use standard instruments validated for use in the prevention field. There are many examples of standard instruments link to Eval TA tables of instruments.

Using existing instruments has a couple advantages. They often are standardized and they usually have been established as valid and reliable.

Disadvantages are that some instruments may not be appropriate for all cultural or ethnic groups and some instruments may not be useful for your program. If an outside consultant suggests an instrument, make sure that you and the evaluation team review each item to make sure the information it asks for is consistent with your expectations and goals.

Custom data collection typically occurs through surveys, in-person interviews, or focus group surveys, which collect self-reported data. Respondents report their opinions on paper, over the phone, or in person to a trained interviewer taking notes.

PROS	CONS
Fairly cheap (not as cheap as digging through files but still reasonable). This method also allows you to choose the way the questions are asked.	If you are studying substance use and aren't able to verify self-reports (subjective measures) with objective measures (such as lab work), some people will lie. For example, kids may underreport use for fear of being caught. Other times, they'll overreport use for fear of being labeled a "geek." Another problem with custom data collection is interviewer bias. Training helps avoid this problem). In telephone interviews, the presence of other family members may bias the answers (or reduce the responses to whispers or unintelligible grunting in front of the person's parent or spouse). It will be important to test any instruments you develop and make needed changes before finalizing them for use in your evaluation.

PROS

CONS

Fairly cheap (not as cheap as digging through files but still reasonable). This method also allows you to choose the way the questions are asked.

If you are studying substance use and aren't able to verify self-reports (subjective measures) with objective measures (such as lab work), some people will lie.

For example, kids may underreport use for fear of being caught. Other times, they'll overreport use for fear of being labeled a "geek."

Another problem with custom data collection is interviewer bias. Training helps avoid this problem).

In telephone interviews, the presence of other family members may bias the answers (or reduce the responses to whispers or unintelligible grunting in front of the person's parent or spouse).

It will be important to test any instruments you develop and make needed changes before finalizing them for use in your evaluation.

Evaluation for the Unevaluated: Program Evaluation 102

Custom Data Collection

Evaluation for the Unevaluated:
Program Evaluation 102