CONTENTS

Table of Contents
Introduction

1. Introduction to Evaluation
2. What Does an Evaluation Project Look Like
3. Understanding Types of Evaluation Projects
4. Preparing for Your Evaluation

Supplements
Resources

Evaluation for the Unevaluated:
Program Evaluation 101

Module Content Objectives Choosing an Evaluator/Evaluation Team Where To Find Outside Help With Your Evaluation The Evaluation Team Selecting an Evaluator Contracts With Outside Evaluators Planning the Evaluation 1. Clarify your program's mission statement, goals, and objectives. 2. Choose the evaluation design. 3. Draft evaluation questions. Module 4 Quiz Next Step

2. Choose the evaluation design.

While the evaluator would be the expert in this area, Jack and his crew did some reading to make sure they could contribute to this decision in an intelligent fashion. They learned that there are four types of designs typically used in evaluating prevention programs:

Pretest/Posttest

In plain English, this means asking the same kids the same questions before they've participated in Cool After School (pretest) and after they've completed the program at the end of the school year (posttest). Although often used, this method doesn't take into account other factors that may contribute to changes in your kids during the year, such as maturity, changes in home life, health problems, and changes in teachers. Although this design won't show exactly what your program did, it will tell you that you aren't doing something "wrong." This design will show you that your program is contributing to change in the right direction. For example, if substance use increased after kids participated in Cool After School, Jack would know he needed to make changes.

Experimental Designs

These involve the use of a comparison group. They can help strengthen the pretest/posttest method. Groups are created by assigning kids at random (for example, through a coin toss) to a participant group (such as Cool After School) or to a comparison group (such as "no after school program").

This method allows you to observe differences between groups before and after participating in your program. It is based on the assumption that all other life factors are "equal" for both groups, and the only thing different in participants' lives is the prevention program.

Quasi-Experimental Designs

These are used when random assignment is not possible, as is often the case when working with human subjects. These include comparison group designs and time-series designs. The usual time series design is simply the collection of quantitative observations at regular intervals through repeated surveys.

Posttest-Only Designs

These are the weakest type of evaluation but still can be better than no information at all. If you aren't able to give pretests, don't have enough participants for comparison and control group s, or can only test participants one time, this method may be used. For example, you simply measure attitudes and substance use following completion of your program.

With this design, you can compare your results to local and national data. This will give you some sense of where your kids are compared to other kids in their age group.

Armed with the knowledge of the different types of evaluation possible, Jack's group was ready to meet with the evaluator. They could discuss:

• Where to find a comparison group in their community
• How the groups would be selected
• When pretest/posttest measures could be taken during the school year

After talking with their evaluator, Jack and his staff determined that a quasi-experimental design would work best. Because youth behavior varies so much, they needed a comparison group. Randomly assigning students to the program wasn't possible, because Jack had already made a commitment to a particular school. Still, they could find 50 kids in the same school district who were similar to the Cool After School kids.