CONTENTS

Table of Contents
Introduction

1. Evaluation and the Program Planning Process
2. Data Analysis, or "What Do You Mean by That?"
3. Special Challenges in Evaluating Prevention Programs

Supplements
Resources

Evaluation for the Unevaluated:
Program Evaluation 102

Challenges in Evaluating Prevention Programs

Given the difficulties in evaluating prevention programs, a carefully planned, science-based evaluation is essential. Even when working with the best team and the best framework possible, note the following challenges:

Timeframe

It takes a long time to determine long-term outcomes of prevention. You can measure the short-term effects of a program, but the real value to the community is the slow change in attitudes toward, and use of, substances. It can take years for norms to change. For example, it took more than a decade for Mothers Against Drunk Driving to help change our norms about drinking and driving.

Measurement

The effectiveness of prevention services can be hard to measure if they are provided to a whole community rather than individuals. As you move through the intermediate and advanced modules, note the pitfalls of evaluating Jack's 50-person program. Then think about how you would measure change across an entire community! Just keeping track of everyone could pose enormous challenges.

Results

You are attempting to prove that something "didn't" happen. Our colleagues in the treatment field have it a little easier, at least in the evaluation department. Once someone has started using drugs, it is relatively straightforward to measure what interventions are effective in helping the person stop.

It's harder to measure substance abuse that might have happened but didn't. Is it your program, individual characteristics, or some combination of the two? One option is to use multiple measures related to your program. For example, Jack could measure changes in truancy and grades. These could bolster his case that Cool After School helps reduce risky behaviors.

Statistical Significance

This can be difficult for small prevention programs to prove. "Statistical" significance requires either a large number of participants in your study or a very large difference between groups in your study.

While statistical significance is required by some grant programs and peer-reviewed professional journals, you may not be able to demonstrate statistical significance. This doesn't mean that your prevention program isn't working. Nor does it mean that an evaluation won't give you practical information to use in improving your program. It just means that you don't have the data to meet this rigorous test.

An alternative to rigorous statistical measures is to use anecdotal reports and case studies. These can be valuable for small pilot programs with too few participants to yield meaningful statistical conclusions. You can still demonstrate program success and give people ideas for implementing your program, perhaps with more participants.

Accountability

You can ensure accountability by documenting your efforts and tying them to results. Documentation may include quarterly and biannual progress reports, annual progress reports, a final programmatic report, and program monitoring visits.

Evaluation, as we use it here, is designed to help you improve your program, not solely to prove that it works. Remember that evaluation is a circular process. You always have opportunities to improve and enhance your program.

Competition

Believe it or not, you face some stiff competition when you set out to provide social services, especially in the prevention field. Many organizations are looking to support programs that help children and communities, and substance abuse is a societal ill everyone would love to abolish.

Prevention programs can be school based, hospital based, or neighborhood based run by for-profit or nonprofit organizations. This is all the more reason to demonstrate to your community that your program is well designed and effective. The scarce dollars can and will go elsewhere!