CONTENTS

Table of Contents
Introduction

1. Descriptive Statistics in Evaluation
2. Subgroup Analysis
3. Variable - Are They Related?
4. Correlation
5. The t-Test of Difference Between Means

Supplements
Resources
Glossary

Wading Through the Data Swamp:
Program Evaluation 201

Module Content Objectives Background Comparing Means Attrition Versus Loss to Followup New Information: Change Scores and Participation How To Construct Scatter Plots How To Interpret Scatter Plots Linear Relationship Strength of the Relationship: Pearson's Correlation Coefficient What Do All the Numbers and Plots Mean? Adding Points to a Scatter Plot Lessons Learned: What Jack Should Have Asked His Evaluator What Did Jack Learn About His Program From These Analyses? How Can Jack Use This Statistical Information to Improve His Program? How Can Jack Use This Statistical Information To Better Promote His Program? Module 4 Quiz

What Did Jack Learn About His Program From These Analyses?

We can infer that the longer the participants stay in the Cool After School Program (or the more sessions of services they attended while in the program), the greater the negative change in alcohol use. It seems reasonable to believe that those kids who stay in the program exhibit a change in alcohol use because they are receiving more services.

But, as you probably have often heard, "correlation is not causation." Two variables can have a high Pearson's correlation coefficient, but we cannot conclude that one variable causes the other one. The fact that participants' alcohol use changes with the number of sessions attended does not mean that the program causes the change.

The observed correlation may be due to another variable that is not observable. For example, most kids also participated in other extracurricular activities during the same time as the Cool After School Program. This unobservable variable could have affected the change in alcohol use.

A correlation between "receiving more services" and "having better outcomes" is often seen in the data from prevention programs. Note, however, that at least some of these relationships stem from participants' "self-selection" into different amounts of program services.

When a program is working for them, people tend to stick around and attend more sessions. Thus, they receive more services. They also tend to have better outcomes.

On the other hand, when people don't like a program's message or style and it is not working, they tend to drop out or miss sessions. Thus, they receive fewer services. They are also more likely to increase their drug use and have worse outcomes.

Because participants have not been assigned to receive different amounts of services, the effects we observe are likely to stem from a mix of the program and personal factors. If, for example, someone can't attend many sessions, the reasons could include more difficult home or living circumstances. These circumstances may also make positive change more difficult. All Jack can really know is that those who attended more also changed more - there is a correlation between services and outcomes.