CONTENTS

Table of Contents
Introduction

1. Descriptive Statistics in Evaluation
2. Subgroup Analysis
3. Variable - Are They Related?
4. Correlation
5. The t-Test of Difference Between Means

Supplements
Resources
Glossary

Wading Through the Data Swamp:
Program Evaluation 201

"Threats" to Research Designs

Validity

Some research designs are vulnerable to more "threats" to validity. Validity means there are few other explanations for the changes you want to attribute to the program. It is important that the observed changes in participants' scores can be confidently attributed to your program or intervention and not to other possible causes.

In the pre-post design, for example, at least some of the change could result from things like trends in drug use in the community or the participants getting older and changing like everyone else their age. Without a comparison group, these "threats" are not well handled by the design.

There are two types of validity:

1. Internal validity is about whether the observed changes can really be attributed to your program or intervention and not to other possible causes. It's anything other than the program that could lead to differences between groups.
   
   These are some of the threats that evaluators have identified:
   
   
   • Selection
   • History
   • Maturation
   • Testing
   • Instrumentation
   • Regression
   • Mortality
     
     
2. External validity is the degree to which the conclusions in your study would hold for other persons in other places at other times. For example, if your evaluation shows that your program prevents drug use, you would hope that others could use it with similar participants and be confident that it would work.

To the extent that you can control for threats to internal and external validity or collect information about how likely they are, your analysis of data on outcome measures will be more likely to lead to valid conclusions about the program's effects.

Source: Trochim, W. The Research Methods Knowledge Base, First Edition. Cincinnati: Atomic Dog Publishing, 1999; and Shadish, W.R.; Cook, T.D.; and Campbell, D.T. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton-Mifflin, 2002.