Pathways Courses - Wading Through the Data Swamp:

Wading Through the Data Swamp:
Program Evaluation 201

Supplements

Checklist of Information To Be Provided by Program Evaluator

I. Research Questions

__ Provide list of research questions to address the desired outcomes (e.g., decreased alcohol and other drug use). List the data collection methods for each question.

II. Research Design

___ Provide an explanation of the research design and the advantages and disadvantages (e.g.,experimental or quasi-experimental; the comparison group used; the pretest and followup timing, etc.).

III. Data Collection Methods

A) Instruments

___ Provide copy of instrument(s).
___ Discussion of why particular instrument(s) selected:
___ validity and reliability
___ ease of administration (i.e., time to complete)
___ use by other agencies/programs (e.g., national surveys to which your data can be compared)
___ age, educational, and cultural appropriateness.
___ Plan for data collection (who, when, where, how).
___ Discussion of confidentiality of results.
___ Cost.
___ Copyright.

B) Other Data Collection Methods (e.g., record reviews, interviews)

___ Discuss data collection plan. (when, who, where, how).
___ Discussion of confidentiality of results.

IV. Pretest Data

___ 1) Provide raw data tables to program staff.
___ 2) For each item/variable of interest (for example, alcohol and drug use):
    ___ a) Provide descriptive statistics on participant and comparison groups: mean, median, mode, and standard deviations.
    ___ b) Provide results of subgroup analyses for participant group. Provide comparisons of means and standard deviation for independent variables such as gender, age, grade, race/ethnicity.
    ___ d) Provide scatter plots to identify outliers and histograms to identify the shape of the distribution.
    ___ e) Provide analysis of baseline equivalence comparing participants and comparison groups. Provide results from t-test and chi-square to establish equivalence between participant and comparison group or show how they differ.
___ 4) Provide discussion of missing data.
___ 5) Provide discussion of how outliers were handled.
___ 6) Provide discussion on how participants will be matched from pretest to posttest!

V. Posttest Data

___ 1) Provide raw data tables to program staff.
___ 2) For each variable of interest (e.g., alcohol and drug use, attitudes toward use, school performance, family bonding, etc.):
    ___ a) Provide descriptive statistics on participant and comparison groups: means, median, mode, and standard deviation.
    ___ b) Provide scatter plots and histograms. Visually display each data point.
    ___ c) Conduct subgroup analyses on variables of interest. Provide comparisons of means and standard deviations. For example, gender, age, grade, race/ethnicity.
    ___ d) Provide contingency tables. For example, comparing drug and alcohol users and nonusers in participant and comparison groups.
    ___ e) Discussion of missing data, if any. Provide discussion of plans to handle missing data due to attrition, loss to followup, or nonresponse.
    ___ f) Discussion of outliers, if any. Discuss the outliers in pretest data.

VI. Outcome Analysis (comparing pre- and posttest scores)

___ 1) Provide for each outcome of interest (e.g., reduced drug use):
    ___ a) the level of measurement of the variables (e.g., nominal, interval, ratio)
    ___ b) the statistical test chosen (what it tells us)
    ___ c) an explanation of why the test was chosen
    ___ d) the assumptions of the statistical test (e.g. normal distributions)
    ___ e) the level of statistical significance (p-value).
___ 2) Provide the sample size for each final analysis conducted and discuss the number and proportion of participants followed from pretest to posttest. (attrition, loss to followup, nonresponse).
___ 3) Provide scatter plots showing the direction of the outcome differences, indicating whether or not the pattern follows the trend specified in the research question.
___ 4) Provide an explanation of the statistical results; the limitations of the design and analysis; and how the statistical results address the questions about outcomes.
___ 5) Provide a list of alternative explanations for the observed results of the evaluation (e.g., end of the school year; differences in loss to followup; problems with the delivery of the program).
___ 6) Provide, for each outcome, recommendations for improving the program using the results of the evaluation and improving later efforts to evaluate it.

Wading Through the Data Swamp: Program Evaluation 201