Understanding Different Types of Experimental Designs

Experimental design

Basic experiments The simplest possible experimental design involves one independent variable (with two levels) and one dependent variable E.g., Bandura s Bobo doll study There are two possible versions of this: Posttest-Only Pretest-Posttest 2

Posttest-Only Design Random assignment is used to create equivalent groups Careful selection of levels of the IV 3

Pretest-Posttest Design A pretest is given before the experimental manipulation This may measure the DV or a participant variable This allows you to ensure the groups are equivalent (important with small samples) It may be necessary for the research design (to assign participants) It allows you to look at change scores It allows you to eliminate alternative explanations 4

Solomon four-group design Half the participants receive only the posttest; the other half receive both 5

6

Independent vs. Repeated Measures Designs Independent groups design (between, uncorrelated) Different participants exposed to each condition Repeated measures design (within, correlated) The same participants exposed to multiple conditions over time 7

Independent groups design Advantages Scores are not influenced by order effects (practice, fatigue, contrast, etc.) Disadvantages Large number of participants required Statistically weaker (a larger effect is required to achieve statistical significance) Individual differences produce high variability and can become confounding variables 8

Confounding variables in Independent designs Individual differences Participant characteristics differ from one group to another E.g. the participants in one group may be older, smarter, taller etc. than the participants in another group Controlled by? Environmental variables Characteristics of the environment differ between groups E.g. one group may be tested in a large room and another group in a smaller room .other examples? Time of day, characteristics of the researcher, location, distraction .

Solutions Random assignment Matching groups (matched assignment) Involves assigning individuals to groups so that a specific variable is balanced or matched across the groups (e.g. IQ, age, ability, history) Holding variables constant Simply hold the variable constant (e.g. restrict the participants to those with IQs between 100-110) Standardize procedures and treatment setting Use a large sample size Increase the differences between the levels of the IV

Other threats to internal validity in Independent design Differential attrition Compensatory equalization Compensatory rivalry Resentful demoralization 11

Repeated measures design Advantages Requires fewer participants Eliminates problems based on individual differences (no differences between groups and each individual serves as his/her own control) Disadvantages Participant attrition History or maturation effects Instrumentation Regression toward the mean Order effects (practice, fatigue, contrast) 12

Solutions Minimize the interval between treatments (where possible) Counterbalancing Complete counterbalancing Number of possible orders = Number of conditions! 4 conditions = 4! = 4 x 3 x 2 x 1 = 24 orders Latin squares Number of possible orders = Number of conditions 4 conditions = 4 orders 13

Caveat Sometimes the topic of study will determine the choice of research design Any manipulation that produces a relatively permanent change in the participant cannot be used with RM design If you are interested in studying developmental changes the RM design is most appropriate 14

MRS. SMITH Maturation Regression to the mean Selection of participants Selection by maturation interaction Mortality Instrumentation Testing History 15

HISTORY: Unplanned events that occur between the first and second measurements. Examples include changes in local drug use patterns, drug supply interruptions, changes in treatment approach, and seasonal drug use patterns. In general, the more time that elapses between measurements, the greater the threat from historical effects. MATURATION: Developmental changes that naturally occur in clients. Growing older, more experienced, or more independent may be especially important maturation effects with adolescent client populations. Other maturation effects raise particular concerns in samples from special population in transition, such as pregnant women, HIV positive clients, and ex-offenders re-entering society from prison. TESTING: Effects of taking a measurement on the results of subsequent measurement. Repeated urine tests for drug use tend to discourage later drug use, for example, while repeated ability tests tend to raise scores and subjects practice their test-taking skills, even if no real increase in ability occurs. INSTRUMENTATION: Effects of changes in a measurement instrument, or in criteria for recording behavior, during the course of an evaluation. Common examples are a change in the cutoff point for a A drug positive@ determination by a urinalysis laboratory or a change in police criteria for making arrests (for example, during a neighborhood crackdown). 16

STATISTICAL REGRESSION: Effects of selecting samples on the basis of extreme behavior -- over time, their behavior tends to A regress@ toward the overall group average. In drug treatment program evaluation, prodrug attitudes in an extremely prodrug sample, heroin consumption in a sample selected during periods of heavy use, and self-esteem in a sample selected on the basis of low self-esteem, will all tend to be less extreme on a second measurement. SELECTION: Effects of unmeasured difficulties between a group receiving treatment and a non-equivalent group not receiving treatment. Common examples of unmeasured difficulties include clients' motivation to seek treatment, family and social support structures, and expectations about future drug use. The problem tends to arise when treatment is given to volunteers and withheld from non-volunteers, instead of assigning volunteers randomly to treatment. ATTRITION: Effects of unequal dropout rates among different subgroups in the sample. In drug treatment program evaluation, common examples include differences between those who receive treatment and those who do not, between heavier drug users and lighter users, or between more and less satisfied clients. HAWTHORNE EFFECTS: Effect of changes which are due to the fact of being included in an evaluation. 17

18

Threat to Internal Validity Controlling Threat History Random selection, random assignment Maturation Subject matching, randomization Testing Control group Statistical Regression Omit extreme scores, randomization Instrumentation Instrumental consistency, assure alternative form reliability Selection Random selection, random assignment Experimenter Bias Double blind study Mortality Subject matching and omission 19