January 11, 2022, Evaluation Observatory
Suppose a teacher wants to measure the impact of their teaching techniques on students' learning outcomes. A simple way of measuring the impact would be as follows:
The teacher will conduct a test for students before teaching the class. After which, they will teach the class using a particular technique, and take another test for the same class with a similar level of difficulty. Finally, the teacher analyses the difference in score of these two tests to measure the impact of their teaching.
The evaluation design used here is referred to as Pre-Post Test Design, and it is an experiment in which outcome or impact indicator measurements are taken before and after the treatment is administered (Zach, 2020) using a standard survey pre and post the intervention. (EMpower, n.d.).
Pre-Post Test Designs are flexible; and can be used across non-experimental, experimental and quasi-experimental research settings. While it has a treatment group, it may or may not include a control group (Zach, 2020). Although, it is especially used in non-experimental settings within the same treatment group when either the interventions are universally applicable, or too complex to generate a matched comparison group (Kultar et al., 2017).
In the presence of a control group, Pre-Post test designs have a strong internal validity but face challenges in achieving external validity (Shuttleworth, 2009). However, when done in a non-experimental setting, since there is no baseline measurement against groups that remained completely untreated, it becomes challenging to find whether the pre-testing process influenced the results (Shuttleworth, 2009), thus weakening internal validity. For example, children giving a pre-test may try a harder in their lectures and try to outperform children who were not given a pre-test; thus making the test results an outcome of the experimental process, rather than administering the treatment alone (Shuttleworth, 2009).
Advantages of Pre-Post Test Design (meera, n.d.):
- Pre-post-test design are relatively simple to implement
- It controls for variables that may influence outcomes prior to administering the intervention
- Compared to retrospective pre and post-tests, or post-tests, they deliver better evidence for effectiveness of the intervention
Disadvantages of Pre-Post Test Design (meera, n.d.):
- In the absence of a control group, it cannot account for confounding variables or non-programmatic influences on outcome indicators.
- When self-reporting measures are used, in the absence of a control group, it may lead to over-estimation of a particular indicator at baseline (due to lack of knowledge) and accurate estimation at endline (after undergoing treatment); thus, creating erroneous conclusions.
List of recommended resources:
For a broad overview
-
Written by Martin Shuttleworth, this article gives a simple overview of the concept and discusses the pros and cons of implementing pre-post test designs.
-
Written by Jessica L. O'Leary and Glenn D. Israel and published by the University of Florida, this provides the introduction of the Prestest-Posttest Model and discusses advantages and limitations of the model.
-
Developed by EMpower, this document aims to help programs avoid some of the most common errors in pre-and post-evaluation and provide tips for analysis and reporting results.
-
Published by Statology, this web article defines Pre-Post Test Design and illustrates the process of using this design with the help of examples.
-
Written by Dimiter, Dimitrov and Phillip D. Rumrill Jr., this article examines issues in comparing groups and measuring change with pre-test and post-test data. Furthermore, it presents designs that can help rehabilitation professionals better understand and determine the effects of select interventions.
For in depth understanding
-
Written by Peter L. Bonate, this book has an in-depth discussion on various aspects of Pre-Post Test Designs, including data analysis, measurement concepts, difference scores, analysis of covariance, statistical tests, etc.
Case Study
-
Published by SAGE Publications, this case study demonstrates the evaluation of recreational therapy program for people with dementia using paired t-test, one-sample t-test and binomial test.
-
Published by National Journal of Physiology, Pharmacy and Pharmacology, this article evaluates the knowledge of didactic lecture among students by giving pre and post-test questionnaire-based evaluation techniques.
-
Written by Tayyaba Gul Malik and Rabail Alam, this case study discusses the effectiveness of the Pre-Post Test in comparison with a post-test-only model in determining the learning outcomes of a 45 minute lecture.
Toolkits
-
Published by Real Statistics, this post guides practitioners on using excel for Pre-Post Test design. It explains on taking a T-test, ANOVA and ANCOVA approaches to Pre Test-Post Test design.
References:
Guidance For Pre- and Post-Test Design. (n.d.). EMpowerweb. Retrieved December 26, 2021, from https://empowerweb.org/assets/uploads/tools-resources/461/5._guidance_for_pre-_and_post-test_design.pdf
Martyn Shuttleworth (Nov 3, 2009). Pretest-Posttest Designs. Retrieved Jan 05, 2022 from Explorable.com: https://explorable.com/pretest-posttest-designs Singh K., Chandurkar D., & Dutt V. (2017). A Practitioners’ Manual on Monitoring and Evaluation of Development Projects83. Cambridge Scholar Publishing.
Types of Evaluation Designs. (n.d.). meera. Retrieved December 26, 2021, from https://meera.snre.u-mich.edu/types-evaluation-designs
Zach. (2020). Pretest-Posttest Design: Definition & Examples. Statology. https://www.statology.org/-pretest-posttest-design/