Program Evaluation Essentials at the Department of State
This collection covers the essentials of program evaluation at the Department of State, including data collection methodologies, evaluation designs, logic models, data integrity, differences in data types, and the importance of reliability and validity in evaluations. It delves into the intricacies of data collection for public diplomacy evaluations and provides insights into quantitative and qualitative data distinctions.
Uploaded on Sep 26, 2024 | 0 Views
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Collecting Data for Program Evaluation at the Department of State By Dean Olsen Social Science Analyst Evaluation and Measurement Unit US Department of State
A data collection overview Evaluation Design, Question(s), and Logic Models Data Integrity Differences in Data Data Collection Methods and Analysis Considerations for selecting data collection methods Data Collection for Public Diplomacy Evaluations at the Department of State
Evaluation designs, questions, and logic models The purpose for the evaluation Who is the audience? How will the evaluation be used? Evaluation question(s) Types of evaluation designs Randomized experiments Quasi-experimental Non-experimental Pre/Post-Test Non-experimental Time Series Analytical framework Logic models
Program Logic Model Your Planned Work Your Intended Results Resources/ Inputs Activities Output Outcome Impact What does success look like?
Evaluation Logic Model Evaluation Question Method for Data Collection Data Analysis Method Type of Answer/ Evidence Needed Sampling/ Selection Approach Source Method Survey Interview Focus Group Observation Test/Exam Document Review Participants Parents Teachers Trainers Stakeholders Program documents
Data Integrity Reliability Is the ability to produce consistent results and indicates whether an evaluation is likely to produce the same results if conducted again. Validity Is about coming as close to accurately measuring something as possible. For example, if you want to assess a child s math skills, the math test you administer to them must be validated such that it is measuring the child s actual math ability and not something else, such as their test taking skills. Source: Corporation for National and Community Service, http://www.nationalservice.gov/sites/default/files/resource/Description_of_Audio_Data_Collectio n_for_Program_Evaluation.pdf
Differences in Data Secondary and Primary Data Data some else collected Data you collect Disadvantages Advantages Quantitative Data Is information about quantities; that is, information that can be measured and written down with or expressed by numbers. Qualitative Data Is information that cannot be expressed by numbers and is descriptive in nature.
Data Collection Methods and Analysis Quantitative Methods Surveys, Polls, Questionnaires Data someone else has already collected Agency databases, public opinion polls, program performance statistics/metrics, publications, journals, books, magazines, etc. Analysis Statistics, graphs, charts Qualitative Methods Structured and unstructured interviews, focus groups, open ended questionnaires, observation, document reviews Analysis Content analysis, word clouds
Considerations for selecting data collection methods Evaluation Question(s) Nature, scope, and purpose of inquiry Type of evaluation design Availability of funds Time factor Precision required
Data collection for Public Diplomacy Evaluations at the State Department Stakeholders and Contacts Understanding the culture, audience and environment Scheduling, Relationship Building, and Snowballing Third Party Vendors Who else do you think I should be talking to?
