The Froyo Effect: Achieving a 51% University-wide Response Rate

The presentation outlines how a small incentive program led to a 51% response rate in a university survey, a milestone not reached since 2005. It details the study background, response rate history, survey populations, marketing plan, administration, outcomes, and more.

• University
• Survey
• Response Rate
• Incentives
• Research

lamarche_p Follow

Uploaded on Sep 18, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. THE FROYO EFFECT: USING SMALL INCENTIVES TO ACHIEVE A 51% UNIVERSITY-WIDE RESPONSE RATE Office of Institutional Research and Analysis Director, Janel Sutkus, Ph.D. Research Designer and Analyst, Hyun Kyoung Ro, Ph.D.

2. Our goal for this presentation To describe in detail how we administered a small incentive program for our spring census survey that resulted in a 51% response rate (a response rate we had not achieved since Spring 2005)

3. Outline Study background Response rate history and goal Survey schedule by cohorts The big RQ Survey populations (on-campus and off-campus) Marketing plan Administration Calendar Invitation and incentive messages Outcomes Response rates and odds of responding Redemption of small incentives Questions and discussion

4. Study background The Carnegie Mellon University-Student Experience Study (CMU-SES) is a university-wide study designed to understand: how well, if at all, we are meeting our educational objectives for students which experiences, resources, and activities affect those educational outcomes when they do affect the outcomes, in what direction and magnitude The CMU-SES was designed to replace the End of Year Survey which was administered to undergraduates in 2000, 2001, 2004, 2005, and 2009. It will be administered to all CMU students on a three-year cycle, with the next administrations in the spring semesters of 2015 and 2018.

5. The core planning team began meeting on June 1, 2011 with these working principles: this will not be a revision of past EOY s, but an entirely new study the development process will be research question-driven, rather than survey item-driven, and the instrument will be created hand-in-hand with the analysis plan (we will hypothesize that certain predictors affect particular outcomes, and will test those hypotheses using regression methods) satisfaction is not one of our primary educational objective, so this will not be a satisfaction survey we cannot study everything we want to know, so we will set our priorities and study a few things in a thorough manner We spent ten months creating and testing this survey.

6. Survey and assessment schedule 11-12 to 18-19 Year 2011-12 2012-13 2013-14 2014-15 2015-16 2016-17 2017-18 2018-19 F S F S F S F S F S F S F S F S Cohort CIRP SES ADV ADV SES Fall 2011 ADV ADV SES ADV Fall 2012 BCSSE NSSE NSSE ADV SES ADV ADV Fall 2013 CIRP SES ADV ADV SES Fall 2014 ADV ADV SES ADV Fall 2015 BCSSE NSSE NSSE Comparisons: Comparisons: Comparisons: FY CIRP SES Fall CIRP to Spring SES BCSSE NSSE /ADV Fall BCSSE to Spring NSSE ADV FY ADV to JR ADV Soph ADV FY SES to SR SES ADV FY NSSE to SR NSSE SES JR ADV to SR ADV Junior ADV Soph ADV to JR ADV SES FY ADV to Soph ADV ADV Senior SES NSSE/ADV Soph ADV to SR ADV ADV CIRP: Cooperative Institutional Research Program Freshman Survey (UCLA-HERI) Spring 2012 SES BCSSE: Beginning College Student Survey of Engagement (Indiana University) Spring 2013 ADV NSSE (FY and senior only) NSSE: National Survey of Students Engagement (Indiana University) Spring 2014 ADV SES: Carnegie Mellon University Student Experience Study (CMU IR&A) Spring 2015 SES ADV: Carnegie Mellon University Undergraduate Advising Survey (CMU IR&A) Spring 2016 ADV NSSE (FY and senior only) Spring 2017 ADV Spring 2018 SES Spring 2019 ADV NSSE (FY and senior only)

7. Response rate history for spring census surveys 58.6% 53.0% 39.7% 33.0% 32.7% 32.3% 31.6% UG pop randomly split in two April 2000 End of Year April 2001 End of Year April 2004 End of Year April 2005 End of Year April 2009 End of Year April 2009 Advising April 2011 Advising

8. The big goal and associated RQ By using a series of small incentives, can we return our spring census response rate to the levels we had in the mid-2000 s? We hedged our bets by also providing a large raffle-type incentive, but our plan was to do analysis at the end to determine whether small incentives could replace large incentives in the future. RQ: Are there differences in the odds of responding during a particular small incentive window for students who are eligible for small incentives compared to students who are not eligible for small incentives?

9. Survey population ON CAMPUS: 5517 students Were eligible for a drawing for one of two iPad3 s (or $500 at Bookstore) AND received a small incentive for completing the survey OFF CAMPUS: 117 students on study abroad were eligible for a drawing for one of two iPad3 s (or $500 at Bookstore) LIMITATION AND CAVEAT: Other than the big difference of being off-campus and having study abroad experience, the two groups were much the same in terms of demographics and ultimate survey responses Off campus students were not aware what they were missing

10. Marketing plan Who says? CMU-SES! branding campaign Banner in University Center Table tents at dining locations Flyers Tabling on the Cut with balloons and branded candy Core team had CMU-SES t-shirts with iron-on transfers

11. Who says? CMU-SES Student Experience Study

12. Administration calendar Sunday Monday Tuesday Wednesday Thursday Friday Saturday 25 26 27 28 29 SURVEYLAUNCH 11:00 PM 30 LAUNCH EVENT AT UC 4-6 PM 31 1 2 GIVEAWAY #1 ENDS AT 10 PM REMINDER #1 SENT AT 10 PM 3 GIVEAWAY #1 AT UC 3-6 PM (FroYo) 4 5 6 7 8 9 10 11 GIVEAWAY #2 ENDS AT 10 PM 12 GIVEAWAY #2 AT UC 3-5 PM (Fitness class card) REMINDER #2 SENT AT 10 PM 13 14 15 16 17 18 GIVEAWAY #3 ENDS AT 10 PM REMINDER #3 SENT AT 10 PM 19 GIVEAWAY #3 AT CARNIVAL MIDWAY 3:30-5 PM (Funnel cake) 20 21 22 23 24 25 GIVEAWAY #4 ENDS AT 10 PM REMINDER #4 SENT AT 10 PM 26 GIVEAWAY #4 AT UC 3-5 PM (Maggie Murph coffee) FINAL REMINDER SENT AT 10 PM 27 SURVEY CLOSES 2 PM WRAP EVENT AT UC 5-7 PM (CMU gear and iPad3 drawing) 28

13. Invitation Dear Andrew, We are writing to invite your participation in the Carnegie Mellon University Student Experience Study (CMU-SES). Every three years the university conducts a study of the entire student body to gain insight into the educational experience at CMU. Your input is extremely valuable and should take fewer than 10 minutes using the link below. Students who complete the survey will be placed into a drawing for one of two iPad3 s (the recipients may choose to substitute $500 of merchandise at the University Stores). In addition, all students who complete the survey between now and 10:00 PM on Monday, April 2, will be eligible to pick up a certificate for a free FroYo at Skibo Caf . Not included for students studying abroad Please click here to participate in the survey: <<link>> Or copy and paste the URL below into your internet browser: <<URL>> This study is sponsored by the Vice Provost for Education and the Dean of Student Affairs and is administered by the Office of Institutional Research and Analysis. Your individual responses will be kept confidential and will only be reported in the aggregate. No information that could identify your responses will be reported. The survey will be open until Friday, April 27, at 2:00 PM. Thank you for your participation. If you have any questions about the study, please contact Janel Sutkus, Director of Institutional Research and Analysis, at jsutkus@cmu.edu. Sincerely, Amy Burkert, Vice Provost for Education Gina Casalegno, Dean of Student Affairs

14. End of survey messages On campus: Thank you for completing the CMU-SES! Watch your e-mail for details on how to claim your thank you gift and for your invitation to the closing party on April 27 where we will announce the winners of the two iPad3's (you do NOT need to attend to win). Studying abroad: Thank you for completing the CMU-SES! You have been entered into the drawing for one of two iPad3's.

15. E-mail upon completion Dear Andrew, Thank you for participating in the CMU-Student Experience Study. This e-mail is your ticket for TWO things: 1) You have won a certificate for a free FroYo at the Skibo Caf . Please pick up your certificate at: The University Center, Wean Commons on Tuesday, April 3, from 3:00 to 6:00 PM. You will need your student ID and this e-mail to receive your certificate. You can either print this e-mail or show it on your smartphone. The certificate will be valid until May 11. 2) You are invited to the CMU-SES closing party at: The University Center, Connan Room on Friday, April 27, from 5:00 to 7:00 PM. The party will include refreshments and music, and students who attend will receive tickets for drawings for CMU gear. We will also announce the winners of the two iPad3's (you do not need to attend to win). You will need your student ID and this e-mail to attend. You can either print this e-mail or show it on your smartphone. We look forward to seeing you there. Janel Sutkus and Hyunny Ro Office of Institutional Research and Analysis Office of the Vice Provost for Education Office of the Dean of Student Affairs Amy Burkert Gina Casalegno and Holly Hippensteel

16. Certificates This certificate is good for one Group X fitness class (includes yoga classes) at the University Center. This certificate is good for one FroYo at the Skibo Caf . Thanks for participating in CMU-SES! Thanks for participating in CMU-SES! Expires 5/11/12 Expires 5/4/12 # # This certificate is good for one funnel cake at the A Phi O Booth at Carnival. This certificate is good for one 16 ounce brewed coffee or tea at Maggie Murph Caf . Thanks for participating in CMU-SES! Thanks for participating in CMU-SES! Expires 4/21/12 Expires 5/11/12 # # Each certificate was hand-numbered in green ink and samples were given to each vendor.

17. Responses by date 662 484 Reminder Survey launched at 11 PM 354 257 Final Reminder Reminder 199 159 163 Reminder 88 Reminder 71 86 41 44 44 48 3/29 3/30 3/31 4/1 4/2 4/3 4/4 4/5 4/6 4/7 4/8 4/9 4/10 4/11 4/12 4/13 4/14 4/15 4/16 4/17 4/18 4/19 4/20 4/21 4/22 4/23 4/24 4/25 4/26 4/27

18. Responses by incentive Invited R5 3104 Responded 315 Invited R4 3209 Invited R3 Responded 105 3669 Invited R2 Responded 460 4301 Invited R1 Responded 632 5635 Responded 1334 Incentive FroYo Fitness class Funnel cake Coffee None Response rate 23.7% 14.7% 12.5% 3.3% 10.1% Odds of responding 89.8% higher odds no difference in odds 688.3% higher odds no difference in odds no difference in odds in the specific incentive window for a student in the small incentive population compared to a student not in the small incentive population, and controlling for gender, race, citizenship, and class year

19. Incentive redemption Week Incentive Days active Cost Responses N % Dist at Closing Party Cost if all redeemed N % Actual cost Collected Collected redeemed redeemed 1 FroYo 4 $3.00 1334 183 13.7% 20 $609 136 67.0% $408 Fitness Class 2 10 Gift 632 17 2.7% 5 29.4% Funnel Cake 3 6 $3.50 460 76 16.5% $266 62 81.6% $217 4 Coffee 7 $1.25 105 7 6.7% 10 $21 12 70.6% $15 ALL 27 2531 283 11.2% 30 $947 215 76.0% $640

20. Total costs Incentives Cost Marketing Cost Closing Party Cost All small incentives $640 Banner $144 Balloons and decorations $146 Food (pizza, popcorn, cookies, hummus trays) Two iPad3 s $998 Balloons $25 $468 T-shirts, candy, supplies $122 Drinks $32 $1638 $291 $646 $2575 Compared to the most recent census survey (April 2011 Undergraduate Advising), we paid $143 per additional response percentage point or $2.61 per additional student

21. Future plans CMU-SES in Spring 2015: GOAL: Maintain response rate in the low 50 s Marketing and branding continues No large incentives Keep small incentives (replace fitness class and coffee) Anticipated budget - $1700 New health behaviors survey in Fall 2013: GOAL: Response rate in low 50 s to match with FY s and sophs from CMU-SES in Spring 2013 Create marketing and branding campaign Guaranteed small incentives (FroYo, other tasty but healthy treats) Raffle every two days for Fitbugs

22. In the time since. . . 58.6% 53.0% 51.0% 39.7% 33.4% 33.0% 32.7% 32.3% 31.6% One large incentive: iPad Mini UG pop randomly split in two April 2000 End of Year April 2001 End of Year April 2004 End of Year April 2005 End of Year April 2009 End of Year April 2009 Advising April 2011 Advising April 2012 Student Experience Study April 2013 Advising

24. For further information Janel jsutkus@cmu.edu Hyunny hro@cmu.edu