Enhancing Interview Techniques and Tools

interviews, surveys, questionnaires Dr. Alissa Antle 1 Credits: Ron Wardell

retrospective testing interviews post-observation interview users reflect on what they did clarify events avoid erroneous reconstruction users often offer concrete suggestions Why didn t you try this link? I didn t see it. Maybe make it a button. 2

critical incidence interviews people talk about incidents that stood out usually discuss extremely annoying problems not representative, but important to them Tell me about the last big problem you had with Word. I can never get my figures in the right place. It s so annoying! 3

types of interviews unstructured / open most like a conversation, no questions planned pros: rich detail that interviewer may not have thought of cons: hard to replicate, hard to analyze semi-structured pre-select topic areas and potential questions pros: can guide interview but not enforce what is discussed cons: can go off topic 4

types of interviews structured worded analyze predetermined questions that are short and clearly confirmatory rather than exploratory pros: easy to replicate, get specific answers, easy to cons: restrictive answers, details can be lost 5

the game of spot spot the good and bad things in an interview 6

tools for interviews flow: things to do when you want the person to keep talking addition encouragement body movement attentive silence i see , yes i understand , that s interesting nod, lean forward, eye contact wait for person to finish talking 7

tools for interviews non-direction: you want conversation to continue but you don t want to give a direction reflecting echo question-to-question attentive listening repeat last few words (~3) give a neutral question, well, what is your feeling on this? form a question to test what they just said, use what they said and change the words slightly 8

tools for interviews transition: you want to close the previous conversation and open a new discussion reversion mutation associate ideas from previous to new point, it sounds like you use email a lot, do you use IM similarly? unstructured question raising without direct connection to previous conversation (abrupt), we re now going to move on and talk about x. 9

tools for interviews specificity: you want to get more specific information situation re-presentation reconstruction talk about something and then show the user something about it (present the stimulus), we ve talked about how you use email, now let s watch you check your email. recall specific reaction at some event, tell me about the last time you checked your email. 10

tools for interviews depth: you want to get detailed feelings about something emotion/feeling projection attentive listening question using a feeling term, did this make you feel frightened? question about feelings of a hypothetical friend if it seems to be sensitive to them, how do you think your friend felt? if you can t tell how they felt, form a question based on what they just said, it sounds like you might have been frightened. is this true? 11

tools for interviews individual context: you want to make the interview about the specific interviewee personal self-description parallel direct request for description of self, i am interested in what you think, not necessarily what you think others feel. ask about a parallel situation with reactions that would be similar or related 12

types of interview distortion internal distortion problems internal to the interviewer e.g., personal traits, lack of experience, listening skills reduce by restating what people say, take notes, corroborate with other sources 13

types of interview distortion external distortion problems with interaction between interviewer and viewee e.g., distractions (finger tapping), violation of social norms, conflicting messages (body language), emotional discomfort reduce by analyzing own behavior, be aware of social norms, arrange appropriate environment 14

how to write and choose questions use understandable language e.g., don t say: In what way do you understand your cultural context? use short questions e.g., don t say: Where do you commonly place your cell phone and then how do other people in the family notify you when it rings? avoid questions that can lead to biased answers e.g., don t say: You must like using your cell phone at home, right? 15

how to write and choose questions avoid yes and no questions e.g., don t say: Do you communicate using your cell phone often? try to get the respondent to tell a story or talk about a specific incident e.g., say: Tell me about the last time you used your cell phone at home. e.g., say: Tell me about the most memorable occasion where you used your cell phone at home. 16

ordering questions temporally: past to recent events complexity: simple to complex group by topic by level of abstraction: concrete to abstract by threat level: least sensitive to most sensitive 17

start the game 18

surveys / questionnaires automated interviews generate wide range of opinions / large sample pros: cheap to administer large sample cons: time consuming to prepare potentially low response potentially low quality of response 19

questionnaires for usability studies pre-test questionnaire learn about participants experiences before using the system find out any potential biases e.g., how often do you use X? post-test questionnaire ask about each aspect of usability ask about areas of specific concern 20

rules of thumb decide the purpose and audience upfront determine how you d analyze the results don t ask questions you won t use the answer to most important questions at 1/3 point space reliability checks widely apart save personal questions for end test the wording, timing, validity, analysis 21

administering in person requires time but highest completion rate take home lack of completion or questionnaire return email less time constraints for respondent response rates vary web-based standard formats, scripts to ensure completeness 22

timing maximum number of questions phone mail group *peer pressure keeps people going, e.g., the other guy is still answering interview max 80 questions max 20 questions max 60 questions max 100 questions 23

styles of questions open-ended asks for opinions, gets subjective responses hard to analyze e.g., how would you improve the interface? closed supply a choice of answers easier to analyze, but details can be lost e.g., how well do you like this course? 1 not at all 2 3 4 5 love it 24

styles of questionnaires funnel start with open questions, then closed questions good for sensitive topics reverse funnel start with closed questions, then open up good for non-threatening topics 25

paired comparisons compare pairs of attributes fast to answer, potential for inconsistent responses select the feature that you think is more important for the user interface. color of buttons location of buttons or or size of buttons color of buttons location of buttons size of buttons or 26

rank order ask respondents to rank up to 10 items understand user s preference magnitude between the steps is not known rank order the importance of each interface item: color of buttons _1___ location of buttons __3__ size of buttons _2___ 27

Likert scale - scalar measure opinions, attitudes, and beliefs scale is carefully constructed and tested to be equally spaced i like the presentation of the interface controls. strongly agree 1 2 3 4 5 strongly disagree 28

Likert scale semantic differential explore bipolar attitudes to items each pair of attitudes is represented as a pair of adjectives the user interface is: clear attractive 1 2 3 4 5 1 2 3 4 5 confusing ugly 29

multiple choice offer explicit responses where is the primary computer in your home? (select one) 1. living room 2. office 3. bedroom 4. kitchen 30

and many others constant sum graphic scales thurstone scales stapel scale 31