Developing Participatory Indicators for Program Evaluation
Sessions led by Dr. Angela Veale aim to establish baselines, identify indicators, and reflect on evaluation practices in psychosocial program evaluations. Understanding indicators at output, outcome, and impact levels is crucial for measuring project success. The process involves defining indicators, measuring results, and assessing changes to achieve project objectives effectively.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
E N D
Presentation Transcript
Establishing Baselines Sources of Information for Indicators and Developing Participatory Indicators DR ANGELA VEALE, SCHOOL OF APPLIED PSYCHOLOGY, UNIVERSITY COLLEGE CORK.
Overview Evaluation is a central feature of programme design and project cycle management. Aim of session is to reflect on how to identify and develop the best indicators for evaluation purposes. Look at some case studies of psychosocial programme evaluations
Measuring What? Examples Critical questions Programme outputs the planned achievements put out in the process of implementing a project that signal work is on track Finalisation of project training Handbook; What are the visible signs that the process of implementing the project is going to plan? Has the project done what it sent out to do Process evaluation No. of people trained; Project delivered in 20 districts. Programme outcomes the results that come about during the course of a project as a result of the outputs achieved Increased knowledge about gender-based violence; Increase in positive pro- social coping skills; Greater community acceptance of refugees Men using vocational skills to support livelihoods in their households What changes have come about for children, their caregivers or communities? Programme impact the overall change in the lives of children, their families and communities that result from a project reflecting the stated goal. What is the ultimate change relevant to children's lives that was hoped for when funding the project? - Impact evaluation Refugee men assume valued social roles within community
Establishing baselines: Developing your evaluation tools Phase 2 Phase 3 Participatory Tool Development Phase 5 Analysis and feedback Revision Pre-existing scales Gaps or cultural sensitivity? If so, goto 3 Phase 4 Phase 1 Pilot testing field instruments Document review
Establishing baselines- role of indicators For each project objective at output, outcome and impact levels, we need to define how we would measure results An indicator is a simple, clear statement that helps measure and communicate change If there is an objective there should be a means of measuring if it has been achieved
T1 T2 Baselin e Post Waitlist Group no intervention (yet) Standard practice Intervention Group
Study Indicators Innovative Features in Conditional Cash Transfers: An impact evaluation of Chile Solidario on households and children , Monetary: household income per capita (thousands of USD) Labour: a) employment for the household head (%); b) employment in the informal sector for the household head Martorano, Bruno, and Marco Sanfilippo, Innocenti Working Papers No. 2012-03, UNICEF Innocenti Research Centre, Florence, 2012. Education: school enrolment of school-aged children Health: enrolment in public health system (yes.no)
Study Indicators The Pakistan Early Child Development Scale Up Trial: Outcomes on Child Development, Growth and Health, Early childhood development: scores on cognitive, language, motor and socio- emotional development assessed using Bayley Scales of Infant and Toddler Development (third edition) Yousafzai, A.K., et al., PEDS Trial Report, written for UNICEF Pakistan, Karachi, 2012. Nutrition: growth as indexed by length/height-for-age, weight-for-age and weight-for-height z-scores.
Study Indicator The South African Child Support Grant Impact Assessment: Evidence from a survey of children, adolescents and their households, Education: a) grade attainment b) Scores on mathematical ability tests and reading and vocabulary tests average days absent from school Child Labour: adolescents employed outside the home (days of employment in the previous month) Health: experience of illness (as measured during a 15 day period prior to the survey Risky sexual behaviours among adolescent: a) sexual activity ( never had sex); b) number of sexual partners c) Pregnancy ( ever pregnant ) UNICEF South Africa, Pretoria, 2012.
When standardised measures dont cut it? UNICEF- Consultancy to Provide Technical Assistance on Pilot to Develop a New Stress Measure for Adolescents and Caregivers UNICEF currently utlising a validated stress scale Programme showing a lack of impacts- may be that existing measures of perceived stress have mainly been developed and validated in the United States and western settings and may not capture the concept of stress in all settings. In order to capture stress as a potential pathway for positive outcomes in poverty alleviation programs and other interventions, it is important to adequately measure stress and thus a A measure of perceived stress suitable for the context is needed- a need for participatory indicator development? How would you go about doing this?
Developing participatory evaluation indicators The project: Supporting the social reintegration of former child soldier young mothers in Liberia, Sierra Leone & northern Uganda The approach: Highly participatory; meaningful participation was a central principle How could we know if the programme was successful in having a positive impact on the social reintegration of young mothers? Who decides what is effective social reintegration ? How best to measure social reintegration? Standardised measures possible (e.g. Harvard Trauma Questionnaire; Child War Trauma Questionnaire; Retention in School etc.) used in other studies; not suitable here Decision to develop participatory and culturally relevant indicators.
Evaluation Methodology Free listing At a workshop with representatives of young mothers from the 3 countries, they generated lists of important indicators of successful reintegration. This generated 47 items Ranking In northern Uganda, indictors were ranked in order of importance. Similar items were grouped together. Low ranking and highly similar indicators were discarded Face validityRemaining items were examined for face validity ambiguous or unclear indicators were discarded. Final survey instrument contained 13 indicators of social reintegration.
Participatory Indicators Participatory Indicators 1. I feel involved with what the group is doing 2. Involvement in the project has made me more liked or loved by my family 3. My boyfriend/husband is supportive of my children 4. Through participating in the group, I can now speak in public more easily 5. I am able to be supportive to my family by buying basic necessities 6. Through the group, I help other people in the community 7. I can take better care of my child than I could before I joined the group 8. Many girls in Sierra Leone/Uganda/Liberia have sex partners to earn money. Is this true of girls in the project? 9. I feel more supported and respected by community members now than I did before the project 10. Has your health changed since you joined the project? 11. Has the health of your child or children changed since you joined the project? 12. I get pregnant even when I don t what to 13. If someone wants to go to bed with me by force, I know how to report it and get help
Cultural indicators with formerly associated girls, Sierra Leone Indicators of good reintegration Is engaged in income-generating activity Married/has a good marriage Goes/has gone to school Has children Good health Eats well Emotional and/or financial support from family Is invited to join community events and celebrations Is invited to women s secret society events Stark et al. (2009) Developing culturally relevant indicators with armed groups, Intevention, 2009, 7, 1, 4-16.
Example 2: Participatory indicator development with Syrian Refugee Families in Lebanon Conduced with male participants and their families who had completed an Engaging Men programme (men, wives of men, children of men- adolescent girls and boys) Men and their wives and teenage children who are due to start the programme Shanahan, Hajazi, Veale (2016)
Do no harm Facilitators invited participants- existing relationships, ongoing support Informed consent process- written consent Voluntariness Right to withdraw at any time Data protection Confidentiality Psychosocial referral pathways Men and boys and women and girls were interviewed separately Interviews were conducted in Arabic with translation into English. Approval obtained from UCC Ethics Committee.
Overall impact: To contribute to the building of a protective environment of conflict-affected Syrian refugees and vulnerable host communities Outputs and Outcomes Indicators Baseline Post- intervention Outcome Male Syrian refugees promote safety in their communities through their positive attitudes and behaviours Outputs Increased capacity of men to use prosocial coping strategies as a result of the training provided No of men who completed course Score on standardised gender scale Syrian and Lebanese men have increased capacity to mitigate and resolve conflicts peacefully. PRM
Participatory Indicator Development Problem analysis (Protection problems affecting men, women, teenage girls, teenage boys and children). Social mapping of vulnerability and situated risk. Free listing and ranking of coping strategies (to reduce stress and support protective capacity) generated and ranked by men, women, teenage girls, teenage boys. Free listing and ranking of conflict resolution capacity generated and ranked by men, women, teenage girls, teenage boys. Reduction in inter-community conflict indicators generated and ranked by men, women, teenage girls, teenage boys.
Problem analysis Male participants explained that due to deprivation and poverty, men fight with their wives and children. They gave specific examples of how after coming home they are greeted by "why didn't you bring food/bread today? Various men endorsed how wives' "nagging" over their inability to supply household needs compounds their frustration over the situation and causes them to hit their wives. Another source of arguments in the family that may lead to hitting are wives frustrations over not being able to visit their family in other parts of Lebanon. This is further exacerbated by if the husband's family are nearby but not the wives. Some men also identified how their guilt and frustration over having to select some of their children to go to school, but not others also frustrates them and increases their risk of fighting with and hitting their wives. Some men also identified early child marriage (around 13-14) as an issue, explaining that they perceive that female children have more expenses than male children (e.g. because they need more clothing). Women identified the impact of stress on men reflecting at home as the most significant family problem, followed by the impact of stress on their children. Women spoke predominantly about verbal abuse with some participants also indicating that physical violence was an issue. Children identified fighting in the home due to the stress their mothers and fathers were under as a significant problem.
Psychosocial problems affecting men Men identified psycho-social problems related to: feelings: male participants identified feeling helpless, weak, broken, being suffocated from bottling up, low mood, feeling pressured, noting how they feel aged because of their burden, noticing how they or other young people have become like old people, feeling that frequent worry would make a "mountain collapse". Feeling like one wanted to "escape one's skin" if for example the children smelled the neighbors barbecuing meat but the fathercan't afford meat. Thinking : male participants reported frequent worrying about the future, worrying about developing chronic diseases like diabetes, or getting a stroke, ruminating about conflict related experiences. Behaviour : increased smoking, irritability, yelling, speaking in a sharp tone, headaches, taking multiple panadols, flashbacks, nightmares, (involuntarily) losing weight, neglecting outside appearance (e.g. not going barber), depriving oneself to provide. Being forced to accept the situation, smoking, nargileh, hash, drinking.
Situated vulnerability and risk mapping Social mapping of potential vulnerability Situated risks Teenage girls 2. Wives with absent husbands (martyred, imprisoned) 3. Wives of husbands who are disabled 4. Wives who are sick but whose husbands are not able to support and pay for medical care. 5. Wives whose husbands left to Europe. 6. Orphaned children and others without fathers 7. Children of poor or disabled fathers. 8. Teenage boys. 9. Men 10. LGBT refugees (raised by NGO actor only) Street harassment from Lebanese and Syrian youth. 2. Sexual exploitation (including rape) by men to obtain to money for fuel. 3. Mothers may marry off young daughters. 4. Having to frequently be out of the house. 4. Having to clean and become a maid in homes 5. Having to engage in various kinds of "legitimate and illegitimate" employment 6. Being removed from school to become a beggar 7. Street violence and fights, particularly when defending teenage girls from street harassment. 8. Detention.
Coping strategies - men Current participants Previous participants Future participants 1. let things go/ignore upsetting things. 2. Socializing, talking with, and helping others 3. Remaining engaged in activities (e.g. raising pigeons, handymen doing things around the house, planting herbs). 4. Playing with children and taking them out on outings 5. Letting grievances things go. 6. Accepting the overall situation 7. Taking her to see her family 8. Finding activities for the kids (9. Promising children with the hope of returning or promising them with buying things Stay silent. 2. Communicate with others but don t share or tell everything because others are suffering. 3. Smoking 4. Keep wife inside stop socialising outside and only for necessities. Keep silent 2. Go to your family to solve problems 3. Try to change the situation
Coping strategies - women Wives of current participants 1. Praying 2. Crying 3. Keep daughters inside 4. Drink cup of coffee with husband 5. Keep ourselves busy 6. Keep silence 7. Talk to school when children bullied 8. Talking to another woman or group of women Wives of previous participants 1. Do something you love in the morning. 2. Don t sit alone will have flashbacks. 3. Make children sit together and joke and have fun with them 4. Keep busy do something new with your children, tell them a new story or play a new game. 5. Move things around in your home Women in new site 1. Keep busy 2. Drink a coffee on the balcony 3. Check whatsapp or facebook 4. Visit neighbour 5. Pray 6. Do something together as a family
Coping strategies teenage girls Teenage girls (daughters of current participants) 1. Talk to friends on Mum s phone. 2. Listen to music 3. Distract yourself (watch TV, walk around, sit on balcony, eat) 4. Dancing with neighbour s daughter. Teenage girls in new site 1. Talk together, listen, help each other make decisions 2. Smoke shisha 3. Eating 4. Read Koran 5. Sleep during the day 6. Talk to person you trust 7. Talk to self in the mirror 8. Cry 9. Dancing 10. Listening to music 11. Drink coffee 12. Walk outside if allowed to leave 13. Isolate self
Coping strategies teenage boys and young men Teenage boys and young men (current participants and sons of older participants) 1. Contentment with what one has. 2. Ignoring/letting go of others' upsetting talk. 3. Being helped by the support of an understanding wife 4. Socializing with family 5. Hobbies and daily activities (e.g. smoking nargileh, playing backgammon, raising chickens, chopping wood, playing play station) 6. Taking children out for outings 7. Having a special meal for the family at home 8. Consoling wife over hardship 9. Showing wife appreciation through sweet statements (thank you for your efforts). 10. Resort to elders who are wise and are able to contain youth and console them. Because perception that elderly sit at home and have their needs met, so they have higher wellbeing.
Conflict resolution capacity - men Current participants Preferring reconciliation 2. Apologizing when mistaken 3. Friends of that person are tolerant and understanding of the situation 4. Eliciting the help of a community elder Previous participants Ignore 2. Thank him. 3. Avoid confrontation. 4. Stay inside to avoid making mistakes. Future participants Shut up. 2. Ignore. 3. Avoid conflicts.
Conflict resolution capacity - women Wives of current participants 1. Seek support from Lebanese elders (50+) 2. Engage a mediator (e.g. landlord, school principle) 3. Call UNHCR and report 4. Report to police 1. Keep silent 2. Make a coffee 3. Leave the room and talk later. 1. Reason with him 2. Tell the child I am sad 3. Withhold allowance 4. Slap him Wives of previous participants and move away. 2. Talk directly to solve problem. 3. Stay silent 4. Talk later 5. Put children s programmes on TV for them. 6. Distract them. Women in new community Move house to avoid conflict 2. Talk to owner/landlord to mediate 3. Turn to good neighbours but don t tell them 4. Ignore it 5. Discuss the issue 6. Keep silent to reduce conflict 7. Let it go 8. Compromise 9. Cry 10. Don t argue in front of the kids 11. Mediator 12. Don t blame them 13. Encourage don t criticise 14. Hug and kiss your children
Conflict resolution capacity teenage girls Teenage girls (daughters of current participants) 1. Talk to father about it, parents will deal with it if they can. Teenage girls in new 1. Keep silent 2. If possible, tell father and brothers 3. Depend on someone out of the family a confidant 4. Write 5. Draw 6. Hit self 7. Run away from the house 8. Sit alone 9. Early marriage 10. Facebook and whatsapp with Lebanese friends
Conflict resolution capacity boys and young men Teenage boys and young men (current participants and sons of older participants) Avoiding arguments with neighbor, being patient and tolerant. 2. Leaving the house for 2-3 hours. 3. going inside a different room. 4. Take her out on outings, for visits to family. 5. Engaging in more communication, and acknowledging mistake. 6. Breaking household items to avoid hitting her. 7. Cuddling child and kissing him/her. 8. Giving them phone to play with. 9. Giving them candy or other small things. 10. Promise with a nice thing to be purchased later
Indicators of reduction in inter-community conflict When I greet them they would greet me back They would smile at me when I see them in the street Reduction in rents charged to Syrian refugees Less street harassment of young girls by both communities I would visit members of the other community in their homes I would feel safer walking in this area Inter-marriage No shouting or beatings in the streets Groups of friends of Lebanese and Syrians People wouldn t say refugee or Syrian in a bad tone I would be allowed to visit my Lebanese friends home Syrians and Lebanese would communicate more often on Facebook and whatsapp
Conclusion Challenges of participatory indicator development Time needs to be factored into the implementation model ideally before starting implementation. Some sensitive topics are hard to put into words in ways that will be acceptable to ask in a survey-style method Works best as complementary to standardised measures Then why do it? Participants/beneficiaries get to define what they would see as sucessful programme outcomes and impact gives a deeper insight into how your programme should and could be working. Includes the perspective of those that the programme is ultimately accountable.
Some resources UNICEF (2016) Developing and selecting measures of child wellbeing. Florence: Innocenti Centre. UNICEF (2009) Guide to the evaluation of psychosocial programming in emergencies. Geneva: UNICEF.
