In this month’s blog entry, we spoke to Eileen Lambourne – the Impact Evaluation Lead for the SPRING evaluation. Eileen reflected on how we devised an adaptive approach to measure the impact of SPRING, and on the lessons from the fieldwork.

The Impact Evaluation was one of the three components of the SPRING evaluation. It focused on understanding how and to what extent girls benefited from the products, services, and income-generation opportunities provided by the SPRING businesses. As part of this evaluation component, the evaluation team completed trips to Rwanda, Pakistan, Nepal, Myanmar, and Kenya to meet the girls at the heart of SPRING.

SPRING was adaptive in its operation and had several interventions, which in turn changed and adapted during the implementation. As evaluators, we used an adaptive evaluation approach to best capture the impact of the interventions. We applied different methods to capture the nuances of the different situations and various contexts in which SPRING businesses operated.

RWANDA: SHEKINA

In some SPRING impact evaluations we included an ethnographic component. In Rwanda, between May 2016 and November 2018, the evaluation team completed the Impact Evaluation (IE) for Shekina – a cassava-leaf processing business – providing employment to girls through a cooperative-based model. Shekina hoped that earnings from the cooperatives would help girls generate income and save, helping them resume their studies or begin their own business.

As part of the IE, we visited the cooperatives to observe the girls in their workplace. We also visited their homes to speak with them directly, and to meet with their parents. We wanted to understand how the interaction with Shekina impacted girls’ lives.
We learned that girls’ earnings had not increased since joining Shekina. But, we also found out that girls’ earnings impacted the household, benefiting their families as girls started contributing to household expenses. Shekina girls also reported enhanced leadership skills and a tendency to save. In addition, a majority of girls felt that Shekina offered greater professional learning opportunities than previous work environments.
From the fieldwork, we learned about how important it is to:

•  Triangulate findings. Findings reported from the business, the girls and the parents were dissimilar and told different stories. We learned that looking at evidence from multiple sources was essential for understanding the bigger picture.
•  Understand the context. Shekina’s was a complicated intervention, which presented difficulties in terms of communications and management. Going to the field was crucial to understanding this operating context.
  
  NEPAL: FIGHTBACK

Adapting the IE approach to understand the impact of light-touch interventions was the challenge we faced during the impact evaluation of Fightback in Nepal. Fightback delivers 3-day safety awareness and self-defense training to girls. During our evaluation, we visited four schools to observe the training and the girl participants.

To determine the effectiveness of Fightback’s intervention, we asked the girls to complete a survey. Our quantitative approach strongly confirmed Fightback’s theory of change: 92% of those surveyed reported an increased ability to escape from possible threats as well as an enhanced feeling of safety and alertness and 88% reported feeling more confident. We triangulated these findings with direct observation of the in-school training, as well as KIIs with teachers, parents, and girls themselves. This qualitative feedback was a great source of data, allowing us to interpret the raw survey figures with personal insights.

“If someone follows me, I would first change direction and see if the guy is still following. If yes, I would look back and look into his eyes and wait for him to go first. In public transport, if someone tries to come close to me, I’ll create distance by using my hand and placing in my waist. For anything more than that, I’d shout and call the conductor for support.”
KII Girl 11 Endline

Impact evaluation is the raison d’etre of any intervention – to see if a programme has made a difference. The impact story can be told in many ways; through figures, through observation, and through personal accounts, to name three. Our impact evaluation used a range of methods to gather a variety of data to try and tell the most complete picture of the SPRING impact.