The pressure to measure impact

It seems everyone is talking about impact and is looking to achieve and demonstrate impact. 

Investors talk of ‘impact investing’, social investors talk about social impact, and social purpose organisations are talking about collective impact.  The international development community is also increasingly preoccupied with impact. Since the early 2000s, the terms ‘impact’ and ‘impact evaluation’ have skyrocketed in use and have become common among development practitioners and development agencies.

Development practitioners have long sought to achieve impact with their work; and they are often highly and intrinsically motivated to create change.  Measuring and demonstrating impact, however, has not always been high on the agenda. With hindsight, this may seem odd but for decades, development assessments were dominated by outputs – such as the number of trainings held, or the value of goods distributed. Showing you were doing what you promised seemed to be sufficient for funders, and impact was more or less presumed to follow.

The growing importance of demonstrating impact in international development over the past 10-15 years has been driven by several economic and political factors: recently there has been a reduction or retargeting of development budgets in many donor agencies. There has also been a drive among major philanthropists and donors for greater demonstration of ‘value for money’ and wanting to get more ‘bang for their buck’. At the same time, there is an increasing public perception that five decades of development assistance have not had effects as hoped. This perception has pressured social investors to do a better job of demonstrating clear, tangible results that can be understood by the public.

The evidence-based policy movement, which has gained momentum over the past few years, has led to more systematic examination of the main assumptions underlying development work. 

Together, these trends have led to much greater attention among development actors to measure and demonstrate what works more and less well, and to use this knowledge to leverage greater effectiveness from development programmes. Impact has become the watch word for this era in social investment and development.

Defining impact

The Oxford English dictionary gives two definitions of the word impact: 

• ‘the action of one object coming forcibly into contact with another’ and 
  
• ‘a marked effect or influence’.

These alternate meanings describe fundamentally different views of causality and evoke very different images. This mirrors how the term is used in international development; the two most widely used definitions of impact show a similar divergence of perspectives:

• The counterfactual definition from statistics and econometrics defines impact by the measured difference of a predefined indicator (e.g. school test score) with the intervention and without the intervention. 
  
• The OECD-DAC definition, on the other hand, defines impact much more broadly in terms of any long-term effect, whether intended, unintended, positive, negative, direct or indirect. In other words – the former understanding looks for ‘causes of an effect’ and the latter looks for ‘effects of a cause’.

Why is this a problem?

In programme design, discussions about impact set the aspirations for a programme and help to build cooperation, inform investment decisions, and are used to identify risks. During or after a programme, the way impact is defined will affect how its success or failure is perceived, and who takes credit or blame. And this then affects what we learn from the programme to help adapt it or contribute to other programmes.

The Impact Management Project (IMP) defines impact as follows: Impact is a change in positive or negative outcome for people or the planet. The IMP has built consensus that to understand any impact, we need to understand five dimensions of performance: What, Who, How Much, Contribution and Risk.

• What tells us what outcomes the enterprise is contributing to and how important the outcomes are to stakeholders.
  
• Who tells us which stakeholders are experiencing the outcome and how underserved they were prior to the enterprise’s effect.
  
• How Much tells us how many stakeholders experienced the outcome, what degree of change they experienced, and how long they experienced the outcome for.
  
• Contribution tells us whether an enterprise’s and/or investor’s efforts resulted in outcomes that were likely better than what would have occurred otherwise.
  
• Risk tells us the likelihood that impact will be different than expected.

In a paper produced by the Overseas Development Institute (ODI), and BetterEvaluation (BE) the following guidance is provided on impact.  The paper identified six dimensions of impact and pose guiding questions to help stakeholders clarify their interpretation of impact to come to a shared understanding for a programme and context.

1. Application.

Impact is a concept that is applied both prospectively and retrospectively. For example:

• In an environmental impact assessment, impact refers to the potential effects of an intervention on the environment and can be used to help decide whether or not to proceed with a planned course of action. 
  
• In programme planning stages, impact can refer to the intended or desirable effects. 
  
• In an impact evaluation, impact refers to measured or observed effects of an
  intervention, which could help decide whether to stop, continue, scale up or adapt the intervention.

Which application are you referring to when you use the term impact?

2. Scope.

Impact can be defined in terms of very specific changes or it can be broad and open.

• Specific impact focuses on a fixed number of pre-defined variables, such as household income, disease status or air quality, and statements of impact discuss about impact according to these variables. 
  
• Broad impact is not limited to pre-defined variables but considers as many changes as makes sense to make a useful judgement (and are feasible to measure), including variables that may not be foreseen.

Are you looking for impact on specific variables or will you include unintended or unforeseen effects?

3. Subject and level of change.

Among the agency definitions the UN, World Health Organisation, UNAIDS and 3ie refer to changes in people’s lives. 

• Yet development programmes are increasingly focused on more mezzo or macro levels, intervening with groups, institutions and policies, rather than or in addition to individuals. As mentioned above, several agencies also include the environment as a potential subject of change.

Where are you looking for impact?

4. Degrees of separation

Degrees of separation between intervention and impact, as illustrated through results chain or logic model, which is related to the subject and level of change. 

• Interventions operate at different distances from individual beneficiaries. For example, there is a direct, immediate link between a programme providing bed nets or vaccinations and individual beneficiaries.
  
• In contrast, institutional capacity building programmes have several degrees of separation between the intervention with staff and institutional policies and end users, such as patients attending health clinics. 
  
• In the former, impact is relatively linear and the pathway is direct. In the latter, impact is more systemic; it confronts and converges with other factors (contextual or programmatic), which, can either resonate and produce greater effects or they can disturb each other producing chaotic effects.

How direct is the causal chain? How far from the intervention do you expect to see impact?

5. Immediacy, rate and durability of change.

Many agency definitions refer to long-term change.

• But how long is long-term? Different ‘arenas of change’ will have more or less rapid manifestations of impact. 
  
• In some cases, impact may be more immediate – an accident caused by malpractice in an infrastructure programme can have very direct and immediate consequences for those involved. 
  
• A conflict prevention programme, on the other hand, may take years or decades to have an appreciative and observable effect. 
  
• In addition to the length of time, impact may not be static; assessments of impact may come back with different results at different times. A vaccine may provide immunity for life. Information and education cannot be subsequently unknown or unlearned. In unstable environments, however, hard-won successes can quickly unfold and the situation can change very quickly.

How soon are changes likely to manifest? Are they permanent or temporary? How variable is impact likely to be over time?

6. Homogeneity of benefits.

Impact can be measured as an average effect across a population. Or, it can consider positive and negative effects separately and disaggregate among different population groups and contexts.

Among whom are you looking for impact? How will the impact of the programme vary across subgroups? How will mixed results be judged?

Each of these six dimensions has specific implications for what, when and how frequently change is measured.

Moreover, these dimensions and measurement implications are related. Assessing impact across multiple subjects and levels of change, among whom benefits may vary by subgroup, will require longer amounts of time to observe changes and disaggregation of results.

In conclusion:

Currently there is too much ambiguity and confusion about what ‘impact’ is, how it should be defined, how to measure it and what kind of measurement is sufficient.  The purpose of this article is not to propose a single, universal definition of impact or to debate existing definitions. Instead, we examine how some definitions focus on very specific and precise understandings of impact while others cast an extremely broad net. 

Our aim with this article is to elevate the discussion about impact, moving beyond the methodological debates that have dominated attention paid to impact so far, and present different perspectives and dimensions that can affect how impact could be framed and evaluated.

Rather than arguing which definition is universally superior, we encourage development and investment practitioners and professionals to structure an explicit conversation about how different stakeholders conceive of and are using the term impact in order to come to a shared understanding.