Revolutionising monitoring and evaluation through impact measurement
The advent of the SDGs, growing dissatisfaction with current economic models, the explosion of impact investing, impact measurement and other innovative finance models coupled with the potential that the fourth industrial revolution presents and the growing ubiquity of digital technology has created the context in which an overhaul of current monitoring and evaluation practices is required.
Robert Picciotto says in his book, The 5th wave: Social impact evaluation: “The evaluation discipline has yet to adapt its methods and processes to the dynamic pace of decision-making favoured by new actors, technologies and ecosystems.”
But things are starting to change. The OECD has acknowledged that the current approach to monitoring and evaluation is failing as evidenced by their decision to review their evaluation standards and draft new criteria for evaluation. This process kicked off with extensive stakeholder engagement that showed that the current approach is ineffective and increasingly irrelevant within the current context.
New approaches to evaluation such as participatory evaluation, democratic evaluation, empowerment evaluation, horizontal evaluation, realist evaluation, utilisation focused evaluation, culturally responsive evaluation seem to be created monthly by leading experts which also suggest that existing approaches may not serve different contexts.
In addition, a number of impact forecasting and modelling approaches are also being developed. For instance, the International Finance Corporation are in the process of piloting a new rating system designed to assist with better allocation of finances and support its existing Development Outcomes Tracking System. The Anticipated Impact Measurement and Monitoring (AIMM) system offers an “end-to-end” framework to help the IFC decide with projects to support based on a range of quantitative and qualitative information, indicators and estimates about the project’s likely impacts on poverty alleviation and market creation.
These moves by influential players in the sector point to the growing need for a global ‘gold standard’ for monitoring and evaluation that takes into account the shifting context and accommodates the needs of both the traditional development sector as well as the growing impact investment sector.
What is the difference between M&E and impact measurement?
M&E has historically looked at monitoring programme implementation and evaluating the outcome of individual programmes.
Impact measurement aims to manage and control the process of creating impact to maximise impact while managing costs.
Key trends
- The SDGs requires M&E and impact measurement practice that speaks to the 17 goals and the 230 sub-indicators, at a country level, a regional level as well as a specific theme.
- Demand for indicators that measure both financial and social/environmental impact that also aligns with or considers new and emerging innovative/blended financial models are gaining ground in the sector.
- The need for standardised, appropriate and effective methodologies and toolkits that respond to the performance management and measurement requirements of a range of new actors operating in the development sector (i.e. venture philanthropists, ESG investors, impact investors, social investors, grantmakers/donors and philanthropists, etc.) is driving the large range of new and customised measurement methodologies and approaches currently being explored.
- Until now traditional M&E practice focused on developmental outcomes, now there is a much greater focus on and need for financial indicators to evaluate the financial value and impact of social and impact investments.
- There is now a need for new methodologies that look for the contribution effect (that tracks collective, attribution, contribution and leverage of multiple contributors and funders) rather than the attribution of a single funder.
- Nuanced data is now required that speaks to the entire investment cycle, not just the programme cycle. In this regard – performance data that will inform due diligence decisions, impact management as well as impact measurement data, is critical. Traditionally the focus has only been on baseline and end line evaluation data.
- New data collection approaches that use the strengths of digital technologies to capture data has become a necessity to collect more data, more cost effectively and efficiently.
- Ethics and principles that guide behaviour, is an especially important topic in evaluation practice. Principled action and the protection of human rights, data privacy and security have become serious considerations for all stakeholders in the value chain. From the commissioners of research and evaluation, to practitioners and professionals in the sector, evaluation outcomes rather than being prescribed or predetermined by funders and investors.
Roberto Picciotto also distils how these trends translate into key policy directions for the sector:
Key policy directions
- Internationalisation: not just across borders but M&E will increasingly need to be adapted to local contexts while remaining aligned to global standards. This will be supported by the growth of global evaluation associations and networks and increasing professionalisation of the sector.
- Diversification: as the development sector attracts new actors, monitoring and evaluation will need to become more inclusive of a diverse range of needs and stakeholders.
- Digitisation: programme funders and implementers will no longer accept slow and lengthy M&E processes and evaluators will be under increasing pressure to the use new technology, and big data approaches to report on impact quickly and more regularly.
M&E and impact measurement is on a precipice and needs to evolve to take into account the multidimensional and complex qualitative context of development work and ensure that it speaks to the overarching goals of the SDGS. M&E must also start to account for the growing tendency for programmes to be implemented across borders and by various stakeholders including the public sector, private sector and philanthropic funds while also tracking more long-term, secondary and indirect outcomes.
Currently M&E and impact measurement are seen as two distinct subject areas. What we can see is that both sectors are dealing with the same trends. In the Next Generation 2020 Research Report Disruption with Impact we advocate for these two branches to come closer together to offer a more cohesive approach to monitoring and evaluation.
With the growth in impact management and measurement approaches (IMM) – there are several implications for practitioners in the sector, most notably:
- New guidelines, frameworks and standards – such as the IFC Principles for Impact Management and Measurement or the Impact Management Project
- New practice requirements, including:
- Investment period – longer investment periods will require longer evaluation periods and even longitudinal studies
- Investment portfolios – integrated portfolios and investment/development themes (such as renewable energy or climate change adaptation and mitigation) will require different skills, expertise and experience
- Investment size – innovative new development finance models will require professionals to evaluate investments in millions and billions and multiple organisations at the same time – not just investments in single interventions managed by single intermediary organisations
- Forecast impact – whereas traditional evaluation practice focused on outcomes and impact that have taken place, impact investors want to use predictive and forecasting models to determine future impact and return
- Manage and measure impact and return – previously impact was assumed – now practitioners will also be required to measure return on investment
- Report impact and return on investment – it is standard practice that evaluation reports are delivered to investors and funders, however, in a new age of transparency, impact reports will be made public
- Compare and benchmark – to fully understand the impact achieved, research questions and hypothesis will have to consider and explore comparative as well as benchmark data (across geographies or industry sectors) – and not the outcomes of single interventions
- New language – impact investors are using different terminologies – whilst there are comparisons to be drawn between social investment and development practices, concepts such as investment thesis, impact mandate, investees will require traditional evaluators to also change their language.
- New methodologies and new metrics/indicators – the development of global indices will not only have an impact on how impact measurement and evaluation is conducted – but also what gets measured. In order to benchmark and compare, uniform indicators must be considered. The development of IRIS+ for instance, brings much greater clarity on what to measure and what unit standards to use in the measurement of impact.
Read more about how Next Generation developed an integrated and end-to-end approach to impact management and measurement over the past ten years. The Investment Impact Index™ was developed over a ten-year period and is aimed at taking evaluation practice to the next level by providing investors with a multidimensional view of impact and return on investment.