What to Expect From Our Impact Assessment Process & Why Is It Important?


6th Nov 2019

What to Expect From Our Impact Assessment Process & Why Is It Important?

What to Expect From Our Impact Assessment Process & Why Is It Important?

Oftentimes social investors, grant-makers and donors are put off by the cost of an impact assessment

But the cost of an impact assessment is directly related to the size of the investment, number of investments made, programs funded, location and the scale & size of these interventions.

In addition, it’s influenced by how much information is available (i.e. strategies, monitoring and evaluation reports) vs. how much information must be gathered and, of course, the number of stakeholders impacted. 

Therefore, it’s important to understand what an impact assessment entails to help put our cost into context. 

What is Impact Assessment?

Impact Assessment is a means of measuring the effectiveness of organisational activities and judging the significance of changes brought about by those activities.

Impact is a measure of the changes made, and impact assessment seeks to establish a causal connection between inputs and changes in terms of magnitude or scale, or both.

The following article aims to shed more light on how an impact assessment is conducted by Next Generation. It follows our methodology - the Impact Investment Index which was developed over 10 years ago.

Why do Clients Conduct Impact Assessments?

Clients have specific reasons why they want to conduct impact assessments. Some of these reasons include that they want to:

  • Know what works or doesn’t, and why
  • Know if they’re following the right strategy, have the right partners, are implementing in the right way and are achieving the intended impact as they envisaged
  • Know how they compare to other funders
  • Know what they can do to improve
  • Know whether they must continue to fund an intervention or exit instead
  • Report to various stakeholder groups on the difference their investments and interventions make
  • Test their strategies, program management and operations and they want to test the effectiveness of their programs.

The Three Levels of Impact Assessments

It’s due to the reasons listed above that we’ve set out to conduct impact assessments on three levels.

Strategic

To test aspects of strategy, vision, objectives, targets, governance, mandate and intent.

Operational

To test aspects of effectiveness & efficiency of implementation - including performance processes, systems, frameworks, guidelines.

Programmatic

To test aspects of materiality, relevance, sustainability, impact & return on investment.

What Are the Variables We’ve Discovered Throughout the Past Decade?

Over the past ten years we’ve encountered several variables, including:

  • 90% of our clients conduct impact assessments at the end of a funding cycle, generally after 3 to 5 years. These clients want to conduct impact assessments on all their investments and development portfolios because they want to understand and compare the various portfolios & focus areas. In other words, they want a collective and integrated overview of their entire investment portfolio in order to inform their new investment strategies. Futhermore, an impact assessment is a reflection of impact at a specific point in time.  Therefore, some of the programs we assess have been concluded, some are still ongoing, and in some cases clients want to conduct a pre-emptive assessment to forecast future impact
  • 10% of our clients conduct impact assessments of particular interventions or portfolios such as flagship or signature initiatives or specific portfolios which include a number of focus areas such as enterprise development, skills development or youth development.

However, not all our clients allocate funds to long term interventions.

In general, investors have a mix of long- and short-term (1 year or once-off) programs, they have 4-5 investment portfolios and themes. Not all contributions are financial investments. Sometimes their contributions include books or equipment, infrastructure (hospitals, clinics, schools), training, volunteer contributions such as products and consumables, skills or time.

What’s the Process?

Impact Planning

At the start of the assignment a workshop is conducted with the client. This is done to confirm the scope and brief of the assignment, determine which portfolios & programs are to be included, to obtain information and to map out the process flow.

It’s important to note that each assignment is different because client’s reasons for doing the assessment varies. 

Some of the reasons why investors and funders would conduct impact assessments include:

  • To understand which of their investments yielded the highest impact and return on investment
  • To understand and identify which programs didn’t perform as expected and why
  • To get an understanding of the breadth and depth of impact achieved
  • To test the validity of their strategy
  • To inform the next cycle of funding
  • To forecast future impact and to identify indicators & targets that can be measured against in the future
  • To have information and data that could inform reporting

During the planning impact stage, we also require the information that will guide and inform our process.

Generally, the planning process takes a month because we’ve come to realise that some of the information we may require may not be documented. For this reason, we work with the client to confirm or co-design the impact strategy, theory of change, identify indicators and develop key questions.

The follow diagram is provided to share some of our expectations and processes:

Depending on the number of projects and stakeholders this process is aimed at obtaining baseline data that’ll inform the impact outcomes and reports.

Generally, this process takes an additional thirty days - depending on the availability of stakeholders, the number of stakeholders and the availability of data.

Workshop

  • Identify the portfolios and programs to be assessed
  • Confirm the scope of the assignment
  • Confirm the outcomes and deliverables of the assignment
  • Obtain information to conduct the impact assessment

Information required

  • Impact strategies, mandates, objectives
  • Theories of change and logic model frameworks
  • Lists of internal and external stakeholders
  • Lists of budgets and programs

Project planning

  • Once all information is reviewed a project management plan is developed
  • Communication tools such as letters, the impact assessment questionnaire is designed and signed off by the client
  • The assessment questions is confirmed
  • And the first data is recorded - from indicators to program details, objectives and deliverables

Impact Management

During this process, we work on three levels: strategic, operational and programmatic.

We have a customised tool with 75 benchmarks that’s used to guide the internal engagement process. 

This process is aimed at understanding what the impact intent and objectives are, how impact is managed through implementation, and how programs are managed.

Internal Engagement

  • Engaging with internal stakeholders to obtain impact objectives and intent

External Engagement

  • Engaging with external stakeholders to obtain impact data and evidence

Strategic, operational and programmatic assessment

  • Assessment of internal and external impact drivers, influencers and determine effectiveness and efficiency

Impact Measurement

This process speaks to the data collections process phase during which we engage with the implementing organisation, and the recipients and beneficiaries of the interventions.

This may require travel as we visit the actual sites, conduct focus groups, look for evidence and engage directly. We also begin documenting the impact and digitising the various pieces of information.

Once all the information is collected it’s loaded on our system and impact dashboard. 

Once it’s on our system, we identify the impact indicators, organise the indicators according to impact dimensions and load additional data that speaks to the actual impact & return on investment achieved. Only once we’re satisfied that the data represents what we’ve observed do we start preparing impact scoresheets and analyse the impact data.

This is the most time-consuming part of the assignment. The validated and verified data is confirmed and analysed. The entire social impact assessment process comes together as data starts turning into impact stories.

Stakeholder engagement and site visits

  • Impact questionairres are completed
  • Interviews and focus groups are conducted
  • Data is recorded

Data collection, verification and validation

  • Data is collected, verified, validated and recorded
  • Indicators and impact/return dimensions are prepared
  • Data is collated according to geographic locations, portfolios, themes and focus areas

Data analysis, synthesization and triangulation

  • Data is analysed from the ground up.  From individual programs, to focus areas, to portfolios to geographic locations
  • Data is analysed according to dimensions of impact and return and scored
  • Cost benefit analysis is conducted
  • Scoresheets are prepared

Impact Reporting

The final stage of the process is when the impact management report is prepared. During this stage quality assurance is conducted and the information is prepared for presentation and handover to the client. 

This process takes about 30 days.

Depending on client requirements, various impact reports can be prepared. For some the information is presented to boards and executives, for others impact stories are created for specific stakeholder groups. This is also supported by deep conversations that speak to our findings, conclusions and recommendations. 

At this stage we conclude the process not only with a deep analysis of impact and return on investment assessment, but detailed guidelines on a strategic, operational and programmatic level for future decision-making.

Cost benefit Analysis

  • During the quality assurance process the impact data is verified again
  • Cost benefit and effectiveness studies are concluded

Conclusions and recommendations

  • Final analysis regarding future impact and return is prepared
  • Conclusions and recommendations for strategic, operational and programatic practices are prepared

Presentation and Reporting

  • The management report is finalised and the electronic dashboard is prepared for the client
  • The dashboard becomes the clients data visualisation of the impact and return achieved
  • The report is shared with the client and, if required, additional reports for specific stakeholder groups are prepared and handed over

In Conclusion

On average an impact assessment can take between 3 and 6 months to complete. This is dependent on:

  1. the number of portfolios and focus areas
  2. the number of stakeholders
  3. the number programs
  4. the location of the interventions
  5. the size of the investment.

We started this article on the premise that investors think impact assessment is expensive. Yes, that may very well be the case however, the benefits far outweigh the costs involved. 

Because whether you invest US$ 1 million or US$ 100 million, if the investment is ineffective, or the negative impact outweighs the positive impact, or your reputation is not enhanced and—most importantly—lives aren’t impacted and no change occurs. It doesn’t matter how much is invested if you don’t know if your impact intent and objective has been met.

Our clients testify that the value of the impact assessment report is so much more than just numbers. It provides credibility and assurance of what actually happened. It provides a baseline for future assessments and critical insights that inform future investments.

Importantly though, beneficiaries and recipients have noted that they feel valued and part of the process, they feel that their story is recognised, and their voices heard.

Implementing organisations felt they had an opportunity to share their success and challenges with an unbiased and independent third party. 

Lastly, an impact assessment confirms and validates investment effectiveness, it confirms impact strategies and provides a guideline for a more impactful future.