How can an organization help make a major road more resilient to floods if it does not build the road? It can advise planners, train government officials, and involve local communities, while others make the final decisions.
The government makes decisions, a development bank finances the project, engineers develop the design, and local authorities and communities influence what gets built. If the road becomes more resilient to floods, how do we understand one organization’s contribution when many actors shape the result? How did the actors’ decisions interact, and did the support justify the resources used?
MSC faced these questions as we evaluated the Global Center on Adaptation’s (GCA’s) in Bangladesh, supported by Global Affairs Canada (GAC).
GCA intended to influence larger investments rather than directly finance major infrastructure. In Bangladesh, its portfolio included nine projects that span locally led adaptation, resilient infrastructure and urban systems, nature-based solutions (NbS), and climate adaptation finance. Seven of these projects were linked to larger investments supported by multilateral development banks (MDBs) and other international financial institutions (IFIs).
GCA worked with government agencies, MDBs, and IFIs during the early project preparation phase to integrate climate resilience into large-scale infrastructure projects. It conducted climate risk assessments, prepared adaptation solutions, provided technical advice and capacity-building support, and contributed to locally led community planning.
The portfolio’s size created only part of the evaluation challenge. How could MSC assess GCA’s influence in a complex multistakeholder ecosystem? GCA provided only part of the support. Other actors influenced whether that support was used, approved, financed, and implemented. GCA’s theory of change recognized this reliance on other actors in the ecosystem. As a result, MSC customized the evaluation approach and learning questions to GCA’s expected change pathway. The team combined project documents, performance data, and stakeholder perspectives to examine each part of the results pathway.
Why do conventional evaluation designs only tell part of the story?
Climate adaptation and NbS depend heavily on context and can take years to show their full effects. A flood-resilient road, an urban drainage plan, or a nature-based intervention depends on climate conditions, government decisions, financing, maintenance, institutional capacity, and community participation. These factors also interact, as technical advice can shape an investment only if partners use it, decision processes incorporate it, and institutions can carry it forward.
Evaluation literature has long recognized these challenges.
The Organisation for Economic Co-operation and Development’s (OECD’s) 2019 research identifies attribution, baseline shifts, and long-term effects as shared challenges in climate adaptation evaluation. This research also states that a comparison of conditions before and after an intervention may be insufficient because climate risks and the wider social and institutional context continue to change.
The same concern appears in the literature on NbS. The International Union for Conservation of Nature (IUCN) Global Standard for Nature-based Solutions uses broad criteria to assess NbS interventions beyond a single outcome. The standard considers the societal problem alongside ecological integrity, economic feasibility, inclusive governance, trade-offs, adaptive management, and the intervention’s ability to last and scale.
This breadth mattered to GCA because it did not implement standalone NbS activities. It sought to embed climate resilience, including nature-based approaches, within larger investment and planning processes. Its contribution could appear first in a revised design, technical assessment, planning method, stronger institutional capacity, or greater consideration of local priorities. Benefits for communities could come later.
The evaluation, therefore, needed to follow this pathway rather than rely on a single measure of success.
How did the evaluation approach respond?
Figure 1: OECD’s Development Assistance Committee (DAC) criteria, Contribution analysis (Mayne, 2001; 2008) and Value for money (OPM, 2019)
Across all three approaches, MSC triangulated documents, performance data, and stakeholder perspectives. This evidence distinguished verified findings from early signals and outcomes that still depended on future uptake and implementation.
What did this mean in practice?
Each part of GCA’s change pathway required different evidence, so a customized design was essential. GCA had completed some activities, but the investments it sought to influence were still developing. The evaluation, therefore, did not treat future community-level benefits as if they had already occurred. Instead, it examined what could be verified at the time and whether credible signs showed that partners had adopted GCA’s approaches going forward.
The evaluation examined what happened after delivery of a technical product. For example, the value of climate risk assessment depended on what followed: Were its recommendations discussed? Did they enter planning or design decisions? Did the relevant institutions have the ability and incentives to use them?
Context also shaped value. A low-cost study offers little value if the decision-makers cannot use it. Similarly, training may appear efficient when it reaches many participants, but contribute little if they do not apply the knowledge. The evaluation therefore considered resources alongside quality, timeliness, results, and inclusion.
This is where customization becomes important. Complex evaluations rarely require a completely new method. They usually need established approaches, applied to the right questions and at the right point in the results pathway.
Customizing complex evaluations to context
The GCA assignment illustrates a broader principle that applies to many complex, multi-stakeholder programs: Evaluation design should align with how a program expects change to occur.
Programs that work through partnerships, technical assistance, policy influence, institutional change, or larger investment systems often face similar challenges. Multiple actors shape results. Some outcomes appear much later than others. The organization being evaluated may influence decisions without directly controlling implementation.
In such settings, when evaluators first select a standard design and then fit the program into it, important questions can go unanswered.
MSC starts with the program’s change pathway, identifies where evidence can realistically be generated, and customizes the evaluation design accordingly. The established methods can then be combined to address different parts of the problem.
This approach makes the evaluation fit the program without adding unnecessary complexity. These three lessons stand out:
- Start with the change pathway: Evaluation questions should reflect how the program expects to influence decisions and results;
- Match methods to questions: Performance, contribution, and value are related, but each question may require a different approach;
- Keep claims proportionate to the evidence: Separate what can be verified from what still depends on future uptake, financing, or implementation.
Programs can apply these principles beyond climate adaptation wherever they work through several institutions, shape a chain of decisions, and influence the outcome without delivering it directly.
For a program that influences a road without building it, a customized evaluation can answer the question posed at the outset. It can trace how change happened, clarify the program’s role, and show what evidence supports the conclusion.
