Over the last 20 years, theory-based approaches — under various names — have increasingly moved into the mainstream of thinking and practice about how interventions i.
During that time, theory-based approaches have demonstrated promise in helping evaluators address a variety of challenges, such as coming to terms with the inherent complexity of certain types of interventions and overcoming the limitations of experimental evaluation designs.
To help ensure that evaluation continues to play a key role in providing a neutral, evidence-based assessment of the value for money i. Theory-based approaches to evaluation represent a potentially powerful tool.
This document introduces some of the key concepts of theory-based approaches to evaluation. It is hoped that readers will be encouraged by the information and advice provided in this document and will explore the use e.
To support this, Sections 1.
This document is neither an exhaustive training program in theory-based evaluation nor a step-by-step guide to undertaking a theory-based evaluation.
Evaluators who wish to integrate theory-based approaches into their practices are encouraged to pursue additional readings including those referenced in this document and, as appropriate, to seek additional support in undertaking a theory-based evaluation.
John Mayne. The Centre of Excellence for Evaluation would like to thank those departments and agencies that reviewed and provided comments on draft versions of this document.
As noted in Program evaluation methods: Measurement and attribution of program results Treasury Board of Canada Secretariat, chap. Experimental evaluation designs aim to address both of these challenges chap.
These designs typically measure both the baseline and the final results associated with an intervention and, by incorporating a counterfactual e. Experimental and quasi-experimental evaluation designs can be quite powerful and should be undertaken when appropriate e. However, there are several shortcomings associated with these designs, in particular:. A theory-based approach to evaluation can help address these shortcomings. In the absence of an overall experimental design, it provides a way to assess the extent to which an intervention has produced or influenced observed results.
It also opens the black box, examining what role the intervention played in producing the observed results.
Theory-based approaches to evaluation use an explicit theory of change to draw conclusions about whether and how an intervention contributed to observed results.
Theory based evaluation is an approach to evaluation i. It is a way of structuring and undertaking analysis in an evaluation. A theory of change explains how an intervention is expected to produce its results. The theory typically starts out with a sequence of events and results outputs, immediate outcomes, intermediate outcomes and ultimate outcomes that are expected to occur owing to the intervention.
This opens the black box of change and allows evaluators to better examine the causal link between the intervention outputs and the observed outcomes. The theory of change can be used to test — with evidence — the assumed causal chain of results with what is observed to have happened, checking each link and assumption in the process to verify the expected theory. Another way to think of a theory of change is as a logic model that has been described and explained, in particular in terms of the causal linkages between outputs and the different levels of outcome.
While there is little agreement either on the terminology or the concepts, there is consistency in the main messages and agreement on the value of theory-based approaches. Two key ideas that distinguish theory-based approaches from traditional approaches are 1 the influence of context on program results, and 2 a mechanistic, rather than counterfactual, approach to determining causality.
Theory-based approaches, more than many other evaluation approaches, pay explicit attention to the context of the intervention. It is acknowledged that contextual factors can help an intervention achieve its objectives or act against the intervention working.
For example, in an intervention aimed at reducing smoking in the population, contextual factors may include the enthusiasm of those required to enforce a ban, the social makeup of the targeted populations, and related supporting legislation. These factors are often essential in making causal inferences and need to be part of the evaluation design. When developing a theory of change, the context may be explicitly identified i.
For most interventions, there are usually multiple causes for an observed outcome. A wide range of other economic and social factors, not to mention other government interventions, may also come into play.
For example, the success of an allowance-based incentive program intended to encourage nurses to live and work in rural areas may be influenced not only by the existence and size of the allowance but also by employment rates in the nursing field, the specific rural settings in which nurses are placed, and the personal backgrounds of the nurses involved.
Rather, the relevant evaluation question is: In light of the multiple factors influencing a result, has the intervention made a noticeable contribution to an observed outcome and in what way?
Understanding contribution , rather than proving attribution, becomes the goal. In theory-based approaches, the specific steps in a causal chain, the specific causal mechanisms, are tested. If these can be validated by empirical evidence, then there is a basis for making a causal inference.
Theory-Based Approaches to Evaluation: Concepts and Practices
At the same time, theory-based approaches seek to identify and assess any significant influencing factors i. Managers of federal programs regularly use logic models to describe the results expected from an intervention.
Footnote 2 Where program managers have not developed their own logic model, evaluators often do so to support their evaluation effort.
The results chains embedded in logic models are key building blocks for developing theories of change.
Theories of change expand on results chains to articulate why the sequence of results is expected to occur, whereas logic models tend to focus solely on the results intended by a program i. Simply put, theories of change explain how the intervention is expected to bring about the desired results rather than just describing the results. This figure illustrates the differences existing between the short-cycle logic of theories of change that is, the differences between the levels of a results chain or logic model and the long-cycle logic of results chains from activities to ultimate outcomes, in logic models.
In the centre of the figure, there are a set of five boxes arranged vertically, representing the results chain. The boxes represent the elements normally found in a results chain or logic model and are labelled, from bottom to top, activities, outputs, immediate outcomes, intermediate outcomes and ultimate outcomes. The boxes are connected with arrows that point upward, from the lower boxes to those above them, representing the movement in the results chain from activities, to outputs, to immediate outcomes, to intermediate outcomes, to ultimate outcomes.
On the right-hand side of the results chain, there is a large oval loop, which starts at the bottom box, which is labelled "activities," and ends at the top box, which is labelled "ultimate outcomes.
The loop is labelled "long-cycle logic," also referred to as program logic. On the left-hand side of the figure, there are four smaller loops arranged vertically and consisting of two curved lines, each ending in an arrow.
The loops represent feedback and are situated so that each aligns with the spaces between the five boxes in the results chain. The loops are labelled, "short-cycle logic," also referred to as the theory of change. Theories of change are referred to by a variety of names, including program theories, impacts pathways, and pathways of change. Assumptions are key events or conditions that must occur for the causal link to happen.
Risks are influences or events outside the intervention that may inhibit the causal link from happening. Mechanisms are the causal processes that enable the program to produce results. In some cases, theories of change are subdivided into two components: the intervention theory, which outlines the underlying behavioural assumptions the mechanisms behind the intervention, and the implementation theory, which identifies how an intervention is expected to operate and trigger these mechanisms.
These components can be developed separately, but are often merged into or developed as one theory of change.
Unfortunately, there is no consistent terminology for theories of change; different authors may use the same term for different concepts. Blamey and Mackenzie discuss this issue. Blamey, A. Theories of change and realistic evaluation: Peas in a pod or apples and oranges. Evaluation , 13 4 , — There is no agreed classification of theory-based approaches; indeed, in recent years, there has been a proliferation of theory-based approaches and numerous variations within each approach.
In this section, two prominent categories of theory-based evaluations, realistic evaluation and theory-of-change approaches, are discussed.
These descriptions are generic and may not always apply. For more information on the similarities and differences in theory-based approaches, readers can consult Blamey and Mackenzie or Stame Realistic evaluation is a form of theory-based evaluation developed by Pawson and Tilley , They argue that whether interventions work depends on the underlying mechanisms at play in a specific context.
For Pawson and Tilley,. Mechanisms describe what it is about the intervention that triggers change to occur. In a smoking cessation intervention, for example, mechanisms might include peer pressure to stop or to not stop, fear of health risks, and economic considerations.
Theory based evaluation pdf file
For realistic evaluators, the key evaluation questions are, What works? For whom? In what circumstances?
In what respects? Realistic evaluators are less interested in the outcome-level question, Did the intervention work at a macro level?
Realistic evaluation develops and then empirically tests the hypotheses about what outcomes are produced by what mechanisms in what contexts. The realistic approach tends to be more research-oriented, focusing on the underlying intervention theory and its behavioural assumptions at work, and the conditions supporting the intervention.
The focus is on the most promising context-mechanism-outcome configurations CMOCs , which show how interventions are meant to work in which populations and under what conditions. These can be viewed as mini-theories of change or links in an overall theory of change of an intervention. Each CMOC is, in effect, the subject of an evaluation and is tested against the available evidence. Blamey and Mackenzie , p.
Step 1: The evaluator, through dialogue with program implementers, attempts to understand the nature of the program: What is the aim of our smoking cessation program?
What is the nature of the target population at whom it is aimed? In what contexts and settings will it operate?
What are the prevailing theories about why smoking cessation services will work for some people in some circumstances? Step 2: The evaluator maps out a series of potential mini-theories that relate the various contexts of a program to the multiple mechanisms by which it might operate to produce different outcomes.
For example, practitioner knowledge and the existing evidence might suggest that focusing the educational component of a midwife-led smoking cessation program on the potential negative effects on babies in utero will be most effective for pregnant women who have no children.
However, young, non-pregnant female smokers may be less likely to respond to concerns about the threat of health effects on non-existent babies, but may be more likely to respond to anti-smoking interventions designed to appeal to their self-image. Step 3: At this stage, the evaluator undertakes an outcome inquiry in relation to these mini-theories. This involves developing a quantitative and qualitative picture of the program in action.
It might, for example, address how different types of smokers fare when it comes to breaking the habit following different types of cessation services delivered in a variety of ways. This picture includes an assessment of the extent to which different underlying psychological motivations and mechanisms have been triggered in particular smokers by specific services. Step 4: By exploring how CMOCs play out within a program, the evaluator refines and develops tentative theories of what works for whom in what circumstances.