OUTCOME AND IMPACT

The differences between Output and Outcome can be clearly understood. But the differences between Outcome and Impact are not so clear. Both refer to the consequences of programs. In fact, for some these terms mean the same and are used interchangeably. On the other hand, many distinguish between the two terms. But the different ways in which the distinctions are made add to the confusion.


This article is to help explain the differences between these two terms.



Outcome


We know that Outcomes are changes in learning, behaviour or condition in participants after undergoing a program, and these can be short-term, medium-term or long-term in nature, and may be positive or even negative changes. Of course, depending on the program, the Outcomes may occur in the community or the environment as well. A detailed discussion on Outcomes is in another article (here).



Impact


What then is Impact? Many define Impact as longer-term, deeper or broader changes resulting from a program. In my opinion, these are not satisfactory distinctions. Outcomes may also be observed to be long term in duration and may also be deep and broad in nature, affecting an organisation, a family, a community or the environment, and not just the participants.


Problems arise when considering longer term and broader effects of a program. Such effects are diluted or polluted by other influences and all the outcomes cannot be attributed to the program. Screening out non-program influences would be difficult. It would be even more difficult, the longer-term or broader the analysis.


Happily, Social Science provides a methodology called Counterfactual Analysis that enables comparison between the outcomes attributable to a program against what would have happened to the participants without the program. The Impact of the program would correctly be defined to be this comparison.


A simple example would be to observe two groups of people with similar demographics in a similar environment. One group goes through a social program and the other (the control group) does not. The Impact of the program would be the observed differences in outcomes between the two groups at any point in time.


By taking the differences in outcomes, the effects of non-program influences are eliminated, as they are equally felt by both groups. The accuracy and value of such Impact measurement would depend on the use of the best methodology that is available, feasible and appropriate to the program being evaluated.


It is not the intent of this Appendix to delve into the methodologies. The intent here is to show that:


Impact = Set A of Outcomes – Set B of Outcomes


Where:


Set A = Outcomes of participants in the program

Set B = Outcomes of the control group who did not participate


The definition of Impact as the difference between the outcomes achieved by a program versus what would have happened anyway without the program is more satisfactory. In any case this definition clearly shows the difference between Impact and Outcome.



Example 1


Here is a simple fictitious illustration. Country ABC has a penal population of 10,000. It costs the government $50 million per year to run the prisons administration. Thus the cost per inmate per year is $5,000.


Typically 12,000 inmates were released per year. In 2008 the recidivism rate was 30% within one year after release, or 3,600 would be readmitted in 2009. The recidivism rate had historically been declining by 0.5% per year due to improving socioeconomic conditions in the country.


In 2008, Country ABC decided to institute a rehabilitation program for all inmates prior to release. In 2009 it was found that the recidivism rate dropped to 28%, or 3,360 readmitted in that year for those released in 2008. The government claimed that arising from the rehabilitation program, the number readmitted dropped by 240, or a savings of $1.2 million.


However this is not correct. If nothing was done, the recidivism rate would have fallen to 29.5% or 3,540. Thus if nothing was done, the government would have saved the cost of 60 inmates or $300,000 in any case.


Thus using the formulas in the previous section:


Set A Outcome = $1.2 million

Set B Outcome = $0.3 million

Therefore Impact = 1.2 – 0.3 = $900,000


Thus the total outcome in 2009 was a savings of $1.2 million, but the impact from the program was actually $0.9 million. An outcome of $300,000 arose from other influences. It would be incorrect to claim that the impact from the program was $1.2 million in 2009.



Example 2


Another example could be a school having a supplementary after-school program for say Class A. Another class, say Class B, in the same level not undergoing the supplementary program would be used as the control group.


We assume that at the beginning of the year, both classes have similar sets of academic results, say Set 0 (zero) Outcome. At the end of the year, Class A, with the supplementary program achieves Set A Outcome. Class B, without the program, achieves Set B Outcome, which ordinarily will be better than Set 0 due to the normal class programs.


The conclusions we can correctly draw from this example are:


oThe outcome of the normal and supplementary programs for Class A were the changes from Set 0 to Set A


oThe outcome of the normal programs for Class B were the changes from Set 0 to Set B


oThe impact of the normal programs is the same as the outcome for Class B


oThe impact of the supplementary program is not the same as the outcome for Class A as they were also influenced by the normal programs


oIf we assume acceptable homogeneity between Class A and Class B, the impact of the supplementary program is the difference between Set A and Set B.


Note the difference between outcome and impact as described in these conclusions. For Class B, the outcome and impact are the same, as nothing more was done for them. For Class A, the outcome and impact are not the same, due to the intervention of the supplementary program. We have to separate out the outcome of the supplementary program from the total outcome, by using Class B as the control group.


If the two groups are properly chosen to have strong homogeneity, the last conclusion would be valid.


Finally, for this discussion, the outcome sets could be a single target outcome (such as a higher maths score), or a group of outcomes (such as a higher maths score, higher self-esteem, greater interest in school, etc) that is being studied.


Side Notes


As a side note, perhaps the more accurate definition would be where:


Set A = Outcomes of participants in the program

Set B = Outcomes of the same participants if they did not participate in the program


This eliminates any differences between the participants and the control group, but requires hypothetical assumptions.


It is also interesting to note how the formula would turn out if we were to measure the Impact immediately after a program, particularly a program of relatively short duration.


The control group would have been subject to little or no influence and hence they would not have experienced any change in learning, behaviour or condition. Thus Set B = 0 and Impact = Set A. We conclude that immediately after a program, the Impact on the participants would be their immediate Outcomes. This explains why one would think of Impact as being synonymous with Outcome.


The difference shows up when non-program influences come into play such as some time after the program, or when impact is analysed on a broader basis. Then Set B 0 as there are other influences, and Impact ≠ Set A, but would have to calculated as Set A less Set B.




Wong Lin Hong

March 2013




NOTE: In “A Practical Guide to Measuring and Managing Impact” published by the European Venture Philanthropy Association (EVPA) in April 2013, it is stated that five factors need to be considered when measuring impact:


  1. Deadweight: This relates to a consideration of what would have happened anyway in the absence of the social program. It includes the progress that beneficiaries would have made without going through the program (which, as mentioned in the above article, reduces the impact of the program), as well as any negative consequences that might arise without the intervention of the program (which increases the impact of the program).

  2. Attribution: This relates to understanding how much of the change observed in the beneficiaries is the result of the program or the effect of other programs or activities taking place at the same time.

  3. Displacement: This relates to the possibility that some positive interventions of a program may be offset by negative effects seen in a different group which is not the target beneficiary of the program. For example, a reduction of crime in a community due to an intervention program might have been because the criminals moved into another community elsewhere.

  4. Drop off: This relates to diminishing impact over time. Therefore measured impact must account for the estimated time period of its effect after the completion of the program. Since drop off may vary from one beneficiary to the next, it would be useful to identify the more common effects to enable future improvements to the program.

  5. Unintended consequences: These are positive or negative effects arising from a program which are not part of the desired effect.

Measurement of impact has to adjust for these five factors to derive the net impact that can rightly be attributed to a social program.


Google “A Practical Guide to Measuring and Managing Impact” to find the Guide.



Wong Lin Hong

1 June 2013