Better evaluation
Angela Hodkinson, who leads Deloitte’s policy evaluation team, talks about how effective evaluation can strengthen policy-making and the need to make sure this task is more than just an ‘add-on’.
The ‘standard’ policy cycle often places evaluation at the end. This summative approach is valuable to provide a snapshot of progress and impact but evaluation can, and should, be undertaken throughout the policy cycle. Ex-ante evaluation establishes a baseline and articulates what a policy hopes to achieve and how intervention will make a difference. Ongoing evaluation allows for revisions as policy is implemented, thus maximising its potential to deliver. Therefore ongoing evaluation and review needs to be built in at regular intervals.
Constrained public finances have encouraged more ‘in-house’ evaluation but there is value in maintaining an independent perspective. Individuals not directly involved in management and delivery can provide an unbiased point of view, challenging assumptions and bringing a fresh perspective.
More partnership approaches to evaluation are needed in the public sector. For example, some organisations in the voluntary sector are working with peers to provide each other with independent evaluation input, while others primarily self-evaluate but engage an external evaluator to guide the process and provide constructive challenge.
In some cases, the evaluation is clearly embedded in an ongoing cycle of policy development e.g. where delivery is contracted to third parties and the contract cycle drives regular reviews. However, it is not unusual to find at the point of evaluation that there is only a rather vague idea of what impact a programme was expected to have and how it was meant to effect change. Defining this in retrospect can be difficult and risks being biased by hindsight. A well-defined evaluation framework not only supports evaluation; it can also support policy implementation.
We worked recently with a voluntary sector organisation which had invested significantly in evaluation. Working with an external evaluator, it had defined an evaluation framework, with supporting monitoring and management processes. What was different in this case was that they had fully integrated this into their strategic planning cycle for the organisation. So evaluation became a tool to support the organisation itself rather than something ‘done to it’ by a funder.
There can sometimes be a ‘disconnect’ between overarching government policy and the interventions put in place to implement it. The majority of formal
evaluation takes place at the programme or project level rather than at the central policy level. It’s not always clear how this flows up into evaluation of overarching policies over a longer period of time or where the findings from individual programme evaluation end up.
Sometimes the overarching policy aim can get buried in the day-to-day realities of delivery. People and organisations invest time and energy in policies, and evaluation needs to get behind the natural instinct for self-preservation and resistance to change to identify what difference has really been made.
Too often evaluation is considered as something imposed from above, to satisfy paymasters or external stakeholders. But if viewed as a positive opportunity to take stock and continuously improve, then going through a well-designed evaluation process can be beneficial in itself. It can provide opportunities for communication with stakeholders, establish useful relationships, encourage the development of a learning culture and, importantly, create different and improved policy outcomes.
