Use of evaluation and financial modelling in Government

HM Treasury is not making full use of the spending levers it has at its disposal to deliver a step change in the use of evaluation across government. We welcome the creation of the Evaluation Task Force and the increased emphasis on evaluation in the 2020 and 2021 Spending Reviews, whereby HM Treasury made greater use of its authority to set requirements for evaluation when it approved funding. However, we are disappointed that HM Treasury did not make use of these powers earlier, despite the NAO recommending this in 2013. HM Treasury’s final recourse is to withdraw funding or to reduce a department’s delegated authority limit if there is non-compliance with evaluation requirements. HM Treasury has not set up formal and standardised ways of tracking all spending review conditions across departments, including those related to evaluation. With few sanctions available to HM Treasury outside of the spending reviews, HM Treasury needs to make as much use of spending review conditions as it can. HM Treasury acknowledges that there is room for improvement in the quality and provision of evaluations across government despite its recent efforts. Recommendation: By November 2022, HM Treasury should set up a formal process for routinely tracking and following up on spending review settlement conditions relating to evaluation. HM Treasury should set out the range of interventions it will use if departments fail to meet the conditions.

No single body is responsible for upholding modelling and evaluation standards and monitoring their implementation. Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting officer. Supporting the accounting officer, the departmental directors of analysis own the application of the Analysis Functional Standard in their departments. Neither HM Treasury nor the Analysis Function 6 Use of evaluation and modelling in government currently monitor if accounting officers are discharging their responsibilities nor how well standards have been upheld. While HM Treasury is an important user of departments’ models, it does not comprehensively scrutinise and challenge a department’s quality assurance arrangements for the models it uses. To address this gap the Analysis Function intends to strengthen the governance framework for the Analysis Functional Standard, including introducing a self-assessment framework to assess performance against the standard. The Function told us it believes it does not currently have enough resources for this, but intends to request additional funding from HM Treasury. Recommendations: The Analysis Function, under the responsibility of the UK Statistics Authority, should: • put in place an appropriate assessment framework to monitor departments’ implementation of the Analysis Functional Standard; • act on areas for improvement identified through its assessment framework; and • agree with HM Treasury the funding it will provide for these roles. HM Treasury should set out how it plans to gain confidence that the outputs it uses from departments’ business critical models have been quality assured appropriately.

Good quality modelling and evaluation is hampered by challenges in sharing data and a lack of common data standards. Government does not always see data as a priority and data quality is often inadequate. We have drawn attention to poor data and a lack of joined up systems in numerous reports over the years, such as on supporting the vulnerable during lockdown, on tackling the tax gap, on effective regulation of gambling, on dealing with rough sleepers, and on identifying and tackling fraud and error. When producing and using models, poor data can lead to additional time and increased difficulty when quality assuring the model inputs. This difficulty is compounded by a lack of data standards across government which means there are inconsistent ways of recording the same information. When dealing with new situations, such as the rapidly changing pandemic, Government needs to be fast and agile at sharing data. Recommendation: The Cabinet Office should set out its progress in using its national data strategy to address the barriers to better sharing and use of data, including its development of cross-government standards for collecting, storing, recording and managing data.

Departments are not meeting government requirements on publishing evaluation plans and findings, and on transparency of models and their outputs. Transparency is critical for the public to understand the evidence behind decisions and how money is being spent. It also allows departments to learn lessons from past projects and from each other. However, ministers have the final say on whether evaluations should be published, and some parts of government are reluctant to Use of evaluation and modelling in government 7 publish their evaluations. It is only recently that HM Treasury has started to ask departments to explain why they do not plan to publish some evaluations. More than one-third of chief analysts say that only sometimes are they able to publish evaluation findings as required. Transparency of the models that departments use to make decisions is also poor: only 9 out of 17 departments have published their list of business-critical models since 2013, and only four have updated this since

Model producers and users do not adequately assess the range of plausible outcomes and are overly reliant on central estimates that do not reflect inherent uncertainty. All modelled information is uncertain, and decision-makers need information on the possible range of outcomes to plan well, manage risks and make better decisions. However, model producers do not routinely evaluate or communicate the uncertainty in their models. There is only limited evidence of departments using uncertainty analysis or developing contingency plans. Departments usually provide HM Treasury with a single central estimate and do not provide a range of uncertainty around it. HM Treasury does not check every model that departments develop. It does request further uncertainty analysis from departments on a case- by-case basis for those models it uses to make policy decisions or to agree additional spending. HM Treasury agrees that dealing with uncertainty properly is central to quality modelling and that it is important to improve standards in this area. Recommendations: As a key user of outputs from models, HM Treasury should routinely require departments to present the range of plausible outcomes. In its self-assessment tool, the Finance Function, under the responsibility of HM Treasury, should include consideration of how analysis and modelling are applied, including expectations on how accountants should analyse, manage and communicate uncertainty. 8 Use of evaluation and modelling in government 1 The centre of government’s role in improving evaluation and monitoring

On the basis of two reports by the Comptroller and Auditor General, we took evidence from HM Treasury, the Finance Function, the Analysis Function and the Evaluation Task Force about the use of evaluation and modelling across government.1 Cabinet Office introduced cross-government ‘functions’ in 2013 to provide professional support to departments. HM Treasury has overall responsibility for the Finance Function, and the UK Statistics Authority is responsible for the Analysis Function. The Evaluation Task Force, a joint HM Treasury and Cabinet Office Unit, was set up in April 2021 to “deliver a step-change in the scale, quality and impact of evaluation practice in government”.

HM Treasury has used the last two spending reviews to increase the emphasis on evaluation in departments, but acknowledges it has a lot more to do to address the persistent problems which prevent the provision and use of evaluation.17 Working with the Evaluation Task Force, HM Treasury asked departments to provide evaluation evidence for each line of capital budget spend, and for each new line of resource budget spend.18 All of this is a welcome step forward, but we are disappointed that HM Treasury did not make use of its powers earlier, despite the NAO recommending it in 2013.19

Outside of the spending reviews, HM Treasury has few levers and exercises little scrutiny to ensure that departments comply with evaluation requirements and expectations.20 6 C&AG’s Report, Evaluating government spending, para 3.33; C&AG’s Report, Financial modelling in government, para 13, Figure 4 7 Qq 1, 18–19 8 Q 25 9 Q 1 10 Q 1; C&AG’s Report, Evaluating government spending, para 7 11 C&AG’s Report, Evaluating government spending, para 19 12 C&AG’s Report, Evaluating government spending, para 23, para 3.3, Figure 5 13 Qq 25,51 14 C&AG’s Report, Evaluating government spending, para 3 15 Qq 6, 24–25 ; C&AG’s Report, Evaluating government spending, para 3 16 Q 43 17 Q 2 18 C&AG’s Report, Evaluating government spending, para 2.8 19 Q 5; C&AG’s Report, Evaluation in Government, Session 2013–14, 20 December 2013, para 29 20 Qq 2, 26–27; C&AG’s Report, Evaluating government spending, para 3.32–3.33 10 Use of evaluation and modelling in government Despite the spending review being the main lever at HM Treasury’s disposal, it is not making maximum use of this authority. It has not set up formal and standardised ways of tracking all spending review conditions across departments, including those related to evaluation. This means that examples of departments not meeting requirements are likely to persist, and no one is tracking and acting on this at a cross-departmental level.21 HM Treasury told us its final recourse is to withdraw funding or to reduce a department’s delegated authority limit if there is non-compliance with evaluation requirements.22 HM Treasury acknowledged that there is room for improvement in the quality and provision of evaluations across government despite its recent efforts.23 Overall responsibilities

Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting officer.24 Managing Public Money states that financial modelling and evaluation are an important part of the accounting officer’s role, and that they should comply with the Functional Standards.25 However, there continues to be important weaknesses in the coverage, quality and use of evaluations, and in the quality and application of models across government.26 The departmental director of analysis supports the accounting officer, and owns the application of the Analysis Functional standard in their department.27 Neither HM Treasury nor the Analysis Function currently monitor if accounting officers are discharging their responsibilities nor how well standards have been upheld.28

While HM Treasury is an important user of department’s models, it does not comprehensively scrutinise and challenge a department’s quality assurance arrangements for the models it uses. There is no requirement for the Aqua Book to be followed as part of the investment approval process nor as part of the spending review process.29 HM Treasury told us it would want assurance that the department had taken the Aqua Book and its guidance into account, but it does not specifically check.30

To address the lack of overall oversight for both evaluation and modelling standards, the Analysis Function intends to strengthen the governance framework for the Analysis Functional Standard, including introducing a self-assessment framework to assess performance against the standard.31 The Analysis Function told us there will be an expectation in the future that departmental directors of analysis use the self-assessment framework to report back to the Function annually on how well the department has applied the standards.32 The Function told us it believes it does not currently have enough resources for this expanded oversight and assurance role.33 In the 2020 Spending Review, 21 C&AG’s Report, Evaluating government spending, para 3.33 22 Q 27 23 Qq 2, 5, 7 24 Q 1; C&AG’s Report, Evaluating government spending, para 3 25 Q 18; C&AG’s Report, Evaluating government spending, para 1.5; C&AG’s Report, Financial modelling in government, para 1.12, Box 4.1A 26 Financial modelling in government, para 14 27 Qq 18–19; C&AG’s Report, Financial modelling in government, para 1.12 28 C&AG’s Report, Financial modelling in government, para 7 29 C&AG’s Report, Financial modelling in government, para 1.10, Figure 4 30 Q 36 31 C&AG’s Report, Financial modelling in government, Figure 4 32 Qq 16, 18; C&AG’s Report, Financial modelling in government, para 1.7 33 Q 62; C&AG’s Report, Financial modelling in government, para 1.7 Use of evaluation and modelling in government 11 the Analysis Function submitted a bid for £4.9 million to fund 71 full-time equivalent staff. With the change of scope of the Spending Review, HM Treasury did not review this bid so the Analysis Function remained funded at the original rate of £1.3 million.34 The Analysis Function told us it intends to request additional funding from HM Treasury after ongoing internal reviews of scope and governance are completed.35 Data challenges

Government does not always see data as a priority and data quality is often inadequate.36 The NAO reported in 2019 that the quality and sharing of data is a clear example of a neglected and poorly planned activity, and there is a culture of tolerating and working around poor-quality data.37 We have drawn attention to poor data and a lack of joined up systems in numerous reports over the years, recently including for example, on supporting the vulnerable during lockdown, on tackling the tax gap, on effective regulation of gambling, on dealing with rough sleepers, and on identifying and tackling fraud and error.38 We also reported in 2019 on how it had become normal to work around poor-quality disorganised data across government and made several recommendations for improvement.39

When producing and using models, poor data can lead to additional time and increased difficulty when quality assuring the model inputs. This difficulty is compounded by a lack of data standards across government which means there are inconsistent ways of recording the same information.40 When dealing with new situations, such as the rapidly changing pandemic, the Analysis Function told us that Government needs to be fast and agile at sharing data.41 The NAO’s survey showed differences in how satisfied chief analysts were with evaluation support from the centre of government and revealed they were least satisfied with support for sharing data.42 The integrated data service, launched in October 2021, aims to help departments better share and access data.43 34 C&AG’s Report, Financial modelling in government, para 9 35 Q 62 36 C&AG’s Report, Financial modelling in government, para 2.9 37 C&AG’s Report, Challenges in using data across government, Session 2017–19, HC 2220, June 2019 38 Committee of Public Accounts, Covid 19: supporting the vulnerable during lockdown, 53rd Report of Session 2019–21, HC 938, April 2021; Committee of Public Accounts, Tackling the tax gap, 20th Report of Session 2019–21, HC 650, October 2020; Committee of Public Accounts, Gambling regulation: problem gambling and protecting vulnerable people, 7th Report of Session 2019–21, HC 134, June 2020; Committee of Public Accounts, COVID-19: housing people sleeping rough, 49th Report of Session 2019–21, HC 934, March 2021; Committee of Public Accounts, Fraud and error, 9th Report of Session 2021–22, HC 253, June 2021 39 Committee of Public Accounts, Challenges in using data across government, 118th Report of Session 2017–19, HC 2492, September 2019 40 Q 45; C&AG’s Report, Financial modelling in government, paras 2.9–2.9 41 Qq 40, 65 42 C&AG’s Report, Evaluating government spending, para 18 43 Q45; C&AG’s Report, Evaluating government spending, paras 18, 3.28 12 Use of evaluation and modelling in govern

Government’s view is that that transparency is critical for the public to understand the evidence behind decisions and how money is being spent, and that transparency should be the default position.44 The Analysis Function consider that transparency of evaluation findings allows departments to learn lessons from past projects and make them available for the future so the same mistakes might be avoided.45 However, the Finance Function explained that unlike the publication of statistics which is governed by regulation, the publication of evaluations depends on ministerial consent. Some parts of government are reluctant to publish their evaluations. For example, the Department for Work & Pensions was recently reluctant to publish insights from evaluations of the benefits system.46 Departments find it difficult to get approval from senior civil servants and the Cabinet Office to publish evaluation and protocols. More than one-third of chief analysts say that only sometimes are they able to publish evaluation findings as required.47

HMT told us it is only recently that it has started to ask departments to explain why they do not plan to publish some evaluations.48 The Evaluation Task Force raised the issue that there were evaluations that the centre of government did not know about because they had never been published.49This Committee has regularly highlighted departments’ poor timeliness in publishing evaluation findings. The Evaluation Task Force has a core responsibility to maintain a public register of evaluation plan summaries and trial protocols and to encourage departments on the timely publication of evaluation results on GOV.UK. This is to manage ‘publication bias’, the risk that evaluations which have negative findings may not get published,50 to support action on evaluation results and to improve accessibility for decision-makers. To meet its core responsibilities, the Evaluation Task Force explained that it is creating a website which sets out what evaluations departments intend to publish when, and whether this has happened.51 It hopes that this will allow the public to easily see whether departments are complying with requirements and allow the Evaluation Task Force to hold departments to publishing what they have promised.52 HM Treasury told us it has recently started to require that departments publish an evaluation strategy and, given that communication is a central part of evaluation, HM Treasury expects the evaluation strategy will include a communication strategy.53

Transparency of the models that departments use is poor: only 9 out of 17 departments have published their list of business-critical models since 2013, and only four of those have updated this since 2017.54 HM Treasury has not published its own list of business-critical 44 Qq 43,55 45 Q 43 46 Q 49 47 C&AG’s Report, Evaluating government spending, para 20 48 Q 43 49 Q 44 50 C&AG’s Report, Evaluating government spending, para 3.14 51 Qq 33, 44 52 Q 33 53 Q 7 54 C&AG’s Report, Financial modelling in government, para 12 Use of evaluation and modelling in government 13 models since 2013. It told us it intends to update its list within the next few months given the importance of leading by example.55 There can be a public interest in publishing a model in its entirety; for example, the Office of Budgetary Responsibility publishes the code and variables for its model of the economy so that others can use it.56 In other cases, publishing details of the model such as the outputs or assumptions gives parliament and the public assurance about how decisions are made. However, 45 models out of the NAO’s sample of 75 had no information published about them.57 HM Treasury told us it is discussing how to improve guidance to clarify what information departments should publish about models.58 Uncertainty

Uncertainty is inherent in modelled information and should be considered as part of all analysis. HM Treasury believes any model that does not explicitly address uncertainty is deficient.59 Decision-makers need information on the range of outcomes that may occur and their relative likelihoods to develop contingency plans and to avoid putting value for money at risk.60 For example, our report Government preparedness for the COVID-19 pandemic: lessons for government on risk found that government lacked detailed plans for several aspects of the response to COVID-19.61 The NAO’s report found the lack of plans reduced the government’s ability to respond to the emergency.62 The Cabinet Office and the Department of Health & Social Care recognised that government needs to set out risks more broadly and be prepared for a wider range of manifestations of any given risk.63

Despite the importance of uncertainty analysis, model producers do not routinely assess or communicate to users the uncertainty in their models. There is limited evidence of departments using uncertainty analysis or developing contingency plans, and departments generally present outputs as single estimates.64 Analysis of uncertainty can give clarity on the reliability of the assumptions and the data used in the model. The Analysis Function told us proper reporting of uncertainty is essential and that the function can provide departments with expert support on understanding uncertainty in complex models. It is raising the profile of the importance of uncertainty analysis for senior civil servants and ministers through its data masterclasses.65 The Finance Function told us it intends to upskill and support finance professions in raising standards, including in relation to estimating financial uncertainty.66

Departments present outputs from their models to HM Treasury as part of the spending review and budget process. HM Treasury uses these outputs for forecasting, budget planning and to monitor emerging risks. Departments usually provide HM 55 Qq 52–53 56 Q 54 57 C&AG’s Report, Financial modelling in government, paras 1.20–1.21 58 Q 54 59 Q 11 60 C&AG’s Report, Financial modelling in government, paras 3.2–3.9 61 Committee of Public Accounts, Government preparedness for the COVID-19 pandemic: lessons for government on risk, 46th report of Session 2021–22, March 2022, para 15 62 C&AG’s Report, Financial modelling in government, para 3.11 63 Committee of Public Accounts, Government preparedness for the COVID-19 pandemic: lessons for government on risk, 46th report of Session 2021–22, March 2022, para 16 64 C&AG’s Report, Financial modelling in government, para 3.5 65 Qq 10–15 66 Q 16 14 Use of evaluation and modelling in government Treasury with a single estimate and do not provide a range of uncertainty around it.67 HM Treasury told us it does not check every model that departments develop. For those models it uses to make policy decisions and to agree additional spending, HM Treasury explained it does interrogate these models by challenging departments on the assumptions and ranges used. HM Treasury agrees that dealing with uncertainty properly is central to quality modelling.68 67 C&AG’s Report, Financial modelling in government, para 18 68 Q 11 Use of evaluation and modelling in government 15