Recommendations & Conclusions
12 items
3
Recommendation
Fourth Report - Use of evaluation and m…
Accepted
No single body is responsible for upholding modelling and evaluation standards and monitoring their implementation. Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting officer. Supporting the accounting officer, the departmental directors of analysis own the application of the Analysis Functional Standard in their …
Read more
No single body is responsible for upholding modelling and evaluation standards and monitoring their implementation. Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting officer. Supporting the accounting officer, the departmental directors of analysis own the application of the Analysis Functional Standard in their departments. Neither HM Treasury nor the Analysis Function 6 Use of evaluation and modelling in government currently monitor if accounting officers are discharging their responsibilities nor how well standards have been upheld. While HM Treasury is an important user of departments’ models, it does not comprehensively scrutinise and challenge a department’s quality assurance arrangements for the models it uses. To address this gap the Analysis Function intends to strengthen the governance framework for the Analysis Functional Standard, including introducing a self-assessment framework to assess performance against the standard. The Function told us it believes it does not currently have enough resources for this, but intends to request additional funding from HM Treasury. Recommendations: The Analysis Function, under the responsibility of the UK Statistics Authority, should: • put in place an appropriate assessment framework to monitor departments’ implementation of the Analysis Functional Standard; • act on areas for improvement identified through its assessment framework; and • agree with HM Treasury the funding it will provide for these roles. HM Treasury should set out how it plans to gain confidence that the outputs it uses from departments’ business critical models have been quality assured appropriately.
Show less
Government response AI summary
The Government Analysis Function is piloting an assessment framework to monitor departments' implementation of the Analysis Functional Standard, and HM Treasury will build on its current training offer to ensure staff can confidently assure the quality of outputs from other departments.
Read full response →
HM Treasury
4
Recommendation
Fourth Report - Use of evaluation and m…
Accepted
Good quality modelling and evaluation is hampered by challenges in sharing data and a lack of common data standards. Government does not always see data as a priority and data quality is often inadequate. We have drawn attention to poor data and a lack of joined up systems in numerous …
Read more
Good quality modelling and evaluation is hampered by challenges in sharing data and a lack of common data standards. Government does not always see data as a priority and data quality is often inadequate. We have drawn attention to poor data and a lack of joined up systems in numerous reports over the years, such as on supporting the vulnerable during lockdown, on tackling the tax gap, on effective regulation of gambling, on dealing with rough sleepers, and on identifying and tackling fraud and error. When producing and using models, poor data can lead to additional time and increased difficulty when quality assuring the model inputs. This difficulty is compounded by a lack of data standards across government which means there are inconsistent ways of recording the same information. When dealing with new situations, such as the rapidly changing pandemic, Government needs to be fast and agile at sharing data. Recommendation: The Cabinet Office should set out its progress in using its national data strategy to address the barriers to better sharing and use of data, including its development of cross-government standards for collecting, storing, recording and managing data.
Show less
Government response AI summary
The Cabinet Office will address barriers to data sharing and use through the national data strategy, with complete delivery by April 2025 and an interim progress report by December 2023; the CDDO has published the Data Sharing Governance Framework and is establishing the Data Sharing …
Read full response →
HM Treasury
5
Conclusion
Fourth Report - Use of evaluation and m…
Accepted
Departments are not meeting government requirements on publishing evaluation plans and findings, and on transparency of models and their outputs. Transparency is critical for the public to understand the evidence behind decisions and how money is being spent. It also allows departments to learn lessons from past projects and from …
Read more
Departments are not meeting government requirements on publishing evaluation plans and findings, and on transparency of models and their outputs. Transparency is critical for the public to understand the evidence behind decisions and how money is being spent. It also allows departments to learn lessons from past projects and from each other. However, ministers have the final say on whether evaluations should be published, and some parts of government are reluctant to Use of evaluation and modelling in government 7 publish their evaluations. It is only recently that HM Treasury has started to ask departments to explain why they do not plan to publish some evaluations. More than one-third of chief analysts say that only sometimes are they able to publish evaluation findings as required. Transparency of the models that departments use to make decisions is also poor: only 9 out of 17 departments have published their list of business-critical models since 2013, and only four have updated this since
Show less
Government response AI summary
The government will publish and maintain an evaluation registry and review procedures around the quality assurance of models and incorporate improved practices across departments.
Read full response →
HM Treasury
6
Recommendation
Fourth Report - Use of evaluation and m…
Accepted
Model producers and users do not adequately assess the range of plausible outcomes and are overly reliant on central estimates that do not reflect inherent uncertainty. All modelled information is uncertain, and decision-makers need information on the possible range of outcomes to plan well, manage risks and make better decisions. …
Read more
Model producers and users do not adequately assess the range of plausible outcomes and are overly reliant on central estimates that do not reflect inherent uncertainty. All modelled information is uncertain, and decision-makers need information on the possible range of outcomes to plan well, manage risks and make better decisions. However, model producers do not routinely evaluate or communicate the uncertainty in their models. There is only limited evidence of departments using uncertainty analysis or developing contingency plans. Departments usually provide HM Treasury with a single central estimate and do not provide a range of uncertainty around it. HM Treasury does not check every model that departments develop. It does request further uncertainty analysis from departments on a case- by-case basis for those models it uses to make policy decisions or to agree additional spending. HM Treasury agrees that dealing with uncertainty properly is central to quality modelling and that it is important to improve standards in this area. Recommendations: As a key user of outputs from models, HM Treasury should routinely require departments to present the range of plausible outcomes. In its self-assessment tool, the Finance Function, under the responsibility of HM Treasury, should include consideration of how analysis and modelling are applied, including expectations on how accountants should analyse, manage and communicate uncertainty. 8 Use of evaluation and modelling in government 1 The centre of government’s role in improving evaluation and monitoring
Show less
Government response AI summary
The Finance Function will work with the Analysis Function to identify appropriate analysis and modelling considerations relating to uncertainty and incorporate these into the Finance Standard and self-assessment tool.
Read full response →
HM Treasury
8
Conclusion
Fourth Report - Use of evaluation and m…
Accepted
Outside of the spending reviews, HM Treasury has few levers and exercises little scrutiny to ensure that departments comply with evaluation requirements and expectations.20 6 C&AG’s Report, Evaluating government spending, para 3.33; C&AG’s Report, Financial modelling in government, para 13, Figure 4 7 Qq 1, 18–19 8 Q 25 9 …
Read more
Outside of the spending reviews, HM Treasury has few levers and exercises little scrutiny to ensure that departments comply with evaluation requirements and expectations.20 6 C&AG’s Report, Evaluating government spending, para 3.33; C&AG’s Report, Financial modelling in government, para 13, Figure 4 7 Qq 1, 18–19 8 Q 25 9 Q 1 10 Q 1; C&AG’s Report, Evaluating government spending, para 7 11 C&AG’s Report, Evaluating government spending, para 19 12 C&AG’s Report, Evaluating government spending, para 23, para 3.3, Figure 5 13 Qq 25,51 14 C&AG’s Report, Evaluating government spending, para 3 15 Qq 6, 24–25 ; C&AG’s Report, Evaluating government spending, para 3 16 Q 43 17 Q 2 18 C&AG’s Report, Evaluating government spending, para 2.8 19 Q 5; C&AG’s Report, Evaluation in Government, Session 2013–14, 20 December 2013, para 29 20 Qq 2, 26–27; C&AG’s Report, Evaluating government spending, para 3.32–3.33 10 Use of evaluation and modelling in government Despite the spending review being the main lever at HM Treasury’s disposal, it is not making maximum use of this authority. It has not set up formal and standardised ways of tracking all spending review conditions across departments, including those related to evaluation. This means that examples of departments not meeting requirements are likely to persist, and no one is tracking and acting on this at a cross-departmental level.21 HM Treasury told us its final recourse is to withdraw funding or to reduce a department’s delegated authority limit if there is non-compliance with evaluation requirements.22 HM Treasury acknowledged that there is room for improvement in the quality and provision of evaluations across government despite its recent efforts.23 Overall responsibilities
Show less
Government response AI summary
The Evaluation Task Force is working with HM Treasury to ensure that departments respond to and deliver on their settlement conditions relating to evaluation, using a central tracking system to monitor progress.
Read full response →
HM Treasury
11
Conclusion
Fourth Report - Use of evaluation and m…
Accepted
To address the lack of overall oversight for both evaluation and modelling standards, the Analysis Function intends to strengthen the governance framework for the Analysis Functional Standard, including introducing a self-assessment framework to assess performance against the standard.31 The Analysis Function told us there will be an expectation in the …
Read more
To address the lack of overall oversight for both evaluation and modelling standards, the Analysis Function intends to strengthen the governance framework for the Analysis Functional Standard, including introducing a self-assessment framework to assess performance against the standard.31 The Analysis Function told us there will be an expectation in the future that departmental directors of analysis use the self-assessment framework to report back to the Function annually on how well the department has applied the standards.32 The Function told us it believes it does not currently have enough resources for this expanded oversight and assurance role.33 In the 2020 Spending Review, 21 C&AG’s Report, Evaluating government spending, para 3.33 22 Q 27 23 Qq 2, 5, 7 24 Q 1; C&AG’s Report, Evaluating government spending, para 3 25 Q 18; C&AG’s Report, Evaluating government spending, para 1.5; C&AG’s Report, Financial modelling in government, para 1.12, Box 4.1A 26 Financial modelling in government, para 14 27 Qq 18–19; C&AG’s Report, Financial modelling in government, para 1.12 28 C&AG’s Report, Financial modelling in government, para 7 29 C&AG’s Report, Financial modelling in government, para 1.10, Figure 4 30 Q 36 31 C&AG’s Report, Financial modelling in government, Figure 4 32 Qq 16, 18; C&AG’s Report, Financial modelling in government, para 1.7 33 Q 62; C&AG’s Report, Financial modelling in government, para 1.7 Use of evaluation and modelling in government 11 the Analysis Function submitted a bid for £4.9 million to fund 71 full-time equivalent staff. With the change of scope of the Spending Review, HM Treasury did not review this bid so the Analysis Function remained funded at the original rate of £1.3 million.34 The Analysis Function told us it intends to request additional funding from HM Treasury after ongoing internal reviews of scope and governance are completed.35 Data challenges
Show less
Government response AI summary
The Government Analysis Function is piloting an assessment framework to monitor departments' implementation of the Analysis Functional Standard, and HM Treasury will build on its current training offer to ensure staff can confidently assure the quality of outputs from other departments.
Read full response →
HM Treasury
12
Conclusion
Fourth Report - Use of evaluation and m…
Accepted
Government does not always see data as a priority and data quality is often inadequate.36 The NAO reported in 2019 that the quality and sharing of data is a clear example of a neglected and poorly planned activity, and there is a culture of tolerating and working around poor-quality data.37 …
Read more
Government does not always see data as a priority and data quality is often inadequate.36 The NAO reported in 2019 that the quality and sharing of data is a clear example of a neglected and poorly planned activity, and there is a culture of tolerating and working around poor-quality data.37 We have drawn attention to poor data and a lack of joined up systems in numerous reports over the years, recently including for example, on supporting the vulnerable during lockdown, on tackling the tax gap, on effective regulation of gambling, on dealing with rough sleepers, and on identifying and tackling fraud and error.38 We also reported in 2019 on how it had become normal to work around poor-quality disorganised data across government and made several recommendations for improvement.39
Show less
Government response AI summary
The Central Digital and Data Office (CDDO) is delivering Mission 3 of the National Data Strategy, outlining how all departments will work towards better use of data, with the strategy completely delivered by April 2025.
Read full response →
HM Treasury
14
Conclusion
Fourth Report - Use of evaluation and m…
Accepted
Government’s view is that that transparency is critical for the public to understand the evidence behind decisions and how money is being spent, and that transparency should be the default position.44 The Analysis Function consider that transparency of evaluation findings allows departments to learn lessons from past projects and make …
Read more
Government’s view is that that transparency is critical for the public to understand the evidence behind decisions and how money is being spent, and that transparency should be the default position.44 The Analysis Function consider that transparency of evaluation findings allows departments to learn lessons from past projects and make them available for the future so the same mistakes might be avoided.45 However, the Finance Function explained that unlike the publication of statistics which is governed by regulation, the publication of evaluations depends on ministerial consent. Some parts of government are reluctant to publish their evaluations. For example, the Department for Work & Pensions was recently reluctant to publish insights from evaluations of the benefits system.46 Departments find it difficult to get approval from senior civil servants and the Cabinet Office to publish evaluation and protocols. More than one-third of chief analysts say that only sometimes are they able to publish evaluation findings as required.47
Show less
Government response AI summary
The Evaluation Task Force is developing a publicly accessible online registry of UK government evaluations, allowing departments to publish evaluation plans/protocols and reports, and indicate estimated publication dates. The Analysis Function is reviewing its Functional Standard to incorporate improved practices across departments by the end …
Read full response →
HM Treasury
15
Recommendation
Fourth Report - Use of evaluation and m…
Accepted
HMT told us it is only recently that it has started to ask departments to explain why they do not plan to publish some evaluations.48 The Evaluation Task Force raised the issue that there were evaluations that the centre of government did not know about because they had never been …
Read more
HMT told us it is only recently that it has started to ask departments to explain why they do not plan to publish some evaluations.48 The Evaluation Task Force raised the issue that there were evaluations that the centre of government did not know about because they had never been published.49This Committee has regularly highlighted departments’ poor timeliness in publishing evaluation findings. The Evaluation Task Force has a core responsibility to maintain a public register of evaluation plan summaries and trial protocols and to encourage departments on the timely publication of evaluation results on GOV.UK. This is to manage ‘publication bias’, the risk that evaluations which have negative findings may not get published,50 to support action on evaluation results and to improve accessibility for decision-makers. To meet its core responsibilities, the Evaluation Task Force explained that it is creating a website which sets out what evaluations departments intend to publish when, and whether this has happened.51 It hopes that this will allow the public to easily see whether departments are complying with requirements and allow the Evaluation Task Force to hold departments to publishing what they have promised.52 HM Treasury told us it has recently started to require that departments publish an evaluation strategy and, given that communication is a central part of evaluation, HM Treasury expects the evaluation strategy will include a communication strategy.53
Show less
Government response AI summary
The Evaluation Task Force is developing a publicly accessible online registry of UK government evaluations, allowing departments to publish evaluation plans/protocols and reports, and indicate estimated publication dates. The Analysis Function is reviewing its Functional Standard to incorporate improved practices across departments by the end …
Read full response →
HM Treasury
17
Conclusion
Fourth Report - Use of evaluation and m…
Accepted
Uncertainty is inherent in modelled information and should be considered as part of all analysis. HM Treasury believes any model that does not explicitly address uncertainty is deficient.59 Decision-makers need information on the range of outcomes that may occur and their relative likelihoods to develop contingency plans and to avoid …
Read more
Uncertainty is inherent in modelled information and should be considered as part of all analysis. HM Treasury believes any model that does not explicitly address uncertainty is deficient.59 Decision-makers need information on the range of outcomes that may occur and their relative likelihoods to develop contingency plans and to avoid putting value for money at risk.60 For example, our report Government preparedness for the COVID-19 pandemic: lessons for government on risk found that government lacked detailed plans for several aspects of the response to COVID-19.61 The NAO’s report found the lack of plans reduced the government’s ability to respond to the emergency.62 The Cabinet Office and the Department of Health & Social Care recognised that government needs to set out risks more broadly and be prepared for a wider range of manifestations of any given risk.63
Show less
Government response AI summary
The Finance Function will work with the Analysis Function to identify appropriate analysis and modelling considerations relating to uncertainty and incorporate these into the Finance Standard.
Read full response →
HM Treasury
18
Recommendation
Fourth Report - Use of evaluation and m…
Accepted
Despite the importance of uncertainty analysis, model producers do not routinely assess or communicate to users the uncertainty in their models. There is limited evidence of departments using uncertainty analysis or developing contingency plans, and departments generally present outputs as single estimates.64 Analysis of uncertainty can give clarity on the …
Read more
Despite the importance of uncertainty analysis, model producers do not routinely assess or communicate to users the uncertainty in their models. There is limited evidence of departments using uncertainty analysis or developing contingency plans, and departments generally present outputs as single estimates.64 Analysis of uncertainty can give clarity on the reliability of the assumptions and the data used in the model. The Analysis Function told us proper reporting of uncertainty is essential and that the function can provide departments with expert support on understanding uncertainty in complex models. It is raising the profile of the importance of uncertainty analysis for senior civil servants and ministers through its data masterclasses.65 The Finance Function told us it intends to upskill and support finance professions in raising standards, including in relation to estimating financial uncertainty.66
Show less
Government response AI summary
The Finance Function will work with the Analysis Function to identify uncertainty analysis considerations and incorporate these into the Finance Standard and self-assessment tool.
Read full response →
HM Treasury
19
Conclusion
Fourth Report - Use of evaluation and m…
Accepted
Departments present outputs from their models to HM Treasury as part of the spending review and budget process. HM Treasury uses these outputs for forecasting, budget planning and to monitor emerging risks. Departments usually provide HM 55 Qq 52–53 56 Q 54 57 C&AG’s Report, Financial modelling in government, paras …
Read more
Departments present outputs from their models to HM Treasury as part of the spending review and budget process. HM Treasury uses these outputs for forecasting, budget planning and to monitor emerging risks. Departments usually provide HM 55 Qq 52–53 56 Q 54 57 C&AG’s Report, Financial modelling in government, paras 1.20–1.21 58 Q 54 59 Q 11 60 C&AG’s Report, Financial modelling in government, paras 3.2–3.9 61 Committee of Public Accounts, Government preparedness for the COVID-19 pandemic: lessons for government on risk, 46th report of Session 2021–22, March 2022, para 15 62 C&AG’s Report, Financial modelling in government, para 3.11 63 Committee of Public Accounts, Government preparedness for the COVID-19 pandemic: lessons for government on risk, 46th report of Session 2021–22, March 2022, para 16 64 C&AG’s Report, Financial modelling in government, para 3.5 65 Qq 10–15 66 Q 16 14 Use of evaluation and modelling in government Treasury with a single estimate and do not provide a range of uncertainty around it.67 HM Treasury told us it does not check every model that departments develop. For those models it uses to make policy decisions and to agree additional spending, HM Treasury explained it does interrogate these models by challenging departments on the assumptions and ranges used. HM Treasury agrees that dealing with uncertainty properly is central to quality modelling.68 67 C&AG’s Report, Financial modelling in government, para 18 68 Q 11 Use of evaluation and modelling in government 15
Show less
Government response AI summary
The Finance Function will work with the Analysis Function to identify analysis and modelling considerations relating to uncertainty and incorporate these into the Finance Standard by March 2023.
Read full response →
HM Treasury