Fourth Report - Use of evaluation and modelling in government

HM Treasury is not making full use of the spending levers it has at its disposal to deliver a step change in the use of evaluation across government. We welcome the creation of the Evaluation Task Force and the increased … Read more HM Treasury is not making full use of the spending levers it has at its disposal to deliver a step change in the use of evaluation across government. We welcome the creation of the Evaluation Task Force and the increased emphasis on evaluation in the 2020 and 2021 Spending Reviews, whereby HM Treasury made greater use of its authority to set requirements for evaluation when it approved funding. However, we are disappointed that HM Treasury did not make use of these powers earlier, despite the NAO recommending this in 2013. HM Treasury’s final recourse is to withdraw funding or to reduce a department’s delegated authority limit if there is non-compliance with evaluation requirements. HM Treasury has not set up formal and standardised ways of tracking all spending review conditions across departments, including those related to evaluation. With few sanctions available to HM Treasury outside of the spending reviews, HM Treasury needs to make as much use of spending review conditions as it can. HM Treasury acknowledges that there is room for improvement in the quality and provision of evaluations across government despite its recent efforts. Recommendation: By November 2022, HM Treasury should set up a formal process for routinely tracking and following up on spending review settlement conditions relating to evaluation. HM Treasury should set out the range of interventions it will use if departments fail to meet the conditions. Show less

View Details →

No single body is responsible for upholding modelling and evaluation standards and monitoring their implementation. Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting officer. Supporting the accounting officer, the departmental directors of analysis … Read more No single body is responsible for upholding modelling and evaluation standards and monitoring their implementation. Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting officer. Supporting the accounting officer, the departmental directors of analysis own the application of the Analysis Functional Standard in their departments. Neither HM Treasury nor the Analysis Function 6 Use of evaluation and modelling in government currently monitor if accounting officers are discharging their responsibilities nor how well standards have been upheld. While HM Treasury is an important user of departments’ models, it does not comprehensively scrutinise and challenge a department’s quality assurance arrangements for the models it uses. To address this gap the Analysis Function intends to strengthen the governance framework for the Analysis Functional Standard, including introducing a self-assessment framework to assess performance against the standard. The Function told us it believes it does not currently have enough resources for this, but intends to request additional funding from HM Treasury. Recommendations: The Analysis Function, under the responsibility of the UK Statistics Authority, should: • put in place an appropriate assessment framework to monitor departments’ implementation of the Analysis Functional Standard; • act on areas for improvement identified through its assessment framework; and • agree with HM Treasury the funding it will provide for these roles. HM Treasury should set out how it plans to gain confidence that the outputs it uses from departments’ business critical models have been quality assured appropriately. Show less

View Details →

Good quality modelling and evaluation is hampered by challenges in sharing data and a lack of common data standards. Government does not always see data as a priority and data quality is often inadequate. We have drawn attention to poor … Read more Good quality modelling and evaluation is hampered by challenges in sharing data and a lack of common data standards. Government does not always see data as a priority and data quality is often inadequate. We have drawn attention to poor data and a lack of joined up systems in numerous reports over the years, such as on supporting the vulnerable during lockdown, on tackling the tax gap, on effective regulation of gambling, on dealing with rough sleepers, and on identifying and tackling fraud and error. When producing and using models, poor data can lead to additional time and increased difficulty when quality assuring the model inputs. This difficulty is compounded by a lack of data standards across government which means there are inconsistent ways of recording the same information. When dealing with new situations, such as the rapidly changing pandemic, Government needs to be fast and agile at sharing data. Recommendation: The Cabinet Office should set out its progress in using its national data strategy to address the barriers to better sharing and use of data, including its development of cross-government standards for collecting, storing, recording and managing data. Show less

View Details →

Transparency of the models that departments use is poor: only 9 out of 17 departments have published their list of business-critical models since 2013, and only four of those have updated this since 2017.54 HM Treasury has not published its … Read more Transparency of the models that departments use is poor: only 9 out of 17 departments have published their list of business-critical models since 2013, and only four of those have updated this since 2017.54 HM Treasury has not published its own list of business-critical 44 Qq 43,55 45 Q 43 46 Q 49 47 C&AG’s Report, Evaluating government spending, para 20 48 Q 43 49 Q 44 50 C&AG’s Report, Evaluating government spending, para 3.14 51 Qq 33, 44 52 Q 33 53 Q 7 54 C&AG’s Report, Financial modelling in government, para 12 Use of evaluation and modelling in government 13 models since 2013. It told us it intends to update its list within the next few months given the importance of leading by example.55 There can be a public interest in publishing a model in its entirety; for example, the Office of Budgetary Responsibility publishes the code and variables for its model of the economy so that others can use it.56 In other cases, publishing details of the model such as the outputs or assumptions gives parliament and the public assurance about how decisions are made. However, 45 models out of the NAO’s sample of 75 had no information published about them.57 HM Treasury told us it is discussing how to improve guidance to clarify what information departments should publish about models.58 Uncertainty Show less

View Details →

Departments are not meeting government requirements on publishing evaluation plans and findings, and on transparency of models and their outputs. Transparency is critical for the public to understand the evidence behind decisions and how money is being spent. It also allows departments to learn lessons from past projects and from …

View Details →

Model producers and users do not adequately assess the range of plausible outcomes and are overly reliant on central estimates that do not reflect inherent uncertainty. All modelled information is uncertain, and decision-makers need information on the possible range of outcomes to plan well, manage risks and make better decisions. …

View Details →

On the basis of two reports by the Comptroller and Auditor General, we took evidence from HM Treasury, the Finance Function, the Analysis Function and the Evaluation Task Force about the use of evaluation and modelling across government.1 Cabinet Office introduced cross-government ‘functions’ in 2013 to provide professional support to …

View Details →

HM Treasury has used the last two spending reviews to increase the emphasis on evaluation in departments, but acknowledges it has a lot more to do to address the persistent problems which prevent the provision and use of evaluation.17 Working with the Evaluation Task Force, HM Treasury asked departments to …

View Details →

Outside of the spending reviews, HM Treasury has few levers and exercises little scrutiny to ensure that departments comply with evaluation requirements and expectations.20 6 C&AG’s Report, Evaluating government spending, para 3.33; C&AG’s Report, Financial modelling in government, para 13, Figure 4 7 Qq 1, 18–19 8 Q 25 9 …

View Details →

Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting officer.24 Managing Public Money states that financial modelling and evaluation are an important part of the accounting officer’s role, and that they should comply with the Functional Standards.25 However, there continues to be important weaknesses …

View Details →

While HM Treasury is an important user of department’s models, it does not comprehensively scrutinise and challenge a department’s quality assurance arrangements for the models it uses. There is no requirement for the Aqua Book to be followed as part of the investment approval process nor as part of the …

View Details →

To address the lack of overall oversight for both evaluation and modelling standards, the Analysis Function intends to strengthen the governance framework for the Analysis Functional Standard, including introducing a self-assessment framework to assess performance against the standard.31 The Analysis Function told us there will be an expectation in the …

View Details →

Government does not always see data as a priority and data quality is often inadequate.36 The NAO reported in 2019 that the quality and sharing of data is a clear example of a neglected and poorly planned activity, and there is a culture of tolerating and working around poor-quality data.37 …

View Details →

When producing and using models, poor data can lead to additional time and increased difficulty when quality assuring the model inputs. This difficulty is compounded by a lack of data standards across government which means there are inconsistent ways of recording the same information.40 When dealing with new situations, such …

View Details →

Government’s view is that that transparency is critical for the public to understand the evidence behind decisions and how money is being spent, and that transparency should be the default position.44 The Analysis Function consider that transparency of evaluation findings allows departments to learn lessons from past projects and make …

View Details →

HMT told us it is only recently that it has started to ask departments to explain why they do not plan to publish some evaluations.48 The Evaluation Task Force raised the issue that there were evaluations that the centre of government did not know about because they had never been …

View Details →

Uncertainty is inherent in modelled information and should be considered as part of all analysis. HM Treasury believes any model that does not explicitly address uncertainty is deficient.59 Decision-makers need information on the range of outcomes that may occur and their relative likelihoods to develop contingency plans and to avoid …

View Details →

Despite the importance of uncertainty analysis, model producers do not routinely assess or communicate to users the uncertainty in their models. There is limited evidence of departments using uncertainty analysis or developing contingency plans, and departments generally present outputs as single estimates.64 Analysis of uncertainty can give clarity on the …

View Details →

Departments present outputs from their models to HM Treasury as part of the spending review and budget process. HM Treasury uses these outputs for forecasting, budget planning and to monitor emerging risks. Departments usually provide HM 55 Qq 52–53 56 Q 54 57 C&AG’s Report, Financial modelling in government, paras …

View Details →