Recommendations & Conclusions
11 items
6
Conclusion
Fourth Report - The Department for Work…
Rejected
DWP has not yet done enough to understand the impact of machine learning on customers and provide them with confidence that it will not result in unfair treatment. DWP is expanding its use of advanced data analytics to tackle fraud. This includes machine learning algorithms to flag potentially fraudulent benefit …
Read more
DWP has not yet done enough to understand the impact of machine learning on customers and provide them with confidence that it will not result in unfair treatment. DWP is expanding its use of advanced data analytics to tackle fraud. This includes machine learning algorithms to flag potentially fraudulent benefit claims, so the system learns and adapts without following explicit instructions. DWP says it is in an early stage of implementing these tools, but has already piloted them to tackle fraud in Universal Credit advances. There are legitimate concerns about the level of transparency around DWP’s use of these tools and the potential impact on claimants who are vulnerable or from protected groups. DWP has not made it clear to the public how many of the millions of Universal Credit advances claims have been subject to review by an algorithm. Nor has it yet made any assessment of the impact of data analytics on protected groups and vulnerable claimants; though we acknowledge it has recently committed to provide such an assessment in next year’s annual report. Although DWP has internal governance arrangements over its use of machine learning and performs some ongoing analysis of bias, the results so far have been largely inconclusive. 8 The Department for Work & Pensions Annual Report and Accounts 2022–23 Recommendation 6: DWP should, as part of the assessment in its annual report, consider explicitly the impact of data analytics and machine learning on legitimate claims being delayed or reduced, the number of people affected, and whether this is affecting specific groups of people. The Department for Work & Pensions Annual Report and Accounts 2022–23 9 1 The scale of fraud and error in the benefit system
Show less
Government response AI summary
The government disagrees with detailing specific metrics for publication, citing a need to avoid compromising fraud detection. However, it will report annually on the impact of data analytics on protected groups and vulnerable claimants, starting with the 2023-24 Annual Report.
Read full response →
HM Treasury
1
Conclusion
Fourth Report - The Department for Work…
Rejected
On the basis of a Report by the Comptroller & Auditor General (C&AG), we took evidence from the Department for Work & Pensions (DWP) on its 2022–23 Annual Report & Accounts and the level of fraud and error in the benefits it administers.2 We also took evidence from HM Revenue …
Read more
On the basis of a Report by the Comptroller & Auditor General (C&AG), we took evidence from the Department for Work & Pensions (DWP) on its 2022–23 Annual Report & Accounts and the level of fraud and error in the benefits it administers.2 We also took evidence from HM Revenue & Customs (HMRC) due to its role in administering National Insurance records.
Show less
Government response AI summary
The government disagrees with an implied recommendation regarding a 5% assumption, stating it cannot be compared to official fraud and error statistics in isolation due to various influencing factors.
Read full response →
HM Treasury
7
Conclusion
Fourth Report - The Department for Work…
Rejected
DWP estimates that it overpaid 12.8% (£5.5 billion) of all Universal Credit payments in 2022–23, which is much higher than any other benefit.10 We challenged DWP to explain why the fall in fraud and error promised in the Universal Credit business case has failed to materialise. DWP told us that …
Read more
DWP estimates that it overpaid 12.8% (£5.5 billion) of all Universal Credit payments in 2022–23, which is much higher than any other benefit.10 We challenged DWP to explain why the fall in fraud and error promised in the Universal Credit business case has failed to materialise. DWP told us that where legacy benefit claimants have migrated to Universal Credit, specific types of fraud—including earnings from employment and childcare— have fallen significantly.11 But it acknowledged that there has been a large build-up in what it calls the ‘stock’ of overpaid Universal Credit claims, which it says relate mostly to the pandemic. It added that it expects to address this primarily through Targeted Case Reviews.12 DWP estimates that the overpayment rate is particularly high for claims that started at the beginning of the pandemic - March 2020 to June 2020 - which during 2022– 23 were overpaid by 21.0%.13
Show less
Government response AI summary
The government rejects the committee's implied direction to explain the failure of fraud and error reduction, stating its commitment to a cost-effective control environment but highlighting external fraud trends beyond its direct control.
Read full response →
HM Treasury
8
Conclusion
Fourth Report - The Department for Work…
Rejected
We asked DWP to what extent the fact that 1 in 3 Universal Credit claims is incorrect is a result of the complexity of the system. DWP told us it is trying to make it easier for claimants to declare changes of circumstances through continuous improvements of the Universal Credit …
Read more
We asked DWP to what extent the fact that 1 in 3 Universal Credit claims is incorrect is a result of the complexity of the system. DWP told us it is trying to make it easier for claimants to declare changes of circumstances through continuous improvements of the Universal Credit system. It gave as an example where it has made changes to simplify the process for claimants to declare their level of savings. DWP added that Universal Credit underpayments are relatively low, which it believes is a result of bringing together legacy benefits.14
Show less
Government response AI summary
The government rejects the committee's implied criticism regarding system complexity, stating it is committed to reducing fraud and error but acknowledges external trends impacting fraud levels are not directly in its control.
Read full response →
HM Treasury
9
Conclusion
Fourth Report - The Department for Work…
Rejected
We challenged DWP to explain whether it still expects Universal Credit overpayments to fall to 6.5% as it had previously committed to. DWP explained that 6.5% was the level implied in the business case as a result of the expected reduction in fraud and error from merging legacy benefits into …
Read more
We challenged DWP to explain whether it still expects Universal Credit overpayments to fall to 6.5% as it had previously committed to. DWP explained that 6.5% was the level implied in the business case as a result of the expected reduction in fraud and error from merging legacy benefits into Universal Credit. It added that there is no reason to think that it cannot still achieve the expected reduction, but that this would now result in a higher rate than 6.5% because the baseline level of fraud and error has increased. It concluded 7 Qq 13–14 8 Q 15 9 Q 13 10 DWP ARA 2022–23, page 272 11 Q 16 12 Q 16 13 DWP ARA 2022–23, page 111 14 Q 17 The Department for Work & Pensions Annual Report and Accounts 2022–23 11 that it might be that DWP is doing everything it possibly can but still does not achieve 6.5%, and that the key is doing that all it reasonably can and clearly demonstrating that its control activities are cost-effective.15 Forecasting future levels of overpayment
Show less
Government response AI summary
The government rejects the committee's implied direction to explain how it will achieve the 6.5% overpayment target, stating it's committed to reducing fraud and error but external trends impact the level of fraud, which is outside its direct control.
Read full response →
HM Treasury
13
Recommendation
Fourth Report - The Department for Work…
Rejected
We have previously found that the DWP lacks the ability to demonstrate that its counter-fraud activities are having the intended impact and are cost-effective.22 Alongside its forecast that benefit overpayments will not return to pre-pandemic levels until 2027– 28, DWP has set a target to achieve £1.3 billion of fraud …
Read more
We have previously found that the DWP lacks the ability to demonstrate that its counter-fraud activities are having the intended impact and are cost-effective.22 Alongside its forecast that benefit overpayments will not return to pre-pandemic levels until 2027– 28, DWP has set a target to achieve £1.3 billion of fraud and error savings in 2023–24. It has also published a new estimate of the amounts saved by its counter-fraud activities23. 15 Q 100 16 Q 14 17 DWP ARA 2022–23, page 300 18 Q 13 19 Q 98 20 Q 99 21 Correspondence from HMRC dated 28 September 2023 22 Committee of Public Accounts, The Department for Work and Pensions’ Accounts 2021–22 – Fraud and error in the benefit system, Twenty-Sixth Report of Session 2022–23, HC 44, 9 November 2022 23 DWP ARA 2022–23, pages 302, 303 12 The Department for Work & Pensions Annual Report and Accounts 2022–23 The NAO has reported that, taken together, DWP’s forecast, target and savings estimate should improve accountability by providing transparency on its performance in tackling fraud and error.24 24 DWP ARA 2022–23, page 274 The Department for Work & Pensions Annual Report and Accounts 2022–23 13 2 Systemic underpayments of State Pension Progress correcting underpayments relating to historical error by DWP staff
Show less
Government response AI summary
The government rejects the recommendation, stating that while DWP is committed to reducing fraud and error, external trends impacting the level of fraud in the benefit system are not directly within its control.
Read full response →
HM Treasury
29
Conclusion
Fourth Report - The Department for Work…
Rejected
DWP is investing some £70 million to March 2025 in expanding its use of advanced analytics to tackle fraud. This includes using machine learning algorithms to flag potentially fraudulent benefit claims. DWP has already piloted an algorithm to detect fraudulent Universal Credit advances claims.68 The NAO reports that DWP is …
Read more
DWP is investing some £70 million to March 2025 in expanding its use of advanced analytics to tackle fraud. This includes using machine learning algorithms to flag potentially fraudulent benefit claims. DWP has already piloted an algorithm to detect fraudulent Universal Credit advances claims.68 The NAO reports that DWP is now actively developing similar tools for the four main risk areas of Universal Credit. We have reported previously that DWP could be more transparent in its use of machine learning in order to support public trust in the fairness of the benefit system.69
Show less
Government response AI summary
The government rejects detailing specific metrics for publication on data analytics' impact, citing the need to avoid compromising fraud detection. However, it reaffirms its commitment to reporting annually on the impact of data analytics on protected groups and vulnerable claimants, with the first assessment in …
Read full response →
HM Treasury
30
Conclusion
Fourth Report - The Department for Work…
Rejected
We received written evidence from the Child Poverty Action Group and from the Public Law Project expressing concern about the potential unfairness of machine learning, particularly with regard to vulnerable claimants and people with protected characteristics.70 We asked DWP whether it understood the concerns of people who have warned of …
Read more
We received written evidence from the Child Poverty Action Group and from the Public Law Project expressing concern about the potential unfairness of machine learning, particularly with regard to vulnerable claimants and people with protected characteristics.70 We asked DWP whether it understood the concerns of people who have warned of unintentional bias in its use of machine learning. DWP assured us it shared these concerns and that is why a human always makes the final decision on whether to make a benefit payment.71
Show less
Government response AI summary
The government rejects detailing specific metrics for publication on data analytics' impact, citing the need to avoid compromising fraud detection. However, it reaffirms its commitment to reporting annually on the impact of data analytics on protected groups and vulnerable claimants, with the first assessment in …
Read full response →
HM Treasury
31
Conclusion
Fourth Report - The Department for Work…
Rejected
We challenged DWP to explain how it would address the risk that legitimate benefit claims are unfairly delayed or reduced as a result of an algorithms targeting innocent behaviour, such as frequent changes of circumstances. DWP acknowledged that some level of algorithmic bias is to be expected because of how …
Read more
We challenged DWP to explain how it would address the risk that legitimate benefit claims are unfairly delayed or reduced as a result of an algorithms targeting innocent behaviour, such as frequent changes of circumstances. DWP acknowledged that some level of algorithmic bias is to be expected because of how benefit payments work, for example Universal Credit payments are higher for people aged over 25, so older claimants are more likely to be flagged because fraudsters will tend to claim to be older. It asserted that while there “clearly is a hypothetical risk” of unfair impacts on claimants, that there is no evidence of that risk manifesting now. It explained that it is performing analysis regularly to identify bias in the outputs of its algorithms.72 But the NAO has reported that so far this analysis has been largely inconclusive because of limitations in the available data about claimants.73 DWP told us it did not want to provide any further detail on how it will prevent unfair impacts to avoid tipping off potential fraudsters.74
Show less
Government response AI summary
The government rejects detailing specific metrics for publication on data analytics' impact, citing the need to avoid compromising fraud detection. However, it reaffirms its commitment to reporting annually on the impact of data analytics on protected groups and vulnerable claimants, with the first assessment in …
Read full response →
HM Treasury
32
Conclusion
Fourth Report - The Department for Work…
Rejected
DWP also told us it did not want to reveal when it planned to go live with machine learning on a large scale to avoid informing potential fraudsters, but added it was 65 Qq 84–90 66 Q 90; DWP ARA 2022–23, page 308 67 Qq 84–85 68 DWP ARA 2022–23, …
Read more
DWP also told us it did not want to reveal when it planned to go live with machine learning on a large scale to avoid informing potential fraudsters, but added it was 65 Qq 84–90 66 Q 90; DWP ARA 2022–23, page 308 67 Qq 84–85 68 DWP ARA 2022–23, pages 102, 308 69 Committee of Public Accounts, The Department for Work and Pensions’ Accounts 2021–22 – Fraud and error in the benefit system, Twenty-Sixth Report of Session 2022–23, HC 44, 9 November 2022 70 DWP0007; DWP0008 71 Q 101 72 Qq 101–103 73 DWP ARA 2022–23, page 309 74 Q 103 The Department for Work & Pensions Annual Report and Accounts 2022–23 19 working closely with the relevant authorities and that Ministers would be aware of its plans.75 However, DWP claimed that it is “taking it very slowly” with regards to rolling out machine learning. It explained that its pilot algorithm to detect fraud in Universal Credit advances did not work very well at first and needed to be tested and iterated using a small number of cases before being released for wider use. It added that it intends to follow this approach going forward and will not roll out new algorithms more widely until they have reached a level of accuracy that avoids unnecessarily holding up legitimate payments.76
Show less
Government response AI summary
The government rejects detailing specific metrics for publication on data analytics' impact, citing the need to avoid compromising fraud detection. However, it reaffirms its commitment to reporting annually on the impact of data analytics on protected groups and vulnerable claimants, with the first assessment in …
Read full response →
HM Treasury
33
Conclusion
Fourth Report - The Department for Work…
Rejected
In our November 2022 report on DWP’s 2021–22 accounts we recommended that DWP should report annually to Parliament on its assessment of the impact of data analytics on protected groups and vulnerable claimants.77 DWP told us it thought the right way to do this would be to report annually in …
Read more
In our November 2022 report on DWP’s 2021–22 accounts we recommended that DWP should report annually to Parliament on its assessment of the impact of data analytics on protected groups and vulnerable claimants.77 DWP told us it thought the right way to do this would be to report annually in its annual report and accounts.78 In correspondence after our evidence session it confirmed that would be the case, and that its first such assessment would be included in its 2023–24 report and accounts. DWP stated that its first assessment would provide a view on any bias detected, and whether this is in line with its expectation. DWP also stated that the assessment would provide indications of the type of mitigations put in place to reduce the risk of unfairness within the overall system, or actions taken to address issues.79 75 Qq 105–106 76 Q 102 77 Committee of Public Accounts, The Department for Work and Pensions’ Accounts 2021–22 – Fraud and error in the benefit system, Twenty-Sixth Report of Session 2022–23, HC 44, 9 November 2022 78 Q 104 79 Correspondence from DWP dated 24 October 2023 20 The Department for Work & Pensions Annual Report and Accounts 2022–23
Show less
Government response AI summary
The government rejects detailing specific metrics for publication on data analytics' impact, citing the need to avoid compromising fraud detection. However, it reaffirms its commitment to reporting annually on the impact of data analytics on protected groups and vulnerable claimants, with the first assessment in …
Read full response →
HM Treasury