2nd Report – Social media, misinformation and harmful algorithms

In the course of this inquiry, we identified five key principles that we believe are crucial for regulation of social media and related technologies: 1) Public safety: Algorithmically accelerated misinformation is a danger that companies and government need to address—the government and platform companies should work together to protect the public from it. 2) Free and safe expression: Neither government nor private companies should be arbiters of truth. Steps to tackle amplified misinformation should be in line with the fundamental right of free expression, with restrictions where proportionate and necessary to protect national security, public safety, health, or to prevent disorder and crime. 3) Responsibility: Users should be held liable for what they post online, but the platforms they post on are also responsible, especially with regard to the systems used to moderate, circulate or amplify content. 4) Control: Users should have control over both their personal data and what they see online. This includes the right to delete the data stored by platforms and services which is used to drive content and advertisement recommendation algorithms. 5) Transparency: The technology used by platform companies, including social media algorithms, has huge public safety implications, and should be transparent and accessible to public authorities. (Conclusion, Paragraph 6) Misleading and harmful content on social media

We launched this inquiry in the wake of the riots that followed the horrific attack in Southport in 2024. We received overwhelming evidence that online activity, including social media recommendation algorithms amplifying harmful and misleading content, played a key part in driving the unrest and violence. Social media companies’ responses were inconsistent and 52 inadequate, often enabling, if not encouraging, this viral spread, with evidence that they may have profited due to the heightened engagement. The evidence supports the conclusion that social media business models incentivise the spread of content that is damaging and dangerous, and did so in a manner that endangered public safety in the hours and days following the Southport murders. (Conclusion, Paragraph 14)

The Online Safety Act was not designed to tackle misinformation—we heard that even if it had been fully implemented, it would have made little difference to the spread of misleading content that drove violence and hate in summer 2024. Therefore, the Act fails to keep UK citizens safe from a core and pervasive online harm. (Conclusion, Paragraph 18)

We welcome Ofcom’s consultation on a ‘crisis response protocol’ for companies to follow in response to events such as the 2024 unrest. The protocol should directly address misinformation by including all online services at risk of contributing to the spread of false or harmful information, including large online social media, search and messaging services; those with smaller user numbers but high-risk profiles; and others, such as generative AI platforms. In establishing the mechanism, Ofcom should acknowledge the different ways in which different services operate. Following our Principle 2, it should hold platforms responsible for: decelerating the spread of harmful misinformation without censoring lawful speech; ensuring substantial and continuous engagement with law enforcement and government bodies; giving users control over the content they see; and providing transparency around their actions. (Recommendation, Paragraph 19)

Social media and other online platforms have huge power and reach into our lives, with positive and negative impacts. They can democratise knowledge and access to the public sphere, and help to build social connections and global communities. Generative AI provides further opportunities in terms of productivity, creativity and content moderation. For these reasons, it is imperative that we regulate and legislate these technologies based on the principles set out in this report, harnessing the digital world in a way that protects and empowers citizens. (Conclusion, Paragraph 25)

Internet users are exposed to large volumes of harmful and misleading content which can deceive, damage mental health, normalise extremist views, undermine democracy, and fuel violence. We are concerned by the evidence that recommendation algorithms—integral to the advertisement- and engagement-driven business models of social media companies—play a role in this. Young people are particularly vulnerable to these harms, and 53 those born today will never have known a world without AI—we plan to explore in detail the impact the online world has on their developing brains in our future work. (Conclusion, Paragraph 26)

The technology used by social media companies should be transparent, explainable and accessible to public authorities, as stated in our Principle

Social media companies have often argued that they are not publishers but platforms, abdicating responsibility for the content they put online. We believe that these services, with sophisticated recommendation algorithms that directly amplify and push content to users, are not merely platforms but curators of content. As we have seen, the amplification and spread of this content can have serious, large-scale impacts. We recognise that this is a complex area of law and that defining social media companies as publishers would have major consequences, but the current situation is deeply unsatisfactory. We call on the government to set out its position on this question in its response to this report. (Conclusion, Paragraph 28)

There is a shortfall in data needed to accurately analyse the scale of the problem and identify policy solutions. In line with our Principle 4, the government should commission a large-scale research project into how far social media recommendation systems spread, amplify or prioritise harmful content. This should be undertaken by a group of credible independent researchers, bringing diverse perspectives, with full access to the inner functions of the systems that major platforms use to algorithmically recommend content, including the private, external, and third party data used to train their systems; the user, content and engagement attributes the algorithms rely on and how these are weighted, and the objectives the algorithms are optimised for; where user interactions reinforce future recommendations; and any curation rules or interventions that influence promotion or suppression of content. We expect full cooperation from all major services that employ recommendation algorithms. (Recommendation, Paragraph 29)

Based on the research described above, the government should publish conclusions on the level and nature of harm that these platforms promote through their recommendation systems. Following our Principle 3, if significant harm is found, the responsible online services should publish the actions they will take to address these harms. Ofcom should be given the power to serve penalty notices to services that fail to comply, either 10% of the company’s worldwide revenue, or £18 million, whichever is higher. (Recommendation, Paragraph 30) 54

Following our Principles 2 and 3, the government should compel social media platforms to embed tools within their systems that identify and algorithmically deprioritise fact-checked misleading content, or content that cites unreliable sources, where it has the potential to cause significant harm. It is vital that these measures do not censor legal free expression, but apply justified and proportionate restrictions to the spread of information to protect national security, public safety or health, or prevent disorder or crime. (Recommendation, Paragraph 31)

As per Principle 4, users should have more control over the content that is pushed to them online. Government should mandate all online services with a content recommendation algorithm to give the user a ‘right to reset’, which would delete all data stored by their recommendation algorithm, in the manner that users can clear their cookie history. This option should be displayed prominently on the platform’s main feed or homepage. (Recommendation, Paragraph 32)

The UK government—like its counterparts around the world—is facing the challenge of attempting to regulate hugely powerful technology companies that operate across the world, providing technologies that transform societies, with bigger budgets than many countries. It is essential that their impact on our society be understood, effectively scrutinised and, where necessary, regulated in the public interest by Parliament. The committee has experienced some of the challenges of this in its engagement with these companies. We were reassured by the statements from Google, Meta, TikTok and X in our evidence session that they accepted their responsibility to be accountable to Parliament, and we hope that this will be put into practice. (Conclusion, Paragraph 39)

We are concerned by disjointed approaches from platforms to false and harmful content; in particular by recent moves from X and Meta to water down their Terms of Service and approach to content moderation. While there are merits to crowd-sourcing models of context provision and fact-checking—as part of a wider policy on misleading and harmful content—these platforms seem to be prioritising this method over third- party fact checking without clear evidence on whether it will adequately protect users from algorithmically amplified harm and misinformation. (Conclusion, Paragraph 40)

In line with our Principle 1 of tackling amplified misinformation, the government should compel platforms to put in place minimum standards for addressing the spread of misleading content online. More information is needed on the merits of different approaches to this. The government should commission research into the relative benefits of independent third-party fact-checkers, crowd-sourced context provision, and AI driven detection of misinformation, using researchers who are independent, bring 55 diverse perspectives on the issue, and have full access to the data of these systems. The research should enable Ofcom to offer guidance on the most effective method, or combination of methods, to address misinformation. (Recommendation, Paragraph 41)

The Online Safety Act will lead to some improvements, but is designed only to protect users from harm that is illegal or affects children. The decision not to include measures related to the algorithmic amplification of “legal but harmful” content, such as misinformation, means that full enforcement of the Act would have made little difference to the online environment that helped to incite the violence of last summer. (Conclusion, Paragraph 45)

It is vital that platforms are held responsible for the algorithmic spread of misleading or deceptive content that can radicalise and harm users. The few measures in the Act that address misinformation fall short. The False Communications offence is vaguely worded and will be difficult to implement; the advisory committee on mis/disinformation has been renamed, suggesting a change of emphasis; and the media literacy strategy does not set ambitious goals. (Conclusion, Paragraph 46)

The broad scale—and serious impact—of misinformation online requires greater transparency and accountability from the government. In line with our Principle 1, the government should submit an annual report to Parliament on the state of misinformation online, tracking trends and issues from the year, and setting out successes and failures in addressing them. (Recommendation, Paragraph 47)

In line with Principle 5, transparency, the government should introduce duties for platforms to undertake risk assessments and reporting requirements on legal but harmful content, such as potentially harmful misinformation, with a focus on the role of recommendation algorithms in its spread. (Recommendation, Paragraph 48)

To ensure true responsibility from platform companies, as per Principle 3, Ofcom and DSIT should confirm that services are required to act on all risks identified in risk assessments, regardless of whether they are included in Ofcom’s Codes of Practice. (Recommendation, Paragraph 49)

The Online Safety Act does not do enough to address the risks posed by small platforms due to its exclusive focus on size. Ofcom should create an additional category to cover ‘small but risky’ platforms, based on analysis of the role that harmful smaller platforms can play in the online ecosystem, interacting with the recommendation algorithms of large platforms to spread harms such as misinformation, and disinformation campaigns. This regulation of small platforms should be in line with our Principles 1, 3 and 5. (Recommendation, Paragraph 55) 56

Foreign interference and disinformation campaigns, with use of technology such as bots and AI, put UK citizens at risk. The possibility that some of the divisive messages and deceptive content spread by users—and amplified by algorithms—last summer were part of such an influence operation is deeply concerning. In order to tackle amplified disinformation, identified by Principle 1, the government and Ofcom should collaborate with platforms to identify and track disinformation actors and the techniques and behaviours they use to spread adversarial and deceptive narratives online. (Recommendation, Paragraph 58)

Responsibility for tracking foreign disinformation campaigns appears to be split between several departments, including DSIT. This suggests that the Intelligence and Security Committee’s 2020 characterisation of countering Russian influence operations as a “hot potato”, passed between different bodies, has not been addressed. To meet our Principle 1, the government should clarify which department has ownership over tracking and countering online narrative operations. It should consider consolidating responsibility within a single entity, for example the National Security Online Information Team, or establishing a clear chain of command, and in its response to this report the government should set out the actions it intends to take in this regard. (Recommendation, Paragraph 59)

The NSOIT is an important tool in protecting citizens from disinformation and needs appropriate scrutiny. Government should place NSOIT on a statutory footing and bring it under the remit of the Intelligence and Security Committee, to ensure that our Principle 1 is being effectively and safely pursued, in line with Principle 2. (Recommendation, Paragraph 62) Generative AI

The Online Safety Act does not protect users from the commodification of synthetic mis/disinformation, or provide effective transparency for the systems that produce them. It fails to address the issue of tech companies rolling out experimental features that can feed false or harmful information to their enormous audiences, further threatening information integrity online. This has damaging effects on all users, undermining the reputation of the companies that introduce them. (Conclusion, Paragraph 74)

We are concerned at what appears to be contradiction and confusion between regulators and government over the capabilities, limitations and principles behind the Online Safety Act. We expect senior Ofcom officials and ministers to be fully aligned in their understanding of the Act—particularly in relation to critical issues such as misinformation and generative AI. Ofcom at times appeared complacent in its approach to 57 public safety online, failing to live up to its role as the UK’s online safety regulator and slipping into the role of mediator between the industry and the consumer. (Conclusion, Paragraph 75)

To protect citizens from the AI-exacerbated spread of misinformation and harm, the government should pass legislation that covers generative AI platforms, bringing them in line with other online services that pose a high risk of producing or spreading illegal or harmful content. Following the Principles identified by this report, this legislation should require generative AI platforms to: provide risk assessments to Ofcom on the risks associated with different prompts and outputs, including how far they can create or spread illegal, harmful or misleading content; explain to Ofcom how the model curates content, responds to sensitive topics and what guardrails are in place to prevent content that is illegal or harmful to children; implement user safeguards such as feedback, complaints and output flagging; and prevent children from accessing inappropriate or harmful outputs. (Recommendation, Paragraph 76)

Principle 5 is crucial for addressing potential harms from generative AI, as there is currently a serious shortfall in transparency and oversight of the platforms and systems that allow users to create AI-generated content. The government should require providers of generative AI services to provide information to those carrying out independent research into online safety. This should include data such as platforms’ internal decision-making processes, training datasets, optimisation objectives, safety mechanisms and guardrails on outputs. (Recommendation, Paragraph 77)

To effectively tackle amplified misinformation as per Principle 1, the government should work with relevant experts and platforms to develop technology that automatically detects AI-generated media, meeting mis/ disinformation at its source. It should mandate all generative AI platforms, and platforms that employ generative AI technologies, to automatically label AI-generated media with metadata and visible watermarks that cannot be removed. (Recommendation, Paragraph 78) Digital advertising market

Advertising is crucial to major social media companies, which depend on recommending engaging content to increase time spent on their platforms and draw attention to adverts. Their recommendation algorithms do not effectively differentiate between harmless and harmful engaging content, which can result in promotion of misleading, damaging, or hateful material. The effects spread through the online ecosystem, helping to incentivise the production and spread of harmful content. (Conclusion, Paragraph 85) 58

The global digital advertising market is overcomplicated, opaque and under-regulated, operating through an enormous, automated and inaccessible supply chain. This directly leads to the production, viral spread and monetisation of harmful and deceptive content, often without advertisers’ knowledge. Platforms and advertisers appear to be either unable or unwilling to address this problem. We heard evidence that platforms may have profited from misinformation and hateful content after the Southport attack. (Conclusion, Paragraph 95)

In particular, we were concerned by evidence that Google may have helped to monetise misinformation relating to the attacks, contributing to the violence. This is unacceptable, and is just one example of a much wider problem with the digital advertising industry. We are concerned that Google was seemingly unaware of the chain of events when we asked them about it; failed to tell us how much revenue was earned from this; and failed to reassure us that the company would prevent this from happening again. (Conclusion, Paragraph 96)

There is a regulatory gap around digital advertising, as much of the regulation and interventions have been industry-led and focused on tackling harmful advertising content, as opposed to the monetisation of harmful content through advertising. We are not convinced that the digital advertising industry is able, or willing, to effectively self-regulate. The government’s reliance on industry-led, content-focused solutions, is insufficient to meet the current scale of harm. One industry-led intervention that saw some success in increasing transparency and reducing monetisation of harmful content, the Global Alliance for Responsible Media, ended following legal challenge. (Conclusion, Paragraph 97)

Tackling online harm means addressing the principles that incentivise and monetise its spread. In line with Principle 3, responsibility, the government should create a new arms-length body—not funded by industry—to regulate and scrutinise the process of digital advertising, covering the complex and opaque automated supply chain that allows for the monetisation of harmful and misleading content. Or, at the least, the government should extend Ofcom’s powers to explicitly cover this form of harm, and regulate based on the principle of preventing the spread of harmful or misleading content through any digital means, rather than limiting itself to specific technologies or sectors. (Recommendation, Paragraph 98)

To tackle the incentive behind amplified misinformation—namely, the monetisation of harmful content—there should be clear and enforceable standards for digital advertising market processes, as well as advertising content. Following our Principles 1, 3 and 5, government should ask the Advertising Standards Authority to establish comprehensive guidelines for 59 all actors within the digital advertising ecosystem and supply chain. These should be informed by the UN’s 2024 Guiding Principles for Information Integrity and developed in consultation with civil society, academics, experts, industry and policymakers. It should be designed to remove incentives for algorithmic acceleration of harmful or misleading content whilst upholding freedom of expression; ensure advertisers can avoid harmful content; and ensure transparency in technologies with public safety implications, such as digital advertising. (Recommendation, Paragraph 99)

The internet, and social media, could not operate without digital advertising. Given its implications for public safety, as per Principle 5, there needs to be heightened transparency in the market processes of online advertising. Government should mandate ‘Know Your Customer’ checks for participants in the programmatic advertising supply chain, as exists in other large markets. The government should also ensure that platforms disclose full ad campaign data, and allow independent third-party audits and vetting of ad exchange supply partners. (Recommendation, Paragraph 100)

There are insufficient disincentives for bad practice in the digital advertising market. Bad actors can exploit the ecosystem, monetising harmful content through major platforms. Following Principle 3, Ofcom should be empowered to give penalty notices to platforms when they allow harmful content to be monetised through their services. These penalties should be based on a formula that considers: the severity of harm, the amount of revenue the publisher received, the amount of revenue the platform received, and the number of individuals that encountered the harmful content. The revenue generated from these penalties should be used to support victims of online harms. (Recommendation, Paragraph 101) 60