Goldacre Review

Adopt the principles of RAP This is covered in the Open Working chapter, and is crucial. Data curation is a data management task. Data management is done in code: where there is a desire to share access to curated data, this means sharing access to re-usable data management code with adequate technical documentation, as per RAP. This must be more than an ethos: teams and individuals looking to change their way of working must be provided with training, tools, and platforms that support these working practices.

Set up an NHS Data Curation planning and delivery team, or similar, to own the challenge Regardless of the scale of work planned, the system has longstanding ambitions and needs in this space. A single source of knowledge and oversight on curation can provide open documentation on current state of the art, run deep dives on single topics, oversee investments from across the system, and identify opportunities. This should coordinate initial deep dive and ideas meetings with those parts of the community with existing expertise including: AphA, NHSR Community, academic groups, local and national analyst teams, NCDS, MCBK, NICE, EHDEN, OHDSI, and more.

Produce and maintain an open public library of data curation code The NHS should commit to produce and maintain an open public library of all data curation code, accompanied by appropriate technical documentation, that can be populated by any user of NHS data. This library should have the ability to store curation code, associated with a range of additional information including documentation and information validity checks; and provide facilities for annotation, “forks” (a technical term for derivatives of code), and citation or other forms of credit and tracking of use and derivative use. Code should be shared on a “user beware” basis, but capture the provenance of each entry including the user or organisation, so that those evaluating prior curation work can use this - alongside technical documentation and validity checks - in considering the extent to which they are happy to trust and re-implement existing code. It is crucial that this library is not solely a repository of accredited or approved curation code, or the outputs of a small number of pre-selected groups: it should admit all code, but have the facility to display to users which variables have been “assured” by specific organisations, as a tagged subset of all code within the library. This library should be maintained by a small team with domain knowledge around NHS data and its curation or use, attending closely to curation of collections, improvement of documentation, commissioning and evaluation of validity checks on higher value variables, and meeting library users’ needs. This team should engage in close informative 2-way dialogue with platform and curation teams. Generate NHS data curation code, and surface existing work

Mandate that all publicly funded data curation code is shared openly All data curation code written using standard tools against standard datasets in standard computational environments should be shared, mandatorily, for all publicly funded work, including all academic research, and all NHS service analytics, whether delivered by public or commercial organisations.

Identify five Data Pioneer teams to adopt open curation methods These should come from a mix of local NHS analyst teams, national NHS analyst teams, and academic teams, with prior evidence of working to RAP principles (or strong potential to do so), who can work to RAP principles in a TRE that supports RAP.

Ensure national programmes lead by example National data analysis programmes including QoF and the key national “variation in care” audits such as Model Health System, GIRFT and RightCare, all use national datasets, as well as some bespoke datasets, for their regular data reports. They are in a good position to lead by example on adopting new RAP working methods with standard data management tools in standard shared environments.

Capture, and openly share, existing curation knowledge around commonly used national datasets National datasets such as SUS/HES and GP data are commonly used: there is extensive knowledge and best practice from local and national teams around converting raw data into usable variables including in NHS Digital, the work done on the National Commissioning Data Repository, many national audits, and some academic projects. This should be identified and captured in shared, re-usable curation code and portable representations alongside technical documentation and any existing validity checks for each variable. To ensure delivery this work should be done under the aegis of one team tasked with identifying and surfacing best practice from these communities in the NHS Curation Library.

Use consistent environments to facilitate re-usable curation code Currently national datasets such as GP data and HES/SUS are stored and made available in many very different ways in a huge number of different national and local data centres. The system should aim to minimise variation wherever possible. This means minimising the number of locations where data is stored. Where multiple locations are needed, the same health data should always be stored in the same way; and it should be interrogable through a consistent computational mechanism.

Require use of national TREs for tasks using national datasets wherever possible This is likely to address a large number of local analyses, especially those aimed at benchmarking local activity against national activity.

Create and enforce consistent standards for local implementations of national datasets Wherever possible the same health data should always be held in the same form wherever it is stored or made accessible, not the many slightly different cuts, data models, or even column names that are seen across different settings. This will mostly require that all data, but especially national commonly used datasets such as GP data or SUS/HES, is held in in TREs, in its rawest forms, closest to what is collected at the coalface. Doing this will help to address the needless and uninformative heterogeneity of current data structures that obstruct code sharing.

Create and enforce standards for local TREs Local TREs should be either one single open source NHS TRE design, or conform to an extensive range of open standards, as per the TRE chapter (recognising that developing such standards in isolation may be more complex than developing a template open source TREs for local use). Doing this will help to ensure that code and skills are portable between teams and settings. Develop tools to facilitate re-usable curation code There will always some duplication of implementation of the same datasets in multiple locations. Two modest actions can ensure that data curation code is portable settings.

Develop standard tools to convert raw data into analysis-ready datasets The conversion of raw data into analysis-ready datasets should be done with common tools, regardless of setting, to ensure that all data management code is intelligible and re-usable by others. This will require the rapid development of standard functions and libraries (re-usable code and tools), most likely implemented in python or R, by a small range of national experts in collaboration with a broad group of technical users representing a wide variety of data curation and analysis needs, supporting command line and GUIs, that can be implemented in diverse settings. This will be radically more straightforward if all local settings can be obliged to adopt a standard local TRE model: ICS’s are a new set of organisations in the system, using data to improve care, and provide the perfect opportunity for such standardisation (see TRE chapter).

Develop portable representations of data management code There is a need for portable representations of data management actions, as described above. This will ensure that curation code is readable, understandable, re-usable, and portable between teams and settings, and let organisations communicate dataset specifications, or clinical popup and alert specifications, between them. This work must go beyond mere codelist sharing, and address complex phenotypes. It should be led by technical teams, and coordinated by the NHS, specifically the NHS Transformation Directorate; delivery should be led by a team of individuals with proven prior expertise in technical aspects of data management, open standards, informatics, and - lastly - health data management. This should start with a minimum viable product addressing the simplest variable types; and any standards should permit collaborative extension into more complex variables. The latter will require open competitive funding through traditional research funders to develop methodological work and applied code. Work should begin by rapidly delivering a detailed overview of prior art in this space including the related open work done for interoperability around projects such as FIHR and HL7, the excellent example of productive open working practices embodied by EHDEN and OHDSI, and evaluate opportunities in the complex work done for OpenEHR.

Run an open competitive funding call for foundational work on data curation Data curation is a complex methodological challenge. There are simple aspects of the work that can be met with simple “implementation” in code. There are also more complex aspects that require the development of new methods and abstractions. NIHR and/or UKRI should rapidly develop open research calls on a range of prioritised areas including work to address the following challenges: • Describing the quality and completeness of coding on key clinical areas. • Describing variation in coding behaviour between settings. • Developing methods and code for validation and description of data at scale. • Developing and evaluating interventions to improve the quality of coding, focused on specific clinical or geographical areas. • Developing optimal methods, tools, and training for codelist creation and related curation tasks. • Developing and implementing optimal methods for portable representations of complex clinical and demographic phenotypes. Any funding should also invite other innovative approaches to developing better curation of NHS data, but require a clear pathway to implementation in real NHS data analytics within two years. In order to access the best expertise and ideas, and surface the largest amount of prior knowledge, it is vital that all funding is open to applications from all, rather than granted to a closed organisation or group; and that resource is available on realistic timescales (6 months delay from award to commencing work; minimum two years resource). This is crucial, as informatics is a historically neglected space with pockets of excellence and innovation that are under- resourced, and will need time to scale as with other areas of innovative research.

Insist that all dataset requests are made in code Dataset requests should be made in code, not conversation. Data providers such as NHS Digital must be capable of receiving and supporting such requests; requesters must be capable of making them. The process of making a request from a data provider such as NHS Digital should entail writing in code the characteristics of the dataset requested for preparation, and how it is created from the underlying raw data. The public log of datasets released should include this code, not just a free text description of the project and dataset requested. This should be a non-exclusive arrangement, with the prior method of discursive dataset requests persisting in parallel, but such requests should be met by writing open standard code in-house which is then shared as with all other dataset request and preparation code. Delivering this will require that organisations providing data such as NHS Digital also provide well documented details of their underlying data models, datasets, data dictionaries, and so on. Develop capability in Clinical Informatics

Ensure there is clinical informatics training on medical school, post-graduate, and other clinical curricula Clinicians enter and use clinical data about their patients. There is a need for adequate core training on the purpose and importance of this work, how clinical data is stored, and how it is used for analytics and research. Identify and resource existing organisations such as Faculty of Clinical Informatics to develop training in undergraduate and postgraduate curricula, with openly accessible online training for those out of formal training.

Ensure universities have core capacity in clinical informatics Evaluate the best means to develop core capacity in universities, capitalising on any training work to embed practical and theoretical informatics research alongside it.

Support core capacity in clinical informatics There is a need for profession-building in this space. As an example, the Faculty of Clinical Informatics is small and funded by members. This limits its impact. It is unrealistic to expect individuals to resource the development of professional structures to meet national strategic needs: this should be supported by core investment.

Create a single form for all ethics, IG, and other data access permissions Researchers are regularly required to describe the same aspects of the same project in multiple different ways for multiple different organisations to approve multiple different aspects of their permissions separately. Researchers should be able to fill in a single form from a single starting point: the relevant sections of the form on different aspects of IG, ethics, and related issues should be accessible to each of the different relevant organisations from whom approval is sought. The current patchwork approach is duplicative, inefficient and risks issues falling through the gaps.

Streamline the number of approvals meetings Researchers regularly have applications rejected because of the need for clarification of another aspect of their permissions from another approval body. As with the single form above, where possible, whenever one project requires multiple permissions from multiple bodies, this should be addressed at a single meeting where all relevant bodies can collectively review and discuss all aspects of applications in one go. This will be more or less practical depending on the regional coverage of each committee: however, coordinating timings, and overlap, should be readily achievable in many settings. Different organisations can and should take responsibility for different considerations within their own remit: but there should be open conversation between them, and researchers should not have to repeat themselves, or experience delays because of complex concerns about overlap or non-overlap between different varieties of decision-making body.

Get researchers in the room Researchers regularly have applications rejected because of a misunderstanding, or a need for a clarification; when this happens, they may have to wait several months for the chance to have their application re-considered, after addressing the misunderstanding. Whenever a meeting is taking place, the applicants should be ideally be present: this should pose no obstruction to open and considered deliberation. Failing this, the applicant should be informed of the time and date that their application is being considered, and the committee given a telephone number where they can call the applicant in case any factual aspect of their application requires disambiguation.

Create an arbitrator for disagreements over specific access requests Disagreements over access should reduce when data controllership has been streamlined across the NHS, and where strong TREs reduce privacy risks. However, issues may still arise, especially when trying to link to datasets outside of NHS control. In these circumstances, an arbitrator should be able to step in and make the final decision. This, in combination with other recommendations, should help tackle issues related to conflicts of interest and monopolistic behaviour among those holding patient data.

Create a single map of all approvals The approvals process may seem simple to those administering it: even then, they may only have clear oversight of their own component. For those navigating the system, it is often profoundly confusing and complicated. A single map should be created of all required approvals, with links to access further detail at each stage. This should be reviewed by the organisations implementing each regulatory step: they should collaborate on and contribute to the descriptions and roles of their own processes. This map should be regularly reviewed for accuracy, and for opportunities to simplify the system.

Provide rapid unambiguous guidance when approval is not required When a piece of work is believed to sit outside the remit of a given organisation or process, the organisation responsible for that aspect of governance should confidently provide a clear statement that their involvement is not required. Complex aspects of governance may sometimes be a matter of judgement calls; organisations administering the governance processes are in a strong position to make those judgement calls and share their insights. (For clarity, this is often seen already: it should be welcomed, applauded, and its value recognised).

Establish two modest Centres for Regulatory Science Regulation of research, and information governance, will always be complex. The trade- offs inherent in different options will always be challenging to unpick. It is unrealistic to expect that this will ever go away, and it is unhelpful that there is very little “commons of knowledge,” advice, or critical review of current rules in public. At present, there is largely only “folk knowledge” on the part of applicants, rather than a rich commons of knowledge. In the US, the FDA (Food & Drug Administration) commissions, resources, and tasks a small number of practical research units evaluating the efficacy and utility of regulations, legislation and processes managing risks in healthcare, including through the Advancing Regulatory Science Initiative, and the bioethics research communities at universities. These have challenged and improved regulations in several areas. We should replicate this effort on a modest scale: UKRI or NIHR could fund small groups responsible for producing detailed critical reviews of interesting cases for discussion; critical analysis of existing and proposed regulations; clear advice for individual researchers and organisations seeking access to data; and feedback to the policy community on what is and is not working. The objective should be to create a rich, practical, public, critical commons of knowledge around governance. This should be modelled on the welcome expansion over the preceding decades of professional medical ethicists in universities. Staff should be multidisciplinary, technologists, clinicians, researchers, social scientists, and lawyers with experience of regulation and information governance.

Establish a clinic to help users who are blocked on data access MRC has previously funded excellent work at their Regulatory Support Centre to give support and guidance to individual MRC-funded groups encountering barriers to their work around data access or usage. UKRI and/or NIHR should expand this work, or augment a group from the Centres for Regulatory Science, to create an open problem-solving unit that invites reports on blocked projects, locally and nationally. This group should be focused on helping patients, clinicians, commissioners, and researchers overcome the practical, technical, regulatory, and cultural barriers they encounter when they are trying to access data, with practical guidance on how to ‘unblock’ access in ways that are safe, legally compliant, ethically viable and technically feasible for each situation. The aim should be to share this work, and create a growing library of themed insights and solutions, for each project describing the barriers encountered, and how they were overcome. This group should provide an annual open report to policymakers describing their work and any systemic issues or recurring themes that they have encountered. Two-track approval for TREs TREs substantially reduce the privacy risks - but not the ethical risks - involved in using NHS patient data. This should be reflected in the governance arrangements around access to data through a TRE.

Create a 2-track approval system to incentivise use of TREs The current complex IG arrangements were principally devised to manage the privacy risks that are inherent to more insecure methods of data analysis through dissemination, where there is a greater risk that data could be leaked, illegally viewed, or otherwise misused. TREs substantially address many of these issues; and many governance questions can be addressed once in a formal review of the TRE, rather than bespoke for each individual project. The NHS Transformation Directorate should conduct a formal review of which existing safeguards and processes can be accelerated or retired for projects conducted exclusively within a strong TRE. This should result in a substantially less onerous and faster access and approvals process than for work using conventional and less secure methods with data dissemination. This is proportionate, will explode productivity, and will actively incentivise better, safer ways of working.

Maintain excellent standards around governance issues not addressed by TREs Not all aspects of governance are rendered obsolete, or less important, by the introduction of TREs. Regardless of how data is analysed, the purpose for which it is used, and potential harms from this, must still be subject to careful scrutiny for each single analysis. Ethical review and PPIE should therefore persist for single analyses, subject to the efficiency improvements described above. However, any ethics and PPIE work on the facts and processes of data access itself, rather than the single specific analysis, should be done once only for the TRE as a whole, wherever this is feasible and appropriate for the single analysis in question.

Review the National Data Opt Out Policy after TREs are established The National Data Opt Out policy was introduced in response to the crisis in public trust caused by the problematic implementation of care.data in 2013. It has, however, been problematic in implementation, inconsistently applied, and can give patients the impression that their data will never be used for purposes other than direct care. It should be reviewed, but only after a strong national TRE has been established for use of GP data and other commonly used national datasets, as above. If patient data is only ever stored securely, never directly ‘seen’ by researchers, and used transparently, then there may be fewer circumstances in which it is logical to allow people to opt out; or opt-outs could be reviewed to cover different classes of use, rather than different classes of data flow. Any changes should be carefully considered with meaningful input from patient and public representatives; following adequate research into the motivations of opt-out usage; and retrospective changes to current opt-outs should be handled with great caution. Nonetheless, TREs present an opportunity, if not a guaranteed route, to carefully develop a new accord with the public around restrictions on use of their data.

Uphold the commitment that the NHS Digital GPDPR dataset will not be disseminated This dataset is unprecedented in detail and coverage, it cannot be meaningfully pseudonymised by the removal of direct identifiers. It is very welcome that this data will now only be accessible in a TRE. It is critical that this commitment is adhered to. Regulation and Legislation

Revise the definitions of “anonymous” “identifiable” and “linked” data; add a new category of “pseudonymised but re-identifiable” There has been a widespread misapprehension across the system around pseudonymised data, and an excessive confidence in the privacy protections provided by the removal of direct identifiers. This misunderstanding has been pivotal in a range of problematic decisions around risk management. However, it is in part a consequence of the currently available formal categories of risk for datasets relating to patients. There have historically been attempts to create simple nomenclatures that describe whether data is identifiable or not on the basis of the data alone. Current commonly used categories of data are, in brief, limited to: “anonymous data” (for example, “3,200 people died of cancer last year in Norfolk”); “identifiable data” (for example, data with name and address openly stated for each patient); and “linked data” (where one dataset has been linked to another). These three categories are insufficient to describe the challenges faced in secure management of detailed NHS electronic health records data, and in particular they do not capture one of the most commonly encountered forms of data. There is a fourth category - “pseudonymised but readily re-identifiable data” - which should be formally added into common parlance, regulation, and legislation. The extent to which this data presents a privacy risk is a function of the data itself, and the context in which it is being accessed: if disseminated off-site, where a user can interact with it as they please, with no logs of their activity, this pseudonymised rich data is profoundly insecure; in a robust TRE with barriers to viewing disclosive data and logs of all activity, then it is securely held and presents few privacy concerns. It is crucial that the system recognises and describes this category of data as a central privacy risk to be mitigated: recognising this will allow the system to make informed choices and earn the trust of campaigners, professionals, and the public. This issue is increasingly discussed, and is the subject of a current ICO document on anonymisation, put out for consultation in Autumn 2021. However, the concept remains overall rather nameless in regulation and legislation, falling between stools, with work largely guided by interpretations and guidance: it should be a central focus of legislation and regulations that aim to address privacy issues.

Consider including health data in the Digital Economy Act Increasingly health data needs to be linked with other administrative datasets to understand, for example, the social determinants of health. This would be made easier if all public datasets were governed in the same way. There will need to be extra protections for health data, as outlined in detail elsewhere, but it may be unnecessary to have an entirely separate legal framework. The Digital Economy Act (2017), and the inclusion of health data within it, should be formally evaluated with this in mind. As with the above recommendation about the National Data Opt Out Policy, any changes should be carefully considered with meaningful input from patient and public representatives. Health data, linked to non-health data, should only be accessible in a robust TRE that prevents direct access to patient data and shares informative logs with the public to ensure complete transparency about all uses to which the data are put. ONS has deep technical knowledge and history in this space, and should be regarded as a beacon for future work, including through their forthcoming Health Strategy. As with so many recent projects where data access has been facilitated in a time-limited fashion by pandemic legislation, the Public Health Data Asset work between ONS, the NHS and Public Health England shows the power of wider linkage.

Appropriately sanction those who are caught deliberately and maliciously attempting to re-identify individuals in patient records There are numerous accounts of people inappropriately accessing fully identifiable records without consent or legal basis, or otherwise misusing patient records. When this happens - in the case of, for example, celebrities being admitted into hospital - the individuals are often caught, and disciplinary action is taken. However, while it continues to be possible to download detailed, pseudonymised but readily re-identifiable patient records to local machines the true scale of inappropriate use is unknown. Wider use of strong TREs will make it easier to detect deliberate attempts to reidentify individuals, but this may not be a sufficient deterrent if the consequences remain minimal. Regulators across the system, including professional bodies, the ICO, MHRA (Medicines Healthcare Regulatory Agency), and HRA (Health Research Authority) should coordinate to develop and implement appropriate fines and sanctions for those caught deliberately breaching patient privacy. A strong deterrent for any individual person misusing health data must be regarded as a crucial component of any robust regulatory framework if it is to achieve its objectives. This should not be regarded as a barrier to wider and better use of data to improve patient care: it should be regarded as a key facilitator of that objective.

Disclose all data flows leaving NHS organisations in one place Throughout this review we have received detailed descriptions of many substantial bulk flows of NHS patient data for service research and research, including complete patient records, outside of local NHS and associated organisations, including GP practices, on the basis of approval by local organisations. Current recipients are diverse and include NHS organisations (such as NHS Clinical Commissioning Groups); NHS adjacent organisations such as GP Federations, NHS Commissioning Support Units, Academic Health Sciences Networks; large and small scale commercial providers of analytic assistance to NHS organisations (individually or as groups) running their own data centres with NHS patient records; and a broad range of less visible participants including private providers of research services, private providers of case-finding services, private providers of administrative services to GP practices, and several privately and publicly owned GP research datasets, extracting and granting onward access to millions of patients’ data without patient consent. From our interviews, it was clear that many in the system are unaware of the extent of this data dissemination. This may be because attention has historically focused more on national data moves administered by national organisations such as NHS Digital. Some of these data flows were in our view, and others’, disproportionate to the apparent stated objectives of the work. There are strong grounds to believe that the work done or intended to be done from these data flows could be achieved more effectively, and more securely, in a national secure TRE. The NHS Transformation Directorate and DHSC should commission or conduct a detailed open review of these data flows to establish in some detail whether they are safe, proportionate, and can be replaced with more secure options. This work may raise some currently undocumented concerns; however, these are likely overall to form the basis of a stronger case for a single GP data flow into a national secure TRE. Once reviewed, all data flows should be logged in a central system that is visible and public.

Create a central repository of DPIAs, DSAs and related documents for local NHS data flows Many of the NHS data flows currently in place, especially for local projects, are not well known even to clinicians, policymakers and researchers. There are detailed governance requirements around paperwork and disclosure for individual organisations, however information about these flows is often not easily discoverable. The NHS Transformation Directorate should establish a central indexable repository of DPIAs, DSAs, privacy notices, and other related documentation for all small local data flows so that this information can be searched and viewed by interested parties. This should not impose any additional burden on NHS organisations, as it entails solely the sharing of existing documents. Alongside greater transparency and visibility it is likely that this will also help build a more robust commons of knowledge around best practice for such activities, and their appropriate documentation.

Produce boiler-plate templates for patient consent for data use and dissemination There will be certain circumstances where it may be necessary to provide local downloads of patient data, for example, for patient follow­ up in clinical trials. In these circumstances the patients must have given explicit consent for their data to be accessed for research purposes outside of a TRE. Current mechanisms for gaining such consent are variable in quality. Central provision of clear boiler-plate templates for consent to disseminate patient records will raise standards and improve public trust.

Simplify the rules governing use of posthumous data Posthumous medical records are an essential resource for almost all health data research and analysis: studying the records of people who have died is one of the most effective ways to understand how to prevent death. Yet the rules governing the use of posthumous records are confusing and inconsistent. Different types of record are kept for differing lengths of time, in different circumstances, with different access mechanisms, before being destroyed. One team should be charged within he NHS Transformation Directorate or similar to harmonise all requirements, with a view to improving research; data should be held securely in TREs as with living patients’ data; there should be an assumption that records are preserved, with a clear commitment that they will be securely managed for all time, to the same standards as for living people; patient opt-outs should be respected for this class of data to the same standards as those used for living people.

Address the “multiple permissions” problem NHS patient data is a vital and powerful resource for improving the quality, safety, and efficiency of healthcare. This requires that data is accessible. The current requirement to obtain permission separately from each organisation for each act of data sharing is a substantial practical barrier to better harmonisation, and better access. It is driven by the current legal reality that each organisation is the Data Controller for the records they hold. Two options may help to make this more manageable, subject to detailed legal and policy evaluation, and public and professional consultation. Firstly, consideration should be given to whether a national organisation could become Data Controller for a copy of all NHS patients’ records, to be held only ever in a secure national TRE (as is planned for some GP records in the General Practice Data for Planning and Response programme), where it can be worked on for the purposes of service improvement, academic research, and foundational work into data curation and harmonisation. Secondly, and less effectively, consideration should be given to creating an “approvals pool”, where large numbers of Trusts, GP practices and other NHS organisations can voluntarily nominate, with strong national and system-wide support, a single entity that is legally empowered to review and approve data access requests on their behalf, according to shared common principles, with the detailed consideration that comes from a robust economy of scale in making a large number of decisions for a large number of settings, rather than a small number of reviews in a single organisation. Addressing specific roles and uses

Start an overdue public and professional discussion on performance management The issue of data being used for performance management is informing a number of strategic choices around data access and analytics without open public discussion. Data plays a vital role in contributing to the improvement of quality, safety, and efficiency in public services. There should be a more robust and professionalised commons of knowledge around this work, as per the chapter on Modernising Service Analytics. Overall, this is a complex and also political area: it should therefore be the subject of a single piece of policy work by the NHS Transformation Directorate in consultation with the wider community to facilitate frank discussion. In advance of detailed work, the following outline principles are proposed, but only as subject to more detailed evaluation: 1. Legitimate skilled users from national government and national NHS organisations should be entitled to access NHS data for performance monitoring. 2. Organisations whose data is used in such projects should be carefully consulted. 3. Any concerns they raise should be carefully considered 4. Any concerns they raise ahead of analysis should be clearly recorded. 5. Any national organisation imposing costs or inconvenience on health services with metrics claimed to be substantially flawed or uninformative should be subject to expert review and possible censure by an appointed body with appropriate technical skills (such as the UK Statistics Authority). 6. Any prior track record of poor quality or misleading analytics should be considered when considering future data access requests; serious breaches should result in revocation of access to data from patients’ records.

Ensure DHSC can access data when appropriate The management and delivery of NHS care is complex, changeable, and spread across a wide range of organisations. However, we have not encountered any convincing argument why analysts in the Department of Health and Social Care should not be entitled to a legal right to access and process health data for the purposes of operational research and policy analysis, as set out in the NHS Act 2006. A combination of technical and organisational factors means that DHSC analysts are often blocked from accessing data, or at least substantially delayed, which can have significant ramifications for timely policy development, ministerial advice, and operational improvement. Rules regarding the mechanism of access, for example only within a licensed TRE and only by accredited researchers, should be the same as they are for the rest of the system. If the broader access afforded by secure national TREs cannot resolve this problem then a review of legislation and regulation should explore other means by which DHSC can gain access to run analyses on patient data, without viewing individual patients’ records directly.

Start an overdue public discussion about commercial access TREs make a profound contribution to commercial work with NHS data, because they completely separate two distinct issues: the protection of patients’ privacy; and the wider ethical and political judgement about the appropriateness of commercial access. However, while addressing the issue of privacy - and detaching it from the wider ethical, political and strategic issues - TREs do not address those other issues. There is a need for a frank public discussion about commercial use of NHS data (with due consideration to the upcoming National Data Guardian work on Public Benefit). This should not be regarded as deciding all aspects, but should inform the decision. It should be executed through citizens’ juries alongside other forms of public consultation, and include a frank and informative explanation of the key role that commercial actors play in innovation of medicines, services, and digital technologies; due recognition of their potential conflict of interest (as is already well recognised); and due consideration of the best means to mitigate that risk. This discussion should take place after the system has adopted TREs, so that the ethical or public preference aspects of data use are not confused with privacy challenges. There is a very extensive array of detailed policy work with large teams in diverse settings across government around the best mechanisms for appropriate cost recovery, and commercial engagement; these arrangements sit outside the terms of reference of this review. Nonetheless it is the overall view of the chair (BG) that commercial use of data should be welcomed within a sensible legal and regulatory framework; that the risks of misuse, poor quality analytics, and COI are substantially present in non­ commercial users of data; that all uses should be in a TRE that shares complete activity logs; that the NHS should negotiate intellectual property rights in any innovations derived from access to NHS patient records data.

Negotiate co-ownership of claimed commercial innovations from NHS data When a company develops an “algorithm” such as a risk prediction tool they typically apply existing tools, techniques and code libraries to huge, richly detailed health datasets that were collected at great cost. The code libraries used for this work, such as those to implement random forest “AI” models, are themselves often open source. This is not to diminish the effort, application, creativity and problem solving that such work entails. However, it is a fact that the successful delivery of such an algorithm is driven in very large part by the availability of extremely detailed health data to drive the models. This data did not just appear in the lap of the NHS. It has been collected at great expense, over many decades: it has been created, curated, matched, enhanced by the collective effort of hundreds of thousands of clinical and administrative staff in the NHS, and tens of thousands of technical staff. It would be very wrong for the value in such work to be captured exclusively by the group that executed the code. The relative contributions of the underlying data, and the individual teams using it, will vary from project to project. We suggest that if commercial users approach the NHS or government seeking access to data to develop proprietary code or tools in this way then a negotiation should be conducted at the outset around the profit share between the NHS and the users; that these should be made public not solely for transparency but more to help the NHS get informative public discussion and feedback from technical and IP experts on whether they are managing the share correctly; and that all data management and curation code created during the project should be shared as open, to maximise network benefits, avoid the current repeated duplication of curation effort, and ensure that benefits accrue to the NHS even from projects that do not deliver a commercial output or attendant revenue. This data management code can be captured simply as all activity will be within the TREs. For clarity, this is a very different scenario to publicly funded code: this should be managed as above.

Address exclusive commercial arrangements This is an evolving space that has been the subject of various national policy initiatives over time, and substantial ongoing work across government. Overall, exclusive commercial arrangements are likely to be disadvantageous to the NHS as a whole. They will often represent local challenges at well-intentioned organisations: NHS staff have described feeling disadvantaged when negotiating with large commercial organisations, or even academic institutions, who may both have greater legal and commercial expertise on data access issues. The work of the NHS Transformation Directorate business unit ‘The Centre for Improving Data Collaboration’ should help ease this information and power-imbalance, by providing both ‘off­ the-shelf guidance for data partnerships, and bespoke advice. This work is important and should continue to be supported, underpinned by an acceptance that there is unlikely to ever be a ‘one-size-fits-all’ contract and so there will need to be a degree of pragmatism and contextual flexibility. Given the sensitivity of the topic, any guidance regarding commercial data partnerships should be informed by careful deliberate engagement with patients and different publics. Patient and Public Involvement and Engagement Good PPIE is crucial, and valuable. The team do not claim deep, detailed, technical knowledge around best practice in PPIE specifically. However, from the perspective of extensive prior engagement work - and multiple detailed discussions across the community of those conducting, using, and supporting research - we respectfully suggest that the following changes to the ways in which PPIE is funded, commissioned, and conducted could be considered.

Ensure PPIE expectations are proportionate to the sensitivity and scale of the project. Expectations from funders and regulators around PPIE can sometimes be the same for both large and small projects: this can be unrealistic, and may sometimes act as a barrier to younger and less resourced individuals or teams accessing funding and data. This is especially the case when funders require long and detailed PPIE work to be done in advance of any funding being acquired, at the application process. At minimum this additional cost prior to funding requests should be recognised and - if extensive pre-application work is deemed appropriate - this should receive specific resource that is accessible to all applicants at all levels, through all university departments; there should be due audit or exceptions reporting to determine whether this PPIE facility is available to all. More broadly: expectations regarding the quantity - not quality - of PPIE can and should be contextually flexible to the size of the project, and this should be clearly stated.

Provide researchers with easy access to practical guidance, and examples of best-practice. Good engagement requires deep and specific professional skills that may fall outside the skillset of data scientists or research software engineers. Ideally this skills-gap should be filled by embedding a social scientist, data ethicist, and/or engagement professional in the research team. This is not, however, always possible. PhD students, for example, do not necessarily have this option available to them. In these instances, it would be helpful for these researchers to have access to practical tools, resources, and guidance that can help them conduct high- quality PPIE. Examples of practical guidance do exist, but these can be hard to find for those who do not know where to look. Developing a central repository where resources can be shared and signposted to would help significantly.

Resource and give a platform to experts in building public understanding. Engagement as a two-way process is extremely important; but understanding is important too. Patients are entitled to a clear, adequately detailed, accessible description of what their data is actually being used for. The Understanding Patient Data website is an extremely strong example of good practice in this regard, with their diverse and thorough case studies at varying levels of technical detail. It is disappointing to see that this programme has recently been closed by Wellcome; it is to be hoped that the work will find a strong new setting.

Consider centrally commissioning PPIE on common causes of concern Multiple standalone PPIE projects on multiple individual analytic projects have a clear role, and strong support. However, for commonly raised concerns, and large topics, it may be useful to consider central commissioning of very thorough and detailed PPIE projects using agreed methodologies on challenging recurring questions in this space. This will help ensure everyone is able to benefit from the knowledge generated, and that the topics are given the level of attention, space, and consideration required.

Create an NHS Analyst Service modelled on GES, GSS, GORS The system must capitalise on the dispersed talent throughout the NHS and let inspiring individuals lead their colleagues. The Government Economic Service, Government Statistical Service, Government Operational Research Service and Government Social Research Service provide an appropriate model for a new body. These professions each have a head of profession, clear career paths and progression opportunities supported by continuing professional development. They hold their staff to high standards by setting out clear best practice guidance, offer accreditation, and set out a clear code of conduct. This service should be responsible for delivering most or all of the following tasks, set out in recommendations NHSA 2 - NHSA 9.

Create clear job descriptions for NHS analysts at a range of levels In collaboration with NHS analysts, Association of Professional Health Analysts, NHS R Community, Royal Statistical Society and the Cabinet Office Central Digital and Data Office, this proposed NHS Analyst Service should create clear job descriptions for NHS analysts from entry level to head of profession. These should be used nationally to clarify roles, recruit staff, and help senior managers (who likely lack technical skills) to identify, appoint, and train data analysts. The job descriptions should be underpinned by a clear competency framework outlining the specific technical skills required to complete specific technical tasks. Such tasks may include: data communication, data analysis, data management, statistical modelling, risk prediction or service evaluation. The government functional standard provides examples of high- level descriptions including analyst, analytical assurer, analysis commissioner, senior officer accountable for analysis in an organisation. The Digital, Data and Technology Profession Capability Framework provides a good foundation to build on, with detailed and specific examples for different levels of data analyst, data engineer, data scientist and performance analyst. Analysts need clear career paths to seniority that allow them to become senior highly skilled analysts rather than generalist managers.

Revise Agenda for Change, and ensure technical staff are paid realistic salaries NHS analysts, software developers, engineers, and other technical staff should no longer be classified as “administrative / clerical” staff. Technical roles require their own category within the Agenda for Change pay scale framework, with their own job titles, capabilities, competencies, KPIs (Key Performance Indicators), and competitive remuneration packages. The NHS must stop expecting to pay highly skilled technical staff in data science and software development on salary scales devised for low and intermediate level IT technical support. Data scientists and software developers in the commercial sector routinely earn more than their manager, customer, or commissioner: this reflects market value, and is no different to employment of senior clinicians, accountants, lawyers, or other technical specialists by organisations. If barriers are hit when discussing offering higher salaries to senior developers with longstanding experience, the anchor point for negotiations should be NHS clinicians’ salary.

Support an NHS Analyst Community Learning from existing community building and CPD activities, including that conducted by medical school deaneries and NHSx, and in collaboration with key organisations such as APHA, NHS-R community, Royal Statistical Society, ensure NHS analysts have access to a range of community building activities, including regional and / or organisational CPD groups, such as the RAP meetups run by the Government Statistical Service.

Develop an annual data conference for NHS service analysts This should be a high-status event with training, presentations, awards, possibly as part of NHS Expo, giving NHS analysts an opportunity to come together, learn, share, and celebrate examples of excellence, create a community, and raise the status of data analysis across the health and care system. The conference should be held during work hours and should be free to attend.

Find good staff, and empower them quickly with “Data Pioneer” fellowships The system has a challenge: to rapidly foster the development of complex new behaviours and teams. It is hard to meet this challenge through central edicts; during the review it has become clear that there are many pioneers throughout the system who are already exhibiting the desired working practices, often without recognition or support. In other parts of medicine, the system uses “fellowships” to give independence, status and influence to clinicians with specific desired skills in analysis, teaching, or research. This is a powerful opportunity to find people with strong existing analytic skills, using RAP or open methods, and rapidly empower them in practical terms. This should be an open competitive programme where applicants can seek resource to cover half of their salary for 3 to 5 years so that they can spend half of their time spreading and developing their working methods, teaching, developing teaching materials, or receiving analysts for supervision and mentorship on placements.

Identify three “Data Pioneer” analytics teams in ICSs and Trusts As per current policy, the future structure of regional analytics and service management will revolve around Integrated Care Systems (ICSs), each covering a population of approximately 1-4 million patients, and analysts in NHS Trusts. To demonstrate the power of modern open methods in NHS service analytics, the NHS Transformation Directorate should identify three Integrated Care Systems and/or hospital trusts where there are strong existing skills in analytics, informatics, and/or software engineering to act as Data Pioneer teams. 2-4 individuals from each group should be provided with advanced training in modern, open, computational and collaborative working methods, including RAP, with the rest of the team given training in the foundations so that they can learn from ‘doing’ under the direction of the group leaders with advanced training. These Data Pioneer teams can lead by example, providing open documentation of their work for others to learn from, make the methods and code local service analytics more visible to the wider community, and feed into the wider programme of modernisation around the NHS analyst service. It may be useful to choose teams and individuals who are close to working with raw NHS records data, as they will have substantial internal knowledge around data management that will be widely applicable.

Commission intermittent code and analysis audits of organisations and analyst teams for service improvement In collaboration with academics, and key organisations such as AphA and the NHS-R community, the proposed NHS Analyst Service or NHS Analyst Head of Profession should commission regular code audits of all organisations that have received public funding for health data research or analysis, including funding for the development of intermediate knowledge objects (such as re-usable code, documentation, or functions). These audits should follow a set methodology; be published openly; and be used for the explicit purpose of improving performance, rather than penalising poor performance. Specific criteria should be developed in collaboration with the community but include: delivery and use of open code; open methods; open data where possible; sharing insights; support for CPD in work time; whether staff meet JDs with training, CPD or other proof. Good performance should be further incentivised, by highlighting best practice examples.

Create an Analytical Capability Index This should be developed independently, and used nationally, to track whether individual organisations have room to improve and signal to leadership where gaps lie in their organisation, how they compare to peers, who they can learn from. Careful consideration should be given as to how best to present the results, and whether this should be made public or not. It is important that the results are only used to drive genuine improvements, and not used for arbitrary contextless performance management. Training There is a need for a strategic and structured approach to training of NHS service analysts.

Create an Open College for NHS Service Analysts This brand will emphasise that the training is open to all interested parties, and that the analytic methods promoted are themselves modern, open approaches to data science. This Open College should contain the following activities, set out in NHSA 11 - NHSA 21.

Devise the content of a national training programme for NHS analysts: initial and CPD Clear job descriptions and pathways must be tied to training and, where appropriate and non-onerous, proof of competencies. Health data is complex, as are health services: working as an analyst in this setting requires a range of specific knowledge around practical health data analytics, alongside more general technical skills in data management, analysis, and visualisation. The NHS Analyst Service should be tasked with devising a curriculum and training requirements for the key competencies associated with job roles, with clear recognition of existing experience or training in and outside of health, and so on. This should be facilitative rather than restrictive, and be focused on informing high quality training, rather than imposing onerous requirements to gather paperwork as proof of skills.

Oversee funding and delivery of training, both open online and one-to-one Having identified the training required, this must be delivered and resourced. Training pathways must be more than an ad hoc list of standalone links to existing online resources. Training should be an appropriate blend of openly accessible online training, such as MOOCs, accompanied by formal one-to-one or group work to support feedback, supervised practical work, and evaluations, in the situations and skillsets where this more expensive in-person training can be shown to deliver better outcomes than open online work alone. Both MOOCs and in- person training should be resourced through a framework where providers can compete to receive funding and offer training as in other sectors. This should include a range of activities at a range of levels including: core training through new post-graduate certificates, diplomas, and degrees in applied practical analytics for health and social care; and Continuing Professional Development (CPD) opportunities for in-career analysts, consisting of refresher courses, opportunities to learn new skills, and so on. CPD courses should award completion certificates, proof of CPD, recognised or even required by managers, and these should be matched where relevant to competencies in analyst job descriptions. This should be overseen by a governing body and developed in close partnership with AphA, RSS (Royal Statistical Society), and the NHS-R Community.

Establish new core training for analysts Replace the Health Education England Graduate Management Training Scheme in health informatics and health analysis specialisms with a specific graduate training scheme in health data analysis that should include: core training; rotation in different parts of the NHS (for example, in primary care vs. secondary care); the opportunity to specialise (for example,, in data engineering vs. data management vs. data analysis); and specific training in the use of modern open computational methods. This will require funding and coordination from national Arms Length Bodies (ALBs), local NHS organisations, national funders, NHS Leadership Academy, HEE, academic organisations (where they can demonstrate a specific commitment to practical NHS service analytics) and more.

Outline clear, non-onerous CPD training requirements for analysts Once CPD opportunities have been made available, it should be a requirement that all analysts gather CPD points. Progress should be evaluated annually and tied to progression opportunities to ensure that participating is appropriately incentivised. This process should be as light-touch as possible, for example, confirmation of attendance at a conference or completion of an online module. Expectations regarding the amount of CPD points required, and the type of activity that ‘counts’ should be adjusted accordingly to job description and level of seniority. This training cannot be delivered by simply linking out to generic data science training resources from other suppliers covering work in other sectors outside of health, albeit that these may well be fertile starting points for modification into bespoke training on RAP and computational methods for NHS data.

Embrace RAP and modern, open working methods Excel has its place, and training will be required at a range of levels for a range of skills. However, there is a clear need to move away from inappropriate use of inefficient and outdated “point and click” methods for analysis, and towards a model based on Reproducible Analytical Pipelines with modern, open, collaborative approaches to data science. Intermediate, and advanced analyst training should focus on enabling the workforce to develop skills in modern, open, collaborative computational data science with an emphasis on reproducible analytic pipelines covering concepts and skills such as R, version control, GitHub, Jupyter notebooks, Pandas, and similar. This does not mean that everyone in the system must become an expert software developer: but it does require some changes in skillsets and emphasis. RAP has a proven track record in other parts of government and in Public Health Scotland, with a strong model for spreading change in organisations. These will be new skills for many and so training will entail more than links to external generic data science guides. Training should be practical and include completion of tasks inside sandboxes so that mistakes can be made safely. The training provided by the ONS Data Science Campus provides an excellent example, and training should be developed in close collaboration with the RAP teams. The chapter on Open Working discusses these issues in more detail.

Ensure training focuses on RAP as much as Machine Learning There is a tendency for training to be diverted into more exotic forms of data analysis such as Artificial Intelligence or Machine Learning. These have their place, and there are many existing resources that meet these training needs very well outside of health analytics for those who have already developed outstanding skills in data science. However, the key unmet training need in the service is RAP, and the delivery of analytics using modern, open methods to achieve improvements in efficiency, sharing, quality, transparency, documentation, and reproducibility. This must be the priority for any training programme.

Create a technical team to house and develop continuing professional development resources Training in technical skills needs to be delivered by those with technical skills in data science as applied to NHS data. It cannot be delivered by generalist data scientists alone. Training also needs to be kept up to date. The aim should always be to provide training in the most advanced computational data science skills; what these are will change over time. Providing a team of technical specialists with adequate funding to develop, deliver, share, and curate training, including the development of tools such as sandboxes where analysts in training can practice their coding and analysis against dummy data that reflects real NHS data, will be essential if training is to be high-quality and up to date.

Ensure all training is open by default The traditional funding model for training from universities and many other providers is to charge per-attender. Wherever online training resources are commissioned they should be open by default, with all video lectures, training materials, written content, exercises, and code shared openly. This may result in a somewhat higher unit cost for teaching resources procured on a buy-out basis but will represent a better investment in the medium term. It is necessary to remove access barriers to knowledge and training, in a space that urgently requires up-skilling, and to avoid imposing a requirement on analytic staff to ask permission of generalist managers, who may lack technical skills themselves, for access to training budgets that require onerous engagement with bureaucracy. This is particularly important in a complex ecosystem such as the NHS where behaviours, expectations, resourcing, management styles and training availability may vary widely between local and national NHS organisations. Fully open access to all NHS analyst training resources will also create substantial network benefits. It will make these training resources accessible to NHS staff in adjacent specialties who wish to up-skill, including managers and clinicians, enabling them to drive forward better use of data in their own teams and organisations; and to outside elements from the public and private sector making it clearer to them how the NHS uses data to improve care, and how they can interact to offer help and support, or improve analytic work with better tools, algorithms, services, or insights.

Create and maintain a curated national open library of NHS Analyst Code Hire a team of 10 people to create an open library of code and workbooks for key recurring tasks, examples of best practice, ‘how-to guides’, code for common analytical queries, codelists, variables, and so on. It must be unashamedly technical but meet the needs of staff with a range of abilities. The library should be presented as a flexible open online website, with clear tagging, careful thought around discoverability of resources, and careful curation of individual resources into “training arcs.” The library delivery team should be led by an editor experienced in producing good open online technical resources; it should include analysts but also include expertise in technical writing, knowledge management, online education, and publishing. An MVP (Minimum Viable Product) should be created within 6 months by pulling together the best existing resources from national and local teams in close collaboration with all key stakeholders and teams already listed above. This library should be closely tied to (and feeding into) online teaching and CPD. CPD points should be provided for contributing to the library and there should be an obligation for any analyst developing code with public resources to contribute the outputs to the library for re-use. The need for better knowledge management around code and methods for NHS data analysis is also discussed in the sections on Open Methods and Data Curation.

Create training specifically for senior leaders to help them become better customers for data analysis There should be an expectation that non-analyst staff, especially those managing analysts, have sufficient data literacy to conduct informed conversations about data. This will require that non-analyst managerial staff and clinicians have access to training to help them make better use of data in their day-to-day jobs; enable them to make smarter decisions about how to use data for performance management; enable them to ask better questions of their analysts; and to provide them with the skills they need to distinguish between high-quality and poor-quality analysis. Undergraduate and postgraduate training for clinicians and managers should include knowledge of how data is captured and analysed to improve care. Funders and employers should resource collaborative teaching and training between analysts and clinicians/managers. This training will bring the management of NHS service analysts into alignment with the Government functional standard for analysis. It should ensure that non-analyst staff are, at a minimum, able to confidently evaluate: whether commissioned analysis is compliant and appropriate for intended use; the risks, limitations, and major assumptions of particular analytical methods; whether the output is appropriate for the analytical need.

Commission a rapid review of medical school curricula and similar The content of the curriculum in medical schools, allied health professional training, and clinical post-graduate training is always hotly contested, with a wide range of competing communities advocating for their speciality to receive more prominence. Nonetheless there are good grounds to believe that clinical informatics and data science for service improvement are under­ represented. A rapid review of current curriculum content in all medical schools and a range of other clinical training will identify where there is need for augmentation and should aim to recommend how training in essential technical skills can be incorporated without compromising other aspects of the curriculum. This can be done by Health Education England. Platforms and data access Analysts need access to data in platforms that support modern open working.

Improve the provision of data analysis environments In the sections on Trusted Research Environments and Information Governance there are a range of detailed recommendations to ensure that local service analysts have access to data in environments that support modern, open approaches to computational data science, and to ensure that data is not unreasonably withheld.

Revise NHS IT policy for analysts to ensure it is fit for purpose Analysts need to be able to use modern computational data science tools such as python, GitHub and docker on their NHS computers. Current IT policies often block the use of such tools. A similar challenge has been faced and recently overcome by the analytic community in government outside of health. This must be addressed in national and local IT policies with clear statements on assurance and risk from the NHS Transformatin Directorate to local decision makers, to make it the norm for work to be delivered using modern computational data science tools and avoid the apparently prevalent problem of analysts using these tools outside of the formal permissions and policies of their workplace. Many but not all of these challenges will be met by delivering better national and local infrastructure for data access; however, there will likely still be a role for individual local machines that facilitate the use of standard modern tools.

Rationalise national audits, RightCare, GIRFT, and Model Health System As described above these projects are presently implemented as “full service” arrangements where all data collection, extraction, management, and analysis is done inside one organisation or team in order to produce an intermittent single output such as an annual report. These teams have deep knowledge around their datasets, and the clinical context. However the current working style risks duplication of effort (and risk) around data extraction, data hosting, and data management; blocks sharing of detailed technical knowledge and code around methods for data analysis and outlier detection; does not use the most efficient methods to produce a commons of knowledge; and risks creating or reinforcing monopolies around access to data and knowledge that can block innovation and high quality analytics for patient care. A better model would be for all these services to operate in common analytic environments; where all have access under reasonable constraints; and where all code is shared alongside documentation. This is best delivered through identification of a small number of Data Pioneers among these national audit projects, who are ready to embrace RAP working methods and work in a national TRE or similar to deliver their outputs. This is discussed in more detail in the chapter on TREs.

Make change practical The NHS should identify three Data Pioneer ICSs that can move to a full TRE and RAP working style within 6 months; and three Data Pioneer national quality improvement audits (at least one within NHS England) that can move to full TRE and RAP working within 6 months. External collaborations External collaborations with the private and public sector are valuable but should be handled thoughtfully with an emphasis on open delivery.

Commission and promote best practice on outsourcing analytics It is reasonable for NHS organisations to sometimes seek help from external commercial and public sector organisations to improve the productivity or scope of their analytic outputs, especially in a period of transition while the NHS analyst profession is being developed. However as discussed above this work brings risks around quality, transparency, and development of wider open knowledge on NHS data, whether the external partners are commercial or academic. The NHS Transformation Directorate should coordinate the development of Best Practice guidance on outsourcing to cover the range of scenarios where such external collaborations are and are not beneficial to the system, boilerplate contractual requirements, and red flags around working methods and delivery.

Require all outsourced or external work to comply with RAP and open working methods Currently when analytic projects are outsourced to consultancies, academic collaborators or other agencies it is common for only the results to be reported, for example in a PDF or slide deck, without the accompanying methodology or code used to conduct the analysis. This prevents the NHS from error-checking the work, learning from it, or being able to replicate it internally, whether in the organisation that originally commissioned the work or elsewhere in the system. This creates duplication of work, and the loss of knowledge that can create efficient analyses and drive innovation. This cannot solely be addressed by asking external partners for “training” or more detailed narrative descriptions of the methods used. As discussed in the chapter on Open Methods, all NHS data management and analysis code should be accompanied by adequate technical documentation alongside the code, as required by the minimum standards of RAP, openly available for re-use and external scrutiny. All outsourced work should adhere to this requirement.

Support NHS/academic collaborations on RAP data science for NHS service improvement UKRI/NIHR should consider running an open funding call specifically for academic teams to collaborate with national or ICS NHS data analysis teams on using RAP and modern open data science techniques to improve the quality of NHS care, to deliver specific outputs, and to build mutual relationships and capacity building around applied analytics. The targeted outputs should be a range of Jupyter notebooks or similar with well-documented open code describing – with appropriate technical documentation – how NHS data was prepared, analysed, and used to identify or address opportunities to improve NHS clinical activity or outcomes.

Create a RAP and Open Code Oversight Group There is clear evidence of inertia on this issue, and coordinated activity across a range of organisations including funders is required to deliver change. The NHS Transformation Directorate should convene a small group to ensure change happens by commissioning, monitoring, promoting or delivering the initiatives below as appropriate.

Create a public policy setting out expectations on open code Create a public policy setting out expectations on open code for public organisations and individuals to support, with broad brush expectations, links to further technical information, commitments which organisations can be held to, and a clear statement on the limits of open code and the compatibility of open code and commercial models.

Make open code a boilerplate feature of all public contracts Open code sharing should be a required feature of all standard contracts between the NHS and any external provider of code for health data management and analysis; with a similar arrangement for academic funders; and for any university or other body sub-contracting such work.

Create an “exceptions framework” whereby publicly funded code can be closed by prior arrangement if this meets NHS and UKplc strategic objectives Individual researchers, organisations or funders should be able to actively apply for a single project to be closed, under a carefully designed “exceptions” framework, where each exceptional request is evaluated to determine whether this exception meets reasonable national, individual or organisational interests around commercialisation and collaboration, and whether the network damage inflicted by a closed license is justified by another greater public benefit. This expert group should include intellectual property specialists, software developers, researchers, key stakeholders with expertise (such as the Open Data Institute), policy experts and funders.

Create an Open Code Ombudsman and Assistance Unit It is possible, particularly whilst the NHS analytics workforce is in a transition period, that there will be occasions when an analyst or team of analysts in one NHS organisation needs access to code that has not yet been made open by a different NHS organisation - potentially including central Government organisations. There is a need for an independent body that is tasked with dealing with these, and other, disputes. This unit should listen to complaints, and feedback common themes to the relevant policy, funding and commissioning teams so that appropriate guidance can be developed. To make this practical, the unit must have appropriate status and power over even the largest NHS organisations.

Assert that publicly funded code is publicly owned: cautiously consider “Crown Copyright for code” At present decisions about sharing and licensing code are made, at small scale, in huge numbers, across the health and research landscape, often by people who do not understand the impact and implications of these choices for themselves and/or the wider community of data users. This has very substantially blocked code sharing, which should be the norm, and is the norm in many adjacent research specialities. An expert group should be convened to formally consider a new national standard: that public funded code is publicly owned, under a formal license that covers all code produced on public funds; with an expectation that all publicly funded code should be shared under the MIT open license; and exceptions to be decided by prior arrangement with a prespecified ruleset.

Data Controllers should require RAP and open code sharing from data users Data controllers and especially the NHS should require all those accessing patients’ data to share all analysis code, or openly argue for exceptions in single cases. Where the system is in a period of transition, this should be strongly considered in all data access requests, and where there is no plan to share code immediately there should be a credible plan to ensure that this is done later. As with other mandates around open code this should have a clear pre-specified exceptions framework.

Amend the Code of Practice for the Research Powers of the Digital Economy Act The Department of Health and Social Care and the UK Health Security Agency should work with the Department of Digital, Culture, Media and Sport to make a minor amendment to the Code of Practice for the Research Powers of the Digital Economy Act (the primary legal gateway for accessing non-health data) requiring data analysts to make their code available, with a robust exceptions framework as discussed elsewhere.

Make it “Okay to Ask” about access to publicly funded code The team heard from several interviewees that at present it is commonly regarded as provocative to ask for access to the code used to implement an analysis, especially in some parts of the academic community, despite general positive statements on open working. Culture change will only be possible if it is deemed socially acceptable to question when deviations from the ‘new norm’ of open working are identified. This should be made clear in all relevant policies, and codes of conduct, across academic and NHS organisations. Develop Guidance on Open Code from Specific Key Organisations

Health and Care Information Governance Panel guidance should facilitate open code It is important that all NHS organisations are given clear direction that code sharing is not the same as data sharing and that it is entirely possible to share code routinely and safely without the organisation incurring significant costs or reputation risks. To make this clear, the Health and Care Information Governance Panel should create guidance on the importance (and permissibility) of code sharing, to go on the Information Governance Portal, emphasising that transparency is a crucial means to build public trust and clinical safety.

The Information Commissioner’s Office should produce guidance to facilitate code sharing The Information Commissioner’s Office (ICO) is currently working to produce new and updated guidance on anonymisation. Alongside this, the ICO should also produce guidance regarding code sharing. This should make clear that sharing code is not a disclosure risk, and that those writing code have a clear responsibility to ensure that their analytic code is not disclosive of any personal information. Ideally this guidance should also make clear that code sharing and the practices associated with Reproducible Analytical Pipelines are an important aspect of good citizenship around data usage. There is also a need for better guidance and training on ensuring that analytic code is not disclosive of any personal information.

The Medicines and Healthcare products Regulatory Agency should address code sharing and device regulation It is not practical or useful that there should be conflict - or perceived conflict - between code sharing and medical device regulations. The Medicines and Healthcare products Regulatory Agency (MHRA) should deliver absolute clarity to reassure all that where they share code for critical review, re-use and improvement or modification that they do not have liabilities around how it is re-used by others, especially where they have given a clear statement that their code is not intended to be used as a medical device, and especially where the code is shared alongside an analytic or research paper output. Without this clarity, researchers are likely to be more cautious than is necessary. More broadly it is clear that there is a general lack of clarity around medical device regulation and code, and in particular a lack of clarity on remit and what constitutes a medical device, which is likely to lead to some code being both inappropriately included and excluded from the definition, potentially exposing patients to harm. The MHRA is working to provide this clarity, add strong support, and encourage the organisation to prioritise this work. There is an important role for regulation in this space, and some organisations sharing codelists that they have assured and produced explicitly for implementation in NHS service in use as a medical device (such as in a popup trigger) would benefit from being able to wear that assurance and regulatory status prominently.

Negotiate co-ownership of claimed commercial innovations from NHS data See Information Governance and Ethics chapter. Support NHS service analysts to work with RAP and open methods NHS analysts are a crucial part of the wider analytic community and can lead by example.

Write an ‘OpenAnalytics Policy for the NHS’ Bring together DHSC, and the NHS Tranformation Directorate to write a policy that makes it clear to all analyst teams across the NHS, and all general managers, that sharing code is not the same as sharing data and that open is the preferred and default method for all analysis conducted using public data and public funding. This policy should set out best practice for using open working methods, for openly sharing code and for writing documentation. It should also cover more complex areas such as licensing and the protection of IP where applicable. It should be kept under regular review to ensure it remains up-to-date and should signpost to further sources of help and advice where necessary. All external procurement of data science services, whether data management or analysis, should require that all code and codelists produced to deliver the work are shared openly by default. Exceptions to this should be rare and explicitly pre-arranged, with clear justification under pre­ specified criteria set by the NHS Transformation Directorate and DHSC.

Make Open a Standard Contractual Requirement Government departments should require all those conducting data analysis on their behalf, in-house and as contractors, to share all code, consistent with RAP and the computational methods above, with adequate technical documentation as per RAP criteria. To aid with this Intellectual Property assignment and publication requirements should be laid out in template and framework contracts so that all organisations commissioning or contributing analysis and code to the public sector are held to the same standards. These contractual requirements should be developed in collaboration with members of the research software engineering community to ensure that they are fit for purpose. It should also be borne in mind that all such requirements, policies, and guidance documents are likely to be ‘living’ and regularly iterated best on the evolving nature of best practice in this field.

Commission intermittent open code audits to drive improvement In collaboration with academics and key organisations such as AphA and the NHS-R community the NHS Transformation Directorate should commission regular code audits of all organisations that have received public funding for health data research or analysis, including funding for the development of intermediate knowledge objects (for example datasets, or TREs). These audits should evaluate adherence to RAP and open computational methods; follow a set methodology; be published openly; and be used for the explicit purpose of improving performance on code sharing, rather than penalising poor performance. Specific criteria should be developed in collaboration with the community but include: all code shared on GitHub or similar; adequate technical documentation embedded within code as per RAP criteria; use of version control and appropriate methods; sharing of non-disclosive open data where possible; support for continuing professional development in work time; whether staff meet job descriptions with training, continuing professional development or other proof. These elements should be split out into overarching themes to inform targeted interventions for improvement. Good performance should be further incentivised, by highlighting best practice examples. The intention should be to identify blockers to sharing and opportunities generated by sharing - specifically high performers should be asked how they adopted open working methods and the benefits they have seen and poor performers should be asked what help they need in order to ‘go open’ and then provided with the relevant assistance.

Establish a technical writing and documentation team for the NHS Too often the completion of routine local or national NHS analysis tasks, or the reproduction of key technical platforms in separate locations, relies on an “oral tradition” with documentation passed in conversation or email. This is unsustainable, introduces risk and inefficiencies, and impractical in a massively federated system. Technical documentation improves reproducibility and sustainability but writing good documentation is a skill in and of itself. Hiring a central team to train others and write documentation for key technical platforms & tools, including TREs, python libraries, and more, used across the NHS, would greatly facilitate collaboration, and reproducibility. Build workforce capacity for RAP and modern, open, collaborative working The system as a whole is held back by a striking shortage of individuals with crossover capability, covering both technical skills (around data science and software development) and domain knowledge (around epidemiology, NHS service analytics, health services research, and the broader NHS and clinical context of technical and analytic work). This will require training for staff at a range of levels, and for a range of personnel. In outline three key programmes are required: software skills for analysts; NHS domain knowledge for developers; and brief training for senior generalist leaders.

Create a “Code For Health” training programme for NHS service analysts and academic researchers This should include advanced computational data science covering RAP, software carpentry, version control, functions, code documentation, and similar. This should be at a range of levels including MOOCs, short courses, and long courses, with strong practical elements. The following principles and working practices are strongly recommended: Combine NHS and Academia Although historically NHS service analysts and academic analysts are considered separately, this is a strong strategic opportunity to begin building robust technical bridges between the two: both work on similar data, often with similar methods or tools; and both should ideally work in similar data analysis environments, as discussed in the Trusted and Shared Research Environments chapter. MOOCs and Practical Work Traditional teaching and training has a role. However, for scope and access, a priority should be placed on delivering training as a combination of Massive Open Online Courses (MOOCs) and in- person teaching, both developed specifically for this purpose. MOOCs can cover factual content, and with practical code development exercises can be procured outright: they should be made openly available to all. In-person supervision is necessary for a subset of attenders, especially those seeking certification, to supervise practical work, and for marking work to evaluate competences. This will necessitate a per- person fee, which will create a revenue stream for providers, but necessitate a system for unit payment from the NHS which may obstruct NHS analyst attenders given current lower status of this group in some organisations (see NHS Analysts Service chapter): thought should be given to block purchase or training budgets as seen in other parts of the NHS. Build on Prior Work but Maintain Focus on RAP This training should build appropriately on various existing resources and expertise including: the work by existing RAP teams; the work by the NHS-R Community; and other work specifically on RAP and computational methods. It should focus specifically on RAP and computational data science techniques as directly applied to working with NHS data. Caution should be taken that resource is not diverted into training on other issues such as general research methods, specific statistical methods (except as specifically embodied of RAP training), or newer methods such as Machine Learning which are useful but different subjects. Similarly this training cannot be delivered by universities rebadging existing training on other topics such as bioinformatics; or by simply linking out to generic data science training resources from other suppliers; albeit that these may be fertile starting points for modification into bespoke training on RAP and computational methods in NHS data. Open Competitive Procurement Training should be procured by an open competitive process, amenable to the best of either public or private providers. All training can be commissioned by either UKRI, the NHS, or both. A rapid technical review should be conducted of recent very welcome UKRI investment in this space to evaluate whether it addresses RAP and computational methods as above, and whether outputs are open. There is currently a rich diverse ecosystem of training in many other aspects of analytics, with no concern about overlap between offers, and similarly a diverse range of providers with different focuses should be acceptable here. An emphasis should be placed on finding providers with strong previous track record of using RAP or open computational methods.

Create a “Data for NHS Leaders” training programme For senior leaders and those in adjacent skill groups training should be accessible on the basics of data analysis, RAP and computational methods so that this group can understand the principles and purpose of the work done by those in their team. This should be MOOCs and short courses. Some of the excellent recent work at the Number 10 Data Science Unit (10ds) “data for leaders” programme may be adaptable.

Create an “NHS Data for Developers and Data Scientists” training programme Very senior leaders in the system repeatedly expressed to us the view that individuals with deep developer skills and NHS domain knowledge are extremely hard to recruit, if not impossible. Strong software development and data science skills are developed over many years and often require a specific disposition: it is not realistic to expect that current NHS analysts (with a deep but very different knowledge base) can be trained in such skills to the standard required by the NHS and wider community. New knowledge for generalist Software Developers and Data Scientists should be addressed with health data “Transfer Training” so that individuals with strong technical skills can develop deep NHS domain knowledge. This training should include the basics of: non-communicable and communicable disease epidemiology, focusing on the specific methodological issues that arise when analysing health data; NHS service analytics; the operation of the NHS; the clinical context; how NHS systems collect and store information; and the structures, strengths and weaknesses of NHS EHR data. This training can likely be adapted from existing teaching and training on these topics. Priority should initially be given on these courses to developers and Research Software Engineers already working on health data. Attention should be paid to resourcing: the NHS and research sector is actively trying to recruit in and train skilled individuals, from some of the most competitive global jobs markets; this may necessitate paying developers for their training time, and the course fees; it is reasonable to adopt a similar approach to for example MBA funding in the Civil Service, where fees and time are paid, with a contractual obligation for subsequent public service. The developer and data science community are used to working with well-documented and open code; therefore insofar as this can be rapidly replicated in the health data space, onboarding times will be greatly attenuated. Fixing this problem will take a short period of years but pay huge dividends for the NHS and the wider ecosystem of innovators. Make code a central feature of work in universities using health data Universities, research funders, and associated governing bodies need to recognise that much of health data science now involves academic researchers effectively writing software, and that this is deeply enmeshed with the work of iteratively developing, implementing and evaluating new research methods. University staff need to be able to access training in these skills and be given protected time and resources to make the most of training opportunities. Those already demonstrating best practice in this domain should be recognised and rewarded appropriately. The following specific interventions are recommended.

Modify the Research Excellence Framework (REF) to reflect computational work REF is used to assess research quality at UK higher education institutions and is known to drive university strategy and promotions for staff. Generally evidence of quality is provided in academic papers, with other sources admissible, though principally through other elements of REF such as Impact Case Studies (which are fewer in number, and therefore tend to be very large suites of work). It is theoretically possible to return outputs such as well-curated GitHub repositories, but in practice this is not often done in work using health data. This can be addressed in broad terms (making RSE and code a required or recommended form of proof of a strong research environment in REF guidance); but also with concrete quantitative requirements. For example, given that almost every quantitative research paper will entail the production of analytic code, there should be a requirement that a percentage of quantitative papers are accompanied by a link to an openly accessible GitHub repository or similar open resource containing the code.

Embrace Research Software Engineering (RSE) in health data work The RSE movement has had high impact in other sectors, and warrants strong support across all sectors; in particular it should be strongly supported to expand into work using NHS data.

Pay realistic salaries to software developers Software developers are among the highest paid staff globally, but university pay scales typically do not recognise the skillsets required to be a research software engineer, and often attempt to hire engineers and developers on pay-grades similar to those of IT support staff. This makes it hard to recruit people with outstanding software skills and serves to further undermine the value of software development, data management, data curation, and code development. It is commonplace for universities to pay those with technical skills, such as clinicians or accountants, something closer to their realistic market salaries. Universities should develop pay scales for developers in the same way that they have done for clinical academics, recognising the specialist skills and outside options of those they are seeing to recruit.

Create a working group to develop an attribution model for code and data Academic behaviour is inevitably driven to a greater or lesser extent by crude metrics. There has been extensive discussion about recognising contribution to code and datasets, and attributing credit where work is re-used. None of this has delivered concrete change. Examples of options include: Direct Object Identifiers (DOIs) for code and data; recognising citation of these objects in the h-index and other commonly used metrics; supporting publication fees for publishing code and data, or links thereto, as placemarkers in conventional academic journals for citation and recognition. A Working Group should be established with support from UKRI to review prior work and set out options for an implementable set of detailed proposals within 12 months in an openly published document for consultation. This working group should consist of policymakers, senior and junior RSE experts, senior and junior researchers, representatives from academic publishers, representatives from funders, and representatives from organisations producing data for re-use such as the prospective longitudinal research cohorts. This group should produce an implementable model within 24 months and monitor implementation, in order to create and maintain incentives and recognition for those sharing access to data and code.

Clarify the need for authorship for software developers and data scientists as equal core contributors There is currently a wide range of norms around whether those making deep contributions to the software elements of a research project warrant authorship. Many of the current working practices are also inconsistent: “statistical programmers” are included by many groups; but not if there is a very large number of them; or if they contribute to a wide range of outputs; and so on. Current norms around authorship also tend towards regressive attribution, being more likely to include senior authors than junior contributors. Overall, having discussed this issue extensively, it is clear that there should be a presumption to include software developers and data scientists who have contributed to the delivery of the paper in authorship, not least because this work commonly entails a wide range of creative input to deliver the work informed by deep technical knowledge of the analysis, but also in very many cases the clinical domain, and the statistical context, and similar issues. Where this is argued to conflict with other current documentation on principles such as guidance from the International Committee of Medical Journal Editors - which is itself commonly breached across a range of other topics - this should be robustly addressed and discussed. Ownership of this task is complex, reflecting slow progress on this issue: initially this should be managed by the same group as above, or a prominent organisation from the RSE community.

Proactively address sharing during the pandemic Academic researchers have played an essential role in the response to the global COVID-19 pandemic. Yet, despite the global nature of the emergency, not all research, code, and other outputs have been made openly available for others to re-use or learn from. In some instances there may be good reasons for this, but in others it may simply be that the barriers to sharing have proved too high for some research groups - for example, paying for GitHub or for open- access publication. University administration teams and innovation offices should discuss with researchers providing research outputs on COVID-19 whether they can share code, methods, documentation, libraries, or more, for recent and future outputs.

Academic journals should be encouraged to make code- sharing a requirements In recent years, academic journals have begun to embrace the idea of open knowledge, introducing options for open-access publication, allowing for pre-printing and self-archiving, changing the copyright for articles, and - in some instances - introducing data sharing requirements. Less progress has been made in the domain of code sharing. This can be detrimental, as even reviewers sometimes do not have access to analytic code to check the work when making decisions about which articles to publish. Journals should begin consultations with the wider academic community about the development of code-sharing policies.

Embrace Research Software Engineering with three Data Pioneer groups leading by example The RSE model is strong and proven, but under­ used in the field of health data. RSE groups can work collaboratively with teams to help them adopt RAP methods with hands on assistance and with training. They can help with larger code projects to identify the right level at which to “abstract” the task into re-usable functions. RSE groups developing more resource in universities will mean that they naturally develop more influence over the culture of how research is done, and rewarded. This is best achieved by centrally resourcing three RSE groups via UKRI focusing on health in three universities, prioritising teams where there is established capability to augment. This group of three Health Data RSE groups should meet regularly to coordinate advice to UKRI, the NHS, and the university sector more broadly, on how to expand software capability and productivity in the broader health data community.

Use research funding to drive modernisation around better use of code and data Research funders have a unique ability to shape the behaviours, priorities and delivery of the research community. The following recommendations will help them to drive positive change around adoption of RAP and modern, open computational approaches to data science in healthcare.

Make code sharing a core feature of all standard funding contacts Academic funders including NIHR and UKRI must facilitate and require open code as standard practice. There will be some circumstances where making this mandatory will be inappropriate, so all contractual obligations must include an exceptions process, whereby applicants can openly argue for exceptions in single cases, according to a pre-specified set of criteria, as discussed elsewhere in this Review. Exceptions should be agreed before work is funded. Applicants for funding for quantitative research projects should be required to state how they will share code the project, as some funders require applicants to do with data and dissemination.

Provide guidance and training on RAP and code sharing The training above covers broader issues around RAP and code sharing. Some researchers currently funded by for example NIHR and UKRI may be willing to share code but require bespoke training and support to do so. To help these individuals with this transition, funders should provide guidance and funding to attend training on ‘good enough’ code sharing, alongside guidance and examples of ‘perfect world’ code sharing. This should include supporting existing work, such as the Turing Way, RAP, and other similar projects.

Fund a fellowship programme around Code for Health Data There is currently a need for more teams and individuals who combine skills in computational methods, alongside domain knowledge around epidemiology, NHS analytics, and EHR data. This can be addressed by applying the normal mechanisms for capacity building, especially fellowships which bring independent status within the university sector for individuals and a field; support career progression; and support individuals to make their own choices about where they can best contribute and innovate collaboratively alongside pure research colleagues, to get away from a paradigm of developers being seen as “instructed by” researchers. UKRI and/or NIHR should fund the following: Fellowships for software developers in health data UKRI in particular already have some fellowships for developers and engineers; this should be energetically expanded into health, with a ring- fenced count of fellowships for this purpose. To maximise the talent pool, it is crucial that access to apply for this funding is open to all, and not limited to applicants from a specific set of academic groups or educational institutions. Entry Fellowships scheme for developers from other sectors To solicit inward movement from other sectors it will be valuable to have a small range of fellowship schemes were new entrants can have their salary covered for the first year of their inward movement and training for domain- specific knowledge such as epidemiology, EHR structure, NHS service analytics, and related activities. Training fellowships in computational methods To help individuals in conventional academic positions develop their computational skills it will be valuable to have a small range of fellowships, ideally attached to a curriculum of training, for those developing such skills. To maximise the talent pool it is crucial that access to apply for these fellowships is open to all, and not limited to applicants from a specific set of academic groups or educational institutions.

Open funding calls for projects and programmes around Code for Health Data. Work related to technical infrastructure, data management, and TREs is not universally regarded as legitimate academic or methodological activity. There are multiple sources of funding for Individuals and teams to address specific single clinical research questions, but almost no open competitive funding for the development of code, infrastructure, or innovative methods in this space. It is crucial that great code and data infrastructure is developed in close collaboration with the delivery of strong single academic research paper outputs. However strong code is not produced when this aspect of the work is left to fend for itself as a junior party to single topic research projects, especially during a period of transition towards more computational approaches. UKRI and/or NIHR should launch an open funding call specifically for open code projects in health data science, and consult closely with the Wellcome Data for Science group on the best means to achieve this. An initial list of example projects is given in a box at the end of the TRE section for initial guidance and illustration only. To maximise the talent pool it is crucial that access to apply for this funding is open to all, and not limited to applicants from a specific set of academic groups or educational institutions. The following is recommended for any funding programme.

Treat “Data Infrastructure” as Open Code Recent work from UKRI increasingly recognises that modern infrastructure is less about computers and buildings, and more about about code and teams. It is crucial that data infrastructure is handled in this way, with delivery of open code at its core, accompanied by detailed technical documentation alongside it, consistent with RAP and the computational approaches outlined above. It is equally crucial to recognise that this approach can and should viewed as modular (with different re-usable elements for different tasks produced by different teams) and methodological (with approaches to specific data management, data analysis, and privacy preservation tasks developed iteratively through innovation and delivery). Most importantly, the system must steer carefully away from procuring infrastructure such as TREs as “black box services” where only comms material and finished academic manuscripts can be seen from the outside; this is covered more in the chapter on Trusted Research Environments. Some initial outline examples of the types of activity that funders might solicit or respond to in the health data space are given in the box at the end of this section.

Use Open Competitive Funding for code projects It is vital to support an open collaborative ecosystem where those with the best ideas and delivery can compete to propose the best approaches to diverse challenges in data management, analysis, analytic platform components, and so on. An approach where one organisation, or a small number of organisations, is viewed as the sole supplier will not lead to excellence. Open competitive funding, where all can propose projects, is a key route to ensuring we identify and resource the best ideas and teams.

Review prior delivery of open code by applicants During a period of transition to new ways of working it is important that those leading by example are enabled to drive change; and prior delivery in this comparatively new space will likely be a strong predictor of future delivery. Particular caution should be used around teams with prior substantial resources for health data research projects that have not delivered, or cannot retrospectively showcase, a commensurate set of documented code.

Ensure experts on code select and oversee code projects Individuals with direct technical expertise in software projects, data infrastructure, RSE, RAP and related work should lead or very strongly guide all funding for projects in this space, just as conventional academics guide awards and oversight for projects delivering single academic paper outputs.

Ensure the objectives and outputs of all investments are open All funding for code projects, especially those around infrastructure, should be openly and publicly disclosed, with a brief description of the amount, the recipient, the expected work and timelines, and a link to the repositories where development and documentation is anticipated to reside.

Ensure funding for code and platforms does not get diverted When providing bespoke funding for developing open code and appropriate technical platforms it is necessary to ensure resource is not diverted to fund single research paper outputs for which there is extensive funding through other routes. This will require oversight at a number of different levels. When applications are being reviewed, funders should make sure that they have relevant technical experts on the panel reviewing the applications - for example, research software engineers and data scientists - evaluating proof of skills such as GitHub profiles and repositories in addition to CVs.

Avoid “regressive funding models” built around short-term bursts of funding There is a concerning tendency for code projects to be resourced with short-term urgent bursts (for the avoidance of doubt, outside of COVID-19) with funding arrangements that require successful applicants in extremis to spend a large amount of money very suddenly, over less than a year, starting with only a few weeks’ notice. This strongly suggests a lack of strategic thinking in the organisations offering funding, and will tend to preferentially resource large incumbents who can rapidly absorb such costs, but who may not have the best prospects of delivering for the wider community. More specifically this last-minute funding strategy actively mitigates against new entrants to the market, field, and creative academic pool, while favouring incumbents; and disadvantages those in more junior roles, or without large existing teams, who may have the strongest ideas, newest skills, and best delivery prospects. The preferred funding approach should operate on 2-5 year cycles to build capacity and open delivery, with the conventional option of 6 months from award to commencement to facilitate inward recruitment of staff.

Focus on sustainability for software projects: set aside a third of resource for this task Funding should work hard to identify teams and projects with broad user bases and impact, then support them to continue their work with 2-5 year funding. This should not entail unrealistic proposals for commercialisation of data management and analysis code used solely for research and health data analytics unless there are clear grounds for a standalone commercial spinout; and it should not necessarily entail extensive commitments to specific new features simply to justify sustaining a project. Iterative improvement and sustained delivery are sufficient goals in themselves, build capacity in the system, and build a broader ecosystem of productive outputs. A third of any budget should be set aside for sustainability. Encourage open working through Trusted Research Environment design and implementation These issues are also discussed in the TRE chapter.

All Trusted Research Environments for NHS data must facilitate and require code sharing All national TREs should be designed such that researchers are able to use services such as GitHub and Gitlab; and ideally require such that code is shared by default. This must be a core part of the core design of any TRE for NHS data, and should be a key feature of any accreditation or recognition criteria for a TRE.

TREs themselves should be built on principles of RAP and open code It is very reasonable for TREs to use general purpose closed commercial products (such as enterprise database tools) where these reflect the best procurement choice. However, any tooling built for the TREs themselves - especially the components for data management, analysis, and task execution - should be built using the principles of RAP and open computational working as above, with deep technical documentation and open code to facilitate access, usability, accountability on delivery, and modular development and augmentations within the environment itself.

Produce clear guidance on disclosure risk and open code If code is not written thoughtfully, and consciously, there are small risks of disclosing small amounts of personal information. Release of open code from TREs should efficiently manage disclosure risks, and TREs should give clear and enforced guidance to users on the requirement for their code to not contain any disclosive information, alongside all their other guidance on secure analysis of sensitive personal data. Writing non-disclosive code should be an absolute requirement for those requesting access to NHS patient data, with a robust and public exceptions framework for specific situations where there is no alternative, with appropriate alternative steps to manage code sharing and privacy. Those accessing NHS patient data should be required to commit to this, and demonstrate understanding of how to achieve it (for example in the training and tests required for access).

Build trust by taking concrete action on privacy and transparency: trust cannot be earned through communications and public engagement alone.

Ensure all NHS data policies actively acknowledge the shortcomings of “pseudonymisation” and “trust” as techniques to manage patient privacy: these outdated techniques cannot scale to support more users (academics, NHS analysts, and innovators) using ever more comprehensive patient data to save lives.

Build a small number of secure analytics platforms - shared “Trusted Research Environments” - then make these the norm for all analysis of NHS patient records data by academics, NHS analysts, and innovators, wherever there is any privacy risk to patients, unless those patients have consented to their data flowing elsewhere. Every new TRE brings a risk of duplicated effort, duplicated information governance, duplicated privacy risks, monopolies on access or task, and obstructive divergence around data curation and similar activity: there should be as few TREs as possible, with a strong culture of openness and re-use around all code and platforms.

Use the enhanced privacy protections of Trusted Research Environments (TREs) to create new, faster access rules and processes for safe users of NHS data; ensure all TREs publish logs of all activity, to build public trust.

Map all current bulk flows of pseudonymised NHS GP data; then shut these down, wherever possible, as soon as TREs for GP data meet all reasonable user needs.

Use TREs - where all analysts work in a standard environment - as a strategic opportunity to drive modern, efficient, open, collaborative approaches to data science. Modern, open working methods for NHS data

Promote and resource “Reproducible Analytical Pipelines” (RAP, a set of best practices and training created in GDS and ONS) as the minimum standard for academic and NHS data analysis: this will produce high quality, shared, reviewable, re-usable, well-documented code for data curation and analysis; minimise inefficient duplication; avoid unverifiable “black box” analyses; and make each new analysis faster.

Ensure all code for data curation and analysis paid for by the state through academic funders and NHS procurement is shared openly, with appropriate technical documentation, to all data users. Data preparation, analysis and visualisation is complex technical work, requiring collaboration by many individuals, who may never meet, in a range of organisations, across the NHS and other sectors. The only way to manage this shared complexity is by sharing information, as in other technical fields.

Recognise software development as a central feature of all good work with data. UKRI/ NIHR should provide open, competitive, high status, standalone funding for software projects and developers working on health data. Universities should embrace Research Software Engineering (RSE) as an intellectually and academically creative collaborative discipline, especially in health, with realistic salaries and recognition.

Bridge the gap between health research and software development: train academic researchers and NHS analysts in contemporary computational data science techniques, using RAP where appropriate; offer “onboarding” training for software developers and data scientists who are entering health services research and epidemiology; use in-person and online training; make online resources openly available where possible.

Note that “open code” is different to “open data”: it is reasonable for the NHS and government to do some analyses discreetly without sharing all results in real time. Data Curation and Knowledge Management

Stop doing data curation differently, to variable and unseen standards, duplicatively in every team, data centre, and project: recognise NHS data curation as a complex, standalone, high status technical challenge of its own.

Meet this challenge with systematic curation work, devoted teams, shared working practices, shared code, shared tools, and shared documentation; driven by open competitive funding to develop new shared curation methods and tools, and to manually curate data for individual datasets and fields.

Use TREs as an opportunity to impose standards on how commonly used datasets are stored, and curated into analysis-ready tables.

Create an open online library for NHS data curation code, validity tests, and technical documentation with dedicated staff who have appropriate skills in data science, curation, and technical documentation; so that new analysts, academics and innovators can arrive to find platforms with well curated data and accessible technical documentation. NHS Data Analysts

Create an NHS Analyst Service modelled on the Government Economic Service and Statistical Service, with: a head of profession; clear job descriptions tied to technical skills; progression opportunities to become a senior analyst rather than a manager; and realistic salaries where expensive specific skills are needed. Analyst Service modelled on GES and GSS can be found in NHSA 1; job roles NHSA 2, 3; supporting an NHS Analyst community NHSA 4, 5.

Embrace modern, open working methods for NHS data analysis by committing to Reproducible Analytical Pipelines (RAP) as the core working practice that must be supported by all platforms and teams; make this a core focus of NHS analyst training. Health System NHSA 24; making change practical NHSA 6, 7, 25.

Create an Open College for NHS Analysts: this should devise (and coordinate delivery of) a curriculum for initial training and “continuing professional development”, tied to job descriptions; all training content should be shared openly online to all; and cover a range of skills and roles from deep data science to data communication.

Recognise the value of knowledge management: create and maintain a curated national open library of NHS analyst code and methods, with adequate technical documentation, for common and rare analytic tasks, to help spread knowledge and examples of best practice across the community; use this in training.

Seek expert help from academia and industry, but ensure all code and technical documentation is openly available to all, procuring newly created “intellectual property” on a “buy out” basis. Commission “Best Practice Guidance” on outsourcing data analytics to cover: where external collaborations can be most helpful; the role of skilled analysts in guiding procurement; common red flags for delivery; and why RAP builds capacity, quality, and continuity of service.

Train senior non-analysts and leaders in how to be good customers of data teams. Governance

Rationalise approvals: create one map of all approval processes; require all relevant organisations to amend it until all agree it is accurate; de-duplicate work by creating a single common application form (or standard components) for all ethics, information governance, and other access permissions; coordinate shared meetings when approval requires multiple organisations; have researchers available to address misunderstandings of their project; build institutions to help users who are blocked; recognise and address the risk of data controllers asserting access monopolies to obstruct competitors; publish data on delays annually; ensure high quality Patient and Public Involvement and Engagement (PPIE) is done.

Have a frank public conversation about commercial use of NHS data for innovation, but only after privacy issues have been addressed through adoption of TREs; ensure the NHS gets appropriate financial return where marketable innovations are driven by NHS data, which has been collected at great cost over many decades; avoid exclusive commercial agreements.

Develop clear rules around the use of NHS patient records for performance management of NHS organisations, aiming to: ensure reasonable use in improving services; avoid distracting NHS organisations with unhelpful performance measures.

Address the problem of 160 Trusts and 6,500 GPs all acting as separate data controllers: either through one national organisation acting as Data Controller for a copy of all NHS patients’ records in a TRE; or an “approvals pool” where Trusts and GPs can nominate a single entity to review and approve requests on their behalf. Approaches and strategy

Use people with technical skills to manage complex technical problems: create very senior strategic leadership roles for developers, data architects and data scientists; offer leadership training to those in existing technical roles. (Also: train senior leaders in the basics of data analysis, software development, and clinical informatics; but recognise the limitations of that approach).

Build impatiently, but incrementally, accepting that new ways of working are overdue, but cannot replace old methods overnight: we must build skills, and prove the value of modern approaches to data in parallel to maintaining old services and teams.

Identify a range of “data pioneer” groups from each key sector: three ICS analyst teams; three national quality improvement registry or audit teams; three academic birth cohort or electronic health record analysis teams; and 1-3 national NHS analytic teams. These should be selected competitively as those with the best current technical skills. Resource them to adopt modern working practices (Reproducible Analytic Pipeline working methods in a Trusted Research Environment alongside Research Software Engineer support) and to develop shared re-usable methods, code, technical documentation and tools; this can be in parallel to “business as usual” in their organisation, but should incrementally subsume it.

Build TRE capacity by taking a hands- on approach to the components of work common to all TREs. Avoid commissioning multiple closed, black box data projects from which little can be learned, or framing these as “experiments”. Experimentation is only powerful where it delivers openly shared working methods, code, outputs and technical documentation from which all can learn. • Develop a common “service wrapper” for TRE access, with civil servants. TRE governance team TRE 11; single standard Service Wrapper model TRE 12; local TRE service model TRE 26. • Develop common working practices for the “generic compute and database layer” of TREs with generic skilled technical teams from private and public sectors.

Focus on platforms by resourcing teams, services and institutions who are focused solely on facilitating great analytic work by other people, working closely with users. Data curation, secure analytics, TREs, libraries, RAP training, and platforms are the key missing link: they will only be delivered if they become high status, independent activities. Conclusions In the past, “data infrastructure” meant beige boxes in large buildings. In the 21st century, data infrastructure is code, and people with skills. As noted in previous reviews, many shortcomings in the system have been driven by a “destructive impatience”: constantly chasing small, isolated, short-term projects; at the expense of building a coherent system that can deliver faster, better, safer outputs for all users of data. If we invest in platforms and curation - at less than the cost of digitising one hospital - and engage robustly with the technical challenges, then we can rapidly capitalise on our skills and data. New analysts, academics and innovators will arrive to find accessible platforms, with well curated data and accessible technical documentation. The startup time for each new project will shrink, productivity will rocket, and lives will be saved. 73 years of complete NHS patient records contain all the noise from millions of lifetimes. Perfect, subtle signals can be coaxed from this data, and those signals go far beyond mere academic curiosity. They represent deeply buried treasure, that can help prevent suffering and death, around the planet, on a biblical scale. It is our collective duty to make this work.

Rapidly create a substantial multidisciplinary TRE technical delivery team This should be rapidly convened by the NHS Transformation Directorate. It is crucial that this team has the right people: it should combine skills in software development, data architecture, clinical informatics, data science, data management, information governance, cybersecurity, and open source software. This work cannot be done by intermittently consulting people with domain knowledge and technical skills as external advisors; these skills must be strongly represented on the core project team as internal staff. Hands-on NHS service analysts and academic researchers with strong skills in some of the prior listed domains must be closely involved in the core team from the outset, but only as expert users: the project must be led by software developers and technologists, not researchers. This team should seek out and involve, by secondment or close advice, the best teams nationally and globally with a proven record of successful completed delivery of TREs in health and other domains including: ONS Secure Research Services and OpenSAFELY as per Secretary of State’s letter to GPs; along with UK-SERP, Genomics England, Public Health Scotland, and others. It is important to cautiously avoid, from this expert group, teams who have had resource to build TREs but not yet displayed any public code, outputs, or technical documentation; or organisations who have simply been intermediaries for funding to other organisations who have themselves built the TRE tooling.

Rapidly agree and publish features for the minimum viable National TRE This team should agree and publish within 2 months, then finalise within 2 months, the core technical design features of a strong MVP for the National TRE that can meet all use cases for the national GP data extract. This MVP must support RAP working, and have a robust service wrapper. It should ideally have underlying compute infrastructure that can scale for a full National TRE. TRE 4. Agree and publish proposed features of the full National TRE This should be a flexible framework that is open to iteration as new user needs are identified, and as experience is gathered from delivery of the MVP.

Manage diverse local datasets by creating and sharing standard data curation tools and methods 5. Ensure all local implementations of national or commonly used datasets such as SUS/HES conform to a single standard 6. Ensure all datasets extracted from national datasets in NHS Digital are requested using standard data management code 7. Ensure local analysts use a national TRE wherever possible 8. Work towards federated analytics with standard local TREs 9. Listen carefully to local NHS analysts and TRE managers who describe shortcomings in standard approaches; and address these wherever possible.

Produce a minimum viable TRE in 6 months Examples of national TREs for NHS data already exist, so MVP implementation that is capable of scaling, and demonstrating rapid delivery, should take no more than six months. This should include a full secure data management and analysis pipeline, most core features, code executing against live data, sharing code as per RAP, and delivering a small range of completed outputs from a range of user groups. Initially this should be delivered as: a performant service wrapper; underlying core services such as database and provisioning of compute; and a range of 2 to 4 TRE service options (a remote desktop, and code execution environments) operating as software layers that use these underlying core services. This de-risks the project as it reduces lock-in to any one TRE platform option, incentivises delivery, permits iterative improvements, and allows the best features of each system to be used.

Rapidly scale over 18 months The following 18 months should be spent onboarding external users, iterating the platform, iterating detailed documentation for users, actively recruiting users, and adding features in response to user feedback. Proof of delivery should be in the form of outputs, code, and technical documentation, not PowerPoint slides or communications material. The team should aim to support the delivery of at least 100 completed and publicly available end-to-end analyses of real NHS data, that meet genuine NHS service or academic research needs, from ten different teams of users, with the data management and analysis code shared openly with adequate technical documentation, by the end of year one. There should be at least 10 python functions, or similar, available on GitHub, that meet core common user tasks within the TRE, with more than one user. There should be full technical documentation of all platform features, and underlying datasets, openly accessible and associated with the relevant underlying code.

Include GP data and certain commonly used national datasets from the start The first iteration of a national TRE should contain the key, commonly used national datasets including GP data, HES/SUS, and prescribing data. GP data is by far the most challenging dataset to support; the others represent a marginal increase in work. This data alone will represent unprecedented depth and breadth of NHS data, supporting a broad array of innovative outputs. NHS service analysts have not previously been able to work systematically with GP data. This alone represents a phenomenal opportunity. Early outputs are likely to include: work evaluating variation in service activity and clinical outcomes between different organisations and regions; monitoring the resumption of clinical activity following the COVID-19 pandemic; identifying opportunities to improve the quality, safety and efficiency of prescribing; understanding and predicting demand based on detailed data about local patients; and so on. When linked to HES and ONS death certificate data at national scale this resource becomes even more productive, as analysts can create a clear picture of the full patient journey through services. Combined with RAP working methods, to ensure work is systematic and reproducible, this will be a new dawn for NHS data science.

Expand the National TRE in time to accept bespoke datasets The Minimum Viable Product for the National TRE must be to support the core national datasets. However, there are many external datasets - such as the cohorts, registries and audits - that rely on ingesting large volumes of national EHR and NHS data, sometimes without patient consent. These projects should ultimately move into a national TRE. Doing so will require that a national TRE is capable of ingesting data and supporting analysis across various diverse datasets and data structures; this is readily achievable and should be piloted with a small number of pioneer registries, audits, or cohorts after the MVP is delivered. This flexibility is readily deliverable, and has been delivered in other settings, including ONS SRS, OpenSAFELY, and SERP. However, there should not be an expectation that a national general purpose TRE would be immediately capable of importing very large and complex multimodal datasets (such as genomic data) albeit that they may import and link more sparse derivatives of such data. Similarly, there are likely to be a range of social care datasets in local NHS TRE settings; these will be bespoke to each provider and region; however they are likely to become more nationally harmonised if the recommendations in this review are followed; these similarly are an important dataset to consider ingesting into a national TRE.

Evaluate new developments in privacy engineering; adapt accordingly The use of TREs is likely to remain best practice for the protection of disclosive NHS data for some time to come. It is however reasonable to expect that the core technical design features of a good TRE will shift as technology develops. The team should annually review the core technical features expected of national and local NHS TREs to ensure that NHS TRE provision does not fall behind best practice for privacy preservation.

All TREs must support code sharing and RAP Adoption of modern open working practices is a core benefit of moving to TREs. All TREs at all scales must support code sharing and minimum RAP working methods. Develop Trustworthy, Agile, Standard Governance for National NHS TREs

Build a TRE governance team to create a robust framework around TRE access A good TRE must be surrounded by good governance, and support this with relevant technical features. This team must contain experts in information governance, policy, customer workflow, and TRE implementation, with close involvement or direct inward secondment from other TRE teams at for example, NHS Digital, Office for National Statistics, Genomics England, and UK-SERP. This team must be tasked with building processes to ensure that: all users are appropriately qualified and have relevant permissions; all projects are appropriate and have relevant permissions; all data access is limited to the minimum participant count and granularity necessary to achieve the analytic objectives to a high standard; all access arrangements are appropriately time-limited; all lapsed or otherwise incomplete projects have their permissions reviewed and revoked; all projects appropriately involve patients in their design; and any additional aspects of work they identify in the first 2 months. This should build on the best prior workflows in for example, NHSD, ONS, and SAIL/UK-SERP. The team must be specifically tasked with ensuring that access is swift: specifically, they should be required to report back formally to senior leaders every month for the first 6 months on any barriers to fast platform access - whether these are regulatory, legislative, technical, practical, resourcing or organisational - so that these barriers can be rapidly addressed.

Create a single standard Service Wrapper model for NHS TREs This should cover issues such as safe people and safe projects; a standard approach to ethics and governance; and a standard approach to Information Governance (IG) and onward access arrangements for datasets ingested into a national TRE.

Create a national standard approach to “output checking” and support automation All results tables and graphs leaving TREs under best practice must currently go through manual “output checking” to ensure no disclosive material is accidentally released: this is time consuming and implementation is variable. The National TRE team should develop a single national standard on best practice for output checking, then require and monitor adherence. They should also collaborate closely with academic teams and funders aiming to automate aspects of this work (as per the list below of illustrative funding priorities for data science infrastructure).

Establish a standard scheme to accredit NHS TRE users There is a need for a single accreditation framework to identify that individuals have appropriate organisational credentials and appropriate training to work safely with patient data. This should mirror the accredited researcher scheme run by the ONS under the Digital Economy Act (2017). Researchers wishing to access data via any NHS TRE should become accredited NHS researchers. Once accredited, researchers should be able to use any of the licensed TREs for approved research projects without having to apply individually for access.

Ensure TRE access is faster and easier than data dissemination TREs are lower risk than data dissemination. They present a range of safeguards that substantially increase patient privacy, and prevent analyses outside of those permitted, by sharing information about all activity in the platform, and by using tools that obstruct and detect misuse of data. The current IG arrangements - widely regarded as slow and obstructive - were developed to manage the risks of data dissemination. They should not apply to TREs. As discussed in the IG chapter, TREs should be subject to a lighter touch regime that is proportionate to their lower risk. This will incentivise the use of this safer and more open approach.

All TREs must share live detailed activity logs These should openly disclose, at minimum: the individual executing code; a link to the documentation for their legal basis to process the data; the datasets against which code is being executed; and ideally the data management and analysis code itself, as this provides the clearest and most unambiguous record of what has been done. This need not include the results of the analysis; only the code, and ideally accompanying documentation, as a public record of what has been done. This will allow the system to maintain public trust that users are only conducting appropriate analyses with their NHS patient records.

Create clear rules for undeclared analyses in TREs A concern has been raised that sometimes there is a legitimate need for certain users to conduct certain forms of data analysis discreetly, without openly declaring their activity at the time. This is a reasonable user need in certain circumstances. However, this TRE activity should be logged as usual, and published at a later date; appropriate rules around undeclared analyses and delayed disclosure should be created by the TRE governance team.

Switch off data disseminations, without undue panic There is no need for potentially re-identifiable disclosive data at national scale to flow outside of secure environments. TREs are the only way to build public trust and get larger numbers of people working on NHS data for innovation, service improvement and research. No new data disseminations outside a TRE should be established. All current large-scale disseminations of GP and other granular patient data - both national and local - should be reviewed within six months for replacement with a TRE option. Any needs that are claimed to fall outside of TRE usage should be published and considered by the TRE technical team for iterative development to meet the proposed shortfall. Despite the public and professional concern raised about the GP data extraction, there should be no panic or repeat of what happened in 2013 following the suspension of Care.Data, when several unrelated data flows from NHS Digital to research users (such as HES) were suspended or delayed with no alternative plan for access. TREs are needed to meet the new risks of more detailed GP data, and wider access to data; they also address the longstanding shortcomings of pseudonymisation and dissemination; but there is no new emergency.

Conduct an annual access audit The TRE Technical Team and Service Wrapper Team should conduct a collaborative annual “audit for improvement” looking at TRE access request and approvals, the count of completed outputs from each TRE, the extent of code sharing and technical documentation, adherence to national standards around service wrappers, and similar outputs.

Publish all technical steps taken to prevent and detect misuse of data All TREs must take technical steps to prevent and detect misuse of patients’ data, such as by obfuscating access to raw data; and disclose these technical methods openly. Ensuring National TREs are Accessible, and Used

The National TRE should be open to all legitimate users Any national TRE containing national NHS data should be open to all legitimate applicants following a rules-based system including: national NHS service analysts; local NHS service analysts wanting to work on their own local data in a national context; academic researchers; government analysts from outside the NHS; and, following appropriate and positive consultation with the public, users from the life sciences sector.

No special cases for working outside a TRE Since the announcement that the planned GP Data for Planning and Research data collection will be TRE-only some organisations have been arguing that they should be regarded as an exception. There is no reason why a TRE cannot meet all users’ needs. Any organisation raising concerns - whether due to technical or governance issues - should be fully listened to, their concerns fully understood, and addressed in the design of a national TRE. Any organisation hoping to build a closed data analysis environment for their own internal use should be encouraged to support and facilitate work on a full TRE so that it meets their needs; or to deliver a full TRE themselves as a piece of national data infrastructure. The biggest challenges in delivering complex national data infrastructure are technical: any group that can deliver on that challenge to produce a closed internal Data Access Environment can be supported by those with generalist and governance skills to add the service wrapper needed to produce a TRE. Any relaxation of this approach is highly likely to result in repeated duplication of work and risk, reduced standards on governance and transparency, obstructions to open working and re-use of code, and monopolies around access that do not reflect the needs of patients or the wider system through arbitrary rules - or arbitrary application of rules - around access to data.

Ideally one national TRE, never more than three Ideally there should be one national TRE. This will minimise duplication of effort around the IG service wrapper, the underlying technical infrastructure, and cybersecurity risks, and minimise unnecessary variation in implementation that has historically obstructed re-use of code. However, in pragmatism, the system should accept the possibility of having very slightly more than one national TRE, only insofar as this is necessary to avoid the risks of non-delivery and inertia caused by a single monopoly provider, and to stimulate competitive innovation and creative diversity. It is vitally important that the number of national TREs is kept to the smallest number possible, and there should never be more than three TREs containing national scale NHS data. TREs containing data of this scale and depth should only be delivered by national government organisations, ideally within the NHS, with clear lines of accountability into government. Consideration should be given to closing any national TRE that is not delivering where others are. Two principles should be followed whenever considering any new national TRE. Firstly, a new TRE should only be considered where it genuinely offers new features. Any proposal for a new TRE should be required to demonstrate with substantial evidence that they will deliver new features, meeting genuine unmet user needs; and that they have tried to deliver this additional functionality at similar pace by adding additional features to an existing TRE, but found it to be impossible. This will block needless duplication. Secondly, any TRE must be a shared environment open to all legitimate users. No organisation should be permitted to create a “nearly TRE” for their own internal use. Any TRE containing national NHS data must be a shared national resource where all NHS and other users can apply for access on an equal footing. Local NHS TRE provision

Create a Local NHS TRE Programme This programme should have a single clear leader and coordinate work on local NHS TREs ensuring that they follow a common governance wrapper, and a single common open technical model to facilitate portability of code, staff, analyses, curation, and (where needed) federated analytics.

Work to rapidly standardise local TRE and DAE provision, starting with ICSs Currently local data analysis work is conducted with highly variable working methods, in highly variable computational environments, with even the same national datasets stored and used in very different ways. This is not informative or fertile diversity: it is largely hidden, and happenstance. It obstructs sharing of code, methods, curation, and learning. This problem can be readily addressed by the following actions: 1. Create a standard service wrapper model for local NHS TREs 2. Ensure all ICSs use a standard TRE approach 3. Encourage other local NHS data centres to use the same standard

Create a single Service Wrapper model for local NHS TREs Based on the work for the National TRE, a standard service wrapper should be created by the same team - in consultation with local NHS and academic TREs - to cover access to work on data in these environments. All local and academic TREs should be obliged to use this standard service wrapper for access, to avoid needless duplication of IG activity, and to avoid (or at least document) activity that may lead data access monopolies. Exceptions to the Standard TRE Service Wrapper should be possible, but to a prespecified set of criteria. Exceptions should be publicly disclosed, alongside the justification, under a robust exceptions framework. Any exceptions and modifications made in any local or academic TRE should be reviewed at six monthly cycles by the team to consider whether they justify a revision of the Standard TRE Service Wrapper. The data ingestion elements of this local TRE service wrapper should aim to impose some standards on the current highly variable and expensively duplicative array of approaches to IG for local data flows. All local NHS TREs should recognise the standard accreditation for NHS TRE users.

Ensure all ICSs use a standard TRE approach Integrated Care Systems (ICSs) are new organisations in the NHS, and are now the primary locus of work for local planning and delivery of care across a region. All ICSs are expected to use data to improve the quality, safety and efficiency of care. This is an outstanding opportunity to drive change. All local TREs for ICSs should be required conform to a single national model of TRE, rapidly developed, with pragmatic flexibility to account for diverse local datasets; then all analytics in these settings can readily move to conform to RAP. The National TRE technical team should rapidly review current planned or actual provision of data analytic services in ICSs, and identify the best opportunities for harmonisation consistent with the principles above. This is likely to entail: a standard open source TRE approach created for ICSs; standard tools (in the formal sense of functions and libraries) for data management to ensure best-case sharing of code, methods and documentation for data curation where a standard TRE is impossible; but not necessarily a standard underlying compute infrastructure. The system should strongly resist the urge to believe that creating a “diversity of approaches” will allow the best model to emerge and spread: previous experience shows that local data aggregation programmes and data analysis environments tend to be closed, black box services with little or no technical documentation from which others can learn, and only arbitrary variations in approach.

Ensure any other local NHS TREs use the same standard TRE approach There is currently a diverse array of local NHS data aggregation projects as legacy from a range of prior commissioning choices. Many of these are delivering strong service; some may warrant further review. All should be strongly encouraged to adopt a single national approach of a standard NHS local TRE that supports RAP and conforms to a standard NHS TRE service wrapper. This will help to address concerns expressed about transparency, open working, access barriers, and duplication of work. Mixed NHS and academic projects built principally or wholly around NHS data should fall under NHS control. Non-standard TRE or DAE projects containing NHS patient records should be reviewed annually, following commencement of an NHS TRE programme, to establish whether they add value to standard NHS TREs: this review should pay due attention to any duplicated work, duplicated risk, or any identified obstructions to open RAP working or user access.

Manage diverse local datasets by creating and sharing standard data curation tools and methods As discussed above, local NHS service analysts work with a diverse array of local datasets that can vary widely between regions. Examples include: detailed data about admissions and discharges as bespoke feeds from specific local hospital EHR systems; regular extracts from local authorities about payments for individuals’ social care, or more detailed but bespoke social care records from some care providers; intermittent reports from single local services about activity and costs. This presents a profound challenge for any efforts to bluntly standardise all local NHS and care data to a single model. Nonetheless many different data centres, widely separated by geography, will have similar underlying datasets, where shared data management and analysis approaches can be valuable; and nearly all will share common over-arching analytic goals. As per the chapters on Open Working and Data Curation, the best approach to making this curation and analysis activity open, efficient and generalisable is to adopt RAP, and create a small range of standard data management functions and libraries that can operate in any NHS data centre, so that curation work can be shared alongside appropriate technical documentation.

Ensure all local implementations of national or commonly used datasets such as SUS/HES conform to a single standard As discussed in Data Curation, much local work is done using national datasets such as HES/ SUS and GP data. At present the same data is needlessly held in different structures, models, formats, and services in each setting. The National NHS TRE technical team should identify the best flexible approach to harmonisation and this should be adopted wherever local representations of national datasets are required (for example in linkage to local datasets that are not available in a national NHS TRE).

Ensure all datasets extracted from national datasets in NHS Digital are requested using standard data management code As discussed in Data Curation, much of the data used by local NHS service analysts in local NHS data centres has been provided from NHS Digital; often this is provided in different forms at different times; often this is the product of a discussion, rather than simply submitting the formal specification of a requested dataset in code. This adds to needless duplication of work at all sites and should be addressed by NHS Digital developing a service to accept complex derived dataset requests in code.

Work towards federated analytics with standard local TREs “Federated analytics” is a secure approach to executing data analysis against locally held datasets in situ, without extracting all local data to a central repository. At its best, a single set of instructions for data management and analysis are written in one location, then sent out to each smaller data centre, where they execute successfully, producing a local version of the results from the analysis using the local data in each setting; these non-disclosive aggregate outputs are then sent back to a central location for aggregation. Successfully executing federated analysis requires that a number of conditions are met: all data in each local data centre must be in a common data model, or capable of being curated into a common model for the purposes of the single analysis, ideally using curation code written centrally; each local data centre must be capable of receiving instructions, verifying that they can be run, and executing them correctly; and each local data centre must be capable of securely sharing completed summary results to the central location for aggregation. A standard local NHS TRE, and even many services that fall short of this goal, can deliver federated analytics in this manner. This approach could be used, for example, to execute a single analysis describing NHS bed occupancy, by calling local NHS data centres containing such data, even in diverse local formats and implementations; or atlases of variation in activity and clinical outcome using data not present in national datasets. Federated analytics is readily achievable and has already been successfully implemented on NHS data in some form, delivering a range of completed analytic outputs in open code projects such as OpenSAFELY and DataShield (the latter in particular with a large existing user-base).

Ensure local analysts use a national TRE wherever possible Many local NHS service analytics tasks entail using only a local cut of NHS patient record datasets that are currently held at national scale by NHS Digital (in the case of SUS/HES, which is then sent out to local NHS users); or datasets that will imminently be held by NHS Digital (in the case of GP data). This work should all move to be conducted in the National NHS TRE as soon as possible. This will eradicate duplication of risk and cost, and permit rapid collaborative open development of shared tools and learning.

NHS Trusts and Data access Environments Many NHS trusts have arrangements for academic and service analysts to use their internal data for a range of projects involving research and service improvement. These projects should be encouraged to adopt the standard NHS TRE service wrapper, and standard NHS TRE technical approaches, with all caveats above around thoughtful structured extension to these standards where they are needed.

Listen carefully to local NHS analysts and TRE managers who describe shortcomings in standard approaches; and address these wherever possible. None of the above should be taken to be a panacea for all possible data uses; there will always be edge cases. Where these present, they should be shared to the National NHS TRE Technical Team. TREs for national audits and registries As above, there is a diverse landscape of different national clinical audit and registry projects, mostly focused on using data for service improvement or monitoring, typically run by organisations outside the NHS, but in close collaboration with the NHS, often in adjacent organisations such as learned societies. These services collect a mixture of routine and bespoke NHS data into various datasets, large and small. The datasets are located in a variety of computational environments, with a range of service wrappers and access arrangements. Often data management, analysis, and visualisation is done largely behind closed doors (excepting the final completed output), reflecting the normal practices of the past. All of these projects have been built and maintained over many years by gifted and committed individuals with positive intentions, often as a labour of love: they are to be admired and praised. However, the current dispersed and diverse arrangements for data management are an accident of history, and far from optimal. Many of those involved in audits and registries are technically minded and passionate about better use of data in healthcare. Many of the datasets are structured as “one row per event” and therefore amenable to hosting in a suitably flexible national TRE.

Use the same TRE approach as above In terms of technical implementation and service wrapper, these projects present essentially the same challenges as local NHS TREs and academic TREs. They are therefore amenable to all the same interventions as above.

Start with Data Pioneers who can demonstrate computational maturity The best route would be to commence with a Data Pioneer project, identifying between one and three national audit or registry projects that are willing to be resourced to move to a TRE, ideally a national TRE, or use a standard “recipe” for a local TRE, and embrace RAP and modern open working methods. Selection should be based on those with the highest computational maturity: specifically, those who currently have the largest amount of openly accessible technical documentation; the largest amount of openly shared and adequately documented code; and the team that can demonstrate the deepest skills - or potential - for RAP and computational methods.

Review the current Registry and Audit landscape; work towards wider access and use More broadly these datasets are likely an under-used resource, and a very dispersed set of projects, of very varying scale: it would be wise to commission a deep dive to describe for each project the data flows, the clinical service improvement and research purpose, where the data is housed, who it is accessible to, and recent outputs. This work should inform future investment and TRE moves.

Work towards audits and registries using national NHS infrastructure, RAP, and TREs The ultimate destination should be that all such datasets are held in one of the national TREs for NHS data, in particular because these projects often involve using patient data without explicit opt-in consent (in contrast to many bespoke data collections for academic research, where active opt-in patient consent has been sought). Academic TREs Academic TREs present a complex challenge, as the work is dispersed, with a wide range of datasets, data structures, tasks, funders, users, norms and working practices, and a strong local powerbase around many single projects. Nonetheless TREs should be regarded as critical national research infrastructure, and they warrant strong strategic direction. Below are a range of proposals around implementation and funding. Academic TRE implementation

Academics should use NHS data infrastructure to access NHS patient records All academic work on NHS patient records alone should always be conducted in NHS TREs, and be compliant with RAP and open working methods. Patient data should only be transferred out to other non-NHS data centres when that patient has consented for this to be done (for example in consented clinical trials or research studies). Any proposed exceptions to this should be considered under a prespecified exceptions framework created by the NHS TRE team.

Academic TREs should use standard NHS TRE Service Wrapper and governance There is a very diverse array of governance and access arrangements across a very wide range of DAEs and TREs delivered or funded through the academic community, including a large number of new projects created only very recently. This duplicates risk, duplicates effort, reduces visibility, risks public trust, and risks reinforcing concerns about monopolies over access. All academic DAEs and TREs should use the standard NHS TRE Service Wrapper and governance arrangements. A limited degree of diversity is justifiable for the smaller subset of older projects where, for example, there may be longstanding differences in the commitments made in consent forms to patients participating in a particular study. However, this is no substantive barrier to harmonisation and any small differences could readily be wrapped within a standard approach on all other matters where there is convergence. Any requirements for substantial deviation from the standard service wrapper should be openly disclosed, alongside the justification; these should be regularly reviewed by the national TRE service wrapper team to identify opportunities to extent the standard governance arrangements.

Academic TREs should use standard NHS TRE and curation approaches where possible NHS data is complex to manage. Wherever any standard approaches have been created by the NHS for TREs or data curation, these should be used by any academic DAE or TRE that is ingesting complex raw NHS patient data with consent. This will ensure they can share in, and contribute to, any curation or analysis code created in the wider community of NHS data users.

All academic TREs should aim to use shared standard infrastructure Where possible all academic TREs should share a common, core, scalable underlying compute infrastructure to avoid needless duplication in procurement and provisioning: this should be discussed in close collaboration with those who have experience in this domain, including SERP for health data TREs, and the UKRI Science and Technologies Facilities Council for shared infrastructure more broadly. This approach may also help to further minimise historic risks around perceived access monopolies, and lack of interoperability and shared code.

All academic TREs must support, and should require, RAP and open working Modern, open, collaborative approaches to computational data science are the norm in other academic fields such as physics, structural genomics, structural biology, and more. Code sharing in the health data and electronic health records research community has fallen substantially behind these other fields. Research with data is done by writing code. The code that underpins scientific research using NHS data must be shared under open licenses for scientific review and efficient re-use as in other sectors. At present many TREs actively obstruct these modern working practices. TREs should be regarded as the main way that researchers in this space can be helped to work in modern open ways by default, by making it easy for them to do so. Supporting and requiring RAP and modern open working practices should be regarded as a core requirement for any academic TRE.

Start with Data Pioneers who can demonstrate computational maturity in research cohorts There have been many attempts by various organisations to harmonise the data hosting and access arrangements around research cohort datasets. There have also been some complex and labour intensive approaches proposed for “harmonising” these diverse datasets. Overall, the most effective first step would be to identify 2-5 cohorts keen to co-develop TRE working, and to deliver their data curation and analysis work in a standard TRE that supports RAP and computational working. By working in the open, executing data curation through RAP processes with appropriate documentation, and using the same openly accessible TRE arrangements as other NHS resources, data harmonisation and federated analytics become substantially more achievable. Participating cohorts should be selected as the most advanced: specifically those who currently have the largest amount of openly accessible technical documentation; the largest amount of openly shared and adequately documented code; and the team that can demonstrate the deepest skills - or potential - for RAP and computational methods (which may include, for example, detailed shared data management code written using less than RAP methods in R or Stata, or demonstrating an ability to think computationally, sharing code resources, and abstracting out tasks). This work should build on the work of the Longitudinal Linkage Collaboration as that is a strong current locus of such work. It should be implemented as an open competitive funding call for a minimum two year working cycle, and up to a six month delay from award to work commencement to allow for inward recruitment and onboarding of software developers and data scientists to join the existing team with domain expertise. Academic TRE funding

All funding for academic work on TREs should pass through a single national organisation Substantial concerns have been expressed by various senior leaders in various sectors around productivity, coordination, and visibility of funding and outputs for TRE work and related data activity in the academic community. TRE delivery, alongside code and methods, is critical national research infrastructure, at the heart of all ambitions to make better use of NHS patient data for public good. All TRE funding activity should be coordinated by a single organisation, most likely inside either the NHS or UKRI. This organisation should have a clear single line of accountability to government and the NHS, and a specific named Minister. It should be open about all accounts, including details on income and disbursements of funds for individual projects. It should be subject to FOI, as an indicator of the organisation’s status and accountability, rather than the specific good of FOIs. All funding for academic work on TREs should pass through this single national organisation. Non-government funders of TRE activity, such as research charities, should be encouraged to participate in this open coordination work, as part of their positive contribution to shared national TRE infrastructure.

All TRE and related funding should be openly disclosed All academic funding for TRE delivery, and delivery of support code within TREs, should be openly disclosed, with adequate technical detail for the community to see the anticipated work and understand its scale. This should include, for each investment: • The source of funding (e.g. council, programme); • The amount of funding; • The recipient (PI, team, organisation); • The headline objectives; • A link to the GitHub repository or website where outputs and work in progress can be seen (including code, technical documentation, or live services)

There should be follow-up on all TRE projects resourced This should be regarded as a positive opportunity to share outputs, methods, code, insights and technical documentation for others to review, re­ use, or improve; and a chance to share barriers encountered for others. It is to be expected that some projects may not meet their initial objectives: this should be accepted as part of the normal process of building in an uncertain and novel space.

Academic work around TREs should be funded through conventional open competition There is a perception from some in the community that academic funding for TRE work to date has been closed and non-competitive. To address this perception, it is crucial that the majority of all academic funding for TRE work is open to all, via open competition, in a process whereby all groups and organisations can present ideas and proposals in open competition. This work should also reflect the reality that many of the productive academic contributions to TRE work are likely to be on innovative methods and code, from those with contemporary skills in Research Software Engineering and modern, open, computational approaches to data science with NHS records.

Funders should avoid short- term funding for infrastructure There is a clear tendency for some academic resource on data infrastructure to be awarded on very short timelines. This may reflect: TRE funding standing outside of conventional competitive funding structures; a lack of strategic coordination; some pressure around short-term funding horizons within funders (although this is overcome for other fields of work). Extreme examples include “sprints”, where resource awarded must be spent at very short notice, starting work within weeks, and completing spend within months. Short term funding creates numerous problems. It obstructs recruitment and capacity building, in a space where capacity is one of the biggest barriers to delivery. It prevents the creation of a stable ecosystem or culture around code, methods, and projects. Perhaps most importantly, short-term short- notice funding is a regressive model, in that it preferentially channels resource to established incumbents rather than new entrants: the groups best able to spend large amounts at short notice are large academic groups with a shortfall in income from other competitive grants. This approach to funding systematically excludes newer entrants and more junior researchers, who may have the strong ideas and contemporary computational skills needed for innovative work in and on TREs. Wherever possible funding for TRE work should be the same as other competitive research projects, with 2-5 year project duration, and a six month delay from award date to start date, in order to permit recruitment into the project. For all that there may be urgency now, longstanding shortcomings will be fixed more quickly by taking this approach, than by reinforcing the procurement problems of the past.

Funding for TREs should be separate to funding for single academic analyses Strong TRE infrastructure can only be delivered in close collaboration with single-subject analysts delivering excellent scientific research papers. However, these are nonetheless two distinct activities. The quality of a TRE project should not be judged by single academic publications on single subjects (except as proof that something has been delivered at all); and the two activities should receive clear delineation in funding and roles. This will help to address the challenge, expressed elsewhere that funding ostensibly earmarked for code, infrastructure, and curation can commonly be diverted into funding academic research projects on single clinical research topics, reflecting the current higher status of the latter activity. An approach with clear delineation of funding, roles and recognition between TREs and single analyses will also help to foster a much-needed, clear, independent community with status around delivery of code, infrastructure, and curation.

All best practice and teams should be identified and augmented There is a historic tendency for academic groups working on TREs and DAEs to be judged by the extent to which they are able to access data at all (reflecting historic challenges around access and IG); or the number of papers they have published. There is a similar historic tendency for TRE teams to want to hold large volumes of data directly, themselves, in single machines under their own control (reflecting historic norms around how value is judged). Lastly there are good grounds to believe that strong technical work on key tasks such as data curation, secure analytics, and other creative aspects of TRE provision, has been done behind closed doors, without open disclosure or recognition (reflecting historic working practices). Insofar as it is possible to create an approach based more on shared underlying compute infrastructure, and collaborative contribution to open code, there is a risk that strong expertise in these existing closed projects will be overlooked. This should be avoided, if necessary with transfer grants to make prior work more open, accessible, and portable.

An overview of prior investments All new funding via national funders such as UKRI of TREs and related resources should begin with an appropriately detailed open inventory of all prior investment in this space over the past decade, with the positive intention to learn helpful lessons around optimal delivery. This should focus on, for each previous investment: the source (e.g. council, programme); the amount; the recipient (PI, team, organisation); the headline objectives (with reference to contemporaneous public relations material and project proposals); and the outputs (including a link to any papers, code, technical documentation, or live services produced; and a brief description of positive learnings around TRE delivery). It should be recognised that delivering infrastructure of this kind is challenging and that there may often be unforeseen barriers to delivery; open documentation of these will help inform future delivery. What to fund Academic funding for TREs should focus on two core themes: (1) support for shared core TRE infrastructure; (2) development of methods and code for core TRE and data management tasks such as curation, secure analytics, and federated analysis, as per the Open Methods chapter.

Standard, national, shared, core compute infrastructure There is - with narrow exceptions - no technical or regulatory justification for any cohort, centre, university or other organisation to insist that all research data they hold should sit on their own machine in their own data centre. Cloud infrastructure is the standard for storing and accessing the most highly sensitive data, including hospital records with patients’ full name and address. Many academic TREs and registry projects could and should share a common, core, scalable underlying compute infrastructure to avoid needless duplication in procurement and provisioning. One national body - likely UKRI, the NHS, or NIHR - should appoint a core TRE infrastructure lead, resourced to create a team that can evaluate the feasibility of a range of standard generalisable approaches. These should not be on nationally owned machines, but rather reflect a standard range of core implementations that can support TRE work for registry and academic projects from a range of commodity suppliers. As above this should be delivered in collaboration with those with experience of prior work in this space.

TRE infrastructure as code and teams Academic funding for TRE work should recognise that digital infrastructure to support efficient, secure, reproducible, high quality science requires delivery of code, and teams who know how to work with that code. Funding should be focused on delivery of innovative methods and working open code with adequate documentation, following the principles set out for funders in the chapter on Open Working chapter. The focus should specifically be on methods and code that are portable, and can be used in all TREs, both academic and NHS. Work should demonstrate that it has avoided uninformative duplication or overlap with NHS TRE activity, and strong contribution to the methods and code used for NHS TREs. To maximise the talent pool and the range of ideas proposed it is crucial that access to funding for this kind of work is open to all, and not limited to applicants from a specific set of academic groups or educational institutions. The list of examples below is not provided as a comprehensive or prioritised programme of work, but rather as an illustrative list of the kinds of work, some reflecting ongoing activity at various universities, that funders could usefully support through open competitive funding to drive a rich, competitive and collaborative ecosystem of code and methodological approaches for key challenges in health data analysis. Methodological innovation and code for Automated Data Release from TREs At present all finished tables and graphs produced in a TRE must be checked manually, twice, to ensure that they do not unintentionally contain any potentially disclosive information about an individual. There is a clear user need for automated approaches to this task, and substantial prior art in this space. A programme of work on this topic might focus on questions such as: what are the core abstracted components of the disclosivity checking task conducted by people; which can be automated; what is the state of the best prior art in this space, for example the more mathematically driven work on uniqueness and disclosivity; what theoretical elements can be implemented swiftly; what are the achievements and problems when this is implemented in practice; how can workflow optimise use of humans where they are needed; and so on. This work combines theoretical work on disclosure and privacy engineering; deep domain knowledge around clinical records, TRE design and user journeys, information governance requirements, and epidemiological or NHS service analytics; and pure software engineering skills. Methodological innovation and code for Data Curation As in the chapter on Data Curation, there is a wealth of work to be done on the best methods for evaluating NHS data, converting it into analysis-ready datasets, systematic approach to validating EHR data at scale, and so on. This work combines theoretical work on informatics; deep domain knowledge around clinical records, TRE design and user journeys, EHR system design, and epidemiological or NHS service analytics; and pure software engineering skills. Methodological innovation and code for data minimisation Minimisation is a commonly used strategy to protect patients’ privacy, but those in decision- making roles at data provider organisations have little formal or specific guidance to help them adjudicate on the correct amount of information to release about each individual in a dataset. Applied methodological work and code tools in this space would meet their needs, drawing on theoretical work for disclosure and privacy engineering; deep domain knowledge around clinical records; information governance requirements; and more. Methodological innovation and code for detection of data misuse Data analysis environments commonly keep logs, but these are currently under-used, or only examined manually. To meet the strong desire for wider access to innovate in NHS data, there is a need for more robust and scalable approaches to monitoring users’ activity. Applied methodological work and code tools in this space would meet this need, drawing in many of the domains and skills listed above. Methodological innovation and code to detect unwarranted variation in care NHS service analysts commonly set out to monitor service activity and clinical outcomes in different organisations, in order to identify which services have the greatest opportunity to improve the quality, safety and cost effectiveness of care, or which services have the greatest to show their neighbours about high quality efficient delivery. There is extensive prior art in this space, and numerous challenges such as avoid over- inclusive or insufficiently sensitive algorithms. As per the chapter on NHS service analytics, there is huge scope to take this prior art, evaluate it, and scale it across the NHS in national and local TREs. Methodological innovation and code for federated analytics Federated analytics is a complex set of tasks. Developing new and effective methods requires deep technical skills around research software engineering, but also very deep technical domain knowledge around the kinds of data to be accessed and curated, and the kinds of analyses to be conducted. For example, simple descriptive statistics can be combined from multiple data centres with simple arithmetic; whereas approaches to combining intermediate outputs from complex statistical models require substantial and creative statistical thought around issues such as mixed effects or fixed effects meta-analysis, the best approaches to combining different intermediate elements, and so on; this becomes substantially more complex in turn when moving from a single analysis to a generalisable framework for multiple similar analyses. These are precisely the kinds of complex methodological challenges that must be overcome to deliver federated analytics for complex analyses on complex datasets: they cannot be met by closed working in siloes.

Exceptions to TRE usage Following the announcement that the GP data extract will only be available in a TRE, some groups began to request exceptions from this rule. Two substantive exceptions to TRE usage are reasonable to consider. Consented cohorts Where participants have already given a very large amount of disclosive and private personal information to a given research project, such as a birth cohort; and they have given their consent for their NHS records to be extracted, transmitted, and matched onto their research records inside whatever data management service the researchers are currently running; then it is unreasonable for their NHS records to be withheld from the researchers. Clinical Trials Patients participating in a randomised controlled trial have already given written and informed consented for their data to be collected as part of the study: again, this should be respected. There are many opportunities to deliver trial follow­ up, and cohort follow-up, inside a national TRE: but there is no need for this to be mandated, as patients’ preferences and consent should be respected. Where there is a clear operational requirement to disseminate data outside of a TRE for purposes that will improve patient care, without consent, it may be reasonable to do so - under a robust exceptions framework - after all relevant IG and ethical processes have been followed, where it can be shown that additional steps have been taken to preserve patient privacy, for example by robustly minimising the data, and releasing data for a sub-sample of the total population. It is important for context to note that census data - for example - is not shared in this way; and that TREs are achievable and bring many benefits. A robust exceptions framework can gather examples of where dissemination was deemed necessary, and review annually the need for any extension or modification to the available TREs to ensure that such dissemination is minimised in the future.

Address TREs for Artificial Intelligence, but as a separate workstream, funded by existing AI resource TREs with the core features described above will readily support all analysis using traditional analytical or epidemiological research techniques. Analysis and research involving techniques that fall under the heading of Artificial Intelligence – particularly unsupervised machine learning – present some different technical challenges for TRE design that need to be considered. These challenges primarily relate to compute power; the use of specific tools such as GPUs; and export controls, because exported random forest models can ( in ways that are hard to detect) sometimes contain disclosive patient data. Overcoming these challenges is essential if patient privacy is to be maintained while using NHS records for AI research at scale.
It is crucial that any bespoke work for an AI TRE is resourced separately from the core TRE work that meets the current analytic needs for the NHS service analytics and research community. AI has very substantial potential (some of it already well demonstrated) around imaging data; it has some potential around EHR data. However, AI work is attention-grabbing, and can sometimes crowd out other work. Substantial national resource has already been devoted to work on AI in healthcare, much of it committed to tasks other than an AI TRE for NHS data. There is a strong case for some resource being spent on creating core infrastructure that allows AI teams – both public and private – to innovate on NHS data in a secure TRE sandbox. This will release economic gains in due course, as with other TRE work. However, the crucial core strategic challenge for better use of NHS data is the overdue need to create strong foundational infrastructure for conventional data analysis in TREs to support current analytic needs with non-AI methods. Overcoming these challenges will likely help deliver an AI TRE.
If separate resource can be found to develop an AI TRE technical team, they should: rapidly evaluate current TRE offers and how well they manage disclosiveness when releasing AI-based models, with open delivery of their detailed technical findings; validate and reproduce prior research evaluating disclosiveness to ensure the results are accurate; expand the specifications of current offerings to overcome any limitations discovered; and implement a TRE capable of safely supporting AI. This work should be delivered through the same methods set out for conventional analytic TREs: approaching the project as methodological innovation and open code with technical documentation, rather than closed black box services; open competitive funding to find the best talent; and leadership from those with appropriate skills and proven delivery on data infrastructure platforms, rather than an excessive focus on single academic skills.