An expert workshop was commissioned to inform this report on the issues around evaluating Learning Health Systems [177]. It became clear during the evaluation workshop that the scope should be expanded to include appraisal: ie deciding if a project should proceed. There is no single route for an organisation to decide to implement a Learning Health System. The initial idea may come from staff within the organisation, who have read about or experienced such systems elsewhere. It may come as part of a partnership or collaboration, such as forming an Integrated Care System [178]. It may be part of a national strategy that advises or mandates the development of such systems.
Implementing a Learning Health System is usually not an “all or nothing” decision. It will often be built on legacy systems and existing Quality Improvement capability and may not require an organisation-wide decision. Smaller initiatives could be rolled out within departments, with delegated authority. Often, the influence of the person or group proposing the idea – or who will eventually implement the idea – is just as important as hard evidence. A Learning Health System is also more likely to be implemented if it aligns with the organisation’s goals. Organisations sometimes seek evidence yet fail to challenge the methodology used to generate it.
The HM Treasury Green Book [18] presents widely used guidance on conducting an appraisal, covering the following topics:
- Rationale for intervention: Why something needs to be done, including objectives and outcomes
- Options generation: A longlist of the options for delivering the stated outcomes
- Shortlist options appraisal: Comparing the different options
- Valuation of costs and benefits: A comparison with “business as usual”
- Distributional analysis: The impact on different groups in society
- Uncertainty: Applying sensitivity analysis to variables that might change
- Optimism bias: Adjusting for the appraiser’s tendency to overestimate benefits and underestimate costs
- Risk: Accounting for the impact and likelihood of risks that might occur and how they can be mitigated
Following an initial appraisal, the organisation might be persuaded to launch a large project, or they could prefer a pilot study with limited scope. Pilot studies can be a powerful way to test hypothesised impacts and feasibility. They can start to identify knock-on impacts on other services, though they sometimes miss important interactions because of their limited scope.