User type: Research Funder
The National Institute for Health and Care Research (NIHR) is the UK’s largest funder of health and social care research. They fund and support research that aims to improve people’s health and well-being and work to translate research findings into improved treatments, diagnostics and services. The NIHR is funded by the Department of Health and Social Care.
We spoke with Dena Shortall – a Senior Monitoring, Evaluation and Learning Manager – in one of the NIHR’s Monitoring, Evaluation and Learning teams, about the use of Overton in NIHR for impact assessment and case study development.
Her team helps to ensure NIHR’s funding programmes, schemes, and initiatives are run effectively, efficiently, and equitably. They also look at the impact of the research funded by the NIHR, finding where it’s being used in policy, practice, knowledge or training. This reporting is used to feed back to the Department of Health and Social Care, to provide an overview of its key activities and achievements.
Need: As a recipient of government money, the NIHR needs to demonstrate the impact of its funded research on the public and patients’ lives, as well as the health and social care system and the economy. Dena’s team uses the platform to discover where NIHR’s funded research is being cited in policy documents, and to gather detailed evidence about its benefits to society.
How does the National Institute for Health and Care Research use the Overton platform?
- To generate evidence to develop case studies and in other impact reporting (for example during government spending reviews) that will be shared with the Department of Health and Social Care.
- To assess the organisation’s progress towards achieving its strategic goals, via the ‘NIHR outcomes framework’. This framework sets out the outcomes that NIHR is aiming to achieve, to improve public health and provide economic benefit, and provides systematic evidence of progress towards achieving them.
What were their problems?
Sourcing information to evidence policy impact can be a huge amount of manual work. Before using Overton, NIHR colleagues had to do desk research or rely on disclosure from the funding award holders to find out about the policy impact resulting from NIHR research.
Even when they did manage to discover previously unknown examples of policy impact through manual search, these often came with little context. Development of impact case studies in particular requires robust evidence and a compelling narrative, and only certain policy citations will be appropriate to form the basis of an impact case study or other impact product. This can mean that extensive research is needed before Dena even decides which projects from their vast portfolio are worth writing impact case studies about.
Before using Overton, they were using several different tools and databases to help them find relevant information on the impact of NIHR research, but they felt that there was an obvious gap in their data. None of these tools focused specifically on policy citations and so the results were often unfocused, inaccurate or incomplete.
This was especially apparent when reporting on the NIHR’s Outcomes Framework. Dena is the Indicator Lead who develops measures around the relevance, timeliness, influence and accessibility of NIHR-funded research. She has to track and analyse their progress in these areas – by, for example, measuring the number of publications produced through NIHR research, and the number of instances these publications are being cited in policy outputs and guidelines – to understand where and how NIHR research is achieving broader impact and to assess how NIHR is progressing to achieve its longer term outcomes.
How does Overton help?
Using Overton has improved how NIHR monitors policy mentions and citations of their funded research. Dena and her team can find this information quickly and easily via a bespoke filter on the app, which pulls through citing policy documents on all their funded publications. This means that they don’t need to be aware of a source or have information from an award holder to find out about policy impact.
On top of this, the contextual data provided by Overton alongside each citation helps Dena and other colleagues at NIHR assess the significance and nature of a policy citation and build a better understanding of the impact of the research, to decide more quickly if the research project should be the subject of a case study, saving time and avoiding wasted effort.
“Overton gives you a lot of detail as to where the research has been cited. You can easily find out a lot about the source citing the research as well as the capacity in which the research has been cited, to help determine what the impact of the policy citation was. Unlike some other tools where you have to go looking for any additional contextual information. It makes it easier if I need to follow up to find out more.”
Collecting information to evidence NIHR’s impact on policy and how it is progressing towards its longer term outcomes is now more systematic rather than ad hoc and piecemeal. The NIHR has a more comprehensive understanding of their impact and presence in policy-related sources as a result.
They can also track and evaluate their policy influence more strategically, identifying trends and patterns in the citation data for health-related areas of interest. Overton’s extensive filters mean that they can refine their searches and do more sophisticated high-level analyses – looking specifically at the evolution of their impact in certain policy types, organisations or countries, or looking at specific time periods. It’s also possible to filter results by Open Access status, which means that Dena can cross reference her results and see how other indicators overlap.
Best features of Overton for NIHR
- A bespoke filter option, showing all NIHR’s funded research. Instead of using the funder filter the NIHR annually sends Overton a list of DOIs representing all their funded work, and Overton builds this list into their view of the app. This is much more accurate than relying on data from external funding registries
- The specific focus on policy documents makes the outputs significantly more accurate and relevant than those of other similar tools
- The use of full-text indexing means that mentions of NIHR-funded research are detected in the body of publications, which unearths results that wouldn’t be found by using search engines
- Advanced filtering function allows NIHR users to break down results and analyse them more comprehensively
- The rich context provided for each citation makes further research and follow-up easier
- An intuitive and easy-to-use interface
