Guide
By Kat Hart, Analyst at Overton
This guide shows you how to use the Overton Index to find how your work and ideas are used by decision-makers and demonstrate research influence in policy.
We’ve drawn on our collective experience of working in research and policy engagement, as well as the top tips and tricks that we share with users, to create a resource to help you tell a compelling impact story.
Here we show you how to use the platform to track policy influence and impact – whether it’s yours, a research group or a wider set of scholars at the department, faculty or institutional level.
We hope there will be something for everyone here – from researchers with no previous experience of policy engagement, as well as policy teams, research managers and other colleagues in research, communications or advocacy roles. As always, we’re keen to hear from the community should you have any questions or ideas for the data! Just drop us a line at support@overton.io
Tracking influence and impact can help you understand how your research might fit within current and future policy debates which can help you identify collaboration opportunities and shape your scholarly focus.
This of course has social value, in that it helps ensure the policy-salience of your research and build an evidence base that can be used by decision-makers in government and civil society.
There are also important personal reasons you might want to track the influence of your research. Many funders ask researchers to provide evidence of the real world application of the project in their grant reporting. Funders are also more likely to award grants to groups or individuals with previous examples of ‘research impact’. On top of this, scientists are facing more pressure to demonstrate the influence of their scholarship outside academia in order to get job offers or promotions – increasingly, research assessment metrics are including ‘real-world impact’ as a key measure.
Demonstrating that impact and tracking it can be a challenge, but there are a few ways to use the Overton Index that you might want to consider and adapt to your own needs. We’ve outlined them below.
Overton Index helps you find where your scholarly ideas have influenced policy. Researchers can shape debate and decision-making in a variety of ways and the Index is designed to help you identify where this may have taken place, automatically identifying citations, alongside full-text search.
Sometimes researchers are cited directly in policy documents, which means that a specific piece of work has been consulted, and is acknowledged in the references.
Sometimes however, a researcher’s body of work is used more broadly to inform the policymaker’s thinking, rather than one paper being referenced as direct evidence. In these instances, the researcher may be named in the body of the document but no formal citation of work appears anywhere.
While other tools or search engines struggle to pick up these name mentions, we ‘full text index’ each document so we can find more examples of influence.
You can also look at the ongoing ‘reach’ of the research via second and third-order citations and mentions, as policy develops. This is where a piece of research is cited or mentioned in a policy document which is then cited or mentioned by other policy documents – a cascading effect, if you will. We allow you to trace this pathway of citations from the original piece of research through the policy landscape.
This allows you to develop a more comprehensive picture of the part your research has played in the policy development process, as well as creating an automated, ongoing record of your influence and impact for grant reporting or promotion applications.
Let’s take a look at how to do this.
In our example below, there are multiple researchers listed with the same name but with different affiliations. We would expect to see this as researchers tend to move between institutions over the course of their career, which is then reflected within OpenAlex’s affiliation data.
There are a couple of options or considerations here.
Perhaps you’re looking at a researcher’s outputs as part of a wider exploration of a research group or institution’s policy engagement or impact. In which case you may only want to see their policy citations for research published when they were affiliated with a specific university.
Here you should select the tick box next to the relevant affiliation from the list and navigate to ‘see all documents linked to X people’.
In other instances you may be looking at all of a researcher’s outputs to understand their historical influence. In which case, you can multi-select each of the authors that are relevant to your search.
In these instances, we believe that they’re all the same author, but without double checking via a third party platform e.g. LinkedIn or ORCID (which it wouldn’t be possible for us to do manually for every researcher that appears in the Index) we can’t be too sure. So we’d encourage users to consider this in their approach to searches.
Once you’ve selected the author(s) you’re interested in, use the ‘explore’ menu to look at the scholarly articles which have been mentioned or cited in policy.
For the purposes of this example, we’re looking at a scholar affiliated with the University of Chicago, ‘Researcher X’.
From here you can select ‘See report’ to get a summary of the author’s most influential outputs. The report shows us that the author has 196 scholarly books and articles which have policy citations or mentions, and these have been cited across 473 sources in 64 countries.
This page gives you an overview of the researcher’s outputs and where they’ve influenced policy. It shows which policy sources cite the researcher’s work and summarises how often their research appears in different source types – e.g. government documents, IGO publications.
Clicking on any of these links allows you to drill down into the policy shaped by this researcher.
For example, the United Nations has used 36 of this researcher’s articles to inform their policy.
If you follow the link on the word ‘United Nations’ you can explore the UN policy that cite the researcher, and see which specific papers were used to inform each document.
However clicking on the number 36 shows you an ‘article first’ view, where you can see all of the author’s research used by the UN as well as any other citing policy document.
An ‘article first’ view, of Researcher X’s paper which are cited by the UN, alongside each paper’s other policy citations
Although the automatic report gives you a handy numerical summary, we recommend that you do follow through and explore each of the citations to understand more about the source, as well as how your research may have been used. This is especially useful when creating research case studies or similar reports, as you will need to give rich detail on the nature of the policy influence and what change it brought about.
Policy is often a convoluted and iterative process. As a result, simply looking at direct citations doesn’t necessarily paint a complete picture of a researcher’s policy influence or impact.
The Index can help you fill in some of those blanks. Though we couldn’t possibly claim to give a full reckoning of scholarly impact – much policy influence simply isn’t recorded, much less quantifiable – we can shed some light on the complex network of influence that occurs in the policy landscape.
Researchers and policymakers don’t always speak directly to each other. Scholarly research isn’t always accessible to decision-makers in government. This means that lots of scientific evidence gets ‘translated’ by intermediaries – organisations like think tanks who repackage, summarise and aggregate research into more public-facing formats which are specifically designed to influence policy.
This can create complex citation pathways in which, for example, a piece of scholarly research is referenced in a think tank report, which is in turn referenced by another policy influencing organisation (and on and on) before being used by policy makers to shape real world decisions.
So when we use the phrase second, third (and so on) order citations, we’re talking about how your research has shaped policy within a given phase of the policy development process. The Overton Index allows users to track where policy documents (or policy influencing documents like think tank reports) cite each other, offering a more comprehensive view of research impact.
This time we’ll look at ‘Researcher Y’ who works at the University of Nottingham.
Navigate to the ‘explore’ menu again – this will give the option of seeing all policy documents citing that researcher.
To assess second and third order citations, you can use the ‘cites others’ and ‘is cited by others’ filter – this will give you the policy documents citing Researcher Y which then have onward impact – i.e. have been cited by other policy documents.
Let’s break this down, using the ‘cites others’ and ‘is cited by others’ filters to narrow our results further:
Immediately, we find all cited policy documents connected to Researcher Y. In this case, they’ve been cited in reports by sources including the IPCC, the United Nations Environment Programme and UNESCO.
These are ‘stage one’ or ‘direct’ citations. These then go on to influence articles produced by other sources, as illustrated in the diagram below.
You could explore this more and more, likely accumulating larger and broader document sets.
This gives you a more holistic picture of how your research has shaped policy at a given development stage, including the citation context and how the policy document it’s been cited in has then shaped further stages of policy (as well as whether or not your work has been reflected in those deliverables).
Of course, being simply cited in a policy document doesn’t immediately equate to ‘policy impact’. But being cited in a policy document which then goes on to be cited a fair bit by others (often including ‘high level’ decision-makers like governments) might be an indication that your work is influential in the broader policy landscape.
There are many reasons why research might be used in policy. There are also many reasons why research might not. Aside from the engagement efforts of the researcher, factors can include the article’s age or an unexpected policy agenda arising (among others). As such, it’s important to be familiar with some of the nuances to influence and impact in the policy sphere, which helps strengthen your use of the data.
Academic-policy engagement happens in lots of ways, many of which aren’t documented in a public forum. Researchers might contribute through a phone call, meeting, submission of written evidence, co-produced research, roundtable discussion, fellowship and more.
Policy publication practices also vary widely. Differences include publication rates, rapid changes in some policy areas, and the lack of standardisation in how research is referenced, cited, or used. On top of this, there is no standardised method for crediting or evidencing the impact or influence of research on policy.
The point is – context is key.
Our data can be used to identify policy influence and reach through quantitative and qualitative analysis. However, for the reasons above we don’t use our data to draw any big conclusions about the ‘most impactful’ scholars or institutions.
(though we know that some users do use our data to create rankings – read our blog on how best to approach this).
Our data is best used to enhance your understanding of the policy landscape and the use of evidence within it. It’s a way to support further investigation, rather than a set of contextless numbers. We provide additional context for each citation in the form of a ‘snippet’ of the policy document, so you can understand more about how your work has been used.
We recommend that you analyse the citations – by following the link to the original policy document or even contacting the document author – to ensure you draw the right conclusions. This is especially important if you’re creating an impact case study or report.
Start, middle, or end?
Can we say anything interesting about the ‘level’ of impact as a result?
Has your research been used to introduce a fundamental principle in the introduction of a policy document?
Or has it been used to bolster policy recommendations or conclusions at the end of an initial consultation/policy evaluation period?
Is the policy agreeing with or challenging your research findings?
How frequently is it mentioned or cited in the same document?
Has it been used in more than one document from the same author on the same topic?
Have there been second and third order citations via subsequent policy documents?
National-level government? Local authority? Think tank?
Where do they ‘sit’ in the policy space?
How influential are they at driving policy outcomes?
Using metrics for policy impact can be a bit of a chicken and egg situation. Sometimes, they can highlight where we’ve unexpectedly been picked up and point us in the direction of ‘new’ impact narratives. Influence may also come about because we’ve actively increased our own engagements with policy audiences, in which case, we can use the data to evaluate those. In any case, the knowledge of your own work and relationships with policy audiences is key to contextualising those data points.
Generally speaking, we’d encourage you to think of the platform as a useful tool for discovery, as well as a novel tool for generating analytical strategies and narratives as part of your path to sharing your research, influencing public policies and having a broader societal impact with your work.
The Overton Index helps you find policy citations and measure the influence of research. But we also have another tool to support your policy engagement efforts.
Overton Engage is a platform to help researchers connect with decision makers and shape policy outcomes. We’ve collected policy engagement opportunities from across the world into one convenient database, to show our users how they can make a difference.
Researchers can get a free version of the tool, which provides access to all of the opportunity listings, while institutions can unlock additional premium features. An institutional subscription allows policy teams, research offices or administrators to match each opportunity to suitable faculty, allowing them to get more strategic about policy engagement.
Subscribers also get access to the Engage training resources, which offers guidance on research translation and network building, to make sure that researchers are ready to respond effectively.
The Stockholm Environment Institute explain how they use Overton to track impact for their annual reporting
Liverpool John Moores University explore how they use the Overton platform, to discover impact for use in promotions and grant applications; to find policy documents and grey literature; to help with strategic planning
How Overton is used by researchers to discover policy documents and inaccessible grey literature
ELIXIR explore how they use the Overton platform, to prove the impact of their research in reporting to stakeholders
The National Institute for Health and Social Care Research explore how they use the Overton platform, to evidence the value of the research they fund in reporting to stakeholders
We believe policymaking should be evidence-based and transparent – we think this is the best way of ensuring policy decisions have a positive impact on the people they affect. We aspire to bridge the gap between the academic and policy worlds.
Through the Overton platform we make it as easy as possible for researchers to understand their influence on the policy process. Our flagship product is the world’s largest database of policy and grey literature. We collect millions of documents from policy sources all over the world (including governments, IGOs, think tanks and NGOs) and pull out the evidence that they used in the publication – we identify references to the research, people, organisations and other policy documents that are cited or mentioned.
The platform can be used as a research tool, for finding policy and grey literature, and as a tracking tool to find and evidence mentions of individual or institutional ‘impact’ in policy. Importantly, this data allows you to analyse how influence works in the policy sphere, to identify trends and extract learnings about the use of evidence. This can help you in your pursuit of policy impact.
Fill out the form below to get a free trial of the platform, and explore the analytical approaches outlined by Kat.
by Kat Hart
Analyst at Overton
Kat helps to find stories in our data. Her background is in Higher Education management, becoming interested in the policy-research interface as a knowledge broker on the Capabilities in Academic-Policy Engagement (CAPE) project. In her spare time, you’ll find her working towards her PhD project at the University of Nottingham, themed around river management policy.
Overton connects research with policy.
The Overton Index is the world’s largest database of policy and grey literature, connected to all the evidence it references. Overton Engage is a database of policy engagement opportunities designed to connect researchers with government decision makers.
Copyright Overton.io 2023 All rights reserved
