Research Impact Assessment
Research Impact Assessments (RIA) play a crucial role in ensuring that research investments are optimised to generate the greatest possible impact for society. RIAs provide a structured approach to evaluating the impact of research, ensuring that the results are valid, reliable, and robust. The information obtained from RIAs can help funders and research organisations allocate resources more effectively and make informed decisions about future research investments. Furthermore, RIAs provide evidence of the impact of research, which can be used to communicate the value of research to various stakeholders, including the public, funding agencies, governments, and industry partners.
Choosing a suitable RIA framework
There are numerous RIA frameworks available. These four principles will help guide the selection of the most suitable frameworks for evaluating the impact of your research (Louder et al., 2021).
1. Be clear about underlying assumptions of knowledge production and definitions of impact. Clearly define your impact goals and a working definition of research impact at the start of the research project. Deciding on a working definition of impact is important to understand what is classed as impact and what can be collected as impact evidence. Clarification of impact goals will help predict impacts and evaluate the projected research impact pathway.
2. Attempt to measure intermediate and process-related impacts. Identify impact indicators that show incremental changes in problem framing and understanding. This could include perspectives on increased capacity, changes in expertise and knowledge co-production activities. Attention to intermediate impacts may indicate stages of progress to determine if a project is on track towards intended outcomes.
3. Balance emergent and expected outcomes. Evaluations should include an open-ended component to allow for unexpected outcomes, whether positive or negative. Outcomes can be subjective and unpredictable and predicted impacts may still be emerging. Focus on the elements which have changed or are still changing.
4. Balance indicators that capture nuance and those that simplify. Review indicators to include quantitative data for ease of benchmarking along with qualitative measures to identify why or how changes occurred. Simplistic evaluations can overshadow changes resulting from knowledge co-production and lead to ‘gaming of indicators,’ where designing research to maximise scoring is incentivised. Comparable balance is needed for the total number of indicators, where fewer indicators may simplify the evaluation, but additional indicators can offer further insight and context.
Conduct a RIA
The International School on Research Impact Assessment (ISRIA) have gathered insights and compiled these guidelines to help conduct an assessment effectively (Ovseiko et al., 2018)
1. Analyse your context. Examine the research context to understand the purpose of the research and potential contributions to the broader research area and beneficiaries. Understanding the context will also help identify which assessment methods and indicators to use.
2. Reflect continuously on your purposes. Regularly evaluate RIA objectives (ie. accessibility, advocacy, accountability, analysis and allocation) to refine the assessment questions and methodology.
3. Identify stakeholders and their needs. Determine stakeholders and analyse their needs to prioritise stakeholder interests, develop engagement strategies and determine assessment requirements. The RIA must address stakeholder interests, opinions and behaviour, where stakeholder prioritisation is considered according to impact, influence and interest.
4. Engage with key stakeholders early and often. Engage with stakeholders early and stay in regular contact throughout the process of RIA to ensure the legitimacy of the RIA. Developing rapport with key stakeholders can improve the RIA results through effective knowledge exchange, engagement and co-production of impact.
5. Choose conceptual frameworks critically. Determine a suitable conceptual framework to simplify the process of data collection, organisation, analysis and communication. The right framework can also help address methodological challenges and research impact comparisons across different disciplines, organisations and countries.
6. Use mixed methods and multiple data sources. Apply a combination of mixed methods and a variety of data sources to enhance the robustness and trustworthiness of the assessment. The value of RIA can be increased by using more than one method and data source to develop rich accounts of research impact.
7. Select indicators and metrics responsibly. Choose quantitative indicators and metrics carefully to avoid gaming of the RIA and unintended negative results. Focus on what counts as impact evidence and use a balanced set of indicators and metrics that focus on key impacts of interest.
8. Anticipate and address ethical issues and conflicts of interest. Predict and prepare to address any ethical issues and conflicts of interests that may arise from the RIA process or implementation of recommendations. This may occur at both an individual and organisational level so it is important to clearly state the purpose of RIA and consider how it will be perceived in any communications.
9. Communicate results through multiple channels. Develop and implement a comprehensive communication strategy to facilitate effective translation of RIA results. Ensure to tailor messaging and use multiple communication channels for different stakeholders to maximise knowledge uptake and effective translation.
10. Share your learning with the RIA community. Distribute results and messaging to maximise reach amongst the research community to develop methods and grow the RIA evidence base. RIA is a multidisciplinary field of practice, sustained by empirical knowledge and practical skills which can only progress and expand with the dissemination, uptake and translation of RIA methodology and results.