Winter is Coming: Countdown to REF 2021 Begins
- 21 September 2017
- Posted in: Education, Science & Technology
The countdown to REF2021, the assessment process for university research funding, has well and truly begun, as delegates attending Research Impact: Strengthening the Research Excellence Framework learned.
In an informative and entertaining panel discussion Professor Simon Tanner, Pro Vice Dean (Impact & Innovation) at King's College London, paraphrased the Game of Thrones slogan for the impending assessment criteria. Winter may be coming but there are just three summers until REF2021! Although universities appear to much better prepared for the impact element than 2014 he said the dragons are circling researchers to ensure submissions reach a high standard.
"At times, the REF has made me feel like Jaime Lannister on the battlefield, helplessly awaiting the screaming Dothraki hordes stampeding over the horizon.“ Professor Simon Tanner
Dr Tim Horne argued that good research will always be recognised above and beyond well produced submissions.
"There are still lots of unknowns for the next REF, but universities should concentrate on creating an environment in which good research can flourish." Dr Tim Horne
r Chris Hewson, Social Sciences Impact Manager at University of York, discussed how an institutional approach was key to positioning the impact element as enabling force rather than an add-on to research activities. He voiced considerations for how we develop plans, people and be nimble arguing different institutions will have different resources and priorities.
Dr Firth, Deputy Manager of REF 2021 within the Research Policy Team at the Higher Education Funding Council for England, used her keynote address to build upon recommendations made on the future operation of the Research Excellence Framework in the Stern Review (2016). Stern focused on the need to reduce game playing, reduce institutional burden and incentive institutional investment.
"Universities will not be able to copy and paste." Dr Catriona Firth
Firth outlined the changes for the next assessment, with greater weighting to impact and value placed on public engagement, and timelines for consultation. Additional research on previous REF submissions will need to be clearly defined in REF 2021. Firth told the audience that interdisciplinary outputs will be a key focus for REF expert panels and that two proposals are being considered for outputs and transition would be needed for the next cycle. Consultations are now in place for staff submissions, portability of outputs with final guidance and criteria for REF2021 expected winter 2018/19.
"We recognise that impact was very successful. Driven by positive behaviours in the sector and allowed the sector to articulate very effectively to government the value of research." Dr Catriona Firth
Positive behaviour and indirect impacts of the impact assessment were later highlighted the by Rand Europe's Susan Guthrie. She shared evidence of key operational changes taking place in universities due to embedding the impact element. Findings from the 2014 Ref Panels found that the relationship of research and impact the most difficult to assess.
Dr Simon Kerridge, Director of Research Services at the University of Kent highlighted the change in ratio of impact and output assessment that could result in monetary impact and stressed there may be other changes in institutional behaviour that might be more negative.
"Changes to REF 2021 mean there are new added burdens - how do we de-select staff submissions" Dr Simon Kerridge
Kerridge argued that there could be stigma for non-selection as research staff and asked if we will see staff shifted to teaching only contracts.
"The driver for gaming is now bigger." Dr Simon Kerridge
Concerns of alt-metrics - new incentives for gaming, irresponsible metrics and narcissism - was raised by The Metric Tide author James Wilsdon, who argued for algorithmic accountability. Is predictive metrics in academia 'Moneyball' for professors? He asked the audience stressing that good metrics is all about governance, supporting not supplying the assessment process.
"HEFCE should be congratulated how they have taken the seriousness of that measure." James Wilsdon
Wilsdon, Professor of Research Policy and Director of Impact and Engagement at the University of Sheffield, told the audience that although Peer Review has limitations it remains key, metrics should not supplant expert judgement. He said that Metrics are flawed but there are positives such as measuring social, cultural impacts but questions remain of how we measure these beyond scholarly impact. Analysis looking at interdisciplinary work, unifying data and linking it correctly, capturing types of citations and impact indicators is needed - More research needed into science of science policy.
Jon Stroll, Director of Research Analytics at Clarivate Analytics, highlighted the rise in indicators such as social media to show how metrics have developed in recent years and influenced new trends, issues and challenges around the REF 2021.
"We are in an age of too much information, ensure metrics are not just responsible but relevant." Jon Stroll
Digital Science's Martin Szomszor shared how UK researchers had evidenced impact in the previous REF 2014 exercise. Szomszor argued how data can make life easier for researchers through research metadata and the open data revolution, outlining how data can be linked up effectively. In his presentation Szomszor shared with the audience the links of case studies that drew on different research disciplines evidencing how grants can be linked directly with publications and citations.
Tom Frostick of University Alliance stressed the complications of institutional-level assessments with better rewards for inter-disciplinary work and having to avoid duplication of submissions. Frostick raised concerns that those institutions with bigger stories to tell have a natural and unjust advantage.
"Institutional-level assessment has problems, pilots are happening, ten years is more than enough for sector to get it right." Tom Frostick
Professor Hugh McKenna CBE of Ulster University offered delegates practical advice and insights on strong submissions and evidence of research impact. McKenna stressed the importance not to confuse dissemination and impact.
"Impact is a team sport - it involves a team and not an individual." Professor Hugh McKenna CBE
Shireen Ali-Khan of Vertigo Ventures and University of Kent's Renata McDonnell offered practical advice on having a "go to" impact person and ensure senior management buy-in and advocacy for impact reporting. Their shared talk demonstrated how University of Kent had embedded impact through reporting and involving changes in human resources, teams, and senior management. Tracking making it easier to identify case studies for REF submissions.
"Where there is impact evidence is appropriate." Shireen Ali-Khan
Throughout Research Impact: Strengthening the Research Excellence Framework, speakers addressed the widely-held concern that the REF has created so called 'research portability'; the ability for academics to take research carried out at one HEI and include it in REF submissions from another. Research portability has been blamed for creating a costly 'transfer window' leading to the REF deadline where universities can pick-up staff and, by extension, their previously published research, to boost their REF scores.
Adam Golberg, Research Development Manager for the Faculty of Social Sciences, University of Nottingham, made several suggestions as to how institutions can counteract the portability of academic staff in his entertaining presentation; fixed term contracts, rolling publication portability windows and a limited carry forward portability based on transfer date and more emphasis on potential when recruiting rather than an existing track record.
"A REF period should be consistent like a 38 game Premier League season, not a moment every few years like the Olympics." Adam Golberg
Golberg argued that research mobility is a good thing but there are problems if institutions receive no credit for submissions as they are entitled to a fair return on investment research. Would a transfer window minimise the effect? Knock-on effects of portability could see recasting recruitment so it is based on potential - Golberg said this was a huge change and possible advantage to early career researchers.
