1 What changes to existing processes could more efficiently or more accurately assess the outputs, impacts and contexts of research in order to allocate QR? Should the definition of impact be broadened or refined? Is there scope for more or different use of metrics in any areas?
1. The core of REF, if it is to remain as a diligent assessment of academic research quality, broadly trusted by the academic community, should remain largely as it is currently configured: in the hands of experts who care passionately about academic research and who have the expertise and integrity to assess research quality, the conditions needed for its delivery and its impact.
2. Trust: It is vital that the processes involved in any future REF, or its equivalent, are valued and accepted by the community on which the outcomes impact. The system needs to be credible and the academic community needs to have confidence in REF. While concerns have been expressed in some quarters about the efficiency of the RAE/REF process, there is broad recognition that much that has been utilised has worked. Outputs and impacts are an appropriate way of assessing two ends of the research process, and on balance, seem reasonable.
3. Metrics of outputs: Geography is a discipline that spans the natural sciences, social sciences and humanities. In common with many other disciplines, we do not believe citation data, or other variants, quantify ‘quality’ independent of peer review (analysis and evidence to this effect is presented in Richards, K. et al. (2009). The nature of publishing and assessment in Geography and Environmental Studies: evidence from the Research Assessment Exercise 2008. Area, 41(3), 231-243). The responses of the RGS-IBG to consultations dating back to 2008 have been critical of the use of metrics in research assessment, with a strong preference for peer review. Our position remains unchanged. There is no evidence we are aware of for metrics, established or new, that can adequately capture originality, significance and rigour of geographical academic outputs. Moreover, metrics discriminate against certain types of publications: books, monographs, edited collections and non-standard academic publications (such as scholarly websites).
4. Metrics and equality: We know from the citation and readership data for papers in our own journals, ‘established’ ‘recognised’ disciplinary figures are privileged. Young/ emerging scholars are disadvantaged by metrics on outputs.
5. Impacts: the REF impact assessment was of concern to academics/departments prior to its implementation. However, the case study approach and the facets of impact required within these statements generally seemed appropriate and has gained acceptance. Metrics cannot and should not substitute for these. Institutions and departments have learnt much from the REF2014 exercise. Substantial changes in processes now would generate neither economies of production nor greater clarity and effectiveness in assessment. However, we urge further clarification on a number of aspects of definition, as follows:
6. Co-production and impact: The definition of impact is broad, in terms of the areas in which impact occurs. However, this definition could be usefully broadened to remove any ambiguity around co-production models of research. Such models are increasingly encouraged by research funders, but researchers who work closely with stakeholders to co-produce knowledge throughout the research process may then struggle to obtain fully independent evidence of impact. Clear guidance is needed for both researchers and panels, to ensure that alternative impact models (i.e., those that stray from a simple linear progression from conception to research to use) can be recognised and rewarded.
7. Public engagement and impact: It would be very helpful to clarify whether and how public engagement can be evidenced through metrics of mainstream or social media. Clearly it is not impact, as currently defined, to have a major documentary series, or to have appeared on Newsnight. But these are categorically different and the REF should be clear on this point and the REF should be more inclusive of media involvement.
8. Research impact on the international academy is one realm in which the UK is excellent. This is not captured adequately in the current formulation of impact. There is a sense too that impact in the UK is privileged, in part because of challenges in developing an appropriate evidence base in some geographical regions overseas and because of the very long lead times often involved in relationship building in the Global South. Contributions with global reach should be encouraged and receive due merit.
9. Ensuring the REF reporting on impact ties more easily with RCUK reporting could save time in compiling evidence for impact case studies.
10. Contexts of research: The environment statement is the most problematic element of assessment. This is where metrics could be utilised further in the assessment process (e.g. PG support, numbers of PhD students, grants, other minor volume indicators, etc), alongside a clear narrative that links to a unit’s strategy. However, care needs to be directed to mitigate any unintended consequences for smaller units, less-capital intensive sub-disciplinary areas (e.g. in geography secondary data analysis) and for compositional effects (i.e. departments which don’t have certain elements of resource/capital intensive sub-disciplines e.g. some parts of physical geography).
11. Combining the environment and impact templates is worth considering as it would prevent repetition, provide space to describe a coherent joined-up strategy and enable greater ownership by the UoA. The impact component could then focus solely on case studies.
12. Complete submissions of staff: One change that would reduce the administrative burden of the REF, and would reduce ‘gaming’ of the system, would be to move to a system of 100% return with all eligible staff within an institution being returned. This would need to be accompanied by robust regulatory requirements and codes of practice, to ensure that knock-on effects (such as wholesale conversion of staff to teaching-only contracts) were minimised.
13. Normalising subpanel grade-profiles would obviate the need for elaborate calibration procedures between sub- and main panels. We strongly urge that this is given attention.
2 If REF is mainly a tool to allocate QR at institutional level, what is the benefit of organising an exercise over as many Units of Assessment as in REF 2014, or in having returns linking outputs to particular investigators? Would there be advantages in reporting on some dimensions of the REF (e.g. impact and/or environment) at a more aggregate or institutional level?
14. Assessment in disciplinary units, and not at institutional level, is a vital component of REF. Universities do not do research - research is conceived and executed, and impact is achieved, by individuals working within sub-institutional units (groups, clusters, departments, and institutes). Whether these units are organised around disciplinary lines or around discrete topics or problems, these are the organisational engines that promote and encourage high-quality research via environment, support, and pathways to impact. Assessment must recognise these units; institutions are too coarse-grained for this type of evaluation.
15. Anonymity of individuals is integral to the system and should not be changed. Expectations (e.g. for four outputs, pro-rated if appropriate) are not unreasonable for the lifetime of an assessment period and, with the focus on excellence and not bulk productivity, this encourages efforts towards research quality, not quantity.
16. Disadvantages when joint authors are at the same institution, not different institutions, need to be addressed. The current approach for Panel C works against establishment of research teams within departments.
17. Where disciplines have been merged in units of assessment, some operational issues have arisen. Units based broadly on disciplines are an advantageous, and a fair, mechanism for research excellence assessment.
3 What use is made of the information gathered through REF in decision making and strategic planning in your organisation? What information could be more useful? Does REF information duplicate or take priority over other management information?
18. REF provides our organisation (a learned society) with useful benchmarking data to assess progress of disciplinary areas. It does not duplicate other management information as there is no other equivalent assessment nation-wide of the research quality of disciplinary areas.
19. REF is useful in making the case for the strength of disciplines in an international context.
20. REF data do not duplicate other data for management processes, and have therefore inevitably become utilised internally in university institutions. Using REF data as a (significant) component of QR distribution within an institution does reward performance, but allows institutions sufficient flexibility to distribute to some degree according to other priorities.
21. The REF data on environment have the least utility generally. This is largely duplicative or derivative of internal data, and can be out of date by the time it is published, although it allows a public realm dataset for peer referencing. At the very least, the numbers of categories of information should be shortened.
4 What data should REF collect to be of greater support to Government and research funders in driving research excellence and productivity?
22. The purpose of REF is not about producing ‘data’ per se, it is about assessing a range of ‘data’ that are available from other sources, making a considered judgement about academic research quality: about how research is represented (in outputs), how it is generated (through environment) and how it may be put to use beyond academia (as impact). If REF is itself turned into a data-producing, data-crunching and data-visualisation device, then it would become something entirely different from REF as it currently is.
23. REF could address the issue of ‘league tables’: and, rather than allowing all manner of league tables to be generated post-REF with highly variable rank orders of institutions/subjects, REF should formally sanction one way of calculating GPAs (grade point averages) from the profiles and one way of arriving at meaningful ‘power’/ ‘intensity’ tables (if, indeed, the latter are warranted). Arguably this would produce a much more useful set of unambiguous results to inform subsequent Government and research council funding decisions.
5 How might the REF be further refined or used by Government to incentivise constructive and creative behaviours such as promoting interdisciplinary research, collaboration between universities, and/or collaboration between universities and other public or private sector bodies?
24. Researchers need space and limited constraints to be creative and constructive. This needs to be recognised by REF and the assessment should not become a tool to drive particular types of behaviours at the exclusion of support for the highest quality research more generally. REF (RAE) has had an influence on the behaviours of institutions and individuals.
25. ‘Impact’ clearly is driving collaborative engagement, so too are collaborative consortial schemes for PhD training, for example. It would be possible to build into the REF explicit assessment of collaboration between universities, and between universities and public/private sector bodies and make it count.
26. REF could be expanded to include public sector bodies drawing on research, such as government research institutes like British Antarctic Survey and British Geological Survey, and assess how they are collaborating with universities. All of the burden should not be placed on universities to do this. If such interaction is perceived to be beneficial, then the issue needs to be approached from both angles – universities and other public/private sector organisations.
27. More guidance should be given to panels (robust and clear guidelines) about how to assess interdisciplinary research, to combat the widespread view among staff that such work falls outside of the disciplinary focus of REF and is thus unlikely to be recognised or rewarded. The current lack of clarity on how interdisciplinary work will be handled is a serious potential disincentive to staff, and sits at odds with increasing pressure from funders to undertake interdisciplinary research.
28. We point to the research reports, in the last three years, by Hefce, Universities and Colleges Union and the Equality and Challenge Unit on REF related issues for BME academics. These reports highlight different exclusionary behaviours which impact negatively on BME academics’ careers and are factors in the consideration for REF. These reports’ findings mirror the experiences of the RGS-IBG BME RACE Working Group members. These reports, and their recommendations, need to be considered very carefully in the context of the REF and any future developments.
6 In your view how does the REF process influence, positively or negatively, the choices of individual researchers and / or higher education institutions? What are the reasons for this and what are the effects? How do such effects of the REF compare with effects of other drivers in the system (e.g. success for individuals in international career markets, or for universities in global rankings)? What suggestions would you have to restrict gaming the system?
29. REF has been a source of tension in some subjects. In geography, for example, while many departments submit fully to the Geography, Environmental Studies and Archaeology subpanel, there is a possibility to submit physical geography to the Earth and Environmental Sciences subpanel (and gain panel B level STEM funding) and human geography to the Geography, Environmental Studies and Archaeology subpanel. It is also possible for an institution to submit a Department of Geography, including a large amount of human geography, to the Earth and Environmental Science subpanel and gain STEM funding for the entire unit. These tensions are not good for the discipline and unfair in how finance is allocated. Appropriate part-STEM funding levels for geography need to be formally put in place ahead of assessments to reduce the problem and the game playing.
30. REF creates a metronome that has too strong an influence on the cycle of recruitment of new academic staff, though the rules around impact are moderating this. The negative impacts are felt in particular by early career researchers (ECR) whose career stage is out of step with REF (the ‘shadow effect’ after a REF deadline). For both the ECR and the institutions, a steadier rate of turnover would be better than the once-every-6-years surge.
31. REF makes it difficult for academics from the Global South to enter into the UK job market as they are less likely to have the requisite journal articles in English speaking journals due to different cultures of publishing and access issues. Inevitable this means the UK is missing out on global talent and a broader global base for academic knowledge production.
32. The single biggest way to restrict ‘gaming’ in the system is to require universities to make a 100% full submission to the REF, and to avoid cliff-hanging thresholds around impact case study staff FTEs.
33. Another change to restrict gaming would be to reduce everything being dependent on who is a salaried staff member on the REF Census Day, and instead embrace the fact that REF is supposed to be an assessment of a unit’s academic research quality over an entire REF period (i.e. not just at the snapshot of one particular date late in the period). For example, if a staff member has been part of a unit for a given proportion of a REF period, then, irrespective of where they are based on the REF Census Day, they could (but need not) be included by that unit as one of its Category A staff, and an agreed number of outputs set against their name, provided that these outputs carry the unit in question as the researcher’s stated institutional affiliation.
34. An important advantage of the existing system involving peer review is that it is based on the judgement of panels as to the quality of individual submissions. Hence it is very difficult for UoAs to pre-empt the results.
7 In your view how does the REF process influence the development of academic disciplines or impact upon other areas of scholarly activity relative to other factors? What changes would create or sustain positive influences in the future?
35. REF provides an opportunity to gain reputational and financial advantage; it provides a reason to stop and assess, at the individual level, the quality and quantity of research output. This should lead us as individuals and as units to reflect on where there is room for improvement; and it provides a stimulus to recruitment in the build-up to REF.
36. Nurturing research for the future is not systematically included within REF. A better means of assessing e.g. early career research and the development of new programmes etc (other than loosely through research environment) is worthy of consideration. More forward looking rather than predominantly reflective on past/current research would be welcome.
37. REF needs to be cognisant of changing government initiatives, for example the Newton programme, and demands on researchers and collaborators particularly around internationalisation agendas and models of international knowledge production and publication. We urge this review to consider how best this be done.
8 How can the REF better address the future plans of institutions and how they will utilise QR funding obtained through the exercise?
38. Past performance is not always a guarantee of future success and provision of QR largely based upon historical performance can stifle growth in institutions with ambitious new plans for the future. An assessment of the long-term forward strategies and goals of institutions could be conducted within REF to account for this, alongside clear evidence of past performance and delivery. However, it must be stressed forward looking can become little more than speculation and it would be easy for the process to become one in which reward is given to who can tell the best story and this could well not match up at all with what actually happens. Attention to what elements are quantifiable would be important.
39. REF is one driver of institutional strategy – it is one element among many. This needs to be recognised.
9 Are there additional issues you would like to bring to the attention of the Review?
40. Double-weighted outputs are important and this form of output measurement should be retained in future REF. However, more sub-panel clarification is needed on the circumstances when double weighting might be allowable. For panels, a more clear distinction between weight and quality needs to be included for assessment of double-weighting.
41. The core of REF, if it is to remain as a diligent assessment of academic research quality, broadly trusted by the academic community, should remain largely as it is currently configured: in the hands of experts who care passionately about academic research and who have the expertise and integrity to assess research quality, the conditions needed for its delivery and its impact.