Q8: Do you consider that the KEF as outlined will fulfil its stated purposes?
To provide universities with new tools to understand, benchmark and improve their performance: Somewhat disagree
To provide businesses and other users with more information on universities: Somewhat disagree
To provide greater public visibility and accountability: Somewhat disagree
Q9. Please provide a commentary in relation to your scores above
The stated purposes of KEF suggest an intention to measure and benchmark how well universities foster knowledge sharing and research commercialisation and capture the rich network of collaborations between universities and businesses. However, the approach proposed does not capture the richness, depth and intangibility of many elements of knowledge exchange.
The perspectives and metrics proposed seem somewhat one-sided – universities gather and report quantitative data for HE-BCI survey, which forms the basis for the majority of benchmark data, but there is very little opportunity for business or other partners to share their view of the quality or nature of collaboration, or measure of satisfaction with the knowledge exchange they have collaborated in; the ‘business’ voice appears absent.
As currently proposed, KEF may also be adding little to REF (and in some cases double-measuring). REF is already bringing impact to the fore as universities have improved both capacity and expertise in generating impact from research.
If a future link to HEIF funding allocation is intended (in terms of fulfilling the greater public visibility and accountability purpose), we would wish to know now in order that the proposed approach could be more appropriately designed to that purpose.
Q10. The KEF consultation document describes the overall approach as being an annual, institutional level, largely metrics driven exercise, although noting that narrative will have an important role. More background may be found in the report summarising the recommendations of the technical advisory group. Do you consider this overall approach to be appropriate?
Somewhat disagree.
Q11. Please provide a commentary in relation to your scores above
Overall, KEF seems to take a narrow definition of knowledge exchange based on what metrics are widely available, rather than defining knowledge exchange broadly and seeking an assessment approach that measures attributes of that broader definition. In addition, many metrics are income-based, which has the potential to imply that monetised knowledge exchange is more valuable than non-monetised forms.
Operating a largely metrics-driven approach at the institutional level lacks sufficient granularity for it to be useful to business or other partners in understanding or assessing potential collaboration opportunities with that institution when the relationships they form are more likely to be at faculty/school/department or with specific academic/s.
We are broadly concerned about the limitations of using snapshot metrics for capturing the impact and knowledge exchange embedded in research that engages stakeholders across the whole lifecycle, which might cross multiple annual data checkpoints with natural fluctuations according to research or knowledge exchange intensity. Not addressing this risks deprioritising the importance of the 'long slow burn' in forming valuable, mutually beneficial knowledge exchange relationships, especially since there can be significant variation between subjects/disciplines or between forms of knowledge exchange in terms of the time and nature of resource it takes to develop those relationships.
Q12. Please indicate your degree of support for the following aspects of our clustering approach
The conceptual framework that underpins the cluster analysis: Somewhat disagree
The variables and methods employed in undertaking the cluster analysis: Somewhat disagree
The resulting make up of the clusters, i.e. the membership: Somewhat disagree
That the overall approach to clustering helps Research England to meet the stated purposes of the KEF and ensures fair comparison: Somewhat disagree
Q13. Please provide commentary on any aspect of your scores above. If relevant please incorporate suggestions for alternative arrangements.
The description of clustering seems to suggest that the clusters have not been fully benchmarked for geographical location and the relative opportunity presented by the local economy. The KEF Cluster Analysis Report showed that clusters might be split by low and high LEP Competitiveness scores, and noted that “It is possible that the KE opportunity potential even within a cluster may thus be different, with those in less competitive LEPs facing different opportunities and challenges to engagement locally than those in more competitive areas.”, but it is unclear whether this will be addressed in benchmarking. We believe that these differences (which may be considerable within some clusters) may be further exacerbated by institutional mission or subject specialisms in relation to the opportunities presented by the local economy.
We suggest that attention be paid to relative opportunity, notably the availability of partners and opportunities across both private and not-for-profit sectors, in order that benchmarking for clusters with similar institutional features are not unduly impacted by their geographical location and significant disparity in external opportunity landscapes.
Perspectives / Research partnerships / Working with business / Working with the public and third sector / Skills, enterprise and entrepreneurship / Local growth and regeneration / IP and commercialisation / Public and community engagement
Q16. Taking into account the overall range of perspectives and metrics outlined in the consultation document, do you agree or disagree that a sufficiently broad range of KE activities is captured?
Somewhat disagree
HE-BCI metrics look to be capable of measuring some of the capacity for knowledge exchange, but not effectiveness over time. Income is an input to a transformative knowledge exchange process. But lots of income poorly spent results in poor knowledge exchange and impact; there appear to be no measures for return on investment. There also appears to be no ability for business or other partners to share their view of the quality or nature of collaboration, or their satisfaction with the knowledge exchange they have collaborated in, which might be a useful alternative or addition. For geography, we especially wish to emphasise the importance and frequency of non-monetised interactions and knowledge exchange that risk being overlooked by KEF, particularly around community and public engagement, partnerships with businesses that do not result in financial transactions (co-location of facilities, student placements), student activities (work experience & internships, employer sponsorship of, or delivery of content for, courses) and the human interaction aspects of forming effective working relationships. While we acknowledge the role of the narrative in documenting some of these, the proposed length of the narrative is insufficient to allow institutions to fully explore this across the breadth of their knowledge exchange activities. We understand that HE-BCI metrics are being reviewed by HESA and other stakeholders and look forward to the outcome of that review in terms of understanding whether they remain fit for purpose in this context.
Q17. Taking into account the range of metrics outlined in the consultation document, please indicate whether you consider that they adequately represent performance in each of the proposed perspectives.
Research partnerships: 50% support
Working with business: 50% support
Working with the public and third sector: 50% support
Skills, enterprise and entrepreneurship: 50% support
Local growth and regeneration: 50% support
IP and commercialisation: 50% support
Public and community engagement: 50% support
Q18. Research partnerships - Taking into account the range of metrics outlined in the consultation document for this perspective, please provided any comments on the balance and coverage of the proposed metrics.
In response to Para 31 of the KEF Consultation, we query whether co-authorship with non-academic partners is a fair metric, on the basis of:
-
The difficulty of defining “non-academic”; by organisational affiliation is not sufficient, nor by qualification. Researchers are found in a wide variety of organisations, with increasing permeability between academia and other forms of [research] organisation, and a rise in “pracademics”.
-
Co-authorship with non-academics is not universally prioritised, possible or available across subjects/disciplines. Institutions with subject clusters in areas that do not prioritise co-authorship may be disadvantaged.
-
Not all knowledge exchange partners wish to publish. Institutions that engage with certain types of knowledge exchange partners may be disadvantaged (or advantaged).
Q19. Working with business - Taking into account the range of metrics outlined in the consultation document for this perspective, please provided any comments on the balance and coverage of the proposed metrics.
Building the relationships required for knowledge exchange to take place can be time-consuming (and costly) to develop, and the process of forming those relationships may form knowledge exchange in and of itself. Such relationships are based on, and to some extent might be valued on the basis of, the degree of trust and familiarity developed between partners.
There appears to be no ability for business (broadly defined across private, public and not-for-profit sectors) partners to share their view of the quality or nature of collaboration, or their satisfaction with the knowledge exchange they have collaborated in, which might be a useful alternative or addition.
Q20. Working with the public and third sector - Taking into account the range of metrics outlined in the consultation document for this perspective, please provided any comments on the balance and coverage of the proposed metrics.
Many knowledge exchange activities with public and third sector can’t easily be quantified (notably public engagement and cultural impact) while others are difficult to trace at all, including those deeply embedded within individual researchers’ methodologies. The proposal to link this metric solely to income (via HE-BCI Table 1b and Table 2) limits capturing the breadth and depth of how universities engage with these important partners.
Building the relationships required for knowledge exchange to take place can be time-consuming (and costly) to develop, and the process of forming those relationships may form knowledge exchange in and of itself. Such relationships are based on, and to some extent might be valued on the basis of, the degree of trust and familiarity developed between partners.
There appears to be no ability for business (broadly defined across private, public and not-for-profit sectors) partners to share their view of the quality or nature of collaboration, or their satisfaction with the knowledge exchange they have collaborated in, which might be a useful alternative or addition.
Q22. Local growth and regeneration
Taking into account the range of metrics outlined in the consultation document for this perspective, please provided any comments on the balance and coverage of the proposed metrics. (400 word limit) Note there is a separate question to consider the use of supplementary narrative.
The description of clustering seems to suggest that the clusters have not been fully benchmarked for geographical location and the relative opportunity presented by the local economy. The KEF Cluster Analysis Report showed that clusters might be split by low and high LEP Competitiveness scores, and noted that “It is possible that the KE opportunity potential even within a cluster may thus be different, with those in less competitive LEPs facing different opportunities and challenges to engagement locally than those in more competitive areas.”. We believe that these differences (which may be considerable within some clusters) may be further exacerbated by institutional mission or subject specialisms in relation to the opportunities for local growth and regeneration presented by the local economy.
We suggest that attention be paid to relative opportunity for local growth and regeneration, in order that benchmarking for clusters with similar institutional features are not unduly impacted by their geographical location and significant disparity in external opportunity landscapes.
Q24. Public and community engagement
Taking into account the range of metrics outlined in the consultation document for this perspective, please provided any comments on the balance and coverage of the proposed metrics.
We are concerned that benchmarking perspective seven “Public and community engagement” will be difficult. Narratives will be central to this [see our comments later], but time per FTE academic staff committed to external-facing events cannot be suitably benchmarked across engagement of different types or intensities (numbers of attendees would pose a similar challenge, and we don’t recommend that either). This appears to be a categorisation error - quantifying how much time spent on a public engagement activity is not a sufficient proxy for the quality of, and impact of, that public engagement.
An NCCPE consultation event regarding REF impact and public engagement highlighted a number of barriers to public engagement/knowledge exchange as part of REF, which included:
-
Issues of resources (time, funding, skills etc.)
-
Structural issues (e.g. lack of strategic investment in areas like reward and recognition or long term investment in capacity building)
-
Issues of language/definitions
-
The challenge of evidencing impacts from PE
-
(Perception) PE not valued/seen as risky
-
The need for additional guidance/examples
Metricising knowledge exchange using the methodology proposed in the consultation, even with a supporting narrative, does not recognise or incentivise ‘risky’, ‘difficult’ or ‘poorly resourced’ forms of engagement, which are still crucial to success of research and towards universities in achieving their mission.
Q25. Do you consider it appropriate for HEIs to provide narrative text to support the metrics in perspectives that don't currently have fully developed metrics?
Somewhat agree
Q26. Public and community engagement narrative
Overall, is the guidance on the provision of narrative text for this perspective clear
Somewhat disagree
Q27. Please comment on the proposal to include narrative from HEIs for the public and community engagement perspective, in particular: - where further clarification is requiredwhere refinements could be made- whether there are areas where more consistency across HEIs could be achieved.
We are concerned by the intention to present a 'provisional' metric score alongside the narrative. We consider the metric score to be an imperfect and incomplete view of the perspective and would suggest that the narrative would be more fully representative of the perspective as a whole.
For geography, non-monetised interactions and knowledge exchange that would fall under "Community and Public Engagement" risk being overlooked by KEF. While we acknowledge the role of the narrative in documenting and highlighting some of these, the proposed length of the narrative is insufficient to allow institutions to fully explore this across the breadth of all their knowledge exchange activities.
Q28. Local growth and regeneration narrative
Overall, is the guidance on the provision of narrative text for this perspective clear?
Somewhat disagree
Q30. The role of further narrative or contextual information We welcome responses on what other types of narrative or contextual information would be helpful. You may wish to consider, for example: Should the HEI or Research England provide other narrative information? How should we use other contextual information, such as information on local economic competitiveness described in section 5 of the cluster analysis report? Would other perspectives benefit significantly from further narrative information? Would the benefit of adding further narrative information be outweighed by the burden of doing so?
Overarching institutional statement - provided by the HEI: Agree
Overarching institutional statement - provided by Research England: No opinion
We are concerned to read in Para 44 that the Narratives are only intended to specifically address the perspectives of ‘Public and community engagement’ and ‘Local growth and regeneration’ and not be allowed to contextualise metrics in other areas more broadly. We encourage the inclusion of an overarching institutional narrative that sets out mission and vision, and provides space for a more integrated understanding of the approach to knowledge exchange, including clarification or contextualisation for metrics across all perspectives. We encourage the re-use of REF impact narratives as evidence for KEF, to more fully and consistently present quantified/qualified impact and end-to-end engagement across a specific area of research. We are concerned by the intention to present a 'provisional' metric score alongside the narratives for the two perspectives that allow this. We consider the metric score to be an imperfect and incomplete view of the perspective, potentially misleading in the context of the visualisation proposed, and would suggest that the narrative alone would be more fully representative of the perspective as a whole. We are concerned that Annexes D and E of the KEF Consultation seem to indicate that the eventual goal is to remove narrative statements entirely once appropriate metrics are developed, which would reduce the ability of institutions to describe areas of work that are more hidden or not quantified by metrics, which remain considerable.
Q31. Visualisation
Please indicate your level of support for the proposed method of comparison and visualisation.
Each of the seven perspectives is to be given equal weighting: 75% support
Metrics under each perspective are to be normalised and summed: 50% support
The performance of each HEI is to be expressed in a radar chart in deciles, relative to the mean average decile of the peer group: 75% support
Perspectives are not intended to be aggregated into a single score: 75% support
Narratives are to be presented alongside the metric score, making it clear that metrics in the two perspectives of public & community engagement and local growth & regeneration are provisional, and should be read in conjunction with the narratives: 75% support
Visualisation is to be delivered through an interactive, online dashboard which will allow exploration of the data underlying the ‘headline’ results in various ways: 50% support
Q32. Please comment on the presentation and visualisation proposals, for example:- where further clarification is required- where refinements could be made- whether there are areas where more consistency across HEIs could be achieved- how narratives could be incorporated?
We are concerned by the intention to present a 'provisional' metric score alongside narratives for the two perspectives that allow this. We consider the metric score to be an imperfect and incomplete view of the perspective, potentially misleading in the context of the visualisation proposed, and would suggest that the narrative alone (perhaps with metric benchmark data embedded within the narrative itself) would be more representative of the perspective, under the proposed methodology.
With reference to Para 45 of the KEF Consultation, we ask whether disaggregated data [benchmark or otherwise] will be available for download, or whether it will only be exclusively available within the visualisation dashboard, as this will impact its usefulness for further analysis and other purposes.
Q34. If you have any other comments, please share them here
As a learned society, professional body and charity, we are a pathway to impact and knowledge exchange partner for many researchers and universities in terms of: public and community engagement; informing policy and practice; fostering collaboration with business, government and not-for-profit sectors; providing support for student learning; delivering events and conferences; and many other forms of knowledge exchange. We would welcome further discussion on this perspective.