1 Background and motivation
University funding by government in Finland has been partly based on performance since the establishment in the 1990s of performance agreements, funding models and the KOTA-database for input and output statistics. The main policy goals have been to enhance efficiency, internationalisation, quality and impact of educational and research activities, gaining insights into the performance of the Finnish higher education institutions, as well as creating accountability and transparency regarding the use of public funds. Adaptation of the Norwegian model is embedded in the funding model that the Ministry of Education and Culture uses for allocating direct core funding annually to universities.
The funding model determines what performance indicators are used, and what is their share of core funding, in a fixed funding formula. The next years’ funding is calculated on the basis of three previous year’s performance outputs (average). Over the years, performance-based funding criteria have undergone several changes. In general, the share of core funding allocated on basis of indicators, and the weight of research indicators compared to education, has been increased. Publication statistics have been collected since 1994. The number of scientific publications related to the number of academic staff was introduced to the model in 2007-2012 as an indicator of research performance.
In the current model used since 2017, 41% of core funding is allocated on basis of indicators measuring performance in education, 33% is based on research performance indicators, and 28% on other policy considerations (Table 1 ). One of four research indicators is the quality-weighted count of scientific publications (share 13%). This indicator was adapted with components of the Norwegian model gradually in 2013-2015.
Table 1 Universities’ funding model 2017-2020. |
| Performance indicators | Share (%) |
|---|---|
| Education | 41 |
| · Master’s degrees | 13 |
| · Bachelors’ degrees | 6 |
| · Study credits in open university, specialization studies, studies based on cooperation and in non-degree programmes | 2 |
| · Number of students who have gained at least 55 study credits | 10 |
| · Student feedback | 3 |
| · Number of employed graduates | 2 |
| · Master’s degrees awarded to foreign nationals | 1 |
| · Student mobility to and from Finland | 2 |
| Research | 33 |
| · PhD degrees | 9 |
| · Scientific publications | 13 |
| · International teaching and research staff | 2 |
| · Competed research funding | 9 |
| Other education and science policy objectives | 28 |
| · Strategic development | 12 |
| · Field-specific funding | 9 |
| · National duties | 7 |
2 Organization of (re)design, implementation and operations
Development of a publication-based indicator for measuring research performance was discussed and planned since 2005 in several ministry working-groups and reports that investigated publication practices, information sources and international examples. In 2007, a KOTA working-group reviewed the performance-based research funding systems (PRFSs) used at the time in Australia, Flanders, Norway, Sweden and UK. Following the Norwegian model, the KOTA working-group recommended the development of a comprehensive national publication information system to improve the quality of publication data, as well as the development of a national authority list of publication channels to take quality of publications adequately into account in the funding model.
In 2009, the Finnish Council of University Rectors (now Universities Finland UNIFI) set up a working-group to develop the evaluation of quality of publications. Citation analysis, which is well established in science and medicine, was not chosen because of the inadequate coverage of the social sciences and humanities publications in the international databases (Web of Science and Scopus). Based on the Norwegian model adopted already in Denmark, also the UNIFI working-group recommended the development in Finland of a system based on a publication forum, in which peer-reviewed publication channels are identified and classified in quality levels by the experts in the field. Several advantages and disadvantages were identified with this model (Table 2 ).
Table 2 UNIFI working-group’s SWOT analysis of a publication forum-based system. |
| Strengths · promotes quality · quality defined by research community · sensitive to different publishing practices · could be established e.g. only for SSH fields · transparent indication of quality · takes into account quantity and quality · comprehensive information for all fields · dynamic system that can be updated | Weaknesses · burdensome expert-panel organisation · panels tie human resources · requires harmonization of publication data · renewal of field-classification (in Finland) · validity may differ between fields |
|---|---|
| Opportunities · current data about research quality · supports research assessments · steering and management · field and subfield specific analyses | Threats · experts privilege channels of national interest over quality in ratings · ensuring commensurability · difficult to reach consensus about ratings |
3 Data sources and indicator design
There is a clear distinction of responsibilities with regard to the design of the publication indicator and the development of its different components: Federation of Finnish Learned Societies (TSV) produces together with the research community the publication channel authority list with level ratings, CSC - IT Center for Science Ltd. collects the comprehensive national level publication data from universities, while the funding model working-groups constituted by the Ministry Education and Culture decide if and how the publication data and channel ratings are used in the funding model indicator.
TSV, an independent organisation that represents the research community via its over 250 member societies, has been entrusted with the responsibility to produce the national authority list of peer-reviewed publication channels. Funding for this and other TSV activities comes from the Ministry. In 2010, TSV appointed a steering-group for a two-year project, which established a secretariat (2 persons fulltime) and appointed 23 expert panels covering all fields specifically for the task of evaluating the outlets. Panels have around 250 members from Finnish universities and public research institutes. The first authority list was published in the beginning of 2012, and after the project ended, the Publication Forum has become a fixed part of the TSV activities.
The most notable differences between the Norwegian and the Finnish authority lists concern the number of panels, the number of quality levels, the rating of national language channels, and the interval of updates. The rating has three levels for peer-reviewed publication series and book publishers: level 3 = top, level 2 = leading and level 1 = basic. There is also a level 0 for channels not qualifying as level 1. Both foreign and national journals/series, conferences and book publishers can be admitted to level 1 if they meet the following four criteria:
a) specialised in the publication of scientific or scholarly research outcomes
b) editorial board constituted by experts
c) entire manuscripts of scientific or scholarly articles or books subject to peer review
d) registered ISSN or ISBN number
As a main rule, however, even a publication channel meeting these criteria should not be included in Level 1 if they are local (mainly used by researchers of a single research organisation) or the quality and relevance to Finnish research community is questionable (e.g. predatory journals).
The authority list contains more than 30 000 publication series and book publishers, of which levels 2 and 3 include 10% of the most widely respected international journals and book publishers in different fields (table 3). In the social sciences and humanities fields, however, level 2 has also included since 2012 three book publishers and over 20 journals and book series publishing in the national languages (Finnish and Swedish). To balance the ratings across fields, in each expert-panel the publication channels selected for level 2 can represent in total up to 15%, and in level 3 up to 5%, of the world’s publications.
The authority list of publication channels is updated regularly, new channels being evaluated for addition to level 1 every year, and levels 2 and 3 are updated every four years. Also, the panels and the steering-group are appointed for four-year periods. Publication Forum ratings are openly available at the JUFO-portal, which the members of research community can also use for suggesting new additions and changes to the ratings.
Table 3 Publication Forum authority list of publication channels in 2018. |
| Publication Forum level | Journals/series | Book publishers |
|---|---|---|
| 3 | 654 | 15 |
| 2 | 2,242 | 91 |
| 1 | 19,837 | 1,203 |
| 0 | 5,047 | 1,710 |
| All | 27,780 | 3,019 |
CSC, a company partly owned by the state of Finland (70%) and Finnish higher education institutions (30%), has developed technical solutions for the national level integration of publication metadata from the universities since 2009. Finland faced the challenge that many universities had strongly invested in their own information systems, so it was not considered feasible to replace those systems with a national CRIS (such as CRISTIN in Norway). Integrating data from diverse local information systems has involved development of the data model, the publication type and field classifications, as well as the data collection guidelines and procedures to ensure quality and comparability of the publication data. Full publication data from 2011 to the present is currently stored in the VIRTA publication information service, an advanced solution launched in 2016.
In VIRTA, publications are divided in scientific, professional and general publications, and for the scientific publications separate publication types are used for the refereed and not-refereed articles in journals, conferences and books, monographs and edited work (Table 4 ). Publications are assigned to the publication types in universities at the time of registering of publications in the local information systems, from which these classifications are transferred with publication metadata to VIRTA. VIRTA stores records also for artistic and design activities, theses, patents and innovation announcements, as well as audiovisual material and ICT applications, but these have not been taken into account in the publication indicator.
Table 4 Publication type classification in VIRTA publication information service. |
| Scientific publications |
|---|
| A1 Peer-reviewed journal article, original research |
| A2 Peer-reviewed journal article, review |
| A3 Peer-reviewed article or chapter in book |
| A4 Peer-reviewed article in conference proceedings |
| B1 Non-peer-reviewed journal article |
| B2 Non-peer-reviewed article or chapter in book |
| B3 Non-peer-reviewed article in conference proceedings |
| C1 Monograph (peer-reviewed) |
| C2 Edited book or special issue (peer-reviewed) |
| Professional publications |
| D1 Article in a trade journal |
| D2 Article in a professional book |
| D3 Article in professional conference proceedings |
| D4 Published development or research report or study |
| D5 Textbook, professional manual or guide |
| D6 Edited professional book |
| Publications for the general public |
| E1 Popularised article, newspaper article |
| E2 Popularised monograph |
| E3 Edited popularised book |
According to the data collection guidelines published by the Ministry of Education and Culture, scientific publications need to fulfil the following three conditions:
1. The publication must produce new information in relation to previous research data on the same subject.
2. The publication must be presented in a format that enables the verification of the research results and/or use of the research results in a new research, thus allowing other researchers to assess the research results and use them in their own work.
3. The publication channel for the publication specialises in publishing scientific research results, and it has an editorial staff consisting of experts in the field of science as well as a peer review practice.
A pre-publication peer-review, focusing on research quality, by independent experts in the field is required for inclusion of publications among the peer-reviewed publication types (A and C). To support the identification of peer-reviewed publications, TSV introduced in 2014 a national label for peer-reviewed publications inspired by the example of the GPRC-label in Flanders. Ten Finnish book publishers and 167 journals currently use the label to indicate articles and books that have undergone peer-review according to the label requirements.
The funding model is developed by working-groups appointed periodically by the Ministry to consider adjustments to the performance indicators and their share of the core funding. These groups have representatives from the Ministry, the higher education institutions, and the Academy of Finland (the main research funding agency). A funding model proposal is published in a report and sent out by the Ministry for consultation to a wide group of stakeholders. A decree concerning the universities’ core funding criteria is amended according to the new model approved by the Ministry.
The publication information now stored in VIRTA has been used as the data source of publication indicator since 2013. Weighting of publications according to the publication type was also introduced already in 2013, as monographs (C1) were counted with four times larger weight than articles (A1-A4, B1-B3). The Publication Forum levels have been identified for all peer-reviewed articles and monographs published in 2011 or later. In the period 2013-2014, however, the funding model did not yet use the Publication Forum levels to differentiate publications according to quality. Instead, 9% of core funding was allocated on basis of an undifferentiated count of international refereed publications and 4% based on the other publications. The former category included publication types A1-A4 and C1 with country of publication other than Finland, and the latter category included all the other publications belonging to types A, B and C.
The funding model working-group originally planned that in 2015, international refereed publications would be replaced with the count of publications in Publication Forum level 2 and 3 channels, and other publications with publications in level 1 channels. Eventually, a weighting of publications by type and channel was introduced in 2015, and later adjusted for the period 2017-2020 (Tables 5 and 6). The Finnish publication indicator differs somewhat from the Norwegian model as it assigns similar weight to peer-reviewed articles in journals, books and proceedings, and takes into account with small weight not-refereed publications intended for scholarly, professional and general audiences (publication types D and E).
Table 5 Weight of publications in the funding model 2015-2016. |
| Publication type | Level 3 | Level 2 | Level 1 | Level 0 |
|---|---|---|---|---|
| Peer-reviewed monograph (C1) | 12 | 12 | 6 | 4 |
| Peer-reviewed article in journal (A1-2) | 3 | 3 | 1.5 | 1 |
| Peer-reviewed article in book (A3) | 3 | 3 | 1.5 | 1 |
| Peer-reviewed article in proceedings (A4) | 3 | 3 | 1.5 | 1 |
| Not-peer-reviewed monographs | 0.4 | |||
| Not-peer-reviewed articles | 0.1 | |||
Table 6 Weight of publications in the funding model since 2017-2020. |
| Publication type | Level 3 | Level 2 | Level 1 | Level 0 |
|---|---|---|---|---|
| Peer-reviewed monograph (C1) | 16 | 12 | 4 | 0.4 |
| Peer-reviewed article in journal (A1-2) | 4 | 3 | 1 | 0.1 |
| Peer-reviewed article in book (A3) | 4 | 3 | 1 | 0.1 |
| Peer-reviewed article in proceedings (A4) | 4 | 3 | 1 | 0.1 |
| Peer-reviewed edited work (C2) | 4 | 3 | 1 | 0.1 |
| Not-peer-reviewed monographs | 0.4 | |||
| Not-peer-reviewed articles and edited work | 0.1 | |||
Institutional level whole-counts are used, meaning that co-publications with authors from several Finnish universities are counted more than once in the funding model. Fractionalization at institutional or author level has been discussed but not adopted in order not to discourage inter-university collaboration.
The Universities Act and the Polytechnics Act require all higher-education institutions, including 14 universities and 23 universities of applied sciences, to supply publication information to Ministry of Education and Culture. Also five university hospital districts, each consisting of several hospitals, and six public research institutes have agreed to provide their publication data to VIRTA. The publication information is openly available at the JUULI-portal for browsing and searching, and at the VIPUNEN-portal for statistics.
4 Funding implications
The funding model allocates around 1.6 billion euros annually to universities, and this core funding covers almost two-thirds of the universities’ budgets. The share of core funding allocated on basis of scientific publications has increased considerably, being 0.3% in 2007-2009, 1.7% in 2010-2012, and 13% since 2013. Since 2013, the publication indicator distributes annually over 200 million euros. In a report published in 2018 by Economic Policy Council on the university reform and financial Incentives, it was estimated that one publication point (equivalent e.g. to one peer-reviewed article in level 1 journal) is worth approximately 4200 euros to universities (Seuri & Vartiainen, 2018).
5 Other uses of the data
According to the UNIFI working-group, the development of the Publication Forum based publication indicator supported with national level publication data would serve three potential interests:
a) to provide quality indicator of publications for the universities’ funding model
b) to provide universities with comparable data for benchmarking with other universities, monitoring the publication profiles of units, and developing internal funding models
c) to increase the awareness of the researchers about quality of publication channels, thereby raising their level of ambition in publishing
The working-group also pointed out that Publication Forum could be used as information supporting research evaluation, for example in research assessment exercises the universities conduct themselves every few years, and the state of research in Finland reports produced by the Academy of Finland. Publication Forum ratings have indeed been used to supplement Web of Science based bibliometric analyses in some recent research assessment exercises.
Besides the funding model, the Ministry of Education and Culture uses the information based on the VIRTA publication data and the Publication Forum ratings for monitoring the research performance of the universities. Since 2010, the Ministry’s spending proposals submitted to the Ministry of Finance for the formulation of the state budget includes quantitative research performance targets for the universities collectively. Since 2016, targets have been set for the number of scientific level 2 and 3 publications per academic staff, as well as the international co-publications’ share of scientific level 1-3 publications. The same indicators are used by the Ministry as positioning statistics, openly available at the VIPUNEN-portal, for the performance agreements with individual universities.
The Ministry of Education and Culture has emphasized that the internal allocation of funds to faculties and departments should be based on universities’ own strategic priorities rather than replicate the funding model criteria. Nevertheless, universities are autonomous in using data and indicators in evaluation, development and management of their research activities. While the UNIFI working-group envisioned the potential use of the indicator in universities, appropriate local models and uses of the Publication Forum rating have not been defined. Already in 2012, the Publication Forum steering-group published a user-guide, in which it recommended not to use the rating in cross-field comparisons or in the evaluation of individual researchers.
In the fall 2015, TSV and UNIFI conducted a survey among university managers concerning the local use of the Publication Forum rating. The sample consisted of 10 rectors, 19 deans and 68 heads of department. Most Finnish universities make use of the channel rating for monitoring and developing their publishing activities, and in some universities the rating is used also for funding allocation to faculties and/or departments. The use of the ratings for evaluation purposes (recruitment, promotion, personal performance, bonuses) at the individual level is also attested, especially in social sciences and humanities. It is not clear, however, to what extent the rating supports, rather than supplants, qualitative expert assessment.
VIRTA publication data, including the Publication Forum ratings, has provided the source of information for bibliometric research and analyses concerning for example publications patterns and research productivity.
6 Experiences and effects
In Finland, as in the other Nordic countries using the Norwegian model, the positive and negative effects of the publication indicator have been discussed and speculated extensively. The effect of the publication indicator on research performance has been investigated in two recent reports, one evaluating the impact of the Universities Act reform in 2010 (2016), and another by Economic Policy Council assessing specifically the incentives created by the funding model (2018). In both reports, the development of Finnish universities publication activities was analyzed using the VIRTA publication data and Publication Forum levels. Both reports conclude that the research productivity has increased since 2011. Most notably, there has been a clear movement from level 0 channels toward level 1-3 channels. It seems that the publication channel rating has increased the Finnish research community’s awareness of quality of publication channels, even though the quality-weights were formally introduced to the funding model only in 2015.
The Economic Policy Council’s report also established that the average Publication Forum rating of peer-reviewed publications has increased in all universities, and more so in those universities where the average level was weaker in 2011. The differences in performance have decreased between universities, suggesting that the model has been most effective in those environments where full potential in productivity have not yet been reached. When related to the number of teaching and research personnel, it seems that the publication productivity of academic staff has increased but this has not happened at the expense of quality. But also the question is raised, to what extent the level ratings indicate increase in quality, as the selection of channels to level 2 and 3 can be influenced by local interests. The report also pointed out field specific differences in publication performance, making some fields seem more productive to universities.
Although it is regarded an advantage of the publication indicator that research community itself is entrusted the task of rating the publication channels, subjectivity of the expert-ratings is a common concern in Finland. The indicator has been developed for the purpose of providing comprehensive coverage of research outputs across fields, but equal treatment of different fields and specialties in level 2 and 3 nominations is much debated. The indicator is sensitive to the specificity of social sciences and humanities publishing culture, yet the funding models’ incentives are more often linked with decline rather than promotion of book and national language publishing. Other topics of discussion have included the effect of the publication indicator on interdisciplinarity and open access. Many of the concerns probably stem from the the publication indicator’s use for research management and evaluation purposes at universities, rather than from the indicators’ use in the universities’ funding model.