Performance-based research funding systems (PRFSs) have been installed in several countries around the globe, many of which are European (Debackere et al., 2018; Zacharewicz et al., 2018). Hicks (2012) has characterized PRFSs as “national systems of research output evaluation used to distribute research funding to universities”. The Flemish Government introduced the BOF-key (BOF stands for “Bijzonder Onderzoeksfonds” or “University Research Fund” in English), a mechanism to distribute research funding between the Flemish universities, in 1994 (Spruyt & Engels, 2013). Originally the BOF-key was based primarily on input indicators. A major change occurred in 2003 with the addition of bibliometric parameters based on publication and citation data from the Science Citation Index Expanded (SCIE) of Web of Science (WoS) to the system (Debackere & Glänzel, 2004). From then onwards, the BOF-key became a PRFS. This change, however, triggered strong criticism among the scholarly community in Flanders (De Wever, 2007), as the nature of the SCIE caused scholarly publications in the social sciences and humanities (SSH) to be of negligible importance in the BOF-key (Verleysen, Ghesquière, & Engels, 2014). This spurred on two further reforms in 2008, namely (1) the inclusion in the bibliometric parameters of publications and citations indexed in the Social Sciences Citation Index (SSCI), the Arts and Humanities Citation Index (AHCI) and the WoS conference proceedings databases, and (2) the enactment of the legal framework for a comprehensive coverage database of scholarly peer reviewed publications in the SSH, the Flemish Academic Bibliographic Database for the Social Sciences and Humanities (VABB-SHW; Engels et al., 2009). The data collected in the VABB-SHW resulted in the addition of a new bibliometric parameter to the BOF-key as of 2011 (Engels, Ossenblok, & Spruyt, 2012). After the first 5-yearly external evaluation of the VABB-SHW (Zuijdam et al., 2013), the government decided to gradually increase the weight of the bibliometric parameters in the BOF-key to 16.6% for WoS-indexed publications, 6.8% for VABB-SHW indexed publications and 16.6% for WoS-indexed citations of WoS-indexed publications as of 2016 (Spruyt & Engels, 2013). Since 2016, the BOF-key consists of the following parameters:

- Part A: 60%

o Bachelor and master degrees: 23%

o Defended PhDs: 35%

o Diversity: 2%

- Part B: 40%

o Publications in WoS: 16.6%

o Publications in VABB-SHW: 6.8%

o Citations in WoS: 16.6%

Overall, the Flemish PRFS has been well documented, e.g. in the aforementioned publications. The introduction of bibliometric parameters in the BOF-key followed a tradition of studying the possibilities, limitations and implications of the use of bibliometrics as a science policy tool (e.g. Luwel, Noyons, & Moed, 1999; Moed et al., 1998; Van Den Berghe et al., 1998) including in the humanities (Billiet et al., 2004; Moed, Luwel, & Nederhof, 2002) and in law (Luwel et al., 1999). Recently Verleysen and Rousseau (2017) have also discussed to what extent the bibliometric parameters of the BOF-key can help evaluators to act according to the principles
of the Leiden Manifesto. They conclude that, although further improvements are possible, the changes that have been made over the last decade have resulted in ever greater compliance with the 10 principles of the Leiden Manifesto. Complementing the aforementioned references, the focus of this contribution is on the history of the Flemish PRFS, as well as the inter-institutional organizational setup and processes that underpin the bibliometric parameters of the Flemish PRFS.

The Flemish universities played a significant role in the development of the Flemish PRFS since the early 1990s. Initially the Flemish government encouraged the five Flemish universities (i.e. Antwerp University, Ghent University, Hasselt University, KU Leuven, and Vrije Universiteit Brussel) to experiment with research evaluation underpinned by bibliometrics. In 2001 the government decided to fund a group of researchers at KU Leuven with, among other things, the task to develop and implement a bibliometric indicator for inclusion in the BOF-key. In 2007 all Flemish universities became part of this consortium, which became known as the Flemish Center for R&D Monitoring (ECOOM) from 2009 onwards (te Velde
et al., 2018). ECOOM is financed through a five year consortium agreement with the government, stipulating among other things the expectations with regard to the delivery of the bibliometric indicators for the BOF-key. The ECOOM-Leuven bibliometrics group headed by Wolfgang Glänzel is responsible for the WoS-based bibliometric indicators, whereas our own ECOOM-Antwerp group has the task to annually deliver the VABB-SHW. For more information on the tasks of the other ECOOM groups we refer the reader to www.ecoom.be. All members of ECOOM report to its steering committee, which is chaired by a representative of the Flemish Minister for Innovation. At the Flemish level, several working groups operate on behalf of ECOOM, including the interuniversity VABB-SHW working group chaired by Koenraad Debackere, who is ECOOM’s principal investigator.

The BOF-regulation enacted by the Flemish government stipulates the criteria and rules according to which the BOF-key, including its bibliometric parameters, need to be calculated. In addition, the government has established the GP (“Gezaghebbende Panel” or Authoritative Panel), which is composed of 18 professors affiliated with Flemish universities and whose expertise covers the main SSH disciplines. In view of the annual new version of the VABB-SHW, the GP is entrusted by the government with the task of selecting the publication channels which comply with the criteria set in BOF-regulation (Verleysen, Ghesquière, & Engels, 2014). In addition, the GP can apply additional criteria and propose changes to the BOF-regulation to the government. The members of the GP are appointed by the government and hence as a group report to the government only. Administrative support to the GP is provided by the VLIR (“Vlaamse Interuniversitaire Raad” or Flemish Rectors Conference), whereas ECOOM-Antwerp provides technical support. This technical support consist of, among other things, providing the GP with overviews (both overall and per discipline) of publication channels that need to be decided upon, as well as bibliometric analyses that can inform their decision-making processes (e.g. in relation to predatory publishing).

For the WoS-based bibliometric indicators of the BOF-key the government decided not to install any specific working groups or panels. However, in view of the consecutive changes to the BOF-key and the WoS-publications parameter in particular, the research policy working group of the VLIR has taken up an active role in discussing ideas, limitations and possibilities in order to arrive at common proposals to the government (Spruyt & Engels, 2013). Indeed, the government has also repeatedly encouraged the universities to come forward with consensus suggestions to improve the BOF-key and the BOF-regulation. As such, the BOF-key should not be seen as something forced on the universities by the government. Instead, the discussions of the BOF-key over the years have allowed the universities to develop considerable expertise with regard to PRFS, leading to full recognition of pros and cons. It therefore does not come as a surprise that a recent evaluation of the BOF-key and BOF-regulation resulted in only minor suggestions for change. Moreover, the international panel of experts observed that the year-to-year changes in the shares per university are rather small (Korlaar, Bongers, & Groot Beumer, 2018).

The data sources for the bibliometric indicators of the Flemish BOF-key are the WoS and the VABB-SHW. For all three bibliometric indicators, a 10 year time window is used.

Initially in 2003 only the SCIE database of WoS was taken into account in order to calculate the number of publications per university as well as the number of citations of each of these papers (Debackere & Glänzel, 2004). No weighting of publications or citations was applied. Only the publication types article, letter, note and review were considered, and each publication (and its citations) was wholly counted for each university that had contributed to it (according to the addresses mentioned on the published paper).

Already in 2004 a working group of VLIR discussed the possibility of weighting publications according to the Journal Impact Factor (JIF) of the journals in which they appeared, the possibility to weight citations, as well as the possibility to better take stock of publications in the social sciences and humanities (Debackere & Glänzel, 2004). These discussions resulted in the aforementioned expansion of data sources as of 2008 (including the addition of the publication type proceedings paper) and the introduction of weighting of publications according to the JIF of the journal in which they appeared (Spruyt & Engels, 2013). This weighting occurred for 50% according to the field of publication using the Leuven-Budapest classification scheme (a classification of science into 16 fields and 68 subfields; see Glänzel & Schubert, 2003), and for 50% across all fields.

Soon after its implementation this weighting scheme was criticized because the difference in resulting weight attached to publications reached up to a factor of 4000, with an overrepresentation of the biomedical sciences among the publications with the highest weights (Vandevelde et al., 2009). Hence the weighting of WoS-indexed publications changed again in 2013: from 2013 onwards, these publications are weighted according to a binning of JIFs per research subfield (the 68 subfields as distinguished in the aforementioned Leuven-Budapest scheme) into twenty equally sized percentile classes. For the final weighting, each publication is weighted according to the best weight that a journal receives in any of the subfields to which it has been assigned.

Although this weighting still results in differences up to a factor 100, it has proven less controversial, presumably because every year around half of the publications are assigned to the top weights of 10, 6, 3 or 2. Thus, the weighting seems in line with the focus of many researchers on publishing in prestigious journals. Yet this weighting of WoS-indexed publications has also been criticized for being in conflict with several of the principles of the Leiden Manifesto (Verleysen & Rousseau, 2017). The use of the JIF is in itself controversial, yet criticisms such as sensitivity to outliers are less applicable, since the weighting scheme does not directly use the JIF value and takes a 10 year time window into account. Another point of criticism concerns the absence of a weighting of citations, a feature that has been repeatedly discussed (Spruyt & Engels, 2013). This seems problematic in the light of the principles of the Leiden Manifesto, in particular principle 6, which advocates to account for variations by field in publication and citation practices (Verleysen & Rousseau, 2017).

The VABB-SHW is a comprehensive database of peer-reviewed research output written by authors who are affiliated to a Flemish university department from the social sciences and humanities (SSH). As such, it contains both publications that are indexed in the Web of Science (SCIE, SSCI, AHCI, conference proceedings databases) and publications that are not. Publications of the latter type are considered in the VABB-SHW parameter of the BOF-key, while publications of the former type are only considered in the WoS publications parameter. This way it is ensured that no publication can be counted twice in different parameters. In recognition of the wider variety of publication types used in the SSH, the following five publication types are distinguished in the VABB-SHW: journal articles, books as authors (monographs), books as editor, book chapters, and articles in proceedings. VABB-SHW publications are weighted in the BOF-key according to a straightforward scheme based on publication type. While journal articles, books as editor and book chapters each have a weight of 1, books as author carry a weight of 4 and articles in proceedings have a weight of 0.5. Contrary to the original Norwegian model, there is no differentiation in terms of quality levels.

According to the BOF-regulation, a publication that belongs to one of these types and is authored by a researcher affiliated to an SSH unit of a Flemish university can be included in the VABB-SHW, on the provision that four criteria are met. The publication must:

a) be publicly accessible;

b) be unambiguously identifiable by an ISSN and/or ISBN;

c) contribute to the development of new insights or the application thereof;

d) be peer-reviewed by independent experts in the field prior to publication. Peer review must be carried out by an editorial board, a reading committee, external referees, or a combination of these. Peer review must primarily be external to the own research group and independent of the author(s). Peer review is not organized by the author themselves.

The VABB-SHW is updated on a yearly basis: each year, the five Flemish universities send the metadata of SSH publications of the previous two years to ECOOM-Antwerp. ECOOM-Antwerp processes, de-duplicates, and stores the data. From these data, several lists of publication channels (in which Flemish SSH scholars have published in the 10-year time window) are compiled for the GP; the most important ones are the lists of journals, of publishers, and of book series.

The GP’s main task is to decide which publication channels conform to the abovementioned criteria. Especially the identification of peer-reviewed publications and publication channels constitutes the bulk of the GP’s work in this regard. In addition, the GP can install extra regulations and criteria. The most important example of this is the addition of an extra criterion from the first version of the VABB-SHW onwards: a publication must have at least four pages. The rationale behind this is to exclude the myriad of editorials, opinion pieces etc. that are often registered as regular articles but typically count less than four pages. Furthermore, the GP can propose changes to the government. In 2012, this has led to changing the weight of books as editor from 2 to 1.

Several changes have occurred to the workflow since the first version of the VABB-SHW.

In 2010 the Flemish Publishers Association installed the Guaranteed Peer Reviewed Content (GPRC) label, a quality label stating that a book has been peer-reviewed according to international standards (Verleysen & Engels, 2013). The GP works together with the publishers for the inclusion of GPRC-labelled books in the VABB-SHW.

From the second version of the VABB-SHW onwards the GP has decided to allow inclusion of individual peer-reviewed books that have not been published by a publisher exclusively working with peer review. From 2013 onwards, the GP has been working with a selection at the level of book series, in addition to the selection at the level of publishers and individual books. Since book series are in many ways akin to journals, this “intermediate” level lends itself well to making decisions on peer review status. A book series can be included if the GP has established that it has an academic editorial board and/or series editor, and employs peer review.

In 2013 the GP asked ECOOM-Antwerp to screen the VABB-SHW for the occurrence of so-called predatory open access publications. This has grown into a yearly exercise, on the basis of which over 200 journal articles submitted to the VABB-SHW have been identified as predatory open access and excluded from the VABB-SHW (Eykens et al., 2018).

In 2017 the GP decided to treat all journals that are indexed in the WoS Emerging Sources Citation Index (ESCI) as peer-reviewed; hence, publications that appear in these journals can be counted for the VABB-SHW part of the BOF-key (if they adhere to all criteria). Note that journals in this index are not counted in the WoS part of the BOF-key, and citations to these publications are not taken into account.

The BOF-money as distributed through the BOF-key is essential for each of the Flemish universities in view of their research mission. In 2017 the BOF-money amounted to 170.8 million Euro, representing 8.9% of the total turnover of the Flemish universities (Korlaar et al., 2018). As such, the BOF-money has a structural impact on the research capacity of each of the Flemish universities. However, the direct funding implications of the bibliometric parameters, which together amount to 40% of the BOF-key, have been limited in recent years. One reason is that the BOF-regulation guarantees a minimum share for the three small and medium sized universities (SMUs) in Flanders, because of the importance of a regionally diverse university landscape. At the same time, this has may have led to increased competition between the two largest universities. Another reason is that, since the increase of the weight of the VABB-SHW parameter in 2013, the relative weight of each of the bibliometric parameters has been stable. Hence after the initial substantial impact of the addition of the VABB-SHW (Zuijdam et al., 2013), the BOF-key has stabilized since 2013.

The data collected for calculation of the BOF-key are also used for monitoring purposes. In particular, the annual Flemish Indicator Book contains sections on publication output and impact (Debackere, Glänzel, & Thijs, 2017; Guns & Engels, 2017). The extent to which the universities use the data for internal allocation or evaluation purposes is unknown. Already upon the introduction of the bibliometric parameters in the BOF-key, Debackere and Glänzel (2004) warned against the intra-institutional use of the data in view of allocation and evaluation. Nevertheless, successive evaluation panels have observed that such use does occur (Korlaar et al., 2018; Zuijdam et al., 2013), notwithstanding the clear opposition of the GP against such use. The data are also used as a benchmark when researchers apply to become panel members at the Research Foundation Flanders (FWO, Fonds Wetenschappelijk Onderzoek Vlaanderen). In almost all cases, however, such use of the data is informing rather than replacing peer review.

The data collected for the VABB-SHW have also been used to study publication patterns in the SSH. Engels, Ossenblok and Spruyt (2012) studied changes in publication patterns in the SSH using VABB-SHW data. Among other results, they find that the number and share of English-language publications is increasing, to the detriment of Dutch (the local language) and other languages. The use of different publication types differs from discipline to discipline; on the whole, the share of book publications in the humanities is higher and, contrary to the social sciences, not decreasing. In a separate study (Ossenblok, Verleysen, & Engels, 2014), it was found that patterns of coauthorship can also be very different among disciplines. Follow-up studies have focused on the role of editors of edited books in collaboration and the prevalence of alphabetical publishing (Guns, 2016; Ossenblok & Engels, 2015). The VABB-SHW has also allowed for several studies of the characteristics of author types that have hitherto remained largely unstudied, such as monograph authors or book editors (Ossenblok, Guns, & Thelwall, 2015; Verleysen & Ossenblok, 2017). In two studies, Verleysen and Weeren (2016a, b) clustered senior authors in the VABB-SHW by their publication patterns. The results show that most disciplines are heterogeneous in terms of publication patterns of the authors. Given the importance of book publications in the SSH, several studies have been undertaken that investigate the characteristics of book publications and publishers: internationalization of book publishing (Verleysen & Engels, 2014a, 2014b), the role of book publications in History (Verleysen & Engels, 2012), and concentration of academic book publishers (Guns, 2018). Finally, given the prevalence of similar national databases across Europe (Sīle et al,, 2017; Sīle et al., 2018), many possibilities have opened in the last few years for international comparison (Engels et al., 2018; Giménez-Toledo et al., 2016; Kulczycki, Engels, & Nowotniak, 2017; Kulczycki et al., 2018; Ossenblok, Engels, & Sivertsen, 2012; Pölönen et al., 2018; Pölönen et al., 2017; Pölönen et al., 2017). ECOOM-Antwerp has also setup an online overview of national bibliographic databases for the SSH at https://ecoom.uantwerpen.be/sshdatabases.

While the Flemish PRFS has developed independently, some of the most substantial changes, especially those relating to the VABB-SHW, have been strongly inspired by the Norwegian model. Specifically, the comprehensive coverage of
the VABB-SHW database as well as the different weights assigned to different publication types in the VABB-SHW are similar to the Norwegian system. What is different from the Norwegian model, however, is the split of publications over two databases—Web of Science and VABB-SHW—as well as the use of WoS citations as a parameter. Another difference is that publication channels in the VABB-SHW are not weighted by quality level: either a channel is included or it is not. Journal articles that are indexed in WoS are weighted by the JIF rather than a predefined set of quality levels established by panels. Furthermore, the Flemish PRFS uses whole counting instead of the fractional counting one encounters in the original Norwegian model (Sivertsen, 2016).

It is well understood that PRFSs can have an effect on the publishing characteristics in a country (Hicks, 2013). For the case of Flanders, this has been empirically studied by Guns and Engels (2016). While causal effects are impossible to establish, they find that a strong emphasis on WoS-indexed publications since 2003 has been accompanied by a growth in WoS publications that is greater than what can be observed in other countries. However, during the same period the number of researchers in Flanders also increased rapidly (Delanote et al., 2017). Remarkably, the introduction of the VABB-SHW has not led to slower growth of WoS publications in the SSH. Publication types that only count in the VABB-SHW parameter of the BOF-key appear to follow a different mechanism; especially the growth in yearly number of book chapters is comparable to that of WoS articles.

One particularly interesting effect to study is that of the whole counting of publications per university. Debackere and Glänzel (2004) warned that any systematic collaboration between institutions could distort the bibliometric parameters of the BOF-key and hence make it unacceptable. Sivertsen (2017) also suggested that counting methods should incentivize collaboration without stimulating the inclusion of authors with minimal contributions. Indeed, the difference in counting method between the Norwegian model as implemented in Norway and the counting method in Flanders is striking, calling for a systematic analysis of the possible differential effects of both.

The overall effects of the Flemish PRFS range much wider, however.

Since 2004 the bibliometric parameters discussed in this article are also used in another interuniversity allocation scheme, namely the IOF-key (Industrial Research Fund). The IOF-money, which amounted to 32 million in 2018 and is thus much less than the BOF-money, stimulates the universities to develop new technologies and prepare them for use in the market place. In addition to bibliometric indicators, the IOF-key also takes into account PhDs defended, industrial research funding and EU-framework program funding, license income, patents, and spin-off creation. As such, the BOF-key became a gateway for the introduction of performance-based funding also in the area of valorization and development.

In 2008, the government went one step further by taking the bibliometric parameters of the BOF-key also into account in the calculation of the government block grant. This expansion of the reach of the bibliometric indicators was not without controversy and still regularly causes debate. One often voiced concern is that the balance between the education and the research mission of the university has tilted too much towards the research aspect. Inversely, one could also argue that more emphasis on research was needed, at least at the time when the bibliometric parameters were introduced into the BOF-key. It seems impossible, however, to pinpoint such arguments in one or the other direction.

There seems to be a broad consensus, however, that the research performed at the Flemish universities is of high quality, and that this is partially thanks to a close monitoring of research activity and impact. Indeed, the datasets that have been set up in view of the bibliometric parameters of the BOF-key have become references for many other processes at the level of institutions and the government. Still, the conceptual definition and delimitation of research publications remains challenging given the constant changes in scholarly communication practices. In the current era of digital and open science, these definitions and delimitations will also need to co-evolve with the broader evolutions of the ways in which research is taking place.

Among researchers such overall benefits of the PRFS are not generally shared. One argument is that the PRFS has led to undue pressure to publish. This point of view is, for instance, regularly expressed in opinion pieces in the Flemish press, yet it is hard to directly ascertain to what extent such a trickle-down effect takes place. A first consideration is that almost all researchers operate in the international context of their discipline. The need and wish to contribute to the development of the field generates pressure for all researchers who are at the forefront of their field. The degree to which the epistemic cultures of the researchers are in line with the workings of the PRFS may differ from specialism to specialism, thus causing inconsistency of expectations more for some than for others. The weighting of WoS-publications, for example, may be more in line with the hierarchical structure of the STEM journal landscape than with the publishing landscape of the SSH (Bonaccorsi, 2018). The perceived unintended consequences of the PRFS have been stressed in particular in relation to violations of research integrity: if too much pressure is put on researchers, they might lose their integrity. Although this is a general and international concern, some have pinpointed blame in particular on the Flemish PRFS (Cornelis, 2013). To the best of our knowledge, no evidence for this assertion has been produced, and cases of infringement of research integrity have not been exceptionally common in Flanders.

Still, the PRFS might be one of several elements in the evolution of the university into an environment that exerts many short term expectations on researchers. A survey among PhD students in Flanders found that many of them suffer undue levels of stress (Levecque et al., 2017). Also among postdocs and professors, high levels of stress and chronic stress seem to be common. One reason might be that the PRFS is part of a much broader evolution towards more measured and monitored environments (Power, 1999). Dahler-Larsen (2017) has also pointed out that PRFSs may cause ambiguity in terms of expectations for researchers. So while publications become closely monitored and measured, in some cases it may not be clear what to expect in terms of ‘credits’ from a certain publication. Researchers may also disagree with the credits that are assigned, which may cause distress in itself.

From a government perspective, all these issues are important. Above all, however, the government expects investment in universities to translate into economic and societal impact. According to a recent analysis, every Euro that the government invests in Flemish universities generates 6 Euros of wealth in Flanders (BiGGAR Economics, 2017). This occurs through education, research, and knowledge and technology transfer. While the competition for universities is international (in particular for research), the PRFS compares universities within the region. Even though whole counting is used for all parameters—and collaboration is hence encouraged, the fact that the reference for the funding distribution is regional often reinforces the idea that the PRFS is not encouraging collaboration. When the Flemish Minister of Innovation Philippe Muyters announced on the 26^th of September 2018 extra investments in research and innovation, including 35 million Euros of additional BOF-funding, it therefore came as no surprise that he stressed the importance of collaboration, both among universities and internationally. In addition, he invited the universities to come forward with suggestions on how to encourage more interdisciplinarity and excellence through the BOF-means. At the time of writing, no decision has been taken on whether or how to fine-tune the BOF-key in response to these expectations. One possibility appears to be more emphasis on international benchmarking and/or allocation of funding according to goals and strategies rather than measured (by definition: past) performance. It is an open question whether such an approach would in the end lead to more economic and societal impact. Such unknowns notwithstanding, the impact of the Flemish universities is high already.