Following a general trend in Europe as well as advise from the European Commission, Norway implemented a performance-based funding system for its Higher Education Sector in 2002. The system only affects a small part of the funding, and most weight is given to indicators of educational activity. The indicators representing research were initially based on numbers of doctoral degrees, amounts of external funding, and numbers of tenured research personnel. However, neither the funder—the Ministry of Education and Research—nor the funded organizations were happy with the latter staff-dependent indicator. They wanted an indicator that would more directly represent research activity and contribute to research quality. Starting in 2003, with myself as expert advisor, the funding and funded organizations collaborated on developing an indicator based on scientific publishing activity. It was implemented in 2005 as a model for data production, measurement and funding with three main components:

(A) A complete representation in a national database of structured, verifiable and validated bibliographical records of the peer-reviewed scholarly literature in all areas of research;

(B) A publication indicator with a system of weights that makes field-specific publishing traditions comparable across fields in the measurement of “Publication points” at the level of institutions;

(C) A performance-based funding model which reallocates a small proportion of the annual direct institutional funding according the institutions’ shares in the total of Publication points.

In principle, component C is not necessary to establish components A and B. The experience is, however, that the funding models in C support the need for completeness and validation of the bibliographic data in component A. Since the largest commercial data sources, such as Scopus or Web of Science, so far lack the completeness needed for the model to function properly, the bibliographic data are delivered by the institutions themselves in component A through Current Research Information Systems (CRIS). References from Scopus or Web of Science are imported to the systems, and other references are added according to a definition and validation procedures explained in section 3 below.

Denmark and Finland have also implemented the Norwegian model. It has inspired changes in similar national models in Flanders (Belgium) and Poland, and it is used for local purposes by several universities in Sweden and by University College Dublin in Ireland.

The Norwegian model is designed to represent all areas of research equally and properly. The typical mode of implementation in Denmark, Finland and Norway has been for the governments to involve prominent researchers in each major area of research, e. g. deans appointed by the rector’s conference, to represent the respective faculties at all universities, or experts appointed by the learned societies on the national level. The representative researchers have then been involved directly in the national adaptation and design of the publication indicator (component B). The result of these design processes has been one single and simple pragmatic compromise—the first bibliometric indicator to cover all areas of research comprehensively and comparably—rather than several separate representations of scholarly publishing standards in each individual field. This ‘universalistic’ approach was necessary for funding general universities and other more specialized institutions in the same system.

In the following, we will first present the organization and design of the model with more details about its three components—A, B, and C. Then we will present the results of an evaluation of the Norwegian Model that was undertaken by a Danish team of experts in 2013. They studied the design and effects of the model and suggested changes that have been implemented later. The article ends with an overview of a new type of bibliometric studies that are based on the type of comprehensive national publication data that the Norwegian Model provides.

The data for the indicator (component A) are produced by the institutions themselves in a shared quality assurance system that creates an open and transparent national database with references to all scientific publications from all institutions. The data are not only used by the government for funding, but also by the institutions themselves for internal purposes at various levels, e.g. in redistribution of funds at local levels, in annual reports and statistics, in CV’s and applications, in open Current Research Information Systems, and as bibliographical references that lead on to the available full text of publications.

The creation of a national database of this kind starts with acknowledging that not only the government, but also the research institutions themselves, need continuous and structured information about their research activities for internal and external purposes. In bibliometrics, we are used to serve the institutions from the outside with professionally processed data and analysis of their activities. But institutions also produce their own data. At the department level, there is a tradition for listing the staff’s scientific production as bibliographic references in annual reports. For some years now, this tradition has evolved into creating information systems with databases that record all scientific publications at the level of institutions. Since these databases demand input from the researchers themselves, there is often a problem with incompleteness, and the data may lack the necessary structure and coherence that can be seen in professional bibliographic data sources like the Web of Science (WoS) or Scopus. On the other hand, institutional databases may include scientific and scholarly publications in books, series or journals that are not covered by WoS or similar data sources. Institutional databases may also solve the problem with attributing author names and addresses to unique persons and institutions, and they may add routines of quality assurance and validation to this process.

The design of the Norwegian Model started with an ambition to get the best from both types of data sources, creating complete and quality-controlled structured bibliographic data at the institutional level, not only for each of the institutions, but in one and the same national database for all institutions. An agreement on this ambition was reached between the Ministry of Research and Education and the institutions in 2003-2004 on the basis of a design of the indicator and its database that was developed by the Norwegian Association of Higher Education Institutions (Sivertsen, 2010). This organization—a parallel to the Rectors’ conferences in other countries—has since then had the responsibility for the maintenance and further development of the indicator and its database. This responsibility is carried out by a National Publishing Board with representatives at the level of deans from all types of institutions and major research areas. My own role has been to contribute as an expert in the design phase and as a consultant to the National Publishing Board after the implementation. I have had a similar role in Denmark since 2007.

As explained in the introduction, the Norwegian model is primarily designed to serve a partly indicator-based funding system for research institutions. Since institutions have different research profiles (e. g. a general university versus a technical university), the model needs to represent all research areas in a comprehensive and comparable way.

There is no single comprehensive international data source for all scholarly publications in all research areas. Figure 1 exhibits the patterns and degrees of coverage in the two largest commercial data sources, Scopus and Web of Science. We know from the complete data set that we use here for comparison, the Norwegian Science Index, representing all peer-reviewed scientific publications in the Current Research Information System in Norway (Cristin), that the deficiencies in coverage of the social sciences and humanities are mainly due to incomplete coverage of the international journals, limited or no coverage of national scholarly journals and very limited coverage of peer-reviewed scholarly books (Sivertsen, 2014).

The data for the Norwegian model are delimited by a definition which all areas of research contributed to develop and agree on before it was published in 2004 (Sivertsen & Larsen, 2012). According to this definition, a scholarly publication must:

1. present new insight

2. in a scholarly format that allows the research findings to be verified and/or used in new research activity

3. in a language and with a distribution that makes the publication accessible for a relevant audience of researchers

4. in a publication channel (journal, series, book publisher) which represents authors from several institutions and organizes independent peer review of manuscripts before publication.

While the first two requirements of the definition demand originality and scholarly format in the publication itself, the third and fourth requirement are supported by a dynamic register of approved scholarly publication channels at http://dbh.nsd.uib.no/kanaler/. Suggestions for additions can be made at any time through the same web page. Publications in local channels (serving only one institution’s authors) are not included in the definition, partly because independent peer-review cannot be expected in local channels, and partly because the indicator connected to institutional funding of research is not meant to subsidize in-house publishing. Publication channels with questionable procedures for peer review are also excluded from the register of approved publication channels.

The definition of scientific publications is not meant to cover the researchers’ publishing activities in general. It is meant to represent research, not publications. Accordingly, it is limited to original research publications.

In addition to a definition, there is need for a comprehensive data source with bibliographic data that can be connected to persons and their institutional affiliations. These data need to be well-structured (thereby comparable and measurable), verifiable (in external data sources, e. g. in the library information sources) and validated (inter-subjective agreement on what is included according to the definition). These needs are now possible to serve due to the development during the last two decades of Current Research Information Systems (CRIS). They can be designed to produce quality assured metadata at the level of institutions or countries.

CRIS systems on the institutional level have become widespread recently, both in locally and commercially developed solutions (Sivertsen, 2018). Norway is one of a few countries that has a fully integrated non-commercial CRIS system at the national level. Cristin (The Current Research Information System in Norway; cristin.no) is a shared system for all research organizations in the public sector: universities, university colleges, university hospitals and independent research institutes. The Norwegian model, which is now used for institutional funding in all sectors, was a driver in the development of a shared system. One reason is that many publications are affiliated with more than one institution and need to be treated as such in the validation process and in the indicator. Another reason is that transparency across institutions stimulates data quality. Every institution can see and check all other institutions’ data. The publication database in the CRIS system is also online and open to society at large.

The costs of running Cristin would not be legitimate without multiple use of the same data. References to publications are registered only once, after which they can be used in CV’s, applications to research councils, evaluations, annual reports, internal administration, bibliographies for Open Archives, links to full text, etc.

In the measurement for the funding formula by the end of each year, the publications are weighted as they are counted. The intention is to balance between field specific publishing patterns, thereby making the publication output comparable across research areas and institutions that may have different research profiles. In one dimension, three main publication types are given different weights: articles in journals and series (ISSN), articles in books (ISBN) and books (ISBN). In another dimension, publication channels are divided into two levels in order to stimulate publishing in the most prestigious and demanding publication channels within each field of research. The highest level is named “Level 2”. It includes only the leading and most selective international journals, series and book publishers. There is also a quantitative restriction, since the publication channels selected for Level 2 can only in total represent up to 20% of the world’s publications in each field. The weighting of publications by type and channel is shown in Table 1.

Publication points are measured at the level of institutions, not at the level of individual researchers. The points for publications with multiple authors representing several institutions are fractionalized among the participating institutions according to their number of participating authors. Since 2015, the fractional counting method has been modified by using the square root of fractions—see section 5 below.

The list of journals, series and book publishers on “Level 2” is revised annually in collaboration with national councils in each discipline or field of research (Sivertsen, 2010). These councils propose changes to an interdisciplinary National Publishing Board, which governs the process on behalf of all institutions and has the final decision. Bibliometric statistics (world production versus national production in channels on both levels, and citation statistics for publication channels) are used as an aid in this process, but not as criteria by themselves.

From the start, the list of journals, series and book publishers has needed quality assurance on level 1 as well. Journals with only local authorship or questionable peer review and publishing procedures were not included. With the increase of “predatory journals” where one can publish rapidly against payment, the list also serves the purpose of good quality open access publishing.

There are two main variants of performance-based funding of research institutions in Europe: the evaluation-based variants (Italy, United Kingdom, Portugal), and the indicator-based variants (most other European countries) (Hicks, 2012; Jonkers & Zacharewicz, 2015). The Norwegian model was developed for indicator-based funding. It is, however, not an alternative to research evaluation. Research evaluations with expert panels are also practiced for formative purposes and with no with direct consequences for institutional funding (Sivertsen, 2017). In general, countries with indicator-based funding of research institutions do not rely solely on bibliometric indicators. Other indicators may be for example be external funding or the number of doctoral degrees. In addition, the indicators usually reallocate only a minor part of the total funding. Consequently, the economic consequences of an institution’s score on the publication indicator in the Norwegian model are therefore relatively small in all countries. In Norway, the publication indicator reallocates less than 2% of the total expenses in the Higher Education Sector. One publication point represents less than 3,000 Euro.

Still, the publication indicator receives a lot of attention from the researchers, much more attention than is given other and more consequential parts of the funding system. A reason might be that this indicator can be influenced directly by the researchers themselves, and that the indicators resonates with other incentives in the research system.

There have been several studies already of the effects of the Norwegian model in different contexts in Denmark, Flanders, Norway and Sweden (Ahlgren et al., 2012; Hammarfelt & de Rijcke, 2015; Ossenblok et al., 2012). In addition, there have been three evaluations commissioned by governments: After “light” assessments of the experiences with the model in the Danish and Flemish contexts in 2012 (Sivertsen & Schneider 2012; Technopolis Group, 2013), the Norwegian model was evaluated extensively in Norway in 2013 the Danish Centre for Studies in Research and Research Policy at Aarhus University (Aagaard et al., 2014). As well as advising improvement and further development, the exercise provided the basis for four in-depth studies of internationally relevant questions (Aagaard, 2015; Aagaard et al., 2015; Schneider et al., 2015; Bloch & Schneider, 2016).

The evaluation focused on the design, effects, organization, and legitimacy of the Norwegian model in Norway. Interviews with researchers and surveys to a large number of them was part of the evaluation. Since no broad general discontent with the model was found except for the identified problems (see below), and since unintended changes in the researchers’ behaviour could not be detected, at least at the macro level, the Ministry of Education and Research has decided to continue using the model as part of the performance-based funding.

The evaluation identified one major effect of the indicator, increased productivity, along with three major problems, all of which I will discuss shortly here.

A main finding was an increased publication rate above what could be expected from the increase of funding. Figure 2 below shows the increase in publication points in the higher education sector since 2004. Figure 3 below has a more independent measurement based on Web of Science. It shows the development in world shares of articles for four Scandinavian countries. Note that the incentive to publish was introduced in Norway in 2004, in Denmark (Norwegian model) and Sweden (only WoS-based) in 2009, and in Finland (Norwegian model) in 2015.

The evaluation in Norway found no other changes in the publication patterns than the increase. The balances between publication types (books, articles in books, articles in journals and series) and publication languages (the native language versus international languages) had remained the same. Collaboration in authorship is increasing at the same rate as in other countries of the same size. The length of publications remains the same. The citation impact on country level is also stable. However, as seen in Fig. 2, both the absolute number and the percentage publications in the most internationally influential publication channels has increased.

The evaluation in Norway identified three major problems with the model; one problem in the design of the indicator, and two problems with how the model is practiced.

Regarding the design problem, the evaluation (Aagaard et al., 2014) found an imbalance in the indicator’s representation of productivity across fields, thereby confirming an observation in an earlier study (Piro et al., 2013). The humanities and social sciences seemed to be more productive than the sciences, probably because the indicator uses fractional counting of co-authored publications. Even if co-authorship practices seemed unaffected at the macro level, the evaluation also expressed concern about the risk of discouraging research collaboration when using fractional counts (Bloch & Schneider, 2016).

The indicator needed to be balanced because it is used for measuring productivity across institutions with different research profiles (e.g. general versus technical universities, universities with and without medical faculties). To study the problem and simulate its solution, we used data from CRIStin (Current Research Information System in Norway), covering the two years 2011-2112. In this system, authors are identifiable as real persons at Norwegian institutions, not just as author names with addresses. We studied the average productivity among 14,441 active researchers who had contributed to a minimum of two publications in the two years. We found that an intermediate solution—making different co-authorship practices in different fields comparable—is to use the square root of the institution’s fraction of the publication (Sivertsen, 2016a). This solution has recently proved to be generalizable in a study based on data from Web of Science (Sivertsen et al., 2018).

The transparency and thereby the legitimacy of the annual nomination process for Level 2 (described above in section 3) is a second problem identified in the evaluation. Here, the Norwegian Association of Higher Education Institutions has implemented a solution in which the whole process of decisions (and their explicit grounds) are available in an internet portal open to all researchers, where they can both influence and gain information about the process: https://npi.nsd.no/.

The third problem identified is the local use of the indicator. Although the Norwegian model was developed for institutional funding on the national level, the indicator has become widely used also for internal purposes at the level of institutions, faculties, departments, etc. Some of these practices may be reasonable; other practices can be highly problematic, especially if the indicator replaces responsible leadership and human judgment. Norwegian research institutions are relatively autonomous and cannot be instructed from the outside with regard to leadership practices. However, a large national conference was arranged early in 2015 where leaders of research organizations at all levels shared their views and experiences related to the use of the publication indicator at the local level. National guidelines for local use have also been published by the National Publishing Board.

As described in section 3, the Norwegian model is based on a nationally integrated research information system. It still remains to make the data from such systems comparable across countries, but recent developments in Europe are pointing in this direction (Puuska et al., 2018).

However, we can see already now that the establishment of non-commercial national CRIS in some countries, particularly countries with data validation connected to performance-based funding, has supported an increase in output-based studies of research, particularly in the humanities and social sciences, where CRIS can provide a more complete representation of scholarly publications than we find in the commercial data sources (Sivertsen, 2016b). The field is quite new with most of the publications appearing only the last two years, after the establishment of the European COST network ENRESSH (European Network for Research Evaluation in the Social Sciences and Humanities). The increased activity is now the basis for a new series of biannual conferences focusing on Research Evaluation in the Social Sciences and Humanities (RESSH, established in Rennes 2015, continued in Antwerp 2017).

Particularly active is the ECOOM group at the University of Antwerp. This group developed and is running the Flemish Bibliographic Database for the Social Sciences and Humanities (VABB-SHW) for a similar purpose as the Norwegian CRISTIN system. They have used the data for studying several aspects of the publishing patterns of the social sciences and humanities that have rarely been studied before. Here are some examples:

The ECCOM group studied general changes in the publication patterns of the social sciences and humanities over a decade (2000-2009), finding growth in the output, particularly a steady increase in the number and the proportion of publications in English, however with no overall shift away from book publishing (Engels et al., 2012). They found almost identical evolutions in the use of English as a publication language by comparing data from CRIS in Flanders and Norway, however WoS coverage was stable for Norway but had been increasing rapidly for Flanders, probably because of differences in the parameters used for performance-based funding of universities (Ossenblok et al., 2012). Internationalization was also found in book publishing. Whereas peer reviewed books were increasingly published abroad and in English, non-peer reviewed book literature remained firmly domestic and published in the Dutch language (Verleysen et al., 2014a). Whereas the humanities are more continentally oriented in their book publishing, the social sciences are firmly Anglo-Saxon oriented (Verleysen & Engels, 2014b). A study of co-authorship patterns in the social sciences and humanities indicated that collaborative publishing in the SSH is increasing with a sharp decline in single-author publishing (Ossenblok et al., 2014). A study of 753 peer reviewed edited books and the 12,913 chapters published therein revealed that not only co-authorships, but also co-editing and publishing different chapters in the same books are indicators of scholarly collaboration in the social sciences and humanities (Ossenblok & Engels, 2015). The editors of scholarly books are mostly established researchers, produce more book chapters and monographs than do other researchers, and are more productive (Ossenblok et al., 2015).

A new study based on CRIS data (Kulczycki et al., 2018) investigates publication patterns in the language and type of social sciences and humanities across a much wider range of non-English speaking European countries, including Eastern Europe: Czech Republic, Denmark, Finland, Flanders (Belgium), Norway, Poland, Slovakia, and Slovenia. The study demonstrates that publication patterns are related not only to discipline but also to each country’s cultural and historic heritage. This finding corrects an assertion in an earlier CRIS-based study (Sivertsen, 2016b) that publication patterns vary by discipline, but less across countries in the same discipline.

Furthermore, on the basis of CRIS data, other researchers have provided deeper insight into the publishing patterns of particular fields of research, such as political science (Chi, 2014; Chi, 2015) and law (van Leeuwen et al., 2016).

There are some studies based on CRIS data that investigate policy-related questions across all fields of research, not only the social sciences and humanities. With data from the CRIS of the University of Helsinki, Puuska (2009) examined the effects of a scholar's position and gender on publishing productivity in several types of scientific publications: monographs, articles in journals, articles in edited books, and articles in conference proceedings. Aksnes et al. (2013) studied the mobility of researchers on the basis of CRIS data from the four main Norwegian universities.

Other studies have contributed to a critical examination of how CRIS data are used for statistics, evaluation and funding in research management, most often with suggestions for further development of data and indicators (Sivertsen & van Leeuwen, 2014; Diaz-Faes et al., 2016; Kulczycki, 2017; Savic et al., 2017; Giménez-Toledo et al., 2016; Giménez-Toledo et al., 2017), sometimes only describing potential negative effects of such use (Hammarfelt & de Rijcke, 2015).

Finally, to illustrate in more detail that the use of CRIS data in studies of research may also have a broader interest beyond bibliometrics and studies of the social sciences and humanities in particular, we will end this chapter by presenting two examples showing that bibliographic data in CRIS can be combined with other data (personal variables, backgrounds, resources, projects, networks, memberships) within or outside of the CRIS system, thereby serving science studies more generally.

The first example is a little study of gender, age and productivity that we did some years ago, based on data in the Norwegian CRISTIN system (Sivertsen, 2016c). Here, gender, age and complete records of all peer reviewed scientific publications is among the available information for each active researcher. We studied the productivity of 17,212 researchers (10,279 men and 6933 women) aged 27-67 who published in 2011. Altogether, they contributed to 12,441 unique publications. There was no double counting if two or more researchers contributed to the same publication. Instead, publications with multi-authorship were fractionalized by the number of authors. Figure 1 shows the result by presenting the women’s share of among Norwegian researchers and their publication output in each 1-year age cohort between 27 and 67.

We can see the gender gap decreasing as younger generations are recruited to research. We also observe that the difference in productivity between men and women is somewhat larger in the younger age cohorts. This is not a new finding. The same observation and its possible explanations have been studied more extensively in previous studies, e.g. by Kyvik and Teigen (1996) with the telling title ‘‘Child Care, Research Collaboration, and Gender Differences in Scientific Productivity’’. That study, however, was based on a survey and interviews with relatively few researchers. Figure 4 is based on complete data for all active researchers in a country. With the help of the CRIS system, we can see that the difference in productivity between men and women is so far consistent across all types of institutions (universities, university colleges, research institutes, hospitals) and across all fields of research (humanities, social sciences, health sciences and natural sciences). This could be an indication that gender equality in research is dependent also on the degree to which gender equality has been achieved in society.

The last example is a research project that was be presented at the 16^th conference of the International Society of Scientometrics and Informetrics (ISSI) in Wuhan in 2017 (Zhang & Sivertsen, 2017). The project is a response to two independent studies that were first presented at ISSI 2015 and published the year after in PLOS ONE. Larivière & Costas (2016) and Sandström & van den Besselaar (2016) observe similarly that productivity among individual researchers is correlated with citation impact in large datasets from Web of Science (WoS). While the latter study draws the policy implication that productivity should be incentivized, the first study explains their finding by the Mertonian theory of cumulative advantages and maintains that research assessment should be qualitative and focus on research quality. Both studies are based on author name disambiguation in Web of Science data.

As acknowledged by the authors, there are several problems with studying individual productivity by using author name disambiguation. Here, CRIS can come to aid. In our study of the same general research question, we match Web of Science with records in The Norwegian Science Index and The Norwegian Research Personnel Register. Hence, we can study real persons, not just authors. We know their age, gender, position and affiliation, as well as their former career and educational background in the higher education sector. Productivity depends on what roles researchers actually take in research, what position they have, what resources are available, and what they achieve in their careers. We also have broader basis for measuring productivity across fields, building on the completeness of CRIS data. We found that productivity and citation impact is much less correlated if publications beyond WoS are also included in the measurement of productivity. We also found higher average citation rates among post docs than among professors. Young researchers are on the average more cited than seniors, if measured per publication. On the other hand, senior researchers and professors are more productive within WoS.

The Norwegian model is an example of how well integrated and structured research information systems at the institutional or national level can serve several purposes, including local management, national funding, statistics, research information, and bibliometric studies. One strength of these systems is connected to the completeness of bibliographical records, the automatic disambiguation of authors/persons and addresses/affiliations, and the possibility of thereby to connect with other data describing the researchers, their institutions and resources, and the outcomes of their research. Another strength is that they empower the academic communities in influencing the evaluation and funding system by allowing them to define and prioritize what is valued and counted in these contexts, and by creating an overview of the scientific production. Nevertheless, any research information system connected to funding will also create incentives towards certain behaviours in research and scientific publishing. The Norwegian model was designed to balance between different publication patterns rather than changing them, by being inclusive and at the same time stimulating publishing in the most prestigious international publication channels. The design and effects of the model will still need to be monitored and discussed. Fortunately, this discussion is prevalent and vivid in the international literature, as demonstrated in this contribution and its references.