1 Introduction
Quantitative evaluation studies monitoring the scientific performance of actors (countries, institutes, individuals and also journals) regard primarily scholarly output and impact. Output is usually quantified by counting publications. The impact measurement usually regards scientific impact. Such impact, framed as a proxy for quality, is then measured by citations.
Assessing other impacts, for instance societal impact, is much more complicated and certainly less developed. Traceable effects of research in the societal context varies from one field to the other, from application to application and from case to case. For that reason, such impact is often assessed by case studies or other qualitative approaches.
Research assessments mostly deal with organizational entities, such as universities, research institutes or individual researchers. Another type of assessed entity we can discern is the journal. Its role and quality is evaluated in many studies and reports. The most common assessment of journals is done by using the Journal Impact Factor (JIF). In short and simplified, JIF measures the average number of citations received by papers in a journal within the first two years. This indicator has often been used to assess the quality of individual publications. Being accepted by a journal a paper meets the standard of that journal. Recently, this particular use of JIF has been criticised heavily leading to, for instance, “The San Francisco Declaration on Research Assessment”, DORA (Cagan, 2013).
On top of that, a group of bibliometric and evaluation specialists, scientists, publishers, scientific societies and research-analytics providers recently promoted the need to develop a new suite of journal indicators and other ways to assess the quality of journals (Wouters et al., 2019). The authors state that these indicators and other ways to assess journals are to be developed in the context of the journals’ primary tasks: registering, curating, evaluating, disseminating and archiving.
In this paper, I frame the assessment of journals from a broader perspective. The approach is based on a recent proposal to assess the connectedness of research actors to society (Noyons, 2018). The approach does not intend to cover all aspects of a journal’s primary tasks but it does cover parts of the task of dissemination.
A key point I make in this paper is that the broader impact of journals should not be limited to the output of the publications in the journal itself. We can open a variety of options to assess the impacts of journals and science (other) actors if we consider the content of the entire research areas to which a journal’s output belongs and contributes. In other words, the connectedness and broader impact does not depend on the output and impact of the journal itself alone. The community of research around the journal matters as well.
2 Data and method
2.1 Web of Science (WoS) and publication-based classification
For this study, I use the data from the bibliometric version the Web of Science core collection hosted at CWTS. It is a dedicated version of the Web of Science online for bibliometric studies, with additional features and improvements. The important add-on for this study is the publication-based classification (Waltman & van Eck, 2012) in which publications are clustered on the basis of their citation-based context into coherent groups (Traag, 2015; Traag, Waltman, & van Eck, 2019). Over 4000 clusters (CWTS classification, version 1813, available at www.leidenranking.com) provide a fine-grained structure of all sciences independent of the journals in which research papers are published. This structure enables a topic basic-based analysis and will be used to identify research communities, i.e., clusters of publications on a certain topic.
The VOSviewer map below shows the distribution of the 4,000 clusters. Each cluster is represented by a circle in the map. The position of a circle indicates its relatedness to all others, based on their mutual citation traffic. The denser the citation traffic, the closer two circles are. The size of a circle represents the number of publications included.
Figure 1. Landscape of science (data: Web of Science 2000-2017).^Circles represent clusters of publication (areas), size represents relative volume (numbers of publications), color represents main fields, disciplines. |
Another add-on to the CWTS database is the direct link of WoS publications to Altmetric data and (PATSTAT) references in patents. For each paper we determine whether it has been mentioned on Twitter, in a news item, in a policy document, in a Wikipedia article etc or whether it was cited in a patent. I consider such mentions or citations as non-academic events or actions related to the research that was mentioned, cited etc. This information I will therefore use as signals of connectedness between research and society.
2.2 ABC method
In the approach to characterize journals I apply a method, coined as area-based connectedness (ABC) to society (Noyons, 2018). This approach assesses the connectedness of the research covered based on the output of an actor, in this case a journal. This connectedness is captured by the communication signals we detect between research output and society. These signals are diverse and represent a variety of dimensions.
To determine connectedness of research areas or communities to society, we measure a set of signals. The individual signals represent different aspects, dimensions of this connectedness. For instance, (co-)authorship of industry in publications indicates technological application and/or commercial use of research. Mentions in policy documents of publications signifies societal or political interest or value. I will demonstrate that each signal has its own specific profile. The profiles relate to their distribution across the science landscape. Different signals connect different research areas.
Until now, such connectedness signals are measured at the level of actors. For instance, the share of output of actor A co-authored with industry would then be a an indication of the societal, economic, technological impact of A. Or the number of twitter mentions of articles by actor B would be an indication of B’s societal impact (Bornmann, 2013; Seppo & Lilles, 2012).
Such approaches have at least two flaws. Firstly, social media metrics are easily manipulated and therefore vulnerable for malpractices (Costas, Zahedi, & Wouters, 2015; Haustein et al., 2016). Moreover, such indicators may be affected by practical limitations. For instance, co-authorship with industry often depends on the willingness of the company involved to co-author. Similarly, mentions of scientific output in non-scholarly output often depends on the accessibility of scholarly literature. If a research paper is not available or too technical, the author of a policy report may decide to cite another publication on the same topic.
The ABC method measures the output of a journal using the character of research communities (areas) to which it belongs. A community is defined as a set of publications on the same topic, i.e. a cluster of publications from the CWTS publication-level classification.
Weighted by the number of publications a journal has within a community or area it “inherits” the character of the community/area. This character involves the connectedness to society as measured by the signals.
The approach may not meet all the criteria for proper journal indicators and/or responsible use thereof, but it does provide a broad range of the indicators to assess the role of journals which are less vulnerable for gaming and manipulation than the existing (mostly) one-dimension journal indicator.
Example
Journal A contains 100 publications in the period 2015–2018. These publications are distributed over 20 classes of the CWTS publication level classification. The 20 classes contain between 200 and 1,000 publications in the period 2015 to 2018. Within that period a certain share of the output in a class is cited by patents, is authored by industry, is mentioned in a policy document, etc. Every class can be characterized along each of these dimensions. Moreover, there is a global average for each dimension. Let’s suppose the global average of publications being cited in patents is 10%. In the 20 classes of Journal A, the share of publications being cited ranges between 0% and 20%. So, some classes have a relatively high share of papers being cited by patents, while other have a low share or even none. The connectedness of Journal A is to be measured by a combination of its output distribution and the connectedness its (20) classes, for instance as follows:
Journal A contains 100 publications in the period 2015–2018. These publications are distributed over 20 classes of the CWTS publication level classification. The 20 classes contain between 200 and 1,000 publications in the period 2015 to 2018. Within that period a certain share of the output in a class is cited by patents, is authored by industry, is mentioned in a policy document, etc. Every class can be characterized along each of these dimensions. Moreover, there is a global average for each dimension. Let’s suppose the global average of publications being cited in patents is 10%. In the 20 classes of Journal A, the share of publications being cited ranges between 0% and 20%. So, some classes have a relatively high share of papers being cited by patents, while other have a low share or even none. The connectedness of Journal A is to be measured by a combination of its output distribution and the connectedness its (20) classes, for instance as follows:
2.3 ABC signals and dimensions of connectedness
The dimensions considered in the ABC method discussed in this paper are based on the signals available to measure societal connectedness. For journal indicators we implement the following dimensions with the according signals between parentheses:
- News (papers being mentioned in news items)
- Policy (papers being mentioned in policy documents)
- Industry R&D (industry authorship)
- Technological or commercial application (papers cited in patents)
- Local scope (papers in local languages, not in English)
In the remainder of this section I discuss the dimensions one by one. For each dimension I discuss the signal being used as well as their distribution across the science landscape.
2.4 News
The News dimension relates to research being relevant in the broadest sense for the general public. It is assessed by the proportion of publications in a research area mentioned in news items. These news items are collected and included in Altmetrics from a large set of sources. We detect around 2,500 news sources. The top 10 most common sources are:
1. The Conversation
2. EurekAlert!
3. Phys.org
4. MedicalXpress
5. Health Medicinet
6. Science Daily
7. Huffington Post
8. Medical News Today
9. Bioportfolio
10. Yahoo! News.
These approximately 2,500 sources are consulted for mentions of publications.
I use these mentions as signal for connectedness in the following way. For each area, the proportion of papers being mention in a news item is calculated. These proportions range from 0 to 50% across the 4,000 areas. Their distribution is visualized inFig. 2 .
I use these mentions as signal for connectedness in the following way. For each area, the proportion of papers being mention in a news item is calculated. These proportions range from 0 to 50% across the 4,000 areas. Their distribution is visualized in
In Fig. 2 , I use a color coding indicating the proportion in such a way that the Blue areas have a lower proportion or even zero in Dark Blue, while the Yellow areas have a relatively high proportion.
Figure 2. Landscape of science (data: Web of Science 2000-2017).^Color represents share of publications mentioned in news items |
2.5 Policy
The policy dimension relates to research being relevant to political issues. For this dimension I use the signal of mentions of research output in policy documents. These documents are retrieved from 56 sources. The top 10 most commonly used organizations are:
1. National Academies Press
2. World Health Organization
3. Centers for Disease Control and Prevention (CDC)
4. National Institute for Health and Care Excellence
5. UK Government (GOV.UK)
6. National Bureau of Economic Research
7. Analysis & Policy Observatory (APO)
8. Food and Agriculture Organization of the United Nations
9. The Publications Office of the European Union
10. World Bank
The distribution of proportions of papers mentioned in policy documents across the landscape ranges between 0 and 19% and is visualized in Fig. 3 .
Figure 3. Landscape of science (data: Web of Science 2000-2017).^Color represents share of publications mentioned in policy documents. |
The policy signal shows a preferred position in the social sciences, life & earth sciences and biomedical sciences. This preference is different from the news dimension. Such differences support the fact that one should look at more than just one dimension when considering connectedness to society.
2.6 Industry R&D
The dimension related to the interest from the private sector is assessed by the authorship of companies in publications. For this dimension I calculate the proportion of output in which the author affiliation is a company in an area, ranging between 0 and 30%. The distribution of this signal is shown in Fig. 4 .
Figure 4. Landscape of science (data: Web of Science 2000-2017).^Color represents share of publications (co-)authored by Industry (a company). |
In Fig. 4 , we see quite a different picture than in the other dimensions. Here, the preferred areas are in computer sciences, physical sciences, engineering and, to a lesser extent, in biomedical sciences.
2.7 Technology
The technology dimension relates to the support or enabling strength of research. The signal I use to monitor this is citations from patents, operationalized by measuring the proportion of publications being cited in patent as Non-Patent Literature references. The distribution of this signal, ranging between 0 and 40%, is shown in Fig. 5 .
Figure 5. Landscape of science (data: Web of Science 2000-2017).^Color represents share of publications cited by at least one patent. |
In this case the signal is primarily detected in biomedical sciences and the chemistry part of the physical sciences. On top of that, we find somewhat stronger signals in engineering.
2.8 Local interest
The dimension of local interest relates to the extent to which research is relevant for local issues. Most likely, research with a local interest is published in a local language. This maybe any language so that I defined it as non-English. English is the most international, i.e. non-local, language. That does not imply, however, that English language papers cannot be aiming at local issues. This in particular is an example why the ABC method is an interesting approach. A journal is usually in one language only. Therefore, characterizing journals by the proportion of non-English papers would neglect journals in English with a local interest. By using the characterization of the areas in which a journal is covered, inheriting their character, we can identify English language journals with a local interest.
The proportion of papers not in English by area ranges between 0 and 83%. The areas with really high proportions are often small. The distribution over the landscape is shown in Fig. 6 .
Figure 6. Landscape of science (data: Web of Science 2000-2017).^Color represents share of publications not in English language. |
The preferred areas for non-English publications are in social sciences and clinical and health sciences.
3 Selected results
In this section, I will demonstrate a way to measure the connectedness of journal using the ABC approach. For each journal I count the number of publications and their distribution across the 4,000 research areas. The output of a journal will then be characterized by the connectedness of the areas in each dimension, which means that the connectedness of a journal is the average of the sum of the product of the number of publications in an area and the score on a dimension in an area (weighted average).
The final connectedness on each dimension will then be compared to the overall average of connectedness of that dimension.
As a first example, consider the Journal of Fluency disorder. This journal provides comprehensive coverage of clinical, experimental, and theoretical aspects of stuttering, including the latest remediation techniques (www.journals.elsevier.com). Obviously, this is a specialized journal with a clear clinical focus. Usually, such journals are less well cited than more fundamental journals. Its journal impact factor is 1.73 (JCR 2017/18).
The distribution of publications from this journal across the science landscape is depicted in Fig. 7 . The structure is as explained in the previous section. The size of the circle indicates, proportionally, the number of publications from this journal in this cluster.
Figure 7. Landscape of science (data: Web of Science 2000-2017).^Size represents number of publications in Journal of Fluency disorder (2014-2017). |
The cluster to which almost all publications from this journal belong has a relatively high score on the news dimension, while the share of non-English papers is around average (baseline). In all other dimensions the connectedness of this cluster is low. This leads to the ABC profile as depicted in Fig. 8 .
Figure 8. Area-based connectedness to society profile of the Journal of Fluency Disorders (2014-2017). |
This profile tells us that the research published in this journal has a connectedness to society through the news dimension (almost 40% above average) and an average connectedness on the dimension of local interest.
A second specialized journal, Solar Physics, has 95% of its output in one cluster (See Fig. 9 ) and a modest impact factor (2.6).
Figure 9. Landscape of science (data: Web of Science 2000-2017).^Size represents number of publications in journal Solar Physics (2014-2017). |
The area (ID 43) has a high connectedness through its content on the Industry dimension as shown in Fig. 10 below (20% above average). Hence, this journal may not have the high impact one may expect or want for this journal but the research published in this journal does have its value for application, as indicated by the industry dimension.
Figure 10. Area-based connectedness to society profile of journal Solar Physics (2014-2017). |
Finally, I illustrate the approach using the Lancet, a journal with an impact factor over 50 and a much broader scope. The broader scope of this journal is demonstrated by the distribution across areas in Fig. 11 .
Figure 11. Landscape of science (data: Web of Science 2000-2017).^Size represents number of publications in The Lancet (2014-2017). |
The Lancet is used in biomedical and health sciences in the broadest sense. This yields quite a different profile of connectedness from the other, specialized journals. The “Lancet areas” show particularly a very high connectedness on the policy dimension with almost 4 times the average (see Fig. 12 ). This is also visible in the news dimension at 75% above average. Maybe a bit more surprising is the high connectedness on the local interest dimension at 50% above the baseline, which most likely relates to clinical research.
Figure 12. Area-based connectedness to society profile of The Lancet (2014-2017). |
4 Conclusion & discussion
Journals are usually assessed by the Journal Impact Factor (JIF). Besides the issues related to this indicator (e.g., Moed, 2002), such an approach is one dimensional and overlooks the other tasks and values a journal has (Wouters et al., 2019). In this study I introduce an alternative and more diverse way to assess journals. The approach is not actor-based but areas-based. This means that the output of a journal is not assessed by the characteristics of publications from that journal but rather by the characteristics of the areas their publications are published in. In other words, journals inherit the characteristics from the areas to which they belong.
The measures I introduce, relate to the connectedness of journals to society or more general, to their dissemination potential. To monitor this quality, the proposed area-based connectedness (ABC) to society approach introduces 5 dimensions. These dimensions are based on signals we can detect between research outputs and society. There are many more possible signals and dimensions to study and the task we take upon us in the near future is to explore as many as possible to cover a broader spectrum of connectedness. The dimensions I introduce in this paper are civil interest, policy interest, industry R&D, technology and local interest. With a few illustrative examples, I demonstrate that the ABC approach provides an interesting new way to assess potential of journals. The approach can be based on the most recent output, and as such the assessment is much more up-to-date than citation-based indicators. Furthermore, by evaluating a journal’s output through the inherited ABC character of the (algorithmically created) areas, the assessment is less vulnerable for gaming and manipulation (Falagas & Alexiou, 2008).