Research Papers

Gauging scholars’ acceptance of Open Access journals by examining the relationship between perceived quality and citation impact

  • William H. Walters ,
Expand
  • Mary Alice & Tom O’Malley Library, Manhattan University, Riverdale, NY 10471, USA
†William H. Walters (Email: ).

Received date: 2024-08-31

  Revised date: 2024-11-01

  Accepted date: 2024-11-07

  Online published: 2024-11-14

Abstract

Purpose: For a set of 1,561 Open Access (OA) and non-OA journals in business and economics, this study evaluates the relationships between four citation metrics—five-year Impact Factor (5IF), CiteScore, Article Influence (AI) score, and SCImago Journal Rank (SJR)—and the journal ratings assigned by expert reviewers. We expect that the OA journals will have especially high citation impact relative to their perceived quality (reputation).

Design/methodology/approach: Regression is used to estimate the ratings assigned by expert reviewers for the 2021 CABS (Chartered Association of Business Schools) journal assessment exercise. The independent variables are the four citation metrics, evaluated separately, and a dummy variable representing the OA/non-OA status of each journal.

Findings: Regardless of the citation metric used, OA journals in business and economics have especially high citation impact relative to their perceived quality (reputation). That is, they have especially low perceived quality (reputation) relative to their citation impact.

Research limitations: These results are specific to the CABS journal ratings and the four citation metrics. However, there is strong evidence that CABS is closely related to several other expert ratings, and that 5IF, CiteScore, AI, and SJR are representative of the other citation metrics that might have been chosen.

Practical implications: There are at least two possible explanations for these results: (1) expert evaluators are biased against OA journals, and (2) OA journals have especially high citation impact due to their increased accessibility. Although this study does not allow us to determine which of these explanations are supported, the results suggest that authors should consider publishing in OA journals whenever overall readership and citation impact are more important than journal reputation within a particular field. Moreover, the OA coefficients provide a useful indicator of the extent to which anti-OA bias (or the citation advantage of OA journals) is diminishing over time.

Originality/value: This is apparently the first study to investigate the impact of OA status on the relationships between expert journal ratings and journal citation metrics.

Cite this article

William H. Walters . Gauging scholars’ acceptance of Open Access journals by examining the relationship between perceived quality and citation impact[J]. Journal of Data and Information Science, 2025 , 10(1) : 151 -166 . DOI: 10.2478/jdis-2025-0002

1 Introduction

By eliminating subscription fees, Open Access (OA) journals make research more accessible to scholars and to the general public. This has the potential to increase citation impact as well as readership. Although OA status has no independent impact on citation rates, authors can maximize the impact of their research by choosing OA journals for the papers that are most likely to benefit from a broad, interdisciplinary readership (Basson et al., 2021; Davis, 2011; Davis et al., 2008; Gaule & Maystre, 2011; Hubbard, 2017; Piwowar et al., 2018; Wang et al., 2021; Wray, 2016). In terms of citation impact per article, OA journals can now be found among the top five journals in nearly 200 fields including chemical engineering, economics, electrical engineering, environmental science, genetics, history, management science, molecular medicine, music, political science, public health, soil science, and statistics (Elsevier, 2023).
Many scholars were initially skeptical of OA journals, however, especially with regard to the “authors pays” model, in which publishers earn revenue through publication fees paid by the authors, their institutions, or their funding agencies (Borrego, 2023; Dalton et al., 2020; Halevi & Walsh, 2021; Togia & Korobili, 2014). This skepticism has led to a bias against OA journals on the part of some authors and evaluators. For example, 55% of the respondents to a 2006 survey of authors who published in three well-known medical journals reported that they would not submit again to their journal if it became OA and charged fees, mainly because the reputation of the journal would suffer (Schroter & Tite, 2006). More recent surveys reveal widespread negativity toward the author-pays model as well as a continuing belief that OA journals suffer from weak peer review standards and that “anyone can pay to get published” in OA journals (Feenstra & López-Cózar, 2022; Hayman, 2016; Segado-Boy et al., 2022).
These attitudes suggest that OA journals may have poor reputations relative to subscription journals of similar quality or impact. Moreover, there is some evidence that OA journals have relatively high citation impact due to factors other than scholarly quality, such as accessibility to a broad audience (i.e., the removal of barriers to reading and citing). (See section 2 for details.) This study evaluates whether OA journals in business and economics have especially high citation impact relative to their perceived quality (reputation)—or, stated differently, whether they have especially low perceived quality (reputation) relative to their citation impact.
For a set of 1,561 journals, multiple regression is used to estimate the ratings assigned by expert reviewers for the 2021 journal assessment exercise of the Chartered Association of Business Schools (CABS). The only independent variables are four citation metrics, evaluated separately, and a dummy variable representing the Open Access status of each journal (OA or non-OA). We expect that for each analysis, the OA regression coefficient will be negative, indicating that OA business and economics journals have low CABS ratings relative to their citation impact.

2 Context and previous research

2.1 Relationships between expert assessments and citation metrics

Quite a few studies have evaluated the relationships between subjective assessments of reputation (or “quality”) and citation impact. These investigations have been undertaken with regard to entire journals (e.g., Haddawy et al., 2016; Hall, 2011; Hodge et al., 2021; Kulczycki & Rozkosz, 2017; Mahmood, 2017; Saarela & Kärkkäinen, 2020; Walters, 2017b, 2024; Walters & Markgren, 2019), individual articles (e.g., Abramo et al., 2019; Weitzner et al., 2024), individual researchers (e.g., Derrick et al., 2011; Guba & Tsivinskaya, 2023), and institutions (e.g., Szluka et al., 2023). However, a comprehensive literature search revealed no prior studies of the impact of OA status on the relationships between expert journal ratings and journal citation metrics. Specifically, there is no clear evidence that OA journals have especially high (or low) citation impact relative to their subjectively assessed quality.

2.2 The influence of OA status on expert assessments

As noted in section 1, skepticism about OA publishing can generate negative biases that affect scholars’ opinions of OA journals (Borrego, 2023; Dalton et al., 2020; Feenstra & López-Cózar, 2022; Halevi & Walsh, 2021; Hayman, 2016; Segado-Boy et al., 2022; Togia & Korobili, 2014). In turn, these negative attitudes may lead expert evaluators to assign lower reputation scores to OA journals than to subscription journals of comparable objective quality or impact. Although this assertion is reasonable, it has not been tested directly. Just one study, of limited scope, has addressed the question. Carvalho Neto et al. (2016) evaluated a random sample of 450 Brazilian journals included in the Qualis assessment exercise, which rates more than 24,000 journals on an eight-point scale based on their citation impact, their index coverage, and the subjective assessments of expert reviewers. Perhaps surprisingly, the OA journals in their sample had substantially higher Qualis ratings than the subscription journals. In fact, the OA journals—68% of the 450—comprise at least 80% of the journals in each of the top four rating categories but fewer than 60% of the journals in each of the lowest four categories. Although this finding is contrary to the expectations that underpin this investigation, it is based on data for just 450 journals published in Brazil—journals that may be systematically different from most international journals.

2.3 The influence of OA status on citation impact

This study also draws on the idea that the accessibility of OA journals may give them greater citation impact than subscription journals of comparable quality. This assertion, set forth by Lawrence (2001) and subsequent authors (e.g., Copiello, 2019), is supported by some empirical investigations but not by others.
At the level of the individual article, OA status does tend to be associated with higher citation rates. However, this advantage can be fully attributed to factors other than OA status itself. For instance, OA and non-OA articles vary systematically with regard to field of inquiry, external funding, authors’ willingness to pay article processing charges, institutional support for OA publishing, and whether authors believe their findings are especially important or especially likely to appeal to a non-scholarly audience (self-selection). Many well-known studies have reported an OA citation advantage while controlling for just one or two (or none) of these confounding factors (e.g., Antelman, 2004; Dorta-González & Dorta-González, 2022, 2023a, 2023b; Eysenbach, 2006; Gargouri et al., 2010; Hadad & Aharony, 2023; Hadad et al., 2024; Koler-Povh et al., 2014; Sotudeh, 2020).
Studies that gauge the independent effect of OA status at the article level—those that include a full range of control variables or that randomly assign papers to OA or non-OA groups—reveal that all else equal, OA status has no independent impact on citation rates (Davis, 2011; Davis et al., 2008; Gaule & Maystre, 2011; Piwowar et al., 2018). This is not the same as no impact, of course. Moreover, there is some evidence of an independent effect for certain types of articles—those in particular subfields, for instance (Basson et al., 2021; Hubbard, 2017; Wang et al., 2021; Wray, 2016).
It is also possible that the influence of OA status on citation rates may be greater for entire journals than for individual articles. For instance, the title of an OA journal (e.g., PLoS Biology) often signals that the journal content is freely available, so potential readers know in advance that there will be no barriers to access. In contrast, OA papers published elsewhere—in hybrid journals or OA archives, for instance—have no such advantage. Simple comparisons of OA and non-OA journals do not support the idea that OA journals have greater citation impact (Björk & Solomon, 2012; Khan et al., 2023), although most studies of subscription journals that have “flipped” to OA status show that the transition to Open Access does bring a citation advantage (Asai, 2023; Bautista-Puig et al., 2020; Momeni et al., 2021). One recent paper reports mixed results (Wang et al., 2019), and one study of journals that have “reverse flipped” suggests that the transition from OA to subscription status also brings an increase in citations (Ming & Zhao, 2022).
Finally, two recent studies have shown that OA articles in hybrid journals—journals that publish both OA and non-OA articles—have higher citation rates than those published in fully OA journals (Hadad et al., 2024; Maddi & Sapinho, 2022). This challenges the idea that fully OA journals gain a citation advantage by signaling their OA status to potential readers. On the other hand, it can also be attributed to a high degree of self-selection—that authors choose to publish only their higher-impact papers as OA articles in hybrid journals.

3 Methods and materials

3.1 CABS ratings

While at least five expert ratings of business journals are in current use, CABS is perhaps the most widely adopted by universities and other organizations that evaluate research contributions in business and economics. Moreover, CABS is the expert rating most closely related to the other four, based on the correlations among them (Walters, 2024). That is, CABS best represents the set of all five ratings.
The CABS ratings, assigned in March 2021, are based on the opinions of 53 experts appointed by the Chartered Association of Business Schools (UK), who consulted with other researchers as well as representatives of scholarly and professional societies. The rating criteria include reputation, selectivity, perceived scholarly impact, adherence to high methodological standards, and rigor and transparency of peer review. Raters were provided with a range of citation data, but those data were not formally incorporated into the ratings. The final ratings, from 4* (most important) to 1 (least important), were assigned after discussion and consensus among the raters. Overall, 3% of the CABS journals were assigned ratings of 4* (recoded as 5 in the current study). Six percent have ratings of 4, 19% have ratings of 3, 32% have ratings of 2, and 41% have ratings of 1 (Chartered Association of Business Schools, 2021).

3.2 Citation metrics

Four citation metrics were selected for this investigation. Two of them, 5IF (five-year Impact Factor) and CiteScore, are among the indicators most often reported by publishers and used by research evaluators. Two others, AI (Article Influence score) and SJR (SCImago Journal Rank), are the citation metrics most closely related to CABS (Walters, 2024). 5IF and CiteScore are straightforward; they represent, essentially, the average number of citations accrued in the 1-3 years since publication. In contrast, AI and SJR are weighted and normalized; they account for both (a) the impact of the journals that cite each article and (b) differences in citation rates across fields of study (Bornmann, 2020; González-Pereira et al., 2010; Guerrero-Bote & Moya-Anegón, 2012; SCImago Research Group, 2007; Walters, 2017a; West et al., 2010, 2013). Further details about the CABS ratings and the citation metrics, including frequency distributions and correlations, are presented elsewhere (Walters, 2024).

3.3 OA and non-OA journals

The journals included in the analysis are those covered by both CABS and either Journal Citation Reports (Web of Science) or Scopus (Chartered Association of Business Schools, 2021; Clarivate, 2023; Elsevier, 2023). The selection standards of Web of Science and Scopus ensure that predatory or marginal journals are excluded, and none of the OA journals in this analysis appear on Beall’s List (Beall, 2021; Clarivate, 2024; Meester, 2021).
Within the set of 1,561 journals, the 85 OA journals are those that make both current issues and back issues freely available online. Journals with optional or hybrid OA status—those in which only some articles are freely available—were not counted as OA, nor were “subscribe to open” journals, where the journal becomes freely accessible only when a particular number of subscriptions have been sold and maintained. Likewise, journals that adopted OA status in 2021 or later were not regarded as OA, since they were still subscription journals at the time of the CABS evaluations. Information on OA adoption dates was obtained from the Directory of Open Access Journals, the journals’ web sites, and other online sources.

3.4 Regression analyses

Four regressions were conducted for the full set of journals. Each used a single citation metric—5IF, CiteScore, AI, or SJR—and an OA dummy variable (OA or non-OA) to predict CABS. The goal was not to identify the determinants of CABS ratings, but to evaluate the impact of OA status. Likewise, four regressions were conducted for the set of business journals (i.e., excluding those in the CABS economics, general psychology, and social science categories) and four for the set of economics journals.
In each regression, the citation metrics were entered in square root or log10 form. This approach, adopted by several recent authors, was undertaken to maintain linearity, to ensure the best fit, and to account for the strong right skew of the citation distribution (Bradshaw & Brook, 2016; Saarela & Kärkkäinen, 2020; Thelwall, 2022; Thelwall et al., 2023; Walters, 2017b). For this investigation, the substantive results are the same regardless of whether the citation metrics are transformed.
Although we cannot assume measurement invariance across the five CABS ratings—that the difference between ratings of 1 and 2 is equal to the difference between ratings of 4 and 5, for instance—there is good justification for treating CABS as a continuous variable in the regressions. There are three reasons for this. First, the underlying constructs (such as quality or reputation) represented by scores that combine individuals’ ratings may be continuous even when the actual rating categories are not. Second, there is reliable evidence that violations of measurement invariance do not necessarily invalidate the results of techniques such as OLS regression. Finally, the use of methods meant for ordinal variables would result in the loss of valuable information about the shape of the variables’ distributions (Robitzsch, 2020, 2022; Robitzsch & Lüdtke, 2023).
Because the study data include the entire population—all the journals for which both CABS ratings and citation data are available—there is no need to draw conclusions about the population on the basis of sample characteristics (i.e., no need for significance tests). Nonetheless, nearly all the results achieve statistical significance if we assume that our data refer to a larger hypothetical population. The significance levels reported in the regression results are for Student’s t test, two-tailed.

4 Results

4.1 All CABS journals

Tables 1-5 present descriptive statistics and regression results for the set of all CABS journals—the journals in all 22 CABS subject categories. The OA variable was coded 1 for OA journals and 0 for non-OA journals.
Table 1. Descriptives for all CABS journals—the journals in all 22 CABS subject categories.
Variable Non-OA
Mean
Non-OA
SD
Non-OA
N
OA
Mean
OA
SD
OA
N
OA 0.00 0.00 1,476 1.00 0.00 85
CABS 2.06 1.04 1,476 1.32 0.69 85
5IF 4.01 3.33 1,242 2.45 2.76 59
CiteScore 5.50 4.52 1,433 2.96 2.57 80
AI 1.30 1.84 1,242 0.75 1.71 59
SJR 1.34 2.03 1,437 0.63 1.06 77
Table 2. Regression results: Predicting CABS using sqrt(5IF) and OA—all CABS journals. R2=0.30, SEE=0.88, n=1,301.
Variable b SE Beta Sig.
Sqrt(5IF) 0.770 0.034 0.53 < 0.01
OA -0.432 0.119 -0.09 < 0.01
Y-intercept 0.735 0.069 < 0.01
Table 3. Regression results: Predicting CABS using sqrt(CiteScore) and OA—all CABS journals. R2=0.24, SEE=0.90, n=1,513.
Variable b SE Beta Sig.
Sqrt(CiteScore) 0.577 0.028 0.47 < 0.01
OA -0.410 0.105 -0.09 < 0.01
Y-intercept 0.807 0.065 < 0.01
Table 4. Regression results: Predicting CABS using log10(AI) and OA—all CABS journals. R2=0.53, SEE=0.73, n=1,301.
Variable b SE Beta Sig.
log10(AI) 1.865 0.050 0.72 < 0.01
OA -0.219 0.098 -0.04 < 0.01
Y-intercept 2.344 0.021 0.03
Table 5. Regression results: Predicting CABS using sqrt(SJR) and OA—all CABS journals. R2=0.52, SEE=0.72, n=1,514.
Variable b SE Beta Sig.
Sqrt(SJR) 1.362 0.035 0.71 < 0.01
OA -0.294 0.085 -0.06 < 0.01
Y-intercept 0.671 0.040 < 0.01
As shown in Table 1, OA journals in business and economics tend to appear considerably lower in the status hierarchy than conventional subscription journals, in terms of both subjective reputation (CABS) and citation impact (5IF, CiteScore, AI, and SJR). The OA/non-OA differential is more pronounced for CABS than for the citation metrics, when expressed in standard deviations.
Moreover, the CABS ratings of the OA journals are consistently lower than the ratings we might expect based solely on the journals’ citation impact. (See the OA coefficients in Tables 2-5.) This trend can be seen across all four citation metrics. For instance, the OA business and economics journals have, on average, actual CABS ratings about 0.43 points lower than those of non-OA journals with the same 5IF scores. A similar differential can be seen for the CiteScore equation, although the effect of OA status is not as great when AI and SJR are used to predict CABS. Taken together, these results show that expert reviewers evaluating OA journals do assign them ratings that are low relative to their citation impact. That is, OA business and economics journals have high citation impact relative to their CABS reputation scores.

4.2 Business journals

The relationships shown in Table 1 persist when only those journals with a clear business focus are included in the analysis—when the journals in the CABS economics, econometrics and statistics; psychology (general); and social sciences categories are excluded. Like Table 1, Table 6 reveals both low CABS ratings and low citation impact for the OA journals. It also shows an OA/non-OA differential that is greater for CABS than for the citation metrics, in terms of standard deviations. And, as before, the mean CABS rating of the OA journals is consistently lower than the mean rating we would expect based solely on the journals’ citation impact. (See Tables 7-10.) On average, the OA business journals have actual CABS ratings about 0.42 points lower than those of non-OA journals with the same 5IF scores. The OA differential is 0.40 points when CiteScore is used as the citation metric, 0.16 when AI is used, and 0.21 when SJR is used. Again, we can see that the OA journals have low subjective ratings relative to their citation impact, and that this relationship persists across all four citation metrics.
Table 6. Descriptives for the CABS business journals—excluding the journals in the CABS economics, econometrics and statistics; psychology (general); and social sciences categories.
Variable Non-OA
Mean
Non-OA
SD
Non-OA
N
OA
Mean
OA
SD
OA
N
OA 0.00 0.00 1,051 1.00 0.00 52
CABS 2.00 1.04 1,051 1.25 0.48 52
5IF 4.34 3.31 845 2.44 2.38 31
CiteScore 5.79 4.61 1,023 2.94 2.48 49
AI 1.12 1.27 845 0.43 0.34 31
SJR 1.24 1.54 1,027 0.47 0.41 46
Table 7. Regression results: Predicting CABS using sqrt(5IF) and OA—CABS business journals. R2=0.31, SEE=0.88, n=876.
Variable b SE Beta Sig.
Sqrt(5IF) 0.805 0.042 0.55 < 0.01
OA -0.418 0.163 -0.07 0.01
Y-intercept 0.576 0.087 < 0.01
Table 8. Regression results: Predicting CABS using sqrt(CiteScore) and OA—CABS business journals. R2=0.26, SEE=0.90, n=1,072.
Variable b SE Beta Sig.
Sqrt(CiteScore) 0.581 0.031 0.49 < 0.01
OA -0.402 0.133 -0.08 0.01
Y-intercept 0.717 0.076 < 0.01
Table 9. Regression results: Predicting CABS using log10(AI) and OA—CABS business journals. R2=0.51, SEE=0.74, n=876.
Variable b SE Beta Sig.
log10(AI) 1.955 0.066 0.71 < 0.01
OA -0.158 0.138 -0.03 0.25
Y-intercept 2.394 0.027 0.01
Table 10. Regression results: Predicting CABS using sqrt(SJR) and OA—CABS business journals. R2=0.54, SEE=0.71, n=1,073.
Variable b SE Beta Sig.
Sqrt(SJR) 1.536 0.045 0.73 < 0.01
OA -0.214 0.108 -0.04 0.05
Y-intercept 0.485 0.050 < 0.01

4.3 Economics journals

Tables 11-15 present very similar results for the journals in the CABS economics, econometrics and statistics category. On average, the OA economics journals have actual CABS ratings about 0.43 points lower than those of non-OA journals with the same 5IF scores. As the tables show, the disparity between citation impact and reputation is even stronger for OA economics journals than for OA business journals.
Table 11. Descriptives for the CABS economics journals—those in the CABS economics, econometrics and statistics category.
Variable Non-OA
Mean
Non-OA
SD
Non-OA
N
OA
Mean
OA
SD
OA
N
OA 0.00 0.00 287 1.00 0.00 26
CABS 2.11 0.98 287 1.46 0.99 26
5IF 2.87 2.53 266 2.19 3.17 22
CiteScore 4.10 3.35 278 2.86 2.94 25
AI 1.68 2.94 266 1.16 2.74 22
SJR 1.71 3.40 278 0.94 1.75 25
Table 12. Regression results: Predicting CABS using sqrt(5IF) and OA—CABS business journals. R2=0.31, SEE=0.83, n=288.
Variable b SE Beta Sig.
Sqrt(5IF) 0.856 0.079 0.53 < 0.01
OA -0.434 0.186 -0.12 0.02
Y-intercept 0.838 0.135 < 0.01
Table 13. Regression results: Predicting CABS using sqrt (CiteScore) and OA—CABS business journals. R2=0.22, SEE=0.89, n=303.
Variable b SE Beta Sig.
Sqrt(CiteScore) 0.627 0.073 0.44 < 0.01
OA -0.446 0.187 -0.12 0.02
Y-intercept 0.939 0.149 < 0.01
Table 14. Regression results: Predicting CABS using log10(AI) and OA—CABS business journals. R2=0.63, SEE=0.61, n=288.
Variable b SE Beta Sig.
log10(AI) 1.705 0.079 0.79 < 0.01
OA -0.181 0.136 -0.05 0.19
Y-intercept 2.290 0.038 < 0.01
Table 15. Regression results: Predicting CABS using sqrt(SJR) and OA—CABS business journals. R2=0.56, SEE=0.67, n=303.
Variable b SE Beta Sig.
Sqrt(SJR) 1.016 0.054 0.73 < 0.01
OA -0.367 0.141 -0.10 0.01
Y-intercept 1.030 0.071 < 0.01

5 Discussion

As noted in previous research, there are at least six dimensions of journal quality: editors’ and publishers’ intentions, adherence to accepted norms of peer review, adherence to other norms of scholarly publishing, scholarly reputation, impact on subsequent scholarship, and impact on teaching and practice (Walters, 2022). This analysis used data for 1,561 business and economics journals to evaluate the relationships between two of those constructs: scholarly reputation (as assessed by expert evaluators) and impact on subsequent scholarship (citation impact). As expected, OA journals have low reputation scores relative to their citation impact—that is, high citation impact relative to their subjectively assessed reputation.
If we regard quality as an attribute of individual articles and “journal quality” as the aggregate of those individual assessments, there are at least two possible explanations for the findings reported here. First, evaluators in business and economics may be biased against OA journals due to the journals’ unconventional business models, the idea that authors (or their institutions) should pay to publish, or the possibility that OA journals might increase their revenue through higher acceptance rates (more papers published and more fees paid) (Borrego, 2023; Dalton et al., 2020; Feenstra & López-Cózar, 2022; Halevi & Walsh, 2021; Hayman, 2016; Segado-Boy et al., 2022; Togia & Korobili, 2014). Second, OA journals in business and economics may have especially high citation impact, relative to their quality, due to their increased readership—the fact that more potential readers (citers) have easy access to each paper (Asai, 2023; Bautista-Puig et al., 2020; Copiello, 2019; Lawrence, 2001; Momeni et al., 2021). Unfortunately, the available data provide no way of determining whether either, or both, of these explanations are supported.
A third possibility also comes to mind: that despite any anti-OA bias, OA journals have especially high citation rates due to authors’ self-selection of articles during the submission process. That is, authors may send their best work, or the work most likely to benefit from a wide readership, to OA journals rather than subscription journals (Davis, 2011; Davis et al., 2008; Gaule & Maystre, 2011; Hadad et al., 2024; Maddi & Sapinho, 2022). This third explanation is not consistent with the evidence presented here, however, since OA journals in business and economics have mean CABS ratings and mean citation impact values lower than those of non-OA journals. When we say that these OA journals have relatively high citation impact, it is only in comparison with their own CABS ratings. Self-selection may result in higher citation rates for the best OA journals, but probably not for OA business or economics journals in general.
Regardless of the evaluator, reader, and author behaviors underlying these findings, the results of this investigation do have two clear implications. First, authors should consider publishing in OA journals when overall readership and citation impact are especially important, or when journal reputation within a particular academic discipline is less important. Second, by replicating this study every few years, we should be able to determine whether citation impact scores and subjective quality assessments of OA and non-OA journals are converging. The regression analyses are relatively easy to undertake, and the OA coefficients provide a useful indicator of the extent to which anti-OA bias (or the citation advantage of OA journals) is diminishing over time—whether OA journals are achieving full acceptance within the scholarly communication system.
The results reported here are specific to the CABS journal ratings, the four citation metrics chosen for the analysis, and a particular set of subject areas. However, there is strong evidence that CABS is closely related to several other expert ratings, and that 5IF, CiteScore, AI, and SJR are representative of other citation metrics that might have been chosen (Walters, 2024). Very similar findings were obtained for all four citation metrics, and the data source—Web of Science (5IF, AI) or Scopus (CiteScore, SJR)—does not seem to influence the results. This last point is important, since Web of Science is considerably more selective than Scopus in the journals it covers. This suggests that similar results are obtained whether the population includes the full range of journals or just the upper portion of the distribution (based on citation impact). Finally, the results of this study may be limited to business and economics, but the methods can be applied to a wide range of subject areas—any disciplines for which expert journal ratings and citation metrics are available.

Conflict of interests

This research was conducted in the absence of any relationships that could be construed as potential conflicts of interest.

Data availability statements

The data compiled for this study will be supplied by the author on request.
[1]
Abramo G., D’Angelo C.A., & Reale E. (2019). Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications? Scientometrics 121(1), 537-554. doi: 10.1007/s11192-019-03184-y

[2]
Antelman K. (2004). Do Open-Access articles have a greater research impact? College & Research Libraries 65( 5), 372-382. doi: 10.5860/crl.65.5.372

[3]
Asai S. (2023). Revisiting methodology for identifying Open Access advantages. Publishing Research Quarterly 39(2), 107-119. doi: 10.1007/s12109-023-09946-0

[4]
Basson I., Blanckenberg J.P., & Prozesky H. (2021). Do Open Access journal articles experience a citation advantage? Results and methodological reflections of an application of multiple measures to an analysis by WoS subject areas. Scientometrics 126(1), 459-484. doi: 10.1007/s11192-020-03734-9

[5]
Bautista-Puig N., Lopez-Illescas C., de Moya-Anegon F., Guerrero-Bote V., & Moed H.F. (2020). Do journals flipping to gold open access show an OA citation or publication advantage? Scientometrics 124(3), 2551-2575. doi: 10.1007/s11192-020-03546-x

[6]
Beall J. (2021). Beall’s list of potential predatory publishers and journals. https://beallslist.net/

[7]
Björk B.C., & Solomon D. (2012). Open Access versus subscription journals: A comparison of scientific impact. BMC Medicine 10, article 73. doi: 10.1186/1741-7015-10-73

[8]
Bornmann L. (2020). How can citation impact in bibliometrics be normalized? A new approach combining citing-side normalization and citation percentiles. Quantitative Science Studies 1(4), 1553-1569. doi: 10.1162/qss_a_00089

[9]
Borrego A. (2023). Article processing charges for Open Access journal publishing: A review. Learned Publishing 36(3), 359-378. doi: 10.1002/leap.1558

[10]
Bradshaw C.J.A., & Brook B.W. (2016). How to rank journals. PLoS ONE 11(3), article e0149852. doi: 10.1371/journal.pone.0149852

[11]
Carvalho Neto S., Willinsky J., & Alperin J.P. (2016). Measuring, rating, supporting, and strengthening Open Access scholarly publishing in Brazil. Education Policy Analysis Archives 24, article 54. doi: 10.14507/epaa.24.2391.24.2391

[12]
Chartered Association of Business Schools. (2021). Academic journal guide 2021. https://charteredabs.org/academic-journal-guide-2021/

[13]
Clarivate. (2023). Journal citation reports. https://clarivate.com/products/scientific-and-academic-research/research-analytics-evaluation-and-management-solutions/journal-citation-reports/

[14]
Clarivate. (2024). Web of Science core collection: Editorial selection process. https://clarivate.com/products/scientific-and-academic-research/research-discovery-and-workflow-solutions/webofscience-platform/web-of-science-core-collection/editorial-selection-process/

[15]
Copiello S. (2019). The open access citation premium may depend on the openness and inclusiveness of the indexing database, but the relationship is controversial because it is ambiguous where the open access boundary lies. Scientometrics 121(2), 995-1018. doi: 10.1007/s11192-019-03221-w

[16]
Dalton E.D., Tenopir C., & Björk B-C. (2020). Attitudes of North American academics toward Open Access scholarly journals. Portal: Libraries and the Academy 20(1), 73-100. doi: 10.1353/pla.2020.0005

[17]
Davis P.M., Lewenstein B.V., Simon D.H., Booth J.G., & Connolly M.J.L. (2008). Open Access publishing, article downloads, and citations: Randomised controlled trial. BMJ 337(7665), article 568. doi: 10.1136/bmj.a568

[18]
Davis P.M. (2011). Open Access, readership, citations: A randomized controlled trial of scientific journal publishing. FASEB Journal 25(7), 2129-2134. doi: 10.1096/fj.11-183988

PMID

[19]
Derrick G.E., Haynes A., Chapman S., & Hall W.D. (2011). The association between four citation metrics and peer rankings of research influence of Australian researchers in six fields of public health. PLoS ONE 6(4), article 18521. doi: 10.1371/journal.pone.0018521

[20]
Dorta-González P., & Dorta-González M.I. (2022). Contribution of the Open Access modality to the impact of hybrid journals controlling by field and time effects. Journal of Data and Information Science 7(2), 57-83. doi: 10.2478/jdis-2022-0007

[21]
Dorta-González P., & Dorta-González M.I. (2023a). Citation differences across research funding and access modalities. Journal of Academic Librarianship 49(4), article 102734. doi: 10.1016/j.acalib.2023.102734

[22]
Dorta-González P., & Dorta-González M.I. (2023b). The funding effect on citation and social attention: The UN Sustainable Development Goals (SDGs) as a case study. Online Information Review 47(7), 1358-1376. doi: 10.1108/OIR-05-2022-0300

[23]
Elsevier. (2023). Scopus: Sources. https://www.scopus.com/sources.uri

[24]
Eysenbach G. (2006). Citation advantage of open access articles. PLoS Biology 4(5), article 157. doi: 10.1371/journal.pbio.0040157

[25]
Feenstra R.A., & López-Cózar E.D. (2022). Philosophers’ perceptions of pay to publish and open access in Spain: Books versus journals, more than a financial dilemma. Learned Publishing 35(2), 118-129. doi: 10.1002/leap.1426

[26]
Gargouri Y., Hajjem C., Larivière V., Gingras Y., Carr L., Brody T., & Harnad S. (2010). Self-selected or mandated, Open Access increases citation impact for higher quality research. PloS ONE 5(10), article 13636. doi: 10.1371/journal.pone.0013636

[27]
Gaule P., & Maystre N. (2011). Getting cited: Does Open Access help? Research Policy 40(10), 1332-1338. doi: 10.1016/j.respol.2011.05.025

[28]
González-Pereira B., Guerrero-Bote V.P., & Moya-Anegón F. (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of Informetrics 4(3), 379-391. doi: 10.1016/j.joi.2010.03.002

[29]
Guba K., & Tsivinskaya A. (2023). Expert judgments versus publication-based metrics: Do the two methods produce identical results in measuring academic reputation? Journal of Documentation 79(1), 127-143. doi: 10.1108/JD-02-2022-0039

[30]
Guerrero-Bote V.P., & Moya-Anegón F. (2012). A further step forward in measuring journals’ scientific prestige: The SJR2 indicator. Journal of Informetrics 6(4), 674-688. doi: 10.1016/j.joi.2012.07.001

[31]
Hadad S., Aharony N., & Raban D.R. (2024). Policy shaping the impact of Open-Access publications: A longitudinal assessment. Scientometrics 129(1), 237-260. doi: 10.1007/s11192-023-04875-3

[32]
Hadad S., & Aharony N. (2023). Open Access advantages as a function of the discipline: Mixed-methods study. Journal of Academic Librarianship 49(4), article 102746. doi: 10.1016/j.acalib.2023.102746

[33]
Haddawy P., Hassan S.U., Asghar A., & Amin S. (2016). A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality. Journal of Informetrics 10(1), 162-173. doi: 10.1016/j.joi.2015.12.005

[34]
Halevi G., & Walsh S. (2021). Faculty attitudes towards article processing charges for Open Access articles. Publishing Research Quarterly 37(3), 384-398. doi: 10.1007/s12109-021-09820-x

[35]
Hall C.M. (2011). Publish and perish? Bibliometric analysis, journal ranking and the assessment of research quality in tourism. Tourism Management 32(1), 16-27. doi: 10.1016/j.tourman.2010.07.001

[36]
Hayman R. (2016). Some LIS faculty indicate reservations about Open Access. Evidence Based Library and Information Practice 11(3), 96-98. doi: 10.18438/B8890B

[37]
Hodge D.R., Yu M., & Kim A. (2021). Ranking disciplinary social work journals: Comparing faculty perceptions with two citation-based approaches. Journal of the Society for Social Work and Research 12(1), 109-129. doi: 10.1086/713306

[38]
Hubbard D.E. (2017). Open Access citation advantage? A local study at a large research university. Proceedings of the Association for Information Science and Technology 54(1), 712-713. doi: 10.1002/pra2.2017.14505401126

[39]
Khan D., Ashar M., & Yuvaraj M. (2023). Do Open Access journals have a greater citation impact? A study of journals in library and information science. Collection and Curation 42(1), 13-24. doi: 10.1108/CC-03-2022-0010

[40]
Koler-Povh T., Južnič P., & Turk G. (2014). Impact of Open Access on citation of scholarly publications in the field of civil engineering. Scientometrics 98(2), 1033-1045. doi: 10.1007/s11192-013-1101-x

[41]
Kulczycki E., & Rozkosz E.A. (2017). Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland. Scientometrics 111(1), 417-442. doi: 10.1007/s11192-017-2261-x

PMID

[42]
Lawrence S. (2001). Free online availability substantially increases a paper’s impact. Nature 411(6837), 521. doi: 10.1038/35079151

[43]
Maddi A., & Sapinho D. (2022). Article processing charges, altmetrics and citation impact: Is there an economic rationale? Scientometrics 127(12), 7351-7368. doi: 10.1007/s11192-022-04284-y

[44]
Mahmood K. (2017). Correlation between perception-based journal rankings and the journal impact factor (JIF): A systematic review and meta-analysis. Serials Review 43(2), 120-129. doi: 10.1080/00987913.2017.1290483

[45]
Meester W. (2021). The guardians of Scopus. https://www.elsevier.com/connect/the-guardians-of-scopus

[46]
Ming W., & Zhao Z. (2022). Rethinking the open access citation advantage: Evidence from the “reverse‐flipping” journals. Journal of the Association for Information Science and Technology 73(11), 1608-1620. doi: 10.1002/asi.24699

[47]
Momeni F., Mayr P., Fraser N., & Peters I. (2021). What happens when a journal converts to open access? A bibliometric analysis. Scientometrics 126(12), 9811-9827. doi: 10.1007/s11192-021-03972-5

[48]
Piwowar H., Priem J., Larivière V., Alperin J.P., Matthias L., Norlander B., Farley A., West J., & Haustein S. (2018). The state of OA: A large-scale analysis of the prevalence and impact of Open Access articles. PeerJ 6, article e4375. doi: 10.7717/peerj.4375

[49]
Robitzsch A., & Lüdtke O. (2023). Why full, partial, or approximate measurement invariance are not a prerequisite for meaningful and valid group comparisons. Structural Equation Modeling 30(6), 859-870. doi: 10.1080/10705511.2023.2191292

[50]
Robitzsch A. (2020). Why ordinal variables can (almost) always be treated as continuous variables: Clarifying assumptions of robust continuous and ordinal factor analysis estimation methods. Frontiers in Education 5, article 589965. doi: 10.3389/feduc.2020.589965

[51]
Robitzsch A. (2022). On the bias in confirmatory factor analysis when treating discrete variables as ordinal instead of continuous. Axioms 11(4), article 162. doi: 10.3390/axioms11040162

[52]
Saarela M., & Kärkkäinen T. (2020). Can we automate expert-based journal rankings? Analysis of the Finnish publication indicator. Journal of Informetrics 14(2), article 101008. doi: 10.1016/j.joi.2020.101008

[53]
Schroter S., & Tite L. (2006). Open Access publishing and author-pays business models: A survey of authors’ knowledge and perceptions. Journal of the Royal Society of Medicine 99(3), 141-148. doi:10.1177/014107680609900316

PMID

[54]
SCImago Research Group. (2007). Description of SCImago journal rank indicator. https://www.scimagojr.com/SCImagoJournalRank.pdf

[55]
Segado-Boj F., Prieto-Gutiérrez J.-J., & Martín-Quevedo J. (2022). Attitudes, willingness, and resources to cover article publishing charges: The influence of age, position, income level country, discipline and open access habits. Learned Publishing 35(4): 489-498. doi: 10.1002/leap.1455

[56]
Sotudeh H. (2020). Does Open Access citation advantage depend on paper topics? Journal of Information Science 46(5), 696-709. doi: 10.1177/0165551519865489

[57]
Szluka P., Csajbók E., & Győrffy B. (2023). Relationship between bibliometric indicators and university ranking positions. Scientific Reports 13, article 14193. doi: 10.1038/s41598-023-35306-1

[58]
Thelwall M. (2022). Can the quality of published academic journal articles be assessed with machine learning? Quantitative Science Studies 3(1), 208-226. doi: 10.1162/qss_a_00185

[59]
Thelwall M., Kousha K., Makita M., Abdoli M., Stuart E., Wilson P., & Levitt J. (2023). In which fields do higher impact journals publish higher quality articles? Scientometrics 128(7), 3915-3933. doi: 10.1007/s11192-023-04735-0

[60]
Togia A., & Korobili S. (2014). Attitudes towards Open Access: A meta-synthesis of the empirical literature. Information Services & Use 34( 3-4), 221-231. doi: 10.3233/ISU-140742

[61]
Walters W.H. (2017a). Citation-based journal rankings: Key questions, metrics, and data sources. IEEE Access 5, 22036-22053. doi: 10.1109/ACCESS.2017.2761400

[62]
Walters W.H. (2017b). Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact? Journal of Informetrics 11(3), 730-744. doi: 10.1016/j.joi.2017.05.001

[63]
Walters W.H. (2022). The citation impact of the Open Access accounting journals that appear on Beall’s List of potentially predatory publishers and journals. Journal of Academic Librarianship 48(1), article 102484. doi: 10.1016/j.acalib.2021.102484

[64]
Walters W.H. (2024). Relationships between expert ratings of business/economics journals and key citation metrics: The impact of size-independence, citing-journal weighting, and subject-area normalization. Journal of Academic Librarianship 50(4), article 102882. doi: 10.1016/j.acalib.2024.102882

[65]
Walters W.H., & Markgren S. (2019). Do faculty journal selections correspond to objective indicators of citation impact? Results for 20 academic departments at Manhattan College. Scientometrics 118(1), 321-337. doi: 10.1007/s11192-018-2972-7

[66]
Wang H.Y., Wei S.X., Cao C., Wang X., & Ye F.Y. (2021). Scientific value weights more than being open or toll access: An analysis of the OA advantage in Nature and Science. Journal of Data and Information Science 6(4), 62-75. doi: 10.2478/jdis-2021-0033

[67]
Wang M., Zhang J., Chen G., & Chai K.-H. (2019). Examining the influence of Open Access on journals’ citation obsolescence by modeling the actual citation process. Scientometrics 119(3), 1621-1641. doi: 10.1007/s11192-019-03085-0

[68]
Weitzner A.S., Davis M., Han A.H., Liu O.O., Patel A.B., Sites B.D., & Cohen S.P. (2024). How predictive is peer review for gauging impact? The association between reviewer rating scores, publication status, and article impact measured by citations in a pain subspecialty journal. Regional Anesthesia & Pain Medicine, advance online publication. doi: 10.1136/rapm-2024-105490

[69]
Wenaas L. (2022). Choices of immediate Open Access and the relationship to journal ranking and publish-and-read deals. Frontiers in Research Metrics and Analytics 7, article 943932. doi: 10.3389/frma.2022.943932

[70]
West J.D., Bergstrom T.C., & Bergstrom C.T. (2010). The Eigenfactor metrics: A network approach to assessing scholarly journals. College & Research Libraries 71(3), 236-244. doi: 10.5860/0710236

[71]
West J.D., Jensen M.C., Dandrea R.J., Gordon G.J., & Bergstrom C.T. (2013). Author-level Eigenfactor metrics: Evaluating the influence of authors, institutions, and countries within the social science research network community. Journal of the American Society for Information Science and Technology 64(4), 787-801. doi: 10.1002/asi.22790

[72]
Wray K. (2016). No new evidence for a citation benefit for author-pay Open Access publications in the social sciences and humanities. Scientometrics 106(3), 1031-1035. doi: 10.1007/s11192-016-1833-5

Outlines

/

京ICP备05002861号-43

Copyright © 2023 All rights reserved Journal of Data and Information Science

E-mail: jdis@mail.las.ac.cn Add:No.33, Beisihuan Xilu, Haidian District, Beijing 100190, China

Support by Beijing Magtech Co.ltd E-mail: support@magtech.com.cn