Research Papers

Gauging scholars' acceptance of Open Access journals by examining the relationship between perceived quality and citation impact

Expand
  • Mary Alice & Tom O'Malley Library, Manhattan University, Riverdale, NY 10471, USA
† William H. Walters (Email: william.walters@manhattan.edu; ORCID: 0000-0001-9490-4032).

Received date: 2024-08-31

  Revised date: 2024-11-01

  Accepted date: 2024-11-07

  Online published: 2024-11-14

Abstract

Purpose: For a set of 1,561 Open Access (OA) and non-OA journals in business and economics, this study evaluates the relationships between four citation metrics—five-year Impact Factor (5IF), CiteScore, Article Influence (AI) score, and SCImago Journal Rank (SJR)—and the journal ratings assigned by expert reviewers. We expect that the OA journals will have especially high citation impact relative to their perceived quality (reputation).
Design/methodology/approach: Regression is used to estimate the ratings assigned by expert reviewers for the 2021 CABS (Chartered Association of Business Schools) journal assessment exercise. The independent variables are the four citation metrics, evaluated separately, and a dummy variable representing the OA/non-OA status of each journal.
Findings: Regardless of the citation metric used, OA journals in business and economics have especially high citation impact relative to their perceived quality (reputation). That is, they have especially low perceived quality (reputation) relative to their citation impact.
Research limitations: These results are specific to the CABS journal ratings and the four citation metrics. However, there is strong evidence that CABS is closely related to several other expert ratings, and that 5IF, CiteScore, AI, and SJR are representative of the other citation metrics that might have been chosen.
Practical implications: There are at least two possible explanations for these results: (1) expert evaluators are biased against OA journals, and (2) OA journals have especially high citation impact due to their increased accessibility. Although this study does not allow us to determine which of these explanations are supported, the results suggest that authors should consider publishing in OA journals whenever overall readership and citation impact are more important than journal reputation within a particular field. Moreover, the OA coefficients provide a useful indicator of the extent to which anti-OA bias (or the citation advantage of OA journals) is diminishing over time.
Originality/value: This is apparently the first study to investigate the impact of OA status on the relationships between expert journal ratings and journal citation metrics.

Cite this article

William H. Walters . Gauging scholars' acceptance of Open Access journals by examining the relationship between perceived quality and citation impact[J]. Journal of Data and Information Science, 0 : 1 -1 . DOI: 10.2478/jdis-2025-0002

References

[1] Abramo G., D'Angelo C.A., & Reale E. (2019). Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications? Scientometrics 121(1), 537-554. doi: 10.1007/s11192-019-03184-y
[2] Antelman, K. (2004). Do Open-Access articles have a greater research impact? College & Research Libraries 65(5), 372-382. doi: 10.5860/crl.65.5.372
[3] Asai, S. (2023). Revisiting methodology for identifying Open Access advantages. Publishing Research Quarterly 39(2), 107-119. doi: 10.1007/s12109-023-09946-0
[4] Basson I., Blanckenberg J.P., & Prozesky H. (2021). Do Open Access journal articles experience a citation advantage? Results and methodological reflections of an application of multiple measures to an analysis by WoS subject areas. Scientometrics 126(1), 459-484. doi: 10.1007/s11192-020-03734-9
[5] Bautista-Puig N., Lopez-Illescas C., de Moya-Anegon F., Guerrero-Bote V., & Moed H.F. (2020). Do journals flipping to gold open access show an OA citation or publication advantage? Scientometrics 124(3), 2551-2575. doi: 10.1007/s11192-020-03546-x
[6] Beall, J. (2021). Beall's list of potential predatory publishers and journals. https://beallslist.net/
[7] Björk, B.C., & Solomon, D. (2012). Open Access versus subscription journals: A comparison of scientific impact. BMC Medicine 10, article 73. doi: 10.1186/1741-7015-10-73
[8] Bornmann, L. (2020). How can citation impact in bibliometrics be normalized? A new approach combining citing-side normalization and citation percentiles. Quantitative Science Studies 1(4), 1553-1569. doi: 10.1162/qss_a_00089
[9] Borrego, A. (2023). Article processing charges for Open Access journal publishing: A review. Learned Publishing 36(3), 359-378. doi: 10.1002/leap.1558
[10] Bradshaw, C.J.A., & Brook, B.W. (2016). How to rank journals. PLoS ONE 11(3), article e0149852. doi: 10.1371/journal.pone.0149852
[11] Carvalho Neto S., Willinsky J., & Alperin J.P. (2016). Measuring, rating, supporting, and strengthening Open Access scholarly publishing in Brazil. Education Policy Analysis Archives 24, article 54. doi: 10.14507/epaa.24.2391
[12] Chartered Association of Business Schools. (2021). Academic journal guide 2021. https://charteredabs.org/academic-journal-guide-2021/
[13] Clarivate. (2023). Journal citation reports. https://clarivate.com/products/scientific-and-academic-research/research-analytics-evaluation-and-management-solutions/journal-citation-reports/
[14] Clarivate. (2024). Web of Science core collection: Editorial selection process. https://clarivate.com/products/scientific-and-academic-research/research-discovery-and-workflow-solutions/webofscience-platform/web-of-science-core-collection/editorial-selection-process/
[15] Copiello, S. (2019). The open access citation premium may depend on the openness and inclusiveness of the indexing database, but the relationship is controversial because it is ambiguous where the open access boundary lies. Scientometrics 121(2), 995-1018. doi: 10.1007/s11192-019-03221-w
[16] Dalton E.D., Tenopir C., & Björk B-C. (2020). Attitudes of North American academics toward Open Access scholarly journals. Portal: Libraries and the Academy 20(1), 73-100. doi: 10.1353/pla.2020.0005
[17] Davis P.M., Lewenstein B.V., Simon D.H., Booth J.G., & Connolly M.J.L. (2008). Open Access publishing, article downloads, and citations: Randomised controlled trial. BMJ 337(7665), article 568. doi: 10.1136/bmj.a568
[18] Davis, P.M. (2011). Open Access, readership, citations: A randomized controlled trial of scientific journal publishing. FASEB Journal 25(7), 2129-2134. doi: 10.1096/fj.11-183988
[19] Derrick G.E., Haynes A., Chapman S., & Hall W.D. (2011). The association between four citation metrics and peer rankings of research influence of Australian researchers in six fields of public health. PLoS ONE 6(4), article 18521. doi: 10.1371/journal.pone.0018521
[20] Dorta-González, P., & Dorta-González, M.I. (2022). Contribution of the Open Access modality to the impact of hybrid journals controlling by field and time effects. Journal of Data and Information Science 7(2), 57-83. doi: 10.2478/jdis-2022-0007
[21] Dorta-González, P., & Dorta-González, M.I. (2023a). Citation differences across research funding and access modalities. Journal of Academic Librarianship 49(4), article 102734. doi: 10.1016/j.acalib.2023.102734
[22] Dorta-González, P., & Dorta-González, M.I. (2023b). The funding effect on citation and social attention: The UN Sustainable Development Goals (SDGs) as a case study. Online Information Review 47(7), 1358-1376. doi: 10.1108/OIR-05-2022-0300
[23] Elsevier. (2023). Scopus: Sources. https://www.scopus.com/sources.uri
[24] Eysenbach, G. (2006). Citation advantage of open access articles. PLoS Biology 4(5), article 157. doi: 10.1371/journal.pbio.0040157
[25] Feenstra R.A.,& López-Cózar, E.D. (2022). Philosophers' perceptions of pay to publish and open access in Spain: Books versus journals, more than a financial dilemma. Learned Publishing 35(2), 118-129. doi: 10.1002/leap.1426
[26] Gargouri Y., Hajjem C., Larivière V., Gingras Y., Carr L., Brody T., & Harnad S. (2010). Self-selected or mandated, Open Access increases citation impact for higher quality research. PloS ONE 5(10), article 13636. doi: 10.1371/journal.pone.0013636
[27] Gaule, P., & Maystre, N. (2011). Getting cited: Does Open Access help? Research Policy 40(10), 1332-1338. doi: 10.1016/j.respol.2011.05.025
[28] González-Pereira B., Guerrero-Bote V.P., & Moya-Anegón F. (2010). A new approach to the metric of journals' scientific prestige: The SJR indicator. Journal of Informetrics 4(3), 379-391. doi: 10.1016/j.joi.2010.03.002
[29] Guba, K., & Tsivinskaya, A. (2023). Expert judgments versus publication-based metrics: Do the two methods produce identical results in measuring academic reputation? Journal of Documentation 79(1), 127-143. doi: 10.1108/JD-02-2022-0039
[30] Guerrero-Bote, V.P., & Moya-Anegón, F. (2012). A further step forward in measuring journals' scientific prestige: The SJR2 indicator. Journal of Informetrics 6(4), 674-688. doi: 10.1016/j.joi.2012.07.001
[31] Hadad S., Aharony N., & Raban D.R. (2024). Policy shaping the impact of Open-Access publications: A longitudinal assessment. Scientometrics 129(1), 237-260. doi: 10.1007/s11192-023-04875-3
[32] Hadad, S., & Aharony, N. (2023). Open Access advantages as a function of the discipline: Mixed-methods study. Journal of Academic Librarianship 49(4), article 102746. doi: 10.1016/j.acalib.2023.102746
[33] Haddawy P., Hassan S.U., Asghar A., & Amin S. (2016). A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality. Journal of Informetrics 10(1), 162-173. doi: 10.1016/j.joi.2015.12.005
[34] Halevi G.,& Walsh, S. (2021). Faculty attitudes towards article processing charges for Open Access articles. Publishing Research Quarterly 37(3), 384-398. doi: 10.1007/s12109-021-09820-x
[35] Hall, C.M. (2011). Publish and perish? Bibliometric analysis, journal ranking and the assessment of research quality in tourism. Tourism Management 32(1), 16-27. doi: 10.1016/j.tourman.2010.07.001
[36] Hayman, R. (2016). Some LIS faculty indicate reservations about Open Access. Evidence Based Library and Information Practice 11(3), 96-98. doi: 10.18438/B8890B
[37] Hodge D.R., Yu M., & Kim A. (2021). Ranking disciplinary social work journals: Comparing faculty perceptions with two citation-based approaches. Journal of the Society for Social Work and Research 12(1), 109-129. doi: 10.1086/713306
[38] Hubbard, D.E. (2017). Open Access citation advantage? A local study at a large research university. Proceedings of the Association for Information Science and Technology 54(1), 712-713. doi: 10.1002/pra2.2017.14505401126
[39] Khan D., Ashar M., & Yuvaraj M. (2023). Do Open Access journals have a greater citation impact? A study of journals in library and information science. Collection and Curation 42(1), 13-24. doi: 10.1108/CC-03-2022-0010
[40] Koler-Povh T., Južnič P., & Turk G. (2014). Impact of Open Access on citation of scholarly publications in the field of civil engineering. Scientometrics 98(2), 1033-1045. doi: 10.1007/s11192-013-1101-x
[41] Kulczycki, E., & Rozkosz, E.A. (2017). Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland. Scientometrics 111(1), 417-442. doi: 10.1007/s11192-017-2261-x
[42] Lawrence, S. (2001). Free online availability substantially increases a paper's impact. Nature 411(6837), 521. doi: 10.1038/35079151
[43] Maddi, A., & Sapinho, D. (2022). Article processing charges, altmetrics and citation impact: Is there an economic rationale? Scientometrics 127(12), 7351-7368. doi: 10.1007/s11192-022-04284-y
[44] Mahmood, K. (2017). Correlation between perception-based journal rankings and the journal impact factor (JIF): A systematic review and meta-analysis. Serials Review 43(2), 120-129. doi: 10.1080/00987913.2017.1290483
[45] Meester, W. (2021). The guardians of Scopus. https://www.elsevier.com/connect/the-guardians-of-scopus
[46] Ming, W., & Zhao, Z. (2022). Rethinking the open access citation advantage: Evidence from the “reverse‐flipping” journals. Journal of the Association for Information Science and Technology 73(11), 1608-1620. doi: 10.1002/asi.24699
[47] Momeni F., Mayr P., Fraser N., & Peters I. (2021). What happens when a journal converts to open access? A bibliometric analysis. Scientometrics 126(12), 9811-9827. doi: 10.1007/s11192-021-03972-5
[48] Piwowar H., Priem J., Larivière V., Alperin J.P., Matthias L., Norlander B., Farley A., West J., & Haustein S. (2018). The state of OA: A large-scale analysis of the prevalence and impact of Open Access articles. PeerJ 6, article e4375. doi: 10.7717/peerj.4375
[49] Robitzsch A., & Lüdtke, O. (2023). Why full, partial, or approximate measurement invariance are not a prerequisite for meaningful and valid group comparisons. Structural Equation Modeling 30(6), 859-870. doi: 10.1080/10705511.2023.2191292
[50] Robitzsch, A. (2020). Why ordinal variables can (almost) always be treated as continuous variables: Clarifying assumptions of robust continuous and ordinal factor analysis estimation methods. Frontiers in Education 5, article 589965. doi: 10.3389/feduc.2020.589965
[51] Robitzsch, A. (2022). On the bias in confirmatory factor analysis when treating discrete variables as ordinal instead of continuous. Axioms 11(4), article 162. doi: 10.3390/axioms11040162
[52] Saarela, M., & Kärkkäinen, T. (2020). Can we automate expert-based journal rankings? Analysis of the Finnish publication indicator. Journal of Informetrics 14(2), article 101008. doi: 10.1016/j.joi.2020.101008
[53] Schroter, S., & Tite, L. (2006). Open Access publishing and author-pays business models: A survey of authors' knowledge and perceptions. Journal of the Royal Society of Medicine 99(3), 141-148. doi:10.1177/014107680609900316
[54] SCImago Research Group. (2007). Description of SCImago journal rank indicator. https://www.scimagojr.com/SCImagoJournalRank.pdf
[55] Segado-Boj, F., Prieto-Gutiérrez, J.-J., & Martín-Quevedo, J. (2022). Attitudes, willingness, and resources to cover article publishing charges: The influence of age, position, income level country, discipline and open access habits. Learned Publishing 35(4): 489-498. doi: 10.1002/leap.1455
[56] Sotudeh, H. (2020). Does Open Access citation advantage depend on paper topics? Journal of Information Science 46(5), 696-709. doi: 10.1177/0165551519865489
[57] Szluka P., Csajbók E., & Győrffy B. (2023). Relationship between bibliometric indicators and university ranking positions. Scientific Reports 13, article 14193. doi: 10.1038/s41598-023-35306-1
[58] Thelwall, M. (2022). Can the quality of published academic journal articles be assessed with machine learning? Quantitative Science Studies 3(1), 208-226. doi: 10.1162/qss_a_00185
[59] Thelwall M., Kousha K., Makita M., Abdoli M., Stuart E., Wilson P., & Levitt J. (2023). In which fields do higher impact journals publish higher quality articles? Scientometrics 128(7), 3915-3933. doi: 10.1007/s11192-023-04735-0
[60] Togia, A., & Korobili, S. (2014). Attitudes towards Open Access: A meta-synthesis of the empirical literature. Information Services & Use 34(3-4), 221-231. doi: 10.3233/ISU-140742
[61] Walters, W.H. (2017a). Citation-based journal rankings: Key questions, metrics, and data sources. IEEE Access 5, 22036-22053. doi: 10.1109/ACCESS.2017.2761400
[62] Walters, W.H. (2017b). Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact? Journal of Informetrics 11(3), 730-744. doi: 10.1016/j.joi.2017.05.001
[63] Walters, W.H. (2022). The citation impact of the Open Access accounting journals that appear on Beall's List of potentially predatory publishers and journals. Journal of Academic Librarianship 48(1), article 102484. doi: 10.1016/j.acalib.2021.102484
[64] Walters, W.H. (2024). Relationships between expert ratings of business/economics journals and key citation metrics: The impact of size-independence, citing-journal weighting, and subject-area normalization. Journal of Academic Librarianship 50(4), article 102882. doi: 10.1016/j.acalib.2024.102882
[65] Walters, W.H., & Markgren, S. (2019). Do faculty journal selections correspond to objective indicators of citation impact? Results for 20 academic departments at Manhattan College. Scientometrics 118(1), 321-337. doi: 10.1007/s11192-018-2972-7
[66] Wang H.Y., Wei S.X., Cao C., Wang X., & Ye F.Y. (2021). Scientific value weights more than being open or toll access: An analysis of the OA advantage in Nature and Science. Journal of Data and Information Science 6(4), 62-75. doi: 10.2478/jdis-2021-0033
[67] Wang M., Zhang J., Chen G., & Chai K.-H. (2019). Examining the influence of Open Access on journals' citation obsolescence by modeling the actual citation process. Scientometrics 119(3), 1621-1641. doi: 10.1007/s11192-019-03085-0
[68] Weitzner A.S., Davis M., Han A.H., Liu O.O., Patel A.B., Sites B.D., & Cohen S.P. (2024). How predictive is peer review for gauging impact? The association between reviewer rating scores, publication status, and article impact measured by citations in a pain subspecialty journal. Regional Anesthesia & Pain Medicine, advance online publication. doi: 10.1136/rapm-2024-105490
[69] Wenaas, L. (2022). Choices of immediate Open Access and the relationship to journal ranking and publish-and-read deals. Frontiers in Research Metrics and Analytics 7, article 943932. doi: 10.3389/frma.2022.943932
[70] West J.D., Bergstrom T.C., & Bergstrom C.T. (2010). The Eigenfactor metrics: A network approach to assessing scholarly journals. College & Research Libraries 71(3), 236-244. doi: 10.5860/0710236
[71] West J.D., Jensen M.C., Dandrea R.J., Gordon G.J., & Bergstrom C.T. (2013). Author-level Eigenfactor metrics: Evaluating the influence of authors, institutions, and countries within the social science research network community. Journal of the American Society for Information Science and Technology 64(4), 787-801. doi: 10.1002/asi.22790
[72] Wray, K. (2016). No new evidence for a citation benefit for author-pay Open Access publications in the social sciences and humanities. Scientometrics 106(3), 1031-1035. doi: 10.1007/s11192-016-1833-5
Outlines

/

京ICP备05002861号-43

Copyright © 2023 All rights reserved Journal of Data and Information Science

E-mail: jdis@mail.las.ac.cn Add:No.33, Beisihuan Xilu, Haidian District, Beijing 100190, China

Support by Beijing Magtech Co.ltd E-mail: support@magtech.com.cn