Research Papers

Beyond surface correlations: Reference behavior mediates the disruptiveness-citation relationship

Expand
  • School of Information Management, Nanjing University, Nanjing, 210093, China
† Sanhong Deng (Email: sanhong@nju.edu.cn).

Received date: 2025-02-09

  Revised date: 2025-04-21

  Accepted date: 2025-04-24

  Online published: 2025-05-22

Abstract

Purpose: This study examines why papers with high CD indices (measuring research disruptiveness) increasingly show reduced citation impact and investigates whether this represents genuine impact reduction or methodological artifacts.
Design/methodology/approach: We analyzed 29 million papers (1950-2016) using Poisson regression to examine relationships between the CD index and citation count, with controls for fields, team size, and reference count.
Findings: Papers with high CD indices showed reduced citation impact over time. However, when controlling for increasing reference counts in papers, this relationship reversed, revealing a positive association. Papers with more references exhibit lower CD indices owing to the index’s sensitivity to the reference count, while achieving higher citation counts. Alternative innovation metrics consistently show positive correlations with citation impact.
Research limitations: The approach may not adequately capture the reduced citation impact of highly disruptive papers with fewer references. The analysis is limited to journal articles and shows correlation rather than causality.
Practical implications: The apparent undervaluation of disruptive research stems from methodological artifacts in the CD index calculation driven by evolving reference patterns. Researchers should control for the reference count when using this metric.

Cite this article

Alex J. Yang, Fanming Wang, Yujie Shi, Yiqin Zhang, Hao Wang, Sanhong Deng . Beyond surface correlations: Reference behavior mediates the disruptiveness-citation relationship[J]. Journal of Data and Information Science, 0 : 20250029 -20250029 . DOI: 10.2478/jdis-2025-0029

References

[1] Arts S., Melluso N., & Veugelers R. (2025). Beyond citations: Measuring novel scientific ideas and their impact in publication text. The Review of Economics and Statistics, 1-33. https://doi.org/10.1162/rest_a_01561
[2] Bloom N., Jones C. I., Van Reenen J., & Webb M. (2020). Are ideas getting harder to find? American Economic Review, 110(4), 1104-1144. https://doi.org/10.1257/aer.20180338
[3] Bornmann L., Devarakonda S., Tekles A., & Chacko G. (2020a). Are disruption index indicators convergently valid? The comparison of several indicator variants with assessments by peers. Quantitative Science Studies, 1(3), 1242-1259. https://doi.org/10.1162/qss_a_00068
[4] Bornmann L., Devarakonda S., Tekles A., & Chacko G. (2020b). Disruptive papers published in scientometrics: Meaningful results by using an improved variant of the disruption index originally proposed by Wu, Wang, and Evans (2019). Scientometrics, 123(2), 1149-1155. https://doi.org/10.1007/s11192-020-03406-8
[5] Chai S.,& Menon, A. (2019). Breakthrough recognition: Bias against novelty and competition for attention. Research Policy, 48(3), 733-747. https://doi.org/10.1016/j.respol.2018.11.006
[6] Chu, J. S. G., & Evans, J. A. (2021). Slowed canonical progress in large fields of science. Proceedings of the National Academy of Sciences of the United States of America, 118(41), e2021636118. https://doi.org/10.1073/pnas.2021636118
[7] Fortunato S., Bergstrom C. T., Boerner K., Evans J. A., Helbing D., Milojevic S., Petersen A. M., Radicchi F., Sinatra R., Uzzi B., Vespignani A., Waltman L., Wang D., & Barabasi A.-L. (2018). Science of science. Science, 359(6379), eaao0185. https://doi.org/10.1126/science.aao0185
[8] Foster J. G., Rzhetsky A., & Evans J. A. (2015). Tradition and innovation in scientists’ research strategies. American Sociological Review, 80(5), 875-908. https://doi.org/10.1177/0003122415601618
[9] Funk R. J.,& Owen-Smith, J. (2017). A dynamic network measure of technological change. Management Science, 63(3), 791-817. https://doi.org/10.1287/mnsc.2015.2366
[10] Jones, B. F. (2009). The burden of knowledge and the “death of the renaissance man”: Is innovation getting harder? The Review of Economic Studies, 76(1), 283-317. https://doi.org/10.1111/j.1467-937X.2008.00531.x
[11] Jones, B. F., & Weinberg, B. A. (2011). Age dynamics in scientific creativity. Proceedings of the National Academy of Sciences of the United States of America, 108(47), 18910-18914. https://doi.org/10.1073/pnas.1102895108
[12] Jones B. F., Wuchty S., & Uzzi B. (2008). Multi-university research teams: Shifting impact, geography, and stratification in science. Science, 322(5905), 1259-1262. https://doi.org/10.1126/science.1158357
[13] Leibel, C., & Bornmann, L. (2024). What do we know about the disruption index in scientometrics? An overview of the literature. Scientometrics, 129, 601-639. https://doi.org/10.1007/s11192-023-04873-5
[14] Li H., Tessone C. J., & Zeng A. (2024). Productive scientists are associated with lower disruption in scientific publishing. Proceedings of the National Academy of Sciences of the United States of America, 121(21), e2322462121. https://doi.org/10.1073/pnas.2322462121
[15] Li L., Lin Y., & Wu L. (2024). Displacing science. arXiv. https://doi.org/10.48550/arXiv.2402.16839
[16] Lin Y., Evans J. A.,& Wu, L. (2022). New directions in science emerge from disconnection and discord. Journal of Informetrics, 16(1), 101234. https://doi.org/10.1016/j.joi.2021.101234
[17] Lin Y., Frey C. B., & Wu L. (2023). Remote collaboration fuses fewer breakthrough ideas. Nature, 623(7989), 987-991. https://doi.org/10.1038/s41586-023-06767-1
[18] Liu X., Bu Y., Li M., & Li J. (2024). Monodisciplinary collaboration disrupts science more than multidisciplinary collaboration. Journal of the Association for Information Science and Technology, 75(1), 59-78. https://doi.org/10.1002/asi.24840
[19] Macher J. T., Rutzer C.,& Weder, R. (2024). Is there a secular decline in disruptive patents? Correcting for measurement bias. Research Policy, 53(5), 104992. https://doi.org/10.1016/j.respol.2024.104992
[20] Park M., Leahey E., & Funk R. J. (2023). Papers and patents are becoming less disruptive over time. Nature, 613(7942), 138-144. https://doi.org/10.1038/s41586-022-05543-x
[21] Petersen A. M., Arroyave F., & Pammolli F. (2024). The disruption index is biased by citation inflation. Quantitative Science Studies, 5(4), 936-953. https://doi.org/10.1162/qss_a_00333
[22] Radicchi F., Fortunato S., & Castellano C. (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences of the United States of America, 105(45), 17268-17272. https://doi.org/doi:10.1073/pnas.0806977105
[23] Ruan X., Lyu D., Gong K., Cheng Y.,& Li, J. (2021). Rethinking the disruption index as a measure of scientific and technological advances. Technological Forecasting and Social Change, 172, 121071. https://doi.org/10.1016/j.techfore.2021.121071
[24] Uzzi B., Mukherjee S., Stringer M., & Jones B. (2013). Atypical combinations and scientific impact. Science, 342(6157), 468-472. https://doi.org/10.1126/science.1240474
[25] Waltman, L. (2016). A review of the literature on citation impact indicators. Journal of Informetrics, 10(2), 365-391. https://doi.org/10.1016/j.joi.2016.02.007
[26] Wang, J. (2013). Citation time window choice for research impact evaluation. Scientometrics, 94(3), 851-872. https://doi.org/10.1007/s11192-012-0775-9
[27] Wang J., Veugelers R.,& Stephan, P. (2017). Bias against novelty in science: A cautionary tale for users of bibliometric indicators. Research Policy, 46(8), 1416-1436. https://doi.org/10.1016/j.respol.2017.06.006
[28] Wu L. F., Wang D. S., & Evans J. A. (2019). Large teams develop and small teams disrupt science and technology. Nature, 566(7744), 378-382. https://doi.org/10.1038/s41586-019-0941-9
[29] Wuchty S., Jones B. F., & Uzzi B. (2007). The increasing dominance of teams in production of knowledge. Science, 316(5827), 1036-1039. https://doi.org/10.1126/science.1136099
[30] Yang A. J. (2024). Unveiling the impact and dual innovation of funded research. Journal of Informetrics, 18(1), 101480. https://doi.org/10.1016/j.joi.2023.101480
[31] Yang A. J. (2025). Unraveling topic switching and innovation in science. Information Processing & Management, 62(4), 104171. https://doi.org/10.1016/j.ipm.2025.104171
[32] Yang, A. J., & Deng, S. (2024). Dynamic patterns of the disruptive and consolidating knowledge flows in Nobel-winning scientific breakthroughs. Quantitative Science Studies, 1-17. https://doi.org/10.1162/qss_a_00323
[33] Yang A. J., Deng S., Wang H., Zhang Y.,& Yang, W. (2023). Disruptive coefficient and 2-step disruptive coefficient: Novel measures for identifying vital nodes in complex networks. Journal of Informetrics, 17(3), 101411. https://doi.org/10.1016/j.joi.2023.101411
[34] Yang A. J., Gong H., Wang Y., Zhang C., & Deng S. (2024). Rescaling the disruption index reveals the universality of disruption distributions in science. Scientometrics, 129(1), 561-580. https://doi.org/10.1007/s11192-023-04889-x
[35] Yang A. J., Hu H., Zhao Y., Wang H.,& Deng, S. (2023). From consolidation to disruption: A novel way to measure the impact of scientists and identify laureates. Information Processing & Management, 60(5), 103420. https://doi.org/10.1016/j.ipm.2023.103420
[36] Yang A. J., Xu H., Ding Y., & Liu M. (2024). Unveiling the dynamics of team age structure and its impact on scientific innovation. Scientometrics, 129, 6127-6148. https://doi.org/10.1007/s11192-024-04987-4
[37] Yang A. J., Yan X., Hu H., Hu H., Kong J., & Deng S. (2025). Are disruptive papers more likely to impact technology and society? Journal of the Association for Information Science and Technology, 76(3), 563-579. https://doi.org/10.1002/asi.24947
[38] Yang Y., Tian T. Y., Woodruff T. K., Jones B. F., & Uzzi B. (2022). Gender-diverse teams produce more novel and higher-impact scientific ideas. Proceedings of the National Academy of Sciences of the United States of America, 119(36), e2200841119. https://doi.org/10.1073/pnas.2200841119
[39] Zeng A., Fan Y., Di Z., Wang Y., & Havlin S. (2023). Disruptive papers in science are losing impact. arXiv. https://doi.org/10.48550/arXiv.2305.03589
[40] Zeng A., Shen Z. S., Zhou J. L., Wu J. S., Fan Y., Wang Y. G.,& Stanley, H. E. (2017). The science of science: From the perspective of complex systems. Physics Reports, 714, 1-73. https://doi.org/10.1016/j.physrep.2017.10.001
Outlines

/

京ICP备05002861号-43

Copyright © 2023 All rights reserved Journal of Data and Information Science

E-mail: jdis@mail.las.ac.cn Add:No.33, Beisihuan Xilu, Haidian District, Beijing 100190, China

Support by Beijing Magtech Co.ltd E-mail: support@magtech.com.cn