1 Introduction
The impact of research is a crucial aspect in the field of science. While academic impact has long been a primary consideration (Hu et al., 2016), it is now recognized that this is only a fraction of the broader impact of research. There is an increasing call for researchers to articulate the societal value of their work. Since the 1900s, the evaluation of research has expanded to include its societal products, use, and benefits (Mostert et al., 2010). The assessment of impact now encompasses technological, economic, and political outcomes (Bornmann, 2013). As societal impact gains prominence, funding bodies and intermediaries are demanding researchers outline their plans for societal impact (Dance, 2013).
Some fundamental investigations lead to necessary and meaningful contributions to scientific and social progress, a focal point of which is transformative research (TR). TR is also referred to as breakthrough research in the literature. According to the U.S. National Science Foundation, TR involves ideas, discoveries or tools that radically change understanding of existing concepts, create new paradigms or lead to new fields of science, engineering or education. The nature of TR has sparked considerable debate, with its characteristics being summarized from various perspectives. TR, when leading to paradigm shifts, holds the capacity to bring about profound transformations across scientific and sociological realms. In other words, TR possesses the potential to revolutionize established fields, results in radically new technologies, and furnish policymakers with compelling evidence (NSB, 2007). Therefore, it makes sense to investigate the societal impact of TRs. While TR is are relatively rare, it serves as the focal point in this article for two key reasons: firstly, delving into TRs illuminates the intrinsic driving forces behind scientific development; secondly, scientific contributions exhibit a long-tail distribution, wherein a small number of research outcomes contribute to the majority of advancements. TRs, capable of initiating new research disciplines, deserve attention. The Nobel Prize is the most prestigious award in the world, and before research discoveries are awarded the Nobel Prize, the Nobel Committee rigorously evaluates their potential and impact. This study selects Nobel Prize winning publications (NPs) in the fields of gene engineering and astrophysics as examples of TRs (Li et al., 2022) because the transformative nature of Nobel Prize discoveries in these two fields has already been demonstrated. Gene engineering and astrophysics represent distinct realms of academic inquiry (Rafols et al., 2010). Gene engineering, as an applied field, employs biotechnological interventions to directly manipulate organismal genomes, facilitating the modification of cellular genetic material. In contrast, astrophysics serves as a foundational research domain, investigating the physical characteristics of celestial bodies and their interrelations. Application-wise, gene engineering boasts a wide-ranging impact across domains like medicine, agriculture, and life sciences, whereas astrophysics finds more limited applications, predominantly within cosmology and astronomy. Concerning social and ethical dimensions, gene engineering introduces a multitude of considerations due to its involvement in the editing and alteration of life, in contrast to the fewer ethical concerns associated with astrophysics. In terms of the pace of knowledge and technological advancement, gene engineering undergoes continuous updates and rapid development, while astrophysics exhibits a more stable evolutionary trajectory. By examining both of them, we aim to investigate the differences in the technological and policy impacts of TRs of distinct fields.
This contribution focuses on the societal impacts of TR. The emphasis on societal impact resulting from research investments has led to increased political pressure to demonstrate and measure the outcomes of research. Societal impact is defined as the demonstrable contributions that research makes to society, encompassing research output, social awareness through various media, research use and implementation, and societal benefits. However, the understanding of societal impact is limited due to the absence of an accepted framework and quantitative data (Ozanne et al., 2017).
Tracking the path of research on societal impact is challenging, given the time lag between direct outcomes (e.g. articles) and societal benefits. Existing assessments rely heavily on labor-intensive case studies, prompting the call for metrics as proxy measures. Research citations, particularly the number of citations, have been suggested as one such indicator.
This contribution explores two types of societal impact of TRs: impact on technology and on policy. Technological impact is measured by TRs being cited in patents, inspiring technology development. Policy impact, on the other hand, refers to the use of TRs in policy-making processes. While Wilsdon et al. (2015) emphasized that only patent and clinical citations can be used for societal impact, additional indicators, such as policy citations, are deemed essential for a comprehensive impact assessment.
The relationship between knowledge and policy-making has evolved, incorporating concepts like evidence-based policy. Policy citations play a pivotal role in revealing the intricate interaction between science and politics, indicating the research’s “effect on, change, or benefit to policy.” Citation-based approaches have been explored in evaluating impact on technology and policy (Aristodemou et al., 2018; Bornmann et al., 2016). It is important to note that each metric provides only a partial view of the broader societal impact picture. Such analysis of citations has limitations, often focusing solely on direct influences from cited papers (Veugelers et al., 2019). Understanding the indirect impact, which involves disseminating academic knowledge to non-academic audiences through subsequent citing papers (Hu et al., 2018; Olmos-Penuela et al., 2014), is lacking. To address this gap, we’ll examine ego citation networks of NPs and consider both direct and indirect societal impact, spanning two generations of citations (Hu & Rousseau, 2011; Rousseau et al., 2018). This study aims to answer the following questions:
RQ (1): How does NPs’ impact on technology and policy change over generations of citations?
RQ (2): Do these impact characteristics vary across scientific fields?
2 Literature review
2.1 Definition and measurement of societal impact
The definition and measurement of societal impact pose persistent challenges, often referring to demonstrable research contributions to society (ESRC, 2022). Recognized as a complex process (Morton, 2015), demonstrating societal impact is deemed essential (Bornmann et al., 2016). Various payback frameworks have emerged (Martin, 2011). Citation-based approaches, explored in health research (Lewison, 2004; Yin et al., 2021), date back to Narin et al.’s work in the 1980s, revealing knowledge flows from public research to patents, especially in biomedicine (McMillan et al., 2000). Li et al. (2017) support arguments on practical applications, finding NIH grants contributing significantly to patents.
Ke’s recent studies in biomedical science and technology indicate exponential growth in patent-to-paper citations (Ke, 2020). Papers citing diverse fields receive more patent citations (Ke, 2023). Gene engineering research influences policy decisions (Bartkowski et al., 2018), and Belardo et al. (2018) illustrated the impact of biomedical knowledge on health policies.
Astrophysical research, akin to Pasteur’s Quadrant, emphasizes the interaction between basic and applied research, fostering impact diffusion. Venturini et al. (2014) summarized technology transfer in astrophysics, and Petroni et al. (2023) concluded that it generates scientific and technological innovations. Thelwall (2016) explored public interest in astrophysics.
2.2 Direct and indirect citation analysis
Citation-based methods for assessing societal impacts are gaining attention, but they often focus on direct citations. Researchers advocate considering multiple citation generations. Rousseau (1987) is among the earliest to explore both direct and indirect citations. Fragkiadaki et al. (2014) defined citation generations, and Atallah et al. (2006) empirically developed a cumulative citations model. Fragkiadaki et al. (2011) introduced the f-value, incorporating direct and indirect citations. Hu et al. (2012) analyzed patent ego citation networks, proposing indicators based on multiple generations. Recently, Hu et al. (2016) emphasized the significance of indirect citations for research impact. Jones and Hanney (2016) examined five citation generations for four papers, focusing on identifying important citations rather than societal impact assessment.
3 Methodology
3.1 Definition of citation generations
Various citation generation types can be defined in different contexts. For our analysis, we account for potentially overlapping citations. Generations are viewed as multisets, where elements may appear multiple times across different citation generations (Rousseau et al., 2018). Specifically, we designate a NP publication as generation zero (the target article). Publications citing the target article make up the first-generation citations, and publications citing first-generation citations are considered as second-generation citations (Hu et al., 2011).
3.2 Data collection
In this work, we only consider key Nobel Prize publications in gene engineering and astrophysics as a proxy for TRs. Data collecting processes consist of four steps:
Step 1. Identifying Nobel Prizes in gene engineering and astrophysics: Nobel Prizes in Physiology or Medicine (1978, 2007), Chemistry (2020) for gene engineering; Physics (1936, 1974, 1978, 1983, 1993, 2002, 2006, 2011, 2019, 2020) for astrophysics.
Step 2. Searching for key Nobel Prize publications (NPs). For each Nobel laureate, the Nobel official press release will introduce his/her scientific contributions after the Nobel Prize announcements and the laureate will deliver a speech for award ceremony. The key publications related to their contributions written by laureates are usually cited as references to the official press release or the Nobel lecture. Note that NPs must be published before Nobel Prize winning year. Inspired by a reviewer, we present an example to demonstrate the validity of using NPs as a proxy of TRs. The publication (DOI: 10.1126/science.1225829) by Nobel laureates Emmanuelle Charpentier and Jennifer A. Doudna is foundational in its field. It unveils a previously unknown family of endonucleases that utilize dual-RNAs for site-specific DNA cleavage, emphasizing the system’s potential for RNA-programmable genome editing. This article is broadly acknowledged as a fundamental advancement in our understanding of gene editing technologies. In this work, we obtained 92 NPs in gene engineering, and 233 NPs in astrophysics totally. They constitute our initial dataset. In this dataset, we accounted for instances where the same article may simultaneously appear in the laureate’s speech and the official publications of the Nobel Committee, as well as cases where multiple laureates co-authored the same article. As a result, the initial dataset is non-deduplicated, considering multiple counts for the same paper.
Step 3. Removing duplicate NPs as well as those not indexed in the Dimensions. Dimensions integrates extensive bibliometric data (clinical trials, patents, policy documents, altmetrics) with traditional publication and citation data (Herzog et al., 2020). It includes content from Crossref, PubMed, Europe PubMed Central, arXiv, and collaborations with 130 publishers, along with patents from 100+ countries and 50 international patent authorities. Employing a bottom-up approach, it enriches data from diverse sources such as DOAJ, Open Citations, I4OC, clinical trial registries, public policy data, and other Digital Science companies. This broad coverage makes it ideal for examining technology and policy impact. We started by deduplicating repeated NPs, resulting in 74 papers in gene engineering and 202 in astrophysics. Subsequently, we excluded NPs not indexed in the Dimensions, ultimately obtaining 74 NPs in gene engineering and 197 in astrophysics.
Step 4.Downloading information of patents and policies citing NPs and their generations of citations. Dimensions provides dimcli that is an open source Python client for accessing the Dimensions Analytics API. By utilizing dimcli, we downloaded patents and policies citing NPs and their generations of citations (more details can be found at https://api-lab.dimensions.ai/).
Following the above steps, we collected more than 2,000,000 records in total, including more than 1,300,000 publications, 800,000 patents, and 15,000 policies.
3.3 Measurement of impact on technology and policy
We consider an article citation network and focus on one target article the ego, as it is called in network theory (Rousseau, 2011). Based on ego citation network of each NP (as shown in Figure 1 ), we investigate its (in)direct impact on technology and policy.
Figure 1. Direct and indirect impact on technology and policy originating from a NP. |
3.3.1 A network-structural indicator for technological impact
The number of patent citations received by research is not equivalent to its technological impact. Many scholars think that being cited by patents is only an intermediate step to achieving technological impact of an article, rather than the final step (Kim et al., 2017). Instead of counting the number of patent citations, we use the average value of citing patents to assess technological impact of an article.
The value of a patent is complex so that simple citation counting has limitations for comparing potential values of patents across fields. Moreover, neither direct relation with the technological background of a patent nor technological complexity included in a patent could be reflected by citation counting. In our previous article, we introduced the technological span index (TSI), a structural indicator of patent citation network (Hu et al., 2012), to measure the potential value of patents for further innovative development. The TSI takes the subnetwork CR (citation of reference) of a focused patent into account, namely, the references of patent and the number of citations received by those reference items from dynamic networks (as shown in Figure 2 ). The previous study suggested that the TSI could validly indicate the potential technological value of patents for further innovative development. In this work, we use the average TSI value of all citing patents to assess technological impact of an article in this work. It is calculated by,
$\overline{T S I}=\frac{\sum_{i=1}^{n} T S I_{i}}{n}$
Figure 2. The CR subnetwork of patent p citing an article. |
where n is the number of patents citing the target article. The TSI is based on, but different from patent citations. Taking patent p as an example (as shown in Figure 2 ), its TSI is calculated based on the CR (citations of references) subnetwork, which is the subnetwork comprising p’s referenced patents and patents citing these reference items. TSI is calculated by,
$T S I=O I_{P} * T I I$
where OIp means the outgrow index of p and TII denotes the technical interest index of p. OIp reflects the relative position of p in the network comprising p and its reference patents. It is defined as,
$O I_{P}=1-\frac{R_{P}}{T(p)+1}$
where T(p) denotes the number of patents cited by p and RP denotes the rank of p in the ranked list of T(p)+1 elements in decreasing order by the number of patent citations received (Hu et al., 2012).
TII measures the innovative density of technological knowledge flows reflected by patents cited by p. It is defined as the squared root of the total number of citations of all cited patents (see Figure 2 ). Mathematically, it is,
$T I I=\sqrt{C I T_{T(R P)}}$
where CITT(RP) denotes the total number of citations of all cited patents.
We computed the technological impact of more than 1,300,000 publications with Python, and the routine is provided at https://github.com/lx2009xian/TSI_calculation.
3.3.2 Policy impact
Policy impact refers to the use of an article in policy-making processes. We measure the policy impact of an article by policy citations.
3.4 Regression models
To investigate the differences between NPs and their two generations of citations in technological and policy impact, we created two dummy variables to represent three groups, controlling for potential variables to eliminate interference on impact.
For technological impact analysis, we applied a linear regression model as the dependent variable is continuous. The following variables were included as controls:
Publication age: The year interval from publication to 2022. Older articles typically have a longer citation time window, leading to more citations (Wang, 2013).
N-publication citations: The number of citations a publication received. Highly cited publications tend to have a higher technological impact (Veugelers et al., 2019).
Time lag: The time gap between publication and the first patent citation. A shorter time lag is associated with higher technological impact (Ke, 2020).
N-references: The number of references in an article. More references often lead to higher impact (Zhang et al., 2021) and longer reference lists are linked to higher technological impact (Veugelers et al., 2019).
For policy impact analysis, we used a negative binomial regression model to account for the over-dispersed nature of policy citations. We controlled for the following variables:
Publication age: The year interval from publication to 2022 (Wang, 2013).
Publication-type: The type of a publication. Different publication types may have varying numbers of policy citations (Bornmann et al., 2016).
N-research countries: The number of countries collaborating on a publication. International collaborations often result in higher impact (Didegah et al., 2018).
N-publication citations: The number of citations a publication received, as it’s positively correlated with policy citations (Heydari et al., 2019).
N-references: The number of references. More references in an article are associated with higher policy citations (Huang et al., 2022).
4 Results
4.1 Descriptive statistics
Table 1. Descriptive statistics of NPs and their generations of citations. |
| Gene engineering | Astrophysics | ||||||
|---|---|---|---|---|---|---|---|
| NPs | 1st generation | 2nd generation | NPs | 1st generation | 2nd generation | ||
| # Publications | 74 | 29,687 | 854,183 | 197 | 49,497 | 422,840 | |
| # Publications cited by patents | 47 | 4,280 | 81,737 | 4 | 101 | 1,462 | |
| % Publications cited by patents | 63.51% | 14.42% | 9.57% | 2.03% | 0.20% | 0.35% | |
| Average number of patent citations | 130.40 | 103.80 | 106.78 | 1.5 | 2.75 | 6.70 | |
| Average number of citing IPC groups | 7.10 | 4.77 | 4.37 | 3 | 3.31 | 3.41 | |
| # Publications cited by policies | 11 | 500 | 9,008 | 4 | 51 | 745 | |
| % Publications cited by policies | 14.86% | 1.68% | 1.05% | 2.03% | 0.10% | 0.18% | |
| Average number of policy citations | 2.27 | 1.39 | 1.34 | 1 | 1.25 | 1.49 | |
Among scientific publications cited by patents, NPs in gene engineering receive more patent citations and are cited across a broader range of technological fields compared to their citation generations. Conversely, in astrophysics, the trend is reversed. From Table 1 , we also investigate that NPs in both fields tend to receive their first patent citation later than their citing articles.
Regarding policy citations, in gene engineering, 14.86% of NPs are cited by policies, compared to 1.68% for the first generation and 1.05% for the second generation of citations. In astrophysics, these percentages are lower, with values of 2.03%, 0.10%, and 0.18%, respectively.
Among scientific publications cited by policies, NPs in gene engineering receive an average of 2.27 policy citations, surpassing the average for their citing articles. In astrophysics, the average policy citations for NPs are lower than that of their citation generations.
4.2 Direct and indirect impact on technology
In this section, we investigated the impact of NPs and their generations of citations on technology. Figure 3 shows the technological impact of NPs and two generations of citations. Figure 4 shows their differences in technological impact where each dot represents the mean value of technological impact and the vertical line indicates the mean ± standard deviation. Calculated at the mean level, the technological impact for NPs in gene engineering is 9.37, 4.51 for first generation of citation and 3.49 for second generation of citation.
Figure 3. The distribution of technological impact of NPs and their generations of citations. |
Figure 4. The differences of NPs and their generations of citations in technological impact. |
We conducted a series of analyses while controlling for potential confounding factors, as previously described. To ensure the accuracy of model estimates and assess the risk of multicollinearity among variables, we performed variance inflation factor (VIF) tests. The results, presented in Table 2 , indicate that all variables had VIFs lower than 10, suggesting no significant issues with multicollinearity.
Table 2. The VIFs of variables for technological impact. |
| Variables | Gene engineering | Astrophysics |
|---|---|---|
| independent variable (groups) | 1.001 | 1.011 |
| N-categories | 1.000 | 1.017 |
| N-publication citations | 1.010 | 1.094 |
| Time lag | 1.976 | 2.107 |
| N-references | 1.050 | 1.128 |
| Publication age | 2.024 | 2.191 |
We conducted linear regression models with the technological impact as the dependent variable and two dummy variables as the independent variables. The results are displayed in Table 3 .Table 3 and Figure 4a reveal that NPs in gene engineering have a significantly higher technological impact compared to their first generation of citations, and the first generation has a significantly higher technological impact than the second generation. These findings are robust, as they remain consistent after controlling for potential confounding variables (Table 3 , columns 2-7).
Table 3. Linear regression models of direct and indirect impact on technology. |
| Gene engineering | |||||||
|---|---|---|---|---|---|---|---|
| (1) | (2) | (3) | (4) | (5) | (6) | (7) | |
| 1 st generation vs NPs | -4.859*** (1.218) | -4.798*** (1.218) | -4.859*** (1.218) | -4.697*** (1.217) | -4.633*** (1.216) | -2.227* (1.175) | -2.587** (1.168) |
| 2 nd generation vs NPs | -5.882*** (1.212) | -5.816*** (1.212) | -5.881*** (1.212) | -5.725*** (1.211) | -5.626*** (1.210) | -3.423*** (1.169) | -3.817*** (1.162) |
| N-publication citations | 1.040-4*** (3.100-5) | 1.050 -4 *** (3.000 -5 ) | |||||
| N-categories | -0.011 (0.073) | 0.065 (0.070) | |||||
| Time lag | 0.023*** (0.002) | -0.211*** (0.006) | |||||
| N-references | -0.008*** (4.070-4) | -0.001*** (1.116 -3 ) | |||||
| Publication age | 0.232*** (0.003) | 0.324*** (0.004) | |||||
| Constant | 9.373 | 9.286 | 9.384 | 9.048 | 9.601 | 2.282 | 2.337 |
| R 2 | 0.031 | 0.033 | 0.031 | 0.052 | 0.072 | 0.266 | 0.288 |
| N | 883,944 | 883,944 | 883,944 | 883,944 | 883,944 | 883,944 | 883,944 |
| Astrophysics | |||||||
| (8) | (9) | (10) | (11) | (12) | (13) | (14) | |
| 1 st generation vs NPs | -1.263 (3.949) | -1.264 (3.949) | -0.809 (3.949) | -0.624 (3.916) | -1.164 (3.948) | 1.020 (3.777) | 1.303 (3.757) |
| 2 nd generation vs NPs | -1.422 (3.878) | -1.411 (3.878) | -0.922 (3.879) | -0.665 (3.846) | -1.344 (3.877) | 1.509 (3.712) | 1.955 (3.693) |
| N-publication citations | -3.360-4 (3.170-4) | -4.457-5 (3.150-4) | |||||
| N-categories | 0.705** (0.313) | 0.332 (0.300) | |||||
| Time lag | 0.149*** (0.028) | -0.177*** (0.039) | |||||
| N-references | -0.003 (0.002) | 0.002 (0.002) | |||||
| Publication age | 0.197*** (0.016) | 0.276*** (0.023) | |||||
| Constant | 4.531 | 4.596 | 3.121 | 2.813 | 4.603 | -2.116 | -3.459 |
| R 2 | 0.010 | 0.029 | 0.058 | 0.135 | 0.039 | 0.297 | 0.320 |
| N | 472,534 | 472,534 | 472,534 | 472,534 | 472,534 | 472,534 | 472,534 |
Standard errors are in parentheses. ***p<0.01. **p<0.05. *p< 0.1. |
Similar regression analysis was performed for NPs in astrophysics and their citing publications. In this case, the differences in technological impact between NPs in astrophysics and their citation generations are not statistically significant (Figure 4 b and Table 3 ). Even after controlling for several variables (N-publication citations, N-categories, Time lag, N-references, and Publication age), NPs in astrophysics still do not exhibit significantly higher technological impact than subsequent citation generations. It’s worth noting that in astrophysics, technological impact positively correlates with Publication age, indicating earlier publications having a longer citation window and higher technological impact (Ke, 2020).
4.3 Direct and indirect impact on policy
Table 4. The descriptive statistical of policy documents citing NPs and their generations of citations. |
| Gene engineering | Astrophysics | |||||
|---|---|---|---|---|---|---|
| NPs | 1st generation | 2nd generation | NPs | 1st generation | 2nd generation | |
| Publication year | [1968,2017] | [1965,2021] | [1966,2021] | [1980,2016] | [1964,2021] | [1958,2021] |
| Number of countries (Gini) | 4 (0.5808) | 10 (0.7920) | 19 (0.6697) | 1 (1) | 6 (0.5490) | 17 (0.6968) |
| Number of organizations (Gini) | 6 (0.5704) | 28 (0.7525) | 78 (0.6650) | 1 (1) | 11 (0.5512) | 56 (0.7269) |
| Number of organization types (Gini) | 3 (0.6016) | 3 (0.8268) | 3 (0.6646) | 1 (1) | 3 (0.6363) | 3 (0.7575) |
| Number of categories (Gini) | 7 (0.8136) | 17 (0.8950) | 22 (0.7602) | 1 (1) | 11 (0.7397) | 19 (0.8435) |
In astrophysics, policy documents within the field of Information and Computing Sciences published by the U.S. National Academies Press cite NPs. For first-generation citations, documents from the United States, Switzerland, and Italy collectively account for 96%, with the World Meteorological Organization, Center for Strategic and International Studies, and Food and Agriculture Organization of the United Nations being the top three institutions in terms of policy publication volume. For second-generation citations, the United States, Switzerland, and Luxembourg publish the highest number of citing policy documents. At the institutional level, the Publications Office of the European Union, World Meteorological Organization, and Analysis & Policy Observatory are the top three institutions in terms of policy publication volume. In terms of policy categories, those citing first-generation citations mostly fall under Physical Sciences, Information and Computing Sciences, and Earth Sciences, while those citing second-generation citations mainly belong to the categories of Physical Sciences, Studies in Human Society, and Earth Sciences.
All citing policy-publishing institutions can be categorized as international, national, or regional, with one exception: only national institutions cite NPs in astrophysics. We also calculated the Gini index for the distribution of citing policies across countries, organizations, categories, and so on.
We explored the policy impact of Nobel Prize-winning papers (NPs) and their citation generations. Figure 5 displays the distribution of policy citations for NPs and their generations, while Figure 6 illustrates the differences. On average, gene engineering NPs have 0.3378 policy citations, with 1st and 2nd generations having 0.0234 and 0.0141, respectively. In astrophysics, these values are 0.0203, 0.0013, and 0.0026.
Figure 5. The distribution of policy citations of NPs and their generations of citations. |
Figure 6. The difference of NPs and their generations of citations in policy citations. |
Negative binomial regression was applied to control for potential confounders that effect policy citations. To avoid correlation among variables, we also performed multicollinearity tests by VIFs. The results in Table 5 show that there were no serious problems of multicollinearity.
Table 5. The VIFs of variables for policy impact. |
| Variables | Gene engineering | Astrophysics |
|---|---|---|
| independent variable (groups) | 1.002 | 1.069 |
| Publication-type | 1.079 | 1.094 |
| N-research countries | 1.034 | 1.091 |
| N-publication citations | 1.003 | 1.074 |
| N-references | 1.093 | 1.116 |
| Publication age | 1.069 | 1.098 |
The regression results, as shown in Table 6 , indicate that in gene engineering, NPs have a significantly higher policy impact than the first generation of citations, which, in turn, has a higher policy impact than the second generation of citations (Table 6 , column 1). We further ensured the robustness of these results by controlling for confounding variables: Publication type, N-research countries, N-publication citation, N-references, and Publication age (Table 6 , columns 2-7). In all cases, the intergroup difference between NPs in gene engineering and their citation generations in policy impact remained consistent, demonstrating the robustness of our findings.
Table 6. Negative binomial regression models of direct and indirect impact on policy. |
| Gene engineering | |||||||
|---|---|---|---|---|---|---|---|
| (1) | (2) | (3) | (4) | (5) | (6) | (7) | |
| 1 st generation vs NPs | -0.490** (0.204) | -0.491** (0.204) | -0.476** (0.204) | -0.469** (0.204) | -0.497** (0.204) | -0.466** (0.204) | -0.428** (0.204) |
| 2 nd generation vs NPs | -0.530*** (0.200) | -0.530*** (0.200) | -0.519*** (0.200) | -0.506** (0.200) | -0.537*** (0.200) | -0.507** (0.201) | -0.473** (0.201) |
| Publication-type | 0.001 (0.009) | -1.378-3 (9.233-3) | |||||
| N-research countries | 0.036*** (0.006) | 3.880 -2 *** (6.193 -3 ) | |||||
| N-publication citations | 1.261-5*** (9.80-7) | 1.236 -5 *** (9.902 -7 ) | |||||
| N-references | 1.626-4** (7.578-5) | 1.676 -4 ** (7.857 -5 ) | |||||
| Publication age | 0.002** (0.001) | 2.307-3*** (7.958 -4 ) | |||||
| Constant | 0.821 | 0.820 | 0.757 | 0.792 | 0.816 | 0.765 | 0.640 |
| Log-likelihood | -12402.77 | -12402.77 | -12387.26 | -12362.29 | -12400.58 | -12400.51 | -12340.97 |
| N | 883,944 | 883,944 | 883,944 | 883,944 | 883,944 | 883,944 | 883,944 |
| Astrophysics | |||||||
| (8) | (9) | (10) | (11) | (12) | (13) | (14) | |
| 1 st generation vs NPs | 0.227 (0.525) | 0.210 (0.525) | 0.231 (0.525) | 0.479 (0.543) | 0.227 (0.525) | 0.261 (0.525) | 0.479 (0.542) |
| 2 nd generation vs NPs | 0.397 (0.510) | 0.384 (0.510) | 0.404 (0.510) | 0.671 (0.532) | 0.397 (0.510) | 0.450 (0.510) | 0.695 (0.531) |
| Publication-type | -0.032 (0.032) | -2.993-2 (3.394-2) | |||||
| N-research countries | 0.006 (0.013) | 7.768-3 (1.383-2) | |||||
| N-publication citations | 4.366-5** (2.214-5) | 4.085-5* (2.246-5) | |||||
| N-references | -1.721-5 (2.408-4) | 9.576-5 (2.546-4) | |||||
| Publication age | 0.004* (0.002) | 3.768-3* (2.121-3) | |||||
| Constant | 1.155-13 | 0.056 | -0.018 | -0.288 | 8.214-4 | -0.137 | -0.377 |
| Log-likelihood | -1143.06 | -1142.54 | -1142.96 | -1141.34 | -1143.06 | -1141.43 | -1139.20 |
| N | 472,534 | 472,534 | 472,534 | 472,534 | 472,534 | 472,534 | 472,534 |
Standard errors are in parentheses. ***p<0.01. **p<0.05. *p< 0.1. |
However, for NPs in astrophysics, the results in Figure 6 b and Table 6 , columns 8-14, indicate no significant difference in policy impact between NPs and their citation generations.
Furthermore, Table 6 highlights that in gene engineering, policy impact is positively correlated with N-research countries, N-publication citations, and N-references. In contrast, in astrophysics, policy impact is primarily associated with N-publication citations.
5 Discussion and conclusion
The achievement of societal impact of research is by generating reliable knowledge and being used in non-academic reports. Some academics pointed out that societal impact is a complex, indirect process with multidirectional influences, in particular, disseminating academic knowledge to a nonacademic audience often relies on indirect channels (Olmos-Penuela et al.,2014), yet the characteristics of indirect channels are not well understood. Based on previous studies (Hu & Rousseau, 2016), we investigate citation networks of NPs and analyze their (in)direct impact on technology and policy. We introduce a network indicator for technological impact and adopt policy citations to measure policy impact. TRs are considered to be truly important scientific discoveries that have promoted many follow-up studies with technological and policy impact. As presented in Table 3 and Figure 4 , the technological impact of NPs tends to decrease across citation networks. The difference between NPs and their subsequent citation generations in technological impact is statistically significant in gene engineering but not in astrophysics. Concerning policy impact, the results in Table 6 and Figure 6 indicate that NPs’ policy impact also tends to decrease with generations of citations, with statistical significance in gene engineering but not in astrophysics. Our findings imply that the direct impact on technology and policy of TRs in applied research is paramount, but the direct and indirect impacts of TRs in basic research are very similar. Our work provided a clue to better understand the characteristics of transformative research in technological and policy impact and also suggested that taking more than one citation generation into account may help peers to recognize the impact of TRs, especially in basic research areas.
Next, four points will be discussed.
The first point is to discuss the role of generations of citations in analyzing NPs’ societal impact. Previous studies, such as those by Hu et al. (2016), emphasize the importance of considering indirect citations to comprehensively assess the impact of scientific publications (NPs). These NPs encapsulate fundamental ideas, and their subsequent citation generations often build on and closely relate to them. Despite their potential influence, many NPs may not receive a high number of direct citations, partly due to the developmental stage of NPs and the cognitive level of scientists involved. The societal impact of NPs is suggested to rely on the knowledge transition facilitated by subsequent citing publications (Fujigaki, 1998). However, the direction of this transition may vary across different fields (Liu et al., 2014). In practical terms, NPs are initially used and cited by other publications. Over time, the potential value of NPs becomes gradually recognized, leading to knowledge transitions from both NPs and their citing publications to technology or policy, as outlined by Hu et al. (2018).
The second point involves discussing why TRs differ from other types of research. According to Dietz et al. (2012), TRs can be seen as a socially-engaged “movement” with significant interdisciplinary and transdisciplinary impacts. A transformative approach to research enhances its impact by providing support for actions that promote social, economic, and environmental justice, as suggested by Mertens (2021). The impacts of research can be assessed across three dimensions: depth, width, and length. Depth refers to the extent to which impact is structurally and culturally embedded. Social change can occur at varying levels of depth, including incremental, reformative, or transformative change. Transformation represents the most profound type of change, involving the alteration of deeply embedded rules or assumptions. Width pertains to the scope and coherence of impact. Transformation can occur at different scales or within various contexts. What might be considered transformational within a specific context or scale may not have the same transformative effect at a different level. Length relates to the durability and evolution of impact. Some changes are temporary or easily reversed, while transformation is characterized by its long-lasting and often irreversible nature, at least for an extended period, as highlighted by Strasser et al. (2020).
The third point is to explain the patterns of technological impact of NPs. Koshland’s typology categorizes discoveries into three types: challenge, charge, or chance discoveries (Koshland, 2007). Challenge discoveries suggest the need for new concepts or theories to explain anomalies, often leading to groundbreaking technologies. Charge discoveries address more obvious problems, while chance discoveries result from serendipity. It’s worth noting that most NPs in gene engineering fall under the category of “challenge” discoveries (Ledford, 2015), while NPs in astrophysics are typically classified as “charge” or “chance” discoveries (Koshland, 2007; Wuestman et al., 2020). This suggests that NPs in gene engineering tend to drive the development of radically new technologies, which helps explain why their technological impact is significantly higher than that of subsequent citation generations. In contrast, NPs in astrophysics, being “charge” or “chance” discoveries, may not have the same transformative impact on technology.
The final point aims to elucidate the characteristics of policy impact associated with NPs. The utilization of research in policy documents can be influenced by various factors. As Newson et al. (2018) explain, research can serve multiple purposes in policymaking, such as instrumental, conceptual, and symbolic uses. Gene engineering, being a promising biotechnology, allows for the manipulation of an organism’s genes, while astrophysics primarily investigates universal phenomena. The distinct nature of research in these two fields influences policy-makers differently, as it shapes awareness, understanding, and attitudes, generating ideas, arguments, and criticisms that contribute to policy debates (Lavis et al., 2002). Furthermore, policy usage of research reflects the intricate relationship between science and policy-making (Bornmann et al., 2016). Policies are particularly relevant as they target specific areas of society. Gene engineering holds the potential for innovative crops and groundbreaking medical treatments. Due to the novelty of NPs in gene engineering, policy-makers may refer to these pioneering researches to regulate their development. As these technologies mature, policy-makers might shift their focus. In contrast, astrophysics is a well-established field, and excessive policy regulation may not be necessary for its scientific development. Therefore, it’s reasonable that policy citations of NPs are higher than those of subsequent citation generations in gene engineering, but not necessarily in astrophysics.
Our analysis shed light on how technological impact and policy impact of NPs change across citation networks. Conducting such research informed by our approaches would feed into discussions on evaluating NPs comprehensively (Jones et al., 2016). However, there are a few limitations in the present study. First, the selection of Nobel Prizes is not balanced. However, since Nobel Prizes are scarce and the fact that we obtained key Nobel Prize papers in gene engineering and astrophysics, the limitation has little influence on the implications of our findings. Second, incompleteness and inaccuracy of citation links seems to be a major problem for Dimensions.
Funding information
This work was supported by the National Natural Science Foundation of China (Grant No. 71974167).
Acknowledgement
The authors would like to express gratitude to Lutz Bornmann and Ronald Rousseau for helpful comments, and Digital Science for data support.
Author contributions
Xian Li (lx2009yet@swu.edu.cn): data collection, data processing, writing the manuscript; Xiaojun Hu (xjhu@zju.edu.cn): initiated the idea, research question proposal, the design of methodology, writing the manuscript.