1 Introduction
At present, China is a front-runner in the number of scientific outputs, but the quality remains to be improved. In order to respond to international competition and to satisfy domestic development needs, the orientation of China’s science and technology evaluation has changed from encouraging international publications to pursuing excellence and encouraging science and technology for promoting development. One of the policies is the introduction of international experts in science and technology evaluation when conditions permit. For instance, the Opinions of Deepening the Reform of Project Review, Talent Evaluation and Institutional Assessment (the “Sanping Opinions”) printed and released in 2018, proposes to “gradually introduce international peer review in some frontier and basic sciences areas in appropriate proportions” and “enhance performance evaluation of national science and technology plans and introduce international evaluation as needed”. In practice, some advanced institutions and universities in China have explored ways to implement international evaluation in various forms since 2000. The Chinese Academy of Sciences (CAS) took the lead in exploring and implementing international peer review at the institutional level. This initiative goes back to 2004 when a pilot international peer review of research institutes was carried out during Phase II of the Knowledge Innovation Project (Peng, 2013). In recent years, universities such as Shanghai Jiao Tong University (Zhang, 2019), Tsinghua University (Cheng, 2019), and Peking University have been exploring international evaluation with regard to the promotion of researchers; moreover, the Beijing Institute of Technology explored and implemented international evaluation of disciplinary (school or department) developments. In 2011, coinciding with the 25th anniversary of the establishment of the National Natural Science Foundation of China (NSFC), an international peer review of the performance of science funding and management was successfully completed (Wu, Liang, & Zheng, 2012), which is the largest comprehensive science and technology performance evaluation so far in China. Internationally, world-known research institutes have explored and implemented various forms of international evaluation very early, for example, evaluation of research institutes and selection of academic leaders by Max Planck Society (Shuai, 2004) and evaluation of discipline-specific research quality by Research Excellence Framework (REF) (Xu, Liu, & Li, 2014). There is no doubt that international evaluation represents a new trend in science and technology evaluation (Marušić, Brezis, & Squazzoni, 2017). Is the time ripe to conduct international peer reviews in China? More importantly, how to design and conduct effective international peer reviews in China? This paper starts with these two critical policy questions about international peer review, and intends to find an answer through a case study of international evaluation practices of acceptance assessment of CAS Center for Excellences. Then, it will put forward considerations about China’s future international evaluations, in order to accumulate knowledge for the development of theories and methods related to such evaluations and provide a reference for future science and technology management exercises.
2 Two critical questions about international peer review
Question 1: Is the time ripe to conduct international peer reviews in China? First, it is meaningless to conduct an international peer review when the research level is too low. Over the forty years since its reform and opening-up, China has raised its influence and made achievements of world interest in science and technology. For instance, China maintains an annual average 10% growth of investment in science and technology, and ranked second in the world in the number of SCI papers published and in citations from 2007 to 2017 (ISTIC, 2019). In some basic and frontier research areas, China’s research is already at the international level, and it is meaningful and possible to conduct an international peer review in these areas. Second, can international peers understand the difference of research management forms and organizational forms of science research in China? Some may argue that international peers know little beyond a vague picture of the enormous and complicated Chinese S&T system. Therefore, they can hardly play effective roles and provide valuable opinions. This leads to Question 2, which is about the design of international peer reviews.
Question 2: How to design and conduct effective international peer reviews? From the point of a researcher manager’s view, international peer review provides useful suggestions by bringing international experience of peer panels into the picture. The challenge is to ensure that the expertise of evaluation panels is congruent with the research interests of the research groups (Rousseau, 2017). On the one hand, it is critical to well design the evaluation procedure and prepare sufficient evaluation materials such as the background introduction of the purpose, content, criteria, supporting documents (Gunashekar, Wooding, & Guthrie, 2017) of a particular evaluation. On the other hand, not always visible conflicts of interest may lead to toning down negative issues. In that case, the research manager needs to find out hidden meanings among words of excessive praise.
These two critical questions raised hot debates before conducting an international peer review of CAS Centers for Excellence. To some extent CAS’s practice provide a possible answer, which may provide useful insights to others.
3 A case study of international peer review of CAS Centers for Excellence
3.1 Overview of CAS Centers for Excellence
In the light of new national and societal requirements, CAS announced the “Shuaixian Plan” in 2014 after examining and analyzing the existing foundation, strengths and weaknesses, and in response to the “four-first” requirement raised by President Xi Jinping. The Plan puts forward measures for a new classification of four categories of research institutes, namely: Centers for Excellence, Innovation Institutes, Centers for Mega-science and Featured Institutes2（2http://scitech.people.com.cn/n/2014/0820/c1057-25501335.html）. Table 1 shows the basic features of these four categories. Distinctions are based on different value orientations. An Innovation Institute is expected to meet major needs of national strategy and industrial development. Consequently, its evaluation system is based on reviews from the market and from users. Centers for Mega-science are focused on supporting sci-tech progress: its evaluation system is based on reviews from peers and users. Featured Institutes are expected to cultivate disciplinary characteristics, and its evaluation system is based on reviews from peers, relevant industry departments and local governments.
Table 1 Basic feature of CAS four categories of research institutes. |
| Centers for Excellence | Innovation Institutes | Centers for Mega-science | Featured Institutes | |
|---|---|---|---|---|
| Research orientation | · Focus on original innovation of science and technology | · Focus on economic development and national security | · Focus on building and running large-scale science and technology innovation platforms | · Focus on serving for sustainable development of society |
| Research areas | · Basic and frontier sciences · Orientation towards major scientific problems. | · Important foundation and technological direction for promoting industrial upgrading, breaking through bottlenecks, influencing or restricting national security · High risk technological frontier directions | · Design, construct and operate international advanced large-scale scientific devices · Comprehensive interdisciplinary and cutting-edge sciences relying on large scientific devices | · Indispensable research areas for sustainable social development · Interdisciplinary study of natural science and social science · long-term observation and continuous accumulation related basic research work |
| Team composition | · Internationally or domestically recognized leading scientists · High-level academic leaders and their support teams · A small group of excellent researchers | · Strategic scientists, academic and technological leaders with broad vision and good grasp of technological direction and organizational tackling key problems · A well-structured support team consisting of scientific researchers, engineers and industrialized personnel | · High level teams in scheme design, technology research and development and engineering organizations · Professional technical support and operational service team · International and domestic multidisciplinary research teams | · Internationally recognized scientific and technology experts who also understand national demand · An integrated research team with multiple subject background |
| Forms of research output | · Solve major scientific problems · Open up new research areas · Invent significant scientific instruments · Break new ground in key experimental methods · Train first-class scientists · Putt forward significant prospective suggestions | · Breakthrough key technologies · Provide systematic integration solutions · Develop new technologies and standards · Incubation of new industries and enterprises · Technological radiation produces significant economic benefits · Important original innovation for national major strategic needs · Create first-class strategic science and technology experts and engineering technology specialists | · Technological services with open sharing, efficient operation and satisfactory users · Major breakthroughs in science and technology relying on the formation of large scientific equipment · Creating first-class scientists and engineers · Propose suggestions on planning of large-scale scientific equipment adopted by the state | · Providing scientific suggestions and constructive solutions for macro-decision-making and sustainable development · Forming new theories, methods, standards and tools in this field · Accumulate basic data and provide an open and shared analytical technology platform · Creating first-class scientists, strategic science and technology experts and technical specialists |
| Research evaluation system | · Periodic evaluation every 5 years, annual monitoring and mid-term diagnosis assessment | |||
| · Focus on research quality and impact · International peer review | · Focus on goal accomplishment and adoption · Based on peers’ and users’ opinions, and market feedback | · Focus on the achievement of construction objectives, operational efficiency and major outputs · Based on peers’ and users’ opinions | · Focus on quality, benefit and impact · Based on peers’ and users’ opinions | |
The construction of CAS Centers for Excellence is intended to “set examples, promote progress and gather talented personnel”, namely, setting up the banner of China’s science and technology innovation in important areas, taking the lead in making major breakthroughs and gathering top experts, so as to build an internationally top-notch innovation platform. Therefore, its evaluation system is based on reviews from international peers. In accordance with the management requirements, CAS organized an international peer review of nine Centers for Excellence after their preparation period in 2016. Based on the experiences gathered in 2016, an acceptance assessment was conducted for another eight Centers of Excellence in 2017. The following sections focus on three issues related to the design of international peer review, namely “what to review, who shall review and how to review”, to analyze the practices of CAS Centers for Excellence from a methodological perspective.
3.2 What to review?
In general, what to review directly depends on the purpose of the review. In the case of CAS, the acceptance assessment is required to implement the construction of four categories of institutions, systematically summarize and promote reform experiences, and study and solve existing problems in a timely manner. Based on this, the acceptance assessment focuses on the overall goals and the results achieved during the preparatory period of the Centers for Excellence, including their mission, orientation, research field layout, development objectives, domestic and international status, systems and mechanisms, and supporting conditions. At the same time, suggestions related to the advantages, disadvantages and future development of the Center are welcome. Generally speaking, the organizer should have a clear understanding about what international experts can and cannot review, to ensure that international experts can exert their professional knowledge and experience to the maximum and play a practical role in the review. In CAS’s case, international experts were required to review the status of the teams, including their quality, level and the rationality of the research structure, as well as the content and impact of major outputs (representative outstanding research outputs, normally no more than three items), which was exactly what international experts are familiar with and excel at. This is similar to current practice in international peer review across the world (De Wit, Van Heffen, & Verhoeven, 1998).
3.3 Who to review?
To some extent, the selection of international experts determines the quality of international peer review. In accordance with international practice, CAS’ Headquarters (in short: CASHQ) first established the principles for setting up international review panels, consisting of field and dedicated peer experts. Field experts, selected based on the positioning of the center, are expected to have an international vision, strong reputation and influence, combined with a profound insight into their research fields. They, moreover, have extensive contact with dedicated peer experts. In addition, they should have the ability to effectively organize peer panels for internal discussions, and have experiences in reviewing, such as holding management positions in large scientific research institutions or universities, or having served as management/review experts of major scientific research projects. The dedicated peer experts must have good judgment in key areas of the center and an in-depth understanding and knowledge of research fields, be actively involved in cutting-edge and pioneering scientific research, and preferably have experience in performing evaluation exercises. In order to understand domestic policies and the local environment for the development of science and technology, one or two ethnic Chinese experts who can comprehensively utilize their local and linguistic advantage are normally included in the panel group.
Taking the Max Planck Society of Germany as a major reference, potential panel members are first recommended by research institutes. These include field experts, peer review experts, and user experts for the main fields. Then the National Science Library of CAS conducted a data analysis on competency and activity of these experts in the preliminary list, which is further submitted to the administrative department of CAS for final decision-making (Fig. 1 ). In all, 86 international experts were invited for international review of eight Centers for Excellence, of which 53 were non-Chinese experts, accounting for 62%, and 33 were ethnic Chinese experts, accounting for 38%. Furthermore, 46 management experts from CASHQ participated in the review.
Figure 1. International panel selection procedure in the case of CAS. |
3.4 How to review?
The preparation of evaluation materials and the design of evaluation procedures is the core part of international evaluation. In the case of CAS, given that the Centers for Excellence are oriented towards basic and frontier areas3（3http://bdp.cas.cn/zhgg/sljgflgg/201802/t20180226_4636557.html）and aim to develop into incubators of major achievements, international experts are invited to participate in the on-site assessment (1-2 days) to evaluate the strategic positioning, major achievements and teams of the centers.
To ensure that international experts can, within a limited period of time, fully understand the background and purpose of the review, as well as acquire information about the centers, while reducing the burden on the centers as much as possible, the centers and CAS Evaluation Research Center jointly prepared evaluation materials. Specifically, the Centers for Excellence are responsible for preparing a self-evaluation report (both in an English and a Chinese version), filling out the Acceptance Self-evaluation Form, and providing a brief introduction and impact evidence of the major outputs (generally no more than three items), as well as an introduction to the leaders and academic support teams in all research fields. The Evaluation Research Center is responsible for preparing a data status report of the centers and other general materials such as a basic introduction to CAS, an introduction to international review, an expert manual and a work manual, to ensure that international experts have a comprehensive understanding of the current situation of CAS.
Drawing on the general practice of international institutional evaluation, five major steps are designed and programmed, namely: preparatory meeting, presentations of key research areas, group discussions, panel discussions, and panel feedback. The preparatory meeting is held to introduce background, purpose and requirements to the experts. It turned out that the success of the preparatory meeting determined to a large extent whether the review could produce the desired result. In particular, in order to improve the efficiency and effectiveness of international reviews, CASHQ and the center jointly held a pre-communication meeting with the panel leader to exchange opinions and answer questions on the overall situation of the review, which helped panel leader’s understanding of the review.
During the on-site evaluation, a set of evaluation forms are designed for collecting panel members’ opinions on strategic positioning and international status, the research level and impact of major research outputs, and the research level of the center’s teams. More specifically, each international peer is expected to give his/her judgement on the center’s international status based on its main research fields, the impact of original work, the impact of leading scientists and major breakthroughs or research outputs according to the principle of originality, significance and rigor. With regard to evaluation of teams, each panel member must identify the team member’s international research impact within their area. Any comments and suggestions on the center, the development of research work as well as the teams are welcome.
4 Feedback and effect of international peer review of CAS’s Centers of Excellence
In 2017, a total of 25 representative achievements, 31 subject leaders and 80 key members from eight Centers for Excellence were reviewed by international panels. From the feedback from CASHQ and the centers, the international peer review of CAS’s Centers of Excellence turned out to be satisfactory. Firstly, international review promoted the construction of four categories of institutions and the understanding of Center of Excellence. The entire review is a process for the leaders and researchers to “re-understand” the planning and key research fields of the Centers for Excellence. In the process of preparing evaluation materials, all the centers and their respective directors, subject leaders and researchers teased out the strategic orientation and major achievements of the research institutes as well as the relationship between various parts of the research work again, so as to deepen their understanding of the Centers of Excellence. Secondly, international peer review helps the development of CAS Centers of Excellence. A number of promising research work and distinguished researchers are identified, which provide valuable opinions for decision-making on resources allocations. Constructive opinions on the development of key research fields of the centers, especially their domestic and international status are pointed out, which are conducive for further improvement of the development plan of the centers. Thirdly, the philosophy of pursuit of research excellence is further promoted through the international peer review. During the on-site review, the international experts paid more attention to the quality and originality of research work rather than quantitative figures. Sometimes experts assess the current research status and level of researchers by asking them whether they know the best performers, competitors and their research content well in their respective research fields. These questions also help researchers to broaden their horizon, think deeply and understand their own advantages and disadvantages. Last but not least, there is no doubt that international reviews promote international exchanges and cooperation (DaiWai, 2019), which also provides an opportunity for the centers to promote themselves, advance the centers to the international forefront and expand international cooperation.
What is more, a questionnaire is conducted to collect opinions about the design of international peer view. Based on 42 valid questionnaires (86 questionnaires were issued, with a 49% recovery rate), the international experts believed that the international review provided a guarantee of independent and unbiased evaluation. According to their opinions, it is important to assess the Centers for Excellence by senior scientists from first rated international institutions as they can best review the status and assess the potential for future developments. Therefore, it is very important to collect evaluation opinions and suggestions through such an international review as a way to increase the quality and motivation of research work. From a personal perspective of experts, this review also provided them with a systematic understanding of CAS and its research institutes. Some experts said, “We also had exchanges and cooperation with some teams in the past, but that was just at the level of a research group or laboratory. This review provides comprehensive and systematic understanding of the overall situation of CAS.”
Moreover, each panel was required to give a score from 1 (the lowest) to 5 (the highest) in terms of panel member structure, preparation of evaluation materials, evaluation forms and standards, organization and implementation of review and finally the procedures and arrangements of review. It turns out that 83% of the international experts believed that the overall operation of international review is good or even excellent. In particular, the highest score is given to the organization and implementation of the international review, which was rated excellent or good by 90% of the international experts. The forms and standards of the international review received the lowest scores, but 79% of the international experts still rate them as excellent or good. In three other aspects, namely the procedures and arrangement of international review, preparation of materials and structure of expert group, 86%, 81% and 86% of the international experts rate them as excellent or good, as shown in Figure 2 . In addition, the international experts made the following suggestions about the organization and implementation of international peer review: in terms of procedure design and arrangement, more time should be reserved for the preparatory meeting to gain a better understanding of the background. This also applies to one-on-one discussions. It is important to simplify assessment standards and forms and keep them consistent with international practice. In terms of the preparation of materials, it is necessary to collect more basic data such as personnel and funding information. Last, female scientists and more junior peer experts must be included.
Figure 2. Opinions from international experts. |
5 Conclusions and discussions
Responding to the two critical questions proposed before, based on the example of CAS, we conclude that international peer review is effective in identifying problems in research activities, determining research level and status, and improving the development of research units with the help of international experts’ intelligence and experience. In order to gain maximal benefit from international panel members, the organizer must pay a lot of attention to the panel selection. On the one hand, the research level and reputation is the key factor when selecting appropriate experts. On the other hand, an expert is also expected to have strategic vision as well as rich evaluation experiences. In some cases, the authority and personal charm of the panel leader can lead to a smooth and rather pleasant experience. Particularly, ethnic Chinese experts play a great role in providing background information about the science and technology system in China. Other considerations during the on-site review such us informal communication with the panel leader in advance, and evaluation standards and language complying with international practices can improve the effectiveness and efficiency of international peer review as well.
In China, science and technology evaluation is at a critical stage. There is an urgent need for shifting the development path, that is, curbing the trend of excessive quantitative evaluation and developing a more quality-focused evaluation system to guarantee more breakthroughs in science and technology and transform from “quantity oriented to quality oriented” (Cheng, Li, & Xu, 2018). International peer review may provide a solution to address the problem of quantitative oriented research evaluation. In order to maximize the role of international peer review in research management, this paper puts forward the following points for discussion.
First, international peer review needs adaptation to specific circumstances to better play its role. International peer review is often applied in universities with which international panels are more familiar than with a large institute such as CAS. This shows that international peer review must adapt according to management needs and evaluation objectives. This also meets the current purpose of developing science and technology. In addition to serve the growth of scientific knowledge, science and technology has to contribute to economic and social development. Therefore, international peer review is required to evaluate not only the excellence and originality of research work, but also its contribution to economic and social progress. CAS has adopted international peer review in its practices, so that it can coordinate with management experts and better serve the needs of science and technology management.
Secondly, the range of applications and implementations of international peer review should be clear. China has issued relevant policies to encourage international peer review when appropriate, but the difficulty lies in how to define the range of applications and implementations of international peer review. Generally, international peer review is applicable to evaluate innovation and influence of non-confidential research work in frontier and basic scientific research areas, and to identify problems and make suggestions. Specifically, when referring to the evaluation of different subjects such as scientists, teams, organizations and projects, one needs to further explore which are suitable for international peer review.
Thirdly, theories and methods of international peer review need improvement. The advantage of international peer review lies in exerting the role of international intelligence and standards, and reducing the influences of quantitative indicators on experts. Given this, theories and methods of international peer review need urgent improvement. For example, how to avoid conflicts of interest while ensuring that selected experts are familiar with the evaluated objects. The use of evaluation results is another problem. If international experts are capable but choose not to provide honest and complete review opinions, international peer review will only play a weak role in promoting management work, allocation of resources and organizational adjustment. Therefore, efforts should be made to further explore and improve theories and methods concerning international peer review as well as normative and standard evaluation procedures (the Standard Evaluation Protocol proposed in Netherlands provides a good example (KNAW, VSNU, NOW, 2014)), in order to improve the efficiency and effect of international peer review and to support better decision-making.
Author Contributions
Fang Xu (xufang@casisd.cn) conceived and designed the research work, contributed data or analysis tools, and wrote the paper. Xiaoxuan Li (xiaoxuan@casisd.cn) conceived and designed the research work, and performed the analysis.