University benchmarking and assessment has long been a heated area of research and practices with many approaches and rankings. However, there exist different types of universities with different missions and sizes, resulting in varied emphases on research, teaching, or industrial/regional development. Even within research-oriented universities, some are more focused on basic research, while other ones aim more at applications. Most general ranking systems cannot show the degree to which ShanghaiTech University (hereafter: ShanghaiTech) as a small scale research university with a mission for cutting edge basic research in internationally competitive areas, fulfils its mission. These ranking systems fail to pinpoint the university’s strengths and weaknesses, and hence are useless for development planning. Hence, a new, individualized, and differentiating assessment method is needed, with special emphasis on research impact and competitiveness.

The past decades witnessed a worldwide growth of research activities and an increasing demand for return on research investment. Considerable effort has been devoted to benchmark research performance at the international, domestic, and intramural levels (Abramo & D’Angelo, 2011). Yet, assessment of research activities is complex and contentious, and researchers, funders and decision makers try to adapt with ever improving and expanding methods and indicators (James et al., 2015). However, measuring research performance remains a challenge all over the world (Huang et al., 2017). Traditional assessments are usually composite index based, taking many dimensions together, such as research investment (in labor force and in funding), output, quality, impact, social impact, etc. (Moed & Plume, 2011). Most indicators are calculated through bibliometric measurements such as publication and citation counts, patent counts, translational contracts, and even alternative metrics. Established citation databases, such as Web of Science (WoS) and Scopus, extended with specialized analytic tools such as InCites and Scival, are the main sources providing commonly used indicators. Waltman (2016) presented an in-depth overview of the main bibliographic databases and indicators, and made a distinction between basic size-dependent and size-independent indicators. Size-dependent indicators, such as total number of citations, number of highly cited publications, and the h-index, provide an overall performance measurement; size-independent indicators, such as average number of citations per publication, proportion of highly cited publications, reflect the research quality of units across different sizes of the target entities (James et al., 2015; Waltman, 2016). Besides these basic indicators, variations and extended indicators such as the normalized citation impact and the number of documents in the first quartile for journals give supplementary options for comparisons. These indicators and metrics have been widely applied to evaluate research entities (e.g. scientists, research groups, departments, institutions, research-intensive enterprises) (Ding et al., 2013; Ibrahim, 2018; Kumar & Pandit, 2018), with funding allocation, strategic planning, and government policy-making as their goals.

These approaches are used in university rankings to benchmark academic institutions. Modern rankings began in 1983, conducted by US News and World Report (Lukman et al., 2010), while today, more than 80 different university rankings have emerged (Jia, 2018). These rankings draw attention from a broad range of users such as would-be students or faculty, employers, government agencies, and institutions themselves. As a result many universities list their rankings in promotion materials. Some even set up targets in the ranking in their strategic plans (The University of Manchester, 2015). Each ranking is based on a variety of indicators, and many but not all of those are linked to research performance. Even with research activities, rankings bundle together multiple indicators different in nature and intended usage. For example, the Academic Ranking of World Universities (ARWU) includes three WoS related indicators, namely highly cited researchers, papers published in Nature and Science, total publications in Science Citation Index (SCI) and Social Science Citation Index (SSCI). Each of these indicators weighs 20% of the final score (ARWU, 2018). Research productivity (publications, 6%) and research impact (citations, 30%) on Scopus count a total 36% of the final weight in THE World University Ranking (Times Higher Education, 2019). In QS World University Rankings, the indicator of citations per faculty (total citations on Scopus) takes a proportion of 20% in the overall score (QS, 2019). Best Global Universities Rankings by U.S. News uses several WoS/InCites indicators, including total publication counts (10%), publications in the 10% most cited (22.5%), international collaboration (10%), highly cited papers (10%), total citations (7.5%), normalized citation impact (10%), leading to a total of 70% of the final score (U.S. News, 2018).

Although these and other ranking systems have continuously improved, they still face a number of intrinsic challenges. First, they do not differentiate institutions with different missions, sizes, and research emphases, leading to a bias toward large and more comprehensive ones. Second, many of them rely on composite indices, so that universities focusing on and excelling in only some aspects are disadvantaged in the overall rankings. Yet, recently some ranking systems began to give individually faceted rankings like research quality or translational capabilities (Shanghai Ranking, 2018). Thirdly, many indicator-based schemes fail to provide diagnostic power which institutions could use to improve their performance. Relying on limited types of documents or impact measures also affects the applicability of rankings when dealing with different types of institutions. For example, books and policy studies play an important role in the social sciences and humanities, but are not or only partially covered in the usual source databases, creating special problems with funders or assessment agencies in those fields. Some indicators are criticized for other reasons. For example, there is the well-known abuse of the journal impact factor (JIF) (Garfield, 1972), and disciplinary differences are often not accounted for. These and other problems contribute to tendencies of simply increasing the number of publications or ranges of research (Amin & Mabe, 2000). Efforts to correct some of the problems include the “Norwegian Model” designed in 2005 (Sivertsen, 2018). However, the Norwegian model still tries to balance the number and the quality of publications at the same time, so it may not be suitable in all cases and for all purposes.

With increasing applications of scientometric methods, the literature related to research indicators is now more inclined towards studying new indicators (Thelwall et al., 2015). However, over-reliance on single or composite indexes, how good they may be, still easily leads to biases and may hinder the development of research entities with specially focused missions (Vieira & Gomes, 2010). Furthermore, rankings and indicators can only present differences via numerical values, while the cause of these differences, stay beneath the veil. In addition, institutions have preponderant disciplines, therefore benchmarking the research performance of entire institutions while ignoring disciplinary or internal unit-based performance could easily fail to serve the diagnosis and planning purpose of institutions.

The San Francisco Declaration on Research Assessment (DORA) (ASCB, 2016) and The Leiden Manifesto (Hicks et al., 2015) discussed the problems and outlined good principles in research evaluation. In China, the General Office of the State Council issued “Opinions on deepening project, talent and institutional evaluation reform” emphasizing the establishment of a faceted assessment structure (Xinhuanet, 2018). Hence, flexible benchmarking methods geared to the specific missions and research emphases of individual and different institutions are needed. This paper describes the methods and practices ShanghaiTech employed for this purpose.

This study is a preliminary exploration for a mission-oriented benchmarking effort. Different from the common ranking methods, our approach aims to focus on research impact and competitiveness by comparing with top research universities. We do this on the university level, then on the level of disciplines, and further on the level of topics. The result was reported to the faculty and accepted by the university as one of the analytic angles in its self-assessment, complementary to international expert reviews, utilized as the main evaluation approach. Improved studies using the same approach are commissioned by the university, at a yearly interval. The basic approach and methodology can also be adapted for other universities.

Clarifying the principles and assessment questions is crucial, as different indicators and analytic methods must be selected accordingly. Traditional composite index based assessment is usually biased toward comprehensive institutions with large number of researchers and subject coverage (Borgman, 2015; Lopez-Illescas et al., 2009; Waltman & Eck, 2013), but our effort takes the nature and purpose of each institution into consideration and aims to design individual assessment schemes. Selection of benchmarking targets is also in line with this mission-oriented approach in the case of ShanghaiTech.

Multiple indicators are used, trying to provide multiple angles and to avoid bias caused by excessive use of a single-categorized indicator. We recall that Rousseau et al. (2018) mentioned that citation measurement may indicate but is not equal to research quality. Even though citation impact indicators are widely accepted, they can be influenced and biased by many factors, such as language and document type. Researchers and whoever uses indicators in assessments must be aware of the consequence of misusing indicators (Rousseau et al., 2018).

Faculty make-up, in the case of ShanghaiTech with a large number of adjunct faculty, needs to be considered when comparing institutions with obvious differences. For this reason we defined multiple datasets according to employee status to enable revealing their performance separately. However, we were unable to distinguish the detailed categories of faculty at the other benchmarking universities. We would argue that for these more established and bigger institutions adjunct faculty may not play a similar role as in ShanghaiTech. Moreover, comparing to a bigger pool of researchers in the target institutions may not be as bad because it works against over-estimating one’s own performance.

As a mission-oriented model, future improvements are needed to provide better diagnostic power and to help research planning. Top institutions in common composite rankings may not be the real leading ones in specific research areas ShanghaiTech focuses on or plans to develop. Finding those niche leaders for research-area-based benchmarking may provide a better understanding of one’s standing in the field. Using the disciplinary categories by ESI is a choice of convenience only. Constructing disciplinary categories according to individual institution’s research focuses would produce more meaningful and diagnostically more helpful results, but with obvious challenges. Other quality or impact indicators may also be needed to avoid influences of journal-centric ones. Topic distribution maps as they are now in the study are yet unable to reveal impacts, degrees of leading, and evolutions. Limiting to research articles and to a certain database creates its own intrinsic problems, especially for institutions with a strong engineering research focus and corresponding investment like ShanghaiTech. We understand that any analytic result like the one presented here is only a beginning for further explorations. Consequently, new techniques and tools are needed to integrate original data, results. Such tools are needed to drill down, expand, connect, fuse, or otherwise analyze data, leading to reports that are read by researchers or decision-makers to explore new questions stimulated by the results. The authors are planning further improvements in the second and future phases of benchmarking.