Can Automatic Classification Help to Increase Accuracy in Data Collection?

Journal of Data and Information Science >

2016 , Vol. 1 >Issue 3: 42 - 58

DOI: https://doi.org/10.20309/jdis.201619

Research Paper

Can Automatic Classification Help to Increase Accuracy in Data Collection?

Frederique Lang ,
Diego Chavarro & Yuxian Liu

Expand

1 Science Policy Research Unit (SPRU), School of Business, Management and Economics, University of Sussex, Falmer, Brighton, BN1 9SL, United Kingdom;
2 Tongji University Library, Tongji University, Shanghai 200092, China

Received date: 2016-07-17

Revised date: 2016-07-25

Online published: 2016-07-29

Supported by

The authors are grateful to Peter Bone for his help with the proofreading of this paper. We wish also to thank Jose Christian for helpful discussion and support. Thanks go also to the colleagues who gave us comments during the 4^th Global TechMining Conference and in the Science Policy Research Unit (SPRU) Wednesday Seminar. Yuxian Liu's work was supported by National Natural Science Foundation of China (NSFC) (Grant No.: 71173154), The National Social Science Fund of China (NSSFC) (Grant No.: 08BZX076) and the Fundamental Research Funds for the Central Universities.

Fold

Abstract

Purpose: The authors aim at testing the performance of a set of machine learning algorithms that could improve the process of data cleaning when building datasets.
Design/methodology/approach: The paper is centered on cleaning datasets gathered from publishers and online resources by the use of specific keywords. In this case, we analyzed data from the Web of Science. The accuracy of various forms of automatic classification was tested here in comparison with manual coding in order to determine their usefulness for data collection and cleaning. We assessed the performance of seven supervised classification algorithms (Support Vector Machine (SVM), Scaled Linear Discriminant Analysis, Lasso and elastic-net regularized generalized linear models, Maximum Entropy, Regression Tree, Boosting, and Random Forest) and analyzed two properties: accuracy and recall. We assessed not only each algorithm individually, but also their combinations through a voting scheme. We also tested the performance of these algorithms with different sizes of training data. When assessing the performance of different combinations, we used an indicator of coverage to account for the agreement and disagreement on classification between algorithms.
Findings: We found that the performance of the algorithms used vary with the size of the sample for training. However, for the classification exercise in this paper the best performing algorithms were SVM and Boosting. The combination of these two algorithms achieved a high agreement on coverage and was highly accurate. This combination performs well with a small training dataset (10%), which may reduce the manual work needed for classification tasks.
Research limitations: The dataset gathered has significantly more records related to the topic of interest compared to unrelated topics. This may affect the performance of some algorithms, especially in their identification of unrelated papers.
Practical implications: Although the classification achieved by this means is not completely accurate, the amount of manual coding needed can be greatly reduced by using classification algorithms. This can be of great help when the dataset is big. With the help of accuracy, recall, and coverage measures, it is possible to have an estimation of the error involved in this classification, which could open the possibility of incorporating the use of these algorithms in software specifically designed for data cleaning and classification.
Originality/value: We analyzed the performance of seven algorithms and whether combinations of these algorithms improve accuracy in data collection. Use of these algorithms could reduce time needed for manual data cleaning.

http://ir.las.ac.cn/handle/12502/8731

Key words： Disambiguation; Machine learning; Data cleaning; Classification; Accuracy; Recall; Coverage

Cite this article

Frederique Lang , Diego Chavarro & Yuxian Liu . Can Automatic Classification Help to Increase Accuracy in Data Collection?[J]. Journal of Data and Information Science, 2016 , 1(3) : 42 -58 . DOI: 10.20309/jdis.201619

References

C havarro, D. & Liu, Y. (2014). How can a word be disambiguated in a set of documents: Using recursive Lesk to select relevant records. Presented in 2014 Annual Global Techmining Conference. Retrieved from http://www.gtmconference.org/abstracts/2014/session1METHODS3.pdf.
Chin, W.S., Zhuang, Y., Juan, Y.C., Wu, F., Tung, H.Y., Yu, T., Wang, J.P., Chang, C.X., Yang, C.P. & Chang, W.C. (2014). Effective string processing and matching for author disambiguation. The Journal of Machine Learning Research, 15, 3037-3064.
Coussens, L., Yang-Feng, T.L., Liao, Y, Chen, E., Gray, A., McGrath, J., …& Ullrich, A (1985) Tyrosine kinase receptor with extensive homology to EGF receptor shares chromosomal location with neu oncogene. Science, 230(4730), 1132-1139.
Daim, T.U., Rueda, G., Martin, H., & Gerdsri, P. (2006). Forecasting emerging technologies: Use of bibliometrics and patent analysis. Technological Forecasting and Social Change, 73, 981-1012.
Friedman, J., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33, 1-22.
Fukushige, S., Matsubara, K., Yoshida, M., Sasaki, M., Suzuki, T., Semba, K., Toyoshima, K. & Yamamoto, T. (1986). Localization of a novel v-erbB-related gene, c-erbB-2, on human chromosome 17 and its amplification in a gastric cancer cell line. Molecular and Cellular Biology, 6, 955-958.
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning. Berlin: Springer. Available at: https://link.springer.com/content/pdf/10.1007/978-0-387-84858-7.pdf.
Huang, S., Yang, B., Yan, S. & Rousseau, R. (2014). Institution name disambiguation for research assessment. Scientometrics, 99, 823-838.
International Society of Scientometrics and Informetrics (ISSI) (2015). International Conference on Scientometrics & Informetris Call for Paper. Retrieved from http://issi2015.ulakbim.gov.tr/.
Jurka, T., Collingwood, L., Boydstun, A., Grossman, E., & Atteveldt, W.V. (2012). RTextTools: A supervised learning package for text classification. The R journal, 5, 6-12.
Kim, B., Gazzola, G., Lee, J.M., Kim, D., Kim, K., & Jeong, M.K. (2014). Inter-cluster connectivity analysis for technology opportunity discovery. Scientometrics, 98, 1811-1825.
Kim, J., & Diesner, J. (2015). Distortive effects of initial-based name disambiguation on measurements of large-scale coauthorship networks. Journal of the Association for Information Science and Technology, 67(6): 1446-1461.
King, B., Jha, R., & Radev, D.R. (2014). Heterogeneous networks and their applications: Scientometrics, name disambiguation, and topic modeling. Transactions of the Association for Computational Linguistics, 2, 1-14.
Liaw, A., & Wiener, M. (2002). Classification and regression by randomForest. R news, 2, 18-22.
Li, C., Sun, A., & Datta, A. (2013). TSDW: Two-stage word sense disambiguation using Wikipedia. Journal of the American Society for Information Science and Technology, 64(6), 1203-1223.
Liu, W., Do?an, R.I., Kim, S., Comeau, D.C., Kim, W., Yeganova, L., & Wilbur, W.J. (2014). Author name disambiguation for PubMed. Journal of the Association for Information Science and Technology, 65(4), 765-781.
Lundberg, J., Fransson, A., Brommels, M., Skar, J., & Lundkvist, I. (2006). Is it better or just the same? Article identification strategies impact bibliometric assessments. Scientometrics, 66, 183-197.
Meyer, D., Dimitriadou, E., Hornik, K., Weingessel, A., & Leisch, F. (2012). Misc Functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien. Retrieved from http://cran-r.c3sl.ufpr.br/web/packages/e1071/e1071.pdf.
Peters, A., Hothorn, T., Ripley, B.D., Therneau, T., Atkinson, B., & Hothorn, M.T. (2012). Package ‘ipred’: Improved predictors. Retrieved from https://cran.r-project.org/web/packages/ipred/index.html.
Porter, A., & Cunningham, S. (2004). Tech mining: Exploiting new technologies for competitive advantage. Hoboken, New Jersey: John Wiley & Sons.
Ripley, B. (2012). Tree: Classification and regression trees. Retrieved from https://cran.r-project.org/web/packages/tree/index.html.
Rotolo, D., Hopkins, M., & Grassano, N. Do funding sources complement or substitute? The case of the UK cancer research. In the 19^th International Conference on Science and Technology Indicators (the STI 2014), (pp 473). Leiden, Netherlands.
Salo, A., Mild, P., & Pentikäinen, T. (2006). Exploring causal relationships in an innovation program with robust portfolio modeling. Technological Forecasting and Social Change, 73, 1028-1044.
Schechter, A.L., Stern, D.F., Vaidyanathan, L., Decker, S.J., Drebin, J.A., Greene, M.I., & Weinberg, R.A. (1984). The Neu Oncogene - An Erb-b-related gene encoding A 185,000-Mr Tumor-antigen. Nature, 312(5994): 513-516.
Semba, K., Kamata, N., Toyoshima, K., & Yamamoto, T. (1985). A v-erbB-related protooncogene, c-erbB-2, is distinct from the c-erbB-1/epidermal growth factor-receptor gene and is amplified in a human salivary gland adenocarcinoma. Proceedings of the National Academy of Sciences, 82, 6497-6501.
Shih, C., Padhy, L.C., Murray, M., & Weinberg, R.A. (1981). Transforming genes of carcinomas and neuroblastomas introduced into mouse fibroblasts. Nature, 290, 261-264.
Slamon, D.J., Clark, G.M., Wong, S.G., Levin, W.J., Ulirich, A., & Mcguire, W.L. (1987). Human breast cancer: Correlation of relapse and survival with amplification of the HER-2/neu oncogene. Science, 235, 177-182.
Tuszynski, J. (2012). caTools: Tools: Moving window statistics. Retrieved from https://cran.rproject.org/web/packages/caTools/index.html.
Wallace, M.L., & Rafols, I. (2014). Research portfolios in science policy: Moving from financial returns to societal benefits. Minerva, 2015, 53(2): 89-115.

Options

Outlines

/

〈

〉