Chinese Journal of Computers   Full Text
Title	A Study on Integrated Evaluating Kernel Classification Performance Using Statistical Methods
Authors	WANG Yong1),2) HU Bao-Gang1),2)
Address	1)(National Laboratory of Pattern Recognition， Institute of Automation, Chinese Academy of Sciences, Beijing 100190) 2)(Graduate University of Chinese Academy of Sciences, Beijing 100049)
Year	2008
Issue	No.6(942—952)
Abstract &   Background	Abstract This paper explores the research on evaluating kernel classification performance using statistical methods. By employing the corrected resample t-test and other two statistical methods—k-fold cross-validation and paired t-test, this paper compares their classification abilities on nine normally used kernels. In addition, a new quantitative criterion of evaluating kernel classification performance based on information gain is proposed, which is proved to be the nonlinear function of traditional criteria. Benchmark tests show that there is difference among different criteria, but by using statistical methods some regulations can be turned up among them. Simultaneously, there is great difference among different statistical methods, which affects the evaluating results more than the difference among different criteria does. So only with the integrated methods and criteria the classification performance of different kernels can be evaluated objectively. Keywords kernel selection; pattern recognition; corrected resample t-test; information gain; nonlinear function Background This research is supported by the National Natural Science Foundation of China, No.60275025 ("Nonlinearity-variation-based Study of Intelligent Systems") and National Natural Science Foundation Outstanding Innovation Group Project, No.60121302. The nonlinear-variation ability of functions refers to the ability of functions to approximate a cluster or multi-clusters of nonlinear functions. Though some methods have been proposed to solve the problem from the aspects of "Non-linear domain analysis" and "the application of apriori knowledge", in view of practical applications, how to measure the "nonlinear-variation ability" quantitatively is still the most difficult and the key part of the research. How to choose the kernel functions is a special case of it, and up to date there is still missing a well-accepted framework to guide kernel selections. Analyzing the nonlinear-variation ability of functions may give us a new choice for kernel selection. In this paper, the authors applied statistical methods to study the problem of kernel selection quantitatively. K-fold cross-validation is a commonly used statistical method for kernel selection, while it is valid only if the independence between the data and the classifiers is guaranteed. In practical applications this premise condition is usually inaccessible. So by employing the corrected resample t-test—A newly proposed statistical method, the authors compare it with other two statistical methods—k-fold cross-validation and paired t-test on nine normally used kernels to measure their classification abilities. In addition, a new quantitative criterion of evaluating kernel classification performance based on information gain is proposed, which is proved to be the nonlinear function of traditional criteria and with wider application range. Benchmark tests show that the proposed quantitative statistical method is valid, and the information gain criterion is simple, stable. Furthermore, it can make up other criteria to a certain extent. Similar systematic studies on statistical methods of kernel selection are seldom reported in now.