Based on oil monitoring technology to collect friction and wear parameters, the failure modes of key friction pairs in wind turbine gearboxes can be evaluated and classified. However, the collected data of failures caused by friction and wear are generally small, which limits the application of machine learning in the monitoring or evaluation of the critical friction pairs of wind turbine gearboxes. To verify the feasibility of machine learning in this application, algorithms including decision tree (DT), random forest (RF), k-nearest neighbors (k-NN), and support vector machine (SVM) are implemented, in the context of a small dataset of 424 samples of normal, adhesive, fatigue, and cutting wear for outcome classification. Compared with k-NN and SVM, DT and RF perform better on both training and test samples. The two models identified the key factors and their quantified values associated with failure state, including ferromagnetic particles, viscosity, iron content, and external hard particle silicon. The classifiers developed in this work classified failure state with an average accuracy of 96%, thus offering an accurate decision support tool for classification and evaluation of the friction pair wear state of wind turbine gearboxes.