Accurate estimation of state of charge (SOC) is the basis of battery management system. The fractional-order theory is introduced into the resistance-capacitance (RC)model of lithium battery, and adaptive genetic algorithm is used to identify the parameters of the RC model. Considering the changes of internal resistance and battery aging during battery discharge, the state of health (SOH) of the battery is estimated by unscented Kalman filter (UKF), and the values of internal resistance and maximum capacity of the battery are obtained. Finally, a novel estimation algorithm of lithium battery SOC based on SOH and fractional-order adaptive extended Kalman filter (FOAEKF) is proposed. An experimental system is set up to verify the effectiveness of the proposed algorithm. The experimental results show that, compared with the traditional SOC estimation algorithms, the proposed method has higher estimation accuracy, with the average error lower than 1% and the maximum error lower than 2%.