The demand for video surveillance has increased rapidly in recent years. Artificial intelligence (AI) algorithms are key enablers for the smart functionalities of a surveillance camera. Typical smart functionalities include human or object detection, tracking and recognition. However, many of the neural network (NN) algorithms for AI require intensive computation. At the endpoint or edge such as a home surveillance camera, the computation power is limited. The intensive computation also causes higher power consumption, which is also problematic for battery powered cameras. In this paper, we introduce a new human detection scheme that requires much less computation while the accuracy is equivalent to other existing algorithms. It obtains datasets and knowledge from a complex NN algorithm at the learning and calibration phase. These datasets are later used to train two cascading lightweight machine leaning algorithms, which will be used for further human detections. It is demonstrated that the proposed scheme can be run by the camera alone and the speed of detection is much faster than other benchmark NN algorithms.