The Rate of Penetration (ROP) is one of the key parameters related to the efficiency of the drilling process. Within the confines of operational limits, the drilling parameters affecting the ROP should be optimized to drill more efficiently and safely, to reduce the overall cost of constructing the well. In this study, a data-driven optimization method called Extremum Seeking (ES) is employed to automatically find and maintain the optimal Weight on Bit (WOB) which maximizes the ROP. The ES algorithm is a model-free method which gathers information about the current downhole conditions by automatically performing small tests with the WOB and executing optimization actions based on the test results. In this paper, this optimization method is augmented with a combination of a predictive and a reactive constraint handling technique to adhere to operational limitations. These methods of constraint handling within ES application to drilling are demonstrated for a maximal limit imposed on the surface torque, but the methods are generic and can be applied on various drilling parameters. The proposed optimization scheme has been tested with experiments on a downscaled drilling rig and simulations on a high-fidelity drilling simulator of a full-scale drilling operation. The experiments and simulations show the method's ability to steer the system to the optimum and to handle constraints and noisy data, resulting in safe and efficient drilling at high ROP.