Hard-magnetic soft active materials have drawn significant research interest in recent years due to their advantages of untethered, rapid and reversible actuation, and large shape change. These materials are typically fabricated by embedding hard-magnetic particles in a soft matrix. Since the actuation is achieved by transferring the microtorques generated on the magnetic particles by the applied magnetic field to the soft matrix, the actuation depends on the interactions between the magnetic particles and the soft matrix. In this paper, we investigate how such interactions can affect the actuation efficiency by using a micromechanics approach through the representative volume element simulations. The micromechanics reveals that particle rotations play an essential role in determining the actuation efficiency, i.e., the torque transmission efficiency. In particular, a larger local particle rotation in the matrix would reduce the effective actuation efficiency. Micromechanics simulations further show that the efficiency of the torque transmission from the particles to the matrix depends on the particle volume fraction, the matrix modulus, the applied magnetic field strength, as well as the particle shape. Based on the micromechanics simulations, a simple theoretical model is developed to correlate the torque transmission efficiency with the particle volume fraction, the matrix modulus, as well as the applied magnetic field strength. We anticipate this study on the actuation efficiency of hard-magnetic soft active materials would provide optimization and design guidance to the parameter determination for the material fabrication for different applications.