In the optical communication, people use the optical fibers to achieve the high bitrate data transmission. In this paper, the AB system for the ultra-short pulses in anonlinear optical fiber is investigated via the Lie symmetry analysis. Lie symmetries and symmetry reductions are derived via the Lie algorithm. Periodic- and solitary-wave solutions are obtained via the qualitative consideration. Nonlinear self-adjointness of the AB system is proved, and conservation laws associated with the Lie symmetries are constructed. For the magnitude of the electric field in the optical fiber and the function associated with the occupation number which gives a measure of the atomic inversion in the nonlinear optical fiber, we can adjust the amplitudes, widths and velocities of the solitary waves via the Lie symmetry transformations. The results would help the engineers select the ultra-short pulses in the optical communication.