Residual stress build-up in metal additive manufacturing (AM) is a well-known problem that can impede the applicability of the AM parts. Residual stress may cause the part to fail due to the crack or fall out of the specified dimension. Thus, in order for a part to be used in a mission-critical application, it is important to predict the stress state within the AM part accurately and rapidly. During the thermal loading, the grain size is altered at the subsurface through dynamic recrystallization (DRx) and subsequent recovery. The yield strength of the alloys is largely determined by the size of nucleated grains, and it has a substantial influence on residual stress build-up. In this work, a physics-based analytical model is proposed to predict the residual stress considering the microstructure of the additively manufactured part. A moving heat source approach is used to predict the temperature field. Due to the high-temperature gradient in this process, the material properties are considered temperature-sensitive to capture the properties gradient affected thermal distribution. The energy needed for solid-state phase change is also considered by modifying the heat capacity using the latent heat of fusion. Due to the high-temperature gradient, the thermal stress is obtained using Green’s function of stresses due to the point body load. The total stress is the combination of three main sources of stress known as body forces, normal tention, and hydrostatic stress. High thermal stress may exceed the yield strength. The yield surface is obtained by modifying the Johnson-Cook flow stress to incorporate the effect of DRx on grain size using the Hall-Patch equation. The DRx and subsequent recovery affected grain size is predicted using Johnson-Mehl-Avrami-Kolmogorov (JMAK), model. The residual stress is then predicted using incremental plasticity and kinematic hardening behavior of the metal according to the property of volume invariance in plastic deformation and in coupling with equilibrium and compatibility conditions. The predicted residual stress considering microstructure evolution is validated by measuring the residual stress via X-ray diffraction for the In718 parts manufactured via the direct metal deposition process.