The phonon Boltzmann transport equation (BTE) is an important tool for studying the nanoscale thermal transport. Because phonons have a large spread in their properties, the nongray (i.e., considering different phonon bands) phonon BTE is needed to accurately capture the nanoscale transport phenomena. However, BTE solvers generally require large computational cost. Nongray modeling imposes significant additional complexity on the numerical simulations, which hinders the large-scale modeling of real nanoscale systems. In this work, we address this issue by a systematic investigation on the phonon band discretization scheme using real material properties of four representative materials, including silicon, gallium arsenide, diamond, and lead telluride. We find that the schemes used in previous studies require at least a few tens of bands to ensure the accuracy, which requires large computational costs. We then propose an improved band discretization scheme, in which we divide the mean free path domain into two subdomains, one on either side of the inflection point of the mean free path accumulated thermal conductivity, and adopt the Gauss–Legendre quadrature for each subdomain. With this scheme, the solution of the phonon BTE converges (error < 1%) with less than ten phonon bands for all these materials. The proposed scheme allows significantly reducing the time and memory consumption of the numerical BTE solver, which is an important step toward large-scale phonon BTE simulations for real materials.