In this study, polymer-bonded sugar (PBS) is used as a substitute material for polymer-bonded explosive (PBX), and the shear failure process of PBS under compressive loading. First, the shear failure process of PBS was analyzed by a series of experiments, and it was found that the shear band appearing on the surface of the specimen was not symmetrical. Further theory analysis showed that it was triggered by the evolution of asymmetric damage caused by internal defects in the material. In addition, through investigating the distribution of experimental scatters, we found that the material undergoes a relatively long period of internal microstructure adjustment before shear failure occurs, this adjustment will undoubtedly affect the evolution of the shear band. More importantly, a data density method was used to quantify the adjustment process. Finally, by using finite element simulation, the effects of matrix–particle interface strength on the mechanical response or damage evolution of the PBS were thoroughly examined. This research has reference significance for understanding the damage evolution process of high particle content composite materials.