This paper studies the effect of covariance regularization for high dimensional data for classification. This is done primarily by fitting a mixture of Gaussians where the covariance matrix is regularized to each class. Three data sets from various domains are used to suggest the results are applicable to any domain where covariance regularization is required. The regularization needs of the data when pre-processed using two dimensionality reduction techniques, the popular principal component analysis (PCA) and the up-and-coming random projection, are also compared. Observations include that using a large amount of covariance regularization consistently provides classification accuracy as good if not better than using little or no covariance regularization as well as the state-of-the-art classification algorithms across all training set sizes and all but the lowest dimensionalities and that pre-processing using random projection gives results equal to or better than PCA. The results also indicate that random projection could be considered a regularization method per se.