In this article, we dissect ensembles in order to provide different insights that are useful for understanding the dynamics of deep neural networks in general.