Retaining nano-sized grains during processing is challenging with conventional sintering processes. Efforts to mitigate grain growth often rely on sintering aids to lower processing temperatures or dopants that segregate to the grain boundary to induce solute drag. However, anisotropic grain boundary energy often results in heterogeneous solute distribution and, thus, variation in grain boundary behavior. Here, we investigate how anisotropic grain boundary energy distributions lead to variations in irregular grain growth behavior in doped Al<sub>2</sub>O<sub>3</sub>, SrTiO<sub>3</sub>, and MgAl<sub>2</sub>O<sub>4</sub>. Grain boundary energy, measured with atomic force microscopy, is correlated with the boundary’s atomic structure and chemistry, which is characterized at the atomic scale with aberration-corrected scanning transmission electron microscopy. The advantages and challenges for using anisotropic grain boundary energy for controlling grain growth will be discussed.