Microstructural features influence material properties, thus characterization of these features is essential to understanding and predicting material performance. One powerful tool for microstructural characterization is electron backscatter diffraction (EBSD). EBSD is a scanning electron microscopy (SEM) based technique that provides information about crystal structure and orientation, allowing quantification of important features such as shape, size and orientation of grains. While EBSD can provide detailed and accurate information about microstructures, this technique is time-consuming and expensive, limiting its utility for high-throughput microstructural analysis. Here, we describe deep convolutional neural networks that take backscatter electron SEM images, which are easier and faster to collect, and perform semantic segmentation to identify grains and grain boundaries, allowing for microstructural analysis typically accessible through EBSD. We demonstrate the utility and accuracy of our models by performing grain size and shape analysis for stainless steel.