Advances in machine learning (ML) are making a large impact in many disciplines, including materials and computational chemistry. A particularly exciting application of ML is the prediction of quantum mechanical (QM) properties (e.g., formation energy, bandgap, etc.) using only the structure as input. Assuming sufficient accuracies in the ML models, these methods enable screening of a considerably large chemical space at orders of magnitude lower computational cost than available QM methods. Despite the promise of ML in chemistry, several key challenges remain in both applying and interpreting the results of ML algorithms. Here, we will discuss our efforts in addressing these issues, including our recent work on opening the black box of ML methods by identifying the domain of applicability, i.e., where a given model is reliable.