Deep network-based material models provide advantageous computational efficiency over classical crystal plasticity simulations. In this work, a deep network-based surrogate model, more specifically, a Gated Recurrent Unit (GRU) model, is proposed to address the microstructure-sensitive and history dependent response. Visco-plastic self-consistent model incorporating the twinning and detwinning (VPSC-TDT) scheme is used for training the GRU to capture the mechanical response of ZEK100 magnesium alloy sheet. Considering disorder of high-dimensional texture data, a similar PointNet is developed to automatically learn low-dimension representation of microstructure. Subsequently, GRU is used to learn the plasticity-constitutive relations such that it can predict the stress histories of various microstructures and loading paths. This work demonstrates that deep network-based models trained by micromechanical simulations capture material behavior and its relation to microstructural mechanisms in a physically sound way. Compared to crystal plasticity models, the proposed model not only significantly accelerates the simulations, but also sustains a comparable accuracy.