We present a continuum framework for dislocation structure, energy and dynamics of low angle grain boundaries which may be nonplanar or nonequilibrium. We define dislocation density potential functions on the grain boundary to describe the orientation dependent continuous distribution of dislocations. The continuum formulation incorporates both the long-range dislocation interaction and the local dislocation line energy. Within this framework, we develop a continuum model to compute the energy of low angle grain boundaries for any given degrees of freedom (arbitrary rotation axis, rotation angle and boundary plane orientation) based on a continuum dislocation structure, in which the grain boundary energy associated with the dislocation structure is minimized subject to the constraint of Frank's formula for dislocations with all possible Burgers vectors. The continuum model is applied to dislocation structure and energy of low angle grain boundaries in fcc Al (L.C.Zhang, Y.J.Gu and Y.Xiang, Acta Mater. 126, 11-24, 2017).