Carbon has long been an important nuclear material, either in the form of graphite for moderation in reactors or as diamond for radiation-hard dosimeters. Despite its relevance, remarkably few molecular dynamics simulations have been performed to study radiation response in carbon-based materials. Using the Environment Dependent Interaction Potential, we have systematically studied radiation damage in graphite and diamond. Even though graphite and diamond are strikingly different solids in terms of mechanical, electrical and optical properties, cascade simulations show that their response under irradiation is remarkably similar. In both materials the cascade lifetime is exceedingly short, no thermal spike is evident, and damage consists of isolated point defects. Furthermore, the traditional NRT model is useful for both cases, in particular for graphite. In a separate but related study, threshold displacement energies are calculated using a robust sampling method which yields statistically sound values for both materials.