Recent evidence suggests that temporal lobe epilepsy affects a neuronal network rather than a single circumscribed structure within the temporal lobe. Thus, in mesial temporal lobe epilepsy (mTLE) with hippocampal sclerosis gray matter abnormalities have been found beyond the hippocampus in extrahippocampal regions by means of voxel-based morphometry (VBM). On the contrary, in cryptogenic or MRI-negative TLE (cTLE) no consistent gray matter abnormalities in VBM studies have been reported. We used optimized VBM with modulation to detect gray matter abnormalities compared to healthy controls in patients with mTLE and cTLE. Twenty-two patients with mTLE (right/left TLE 13:9), 17 patients with cTLE (right/left TLE 7:10), and 12 healthy controls were enrolled in the study. In mTLE we found decreased gray matter volume (GMV) beyond the hippocampus in the ipsilateral thalamus. GMV decrease was more widespread in patients with left-sided seizure focus including the left parahippocampal and superior temporal gyrus, frontal regions, cerebellum, and the right cingulum. In cTLE, decreased GMV was observed in the frontal and orbitofrontal cortex, the cerebellum, neocortical temporal regions, and in the right parahippocampal cortex. Again, patients with left-sided seizure focus had a more widespread and extensive GMV decrease including regions such as the right and left cingulum. We found evidence for distinct neuronal network damage in mesial temporal lobe epilepsy (mTLE) and cryptogenic TLE (cTLE) which is more widespread in patients with left-sided seizure focus. Atrophy of the cingulum was a common feature in left- but not in right-sided mTLE and cTLE.