Developmental dyslexia is presumed to arise from phonological impairments. Accordingly, people with dyslexia show speech perception deficits taken as indication of impoverished phonological representations. However, the nature of speech perception deficits in those with dyslexia remains elusive. Specifically, there is no agreement as to whether speech perception deficits arise from speech-specific processing impairments, or from general auditory impairments that might be either specific to temporal processing or more general. Recent studies show that general auditory referents such as Long Term Average Spectrum (LTAS, the distribution of acoustic energy across the duration of a sound sequence) affect speech perception. Here we examine the impact of preceding target sounds’ LTAS on phoneme categorization to assess the nature of putative general auditory impairments associated with dyslexia. Dyslexic and typical listeners categorized speech targets varying perceptually from /ga/-/da/ preceded by speech and nonspeech tone contexts varying. Results revealed a spectrally contrastive influence of the preceding context LTAS on speech categorization, with a larger magnitude effect for nonspeech compared to speech precursors. Importantly, there was no difference in the presence or magnitude of the effects across dyslexia and control groups. These results demonstrate an aspect of general auditory processing that is spared in dyslexia, available to support phonemic processing when speech is presented in context.