Conflicting reports regarding whether high tumor-associated neutrophils (TAN) are associated with outcomes in colorectal cancer (CRC) exist. Previous investigators have counted TAN using non-neutrophil-specific immunohistochemistry (IHC) stains. We examined whether TAN levels as determined by multi-field manual counting would predict prognosis. IRB approval was obtained and two pathologists, blinded to stage/outcome, counted TAN in 20 high power fields (HPF) per specimen. TAN score was defined as the mean of these counts. High TAN was defined as at or greater than the median score for that stage. Demographics, tumor characteristics, and overall survival (OS) were obtained from the records and examined for association with TAN score. IHC for arginase expression was performed in a subset of samples. 221 patients were included. Stage II patients with high TAN scores had an OS of 232 months as compared to those with low TAN (OS = 85 months, p = 0.03). The survival benefit persisted in multivariable analysis (HR 0.48, CI 0.25–0.91, p = 0.026) controlling for age and sex. Women had increased survival as compared to men, and there were no significant prognostic associations with TAN count in stage III/IV patients, although there were only 12 stage IV patients. Arginase staining did not provide additional information. Stage II colorectal cancer patients with high TAN live nearly 3 times longer than those with low TAN. Women with stage II disease and high TAN counts appear to be driving the survival benefit seen in the stage II patients and have increased overall survival in all stages.