A total of 94 patients with colorectal cancer (CRC) were included in this study. Lymphocytic infiltration of CD45 + cells in the normal colon was more pronounced than that in the paired tumor stroma ( p = 0.0008). The mean immunoscore of CD45 +TILs was decreased in CRC compared with the controls ( p = 0.0010). The percentage of CD3 + cells was higher in stage II than in stage IV ( p = 0.0218) and showed a negative correlation with the TNM classification (r = -0.2867, p = 0.0109). The number of stromal CD4 +TILs was higher in stage I than in stage III ( p = 0.0116) and IV ( p = 0.0104), and there was a negative correlation between this number and the stage (r = -0.3708, p = 0.0008). There was a positive correlation between the Ki-67 and CD45 + (r = 0.2468, p = 0.0294), CD3 + (r = 0.3822, p = 0.0006), and CD4 + cells (r = 0.5465, p < 0.0001). The levels of cancer-associated fibroblast (CAF) markers such as α-SMA, thrombin and fibronectin were significantly higher in CRC than in normal colonic mucosa. The immunohistochemical expression of α-SMA was negatively correlated with TILs, while fibronectin showed positive coexpression. A higher number of cells expressing IL-2Rα, PD-L1, CD33 and CD14 were found in colorectal adenocarcinomas than in controls. The number of CD14 + cells was also dependent on the TNM stage ( p = 0.0444) and tumor budding ( p = 0.0324). These findings suggest a suppressive impact of CRC on the adaptive immune response and emphasize the importance of CAFs in regulating tumor immunity.