This review focuses on the responses of the plant cell wall to several abiotic stresses including drought, flooding, heat, cold, salt, heavy metals, light, and air pollutants. The effects of stress on cell wall metabolism are discussed at the physiological (morphogenic), transcriptomic, proteomic and biochemical levels. The analysis of a large set of data shows that the plant response is highly complex. The overall effects of most abiotic stress are often dependent on the plant species, the genotype, the age of the plant, the timing of the stress application, and the intensity of this stress. This shows the difficulty of identifying a common pattern of stress response in cell wall architecture that could enable adaptation and/or resistance to abiotic stress. However, in most cases, two main mechanisms can be highlighted: (i) an increased level in xyloglucan endotransglucosylase/hydrolase (XTH) and expansin proteins, associated with an increase in the degree of rhamnogalacturonan I branching that maintains cell wall plasticity and (ii) an increased cell wall thickening by reinforcement of the secondary wall with hemicellulose and lignin deposition. Taken together, these results show the need to undertake large-scale analyses, using multidisciplinary approaches, to unravel the consequences of stress on the cell wall. This will help identify the key components that could be targeted to improve biomass production under stress conditions.