We study experimentally a sediment of self-propelled Brownian particles with densities ranging from dilute to ergodic supercooled to nonergodic glass to nonergodic polycrystal. In a companion paper, we observe a nonmonotonic response to activity of relaxation of the nonergodic glass state: a dramatic slowdown when particles become weakly self-propelled, followed by a speedup at higher activities. Here we map ergodic supercooled states to standard passive glassy physics, provided a monotonic shift of the glass packing fraction and the replacement of the ambient temperature by the effective temperature. However, we show that this mapping fails beyond glass transition. This failure is responsible for the nonmonotonic response. Furthermore, we generalize our finding by examining the dynamical response of another class of nonergodic systems: polycrystals. We observe the same nonmonotonic response to activity. To explain this phenomenon, we measure the size of domains where particles move in the same direction. This size also shows a nonmonotonic response, with small lengths corresponding to slow relaxation. This suggests that the failure of the mapping of nonergodic active states to a passive situation is general and is linked to anisotropic relaxation mechanisms specific to active matter.