Soft, disordered, micro-structured materials are ubiquitous in nature and industry, and are different from ordinary fluids or solids, with unusual, interesting static and flow properties. The transition from fluid to solid -at the so-called jamming density- features a multitude of complex mechanisms, but there is no unified theoretical framework that explains them all. In this study, a simple yet quantitative and predictive model is presented, which allows for a variable, changing jamming density, encompassing the memory of the deformation history and explaining a multitude of phenomena at and around jamming. The jamming density, now introduced as a new state-variable, changes due to the deformation history and relates the system's macroscopic response to its microstructure. The packing efficiency can increase logarithmically slow under gentle repeated (isotropic) compression, leading to an increase of the jamming density. In contrast, shear deformations cause anisotropy, changing the packing efficiency exponentially fast with either dilatancy or compactancy. The memory of the system near jamming can be explained by a microstatistical model that involves a multiscale, fractal energy landscape and links the microscopic particle picture to the macroscopic continuum description, providing a unified explanation for the qualitatively different flow-behavior for different deformation modes. To complement our work, a recipe to extract the history-dependent jamming density from experimentally accessible data is proposed, and alternative state-variables are compared. The proposed simple macroscopic constitutive model is calibrated with the memory of microstructure. Such approach can help understanding predicting and mitigating failure of structures or geophysical hazards, and will bring forward industrial process design/optimization, and help solving scientific challenges in fundamental research.