An analysis of the matrix models representing the polarimetric properties of light and material media is carried out by using the concept of the coherency matrix, which leads to the identification and definition of their corresponding physical quantities. For light, cases of homogeneous and inhomogeneous wavefront are analyzed, and a model for 3D polarimetric purity is formulated. For linear passive material media, a general model is developed on the basis that any physically realizable linear transformation of Stokes vectors is equivalent to an ensemble average of passive, deterministic, nondepolarizing transformations. Through this framework, the relevant physical quantities, including the indices of polarimetric purity, are identified and decoupled. Some decompositions of the whole system into a set of well-defined components are considered, as well as techniques for isolating the unknown components by means of new procedures for subtracting coherency matrices. These results and methods constitute a powerful tool for analyzing and exploiting experimental and industrial polarimetry. Some particular application examples are indicated.