To quantify the excellence of multimedia quality, subjective evaluation experiments
are conducted. In these experiments, the tradition of quantitative assessment is the
most dominating, but it disregards the understanding of participants' interpretations,
descriptions, and the evaluation criteria of quality. The goal of this paper is to
present a new multimedia quality evaluation method called Open Profiling of Quality
(OPQ) as a tool for building a deeper understanding on subjective quality. OPQ is
a mixed method combining a conventional quantitative psychoperceptual evaluation and
qualitative descriptive quality evaluation based on the individual's own vocabulary.
OPQ is targeted for naïve participants applicable to experiments with heterogeneous
and multimodal stimulus material. The paper presents the theoretical basis of the
development of OPQ and overviews the methods for audiovisual quality research. We
present three extensive quality evaluation studies where OPQ has been used with 120
participants. Finally, we conclude further recommendations of use of the method in
quality evaluation research.