Back to articles
Articles
Volume: 32 | Article ID: art00010
Image
A Simple Model for Subject Behavior in Subjective Experiments
  DOI :  10.2352/ISSN.2470-1173.2020.11.HVEI-131  Published OnlineJanuary 2020
Abstract

In a subjective experiment to evaluate the perceptual audiovisual quality of multimedia and television services, raw opinion scores offered by subjects are often noisy and unreliable. Recommendations such as ITU-R BT.500, ITU-T P.910 and ITU-T P.913 standardize post-processing procedures to clean up the raw opinion scores, using techniques such as subject outlier rejection and bias removal. In this paper, we analyze the prior standardized techniques to demonstrate their weaknesses. As an alternative, we propose a simple model to account for two of the most dominant behaviors of subject inaccuracy: bias (aka systematic error) and inconsistency (aka random error). We further show that this model can also effectively deal with inattentive subjects that give random scores. We propose to use maximum likelihood estimation (MLE) to jointly estimate the model parameters, and present two numeric solvers: the first based on the Newton-Raphson method, and the second based on alternating projection. We show that the second solver can be considered as a generalization of the subject bias removal procedure in ITU-T P.913. We compare the proposed methods with the standardized techniques using real datasets and synthetic simulations, and demonstrate that the proposed methods have advantages in better model-data fit, tighter confidence intervals, better robustness against subject outliers, shorter runtime, the absence of hard coded parameters and thresholds, and auxiliary information on test subjects. The source code for this work is open-sourced at https://github.com/Netflix/sureal.

Subject Areas :
Views 57
Downloads 12
 articleview.views 57
 articleview.downloads 12
  Cite this article 

Zhi Li, Christos G. Bampis, Lucjan Janowski, Ioannis Katsavounidis, "A Simple Model for Subject Behavior in Subjective Experimentsin Proc. IS&T Int’l. Symp. on Electronic Imaging: Human Vision and Electronic Imaging,  2020,  pp 131-1 - 131-14,  https://doi.org/10.2352/ISSN.2470-1173.2020.11.HVEI-131

 Copy citation
  Copyright statement 
Copyright © Society for Imaging Science and Technology 2020
72010604
Electronic Imaging
2470-1173
Society for Imaging Science and Technology
7003 Kilworth Lane, Springfield, VA 22151 USA