In this paper, we compare the Double-Stimulus Impairment Scale (DSIS) and a Modified Absolute Category Rating (M-ACR) subjective quality evaluation method for HEVC/H.265-encoded omnidirectional videos. These two methods differ in the type of rating scale and presentation of stimuli. Results of our test provide insight into the similarities and differences between these two subjective test methods. Also, we investigate whether the results obtained with these subjective test methods are content-dependent. We evaluated subjective quality on an Oculus Rift for two different resolutions (4K and FHD) and at five different bit-rates. Experimental results show that for 4K resolution, for the lower bit-rates at 1 and 2 MBit/s, M-ACR provides slightly higher MOS compared to DSIS. For 4, 8, 15 MBit/s, DSIS provides slightly higher MOS. While the correlation coefficient between these two methods is very high, M-ACR offers a higher statistical reliability than DSIS. We also compared simulator sickness scores and viewing behavior. Experimental results show that subjects are more prone to simulator sickness while evaluating 360° videos with the DSIS method.
Ashutosh Singla, Werner Robitza, Alexander Raake, "Comparison of Subjective Quality Evaluation Methods for Omnidirectional Videos with DSIS and Modified ACR" in Proc. IS&T Int’l. Symp. on Electronic Imaging: Human Vision and Electronic Imaging, 2018, pp 1 - 6, https://doi.org/10.2352/ISSN.2470-1173.2018.14.HVEI-525