Measuring Perceptual Video Quality at Scale

MEASURING PERCEPTUAL VIDEO

QUALITY AT SCALE

Encoding pipeline with automated quality monitoring

Evaluate state-of-the-art video compression formats

Optimize encoding recipes and algorithms

Experiment and test on subset of titles

Deploy and run A/B tests

Participate in next-generation codec development

The best video quality possible.

Ways to measure quality

subjectivetest.png

Visual Testing

Tek-pqa600a_01a_h_2.jpg

Video Analyzers

PSNR, SSIM, etc.

HEVCperformance.png

[Ohm, et al., 2012]

Comparative measure;

The PSNR block computes the peak signal-to-noise ratio, in decibels, between two images;

Higher the PSNR, the better the quality of the compressed, or reconstructed image;

8 bits =-48dB / 16 bits =-95dB

255 (8-bit representation), value of PSNR = 20*log(255) = 48dB.

PSNR=10log10(R2/ MSE)

Ris the maximum fluctuation in the input image data type

MSE represents the cumulative squared error between the compressed and the original image

Peak signal-to-noise ratio

IMAGE DATA COMPRESSION INFORMATIONAL REPORT CCSDS 120.1-G-1

Comparative measure;

MSE (or RMSE) is a frequently used measure of the difference between values predicted by a model

and the values observed;

Mean squared error

MSE= 1nΣi=1nSi−Oi2

Oiare the references,Sipredicted values of a variable, andnthe number of observations available for analysis

Bruno Aiazzi, Stefano Baronti ,Massimo Selva. Image Fusion., Algorithms and Applications 2008,

Pages 27-66

A perceptual metric that quantifies image quality

degradation;

SSIM measures the perceptualdifference

Between two similar images;

UnlikePSNR (Peak Signal-to-Noise Ratio), SSIM is

based on visible structures in the image.

Structural similarity -SSIM

http://www.imatest.com/docs/ssim/

Is a Global multiscale extension of SSIM;

Multi-ScaleStructural Similarity Index Measurement MS-SSIM

A close up of a flower

Description automatically generated

Screen Shot 2016-10-08 at 7.28.42 PM.png

PSNR 29.1 dB

Screen Shot 2016-10-08 at 7.28.26 PM.png

PSNR 29.3 dB

Screen Shot 2016-10-08 at 7.49.38 PM.png

PSNR 32.9 dB

Screen Shot 2016-10-08 at 7.51.07 PM.png

PSNR 37.3 dB

Design a video quality metric

●Accurately captures human perception of quality

●Consistent across titles

●Can be run at scale

●For our use case

○Compression artifacts

○Scaling artifacts

What is VMAF?

Video Multimethod Assessment Fusion

Full reference video quality metric

Combines multiple elementary quality metrics

Machine-learning regression to predict a final “fused” score

vmafdiagram.png

Visual Information Fidelity

H. Sheikh and A. Bovik, “Image Information and

Visual Quality,” IEEE Transactions on Image

Processing, vol. 15, no. 2, pp. 430–444, Feb. 2006.

Detail Loss Measure

S. Li, F. Zhang, L. Ma, and K. Ngan, “Image Quality

Assessment by Separately Evaluating Detail Losses

and

...