On the Correspondence between Human Vision and Convolutional Neural Networks: A Visual Quality Assessment Perspective

Abstract

Deep features of convolutional neural networks (CNNs), designed for high-level computer vision tasks such as object detection, have been shown to be also effective for image quality assessment (IQA). This motivates further investigations to understand why CNNs are good IQA estimators and, in a broader sense, whether a correspondence exists between CNN's processing stages and the hierarchical mechanism of the human visual system (HVS). In this paper, we stimulate CNNs with a new family of maximally-regular textures to investigate if higher areas of the visual cortex can be approximated within the CNN layers. The results show interesting correspondence between CNNs and the visual cortex, suggesting that these frameworks might be able to serve as suitable baseline candidates for developing a new generation of quality metrics that better replicate the complex stages of the HVS. Developing such IQA models needs in-depth research into the mechanism of the HVS and designing CNNs with better quality-aware feature encoding. As an initial step toward this goal, we established new criteria for improving the performance of pre-trained networks in quality assessment applications by leveraging texture sensitivity. The outcomes illustrate that our feature map weighting and neuron selection criteria could improve the IQA task.