Back to articles
JIST-first
Volume: 29 | Article ID: art00036
Image
New Encoder Learning for Captioning Heavy Rain Images via Semantic Visual Feature Matching
  DOI :  10.2352/J.ImagingSci.Technol.2021.65.5.050402  Published OnlineSeptember 2021
Abstract

Image captioning generates text that describes scenes from input images. It has been developed for high-quality images taken in clear weather. However, in bad weather conditions, such as heavy rain, snow, and dense fog, poor visibility as a result of rain streaks, rain accumulation, and snowflakes causes a serious degradation of image quality. This hinders the extraction of useful visual features and results in deteriorated image captioning performance. To address practical issues, this study introduces a new encoder for captioning heavy rain images. The central idea is to transform output features extracted from heavy rain input images into semantic visual features associated with words and sentence context. To achieve this, a target encoder is initially trained in an encoder-decoder framework to associate visual features with semantic words. Subsequently, the objects in a heavy rain image are rendered visible by using an initial reconstruction subnetwork (IRS) based on a heavy rain model. The IRS is then combined with another semantic visual feature matching subnetwork (SVFMS) to match the output features of the IRS with the semantic visual features of the pretrained target encoder. The proposed encoder is based on the joint learning of the IRS and SVFMS. It is trained in an end-to-end manner, and then connected to the pretrained decoder for image captioning. It is experimentally demonstrated that the proposed encoder can generate semantic visual features associated with words even from heavy rain images, thereby increasing the accuracy of the generated captions.

Subject Areas :
Views 3
Downloads 0
 articleview.views 3
 articleview.downloads 0
  Cite this article 

Son Chang-Hwan, Ye Pung-Hwi, "New Encoder Learning for Captioning Heavy Rain Images via Semantic Visual Feature Matchingin Proc. IS&T 29th Color and Imaging Conf.,  2021,  pp 207 - 218,  https://doi.org/10.2352/J.ImagingSci.Technol.2021.65.5.050402

 Copy citation
  Copyright statement 
Copyright © Society for Imaging Science and Technology 2021
72010350
Color and Imaging Conference
color imaging conf
2166-9635
Society for Imaging Science and Technology
7003 Kilworth Lane, Springfield, VA 22151 USA