Emotional Semantics-Preserved and Feature-Aligned Cycle GAN for Visual Emotion Adaptation
Abstract : Thanks to large-scale labeled training data, deep neural networks (DNNs) have obtained remarkable success in many vision and multimedia tasks. However, because of the presence of domain shift, the learned knowledge of the well-trained DNNs cannot be well generalized to new domains or datasets that have few labels. Unsupervised domain adaptation (UDA) studies the problem of transferring models trained on one labeled source domain to another unlabeled target domain. In this article, we focus on UDA in visual emotion analysis for both emotion distribution learning and dominant emotion classification. Specifically, we design a novel end-to-end cycle-consistent adversarial model, called CycleEmotionGAN++. First, we generate an adapted domain to align the source and target domains on the pixel level by improving CycleGAN with a multiscale structured cycle-consistency loss. During the image translation, we propose a dynamic emotional semantic consistency loss to preserve the emotion labels of the source images. Second, we train a transferable task classifier on the adapted domain with feature-level alignment between the adapted and target domains. We conduct extensive UDA experiments on the Flickr-LDL and Twitter-LDL datasets for distribution learning and ArtPhoto and Flickr and Instagram datasets for emotion classification. The results demonstrate the significant improvements yielded by the proposed CycleEmotionGAN++ compared to state-of-the-art UDA approaches.
? Existing methods mainly focus on a single-source setting, which cannot easily handle a more practical scenario of multiple sources with different distributions.
? Although different sources are matched towards the target, there may exist significant mis-alignment across different sources.
? MADAN also significantly outperforms source-combined DA, in which domain shift also exists among different sources. By bridging this gap, multi-source DA can boost the adaptation performance.
? We can deem this performance as a upper bound of UDA. Obviously, a large performance gap still exists between all adaptation algorithms and the oracle method, requiring further efforts on DA.
? In this paper, we study the unsupervised domain adaptation (UDA) problem of analyzing visual emotions in one labeled source domain and adapting it to another unlabeled target domain.
? In this paper, we study the UDA problem in both image emotion classification and emotion distribution learning tasks.
? Due to the complexity and subjectivity of emotions, we find the accuracies of all domain adaptation methods are not very high and effectively adapting image emotions is still a challenging problem.
? Similar to GAN and CycleGAN based image generation methods, the proposed CycleEmotionGAN++ also suffers from low quality problem.
? We conduct extensive experiments on the ArtPhoto and FI datasets, and the results demonstrate the superiority of the proposed CycleEmotionGAN model.
? The optimization of the proposed CycleEmotionGAN model is achieved by alternating between two stochastic gradient descent (SGD) steps.
? These results demonstrate that the proposed CycleEmotionGAN model can achieve superior performance relative to state-of-the-art approaches.
? We propose to adapt image emotions from one source domain to a target domain in an unsupervised manner.
? A patch-level discriminator architecture has fewer parameters than a full-image discriminator and can work on arbitrarily-sized images in a fully convolutional fashion with good performance.
? The performance comparisons between the proposed Cycle Emotion GAN++ model and state-of-the-art approaches.
? The oracle method achieves the best performance on both emotion distribution learning and dominant emotion classification tasks.
? Dynamic emotional semantic consistency loss boosts the performance by a large margin; after adding it, the performance improves significantly, proving that preserving the emotion label is of vital importance.
? Each of these two components can improve the performance of the model trained in the source domain.
|