Abstract
Recognizing the emotion an image evokes in the observer has long attracted the interest of the community for its many potential applications. However, it is a challenging task mainly due to the inherent complexity and subjectivity of human feelings. Such a difficulty is exacerbated in the domain of visual arts, mainly because of their abstract nature. In this work, we propose a new version of the artistic knowledge graph we were working on, namely \(\mathcal {A}rt\mathcal {G}raph\), obtained by integrating the emotion labels provided by the ArtEmis dataset. The proposed graph enables emotion-based information retrieval and knowledge discovery even without training a learning model. In addition, we propose an artwork emotion classification system that jointly exploits visual features and knowledge graph-embeddings. Experimental evaluation revealed that while improvements in emotion classification depend mainly on the use of visual features, the prediction of style, genre and emotion can benefit from the simultaneous exploitation of visual and contextual features and can assist each other in a synergistic way.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Achlioptas, P., Ovsjanikov, M., Haydarov, K., Elhoseiny, M., Guibas, L.J.: ArtEmis: affective language for visual art. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11569–11579 (2021)
Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: Advances in Neural Information Processing Systems 26 (2013)
Bose, D., Somandepalli, K., Kundu, S., Lahiri, R., Gratch, J., Narayanan, S.: Understanding of emotion perception from art. arXiv preprint arXiv:2110.06486 (2021)
Castellano, G., De Carolis, B., Macchiarulo, N., Vessio, G.: Pepper4Museum: towards a human-like museum guide. In: AVI\(^2\)CH@ AVI (2020)
Castellano, G., Digeno, V., Sansaro, G., Vessio, G.: Leveraging knowledge graphs and deep learning for automatic art analysis. Knowl.-Based Syst. 248, 108859 (2022). https://doi.org/10.1016/j.knosys.2022.108859. ISSN: 0950-7051
Castellano, G., Sansaro, G., Vessio, G.: Integrating contextual knowledge to visual features for fine art classification. In: Proceedings of the Workshop on Deep Learning for Knowledge Graphs (DL4KG 2021) (2021)
Castellano, G., Vessio, G.: Deep learning approaches to pattern extraction and recognition in paintings and drawings: an overview. Neural Comput. Appl. 33(19), 12263–12282 (2021)
Cetinic, E., Lipic, T., Grgic, S.: A deep learning perspective on beauty, sentiment, and remembrance of art. IEEE Access 7, 73694–73710 (2019)
David, L., Pedrini, H., Dias, Z., Rocha, A.: Connoisseur: provenance analysis in paintings. In: 2021 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–8. IEEE (2021)
El Vaigh, C.B., Garcia, N., Renoust, B., Chu, C., Nakashima, Y., Nagahara, H.: GCNBoost: artwork classification by label propagation through a knowledge graph. In: Proceedings of the 2021 International Conference on Multimedia Retrieval, pp. 92–100 (2021)
Eyharabide, V., Bekkouch, I.E.I., Constantin, N.D.: Knowledge graph embedding-based domain adaptation for musical instrument recognition. Computers 10(8), 94 (2021)
Garcia, N., Renoust, B., Nakashima, Y.: ContextNet: representation and exploration for painting classification and retrieval in context. Int. J. Multimedia Inf. Retrieval 9(1), 17–30 (2019). https://doi.org/10.1007/s13735-019-00189-4
Grover, A., Leskovec, J.: node2vec: Scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 855–864 (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hogan, A., et al.: Knowledge graphs. Synth. Lect. Data Semant. Knowl. 12(2), 1–257 (2021)
Li, L., Zhu, X., Hao, Y., Wang, S., Gao, X., Huang, Q.: A hierarchical CNN-RNN approach for visual emotion classification. ACM Trans. Multimedia Comput. Commun. Appl. (TOMM) 15(3s), 1–17 (2019)
Machajdik, J., Hanbury, A.: Affective image classification using features inspired by psychology and art theory. In: Proceedings of the 18th ACM International Conference on Multimedia, pp. 83–92 (2010)
Mikels, J.A., Fredrickson, B.L., Larkin, G.R., Lindberg, C.M., Maglio, S.J., Reuter-Lorenz, P.A.: Emotional category data on images from the international affective picture system. Behav. Res. Methods 37(4), 626–630 (2005). https://doi.org/10.3758/BF03192732
Pelowski, M., Specker, E., Gerger, G., Leder, H., Weingarden, L.S.: Do you feel like I do? A study of spontaneous and deliberate emotion sharing and understanding between artists and perceivers of installation art. Psychol. Aesthet. Creat. Arts 14(3), 276 (2020)
Rao, T., Li, X., Xu, M.: Learning multi-level deep representations for image emotion classification. Neural Process. Lett. 51(3), 2043–2061 (2020)
Rao, T., Li, X., Zhang, H., Xu, M.: Multi-level region-based convolutional neural network for image emotion classification. Neurocomputing 333, 429–439 (2019)
Sartori, A.: Affective analysis of abstract paintings using statistical analysis and art theory. In: Proceedings of the 16th International Conference on Multimodal Interaction,. pp. 384–388 (2014)
Song, K., Yao, T., Ling, Q., Mei, T.: Boosting image sentiment analysis with visual attention. Neurocomputing 312, 218–228 (2018)
Tashu, T.M., Hajiyeva, S., Horvath, T.: Multimodal emotion recognition from art using sequential co-attention. J. Imaging 7(8), 157 (2021)
Tinio, P.P., Smith, J.K., Smith, L.F.: The walls do speak: psychological aesthetics and the museum experience. In: The Cambridge Handbook of the Psychology of Aesthetics and the Arts, pp. 195–218. Cambridge University Press (2015)
Yang, J., She, D., Lai, Y.K., Yang, M.H.: Retrieving and classifying affective images via deep metric learning. In: Proceedings of the AAAI Conference on Artificial Intelligence (2018)
Zhao, S., Gao, Y., Jiang, X., Yao, H., Chua, T.S., Sun, X.: Exploring principles-of-art features for image emotion recognition. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 47–56 (2014)
Zhao, S., Zhao, X., Ding, G., Keutzer, K.: EmotionGAN: unsupervised domain adaptation for learning discrete probability distributions of image emotions. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 1319–1327 (2018)
Acknowledgment
G. V. acknowledges funding support from the Italian Ministry of University and Research through the PON AIM 1852414 project.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Aslan, S., Castellano, G., Digeno, V., Migailo, G., Scaringi, R., Vessio, G. (2022). Recognizing the Emotions Evoked by Artworks Through Visual Features and Knowledge Graph-Embeddings. In: Mazzeo, P.L., Frontoni, E., Sclaroff, S., Distante, C. (eds) Image Analysis and Processing. ICIAP 2022 Workshops. ICIAP 2022. Lecture Notes in Computer Science, vol 13373. Springer, Cham. https://doi.org/10.1007/978-3-031-13321-3_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-13321-3_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-13320-6
Online ISBN: 978-3-031-13321-3
eBook Packages: Computer ScienceComputer Science (R0)