Multimodal Recipe Recommendation with Heterogeneous Graph Neural Networks
Abstract
:1. Introduction
2. Related Work
2.1. Typical Recommendation
2.2. Recipe Recommendation
3. Problem Description and Formulation
4. Methodology
4.1. Embedding Layer
4.1.1. Encoding for User and Recipe
4.1.2. Feature Transformation with Linear Layer
4.2. Recipe Image Feature Extraction
4.3. Feature Fusion
4.4. GraphSAGE Layer
4.4.1. Message Propagation
4.4.2. Feature Aggregation
4.5. Prediction
5. Experiments
5.1. Dataset
5.2. Experimental Settings
5.2.1. Evaluation Metrics
5.2.2. Baselines
- CF [49]: Collaborative filtering is a common method used in recommendation systems to make personalized recommendations based on user behavior and feedback data. The core idea of this method is that if two users have similar behaviors or interests in some aspects, they may also have similar interests in other aspects.
- Content-based Food Recommendation (CFR) [37]: The method is an improvement of collaborative filtering; it proposes multiple methods for calculating user and recipe similarity and we adopt the Pearson’s correlation algorithm in it. And the method incorporates the relationship between recipes.
- Content-boosted Matrix Factorization (CMF) [35]: Matrix factorization is an important technique in linear algebra and mathematical computing. It aims at splitting a complex matrix into the product of several simpler submatrices or vectors with good interpretability, and matrix factorization can effectively handle the sparse data problem in recipe recommendations. The method is an improvement on matrix factorization; it incorporates ingredient information into matrix factorization.
- LightGCN [50]: A Graph Convolutional Network (GCN) is a deep learning model used for processing graph-structured data, such as user–item relationships found in social networks, recommendation systems, and molecular structures in bioinformatics. A GCN is capable of handling complex user–item relationships, provides good interpretability, and is suitable for large-scale recommendation systems. In this approach, the authors extend a GCN to develop a lightweight and effective recommendation model. They eliminate unnecessary elements and modify the approach to neighborhood aggregation and message propagation.
- GTN [51]: Graph Neural Networks (GNNs) have been widely used in recommendation systems and have shown remarkable effectiveness. However, most current GNN-based recommendation systems tend to neglect interactions due to unreliable behavior (e.g., random/clickbait) and treat all interactions uniformly; this approach can lead to suboptimal and unstable performance. To overcome these limitations, the authors introduce a principled graph trend collaborative filtering method. They present Graph Trend Filtering Networks for Recommendations (GTNs), which are specifically designed to capture the adaptive reliability of interactions.
- GraphDA [52]: Graph Collaborative Filtering (GCF) is widely used to capture complex collaborative signals in recommendation systems. However, GCF faces challenges with its bipartite adjacency matrix, especially for users/items with abundant or insufficient interactions; this matrix, which defines aggregated neighbors based on user–item interactions, can introduce noise. Moreover, it neglects user–user and item–item correlations, which limits the inclusion of useful neighbors. In this approach, the authors propose a new graph adjacency matrix that incorporates user–user and item–item correlations. They also introduce a carefully designed user–item interaction matrix that aims to balance the number of interactions across users.
5.2.3. Implementation Details
5.3. Comparative Experiments
5.4. Ablation Experiment
5.4.1. The Impact of Adding Image Modality to Recipe Recommendation
5.4.2. The Impact of Linear Transformation in GraphSAGE with Different Output Dimensions
6. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Harvey, M.; Ludwig, B.; Elsweiler, D. Learning User Tastes: A First Step to Generating Healthy Meal Plans? In Proceedings of the First International Workshop on Recommendation Technologies for Lifestyle Change (Lifestyle 2012), Dublin, Ireland, 13 September 2012; Volume 18. [Google Scholar]
- Trattner, C.; Elsweiler, D. Investigating the Healthiness of Internet-Sourced Recipes: Implications for Meal Planning and Recommender Systems. In Proceedings of the 26th International Conference on World Wide Web, Perth, Australia, 3–7 May 2017; pp. 489–498. [Google Scholar]
- Ge, M.; Ricci, F.; Massimo, D. Health-aware Food Recommender System. In Proceedings of the 9th ACM Conference on Recommender Systems, Vienna, Austria, 16–20 September 2015; pp. 333–334. [Google Scholar]
- Marshall, J.; Jimenez-Pazmino, P.; Metoyer, R.; Chawla, N.V. A Survey on Healthy Food Decision Influences through Technological Innovations. ACM Trans. Comput. Healthc. 2022, 3, 1–27. [Google Scholar] [CrossRef]
- Harashima, J.; Ariga, M.; Murata, K.; Ioki, M. A Large-scale Recipe and Meal Data Collection as Infrastructure for Food Research. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia, 23–28 May 2016; Calzolari, N., Choukri, K., Declerck, T., Goggi, S., Grobelnik, M., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., et al., Eds.; European Language Resources Association: Paris, France, 2016. [Google Scholar]
- Teng, C.; Lin, Y.R.; Adamic, L.A. Recipe recommendation using ingredient networks. In Proceedings of the Web Science 2012, WebSci ‘12, Evanston, IL, USA, 22–24 June 2012; Contractor, N.S., Uzzi, B., Macy, M.W., Nejdl, W., Eds.; ACM: New York, NY, USA, 2012; pp. 298–307. [Google Scholar] [CrossRef]
- Min, W.; Bao, B.K.; Mei, S.; Zhu, Y.; Rui, Y.; Jiang, S. You are what you eat: Exploring rich recipe information for cross-region food analysis. IEEE Trans. Multimed. 2017, 20, 950–964. [Google Scholar] [CrossRef]
- Min, W.; Jiang, S.; Jain, R. Food Recommendation: Framework, Existing Solutions, and Challenges. IEEE Trans. Multimed. 2019, 22, 2659–2671. [Google Scholar] [CrossRef]
- Tian, Y.; Zhang, C.; Metoyer, R.A.; Chawla, N.V. Recipe Recommendation with Hierarchical Graph Attention Network. Front. Big Data 2021, 4, 778417. [Google Scholar] [CrossRef] [PubMed]
- Pacífico, L.D.S.; Britto, L.F.S.; Ludermir, T.B. Ingredient Substitute Recommendation Based on Collaborative Filtering and Recipe Context for Automatic Allergy-Safe Recipe Generation. In Proceedings of the WebMedia ‘21: Brazilian Symposium on Multimedia and the Web, Belo Horizonte, Minas Gerais, Brazil, 5–12 November 2021; Pereira, A.C.M., da Rocha, L.C.D., Eds.; ACM: New York, NY, USA, 2021; pp. 97–104. [Google Scholar] [CrossRef]
- Jia, N.; Chen, J.; Wang, R. An Attention-Based Convolutional Neural Network for Recipe Recommendation. Expert Syst. Appl. 2022, 201, 116979. [Google Scholar] [CrossRef]
- Tian, Y.; Zhang, C.; Guo, Z.; Huang, C.; Metoyer, R.A.; Chawla, N.V. RecipeRec: A Heterogeneous Graph Learning Model for Recipe Recommendation. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, Vienna, Austria, 23–29 July 2022. [Google Scholar]
- Chavan, P.; Thoms, B.; Isaacs, J. A Recommender System for Healthy Food Choices: Building a Hybrid Model for Recipe Recommendations using Big Data Sets. In Proceedings of the 54th Hawaii International Conference on System Sciences, HICSS 2021, Kauai, HI, USA, 5 January 2021; pp. 1–10. [Google Scholar]
- Ranaldi, L.; Pucci, G. Knowing knowledge: Epistemological study of knowledge in transformers. Appl. Sci. 2023, 13, 677. [Google Scholar] [CrossRef]
- Hamilton, W.L.; Ying, Z.; Leskovec, J. Inductive Representation Learning on Large Graphs. In Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA, 4–9 December 2017; Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., Garnett, R., Eds.; Curran Associates, Inc.: New York, NY, USA, 2017; pp. 1025–1035. [Google Scholar]
- Chum, O.; Zisserman, A. An Exemplar Model for Learning Object Classes. In Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 17–22 June 2007; pp. 1–8. [Google Scholar]
- Melville, P.; Mooney, R.J.; Nagarajan, R. Content-Boosted Collaborative Filtering for Improved Recommendations. Proc. Eighteenth Natl. Conf. Artif. Intell. (AAAI) 2002, 23, 187–192. [Google Scholar]
- Pazzani, M.J.; Billsus, D. Content-Based Recommendation Systems. In The Adaptive Web: Methods and Strategies of Web Personalization; Springer: Berlin/Heidelberg, Germany, 2007; pp. 325–341. [Google Scholar]
- Linden, G.; Smith, B.; York, J. Amazon.com Recommendations: Item-to-Item Collaborative Filtering. IEEE Internet Comput. 2003, 7, 76–80. [Google Scholar] [CrossRef]
- Adomavicius, G.; Tuzhilin, A. Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions. IEEE Trans. Knowl. Data Eng. 2005, 17, 734–749. [Google Scholar] [CrossRef]
- Sarwar, B.; Karypis, G.; Konstan, J.; Riedl, J. Item-Based Collaborative Filtering Recommendation Algorithms. In Proceedings of the 10th International Conference on World Wide Web, Hong Kong, 1–5 May 2001; pp. 285–295. [Google Scholar]
- Su, X.; Khoshgoftaar, T.M. A Survey of Collaborative Filtering Techniques. Adv. Artif. Intell. 2009, 2009, 421425. [Google Scholar] [CrossRef]
- Li, S.; Kawale, J.; Fu, Y. Deep Collaborative Filtering via Marginalized Denoising Auto-encoder. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, Melbourne, Australia, 19–23 October 2015; pp. 811–820. [Google Scholar]
- He, X.; Liao, L.; Zhang, H.; Nie, L.; Hu, X.; Chua, T.S. Neural Collaborative Filtering. In Proceedings of the 26th International Conference on World Wide Web, Perth, Australia, 3–7 April 2017; pp. 173–182. [Google Scholar]
- Andrade-Ruiz, G.; Carrasco, R.A.; Porcel, C.; Serrano-Guerrero, J.; Mata, F.; Arias-Oliva, M. Emerging Perspectives on the Application of Recommender Systems in Smart Cities. Electronics 2024, 13, 1249. [Google Scholar] [CrossRef]
- Pecune, F.; Callebert, L.; Marsella, S. A Recommender System for Healthy and Personalized Recipes Recommendations. In Proceedings of the 5th International Workshop on Health Recommender Systems Co-Located with the 14th ACM Conference on Recommender Systems 2020 (RecSys 2020), Online, 26 September 2020; pp. 15–20. [Google Scholar]
- Ma, X.; Gao, Z.; Hu, Q.; Abdelhady, M. Contrastive Knowledge Graph Attention Network for Request-Based Recipe Recommendation. In Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Virtual, 7–13 May 2022; pp. 3278–3282. [Google Scholar]
- Morol, M.K.; Rokon, M.S.J.; Hasan, I.B.; Saif, A.; Khan, R.H.; Das, S.S. Food Recipe Recommendation Based on Ingredients Detection Using Deep Learning. In Proceedings of the 2nd International Conference on Computing Advancements, Dhaka, Bangladesh, 10–12 March 2022; pp. 191–198. [Google Scholar]
- Peng, J.; Gong, J.; Zhou, C.; Zang, Q.; Fang, X.; Yang, K.; Yu, J. KGCFRec: Improving Collaborative Filtering Recommendation with Knowledge Graph. Electronics 2024, 13, 1927. [Google Scholar] [CrossRef]
- Spoladore, D.; Colombo, V.; Arlati, S.; Mahroo, A.; Trombetta, A.; Sacco, M. An Ontology-Based Framework for a Telehealthcare System to Foster Healthy Nutrition and Active Lifestyle in Older Adults. Electronics 2021, 10, 2129. [Google Scholar] [CrossRef]
- Saad, M.H.M.; Hamdan, N.M.; Sarker, M.R. State of the Art of Urban Smart Vertical Farming Automation System: Advanced Topologies, Issues and Recommendations. Electronics 2021, 10, 1422. [Google Scholar] [CrossRef]
- Chen, S.; Cao, Q.; Cai, Y. Blockchain for Healthcare Games Management. Electronics 2023, 12, 3195. [Google Scholar] [CrossRef]
- Zhang, L.; Kim, D. A Peer-to-Peer Smart Food Delivery Platform Based on Smart Contract. Electronics 2022, 11, 1806. [Google Scholar] [CrossRef]
- Berkovsky, S.; Freyne, J. Group-Based Recipe Recommendations: Analysis of Data Aggregation Strategies. In Proceedings of the Fourth ACM Conference on Recommender Systems, Barcelona, Spain, 26–30 September 2010; pp. 111–118. [Google Scholar]
- Forbes, P.; Zhu, M. Content-boosted matrix factorization for recommender systems: Experiments with recipe recommendation. In Proceedings of the 2011 ACM Conference on Recommender Systems, RecSys 2011, Chicago, IL, USA, 23–27 October 2011; Mobasher, B., Burke, R.D., Jannach, D., Adomavicius, G., Eds.; ACM: Tokyo, Japan, 2011; pp. 261–264. [Google Scholar] [CrossRef]
- Bilgin, A.; Hagras, H.; van Helvert, J.; Al-Ghazzawi, D. A Linear General Type-2 Fuzzy-Logic-Based Computing with Words Approach for Realizing an Ambient Intelligent Platform for Cooking Recipe Recommendation. IEEE T. Fuzzy Syst. 2016, 24, 306–329. [Google Scholar] [CrossRef]
- Freyne, J.; Berkovsky, S. Intelligent Food Planning: Personalized Recipe Recommendation. In Proceedings of the 15th International Conference on Intelligent User Interfaces, Hong Kong, China, 7–10 February 2010; pp. 321–324. [Google Scholar]
- Elsweiler, D.; Trattner, C.; Harvey, M. Exploiting Food Choice Biases for Healthier Recipe Recommendation. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, 7–11 August 2017; Kando, N., Sakai, T., Joho, H., Li, H., de Vries, A.P., White, R.W., Eds.; ACM: Tokyo, Japan, 2017; pp. 575–584. [Google Scholar] [CrossRef]
- Ozeki, S.; Kotera, M.; Ishiguro, K.; Nishimura, T.; Higuchi, K. Recipe Recommendation for Balancing Ingredient Preference and Daily Nutrients. In Proceedings of the CEA++@MM 2022: Proceedings of the 1st International Workshop on Multimedia for Cooking, Eating, and Related APPlications, Lisboa, Portugal, 10 October 2022; Yamakata, Y., Hashimoto, A., Chen, J., Eds.; ACM: Tokyo, Japan, 2022; pp. 11–19. [Google Scholar] [CrossRef]
- Yuan, J.; Roy Chowdhury, P.K.; McKee, J.; Yang, H.L.; Weaver, J.; Bhaduri, B. Exploiting Deep Learning and Volunteered Geographic Information for Mapping Buildings in Kano, Nigeria. Sci. Data 2018, 5, 180217. [Google Scholar] [CrossRef]
- Li, M.; Li, L.; Xie, Q.; Yuan, J.; Tao, X. MealRec: A Meal Recommendation Dataset. arXiv 2022, arXiv:2205.12133. [Google Scholar]
- Lin, C.J.; Kuo, T.T.; Lin, S.D. A Content-Based Matrix Factorization Model for Recipe Recommendation. In Proceedings of the Advances in Knowledge Discovery and Data Mining: 18th Pacific-Asia Conference, Tainan, Taiwan, 13–16 May 2014; pp. 560–571. [Google Scholar]
- Khan, M.A.; Rushe, E.; Smyth, B.; Coyle, D. Personalized, Health-Aware Recipe Recommendation: An Ensemble Topic Modeling Based Approach. arXiv 2019, arXiv:1908.00148. [Google Scholar]
- Freyne, J.; Berkovsky, S.; Smith, G. Recipe Recommendation: Accuracy and Reasoning. In Proceedings of the User Modeling, Adaption and Personalization: 19th International Conference, Girona, Spain, 11–15 July 2011; pp. 99–110. [Google Scholar]
- Starke, A.; Trattner, C.; Bakken, H.; Johannessen, M.; Solberg, V. The Cholesterol Factor: Balancing Accuracy and Health in Recipe Recommendation through a Nutrient-Specific Metric. In Proceedings of the 1st Workshop on Multi-Objective Recommender Systems, Amsterdam, The Netherlands, 25 September 2021. [Google Scholar]
- Koren, Y.; Bell, R.; Volinsky, C. Matrix Factorization Techniques for Recommender Systems. Computer 2009, 42, 30–37. [Google Scholar] [CrossRef]
- Herlocker, J.L.; Konstan, J.A.; Terveen, L.G.; Riedl, J.T. Evaluating Collaborative Filtering Recommender Systems. ACM Trans. Inf. Syst. 2004, 22, 5–53. [Google Scholar] [CrossRef]
- Wang, C.; Blei, D.M. Collaborative Topic Modeling for Recommending Scientific Articles. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Diego, CA, USA, 21–24 August 2011; pp. 448–456. [Google Scholar]
- Konstan, J.A.; Miller, B.N.; Maltz, D.; Herlocker, J.L.; Gordon, L.R.; Riedl, J. GroupLens: Applying Collaborative Filtering to Usenet News. Commun. ACM 1997, 40, 77–87. [Google Scholar] [CrossRef]
- He, X.; Deng, K.; Wang, X.; Li, Y.; Zhang, Y.D.; Wang, M. LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2020, Virtual, 25–30 July 2020; Huang, J., Chang, Y., Cheng, X., Kamps, J., Murdock, V., Wen, J.R., Liu, Y., Eds.; ACM: Tokyo, Japan, 2020; pp. 639–648. [Google Scholar] [CrossRef]
- Fan, W.; Liu, X.; Jin, W.; Zhao, X.; Tang, J.; Li, Q. Graph Trend Filtering Networks for Recommendations. In Proceedings of the SIGIR ‘22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, 11–15 July 2022; Amigó, E., Castells, P., Gonzalo, J., Carterette, B., Culpepper, J.S., Kazai, G., Eds.; ACM: Tokyo, Japan, 2022; pp. 112–121. [Google Scholar] [CrossRef]
- Fan, Z.; Xu, K.; Dong, Z.; Peng, H.; Zhang, J.; Yu, P.S. Graph Collaborative Signals Denoising and Augmentation for Recommendation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, 23–27 July 2023; Chen, H.H., Duh, W.J.E., Huang, H.H., Kato, M.P., Mothe, J., Poblete, B., Eds.; ACM: Tokyo, Japan, 2023; pp. 2037–2041. [Google Scholar] [CrossRef]
Item | Description | Example |
---|---|---|
recipe name | name of the recipe | Coconut Poke Cake |
image url | the url of the recipe image | images.media-allrecipes.com/334118.jpg |
ingredients | recipe ingredients | white cake mix; cream of coconut… |
rating | user ratings for recipes | user id: 19; recipe id: 59; rating: 4 |
Methods | ACC | MAE | RMSE |
---|---|---|---|
CF | 5.44% | 0.8827 | 0.8891 |
CFR | 4.62% | 0.8903 | 0.8964 |
CMF | 55.57% | 0.4166 | 0.4899 |
LightGCN | 85.59% | 0.1272 | 0.1990 |
GTN | 88.13% | 0.1102 | 0.1406 |
GraphDA | 86.96% | 0.1258 | 0.1443 |
MHGRR (Ours) | 90.73% | 0.0811 | 0.0923 |
Input Modal | ACC | MAE | RMSE |
---|---|---|---|
Only text modal | 89.59% | 0.0921 | 0.1448 |
Text modal and image modal | 90.73% ↑ | 0.0811 ↑ | 0.0923 ↑ |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Ouyang, R.; Huang, H.; Ou, W.; Liu, Q. Multimodal Recipe Recommendation with Heterogeneous Graph Neural Networks. Electronics 2024, 13, 3283. https://doi.org/10.3390/electronics13163283
Ouyang R, Huang H, Ou W, Liu Q. Multimodal Recipe Recommendation with Heterogeneous Graph Neural Networks. Electronics. 2024; 13(16):3283. https://doi.org/10.3390/electronics13163283
Chicago/Turabian StyleOuyang, Ruiqi, Haodong Huang, Weihua Ou, and Qilong Liu. 2024. "Multimodal Recipe Recommendation with Heterogeneous Graph Neural Networks" Electronics 13, no. 16: 3283. https://doi.org/10.3390/electronics13163283