Artificial Intelligence recent rapid rate of development has greatly affected the innovation of intelligent recommendation systems for various fields such as health care, entertainment, education, and lifestyle management. More recently, AI-based culinary recommendation systems are assisting users with meal planning and recipe selection by providing meaningful assistance. Existing systems used to recommend recipes predominantly utilize static parameters (i.e., cuisine type, preparation time, dietary preference, or popularity) that do not consider facial expressions (and thereby represent a person’s mood at that time) as well as contextual information concurrently with those parameters when recommending recipes.

For example, users typically demonstrate various facial expressions when selecting food to eat, such as cheerful expressions indicating an acceptance of a new or celebratory type of dish, while tired or sad expressions indicate a preference for easy, quick meals; however, the majority of current recipe applications do not utilize real-time visual indicators to provide personalized recommendations to their users. Therefore, the results generated from many existing systems are not based upon real-time situational conditions (i.e., user experience, facial expression, and/or visible mood/state).

To overcome this shortcoming, this paper proposes ChefMoji, a facial expression-based and contextually aware AI-assisted cooking assistant that aims to improve the personalization aspect of recipe suggestion. The proposed system utilizes Facial Expression Recognition (FER) methods to interpret the facial image of a user and identify visible expressions like happy, sad, angry, surprised, and neutral. These identified expressions are used as mood indicators, which are facial cue-based and not emotionally or psychologically profound.

ChefMoji does more than just recognize expressions. It also thinks about the weather, the time of day what you have in your pantry and what you like to eat. It uses a kind of computer program to come up with recipe ideas that are just right for you. This program looks at all the information ChefMoji has about you and your kitchen. Then it gives you a list of recipes that you can make with what you have.

ChefMoji also makes it easy for you to follow the recipes. It tells you what to do in a voice so you do not have to read anything. It shows you the recipes, on a website that is easy to use. The system even knows what you have in your pantry and will suggest recipes that use those things. This makes it very easy to cook with ChefMoji.

The significance of this research is in proving the potential of facial expression recognition and contextual intelligence integration for enhancing the relevance and effectiveness of recipe recommendation systems powered by AI. ChefMoji’s focus on overt facial expressions as opposed to emotional analysis makes it a light, interpretable, and user-friendly solution for a personalized cooking assistant. This study shows the potential of facial recognition and generative AI for developing human-centric and engaging cooking assistant systems.

1. Literature Review

Recent developments in Artificial Intelligence (AI), affective computing, and recommender systems have greatly impacted the creation of intelligent culinary assistants. This section discusses previous research works done on three major domains related to ChefMoji: Facial Emotion Recognition (FER), AI-based recipe generation.

1. Facial Emotion Recognition (FER)

Facial Emotion Recognition (FER) has been researched heavily in areas of computer vision and affective computing. One of the first surveys of the FER literature came from I. M. Revina and W. S. Emmanuel [1] as they examined several traditional categories of FER techniques, such as geometric extraction and appearance-based approaches for feature extraction.

Deep learning methods subsequently yielded far better performance for classifying emotions than traditional techniques. Shu Li and Weihong Deng [2] conducted a survey of CNN-based FER systems and showed their superiority compared to handcrafted features through several different methods of classification. Similarly, Jyoti Kumari et al. [7] conducted an analysis of various model types used to classify the five basic emotions that all humans experience - happiness, sadness, anger, fear and surprise.

Emotion-aware systems have become increasingly relevant to adaptive human-computer interaction. N. Agrawal et al. [8] demonstrated mood detection using FER Pipelines while J. X. Y. Lek and J. Teo [9] explored a number of applications of emotion classification in the academic setting. All these works illustrate how the field of affective computing enhances the ability to provide personalized service, and this is in keeping with one of the core principles of ChefMoji.

1. AI-Based Recipe Generation Systems

However, with the development of generative models, AI-assisted recipe generation has attracted the attention of researchers. D. Patel and K. Patel [4] presented an intelligent culinary assistant using the GPT-2 model, which can generate recipes using user inputs. The authors proved the effectiveness of structured recipe generation using transformer models.

Similarly, S. Navaneeth et al. [3] proposed a machine learning-based intelligent recipe generator that generates personalized recipes using user constraints. Another study by the authors, titled "ChefAI: Generating Indian Cuisine Recipes Using AI Algorithms," focuses on the generation of Indian cuisine using AI algorithms by S. Chaudhary et al. [5].

Recently, multimodal models have been proposed to extend the domain of the intelligent recipe generator. B. Hannon et al. [15] proposed Chef DALL·E, which generates cooking experiences using multimodal models, including text and image models. Another study by S. Aliyan and A. P. D. F. Gongor titled "Intelligent Pantry System Based on Chatbots to Generate Recipes" explores the intelligent pantry system using the chatbot model to generate recipes autonomously.

Collectively, the studies discussed above have established the foundation for the development of generative AI-powered intelligent recipe generators, which ChefMoji extends to the domain of emotional intelligence.

1. Recipe Recommendation Systems

Besides generation, the key aspects of a recommendation model are ranking and personalization. Y. Tian et al. [13] introduced RecipeRec, a heterogeneous graph learning model that considers ingredient relationships and user preferences. The use of a graph that has lots of things in it made the recommendations more accurate. It did this by thinking about the complicated relationships between people who use the system and the recipes.

People usually use two methods to filter recommendations in systems that suggest food. These methods are looking at what's, in the food and making sure it meets certain rules. This is especially important when dealing with food allergies and diets. The way ChefMoji ranks recipes is based on these methods.

1. OpenAI API and Generative Frameworks

Nowadays, AI culinary assistants are based on large-scale API-driven architectures. H. Habib in [12] offered implementation strategies to develop intelligent conversational applications using the OpenAI API. These best practices were followed to design the modular and fallback-based architecture of ChefMoji.

1. Research Gap

Although existing literature has shown good advancements in Facial Emotion Recognition, AI-Based Recipe Generation, Nutrition-Aware Recommender Systems, Smart Pantry Management, most of them work in isolation in their respective domains.

However, there is less work that incorporates emotion detection, contextual awareness (including weather and calendar events), AI-based dynamic generation, smart pantry persistence, and conversational interaction in a single culinary tool. ChefMoji aims to fill this research void by integrating affective computing, AI-based generation, and nutrition-aware recommendation in a single emotionally intelligent cooking tool.

Table 1: Salient Findings of various authors for Facial Expression and Recipe Generation

2. Methodology

The methodology of the ChefMoji system focuses on developing an AI-powered cooking assistant that integrates emotion recognition, voice interaction, and intelligent recipe generation. The system is designed using a modular approach where different technologies work together to provide personalized cooking recommendations to the user. The development process includes emotion detection, speech processing, AI-based recipe generation, and system integration.

Figure1: Conceptual Framework (Methodology)

1. System Overview

ChefMoji is an AI-assisted cooking platform that uses emotion recognition technology, automatic recipe generation, and nutrition checks to provide users with a complete cooking experience.

The system builds on previous research in areas such as facial recognition of emotions [1][2], recipe generation using artificial intelligence [3][4], and chat-based cooking applications. As opposed to traditional recommendation systems, ChefMoji employs emotion-based computing, considers other factors (such as weather and time) when recommending recipes, and maintains a history of past pantry items to give users personalized cooking experiences.

1. Emotion Detection Module

ChefMoji uses a hybrid approach to provide real-time recognition of facial emotions. Primary way to identify emotional expressions is through Local FER-based identification. Fallback (if Local FER-based identification does not meet the requirements) uses OpenAI multimodal API for identification based on the images provided by the user. FER techniques can be found within established research in Surveys [1][7] and in deep-learning models [2] utilized to classify emotions independently. The method classifies identified emotions into predefined mood groups: Happy, Sad, Angry, Neutra, Surprised, Fearful, Nostalgic Interactive and Emotion-aware systems provide a significant improvement to personalize users within any interaction [8][9].

1. Context-Aware Personalization Layer

ChefMoji increases the quality of recommendations by considering multiple conditions of context: time of day, weather (e.g. through the OpenWeather API), upcoming events on the calendar. Context-aware food recommendation systems have been shown to increase relevance when environmental factors are included [10]. The system detects impending events via Google Calendar integration or structured JSON inputs and modifies its recommendations accordingly.

1. AI-Based Recipe Generation

By using OpenAI APIs Large Language Model as the main recipe engine I get structured recipe details.

These details include: Cuisine, Description, Ingredients, Directions, Estimated calories, Nutritional breakdown. The OpenAI Large Language Model helps create recipe objects with all this information. It makes it easy to get recipe details like title, cuisine and description. I can also see the ingredients and directions. I can also see the calories and nutritional information. Some recipes even have a note and pairing suggestion. The OpenAI APIs Large Language Model is really helpful, for generating these recipe objects. It provides all the information in a structured way.

These recipes draw inspiration from both GPT-based culinary assistants [4][11] and from newer AI-powered smart recipe generators [3]. The implementation of prompt engineering techniques follows the best practices used in API based generative systems [12].

In cases where access to the API may not be possible, such as an outage, ChefMoji has a back-up system that will create deterministic mood-based recipes without any disruptions to the system.

1. Pantry-Aware Recipe Ranking

A new pantry-based memory system has been added to ChefMoji using SQLite to store ingredients.

Recipes are scored using a matching algorithm based on the following: Count of matched ingredients and Count of missing ingredients. Recipes are ordered according to their maximum pantry match and minimum amount of missing ingredients. The recommendation strategies used to recommend recipes such as RecipeRec [13] use graph and content approaches at the ingredient level as a basis for the algorithm used by ChefMoji to determine recipe matches.

1. Nutritional and Health Considerations

Each recipe contains calorie metadata and nutrition fields. While AI-generated, nutritional logic follows the structured evaluation framework used in health recommender systems [10] and quantitative validation of AI-generated recipes [14]. Possible future extensions include the integration of nutrient validation APIs, similar to those used in studies on clinical nutrition recommender systems [10].

1. Conversational & Voice Interaction

ChefMoji incorporates: Text to speech (TTS) functionality through pyttsx3, Detecting when the voice has stopped speaking via speech recognition. Predefined conversational personality templates that are aligned with the intended emotion of the user. Smart kitchen systems driven by chatbots and conversational culinary assistants have shown an increase in engagement and accessibility [6][15].

1. Persistent Storage & Analytics

Recipes are stored, as are ingredient mappings, mood context, ratings, weather & event metadata. Basic analytics modules include: Most frequent mood, Most used ingredients. Data persistence can be used for user-adaptive learning, as seen in intelligent recommender systems [13].

3. RESULTS AND DISCUSSION

The proposed ChefMoji system was assessed for its reliability in emotion detection, quality of AI-generated recipes, effectiveness of the pantry-based recommendation approach, and contextual-based personalization. The hybrid approach of the proposed Facial Emotion Recognition (FER) mechanism, where local and API-based detection were employed, showed promising performance under different lighting conditions and facial orientations. This is supported by the improved robustness of deep learning-based FER systems over traditional approaches, as highlighted in [2]. The incorporation of emotion mapping for culinary preferences showed promising results for mood-based recipes, as supported by previous research on the impact of affect-based systems on user engagement and personalization [8][9].

The output from the Large Language Model (LLM) based recipe generation module always followed a logical and structured pattern. The semantic consistency and logical structuring of the output can be attributed to the capabilities of AI-based culinary assistants as reported in previous studies [3][4]. The novelty of the proposed system lies in the incorporation of the emotional state, weather conditions, and calendar events into the prompt structure, as opposed to traditional recipe generation systems that consider only ingredient constraints. The qualitative analysis of the output, based on the inspiration from AI-generated recipe system evaluation [14] shows the high compatibility of the ingredients and logical flow of the procedures.

The pantry-aware ranking mechanism greatly enhanced the practicality of the system by prioritizing the recipes that had higher ingredient matches and fewer ingredients to be purchased. This method is conceptually similar to other ingredient-based recommendation methods such as RecipeRec [13], but with the enhancements of persistence and adaptability.

Moreover, contextual integration with respect to the weather and event detection also enhanced the intelligence of the recommendations. For example, the cold weather conditions led to the recommendation of comfort food for meals, while festive events on the calendar led to the recommendation of celebratory food. Such multi-factor adaptation is also similar to the findings of the health-aware and context-driven recommendation systems [10]. The multi-factor integration of the recommendations, as proposed by the hybrid model, also outperforms the recommendations proposed by the individual AI-based culinary assistants [3][6].

The overall findings of the experiments demonstrate the effectiveness of the proposed hybrid model as a solution for integrating emotion awareness with AI-based culinary intelligence.

4. Conclusion

ChefMoji is a new type of artificial intelligence-based kitchen assistant (culinary assistant) that incorporates multiple technologies (e.g., emotion recognition via facial expression) to create recipes, recommend pans based on available ingredients (also called pantry-aware recommendations), and personalize the culinary experience (also known as contextual personalization). Unlike other systems that use basic recipe recommendation methods (e.g., filtering, immediate neighbor), ChefMoji uses the affective computing process to optimize personalization.

The target system is based on well-established (but still evolving) emotion recognition methods [1][2] and adaptive emotion-aware systems [8][9] in that it continuously aligns suggestions for recipes with detected moods of users. By using well-established structured recipe generation models [3][4] for generating structured recipes, the culinary assistant enhances previous AI-based culinary assistants by being able to generate output that is both sensitive to context and can drive conversation. In addition, the implementation of the pantry-aware ranking method used with the system can minimize the frustration of running out of ingredients by using more effective ingredient recommendation strategies like RecipeRec [13] for ingredient level recommendations.

Applying unique contextual, or situational, attributes (e.g., weather & calendar/seasonal events) plays a major role in increasing the personalization of recommendation’s value proposition by providing the foundational components needed for Developing ~ Health/Aware & Context/Aware Recommendation Systems as described in [10]. While there are consistent structures for the nutritional data generated from AI systems, future clinical validation (analogous to the quantitative evaluation framework described in [14]) will increase the precision of this area of work.

In summary; The integration of Affective intelligence, Generative AI, Contextual Adjustment, & Persistent Memory mechanisms developed in ChefMoji demonstrate how significant improvements were made in the function and engagement with intelligent culinary systems through the four technologies used together. The proposed model contributes toward an Innovative New Model that will advance the Development of smart kitchen Technologies as a holistic, Emotion Centred & Contextually Based Model for a Culinary Assistant.

5. Future Scope

Even though ChefMoji is able to effectively combine emotion detection, generative AI, and context-based personalization, several changes can still be made to improve the system.

One area that can be explored in the future is the quantitative benchmarking of the accuracy of the emotion detection system, as well as the quality of the recipes generated. Advanced deep learning-based facial emotion detection models can also be incorporated into the system to improve the accuracy of the system under adverse conditions such as low lighting, occlusion, and varying facial orientations.

The nutritional value of the recipes generated can also be made better through the incorporation of clinical nutrition information. Scientific evaluation of the accuracy of the recipes generated can also be achieved through the use of metrics such as Cosine Similarity and Mean Absolute Percentage Error.

In the future, multimodal AI features can also be incorporated, such as the identification of the ingredients through their images, which can then be uploaded for the generation of recipes.

Furthermore, long-term user interaction can also be utilized for the enhancement of the quality of recommendations through the use of reinforcement learning and graph-based recommendation systems.

Lastly, the integration of IoT-enabled smart kitchen appliances can also transform ChefMoji into an automated smart kitchen assistant.

References

1. Revina, I. M., & Emmanuel, W. S. (2021). A survey on human face expression recognition techniques. Journal of King Saud University-Computer and Information Sciences, 33(6), 619-628.
2. Li, S., & Deng, W. (2020). Deep facial expression recognition: A survey. IEEE transactions on affective computing, 13(3), 1195-1215.
3. Navaneeth, S., Pragathi, V. R., Deepak, M., & Kumar, R. A. (2025, November). AI-Powered Smart Recipe Generator: A Machine Learning Approach. In International Conference on Computer Science and Communication Engineering (ICCSCE 2025)(pp. 937-949). Atlantis Press.
4. Patel, D., & Patel, K. (2025). AI Recipe Generator: An Intelligent Culinary Assistant Using GPT-2 and Multi-Modal AI.
5. Chaudhary, S., Soni, B., Sindhavad, A., Mamaniya, A., Dalvi, A., & Siddavatam, I. (2022, October). ChefAI. IN: Generating Indian recipes with AI algorithm. In 2022 International Conference on Trends in Quantum Computing and Emerging Business Technologies (TQCEBT)(pp. 1-6). IEEE.
6. Kumari, J., Rajesh, R., & Pooja, K. M. (2015). Facial expression recognition: A survey. Procedia computer science, 58, 486-491.
7. Habib, H. (2024). OpenAI API Cookbook: Build intelligent applications including chatbots, virtual assistants, and content generators. Packt Publishing Ltd.
8. Kalpakoglou, K., Calderón-Pérez, L., Boqué, N., Guldas, M., Erdoğan Demir, Ç., Gymnopoulos, L. P., & Dimitropoulos, K. (2025). An AI-based nutrition recommendation system: technical validation with insights from Mediterranean cuisine. Frontiers in Nutrition, 12, 1546107.
9. Karunamurthy, A., Sulaiha, A., & Begam, A. Karunamurthy, A. Sulaiha, Y. Yashitha Begam (2024)Journal ID, 3851, 5221.
10. Tanwar, D., Nobari, T., Chaparro, P., & Panangadan, A. (2025, August). Quantitative Evaluation of AI-Generated Recipes for Health Recommender Systems. In 2025 IEEE International Conference on Information Reuse and Integration and Data Science (IRI)(pp. 367-372). IEEE.
11. Aliyan, S., & Gongor, A. P. D. F. Towards Smart Kitchens: Recent Developments In Chatbot-Based Intelligent Pantry Systems For Personalized Culinary Assistance And Autonomous Recipe Generation.
12. Lek, J. X. Y., & Teo, J. (2023). Academic emotion classification using FER: A systematic review. Human Behavior and Emerging Technologies, 2023(1), 9790005.
13. Ilyas, C. M. A., Nunes, R., Nasrollahi, K., Rehm, M., & Moeslund, T. B. (2021, February). Deep emotion recognition through upper body movements and facial expression. In VISIGRAPP (5: VISAPP)(pp. 669-679).
14. Ruiz-Rincón, S., & Galpin, I. (2024). SnapChef: AI-powered Recipe Suggestions. In ICAI Workshops(pp. 28-40).
15. Hannon, B., Kumar, Y., Li, J. J., & Morreale, P. (2024). Chef dalle: Transforming cooking with multi-model multimodal ai. Computers, 13(7), 156.
16. Tian, Y., Zhang, C., Guo, Z., Huang, C., Metoyer, R., & Chawla, N. V. (2022). RecipeRec: A heterogeneous graph learning model for recipe recommendation. arXiv preprint arXiv:2205.14005.
17. Mavrogiannis, A., Mavrogiannis, C., & Aloimonos, Y. (2024, May). Cook2ltl: Translating cooking recipes to ltl formulae using large language models. In 2024 IEEE International Conference on Robotics and Automation (ICRA)(pp. 17679-17686). IEEE.
18. Mehendale, N. (2020). Facial emotion recognition using convolutional neural networks (FERC). SN Applied Sciences, 2(3), 446.
19. Akhand, M. A. H., Roy, S., Siddique, N., Kamal, M. A. S., & Shimamura, T. (2021). Facial emotion recognition using transfer learning in the deep CNN. Electronics, 10(9), 1036.
20. Gautam, C., & Seeja, K. R. (2023). Facial emotion recognition using Handcrafted features and CNN. Procedia Computer Science, 218, 1295-1303.