Article Sidebar

Download
PDF
Statistic
Read Counter : 141 Download : 0

Main Article Content

Abstract

Dementia is a challenging health issue for healthcare systems across the globe. Communication disabilities and behavioral changes have been significantly impacting the well-being of patients with dementia. The study formulates an empathy-based conversational intervention approach for patients with dementia using multimodal sentiment analysis. The proposed system applies a cross-modal attention-based model to synthesize speech, facial expressions, and biological signals for effective emotion identification. The synthesis is further augmented with an innovative large language model-based conversational response generation module that can develop appropriate empathetic responses. Experiments conducted on public benchmark databases confirm that the trimodal fusion-based model outperforms state-of-the-art methods with an overall weighted average accuracy of approximately 87.3% for emotion identification. The proposed approach outperforms state-of-the-art methods such as MulT, MISA, and MAG-BERT. The scores on human evaluation of the generated empathetic dialogue reached 4.12 and 4.28 in terms of empathy and coherence, with improvements of 17.0% and 12.3% over baseline models. The meta-analytic synthesis of previous clinical evidence revealed significant beneficial effects of social robot interventions on depression, loneliness, and agitation of people with dementia. The comparison with commercial models such as PARO, Pepper, and NAO showed the superiority of the proposed approach over others in terms of multimodal emotion recognition and dialogue adaptability. These results show that socially interactive robots with high emotional intelligence, equipped with cutting-edge affective computing and natural language processing, have great potential for enhancing the quality of dementia care through personalized emotional support.

Keywords

Multimodal sentiment analysis Socially assistive robots Dementia care Empathetic dialogue generation Affective computing

Article Details

How to Cite
Lei, Z., Yin, Y., & Chen, Y. (2026). Emotionally intelligent social robot for dementia care: empathy-based conversational intervention model using multimodal sentiment analysis . Future Technology, 5(2), 200–210. Retrieved from https://fupubco.com/futech/article/view/781
Download Citation
Endnote/Zotero/Mendeley (RIS)
BibTeX
Bookmark and Share

References

  1. Nichols, E., Steinmetz, J. D., Vollset, S. E., et al. (2022). Estimation of the global prevalence of dementia in 2019 and forecasted prevalence in 2050: an analysis for the Global Burden of Disease Study 2019. The Lancet Public Health, 7(2), e105-e125. DOI: 10.1016/S2468-2667(21)00249-8
  2. Livingston, G., Huntley, J., Liu, K. Y., et al. (2024). Dementia prevention, intervention, and care: 2024 report of the Lancet standing Commission. The Lancet, 404(10452), 572-628. DOI: 10.1016/S0140-6736(24)01296-0
  3. Chen, S., Cao, Z., Nandi, A., et al. (2024). The global macroeconomic burden of Alzheimer's disease and other dementias: estimates and projections for 152 countries or territories. The Lancet Global Health, 12(9), e1534-e1543. DOI: 10.1016/S2214-109X(24)00264-X
  4. Shen, C., Rolls, E. T., Cheng, W., et al. (2022). Associations of social isolation and loneliness with later dementia. Neurology, 99(2), e164-e175. https://doi.org/10.1212/WNL.0000000000200583
  5. Hajek, A., & König, H. H. (2025). Prevalence of loneliness and social isolation among individuals with mild cognitive impairment or dementia: systematic review and meta-analysis. BJPsych Open, 11(2), e44. https://doi.org/10.1192/bjo.2024.865.
  6. Guarnera, J., Yuen, E., & Macpherson, H. (2023). The impact of loneliness and social isolation on cognitive aging: a narrative review. Journal of Alzheimer's Disease Reports, 7(1), 699-714. https://doi.org/10.3233/ADR-230011
  7. Liao, X., Wang, Z., Zeng, Q., & Zeng, Y. (2024). Loneliness and social isolation among informal carers of individuals with dementia: A systematic review and meta‐analysis. International Journal of Geriatric Psychiatry, 39(5), e6101. https://doi.org/10.1002/gps.6101
  8. Goto, Y., Morita, K., Suematsu, M., et al. (2023). Caregiver burdens, health risks, coping and interventions among caregivers of dementia patients: a review of the literature. Internal Medicine, 62(22), 3277-3282. https://doi.org/10.2169/internalmedicine.0911-22
  9. Xie, Y., Shen, S., Liu, C., et al. (2024). Internet-Based Supportive Interventions for Family Caregivers of People With Dementia: Randomized Controlled Trial. JMIR Aging, 7(1), e50847. doi:10.2196/50847
  10. Rodríguez-Alcázar, F. J., Juárez-Vela, R., Sánchez-González, J. L., & Martín-Vallejo, J. (2024). Interventions effective in decreasing burden in caregivers of persons with dementia: A meta-analysis. Nursing Reports, 14(2), 931-945. https://doi.org/10.3390/nursrep14020071
  11. Moyle, W. (2023). Grand challenge of maintaining meaningful communication in dementia care. Frontiers in Dementia, 2, 1137897. https://doi.org/10.3389/frdem.2023.1137897
  12. Hockley, A., Moll, D., Littlejohns, J., et al. (2023). Do communication interventions affect the quality-of-life of people with dementia and their families? A systematic review. Aging & Mental Health, 27(9), 1666-1675. https://doi.org/10.1080/13607863.2023.2202635
  13. Mundadan, R. G., Savundranayagam, M. Y., Orange, J. B., & Murray, L. (2023). Language-based strategies that support person-centered communication in formal home care interactions with persons living with dementia. Journal of Applied Gerontology, 42(4), 639-650. https://doi.org/10.1177/07334648221142852
  14. Zhao, D., Sun, X., Shan, B., et al. (2023). Research status of elderly-care robots and safe human-robot interaction methods. Frontiers in Neuroscience, 17, 1291682. https://doi.org/10.3389/fnins.2023.1291682
  15. Abdi, J., Al-Hindawi, A., Ng, T., & Vizcaychipi, M. P. (2018). Scoping review on the use of socially assistive robot technology in elderly care. BMJ Open, 8(2), e018815. https://doi.org/10.1136/bmjopen-2017-018815
  16. Pu, L., Moyle, W., Jones, C., & Todorovic, M. (2019). The effectiveness of social robots for older adults: a systematic review and meta-analysis of randomized controlled studies. The Gerontologist, 59(1), e37-e51. https://doi.org/10.1093/geront/gny046
  17. Yen, H. Y., Huang, C. W., Chiu, H. L., & Jin, G. (2024). The effect of social robots on depression and loneliness for older residents in long-term care facilities: a meta-analysis of randomized controlled trials. Journal of the American Medical Directors Association, 25(6). DOI: 10.1016/j.jamda.2024.02.017
  18. Wang, Y., Song, W., Tao, W., et al. (2022). A systematic review on affective computing: Emotion models, databases, and recent advances. Information Fusion, 83-84, 198-217. https://doi.org/10.1016/j.inffus.2022.03.009
  19. Luo, Y., Chen, Q., Chen, S., & Yao, L. (2024). Factors influencing technology acceptance for socially assistive robots among older adults: A meta-analysis. Journal of Gerontological Nursing, 50(3), 17-24. https://doi.org/10.1177/07334648231202669
  20. Geetha, A. V., Mala, T., Priyanka, D., & Uma, E. (2024). Multimodal sentiment analysis: A comprehensive review of approaches, challenges, and future trends. Information Fusion, 102, 102016. doi: 10.1109/TNNLS.2023.3294810.
  21. Lian, Z., Liu, B., & Tao, J. (2023). A survey of deep learning-based multimodal emotion recognition: Speech, text, and face. Entropy, 25(10), 1440. https://doi.org/10.3390/e25101440
  22. Sharma, A., Lin, I. W., Miner, A. S., et al. (2023). Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence, 5(1), 46-57. https://doi.org/10.1038/s42256-022-00593-2
  23. Sorin, V., Brin, D., Barash, Y., et al. (2024). Large language models and empathy: Systematic review. Journal of Medical Internet Research, 26, e52597. doi:10.2196/52597
  24. Hashem, A., Arif, M., & Alghamdi, M. (2023). Speech emotion recognition approaches: A systematic review. Speech Communication, 154, 102974. https://doi.org/10.1016/j.specom.2023.102974
  25. Li, Y., Wang, Y., Yang, X., & Im, S. K. (2023). Speech emotion recognition based on Graph-LSTM neural network. EURASIP Journal on Audio, Speech, and Music Processing, 2023(1), 40. https://doi.org/10.1186/s13636-023-00303-9
  26. Gaya-Morey, F. X., Buades-Rubio, J. M., Palanque, P., Lacuesta, R., & Manresa-Yee, C. (2025). Deep learning-based facial expression recognition for the elderly: A systematic review. arXiv preprint arXiv:2502.02618. https://doi.org/10.48550/arXiv.2502.02618
  27. Bohi, A., Boudouri, Y. E., & Sfeir, I. (2025). A novel deep learning approach for facial emotion recognition: Application to detecting emotional responses in elderly individuals with Alzheimer's disease. Neural Computing and Applications, 37(6), 5235-5253. https://doi.org/10.1007/s00521-024-10938-0
  28. Ismail, S. N. M. S., Aziz, N. A. A., Ibrahim, S. Z., & Mohamad, M. S. (2024). A systematic review of emotion recognition using cardio-based signals. ICT Express, 10(1), 156-183. https://doi.org/10.1016/j.icte.2023.09.001
  29. Saganowski, S., Komoszyńska, J., Behnke, M., Perz, B., Kunc, D., Klich, B., ... & Kazienko, P. (2022). Emognition dataset: Emotion recognition with self-reports, facial expressions, and physiology using wearables. Scientific Data, 9(1), 158. https://doi.org/10.1038/s41597-022-01262-0
  30. Sun, L., Lian, Z., Liu, B., & Tao, J. (2023). Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis. IEEE Transactions on Affective Computing, 15(1), 309-325. doi: 10.1109/TAFFC.2023.3274829.
  31. Gan, C., Fu, X., Feng, Q., Zhu, Q., Cao, Y., & Zhu, Y. (2024). A multimodal fusion network with attention mechanisms for visual–textual sentiment analysis. Expert Systems with Applications, 242, 122731. https://doi.org/10.1016/j.eswa.2023.122731
  32. Qian, Y., Zhang, W., & Liu, T. (2023, December). Harnessing the power of large language models for empathetic response generation: Empirical investigations and improvements. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 6516-6528). DOI: 10.18653/v1/2023.findings-emnlp.433
  33. Abdollahi, H., Mahoor, M. H., Zandie, R., Siewierski, J., & Qualls, S. H. (2022). Artificial emotional intelligence in socially assistive robots for older adults: A pilot study. IEEE Transactions on Affective Computing, 14(3), 2020-2032. doi: 10.1109/TAFFC.2022.3143803.
  34. Saganowski, S., Perz, B., Polak, A. G., & Kazienko, P. (2022). Emotion recognition for everyday life using physiological signals from wearables: A systematic literature review. IEEE Transactions on Affective Computing, 14(3), 1876-1897. doi: 10.1109/TAFFC.2022.3176135.
  35. Ba, S., & Hu, X. (2023). Measuring emotions in education using wearable devices: A systematic review. Computers & Education, 200, 104797. https://doi.org/10.1016/j.compedu.2023.104797
  36. Rashid, N. L. A., Leow, Y., Klainin-Yobas, P., Itoh, S., & Wu, V. X. (2023). The effectiveness of a therapeutic robot, 'Paro', on behavioural and psychological symptoms, medication use, total sleep time and sociability in older adults with dementia: A systematic review and meta-analysis. International Journal of Nursing Studies, 145, 104530. https://doi.org/10.1016/j.ijnurstu.2023.104530
  37. Hsieh, C. J., Li, P. S., Wang, C. H., Lin, S. L., Hsu, T. C., & Tsai, C. M. T. (2023). Socially assistive robots for people living with dementia in long-term facilities: A systematic review and meta-analysis of randomized controlled trials. Gerontology, 69(8), 1027-1042. https://doi.org/10.1159/000529849
  38. Nam, S. J., & Park, E. Y. (2025). Effectiveness of robot care intervention and maintenance for people with dementia: A systematic review and meta-analysis. Innovation in Aging, 9(3). https://doi.org/10.1093/geroni/igae110