Engineering and Technology
Hsin-Chieh Lee, MS
Occupational therapist; PhD student
Department of Physical Medicine and Rehabilitation, Shuang Ho Hospital, Taipei Medical University, Taipei; School of Occupational Therapy, College of Medicine, National Taiwan University, Taipei, Taiwan
Taipei, New Taipei, Taiwan (Republic of China)
Cheng-En Tsai, MS
PhD Student
Department of Computer Science and Information Engineering, Graduate Institute of Networking and Multimedia, National Taiwan University, Taiwan
Taipei, Taipei, Taiwan (Republic of China)
Hong-Sian Li, MS
Student
Department of Computer Science and Information Engineering, Graduate Institute of Networking and Multimedia, National Taiwan University, Taiwan
Taipei, Taipei, Taiwan (Republic of China)
Yung-Jen Hsu, PhD
Professor
Department of Computer Science and Information Engineering, Graduate Institute of Networking and Multimedia, National Taiwan University, Taiwan
Taipei, Taipei, Taiwan (Republic of China)
Wenn-Chieh Tsai, PhD
Postdoctoral Research Fellow
Center for Teaching and Learning Development & Digital Learning Center
Taipei, Taipei, Taiwan (Republic of China)
Chien-Te Wu, PhD
Professor
Department of Occupational Therapy, College of Public Health and Health Professions, University of Florida, Gainesville, USA ; Centre for Cognitive Aging and Memory, McKnight Brain Institute, University of Florida, Gainesville, USA
Gainesville, Florida, United States
Hui-Fen Mao, MS
Professor
School of Occupational Therapy, College of Medicine, National Taiwan University, Taipei, Taiwan; Health Science and Wellness Research Center, National Taiwan University, Taipei, Taiwan; Department of Physical Medicine and Rehabilitation, National Taiwan University Hospital, Taipei, Taiwan
Taipei, Taipei, Taiwan (Republic of China)
Large Language Model (LLM)-based recommendation systems are emerging as valuable tools for supporting clinical decision-making, yet issues such as hallucination may reduce acceptance among healthcare professionals. Our team previously developed an LLM-based Cognitive Tabletop Game Analysis and Recommendation System to assist occupational therapists in selecting cognitively appropriate tabletop games for older adults. Earlier validation showed high (≈90%) agreement in cognitive analysis accuracy, but user experience and clinical applicability remained unexplored. This study aimed to assess the practicality and usability of the system in clinical settings. Additionally, expert feedback was collected to identify system limitations and guide future improvements.
Design:
This mixed-methods study recruited 18 occupational therapy experts experienced in cognitive enhancement for older adults. After standardized training, participants completed two testing tasks: (1) evaluation of system-generated game recommendations targeting six key cognitive functions—Visual-Spatial, Attention, Memory, Executive Function, Language, and Processing Speed; and (2) validation of LLM-produced cognitive analysis of a tabletop game. Participants then completed a ResQue-based questionnaire and provided open-ended feedback.
Results:
In Task 1, the system performed best for dual functions such as processing speed + executive function (5.0±0) and complex attention + processing speed (4.92±0.29), while language-related combinations showed lower accuracy (3.56±1.59). In Task 2, the system demonstrated high accuracy in identifying cognitive goals (4.5±0.88), with strong clarity (4.36±0.9) and clinical utility (4.2±0.8). Overall satisfaction was high—especially for novelty (4.89±0.31) and ease of use (4.72±0.45)—but lower for information completeness (3.83±0.96) and trust (3.89±0.66). Experts appreciated the system’s efficiency and radar-chart interface but noted issues such as mismatched titles, incomplete rule descriptions, and inconsistent cognitive weighting. Suggested improvements included expanding the knowledge base, refining analysis logic, and enhancing interface design.
Conclusion:
The system shows strong potential to automate cognitive analysis and streamline game selection for personalized cognitive interventions in occupational therapy.
