Immersive Tailoring of Embodied Agents Using Large Language Models

This IEEE conference talk explores how large language models (LLMs) are being integrated into virtual reality to create dynamic embodied agents. Learn about an innovative architecture that enables users to modify an agent's behavior, personality, and appearance through natural conversation directly within the virtual environment, eliminating the need for external interfaces or pre-development adjustments. The presentation covers performance evaluation results highlighting latency challenges in the speech-to-text-to-speech pipeline and shares insights from a six-week user study. The researchers from Universidad Carlos III de Madrid, Helsinki Institute for Information Technology, University of Helsinki, and the HCI Group at the Department of Computer Science discuss both the promising potential of this technology and future work needed to reduce latency and address the black-box limitations of LLMs for improved system reliability and responsiveness.