View publication
| Title | AU-TextAlign: A Pipeline for Word-Level Facial Expression Tagging in LLMs |
| Authors | Jose Guillen, ValentÃn Barriere, Mauricio Araya |
| Publication date | 2025 |
| Abstract | We present AU-TextAlign, a method for embedding facial expressions into text by aligning speech transcriptions with Action Units (AUs) from the Facial Action Coding System. The approach links detected AUs with individual spoken words, producing structured annotations that can be directly processed by Large Language Models (LLMs). We test whether LLMs can infer facial expressions from text alone and find that, while current models fail to recover most expressions without explicit cues, they respond more affectively when AU tags are provided. This negative result highlights a gap in multimodal understanding and motivates the integration of structured non-verbal signals into language interfaces. Our work opens a pathway toward socially aware systems that bridge verbal and non-verbal communication. |
| Conference name | International Conference on Pattern Recognition Systems |
| Publisher | IEEE-xplore |
| Reference URL |
|