Dialogic reading, which involves interactive exchanges between a parent and a child during picture book reading, has been shown to effectively promote children’s language development. While many support systems for picture book reading have been developed to reduce the burden on parents, existing systems are not yet capable of handling dialogic reading, which requires dynamic parent-child interaction. To develop conversational agents capable of dialogic reading, we constructed a multimodal corpus of parent-child picture-book reading dialogues. The corpus comprises recordings from 36 Japanese parent-child pairs taken during actual picture book reading sessions. In this study, we annotated the corpus with dialogue acts relevant to parent-child communication and categorized the types of quizzes and questions used in the sessions, analyzing the linguistic aspects of parent-child interaction during dialogic reading. After dividing the dialogues into two groups based on the proportion of the child’s utterances, our analyses revealed that dialogue systems should adapt their interaction strategies according to individual child characteristics.

In recent studies, researchers have used large language models (LLMs) to explore semantic representations in the brain; however, they have typically assessed different levels of semantic content, such as speech, objects, and stories, separately. In this study, we recorded brain activity using functional magnetic resonance imaging (fMRI) while participants viewed 8.3 hours of dramas and movies. We annotated these stimuli at multiple semantic levels, which enabled us to extract latent representations of LLMs for this content. Our findings demonstrate that LLMs predict human brain activity more accurately than traditional language models, particularly for complex background stories. Furthermore, we identify distinct brain regions associated with different semantic representations, including multi-modal vision-semantic representations, which highlights the importance of modeling multi-level and multi-modal semantic representations simultaneously. We will make our fMRI dataset publicly available to facilitate further research on aligning LLMs with human brain function.