2022
pdf
abs
Overlaps and Gender Analysis in the Context of Broadcast Media
Martin Lebourdais
|
Marie Tahon
|
Antoine Laurent
|
Sylvain Meignier
|
Anthony Larcher
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Our main goal is to study the interactions between speakers according to their gender and role in broadcast media. In this paper, we propose an extensive study of gender and overlap annotations in various speech corpora mainly dedicated to diarisation or transcription tasks. We point out the issue of the heterogeneity of the annotation guidelines for both overlapping speech and gender categories. On top of that, we analyse how the speech content (casual speech, meetings, debate, interviews, etc.) impacts the distribution of overlapping speech segments. On a small dataset of 93 recordings from LCP French channel, we intend to characterise the interactions between speakers according to their gender. Finally, we propose a method which aims to highlight active speech areas in terms of interactions between speakers. Such a visualisation tool could improve the efficiency of qualitative studies conducted by researchers in human sciences.
pdf
abs
Are Embedding Spaces Interpretable? Results of an Intrusion Detection Evaluation on a Large French Corpus
Thibault Prouteau
|
Nicolas Dugué
|
Nathalie Camelin
|
Sylvain Meignier
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Word embedding methods allow to represent words as vectors in a space that is structured using word co-occurrences so that words with close meanings are close in this space. These vectors are then provided as input to automatic systems to solve natural language processing problems. Because interpretability is a necessary condition to trusting such systems, interpretability of embedding spaces, the first link in the chain is an important issue. In this paper, we thus evaluate the interpretability of vectors extracted with two approaches: SPINE a k-sparse auto-encoder, and SINr, a graph-based method. This evaluation is based on a Word Intrusion Task with human annotators. It is operated using a large French corpus, and is thus, as far as we know, the first large-scale experiment regarding word embedding interpretability on this language. Furthermore, contrary to the approaches adopted in the literature where the evaluation is done on a small sample of frequent words, we consider a more realistic use-case where most of the vocabulary is kept for the evaluation. This allows to show how difficult this task is, even though SPINE and SINr show some promising results. In particular, SINr results are obtained with a very low amount of computation compared to SPINE, while being similarly interpretable.
pdf
bib
abs
Using ASR-Generated Text for Spoken Language Modeling
Nicolas Hervé
|
Valentin Pelloin
|
Benoit Favre
|
Franck Dary
|
Antoine Laurent
|
Sylvain Meignier
|
Laurent Besacier
Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models
This papers aims at improving spoken language modeling (LM) using very large amount of automatically transcribed speech. We leverage the INA (French National Audiovisual Institute) collection and obtain 19GB of text after applying ASR on 350,000 hours of diverse TV shows. From this, spoken language models are trained either by fine-tuning an existing LM (FlauBERT) or through training a LM from scratch.The new models (FlauBERT-Oral) will be shared with the community and are evaluated not only in terms of word prediction accuracy but also for two downstream tasks : classification of TV shows and syntactic parsing of speech. Experimental results show that FlauBERT-Oral is better than its initial FlauBERT version demonstrating that, despite its inherent noisy nature, ASR-Generated text can be useful to improve spoken language modeling.
2020
pdf
abs
Évaluation de systèmes apprenant tout au long de la vie (Evaluation of lifelong learning systems )
Yevhenii Prokopalo
|
Sylvain Meignier
|
Olivier Galibert
|
Loïc Barrault
|
Anthony Larcher
Actes de la 6e conférence conjointe Journées d'Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Volume 1 : Journées d'Études sur la Parole
Aujourd’hui les systèmes intelligents obtiennent d’excellentes performances dans de nombreux domaines lorsqu’ils sont entraînés par des experts en apprentissage automatique. Lorsque ces systèmes sont mis en production, leurs performances se dégradent au cours du temps du fait de l’évolution de leur environnement réel. Une adaptation de leur modèle par des experts en apprentissage automatique est possible mais très coûteuse alors que les sociétés utilisant ces systèmes disposent d’experts du domaine qui pourraient accompagner ces systèmes dans un apprentissage tout au long de la vie. Dans cet article nous proposons un cadre d’évaluation générique pour des systèmes apprenant tout au long de la vie (SATLV). Nous proposons d’évaluer l’apprentissage assisté par l’humain (actif ou interactif) et l’apprentissage au cours du temps.
pdf
abs
Evaluation of Lifelong Learning Systems
Yevhenii Prokopalo
|
Sylvain Meignier
|
Olivier Galibert
|
Loic Barrault
|
Anthony Larcher
Proceedings of the Twelfth Language Resources and Evaluation Conference
Current intelligent systems need the expensive support of machine learning experts to sustain their performance level when used on a daily basis. To reduce this cost, i.e. remaining free from any machine learning expert, it is reasonable to implement lifelong (or continuous) learning intelligent systems that will continuously adapt their model when facing changing execution conditions. In this work, the systems are allowed to refer to human domain experts who can provide the system with relevant knowledge about the task. Nowadays, the fast growth of lifelong learning systems development rises the question of their evaluation. In this article we propose a generic evaluation methodology for the specific case of lifelong learning systems. Two steps will be considered. First, the evaluation of human-assisted learning (including active and/or interactive learning) outside the context of lifelong learning. Second, the system evaluation across time, with propositions of how a lifelong learning intelligent system should be evaluated when including human assisted learning or not.
2018
pdf
Computer-assisted Speaker Diarization: How to Evaluate Human Corrections
Pierre-Alexandre Broux
|
David Doukhan
|
Simon Petitrenaud
|
Sylvain Meignier
|
Jean Carrive
Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)
2016
pdf
abs
Autoapprentissage pour le regroupement en locuteurs : premières investigations (First investigations on self trained speaker diarization )
Gaël Le Lan
|
Sylvain Meignier
|
Delphine Charlet
|
Anthony Larcher
Actes de la conférence conjointe JEP-TALN-RECITAL 2016. volume 1 : JEP
This paper investigates self trained cross-show speaker diarization applied to collections of French TV archives, based on an i-vector/PLDA framework. The parameters used for i-vectors extraction and PLDA scoring are trained in a unsupervised way, using the data of the collection itself. Performances are compared, using combinations of target data and external data for training. The experimental results on two distinct target corpora show that using data from the corpora themselves to perform unsupervised iterative training and domain adaptation of PLDA parameters can improve an existing system, trained on external annotated data. Such results indicate that performing speaker indexation on small collections of unlabeled audio archives should only rely on the availability of a sufficient external corpus, which can be specifically adapted to every target collection. We show that a minimum collection size is required to exclude the use of such an external bootstrap.
2012
pdf
Nouvelle approche pour le regroupement des locuteurs dans des émissions radiophoniques et télévisuelles (New approach for speaker clustering of broadcast news) [in French]
Mickael Rouvier
|
Sylvain Meignier
Proceedings of the Joint Conference JEP-TALN-RECITAL 2012, volume 1: JEP
pdf
Segmentation et Regroupement en Locuteurs d’une collection de documents audio (Cross-show speaker diarization) [in French]
Grégor Dupuy
|
Mickael Rouvier
|
Sylvain Meignier
|
Yannick Estève
Proceedings of the Joint Conference JEP-TALN-RECITAL 2012, volume 1: JEP
2008
pdf
abs
Combined Systems for Automatic Phonetic Transcription of Proper Nouns
Antoine Laurent
|
Téva Merlin
|
Sylvain Meignier
|
Yannick Estève
|
Paul Deléglise
Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)
Large vocabulary automatic speech recognition (ASR) technologies perform well in known, controlled contexts. However recognition of proper nouns is commonly considered as a difficult task. Accurate phonetic transcription of a proper noun is difficult to obtain, although it can be one of the most important resources for a recognition system. In this article, we propose methods of automatic phonetic transcription applied to proper nouns. The methods are based on combinations of the rule-based phonetic transcription generator LIA_PHON and an acoustic-phonetic decoding system. On the ESTER corpus, we observed that the combined systems obtain better results than our reference system (LIA_PHON). The WER (Word Error Rate) decreased on segments of speech containing proper nouns, without affecting negatively the results on the rest of the corpus. On the same corpus, the Proper Noun Error Rate (PNER, which is a WER computed on proper nouns only), decreased with our new system.