Marco Baroni

Also published as: M. Baroni


2024

pdf
MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained Language Models
Nathanael Carraz Rakotonirina | Marco Baroni
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)

Transformer-based language models (LMs) track contextual information through large, hard-coded input windows. We introduce MemoryPrompt, a leaner approach in which the LM is complemented by a small auxiliary recurrent network that passes information to the LM by prefixing its regular input with a sequence of vectors, akin to soft prompts, without requiring LM finetuning. Tested on a task designed to probe a LM’s ability to keep track of multiple fact updates, a MemoryPrompt-augmented LM outperforms much larger LMs that have access to the full input history. We also test MemoryPrompt on a long-distance dialogue dataset, where its performance is comparable to that of a model conditioned on the entire conversation history. In both experiments we also observe that, unlike full-finetuning approaches, MemoryPrompt does not suffer from catastrophic forgetting when adapted to new tasks, thus not disrupting the generalist capabilities of the underlying LM.

2023

pdf
Unnatural language processing: How do language models handle machine-generated prompts?
Corentin Kervadec | Francesca Franzon | Marco Baroni
Findings of the Association for Computational Linguistics: EMNLP 2023

Language model prompt optimization research has shown that semantically and grammatically well-formed manually crafted prompts are routinely outperformed by automatically generated token sequences with no apparent meaning or syntactic structure, including sequences of vectors from a model’s embedding space. We use machine-generated prompts to probe how models respond to input that is not composed of natural language expressions. We study the behavior of models of different sizes in multiple semantic tasks in response to both continuous and discrete machine-generated prompts, and compare it to the behavior in response to human-generated natural-language prompts. Even when producing a similar output, machine-generated and human prompts trigger different response patterns through the network processing pathways, including different perplexities, different attention and output entropy distributions, and different unit activation profiles. We provide preliminary insight into the nature of the units activated by different prompt types, suggesting that only natural language prompts recruit a genuinely linguistic circuit.

pdf
Bridging Information-Theoretic and Geometric Compression in Language Models
Emily Cheng | Corentin Kervadec | Marco Baroni
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

For a language model (LM) to faithfully model human language, it must compress vast, potentially infinite information into relatively few dimensions. We propose analyzing compression in (pre-trained) LMs from two points of view: geometric and information-theoretic. We demonstrate that the two views are highly correlated, such that the intrinsic geometric dimension of linguistic data predicts their coding length under the LM. We then show that, in turn, high compression of a linguistic dataset predicts rapid adaptation to that dataset, confirming that being able to compress linguistic information is an important part of successful LM performance. As a practical byproduct of our analysis, we evaluate a battery of intrinsic dimension estimators for the first time on linguistic data, showing that only some encapsulate the relationship between information-theoretic compression, geometric compression, and ease-of-adaptation.

2022

pdf
Communication breakdown: On the low mutual intelligibility between human and neural captioning
Roberto Dessì | Eleonora Gualdoni | Francesca Franzon | Gemma Boleda | Marco Baroni
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing

We compare the 0-shot performance of a neural caption-based image retriever when given as input either human-produced captions or captions generated by a neural captioner. We conduct this comparison on the recently introduced ImageCoDe data-set (Krojer et al. 2022), which contains hard distractors nearly identical to the images to be retrieved. We find that the neural retriever has much higher performance when fed neural rather than human captions, despite the fact that the former, unlike the latter, were generated without awareness of the distractors that make the task hard. Even more remarkably, when the same neural captions are given to human subjects, their retrieval performance is almost at chance level. Our results thus add to the growing body of evidence that, even when the “language” of neural models resembles English, this superficial resemblance might be deeply misleading.

pdf
Emergent Language-Based Coordination In Deep Multi-Agent Systems
Marco Baroni | Roberto Dessi | Angeliki Lazaridou
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts

Large pre-trained deep networks are the standard building blocks of modern AI applications. This raises fundamental questions about how to control their behaviour and how to make them efficiently interact with each other. Deep net emergent communication tackles these challenges by studying how to induce communication protocols between neural network agents, and how to include humans in the communication loop. Traditionally, this research had focussed on relatively small-scale experiments where two networks had to develop a discrete code from scratch for referential communication. However, with the rise of large pre-trained language models that can work well on many tasks, the emphasis is now shifting on how to let these models interact through a language-like channel to engage in more complex behaviors. By reviewing several representative papers, we will provide an introduction to deep net emergent communication, we will cover various central topics from the present and recent past, as well as discussing current shortcomings and suggest future directions. The presentation is complemented by a hands-on section where participants will implement and analyze two emergent communications setups from the literature. The tutorial should be of interest to researchers wanting to develop more flexible AI systems, but also to cognitive scientists and linguists interested in the evolution of communication systems.

2021

pdf
Controlled tasks for model analysis: Retrieving discrete information from sequences
Ionut-Teodor Sorodoc | Gemma Boleda | Marco Baroni
Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

In recent years, the NLP community has shown increasing interest in analysing how deep learning models work. Given that large models trained on complex tasks are difficult to inspect, some of this work has focused on controlled tasks that emulate specific aspects of language. We propose a new set of such controlled tasks to explore a crucial aspect of natural language processing that has not received enough attention: the need to retrieve discrete information from sequences. We also study model behavior on the tasks with simple instantiations of Transformers and LSTMs. Our results highlight the beneficial role of decoder attention and its sometimes unexpected interaction with other components. Moreover, we show that, for most of the tasks, these simple models still show significant difficulties. We hope that the community will take up the analysis possibilities that our tasks afford, and that a clearer understanding of model behavior on the tasks will lead to better and more transparent models.

2020

pdf
Compositionality and Generalization In Emergent Languages
Rahma Chaabouni | Eugene Kharitonov | Diane Bouchacourt | Emmanuel Dupoux | Marco Baroni
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

Natural language allows us to refer to novel composite concepts by combining expressions denoting their parts according to systematic rules, a property known as compositionality. In this paper, we study whether the language emerging in deep multi-agent simulations possesses a similar ability to refer to novel primitive combinations, and whether it accomplishes this feat by strategies akin to human-language compositionality. Equipped with new ways to measure compositionality in emergent languages inspired by disentanglement in representation learning, we establish three main results: First, given sufficiently large input spaces, the emergent language will naturally develop the ability to refer to novel composite concepts. Second, there is no correlation between the degree of compositionality of an emergent language and its ability to generalize. Third, while compositionality is not necessary for generalization, it provides an advantage in terms of language transmission: The more compositional a language is, the more easily it will be picked up by new learners, even when the latter differ in architecture from the original agents. We conclude that compositionality does not arise from simple generalization pressure, but if an emergent language does chance upon it, it will be more likely to survive and thrive.

pdf bib
Emergent Language Generalization and Acquisition Speed are not tied to Compositionality
Eugene Kharitonov | Marco Baroni
Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

Studies of discrete languages emerging when neural agents communicate to solve a joint task often look for evidence of compositional structure. This stems for the expectation that such a structure would allow languages to be acquired faster by the agents and enable them to generalize better. We argue that these beneficial properties are only loosely connected to compositionality. In two experiments, we demonstrate that, depending on the task, non-compositional languages might show equal, or better, generalization performance and acquisition speed than compositional ones. Further research in the area should be clearer about what benefits are expected from compositionality, and how the latter would lead to them.

2019

pdf
Tabula Nearly Rasa: Probing the Linguistic Knowledge of Character-level Neural Language Models Trained on Unsegmented Text
Michael Hahn | Marco Baroni
Transactions of the Association for Computational Linguistics, Volume 7

Recurrent neural networks (RNNs) have reached striking performance in many natural language processing tasks. This has renewed interest in whether these generic sequence processing devices are inducing genuine linguistic knowledge. Nearly all current analytical studies, however, initialize the RNNs with a vocabulary of known words, and feed them tokenized input during training. We present a multi-lingual study of the linguistic knowledge encoded in RNNs trained as character-level language models, on input data with word boundaries removed. These networks face a tougher and more cognitively realistic task, having to discover any useful linguistic unit from scratch based on input statistics. The results show that our “near tabula rasa” RNNs are mostly able to solve morphological, syntactic and semantic tasks that intuitively presuppose word-level knowledge, and indeed they learned, to some extent, to track word boundaries. Our study opens the door to speculations about the necessity of an explicit, rigid word lexicon in language learning and usage.

pdf
Miss Tools and Mr Fruit: Emergent Communication in Agents Learning about Object Affordances
Diane Bouchacourt | Marco Baroni
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Recent research studies communication emergence in communities of deep network agents assigned a joint task, hoping to gain insights on human language evolution. We propose here a new task capturing crucial aspects of the human environment, such as natural object affordances, and of human conversation, such as full symmetry among the participants. By conducting a thorough pragmatic and semantic analysis of the emergent protocol, we show that the agents solve the shared task through genuine bilateral, referential communication. However, the agents develop multiple idiolects, which makes us conclude that full symmetry is not a sufficient condition for a common language to emerge.

pdf
CNNs found to jump around more skillfully than RNNs: Compositional Generalization in Seq2seq Convolutional Networks
Roberto Dessì | Marco Baroni
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Lake and Baroni (2018) introduced the SCAN dataset probing the ability of seq2seq models to capture compositional generalizations, such as inferring the meaning of “jump around” 0-shot from the component words. Recurrent networks (RNNs) were found to completely fail the most challenging generalization cases. We test here a convolutional network (CNN) on these tasks, reporting hugely improved performance with respect to RNNs. Despite the big improvement, the CNN has however not induced systematic rules, suggesting that the difference between compositional and non-compositional behaviour is not clear-cut.

pdf
On the Distribution of Deep Clausal Embeddings: A Large Cross-linguistic Study
Damian Blasi | Ryan Cotterell | Lawrence Wolf-Sonkin | Sabine Stoll | Balthasar Bickel | Marco Baroni
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Embedding a clause inside another (“the girl [who likes cars [that run fast]] has arrived”) is a fundamental resource that has been argued to be a key driver of linguistic expressiveness. As such, it plays a central role in fundamental debates on what makes human language unique, and how they might have evolved. Empirical evidence on the prevalence and the limits of embeddings has however been based on either laboratory setups or corpus data of relatively limited size. We introduce here a collection of large, dependency-parsed written corpora in 17 languages, that allow us, for the first time, to capture clausal embedding through dependency graphs and assess their distribution. Our results indicate that there is no evidence for hard constraints on embedding depth: the tail of depth distributions is heavy. Moreover, although deeply embedded clauses tend to be shorter, suggesting processing load issues, complex sentences with many embeddings do not display a bias towards less deep embeddings. Taken together, the results suggest that deep embeddings are not disfavoured in written language. More generally, our study illustrates how resources and methods from latest-generation big-data NLP can provide new perspectives on fundamental questions in theoretical linguistics.

pdf
Word-order Biases in Deep-agent Emergent Communication
Rahma Chaabouni | Eugene Kharitonov | Alessandro Lazaric | Emmanuel Dupoux | Marco Baroni
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

Sequence-processing neural networks led to remarkable progress on many NLP tasks. As a consequence, there has been increasing interest in understanding to what extent they process language as humans do. We aim here to uncover which biases such models display with respect to “natural” word-order constraints. We train models to communicate about paths in a simple gridworld, using miniature languages that reflect or violate various natural language trends, such as the tendency to avoid redundancy or to minimize long-distance dependencies. We study how the controlled characteristics of our miniature languages affect individual learning and their stability across multiple network generations. The results draw a mixed picture. On the one hand, neural networks show a strong tendency to avoid long-distance dependencies. On the other hand, there is no clear preference for the efficient, non-redundant encoding of information that is widely attested in natural language. We thus suggest inoculating a notion of “effort” into neural networks, as a possible way to make their linguistic behavior more human-like.

pdf
EGG: a toolkit for research on Emergence of lanGuage in Games
Eugene Kharitonov | Rahma Chaabouni | Diane Bouchacourt | Marco Baroni
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations

There is renewed interest in simulating language emergence among deep neural agents that communicate to jointly solve a task, spurred by the practical aim to develop language-enabled interactive AIs, as well as by theoretical questions about the evolution of human language. However, optimizing deep architectures connected by a discrete communication channel (such as that in which language emerges) is technically challenging. We introduce EGG, a toolkit that greatly simplifies the implementation of emergent-language communication games. EGG’s modular design provides a set of building blocks that the user can combine to create new games, easily navigating the optimization and architecture space. We hope that the tool will lower the technical barrier, and encourage researchers from various backgrounds to do original work in this exciting area.

pdf bib
The emergence of number and syntax units in LSTM language models
Yair Lakretz | German Kruszewski | Theo Desbordes | Dieuwke Hupkes | Stanislas Dehaene | Marco Baroni
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)

Recent work has shown that LSTMs trained on a generic language modeling objective capture syntax-sensitive generalizations such as long-distance number agreement. We have however no mechanistic understanding of how they accomplish this remarkable feat. Some have conjectured it depends on heuristics that do not truly take hierarchical structure into account. We present here a detailed study of the inner mechanics of number tracking in LSTMs at the single neuron level. We discover that long-distance number information is largely managed by two “number units”. Importantly, the behaviour of these units is partially controlled by other units independently shown to track syntactic structure. We conclude that LSTMs are, to some extent, implementing genuinely syntactic processing mechanisms, paving the way to a more general understanding of grammatical encoding in LSTMs.

2018

pdf
How agents see things: On visual representations in an emergent language game
Diane Bouchacourt | Marco Baroni
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

There is growing interest in the language developed by agents interacting in emergent-communication settings. Earlier studies have focused on the agents’ symbol usage, rather than on their representation of visual input. In this paper, we consider the referential games of Lazaridou et al. (2017), and investigate the representations the agents develop during their evolving interaction. We find that the agents establish successful communication by inducing visual representations that almost perfectly align with each other, but, surprisingly, do not capture the conceptual properties of the objects depicted in the input images. We conclude that, if we care about developing language-like communication systems, we must pay more attention to the visual semantics agents associate to the symbols they use.

pdf
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties
Alexis Conneau | German Kruszewski | Guillaume Lample | Loïc Barrault | Marco Baroni
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing. “Downstream” tasks, often based on sentence classification, are commonly used to evaluate the quality of sentence representations. The complexity of the tasks makes it however difficult to infer what kind of information is present in the representations. We introduce here 10 probing tasks designed to capture simple linguistic features of sentences, and we use them to study embeddings generated by three different encoders trained in eight distinct ways, uncovering intriguing properties of both encoders and training methods.

pdf
Colorless Green Recurrent Networks Dream Hierarchically
Kristina Gulordava | Piotr Bojanowski | Edouard Grave | Tal Linzen | Marco Baroni
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)

Recurrent neural networks (RNNs) achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language. We investigate to what extent RNNs learn to track abstract hierarchical syntactic structure. We test whether RNNs trained with a generic language modeling objective in four languages (Italian, English, Hebrew, Russian) can predict long-distance number agreement in various constructions. We include in our evaluation nonsensical sentences where RNNs cannot rely on semantic or lexical cues (“The colorless green ideas I ate with the chair sleep furiously”), and, for Italian, we compare model performance to human intuitions. Our language-model-trained RNNs make reliable predictions about long-distance agreement, and do not lag much behind human performance. We thus bring support to the hypothesis that RNNs are not just shallow-pattern extractors, but they also acquire deeper grammatical competence.

pdf
Jump to better conclusions: SCAN both left and right
Jasmijn Bastings | Marco Baroni | Jason Weston | Kyunghyun Cho | Douwe Kiela
Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP

Lake and Baroni (2018) recently introduced the SCAN data set, which consists of simple commands paired with action sequences and is intended to test the strong generalization abilities of recurrent sequence-to-sequence models. Their initial experiments suggested that such models may fail because they lack the ability to extract systematic rules. Here, we take a closer look at SCAN and show that it does not always capture the kind of generalization that it was designed for. To mitigate this we propose a complementary dataset, which requires mapping actions back to the original commands, called NACS. We show that models that do well on SCAN do not necessarily do well on NACS, and that NACS exhibits properties more closely aligned with realistic use-cases for sequence-to-sequence models.

pdf
Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks
João Loula | Marco Baroni | Brenden Lake
Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP

Systematic compositionality is the ability to recombine meaningful units with regular and predictable outcomes, and it’s seen as key to the human capacity for generalization in language. Recent work (Lake and Baroni, 2018) has studied systematic compositionality in modern seq2seq models using generalization to novel navigation instructions in a grounded environment as a probing tool. Lake and Baroni’s main experiment required the models to quickly bootstrap the meaning of new words. We extend this framework here to settings where the model needs only to recombine well-trained functional words (such as “around” and “right”) in novel contexts. Our findings confirm and strengthen the earlier ones: seq2seq models can be impressively good at generalizing to novel combinations of previously-seen input, but only when they receive extensive training on the specific pattern to be generalized (e.g., generalizing from many examples of “X around right” to “jump around right”), while failing when generalization requires novel application of compositional rules (e.g., inferring the meaning of “around right” from those of “right” and “around”).

2017

pdf
High-risk learning: acquiring new word vectors from tiny data
Aurélie Herbelot | Marco Baroni
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing

Distributional semantics models are known to struggle with small data. It is generally accepted that in order to learn ‘a good vector’ for a word, a model must have sufficient examples of its usage. This contradicts the fact that humans can guess the meaning of a word from a few occurrences only. In this paper, we show that a neural language model such as Word2Vec only necessitates minor modifications to its standard architecture to learn new terms from tiny data, using background knowledge from a previously learnt semantic space. We test our model on word definitions and on a nonce task involving 2-6 sentences’ worth of context, showing a large increase in performance over state-of-the-art models on the definitional task.

pdf
Living a discrete life in a continuous world: Reference in cross-modal entity tracking
Gemma Boleda | Sebastian Padó | Nghia The Pham | Marco Baroni
Proceedings of the 12th International Conference on Computational Semantics (IWCS) — Short papers

2016

pdf
Squibs: When the Whole Is Less Than the Sum of Its Parts: How Composition Affects PMI Values in Distributional Semantic Vectors
Denis Paperno | Marco Baroni
Computational Linguistics, Volume 42, Issue 2 - June 2016

pdf
There Is No Logical Negation Here, But There Are Alternatives: Modeling Conversational Negation with Distributional Semantics
Germán Kruszewski | Denis Paperno | Raffaella Bernardi | Marco Baroni
Computational Linguistics, Volume 42, Issue 4 - December 2016

pdf
Multimodal Semantic Learning from Child-Directed Input
Angeliki Lazaridou | Grzegorz Chrupała | Raquel Fernández | Marco Baroni
Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

pdf
The LAMBADA dataset: Word prediction requiring a broad discourse context
Denis Paperno | Germán Kruszewski | Angeliki Lazaridou | Ngoc Quan Pham | Raffaella Bernardi | Sandro Pezzelle | Marco Baroni | Gemma Boleda | Raquel Fernández
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf
The red one!: On learning to refer to things based on discriminative properties
Angeliki Lazaridou | Nghia The Pham | Marco Baroni
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

2015

pdf
From Visual Attributes to Adjectives through Decompositional Distributional Semantics
Angeliki Lazaridou | Georgiana Dinu | Adam Liska | Marco Baroni
Transactions of the Association for Computational Linguistics, Volume 3

As automated image analysis progresses, there is increasing interest in richer linguistic annotation of pictures, with attributes of objects (e.g., furry, brown…) attracting most attention. By building on the recent “zero-shot learning” approach, and paying attention to the linguistic nature of attributes as noun modifiers, and specifically adjectives, we show that it is possible to tag images with attribute-denoting adjectives even when no training data containing the relevant annotation are available. Our approach relies on two key observations. First, objects can be seen as bundles of attributes, typically expressed as adjectival modifiers (a dog is something furry, brown, etc.), and thus a function trained to map visual representations of objects to nominal labels can implicitly learn to map attributes to adjectives. Second, objects and attributes come together in pictures (the same thing is a dog and it is brown). We can thus achieve better attribute (and object) label retrieval by treating images as “visual phrases”, and decomposing their linguistic representation into an attribute-denoting adjective and an object-denoting noun. Our approach performs comparably to a method exploiting manual attribute annotation, it out-performs various competitive alternatives in both attribute and object annotation, and it automatically constructs attribute-centric representations that significantly improve performance in supervised object recognition.

pdf
Deriving Boolean structures from distributional vectors
German Kruszewski | Denis Paperno | Marco Baroni
Transactions of the Association for Computational Linguistics, Volume 3

Corpus-based distributional semantic models capture degrees of semantic relatedness among the words of very large vocabularies, but have problems with logical phenomena such as entailment, that are instead elegantly handled by model-theoretic approaches, which, in turn, do not scale up. We combine the advantages of the two views by inducing a mapping from distributional vectors of words (or sentences) into a Boolean structure of the kind in which natural language terms are assumed to denote. We evaluate this Boolean Distributional Semantic Model (BDSM) on recognizing entailment between words and sentences. The method achieves results comparable to a state-of-the-art SVM, degrades more gracefully when less training data are available and displays interesting qualitative properties.

pdf
Leveraging Preposition Ambiguity to Assess Compositional Distributional Models of Semantics
Samuel Ritter | Cotie Long | Denis Paperno | Marco Baroni | Matthew Botvinick | Adele Goldberg
Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics

pdf
Squibs: When the Whole Is Not Greater Than the Combination of Its Parts: A “Decompositional” Look at Compositional Distributional Semantics
Fabio Massimo Zanzotto | Lorenzo Ferrone | Marco Baroni
Computational Linguistics, Volume 41, Issue 1 - March 2015

pdf
Do Distributed Semantic Models Dream of Electric Sheep? Visualizing Word Representations through Image Synthesis
Angeliki Lazaridou | Dat Tien Nguyen | Marco Baroni
Proceedings of the Fourth Workshop on Vision and Language

pdf bib
Distributional vectors encode referential attributes
Abhijeet Gupta | Gemma Boleda | Marco Baroni | Sebastian Padó
Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing

pdf
Hubness and Pollution: Delving into Cross-Space Mapping for Zero-Shot Learning
Angeliki Lazaridou | Georgiana Dinu | Marco Baroni
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)

pdf
Jointly optimizing word representations for lexical and sentential tasks with the C-PHRASE model
Nghia The Pham | Germán Kruszewski | Angeliki Lazaridou | Marco Baroni
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)

pdf
A Multitask Objective to Inject Lexical Contrast into Distributional Semantics
Nghia The Pham | Angeliki Lazaridou | Marco Baroni
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)

pdf
Combining Language and Vision with a Multimodal Skip-gram Model
Angeliki Lazaridou | Nghia The Pham | Marco Baroni
Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

pdf
So similar and yet incompatible: Toward the automated identification of semantically compatible words
Germán Kruszewski | Marco Baroni
Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

2014

pdf
Frege in Space: A Program for Composition Distributional Semantics
Marco Baroni | Raffaella Bernardi | Roberto Zamparelli
Linguistic Issues in Language Technology, Volume 9, 2014 - Perspectives on Semantic Representations for Textual Inference

The lexicon of any natural language encodes a huge number of distinct word meanings. Just to understand this article, you will need to know what thousands of words mean. The space of possible sentential meanings is infinite: In this article alone, you will encounter many sentences that express ideas you have never heard before, we hope. Statistical semantics has addressed the issue of the vastness of word meaning by proposing methods to harvest meaning automatically from large collections of text (corpora). Formal semantics in the Fregean tradition has developed methods to account for the infinity of sentential meaning based on the crucial insight of compositionality, the idea that meaning of sentences is built incrementally by combining the meanings of their constituents. This article sketches a new approach to semantics that brings together ideas from statistical and formal semantics to account, in parallel, for the richness of lexical meaning and the combinatorial power of sentential semantics. We adopt, in particular, the idea that word meaning can be approximated by the patterns of co-occurrence of words in corpora from statistical semantics, and the idea that compositionality can be captured in terms of a syntax-driven calculus of function application from formal semantics.

pdf
Dead parrots make bad pets: Exploring modifier effects in noun phrases
Germán Kruszewski | Marco Baroni
Proceedings of the Third Joint Conference on Lexical and Computational Semantics (*SEM 2014)

pdf bib
SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment
Marco Marelli | Luisa Bentivogli | Marco Baroni | Raffaella Bernardi | Stefano Menini | Roberto Zamparelli
Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)

pdf
A practical and linguistically-motivated approach to compositional distributional semantics
Denis Paperno | Nghia The Pham | Marco Baroni
Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf
Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors
Marco Baroni | Georgiana Dinu | Germán Kruszewski
Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf
How to make words with vectors: Phrase generation in distributional semantics
Georgiana Dinu | Marco Baroni
Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf
Is this a wampimuk? Cross-modal mapping between distributional semantics and the visual world
Angeliki Lazaridou | Elia Bruni | Marco Baroni
Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf
A SICK cure for the evaluation of compositional distributional semantic models
Marco Marelli | Stefano Menini | Marco Baroni | Luisa Bentivogli | Raffaella Bernardi | Roberto Zamparelli
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)

Shared and internationally recognized benchmarks are fundamental for the development of any computational system. We aim to help the research community working on compositional distributional semantic models (CDSMs) by providing SICK (Sentences Involving Compositional Knowldedge), a large size English benchmark tailored for them. SICK consists of about 10,000 English sentence pairs that include many examples of the lexical, syntactic and semantic phenomena that CDSMs are expected to account for, but do not require dealing with other aspects of existing sentential data sets (idiomatic multiword expressions, named entities, telegraphic language) that are not within the scope of CDSMs. By means of crowdsourcing techniques, each pair was annotated for two crucial semantic tasks: relatedness in meaning (with a 5-point rating scale as gold score) and entailment relation between the two elements (with three possible gold labels: entailment, contradiction, and neutral). The SICK data set was used in SemEval-2014 Task 1, and it freely available for research purposes.

pdf
Improving the Lexical Function Composition Model with Pathwise Optimized Elastic-Net Regression
Jiming Li | Marco Baroni | Georgiana Dinu
Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics

2013

pdf
Intensionality was only alleged: On adjective-noun composition in distributional semantics
Gemma Boleda | Marco Baroni | The Nghia Pham | Louise McNally
Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers

pdf
Multi-Step Regression Learning for Compositional Distributional Semantics
E. Grefenstette | G. Dinu | Y. Zhang | M. Sadrzadeh | M. Baroni
Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013) – Long Papers

pdf
Sentence paraphrase detection: When determiners and word order make the difference
Nghia Pham | Raffaella Bernardi | Yao Zhong Zhang | Marco Baroni
Proceedings of the IWCS 2013 Workshop Towards a Formal Distributional Semantics

pdf
General estimation and evaluation of compositional distributional semantic models
Georgiana Dinu | Nghia The Pham | Marco Baroni
Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality

pdf
Studying the Recursive Behaviour of Adjectival Modification with Compositional Distributional Semantics
Eva Maria Vecchi | Roberto Zamparelli | Marco Baroni
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing

pdf
Fish Transporters and Miracle Homes: How Compositional Distributional Semantics can Help NP Parsing
Angeliki Lazaridou | Eva Maria Vecchi | Marco Baroni
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing

pdf
Of Words, Eyes and Brains: Correlating Image-Based Distributional Semantic Models with Neural Representations of Concepts
Andrew J. Anderson | Elia Bruni | Ulisse Bordignon | Massimo Poesio | Marco Baroni
Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing

pdf bib
Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity
Mona Diab | Tim Baldwin | Marco Baroni
Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity

pdf
Compositional-ly Derived Representations of Morphologically Complex Words in Distributional Semantics
Angeliki Lazaridou | Marco Marelli | Roberto Zamparelli | Marco Baroni
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf
A relatedness benchmark to test the role of determiners in compositional distributional semantics
Raffaella Bernardi | Georgiana Dinu | Marco Marelli | Marco Baroni
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

pdf
DISSECT - DIStributional SEmantics Composition Toolkit
Georgiana Dinu | Nghia The Pham | Marco Baroni
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations

pdf bib
Visual Features for Linguists: Basic image analysis techniques for multimodally-curious NLPers
Elia Bruni | Marco Baroni
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Tutorials)

2012

pdf
Distributional Semantics in Technicolor
Elia Bruni | Gemma Boleda | Marco Baroni | Nam-Khanh Tran
Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

pdf
Unseen features. Collecting semantic data from congenital blind subjects
Alessandro Lenci | Marco Baroni | Giovanna Marotta
Proceedings of the Workshop on Computational Models of Language Acquisition and Loss

pdf
Entailment above the word level in distributional semantics
Marco Baroni | Raffaella Bernardi | Ngoc-Quynh Do | Chung-chieh Shan
Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics

2011

pdf bib
(Linear) Maps of the Impossible: Capturing Semantic Anomalies in Distributional Space
Eva Maria Vecchi | Marco Baroni | Roberto Zamparelli
Proceedings of the Workshop on Distributional Semantics and Compositionality

pdf bib
How we BLESSed distributional semantic evaluation
Marco Baroni | Alessandro Lenci
Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics

pdf
Distributional semantics from text and images
Elia Bruni | Giang Binh Tran | Marco Baroni
Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics

pdf
A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus.
Kristina Gulordava | Marco Baroni
Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics

2010

pdf
BabyExp: Constructing a Huge Multimodal Resource to Acquire Commonsense Knowledge Like Children Do
Massimo Poesio | Marco Baroni | Oswald Lanz | Alessandro Lenci | Alexandros Potamianos | Hinrich Schütze | Sabine Schulte im Walde | Luca Surian
Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)

There is by now widespread agreement that the most realistic way to construct the large-scale commonsense knowledge repositories required by natural language and artificial intelligence applications is by letting machines learn such knowledge from large quantities of data, like humans do. A lot of attention has consequently been paid to the development of increasingly sophisticated machine learning algorithms for knowledge extraction. However, the nature of the input that humans are exposed to while learning commonsense knowledge has received much less attention. The BabyExp project is collecting very dense audio and video recordings of the first 3 years of life of a baby. The corpus constructed in this way will be transcribed with automated techniques and made available to the research community. Moreover, techniques to extract commonsense conceptual knowledge incrementally from these multimodal data are also being explored within the project. The current paper describes BabyExp in general, and presents pilot studies on the feasibility of the automated audio and video transcriptions.

pdf
Predicting Cognitively Salient Modifiers of the Constitutive Parts of Concepts
Gerhard Kremer | Marco Baroni
Proceedings of the 2010 Workshop on Cognitive Modeling and Computational Linguistics

pdf
Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space
Marco Baroni | Roberto Zamparelli
Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing

pdf
Distributional Memory: A General Framework for Corpus-Based Semantics
Marco Baroni | Alessandro Lenci
Computational Linguistics, Volume 36, Issue 4 - December 2010

2009

pdf
EEG responds to conceptual stimuli and corpus semantics
Brian Murphy | Marco Baroni | Massimo Poesio
Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing

pdf bib
One Distributional Memory, Many Semantic Spaces
Marco Baroni | Alessandro Lenci
Proceedings of the Workshop on Geometrical Models of Natural Language Semantics

pdf
BagPack: A General Framework to Represent Semantic Relations
Amaç Herdağdelen | Marco Baroni
Proceedings of the Workshop on Geometrical Models of Natural Language Semantics

pdf
Measuring semantic relatedness with vector space models and random walks
Amaç Herdağdelen | Katrin Erk | Marco Baroni
Proceedings of the 2009 Workshop on Graph-based Methods for Natural Language Processing (TextGraphs-4)

2008

pdf
Cleaneval: a Competition for Cleaning Web Pages
Marco Baroni | Francis Chantree | Adam Kilgarriff | Serge Sharoff
Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)

Cleaneval is a shared task and competitive evaluation on the topic of cleaning arbitrary web pages, with the goal of preparing web data for use as a corpus for linguistic and language technology research and development. The first exercise took place in 2007. We describe how it was set up, results, and lessons learnt

pdf
Cognitively Salient Relations for Multilingual Lexicography
Gerhard Kremer | Andrea Abel | Marco Baroni
Coling 2008: Proceedings of the Workshop on Cognitive Aspects of the Lexicon (COGALEX 2008)

2007

pdf
ISA meets Lara: An incremental word space model for cognitively plausible simulations of semantic learning
Marco Baroni | Alessandro Lenci | Luca Onnis
Proceedings of the Workshop on Cognitive Aspects of Computational Language Acquisition

pdf
Words and Echoes: Assessing and Mitigating the Non-Randomness Problem in Word Frequency Distribution Modeling
Marco Baroni | Stefan Evert
Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics

pdf
zipfR: Word Frequency Modeling in R
Stefan Evert | Marco Baroni
Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions

2006

pdf
A Figure of Merit for the Evaluation of Web-Corpus Randomness
Massimiliano Ciaramita | Marco Baroni
11th Conference of the European Chapter of the Association for Computational Linguistics

pdf bib
Large Linguistically-Processed Web Corpora for Multiple Languages
Marco Baroni | Adam Kilgarriff
Demonstrations

pdf
WebBootCaT. Instant Domain-Specific Corpora to Support Human Translators
Marco Baroni | Adam Kilgarriff | Jan Pomikalek | Pavel Rychly
Proceedings of the 11th Annual Conference of the European Association for Machine Translation

2004

pdf
Introducing the La Repubblica Corpus: A Large, Annotated, TEI(XML)-compliant Corpus of Newspaper Italian
Marco Baroni | Silvia Bernardini | Federica Comastri | Lorenzo Piccioni | Alessandra Volpi | Guy Aston | Marco Mazzoleni
Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)

pdf
Using Cooccurrence Statistics and the Web to Discover Synonyms in a Technical Language
Marco Baroni | Sabrina Bisi
Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)

pdf
BootCaT: Bootstrapping Corpora and Terms from the Web
Marco Baroni | Silvia Bernardini
Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)

2003

pdf bib
Exploiting Long Distance Collocational Relations in Predictive Typing
Johannes Matiasek | Marco Baroni
Proceedings of the 2003 EACL Workshop on Language Modeling for Text Entry Methods

2002

pdf
Wordform- and Class-based Prediction of the Components of German Nominal Compounds in an AAC System
Marco Baroni | Johannes Matiasek | Harald Trost
COLING 2002: The 19th International Conference on Computational Linguistics

pdf
Unsupervised discovery of morphologically related words based on orthographic and semantic similarity
Marco Baroni | Johannes Matiasek | Harald Trost
Proceedings of the ACL-02 Workshop on Morphological and Phonological Learning

Search
Co-authors