\section{Data and utility computation}
\label{suppsec:data}
This section provides additional details on the dataset we use and the utility function we employ to compute the utilities between fruits and tools. Note that we refer to fruits for conciseness, but some vegetables, such as \emph{carrot} and \emph{potato}, are included.

% \subsection{Utility computation}

There are $11$ fruits features: \emph{is crunchy, has skin, has peel, is small, has rough skin, has a pit, has milk, has a shell, has hair, is prickly, has seeds} and $15$ tools features: \emph{has a handle, is sharp, has a blade, has a head, is small, has a sheath, has prongs, is loud, is serrated, has handles, has blades, has a round end, is adorned with feathers, is heavy, has jaws}. Note that, when we sample instances of each category as explained in Section \ref{sec:thegame} of the main paper, %
%if $\mu=0$, we consider the feature should never be present and leave its value at $0$. Second,
 features are sampled independently. We filter out, however, nonsensical combinations. For example, the features \emph{has prongs}, \emph{has a blade} and \emph{has blades} are treated as pairwise mutually exclusive.

In order to compute the utility for a pair ($tool$, $fruit$), we use three mapping matrices. The mapping matrix $M_T \in \mathbb{R}^{15\times6}$ (Table \ref{tab:mt}) maps from the space of tool features to a space of more general functional features: \emph{(cut, spear, lift, break, peel, pit remover)},
and similarly $M_F \in \mathbb{R}^{11\times6}$ (Table \ref{tab:mf}) maps from the space of fruits features to a space of functional features: \emph{(hard, pit, shell, pick, peel, empty inside)}. Finally, the matrix $M \in \mathbb{R}^{6\times6}$ (Table \ref{tab:m}) maps the two abstract functional spaces of features together. For example, if an axe sample is described by the vector $t_a \in \mathbb{R}^{1\times15}$ and a nectarine sample is the vector $f_n \in \mathbb{R}^{1\times11}$, the utility is computed as $U(t_a,f_n)=(f_n M_F) M' (t_a M_T)'$ where $'$ denotes transpose. We always add a value of $0.01$ to avoid zero utilities. Therefore we can compute the utility of any combination of (possibly new) fruits and tools, as long as it can be described in the corresponding functional representational space. Note that in our case we have the same number of abstract functional features for fruits and tools ($6$), but they need not be the same. In other words, $M$ need not be a square matrix.

Given the values in the mapping matrices, $5$ of the tools features have no impact on the utility computation since they do not affect the scores of the functional tool features (they have only zeros in the mapping matrix $M_T$). These are: \emph{has a handle, is sharp, has a sheath, is loud, has handles, is adorned with feathers}. Such features only represent realistic aspects of objects and act as noise.

\begin{table*}
  \setlength{\tabcolsep}{2pt}
  \begin{center}
\pgfplotstabletypeset[col sep=comma,
/color cells/max=1,
/color cells/min=-1,
/color cells/textcolor=black,
columns/Cut/.style={color cells},
columns/Spear/.style={color cells},
columns/Lift/.style={color cells},
columns/Break/.style={color cells},
columns/Peel/.style={color cells},
columns/Pit Remover/.style={color cells},
columns/Tools Feature/.style={string type, column type=c|},
/pgfplots/colormap/violet,
every head row/.style={before row=\toprule,after row=\midrule},
every last row/.style={after row=\bottomrule},
]{
Tools Feature,Cut,Spear,Lift,Break,Peel,Pit Remover
has a handle,0,0,0,0,0,0
is sharp,0,0,0,0,0,0
has a blade,1,0.5,0,0,1,0
has a head,0,0,0,1,0,0
is small,0,0,0,0,0,0.25
has a sheath,0,0,0,0,0,0
has prongs,0.5,1,0.25,0,0.25,0
is loud,0,0,0,0,0,0
is serrated,0.5,0,0,0,0,0
has handles,0,0,0,0,0,0
has blades,1,0.5,0,0,0.5,0
has a round end,0.25,0,1,0,0,1
is adorned with feathers,0,0,0,0,0,0
is heavy,0,0,0,0.5,0,0
has jaws,0,0,1,0,0,0.5
}
\caption{$M_{T}$. Rows are dataset tool features, columns are functional tool features.}
\label{tab:mt}
\end{center}
\end{table*}

\begin{table*}
    \setlength{\tabcolsep}{2pt}
  \begin{center}
\pgfplotstabletypeset[col sep=comma,
/color cells/max=1,
/color cells/min=-1,
/color cells/textcolor=black,
columns/Hard/.style={color cells},
columns/Shell/.style={color cells},
columns/Pick/.style={color cells},
columns/Pit/.style={color cells},
columns/Peel/.style={color cells},
columns/Empty inside/.style={color cells},
columns/Fruits Feature/.style={string type, column type=c|},
every head row/.style={before row=\toprule,after row=\midrule},
every last row/.style={after row=\bottomrule},
/pgfplots/colormap/violet,
]{
Fruits Feature,Hard,Pit,Shell,Pick,Peel,Empty inside
is crunchy,1,0,0,0,0,0
has skin,0,0,0,0,1,0
has peel,0,0,0,0,1,0
is small,0,0,0,1,0,0
has rough skin,0,0,0.5,0,0,0
has a pit,0,1,0,0,0,0
has milk,0,0,0,0,0,1
has a shell,0,0,1,0,0,0
has hair,0,0,0,0,0.5,0
is prickly,0,0,0,0,0.5,0
has seeds,0,0,0,0,0,1
}
\caption{$M_{F}$. Rows are dataset fruit features, columns are functional fruit features.}
\label{tab:mf}
\end{center}
\end{table*}

\begin{table*}
    \setlength{\tabcolsep}{2pt}
    \begin{center}
\pgfplotstabletypeset[col sep=comma,
/color cells/max=1,
/color cells/min=-1,
/color cells/textcolor=black,
columns/Hard/.style={color cells},
columns/Shell/.style={color cells},
columns/Pick/.style={color cells},
columns/Pit/.style={color cells},
columns/Peel/.style={color cells},
columns/Empty inside/.style={color cells},
columns/Feature/.style={string type, column type=c|, column name={}},
every head row/.style={before row=\toprule,after row=\midrule},
every last row/.style={after row=\bottomrule},
/pgfplots/colormap/violet,
]{
Feature,Hard,Pit,Shell,Pick,Peel,Empty inside
Cut,1,0,0.5,0,0.5,0
Spear,0,0,0,1,0,0
Lift,0,0,0,0.5,0,1
Break,0.5,0,1,0,0,0
Peel,0,0,0,0,1,0
Pit Remover,0,1,0,0,0,0
}
\caption{$M$. Rows are functional tool features, columns are functional fruit features.}
\label{tab:m}
\end{center}
\end{table*}

\section{Implementation details}
\label{suppsec:imp_details}

\subsection{Training and architecture hyperparameters}

We update the parameters with RMSProp \citep{Tieleman:Hinton:2012} with a learning rate of $0.001$ and the rest of the parameters left to their Pytorch default value. We use a scalar reward baseline $b$ to reduce variance, learned with Mean Square Error such that $1+b$ matches the average reward. We clip all gradients at 0.1. For the Message encoder and decoder modules, we embed input and output symbols with dimensionality $50$ and then use a RNN with $100$ hidden dimensions. The Fruit embedder linear transformation is of output size $100$, the Tool embedder is of size $50$. The Body module is of size $100$. We train the agents with batches of $128$ games for a total of $1$ million batches. We validate on $12$ batches of $100$ games, for a total of $1200$ validation games, and similarly for testing. % We use $20$ different training initialization seeds. We consider a training seed successful when final validation performance is above $85\%$ for all settings except the no-memory, no-communication setting (lowest performance) where we take $84\%$.

\subsection{Test procedure details}

The computation of the $\text{ME}$ values involves random sampling in steps \ref{step:sampB} and \ref{step:sampCount} of Algorithm \ref{algo:algoME} so we test using $20$ testing seeds. For each successful training seeds, we compute the average ME over the test seeds, and report the mean and standard error of the mean (SEM) of the average ME. Given two trained agents A and B, there are $C=4$ possibles configurations at test time:
\begin{enumerate}
\item A is Fruit Player/position 1 and B is Tool Player/position 2
\item A is Fruit Player/position 2 and B is Tool Player/position 1
\item A is Tool Player/position 1 and B is Fruit Player/position 2
\item A is Tool Player/position 2 and B is Fruit Player/position 1
\end{enumerate}

We balance the number of test games in each configuration: we use $3$ batches of $100$ test games in each configuration, resulting in $12$ batches for a total of $1200$ test games. The ME value in each configuration $c$ is the average ME over the number of batches in this configuration ($3$ in our case). We then average the ME in each configuration over the four possible configurations to obtain $ME^{1\rightarrow 2}$, $ME^{F\rightarrow T}$ and their reverse.

\section{Message effect metric}
\label{suppsec:me}
\subsection{Causal graph and assumptions}

\begin{figure}[h!]
  \begin{center}
\begin{tikzpicture}[every node/.style={minimum size=15cm,inner sep=0,outer sep=0}]
  % Nodes
  \node[obs]    (sb)  {$s^B_{t-1}$}; %
  \node[latent,below=1cm of sb]    (mb1)  {$m^B_{t-1}$}; %
  \node[latent,right=0.5cm of mb1]    (cb1)  {$c^B_{t-1}$}; %

  \node[latent,below=of mb1]    (sa)  {$s^A_t$}; %
  \node[obs,fill={rgb:orange,1;yellow,2;pink,5},below=1cm of sa]    (ma)  {$m^A_t$}; %
  \node[obs,right=0.5cm of ma]    (ca)  {$c^A_t$}; %
  \node[latent,left=2cm of sa]    (ia)  {$i^A$}; %
  \node[latent,above left=1.6cm of mb1]    (sa1)  {$s^A_{t-2}$}; %

  \node[obs,right=2cm of sb]    (ib)  {$i^B$}; %
  \node[latent,below=of ma] (sb2)      {$s^B_{t+1}$} ; %
  \node[latent, below=1cm of sb2]      (mb)      {$m^B_{t+1}$} ; %
  \node[latent, right=0.5cm of mb]       (cb)      {$c^B_{t+1}$} ;

  \node[latent,below=of mb] (sa2)      {$s^A_{t+2}$} ; %

  \edge {sb} {mb1} ; %
  \edge {sb} {cb1} ; %
  \edge {mb1} {sa} ; %
  \edge {cb1} {sa} ; %
    \edge {ia} {sa} ; %
    \edge {sa1} {sa} ; %
  \edge {sa} {ma} ; %
  \edge {sa} {ca} ; %
  \edge {sa} {ma} ; %
  \edge {sa} {ca} ; %
  \edge {ca} {sb2} ; %
  \edge {ma} {sb2} ;
    \edge {ib} {sb.east} ; %
  \draw[->,>={triangle 45}] (ib) to[bend left=45] (sb2.east);
  \draw[->,>={triangle 45}] (ia) to[bend right=20] (sa2);
  \draw[->,>={triangle 45}] (sb) to[bend left=70] (sb2);
  \edge {sb2} {mb} ; %
  \edge {sb2} {cb} ;
  \edge {mb} {sa2} ; %
  \edge {cb} {sa2} ; %
  \edge {ia} {sa1} ; %
  % \edge {ia} {sa2.north west} ; %
  \draw[->,>={triangle 45}] (sa) to[bend right=25] (sa2);
\end{tikzpicture}
\caption{Causal graph considered when we compute $\text{ME}^{A\rightarrow B}_t$. The orange node $m^A_t$ is the variable we intervene on. Shaded nodes represent the variables we condition on.}
\label{fig:causalgraph}
\end{center}
\end{figure}

Figure \ref{fig:causalgraph} shows the causal graph we consider when computing
$\text{ME}^{A\rightarrow B}_t$. We write all variables that should be considered at this turn $t$. Conditioning on $c^A_t,s^B_{t-1},i^B$ blocks any backdoor paths when we compute the causal influence of $m^A_t$ on $c^B_{t+1},m^B_{t+1}$ \citep{Pearl:etal:2016}. Moreover, the path between $m^A_t$ and $c^B_{t+1},m^B_{t+1}$ through $i^A$ is blocked at the collider node $s^A_{t+2}$. Therefore, we ensure there is no confounder when we compute the influence of $m^A_t$ on $c^B_{t+1},m^B_{t+1}$. As in \citet{Jaques:etal:2018}, we have knowledge of the inputs to the model and the distributions of the variables we consider. Therefore we do not need to perform abduction to update probabilities of unobserved exogenous variables that may alter the causal relations in our model \citep{Pearl:etal:2016}.

We denote agent B's choice and message pair at turn $t+1$ as $z^B_{t+1}=(c^B_{t+1},m^B_{t+1})$. We explain in Section \ref{sec:setup} of the main paper that we compare (i) the conditional distribution $p(z^B_{t+1}|m^A_t)$ and (ii) the marginal distribution $p(z^B_{t+1})$ which does not take $m^A_t$ into account. We intervene on $m^A_t$, and draw counterfactual messages not from agent A but from another distribution over $m^A_t$, the \emph{intervention distribution}. We define $\tilde{p}(z^B_{t+1})$, the marginal computed with counterfactual messages $m'^A_t$, as:
\begin{align}
\tilde{p}(z^B_{t+1})=\sum_{m^A_t}p(z^B_{t+1}|m'^A_t)\tilde{p}(m'^A_t).\label{eqsupp:marg1}
\end{align}
where $\tilde{p}(m'^A_t)$ is the \emph{intervention distribution}. In our experiments we take a uniform intervention distribution. Importantly, $\tilde{p}(m'^A_t)$ is different from the \emph{observational distribution} distribution $p(m^A_t|s^A_t)$ that agent A actually defines over the messages. Contrarily to \citet{Bottou:etal:2013} by feeding the counterfactuals messages to agent B, we access $p(z^B_{t+1}|m'^A_t)$ and need not estimate it from empirical data.

\subsection{Difference with Mutual Information}

\citet{Jaques:etal:2018} train agents to have impact on other agents
by maximizing the causal influence of their actions. They show their
definition of influence to relate to the Mutual Information (MI)
between influencing and influenced agents. \newcite{Lowe:etal:2019} also
define a causal influence metric based on MI.  MI computation
requires counterfactuals drawn from the influencing agent
distribution, and not from an intervention one. In our setting, this
means drawing counterfactuals from agent A's distribution
$p(m^A_t|s^A_t)$, and not $\tilde{p}(m'^A_t)$, in step
\ref{step:sampCount} of Algorithm \ref{algo:algoME} (main paper).

There is an issue with employing MI and drawing counterfactuals from the influencing agent's distribution, e.g., $p(m^A_t|s^A_t)$, that is particularly pressing if message distributions are very skewed (as it is the case with our agents below). Consider a simple setting where agents A and B utter a message that has two possible values $u$ and $v$. A is the influencing agent and B is the influencee. Suppose that the dynamics are such that A almost always says $u$, and B always replies with the message it received. Most of the exchanges we would sample would be: ``A says $u$, B replies $u$". The MI estimate would then be very low, and one might erroneously conclude that B is not influenced by A. Indeed when distributions are very peaky as in this example, it would require very large samples to witness rare events, such as ```A says $v$, B replies $v$". \newcite{Lowe:etal:2019} ensure that all possible messages from the influencing agent are considered. This is computationally expensive when the message space is large. Moreover, the resulting MI can still be small as each message's contribution is weighted by its probability under the influencing agent. By using a uniform intervention distribution, we ensure that B in the current example would receive $v$ and therefore reply $v$ in half the exchanges, easily detecting the influence of A on B.


\section{Additional results: performance and pragmatics}
\label{suppsec:addres}

Table \ref{tab:perftable2} reports a more detailed view of the ME in Table \ref{tab:perftable} of the main paper. $1T/2F$ denotes games where the Tool Player is in first position and the Fruit Player is in second position, and $1F/2T$ denotes games where the Fruit Player in first position and Tool Player in second. This table shows that in the no-memory, with-communication setting (top right quadrant), the difference between the influence of the Fruit Player on the Tool player and its reverse is greater when the Fruit Player is in position $2$. On in-domain data, the Fruit Player has a stronger influence on the Tool Player only when in position $2$. This explains the effect $\text{ME}^{2\rightarrow 1} > \text{ME}^{1\rightarrow 2}$ we mention in Section \ref{sec:prag} in the main paper. We also observe that in the no-memory, no-communication setting (top left quadrant), when the Tool Player is in position 1, $ME^{1\rightarrow 2}~1T/2F \approx 0$. This relates to the artifact we describe in the main paper: in that case the Tool Player stops the game at $t=0$, leaving no room for the Fruit Player to be influenced.

\begin{table*}[h!]
  \begin{center}
\begin{tabular}{|c|c|cc|cc|}
  \hline
  ~& ~&\multicolumn{2}{c|}{\bfseries No communication} &\multicolumn{2}{c|}{\bfseries With communication}\\
\hline
~ & Metric & In & Transfer & In & Transfer \\
\hline
\multirow{ 7}{*}{\rotatebox[origin=c]{90}{\bfseries No memory}}&
Av. perf. (\%)& $84.83\pm0.09$& $84.0\pm0.11$&$96.9\pm0.32$& $94.5\pm0.37$\\
~&$ME^{F\rightarrow T}$& $0.133^*\pm0.01$& $0.14^*\pm0.01$&$5.0^*\pm0.39$& $5.0^*\pm0.36$\\
~&$ME^{T\rightarrow F}$& $0.05\pm0.02$& $0.030\pm0.01$&$3.9\pm0.38$& $3.3\pm0.30$\\
~&$ME^{1\rightarrow 2}$& $0.066\pm0.00$& $0.067\pm0.01$&$3.9\pm0.29$& $3.7\pm0.26$\\
~&$ME^{2\rightarrow 1}$& $0.12^*\pm0.02$& $0.10^*\pm0.01$&$5.0^*\pm0.38$& $4.7^*\pm0.33$\\
~&$ME^{1\rightarrow 2}~1T/2F$& $0.000001\pm0.00$& $0.000001\pm0.00$&$3.7\pm0.46$& $3.0\pm0.34$\\
~&$ME^{2\rightarrow 1}~1T/2F$& $0.13^*\pm0.01$& $0.15^*\pm0.01$&$5.8^*\pm0.50$& $5.7^*\pm0.47$\\
~&$ME^{1\rightarrow 2}~1F/2T$& $0.133\pm0.01$& $0.13^*\pm0.01$&$4.2\pm0.34$& $4.4^*\pm0.34$\\
~&$ME^{2\rightarrow 1}~1F/2T$& $0.10\pm0.03$& $0.06\pm0.02$&$4.2\pm0.39$& $3.6\pm0.29$\\
\hline
\multirow{ 7}{*}{\rotatebox[origin=c]{90}{\bfseries With memory}}&
Av. perf. (\%)& $88.5\pm0.11$& $87.7\pm0.16$&$97.4\pm0.12$& $95.3\pm0.16$\\
~&$ME^{F\rightarrow T}$& $0.11^*\pm0.01$& $0.13^*\pm0.01$&$3.0^*\pm0.29$& $2.8^*\pm0.24$\\
~&$ME^{T\rightarrow F}$& $0.064\pm0.01$& $0.071\pm0.01$&$1.8\pm0.22$& $1.8\pm0.21$\\
~&$ME^{1\rightarrow 2}$& $0.085\pm0.01$& $0.10\pm0.01$&$2.4\pm0.29$& $2.3\pm0.22$\\
~&$ME^{2\rightarrow 1}$& $0.093\pm0.01$& $0.103\pm0.01$&$2.4\pm0.22$& $2.4\pm0.21$\\
~&$ME^{1\rightarrow 2}~1T/2F$& $0.063\pm0.01$& $0.064\pm0.01$&$1.8\pm0.24$& $1.8\pm0.23$\\
~&$ME^{2\rightarrow 1}~1T/2F$& $0.12^*\pm0.02$& $0.13^*\pm0.02$&$2.9^*\pm0.25$& $2.9^*\pm0.25$\\
~&$ME^{1\rightarrow 2}~1F/2T$& $0.106^*\pm0.01$& $0.13^*\pm0.02$&$3.1^*\pm0.35$& $2.8^*\pm0.25$\\
~&$ME^{2\rightarrow 1}~1F/2T$& $0.065\pm0.01$& $0.077\pm0.01$&$1.8\pm0.21$& $1.9\pm0.20$\\
\hline
\end{tabular}
\caption{Detailed ME values (compare to Table \ref{tab:perftable} in main paper). $1T/2F$ denotes games where the Tool Player is in first position and the Fruit Player is in second position, and $1F/2T$ denotes games where the Fruit Player in first position and Tool Player in second.}
\label{tab:perftable2}
\end{center}
\end{table*}


\begin{table*}[t!]
\begin{center}
  \small{
\begin{tabular}{ |c|ccc| }
\hline
 Utterances & Fruit & Tool 1 & Tool 2\\
\hline
Both, A is F & $42\pm2.21$& $32\pm2.04$& $27\pm1.33$\\
Both, B is F & $44\pm2.00$& $28\pm1.58$& $28\pm1.69$\\
\makecell{Both, Train A is F / Test B is F}& $6.8\pm0.61$& $11\pm1.16$& $8.8\pm0.68$\\
\makecell{Both, Train B is F / Test A is F} & $5.9\pm0.53$& $10\pm1.05$& $8.8\pm0.62$\\
\hline
Stats A is F & $6.4\pm0.27$& $8.9\pm0.42$& $8.2\pm0.38$\\
Stats B is F& $6.4\pm0.15$& $9.1\pm0.61$& $9.0\pm0.75$\\
\hline
\end{tabular}
}
\end{center}
\caption{Semantic classifier \% accuracy in inverted-roles setup}
\label{tab:invres}
\end{table*}

\section{Details on the semantics classifier}
\label{suppsec:sem}

\subsection{Classifier training and hyperparameters}

Our classifier consists of an Embedding table of size $50$ which maps the agents' discrete utterances to a continuous space, then uses a RNN with a hidden size of $100$ to map the entire embedded conversation into a hidden state. The hidden state is then fed to a linear classifier that predicts a score for each class, and the number of classes depends on the prediction task (e.g. $31$ classes when the task is to predict the fruit). We consider the setting with communication and with memory. From successful test in-domain conversations, we create train/validation/test partitions for the classifier. We ensure that each fruit is in the train set, and each tool in either of the two positions. We use $20$ different seeds for initializing the classifier dataset partitioning into train/validation/test. For each successful training seed, we compute the average accuracy over these $20$ test initialization seeds, and report the classifier accuracy mean and standard error of the mean (SEM) over the successful training seeds.

The agents were trained with symmetrical roles and random starting agent, but we generate  conversations with fixed roles and positions, so that all conversations follow the same pattern (for example: agent A always starts and agent A is always the Fruit Player).

\subsection{Inverted-roles experiment}

Table \ref{tab:invres} shows the results of the inverted-roles experiment: e.g., we train the classifier on conversations where A is Fruit Player and B is Tool Player, and test on conversations about the same inputs, but where the roles are inverted, that is, B is Fruit Player and A is Tool Player. The performance drops compared to testing on conversations where the roles are not inverted. For this experiment, we consider only the conversations that have at least one utterance from each agent (conversation length $\ge 2$) in order to remove the potential confounding effect of conversation length.

\end{document}
