Added 6.1

rmd/06-message-passing.rmd

@@ -8,4 +8,34 @@ We remark that many of the constructions discussed here are presented in their m
 
 ## Definition of higher-order message passing
 
+Higher-order message passing refers to a computational framework that involves exchanging messages among entities and cells in a higher-order domain using a set of neighborhood functions. In Definition \@ref(def:homp-definition), we formalize the notion of higher-order message passing for CCs. Figure \@ref(fig:homp) illustrates Definition \@ref(def:homp-definition).
+
+```{definition, homp-definition, name="Higher-order message passing on a CC"}
+Let $\mathcal{X}$ be a CC.  Let $\mathcal{N}=\{ \mathcal{N}_1,\ldots,\mathcal{N}_n\}$ be a set of neighborhood functions defined on  $\mathcal{X}$. Let $x$ be a cell and $y\in \mathcal{N}_k(x)$ for some $\mathcal{N}_k \in \mathcal{N}$. A *message* $m_{x,y}$ between cells $x$ and $y$ is a computation that depends on these two cells or on the data supported on them. Denote by $\mathcal{N}(x)$ the multi-set  $\{\!\!\{ \mathcal{N}_1(x) , \ldots ,  \mathcal{N}_n (x) \}\!\!\}$, and by $\mathbf{h}_x^{(l)}$ some data supported on the cell $x$ at layer $l$. \textbf{Higher-order message passing} on $\mathcal{X}$, induced by $\mathcal{N}$, is defined via the following four update rules:
+\begin{align}
+m_{x,y} &= \alpha_{\mathcal{N}_k}(\mathbf{h}_x^{(l)},\mathbf{h}_y^{(l)}), \\ \label{eqn:homp1}
+m_{x}^k &=  \bigoplus_{y \in \mathcal{N}_k(x)}  m_{x,y}, \; 1\leq k \leq n,   \\ \label{eqn:homp2}
+m_{x} &=  \bigotimes_{ \mathcal{N}_k \in \mathcal{N} } m_x^k, \\ \label{eqn:homp3}
+\mathbf{h}_x^{(l+1)} &= \beta (\mathbf{h}_x^{(l)}, m_x).
+(\#eq:homp0)
+\end{align}
+Here, $\bigoplus$ is a permutation-invariant aggregation function called the \textbf{intra-neighborhood} of $x$, $\bigotimes$ is an aggregation function called the *inter-neighborhood* of $x$, and $\alpha_{\mathcal{N}_k},\beta$ are differentiable functions.
+```
+
+```{r homp, echo=FALSE, fig.align="center", fig.cap="An illustration of higher-order message passing. Left-hand side: a collection of neighborhood functions $\\mathcal{N}_1,\\ldots,\\mathcal{N}_k$ are selected. The selection typically depends on the learning task. Right-hand side: for each $\\mathcal{N}_k$, the messages are aggregated using an intra-neighborhood function $\\bigoplus$. The inter-neighborhood function $\\bigotimes$ aggregates the final messages obtained from all neighborhoods."}
+knitr::include_graphics('figures/homp.png', dpi=NA)
+```
+
+Some remarks on Definition \@ref(def:homp-definition) are as follows. First, the message $m_{x,y}$ in Equation \@ref(eq:homp0) does not depend only on the data $\mathbf{h}_x^{(l)}$, $\mathbf{h}_y^{(l)}$ supported on the cells $x, y$; it also depends on the cells themselves. For instance, if $\mathcal{X}$ is a cell complex, the *orientation* of both $x$ and $y$ factors into the computation of message $m_{x,y}$. Alternatively, $x\cup y$ or $x\cap y$ might be cells in $\mathcal{X}$ and it might be useful to include their data in the computation of message $m_{x,y}$. This unique characteristic only manifests in higher-order domains, and does not occur in graphs-based message-passing frameworks [@gilmer2017neural; @bronstein2021geometric]^[The message 'direction' in $m_{x,y}$ is from $y$ to $x$. In general, $m_{x,y}$ and $m_{y,x}$ are not equal.]. Second, higher-order message passing relies on the choice of a set $\mathcal{N}$ of neighborhood functions. This is also a unique characteristic that only occurs in a higher-order domain, where a neighborhood function is necessarily described by a set of neighborhood relations rather than graph adjacency as in graph-based message passing. Third, in Equation \@ref(eq:homp0), since $y$ is implicitly defined with respect to a neighborhood relation $\mathcal{N}_k \in \mathcal{N},$ the function $\alpha_{\mathcal{N}_k}$ and the message $m_{x,y}$ depend on $\mathcal{N}_k$. Fourth, the inter-neighborhood $\bigotimes$ does not necessarily have to be a permutation-invariant aggregation function. For instance, it is possible to set an order on the multi-set $\mathcal{N}(x)$ and compute $m_x$ with respect to this order. Finally, higher-order message passing relies on two aggregation functions, the intra-neighborhood and inter-neighborhood, whereas graph-based message passing relies on a single aggregation function. The choice of set $\mathcal{N}$, as illustrated in Chapter \@ref(combinatorial-complexes), enables the use of a variety of neighborhood functions in higher-order message passing.
+
+```{remark}
+The push-forward operator given in Definition \@ref(pushing_exact_definition} is related to the update rule of Equation \@ref(eqn:homp0}. On one hand, Equation \@ref(eqn:homp0} requires two cochains $\mathbf{X}_i= [\mathbf{h}_{x^i_1}^{(l)},\ldots,\mathbf{h}_{x^i_{|\mathcal{X}^i|}}^{(l)}]$  and $\mathbf{Y}_{j}^{(l)}=[\mathbf{h}_{y^{j}_1}^{(l)},\ldots,\mathbf{h}_{y^{j}_{|\mathcal{X}^{j}|}}^{(l)}]$ to compute $\mathbf{X}^{(l+1)}_i = [\mathbf{h}_{x^i_1}^{(l+1)},\ldots,\mathbf{h}_{x^i_{|\mathcal{X}^i|}}^{(l+1)}] $, so signals on both $\mathcal{C}^j$ and $\mathcal{C}^i$ must be present in order to execute Equation \@ref(eqn:homp0}. From this perspective, it is natural and customary to think about this operation as an update rule. On the other hand, the push-forward operator of Definition \@ref(pushing_exact_definition} computes a cochain $\mathbf{K}_{j} \in \mathcal{C}^j $ given a cochain $\mathbf{H}_i\in \mathcal{C}^i$. As a single cochain $\mathbf{H}_i$ is required to perform this computation, it is natural to think about Equation \@ref(functional} as a function. See Section \@ref(merge and message passing} for more details.
+```
+
+The higher-order message-passing framework given in Definition \@ref(def:homp-definition) can be used to construct novel neural network architectures on a CC, as we have also alluded in Figure \@ref(fig:tdl). First, a CC $\mathcal{X}$ and cochains $\mathbf{H}_{i_1}\ldots, \mathbf{H}_{i_m}$ supported on $\mathcal{X}$ are given. Second, a collection of neighborhood functions are chosen, taking into account the desired learning task. Third, the update rules of Definition \@ref(def:homp-definition) are executed on the input cochains $\mathbf{H}_{i_1}\ldots, \mathbf{H}_{i_m}$ using the chosen neighborhood functions. The second and the third steps are repeated to obtain the final computations.
+
+```{definition, hmpsnn, name="Higher-order message-passing neural network"}
+We refer to to any neural network constructed using Definition \@ref(def:homp-definition) as a *higher-order message-passing neural network*.
+```
+
 ## Higher-order message-passing neural networks are CCNNs