Pytorch Geometric has a really great documentation. It has helper functions for data loading, data transformers, batching specific to graph data structures, and also has several graph neural network implementations. It also comes with easy loading of classic graph datasets like, Cora citation network, Zachary Karate Club and etc.

It also has a base Message Passing class so that we can implement our own Graph Neural Networks.

Data Handling

Graph in pytorch geometric is described by an instance of torch_geomtric.data.Data that has the following attributes.

• data.x: node features tensor of shape [num_nodes, num_node_features]
• data.edge_index: Graph connectivity in COO format with shape [2, num_edges]. Basically represents all the edges, an alternative to the Adjacency matrix representation. This is not a list of edges, if you have a list of edges you can get edge_index by doing this edge_index = edge_list.t().contiguous()
• data.edge_attr: Edge features tensor of shape [num_edges, num_edge_features]
• data.y: Target to train against, it can be anything from node labels of shape [num_nodes, *], or graph labels of shape [1, *]
• data.pos: Node position matrix with shape [num_nodes, num_dimensions]

GNN Layers

The minor difference from report 8 and pytorch documentation is that it has a much more generic equation and different variables, for getting the embeddings of a node at layer k from its neighbours. It goes as follows.

$x_i^{(k)}={\gamma }^k(x_i^{(k-1)},{\square }_{j\in \mathcal{N}(i)}{\varphi }^{(k)}(x_i^{(k-1)},x_j^{(k-1)},e_{j,i}))$

Where $\square$ our aggregation function, $x_i^{(k)}$ embedding of node $x_i$ at $laye{r}_k$, $e_{j,i}\in {\Re }^D$ denoting optional edge features from node $j$ to node $i$, $\gamma$ and $\varphi$ differentiable functions such as MLP's. $\gamma$, $\varphi$ and $\square$ vary according to the Graph neural network we use.

GCN

For example in case of a Graph Convolution Layer(GCN) we defined the embedding equation as

$h_v^k=\sigma (W_k\sum _{u\in N(v)}\frac{h_u^{k-1}}{|N(v)|}+B_k{h}_v^{k-1}),\forall k\in \{1,\cdots ,K\}$

We included the non-linearity $\sigma$ here, but in pytorch-geometric we have it as a separate ReLU layer after the GNN Layer.

Now we can see how we get our GCN equation from the generic equation accordingly

$\square =\sum$

$\varphi (x_i,x_j,e_{i,j})=x_j$

$\gamma (x_i,{\square }_{\mathcal{N}})=B{x}_i+W{\sum }_{\mathcal{N}}$

You can find how to implement GCN Layer from the message passing base class in the documentation here

You can find GCNConv layer from the pytorch geometric documentation here

GraphSAGE

Here the equation we had was

$h_v^k=\sigma ([A_k.AGG(\{h_v^{k-1},\forall u\in N(v)\}),B_k{h}_v^{k-1}])$

Where

$\square =AGG$

$\varphi (x_i,x_j,e_{i,j})=x_j$

$\gamma (x_i,{\square }_{\mathcal{N}})=[A.AG{G}_{\mathcal{N}},B{x}_i]$

Other Conv Layers

You can find the documentation for all the convolutional layers here.

For each layer the documentation has the embedding equation and the link to the original paper.

GCN on Zachary Karate Club Network

Here we trained a GCN Model on the Zachary Karate Club network to do a node classification task. We had 3 GCNConv layers, which implies every node got information aggregated messages from it's 3-hop neighbourhood You can see from the results, before the training the node embeddings were all over the place, but after training we were able to see a clear classification on the nodes. I got this training code from one of the example colab notebooks on pytorch geometric documentation. I forked the notebook to instrument it with wandb. You can find the notebook here

Comparison of Various GNN's on Cora Citation Network

We trained various GNN models on the Cora Citation Network, to see how each perform.

You can see how by using GNN we improved test accuracy from 0.5 to 0.8, normal MLP where just node features were used, in GNN we took advantage of the network structure along with node features..

Chapters

1. Introduction, Structure of Graphs
2. Properties of Networks and Random Graph Models
3. Motifs and Structural Roles in Networks
4. Community Structure in Networks
5. Spectral Clustering
6. Message Passing and Node Classification
7. Graph Representation Learning
8. Graph Neural Networks
9. Graph Neural Networks - Pytorch Geometric
10. Deep Generative Models for Graphs
11. Link Analysis: PageRank
12. Network Effects and Cascading Behaviour
13. Probabilistic Contagion and Models of Influence
14. Influence Maximization in Networks
15. Outbreak Detection in Networks
16. Network Evolution
17. Reasoning over Knowledge Graphs
18. Limitations of Graph Neural Networks
19. Applications of Graph Neural Networks