A Brief Introduction to Residual Gated Graph Convolutional Networks

This article provides a brief overview of the Residual Gated Graph Convolutional Network architecture, complete with code examples in PyTorch Geometric and interactive visualizations using W&B.
Saurav Maheshkar
Created on August 31|Last edited on June 28
Comment
In this article, we'll briefly go over the Residual Gated Graph Convolutional Network architecture proposed in the paper Residual Gated Graph ConvNets by Xavier Bresson and Thomas Laurent, a fundamental model from the Graph Convolutional Network paradigm inspired by traditional Recurrent Neural Network (RNN) and Convolution Network (CNN) Literature.
There are three main classes of models of Graph Neural Networks, namely Message Passing Graph Neural Networks, Graph Convolutional Networks, and Graph Attention Networks. For a brief overview of the three paradigms, you can refer to the following blogs:
An Introduction to Convolutional Graph Neural Networks
This article provides a beginner-friendly introduction to Convolutional Graph Neural Networks (GCNs), which apply deep learning paradigms to graphical data. 
An Introduction to Message Passing Graph Neural Networks
This article provides a beginner-friendly introduction to Message Passing Graph Neural Networks (MPGNNs), which apply deep learning paradigms to graphical data. 
An Introduction to Graph Attention Networks
This article provides a beginner-friendly introduction to Attention based Graphical Neural Networks (GATs), which apply deep learning paradigms to graphical data. 
﻿
﻿Check out the Colab Notebook of these experiments here ﻿⟶\longrightarrow⟶﻿﻿
Table of ContentsResidual Gated GCNs: The MethodImplementing the ModelResultsSummaryRecommended Reading
﻿
﻿
Residual Gated GCNs: The MethodThe main problem that the authors of the paper studied was the case of variable-length graphs. So far, we have looked into classical Graph Convolutional Methods based on the approximations of the spectral graph convolutions as mentioned in our introductory blog on Graph Convolutional Networks and it's a natural extension to deal with missing nodes in the blog on GraphSAGE.
For fixed-length graphs, our first hidden state is simply the vector of all node representations. However, for graphs with variable length, the major differences lie in how we generate our initial representations. In this case, we generate our first hidden state by using a vanilla RNN with an MLP. This paper extends the said formulation by borrowing from LSTM and CNN literature by introducing residual connections like CNN and introducing a gated layer. 
Let's look into how this method can be formulated based on the initial formula introduced in the introductory blog post.
The most general formulation is as follows:
hv=ϕ(xu, ⊕v∈Nu cuv ψ(xv))\huge h_v = \phi(x_u , \, \oplus_{v \in \mathcal{N}_u} \, c_{uv} \, \psi(x_v))hv​=ϕ(xu​,⊕v∈Nu​​cuv​ψ(xv​))﻿
In the case of Residual Gated Graph ConvNets, the update function is of the form:
hu=W1xu+∑v∈Nuηuv⊙w2xv\huge h_u = W_1x_u + \displaystyle \sum_{v \in \mathcal{N}_u} \eta_{uv} \odot w_2 x_vhu​=W1​xu​+v∈Nu​∑​ηuv​⊙w2​xv​﻿
Where η\large \etaη﻿ is the aforementioned gating operation, which is of the following form:
ηuv=σ(W3xu +W4xv)\huge \eta_{uv} = \sigma(W_3x_u \, + W_4x_v)ηuv​=σ(W3​xu​+W4​xv​)﻿
Implementing the Model﻿PyTorch Geometric provides a great implementation of the update rule outlined in the paper (ResGatedGraphConv). 
Let's walk through a minimal example implementation!
class ResGatedGCN(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = ResGatedGraphConv(in_channels, hidden_channels)
        self.conv2 = ResGatedGraphConv(hidden_channels, out_channels)
﻿
    def forward(self, x, edge_index):
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.conv1(x, edge_index).relu()
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.conv2(x, edge_index)
        return x
ResultsWe train some models for 50 epochs to perform Node classification on the Cora Dataset, using the minimal model implementation as stated above, and report the training loss and accuracy comparing the effect of the hidden dimension on the overall performance.
﻿
Run set3
﻿
SummaryIn this article, we learned about the Residual Gated Graph Convolutional Network architecture, along with code and interactive visualizations. To see the full suite of W&B features, please check out this short 5-minute guide.
If you want more reports covering graph neural networks with code implementations, let us know in the comments below or on our forum ✨!
Check out these other reports on Fully Connected covering other Graph Neural Networks-based topics and ideas.
Recommended Reading
Graph Neural Networks (GNNs) with Learnable Structural and Positional Representations
An in-depth breakdown of "Graph Neural Networks with Learnable Structural and Positional Representations" by Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio and Xavier Bresson.
Part 1 – Introduction to Graph Neural Networks With GatedGCN
This article summarizes the need for Graph Neural Networks and analyzes one particular architecture – the Gated Graph Convolutional Network.
An Introduction to GraphSAGE
This article provides an overview of the GraphSAGE neural network architecture, complete with code examples in PyTorch Geometric, and visualizations using W&B. 
Using W&B with DeepChem: Molecular Graph Convolutional Networks
A quick tutorial on using W&B to track DeepChem molecular deep learning experiments
﻿
﻿
Add a comment
Tags: Articles, GNN, Tutorial, Intermediate
Iterate on AI agents and models faster. Try Weights & Biases today.