Graph Transformer: A Generalization of Transformer Neural Network Architectures for Arbitrary Graphs
The Graph Transformer is a method proposed as a generalization of Transformer Neural Network architectures, designed for arbitrary graphs. This architecture is an enhanced version of the original Transformer and comes with several highlights, making it unique in its approach.
Attention Mechanism
The attention mechanism is a crucial part of the Graph Transformer architecture. Unlike the original Transformer, it considers the neighborhood connectivity of each node in the graph. This approach helps to model the importance of the relationships between nodes more accurately.
Position Encoding
Positional encoding refers to the ability of the model to keep track of the location of each node in the graph. The Graph Transformer uses Laplacian Eigenvectors to represent the position encoding, which naturally generalizes the sinusoidal positional encodings often used in natural language processing.
Layer Normalization vs. Batch Normalization
The Graph Transformer uses batch normalization as an alternative to layer normalization. Batch normalization normalizes the activations of a particular layer by utilizing mean and variance computed across the batch. This approach helps to counter covariate shift and makes the model more stable.
Edge Representation
The Graph Transformer architecture is extended to have edge representation, which can be critical to tasks with rich information on the edges. In these situations, pairwise interactions can be represented more accurately by considering bond types in molecules, relationship types in KGs, among others.
In summary, the Graph Transformer is a robust method for modeling arbitrary graphs. It offers an innovative approach to the attention mechanism, positional encoding, normalization, and edge representation, making it unique in its approach.
