Different Types Of Gats

Posted by Packy on October 29, 2019

TYPES OF GRAPH ATTENTION MECHANISM

总的来说 Graph Attention可以被归纳为3种类型：

目的是相同的，只是注意力机制的定义 和 实现有所不同。

6.1 Learn attention weights Given the corresponding attributes/featuresxo,xl, • • , . I for vo, VI, • can be learned via: eug euk • , vlr l, attention weights where eo,J is node v, 's relevance to vo. In practice, this is typically implemented using softmax with a trainable function learning v, 's relevance to vo by considering their attributes. The implementation in GAT [Velickovic et al. 2018] illustrates this: exp (LeakyReLU(alWxollWxJl) exp where a is a trainable attention vector, W is a trainable weight matrix mapping the input features to the hidden space, and Il represents concatenation. An illustration of this is shown in Fig. 3a.

基于相似度的注意力机制。

6.2 Similarity-based attention Again, given the corresponding attributes or features, the second type of attention can be learned similarly as above except for a key difference. We call this approach similarity-based attention as more attention is given to object's that share more similar hidden representations or features, this is also often referred to in the literature as alignment [Bahdanau et al. 2015]. To illustrate this, we use the definition given in AGNN [Thekumparampil et al. 2018]: exp(p • cos Wxo,Wx exp where p is a trainable bias and "cos" represents cosine-similarity, like before, W is a trainable weight matrix to map inputs to the hidden space. Note that this is very similar to the above definition. The difference is that the model explicitly learns similar hidden embeddings for objects that are relevant to each other since attention is based on similarity or alignment.