The foundation, frontier and application of GNN

# In recent years, graph neural networks (GNN) have made rapid and incredible progress. Graph neural network, also known as graph deep learning, graph representation learning (graph representation learning) or geometric deep learning, is the fastest growing research topic in the field of machine learning, especially deep learning. The title of this sharing is "Basics, Frontiers and Applications of GNN", which mainly introduces the general content of the comprehensive book "Basics, Frontiers and Applications of Graph Neural Networks" compiled by scholars Wu Lingfei, Cui Peng, Pei Jian and Zhao Liang. .

1. Introduction to graph neural networks

1. Why study graphs?

Diagrams are a universal language for describing and modeling complex systems. The graph itself is not complicated, it mainly consists of edges and nodes. We can use nodes to represent any object we want to model, and we can use edges to represent the relationship or similarity between two nodes. What we often call graph neural network or graph machine learning usually uses the structure of the graph and the information of edges and nodes as the input of the algorithm to output the desired results. For example, in a search engine, when we enter a query, the engine will return personalized search results based on the query information, user information, and some contextual information. This information can be naturally organized in a graph.

2. Graph structured data is everywhere

Graph structured data can be found everywhere, such as the Internet, social networks, etc. In addition, in the currently very popular field of protein discovery, people will use graphs to describe and model existing proteins and generate new graphs to help people discover new drugs. We can also use graphs to do some complex program analysis, and we can also do some high-level reasoning in computer vision.

3. Recent trends in graph machine learning

Graph machine learning It is not a very new topic. This research direction has been in the past 20 years, and it has always been relatively niche before. Since 2016, with the emergence of modern graph neural network-related papers, graph machine learning has become a popular research direction. It is found that this new generation of graph machine learning method can better learn the data itself and the information between the data, so that it can better represent the data, and ultimately be able to better complete more important tasks.

4. A brief history of graph neural networks

The earliest paper related to graph neural network appeared in 2009, before deep learning became popular. Papers on modern graph neural networks appeared in 2016, which were improvements to early graph neural networks. Afterwards, the emergence of GCN promoted the rapid development of graph neural networks. Since 2017, a large number of new algorithms have emerged. As the algorithms of graph neural networks become more and more mature, since 2019, the industry has tried to use these algorithms to solve some practical problems. At the same time, many open source tools have been developed to improve the efficiency of solving problems. Since 2021, many books related to graph neural networks have been written, including of course this "Basics, Frontiers and Applications of Graph Neural Networks".

The book "Basics, Frontiers and Applications of Graph Neural Networks" systematically introduces the core concepts and technologies in the field of graph neural networks, as well as cutting-edge research and development, and introduces applications in different fields. Applications. Readers from both academia and industry can benefit from it.

##2. The basis of graph neural network

1. The life cycle of machine learning

The above figure reflects the life cycle of machine learning, in which feature learning is a very important link. Its main task is to transform raw data into structured data. Before the emergence of deep learning, everyone mainly completed this task through feature engineering. After the emergence of deep learning, this end-to-end machine learning method began to become mainstream

.

2. Feature learning in the graph

Feature Learning in Graphs is very similar to deep learning. The goal is to design an effective task-related or task-independent feature learning method to map the nodes in the original graph into a high-dimensional space to obtain the embedding of the nodes. representation, and then complete the downstream tasks.

3. The basis of graph neural network

There are two types of representations that need to be learned in graph neural networks:

• Representation of graph nodes

Requires a filter operation, which takes the matrix of the graph and the vector representation of the node as input, continuously learns, and updates the vector representation of the node. Currently, the more common filter operations include Spectral-based, Spatial-based, Attention-based, and Recurrent-based.

• Representation of the graph

Requires a pool operation, Taking the matrix of the graph and the vector representation of the nodes as input, it continuously learns to obtain the matrix of the graph containing fewer nodes and the vector representation of its nodes, and finally obtains a graph-level vector representation to represent the entire graph. Currently, the more common pool operations include Flat Graph Pooling (such as Max, Ave, Min) and Hierarchical Graph Pooling (such as Diffpool).

4. Basic model of graph neural network

There is a context learning in the field of machine learning concept. In graph neural networks, the context of a node is its neighbor nodes. We can use the neighbor nodes of a node to learn the vector representation of this node.

In this way, each node can define a calculation graph.

We can layer the calculation graph. The first layer is the most original information, and the sum is passed layer by layer. Aggregate information to learn vector representations of all nodes.

The above figure roughly describes the main steps of graph neural network model learning. There are mainly four steps:

• Define a Aggregation function;
• Define the loss function according to the task;
• Train a batch of nodes , for example, you can train a batch of calculation graphs at one time;
• # produces the required vector representation for each node, even some nodes that have never been trained. (What is learned is the aggregation function, and the vector representation of the new node can be obtained by using the aggregation function and the vector representation that has been trained).

##Picture above is an example of using average as an aggregation function. The vector representation of node v in the kth layer depends on the average of the vector representations of its neighbor nodes in the previous layer and its own vector representation in the previous layer.

To summarize the above content, the main point of the graph neural network is to generate the target node by aggregating the information of neighbor nodes. Vector representation of points, which takes into account parameter sharing in the encoder and also allows for inference learning.

5. Popular models of graph neural networks

##The classic or popular algorithm essence of graph neural networks The above is to use different aggregation function or filter function, which can be divided into supervised graph neural network and unsupervised graph neural network.

##GCN

is one of the most classic algorithms, It can Act directly on the graph and exploit its structural information. Focusing on improving model speed, practicality and stability, as shown in the figure above, GCN has also gone through several iterations. The GCN paper is of epoch-making significance and laid the foundation for graph neural networks.

MPNN

The core pointis to transform the graph into convolution For the process of information transmission, it defines two functions, namely aggregation function and update function. This algorithm is a simple and general algorithm, but it is not efficient.

GraphSage

is an industrial-level algorithm. It uses sampling to get a certain number of neighbor nodes. Thus the vector representation of the school node.

GAT

is the introduction of the idea of ​​attention, its core The point is to dynamically learn edge weights during information transfer.

In addition to the algorithms introduced above, there is also GGNN. Its characteristic is that the output can be multiple nodes. If you are interested, you can check it out. Related papers.

In the book "Basics, Frontiers and Applications of Graph Neural Networks", Chapters 5, 6, 7 and 8 also introduce how to evaluate the scalability of graph neural networks and graph neural networks respectively. , the interpretability of graph neural networks, and the adversarial stability of graph neural networks. If you are interested, you can read the corresponding chapters in the book.

##3. The Frontier of Graph Neural Networks

1. Graph Structure Learning

Graph neural network requires graph structure data, but it is doubtful whether the given graph structure is optimal. Sometimes there may be a lot of noise, and many applications There may be no graph-structured data, or even just raw features.

So, we need to use the graph neural network to learn the optimal graph representation and graph node representation.

We transform the learning of the graph into the learning of similarities between nodes, and control the smoothness through regularization , system attributes and connectivity, and iteratively refine the structure of the graph and the vector representation of the graph.

#Experimental data can show the advantages of this method.

Through the visualization results of the graph, it can be found that the learned graphs tend to compare similar graphs Objects are clustered together and have a certain interpretability.

2. Other Frontiers

In the book "Basics, Frontiers and Applications of Graph Neural Networks", the following frontiers are also introduced. Research, these cutting-edge research have important applications in many scenarios:

• Graph classification;
• Link Prediction;
• Graph generation;
• Graph conversion;
• Graph matching;
• Dynamic graph neural network;
• Heterogeneous Graph Neural Network;
• AutoML for Graph Neural Network;
• Self-supervised learning of graph neural networks.

4. Application of graph neural network

1. Application of graph neural network in recommendation system

We can use session information to construct a heterogeneous global graph, and then learn the vector representation of users or items through graph neural network learning, and use this vector representation for personalization recommendation.

2. Application of graph neural network in computer vision

We can track the dynamic change process of objects, Deepen your understanding of video with graph neural networks.

3. Application of graph neural network in natural language processing

We can use graph neural networks to understand high-level information of natural language.

4. Application of graph neural network in program analysis

##5. Application of graph neural network in smart cities

##5. Q&A session

Q1: Is GNN an important method for the next generation of deep learning?

#A1: Graph neural network is a very important branch, and the one that keeps pace with graph neural network is Transformer. In view of the flexibility of graph neural networks, graph neural networks and Transformer can be combined with each other to take advantage of greater advantages.

Q2: Can GNN and causal learning be combined? How to combine?

#A2: The important link in causal learning is the causal graph. The causal graph and GNN can be naturally combined. The difficulty of causal learning is that its data size is small. We can use the ability of GNN to better learn causal graphs.

Q3: What is the difference and connection between the interpretability of GNN and the interpretability of traditional machine learning?

#A3: There will be a detailed introduction in the book "Basics, Frontiers and Applications of Graph Neural Networks".

Q4: How to train and infer GNN directly based on the graph database and using the capabilities of graph computing?

A4: At present, there is no good practice on the unified graph computing platform. There are some startup companies and scientific research teams exploring related directions. This will be a very For valuable and challenging research directions, a more feasible approach is to divide the research into different fields.

The above is the detailed content of The foundation, frontier and application of GNN. For more information, please follow other related articles on the PHP Chinese website!