Learnable Global Layerwise Nonlinearities Without Activation Functions

Abstract

In machine learning and neural networks, non-linear transformations have been pivotal in capturing intricate patterns within data. These transformations are traditionally instantiated via activation functions such as Rectified Linear Unit (ReLU), Sigmoid, and Hyperbolic Tangent (Tanh). In this work, we introduce DiagonalizeGNN, an approach that changes the introduction of non-linearities in Graph Neural Networks (GNNs). Unlike traditional methods that rely on pointwise activation functions, DiagonalizeGNN employs Singular Value Decomposition (SVD) to incorporate global, non-piecewise non-linearities across an entire graph’s feature matrix. We provide the formalism of this method and empirical validation on a synthetic dataset, we demonstrate that our method not only achieves comparable performance to existing models but also offers additional benefits such as higher stability and potential for capturing more complex relationships. This novel approach opens up new avenues for research and offers significant implications for the future of non-linear transformations in machine learning