Single-Layer Learnable Activation for Implicit Neural Representation (SL$^{2}$A-INR)
Moein Heidari, Reza Rezaeian, Reza Azad, Dorit Merhof, Hamid Soltanian-Zadeh, Ilker Hacihaliloglu
2024-09-18
Summary
This paper introduces SL²A-INR, a new type of neural network that improves how we represent complex signals by using a single-layer learnable activation function to enhance performance in various tasks.
What's the problem?
Implicit Neural Representations (INRs) are used to transform coordinates into attributes, but they struggle with capturing important details, especially high-frequency components. This limits their effectiveness in tasks like image representation and 3D shape reconstruction, where fine details are crucial.
What's the solution?
The researchers propose SL²A-INR, which incorporates a single-layer learnable activation function in its architecture. This allows the model to better capture fine details and adapt to different types of signals. The new approach significantly improves performance across various tasks, including image representation and CT reconstruction, setting new benchmarks for accuracy and quality.
Why it matters?
This research is important because it enhances the capabilities of neural networks in representing complex data. By improving how these models work, SL²A-INR can lead to better applications in fields like computer graphics, medical imaging, and any area where high-quality signal representation is needed.
Abstract
Implicit Neural Representation (INR), leveraging a neural network to transform coordinate input into corresponding attributes, has recently driven significant advances in several vision-related domains. However, the performance of INR is heavily influenced by the choice of the nonlinear activation function used in its multilayer perceptron (MLP) architecture. Multiple nonlinearities have been investigated; yet, current INRs face limitations in capturing high-frequency components, diverse signal types, and handling inverse problems. We have identified that these problems can be greatly alleviated by introducing a paradigm shift in INRs. We find that an architecture with learnable activations in initial layers can represent fine details in the underlying signals. Specifically, we propose SL^{2}A-INR, a hybrid network for INR with a single-layer learnable activation function, prompting the effectiveness of traditional ReLU-based MLPs. Our method performs superior across diverse tasks, including image representation, 3D shape reconstructions, inpainting, single image super-resolution, CT reconstruction, and novel view synthesis. Through comprehensive experiments, SL^{2}A-INR sets new benchmarks in accuracy, quality, and convergence rates for INR.