For decades, deep learning architectures have been divided into three distinct families—convolutional networks for locality, recurrent networks for sequential memory, and transformers for pairwise interactions. A new paper from researchers Dhor, Ashim, Mondal, Rasel, Chen, and Pin Yu, posted on arXiv, argues that this fragmentation is not fundamental but reflects incomplete views of a single underlying mathematical object: a learnable integral transform.
Their proposed architecture, the Integral Transform Network (ITNet), replaces the separate mechanisms of convolution, attention, and recurrence with a unified kernel that depends jointly on position and features. This kernel is implemented as a small multi-layer perceptron (MLP) that models pairwise interactions, allowing the network to adapt its behavior from data.
The Problem of Architectural Diversity
The paper states that "convolutional networks, recurrent networks, and transformers each encode different inductive biases—locality, sequential memory, and content-dependent pairwise interaction—and have remained mathematically distinct since their inception." This diversity forces practitioners to choose a specialized architecture for each task, limiting reuse and increasing development cost. ITNet aims to recover all three behaviors from a single learned interaction mechanism.
The Integral Transform Network (ITNet)
ITNet is built around a learnable kernel function implemented as an MLP. This kernel models pairwise interactions between inputs, enabling the network to adapt its computation to the data. The authors show that convolution, self-attention (including multi-head), and autoregressive recurrence (including LSTM, GRU, S4, and Mamba) arise as special cases of this integral transform under appropriate parameterizations. Moreover, ITNet is proved to be a universal approximator of continuous operators.
Performance Benchmarks
The paper reports that a single ITNet architecture with a shared operator and lightweight modality-specific encoders matches or exceeds specialized baselines on five major benchmarks:
| Benchmark Domain | Dataset | Performance Claim |
|---|---|---|
| Image Classification | ImageNet-1K | Matches or exceeds specialized CNNs/Vision Transformers |
| Natural Language Understanding | GLUE | Matches or exceeds specialized transformer models |
| 3D Point Cloud Classification | ModelNet40 | Matches or exceeds specialized point cloud networks |
| Visual Question Answering | VQA v2 | Matches or exceeds specialized VQA models |
| Visual Reasoning | NLVR2 | Matches or exceeds specialized reasoning models |
No specific accuracy numbers are provided in the abstract; the claim is qualitative.
Efficiency Innovations
To make the integral transform practical, the authors developed three key techniques:
- Tiled kernel fusion: Groups computation for efficiency.
- Importance-weighted Monte Carlo integration: Approximates the continuous integral with fewer samples.
- Learned low-rank factorization: Reduces the kernel's parameter count.
These methods enable ITNet to scale to large datasets and models while remaining computationally competitive with specialized architectures.
Implications for AI Research
ITNet challenges the long-standing separation between architectural families. If validated, it could simplify model design and transfer learning: a single architecture could handle vision, language, and sequential data without redesign. The paper is available on arXiv under a Creative Commons BY 4.0 license, with code and data associated with the article expected to be released.