Token-Saving Transformer Architecture
Novel attention sparsification and dynamic token reduction mechanism decreasing quadratic computational complexity.
Standard Transformer architectures suffer from quadratic time and memory complexity with respect to input sequence length (\(\mathcal{O}(N^2)\)). This project introduces a novel token-saving mechanism that dynamically identifies and prunes redundant attention tokens across deeper layers without degrading downstream representation fidelity.
Innovations
- Dynamic Token Pruning: Evaluates token saliency scores at intermediate layers to discard low-impact representations.
- Compute Efficiency: Achieves substantial reductions in FLOPs and KV-cache footprint for long-horizon forecasting and sequence tasks.
- High Retention Accuracy: Retains key semantic representations, matching full-attention baselines across benchmark evaluations.