Hybrid Architectures for Language Models: Systematic Analysis and Design Insights

Recent progress in large language models demonstrates that hybrid architectures - combining self-attention mechanisms with structured state space models like Mamba - can achieve a compelling balance between modeling quality and computational efficiency, particularly for long-context tasks. This work presents a holistic evaluation of hybrid architectures based on inter-layer (sequential) or intra-layer (parallel) fusion, across language modeling and downstream task performance, long-context capabilities, scaling analysis, and training and inference efficiency, and proposes optimal design recipes for hybrid models.