Microsoft Researchers Introduce Samba 3.8B: A Simple Mamba+Sliding Window Attention Architecture that Outperforms Phi3-mini on Major Benchmarks

Large Language Models (LLMs) face two primary challenges in terms of complex long-term dependencies and efficient parallelization:

1. Capturing complex long-term dependencies: LLMs need to understand and generate human-like text by learning the intricate patterns and structures inherent in human language4. This requires the models to capture complex dependencies between words and phrases, which can be particularly challenging when dealing with long sequences of text. Existing models, such as attention-based models, struggle with computational complexity and extrapolation to longer sequences.

2. Achieving efficient parallelization for large-scale training: LLMs often require massive amounts of data for training, which can be computationally intensive and time-consuming. Parallelization is crucial for efficiently training these models on large-scale datasets. However, achieving efficient parallelization can be difficult, especially when dealing with complex model architectures and computational constraints.

These challenges have led researchers to explore alternative model architectures, such as State Space Models (SSMs), which offer linear computation complexity and the potential for better extrapolation. However, SSMs also have limitations, such as struggling with memory recall due to their Markovian nature. Hybrid models, like SAMBA, have been developed to combine the strengths of both attention mechanisms and SSMs to address these challenges simultaneously.

SAMBA (Systems for AMBient intelligence enabled by Agents) is a conceptual service-oriented architecture designed to support the interaction and interoperability of systems, applications, and actors in the context of Ambient Intelligence (AmI). The objective of SAMBA is to provide an ecosystem infrastructure that supports the interaction and interoperability of various elements by encapsulating and representing them through agents acting as members of an Ambient Intelligence Elements Society3.

The SAMBA architecture is composed of two interoperability layers bound together by the notion of models used at runtime:

1. AmIE Technical Interoperability Infrastructure (lower layer): This layer views all AmI elements as agents equipped with reasoning and dialogue capabilities, providing individual strategies selection. It integrates in a modular way the functional behavior of AmI components, non-functional execution constraints (particularly for mobile devices), service requests from AmI elements, services provided by AmI elements, and different security policies needed throughout an AmI application.

2. AmIE Semantic Interoperability Infrastructure (upper layer): This layer facilitates interoperability and interactions/communication among the diverse AmI elements. It spans across various heterogeneous, existing, and emerging service- and agent-related technologies and standards. AmI elements in SAMBA expose their functionality and semantics through the use of appropriate description documents and interfaces.

SAMBA integrates the strengths of both State Space Models (SSMs) and attention-based models by using a hybrid approach. Mamba, a variant of SSMs equipped with selective state spaces, is combined with the attention mechanism through layer-wise interleaving. Mamba layers capture time-dependent semantics and provide a backbone for efficient decoding, while attention layers model complex, non-Markovian dependencies. This combination allows SAMBA to achieve unlimited sequence length extrapolation with linear time complexity, while maintaining the ability to precisely recall memories with the attention mechanism.

Overall, SAMBA offers a powerful and efficient architecture for language modeling with unlimited-length extrapolation ability, demonstrating superior performance across various benchmarks and outperforming pure attention-based and SSM-based models.