Lost in compression: the failure of SSMs to propagate factual context in transformer reconstruction

Abstract

State Space Models (SSMs) have emerged as a promising alternative to Transformer architectures, offering improved computational efficiency and scalability for long-context sequence modeling. Furthermore, several hybrid SSM-Transformer architectures have been proposed, aiming to combine the efficient context modeling capabilities of SSMs with the expressive decoding power of Transformers. These hybrid approaches are commonly claimed to offer advantages over Transformer-only or SSM-only models. In this work, we investigate a critical weakness of such hybrid designs by constructing an encoder-decoder language model in which an SSM-based architecture serves as the encoder and a Transformer serves as the decoder. Although this model produces grammatically fluent outputs, we observe systematic degradation in factual accuracy when evaluating it on text summarization tasks. Through controlled experiments, we demonstrate that the latent representations generated by the SSM encoder are insufficient for accurate factual reconstruction during Transformer-based decoding. Our findings suggest that SSM-based encoders compress contextual information in a manner that is not fully decodable for factual and faithful summarization, exposing a fundamental limitation of SSMs in encoding context-rich representations suitable for downstream reconstruction. This work provides insights into the challenges of fact-preserving context compression in neural language models.