This paper explores whether modern hybrid language model backbones can be turned into diffusion language models (DLMs). Diffusion language models are an alternative way to generate text. The usual route to build a DLM is to start from a pretrained transformer that uses full (bidirectional) attention. But many recent autoregressive (AR) models—models that predict text one piece at a time—now use hybrid architectures that mix attention layers with recurrent neural network (RNN) layers. RNNs are naturally causal, meaning they process input in one direction, and that makes them harder to turn into bidirectional components that many DLM methods assume.

The authors take a popular pretrained backbone, Qwen3.5, and adapt it into a family of diffusion models called dQwen3.5. They try four sizes of the backbone: 0.8 billion, 2 billion, 4 billion, and 9 billion parameters. The adaptation experiments compare the hybrid attention+RNN backbones to a full-attention transformer baseline. The team measures training loss (a standard number that shows how well a model is fitting its training data) and tests how the resulting models behave when decoding text in different orders and when generating many tokens in parallel.

The main finding is that hybrid backbones can be efficient starting points for adapting into DLMs. In the experiments, the hybrid models reached a given training loss after seeing roughly half as many tokens during training as the full-attention control. Across the different sizes, the dQwen3.5 models behaved like full-attention DLMs when it came to any-order decoding (that is, producing text not strictly left-to-right) and they performed well under parallel decoding (generating multiple tokens at once).

This matters because many recent and efficient pretrained AR models use hybrid designs. If those backbones can be converted into competitive DLMs, researchers and engineers might reuse existing models rather than train new full-attention transformers from scratch. That could save compute and make diffusion-style text generation more widely accessible.