Sparse retrieval underpins modern search systems, from web search to retrieval-augmented generation. Existing work has introduced Learned Sparse Retrieval (LSR) to push beyond exact lexical matching toward richer semantics. Yet LSR has so far remained tied to encoder-style bidirectional architectures, and its extension to multimodal settings still relies heavily on auxiliary cross-modal modules.
To address these limitations, we introduce UEmbed (Unified Embedding), a decoder-only multimodal embedding model that produces both sparse lexical and dense representations in one causal forward pass. UEmbed appends N learnable special tokens to the input and partitions the vocabulary into N disjoint subsets. Each token's causal hidden state predicts sparse weights over its assigned subset, and the N subsets are concatenated into the full sparse vector. This partitioned projection effectively circumvents the single-token representational bottleneck of causal models.
Trained on public data, we release UEmbed at 2B, 4B, and 9B scales. UEmbed-9B scores 71.8 (dense) and 71.0 (sparse) on MMEB-v2, leading models trained on public data in dense retrieval and setting the state of the art for sparse retrieval. On BEIR, UEmbed also remains competitive with strong dense and sparse baselines. Furthermore, we demonstrate the practical utility of UEmbed across three dimensions: effectiveness, efficiency, and agentic applications.
Overall, UEmbed offers a new paradigm: it unifies dense and sparse embeddings in one model, while further extending sparse retrieval to unify text and multimodal inputs.
The key challenge in adapting sparse retrieval to decoder-only models is that unidirectional attention prevents each position from attending to future tokens, rendering max-pooling over all hidden states ineffective. Relying on a single token (e.g., EOS) to project into a massive vocabulary space severely limits representational capacity.
UEmbed addresses this with vocabulary partitioning: N=16 learnable special tokens are appended to the input sequence, and the vocabulary is partitioned into N disjoint subsets via semantic k-means clustering, which encourages each special token to act as a "soft topic specialist". Each special token attends to all preceding tokens, summarizes the full input, and predicts sparse weights over its assigned subset through a subset-specific sparse head. The N subset vectors are concatenated into the full sparse vector, which plugs directly into standard inverted indexes.
Before partitioning, the vocabulary is compressed via accent stripping, lowercasing, and whitespace collapsing, merging redundant tokens and reducing the vocabulary from 248,320 to 184,016 entries.
Dense representation. The hidden state of the EOS token preceding the special tokens serves as the dense embedding. When only dense retrieval is needed, the special tokens can be omitted entirely, incurring no extra forward computation.
Unified objective. The model is trained with a combined loss over both retrieval modes: dense InfoNCE, sparse InfoNCE, and FLOPS regularizers on queries and documents that encourage sparsity. The jointly trained model closely matches both single-mode specialists, confirming that the dual-mode capability incurs only a negligible performance penalty.
Data curation. UEmbed is trained on 3.94M publicly available samples drawn from Echo-embedding training data, MLDR long-document data, and MMEB training sets, with hard negatives for the multimodal data mined by Qwen3-VL-Embedding-8B as a teacher model.
We instantiate UEmbed from the Qwen3.5 family at 2B, 4B, and 9B parameters; for each scale, a single checkpoint supports both dense and sparse retrieval. As a result, UEmbed unifies dense and sparse retrieval across text, image, video, and visual-document inputs within a single causal backbone.
UEmbed is competitive with the strongest open multimodal embedders at every scale. UEmbed-9B (dense) reaches an aggregate score of 71.8 on MMEB-v2. It is surpassed only by Qwen3-VL-Embedding-8B, which benefits from multi-stage training on vast proprietary datasets. Among models with publicly available checkpoints trained on open datasets, UEmbed-9B (dense) leads the field, outperforming RzenEmbed-V2-7B (71.1) and Ops-MM-Embed-7B (67.1). UEmbed-4B (dense, 70.4) surpasses every other model in the 4B-parameter class, such as Embed-RL-4B (68.1), and even the most compact UEmbed-2B (66.5) outperforms some 7B-scale open baselines like UniME-7B (64.1).
Sparse retrieval remains competitive with dense retrieval in the multimodal regime. The gap between the two modes is at most 1.0 point at every scale (e.g., 71.8 vs. 71.0 for the 9B model). To our knowledge, these are the first reported sparse multimodal results on this benchmark, and they significantly outperform established dense models: UEmbed-4B (sparse, 69.7) already surpasses the dense Ops-MM-Embed-7B (67.1). Sparse embeddings are especially effective on visually-rich document tasks, where the drop is minimal (79.2 vs. 79.1 for the 9B model), while the gap is slightly more pronounced on Video.
On nine BEIR datasets, UEmbed-9B achieves the highest average nDCG@10 of 56.3 (dense), with UEmbed-4B closely following at 56.0, placing both ahead of strong recent baselines like Qwen3-VL-Embedding-8B (55.5) and GME-7B (53.5). UEmbed is particularly strong on individual datasets, reaching 89.9 on Quora (9B) and 40.5 on NFCorpus (4B). In the sparse setting, UEmbed-9B reaches 55.2, effectively matching the strong specialist model Echo-Mistral-SPLADE (55.2), while simultaneously supporting dense retrieval and multimodal inputs within a single backbone.
Ablations. Unless noted, ablations use the 2B backbone trained on a random 500k subset and are evaluated on MMEB-v1. Joint dense-sparse training closely matches single-mode specialists; semantic k-means partitioning outperforms random and max-distance assignment; decoupling the sparse softmax temperature (τs=32) improves sparse performance without degrading dense retrieval; and performance stays stable up to N=16 special tokens, with a clear drop at N=32. UEmbed-2B also surpasses a SPLADE-style bidirectional baseline sharing the same backbone and data in both modes (+3.2 dense, +2.1 sparse on average), with the largest margins on IMG-QA (+8.2 dense, +6.0 sparse).
Cross-modal interpretability. UEmbed produces semantically meaningful lexical activations across modalities: it can deduce an image's location as Singapore based on its skyline and accurately identify a rocket as belonging to SpaceX. We also observe two limitations: the model occasionally emits anomalous tokens (e.g., "_alt") given the very large LLM vocabulary, and it predominantly activates English and Chinese tokens, reflecting the language distribution of the training corpus.
Practical advantages. Hybrid scoring over the dense and sparse modes improves Text (+0.3) and VisDoc (+0.5) at near-zero additional cost, since both representations come from a single forward pass; natural images and video frames carry little surface-form lexical signal, so the aggregate gain is modest. The purely autoregressive design retains compatibility with high-throughput serving stacks (e.g., vLLM) and standard inverted-index search. On the agentic search benchmark BrowseComp-Plus, the sparse mode of UEmbed consistently requires fewer tool-call search rounds than its dense counterpart, with higher recall at the 2B scale and comparable recall at 4B and 9B, validating sparse retrieval for cost-sensitive, iterative reasoning loops.
@misc{song2026uembed,
title={UEmbed: Unified Sparse and Dense Multimodal Embeddings},
author={Tingyu Song and Mingxin Li and Yanzhao Zhang and Dingkun Long and Pengjun Xie and Zhijie Nie and Yilun Zhao and Shu Wu},
year={2026},
archivePrefix={arXiv},
primaryClass={cs.CL},
}