Linkup research team releases SPARSEUP, an open-source learned sparse embedding model. The model runs on a 149M-parameter ModernBERT backbone and ships under Apache 2.0. Linkup team reports 56.4 average nDCG@10 on BEIR-13. It calls this the strongest public vocabulary-based sparse encoder it knows of under 150M parameters.
Is it deployable? Yes. The weights are on Hugging Face under Apache 2.0. The model loads through Transformers or Sentence Transformers with trust_remote_code=True.
Why a Sparse Model, and Why Now
Most open retrieval models are dense: 1 vector per text. Sparse models output weights over a vocabulary instead. Each dimension maps to a real token, so vectors fit inverted indexes and humans can read them. They also tend to match rare words well.
The trigger was LightOn’s DenseOn and LateOn release. LightOn published open data, a training recipe, a dense model and a late-interaction model. SPARSEUP fills the missing sparse slot. It uses the same backbone family and fine-tuning data, so all 3 retrieval styles can be compared side by side.
How is SPARSEUP Built
Training starts from LateOn-unsupervised. That checkpoint had no MLM head, so the team grafted back ModernBERT’s original one. Fine-tuning used LightOn’s fine-tuning mixture with contrastive learning only. Each query gets 7 hard negatives sampled from a pool of 50, and in-batch negatives. There is no cross-encoder distillation, and training fits on a single H100.
A vanilla SPLADE on this backbone produced huge bags full of stopwords. Linkup fixed this with 3 changes:
- Logit shifting: The encoder computes
log(1 + ReLU(x - 15)). ModernBERT’s MLM logits sat too high, saturating the log and making bags dense at initialization. - Per-position top-k: Each input token keeps only its 12 strongest vocabulary dimensions before max pooling. This caps expansion per token, not total vector size.
- Case folding: Byte-level BPE stores
heat,Heat,ĠheatandĠHeatas separate ids. SPARSEUP folds them onto 1 id and keeps the largest weight. Output dimensions drop from about 50k to about 34k.
Queries and documents take [Q] and [D] prefixes, and scoring is a dot product. Evaluation max lengths are 128 tokens for queries and 512 for documents.
Credit: Source link
