Contrastive-LM has released CLM-8B, the first open model in a new class called Contrastive Language Models (CLMs). CLM does not generate text. It scores a set of candidate actions against the current state and returns probabilities. Their main baseline is Jev, the proprietary System One model from TypeSafe AI.
Is it deployable? Yes. The Apache-2.0 head weighs 75 MB. It runs on 1 NVIDIA GPU under Linux, with vLLM serving the Qwen3-8B encoder.
What a System One Model Does
Jev entered limited early access on 15 September 2026. It returns typed values with probabilities instead of text. CLM targets the same interface. The CLM GitHub repo serves CLM-8B behind a TypeSafe-compatible API. It exposes 3 question types:
- Noul: returns the probability that a statement is true.
- Choice: picks one option from a declared set, with probabilities.
- Score: returns an expected level on an ordered rubric.
A request written for TypeSafe’s API can be replayed through CLM’s Python client.
How CLM Works
CLM trains a state encoder and an action encoder with a bidirectional InfoNCE loss. Each encoder is a frozen Qwen3-8B backbone plus a 20M-parameter trainable projection head. Training pulls each state toward the action actually taken and pushes it away from the others.
At inference, CLM scores each candidate by the dot product of the state and action embeddings. A softmax over those scores becomes the answer distribution. The same primitive ranks best-of-N solutions, routes tools and answers typed decisions.
This design disaggregates states and actions. In an agent loop, the state changes every step while the action set stays mostly fixed. clm-serve reserves a slab of GPU memory, similar to vLLM’s KV cache, and reuses cached vectors. On 1 RTX 4090 with 3 actions, revisited states drop from 1.7 ms to 0.6 ms. The model card reports CLM running 13× faster than Jev with about 1,000 candidates.
A 3-Stage Training Recipe
- Pre-training on ~60M Nemotron DQA question-answer pairs.
- Mid-training on ~30M synthetic hard negatives generated by Gemini 2.5 Flash-Lite.
- Post-training on ~1M agent trajectories from Agent Data Protocol, Endless-Terminals and LiteCoder-Terminal-SFT.
On ~100K held-out questions, pre-training alone reaches 52.1% top-1 accuracy. Mid-training lifts it to 69.2%. Training on hard negatives from the start peaks at 62.4%, then overfits.
Zero-Shot Results Against Jev
| Task | CLM-8B latency | Jev latency | CLM-8B success | Jev success |
|---|---|---|---|---|
| T-Rex game | 16.5 ms | 149.8 ms | 5/5 | 5/5 |
| Tool calling (BFCL v4) | 76.8 ms | 125.5 ms | 95.2% | 99.2% |
| WikiRacing | 79.8 ms | 225 ms | 26/30 | 30/30 |
| Super Mario | 33.5 ms | 132.6 ms | 5/5 | 5/5 |
The 9× figure comes from the T-Rex game, where actions repeat across states. CLM matches Jev on T-Rex and Super Mario. It trails on tool calling and WikiRacing while running faster on every task.
CLM as a Verifier for Coding Agents
Here a generator samples several candidate solutions and the verifier picks one. Opus 5 produced DeepSWE candidates (best-of-4). Fable 5 produced Terminal-Bench 2.1 candidates (best-of-5). The team evaluated 38 held-out DeepSWE tasks and 30 held-out Terminal-Bench 2.1 tasks. Latency was measured on an H100.
| Benchmark | Pass@1 | CLM (fine-tuned) | Jev | CLM latency | Jev latency |
|---|---|---|---|---|---|
| DeepSWE | 73.7% | 81.6% | 71.1% | 79 ms | 449 ms |
| Terminal-Bench 2.1 | 84.0% | 87.6% | 83.1% | 32 ms | 131 ms |
The research team reports these as new SOTA verifier results. Jev scores below pass@1 on both benchmarks, so selecting with Jev is worse than taking 1 sample. CLM runs 4.1× to 5.7× faster. These numbers use lightweight fine-tuned heads, not the zero-shot checkpoint. They are held-out subset results, not full leaderboard submissions.
Interactive Explainer
Credit: Source link
