devfactory.musolving real problems with AI you run yourself

LatestBenchmarksintermediate

Qwen3.8-27B on One RTX 3090

Four Qwen3.8-27B configurations measured on the same 24GB RTX 3090: mainline llama.cpp against ik_llama.cpp, unsloth UD-IQ4_XS and UD-Q4_K_M against ubergarm IQ4_KS. Throughput on the club-3090 harness, prompt processing at 10K and 90K, and retrieval through a 106,612-token working context.

The winner changes once the context fills Two panels comparing the fastest configuration under two conditions, each showing the winner's margin over the runner-up as a bar on a shared nought to thirty percent scale. On a short prompt, ik_llama.cpp with ubergarm IQ4_KS is fastest at 63.97 tokens per second on code, just ahead of mainline llama.cpp with unsloth UD-IQ4_XS at 61.96: a margin of 3 percent, drawn as a short bar, and it uses 3.1GB more VRAM to get there. With a 106,612-token working context loaded, the ordering reverses. Mainline decodes at 49.06 tokens per second against 38.26 for ik_llama.cpp, a margin of 28 percent, drawn as a bar nearly ten times longer, while also processing the prompt 22 percent faster and using 3.1GB less VRAM. Both configurations retrieved all six planted codes, so the difference is speed and memory rather than capability. > THE WINNER CHANGES ONCE THE CONTEXT FILLS SHORT PROMPT code decode, thinking off ik_llama.cpp · IQ4_KS 63.97 tok/s llama.cpp · UD-IQ4_XS  61.96 margin: 3 percent, and 3.1GB more VRAM AT 106,612 TOKENS decode with a working context loaded llama.cpp · UD-IQ4_XS 49.06 tok/s ik_llama.cpp · IQ4_KS  38.26 margin: 28 percent, and 3.1GB less VRAM Both retrieved 6/6 planted codes. One RTX 3090 · 131,072 ctx · q8_0 KV · draft depth 4. Margin bars share a 0 to 30 percent scale; the two tests are not comparable to each other.
The fastest configuration on a short prompt is not the fastest one with a working context loaded.

From the lab

Every figure here is measured on this fleet or sourced and attributed, and each article says which. Nine cards in production across two hosts, four more in stock and not yet racked.

GPU compute fabric

inference in progress

  1. 01route
  2. 02load
  3. 03compute
  4. 04tokens

host.aCUDA3× RTX 3090

  • gpu 0
  • gpu 1
  • gpu 2

host.bROCm6× MI100

  • gpu 0
  • gpu 1
  • gpu 2
  • gpu 3
  • gpu 4
  • gpu 5

9 serving4 stagedthe cycle illustrates an inference request; it is not telemetry

Signals

News and releases worth noticing, filtered through what they change for people running AI themselves.

Infrastructure

ROCm 10 turns AMD's software stack into a product.

AMD released ROCm 10 with validated vLLM and SGLang containers, modular packages, ROCm CLI, local observability and AI-assisted workflows.

Why it matters

AMD's hardware was rarely the whole problem. ROCm 10 directly targets the integration and operational friction that kept NVIDIA easier to deploy.

AMD (opens in a new tab)
Models

Ox Alpha was GLM-5.3-Flash all along.

Z.ai revealed the anonymous model as GLM-5.3-Flash, an open-weight 320B/18B-active multimodal MoE quietly tested through public model platforms.

Why it matters

A model that earned attention before anyone knew its name is stronger evidence than another vendor benchmark, and the MIT-licensed weights can now be tested locally.

Unsloth (opens in a new tab)
Models

Qwen just previewed the architecture behind Qwen4.

Qwen3.8-Flash-Next is an open-weight multimodal MoE introducing new attention, residual and embedding designs aimed at cutting compute and memory costs.

Why it matters

This is less about one model release than where Qwen is heading: larger capacity, cheaper long context and architectures built around efficient inference.

Unsloth (opens in a new tab)
All signals →

Also published

All posts →