devfactory.musolving real problems with AI you run yourself

Qwen3.8-27B on One RTX 3090

Four Qwen3.8-27B configurations measured on the same 24GB RTX 3090: mainline llama.cpp against ik_llama.cpp, unsloth UD-IQ4_XS and UD-Q4_K_M against ubergarm IQ4_KS. Throughput on the club-3090 harness, prompt processing at 10K and 90K, and retrieval through a 106,612-token working context.

intermediatemeasuredupdated 2 September 2026

The RTX 3090 remains a useful local AI card because 24 GB of VRAM is enough to run models that would otherwise require much newer hardware. But fitting the weights is only the first test. A practical configuration also has to generate quickly, ingest large prompts, leave usable memory headroom and keep working once the context fills.

That is what we tested with Qwen3.8-27B.

Qwen released the 27B model on 14 August 2026. Its official model card lists a native 262,144-token context and training with Multi-Token Prediction (MTP). For production or high-throughput serving, Qwen recommends dedicated engines such as SGLang, vLLM and TokenSpeed. Qwen3.8-27B model card · Qwen3.8 release repository

Two days earlier, club-3090 had retired the relevant single-card Qwen3.6-27B llama.cpp and ik_llama.cpp recommendation paths while consolidating that surface around vLLM. The old measurements still stood, but those paths were no longer maintained defaults. club-3090 retired Qwen3.6-27B single-card profile

So the question was straightforward: if we still want Qwen3.8-27B on one 24 GB RTX 3090 using GGUF, which configuration is worth running?

What we tested

All published results came from the same RTX 3090 with 24,576 MiB of VRAM. The server context was configured to 131,072 tokens, K/V cache was q8_0, and MTP used a maximum draft depth of four.

ConfigurationEngineWeights
mainline + UD-IQ4_XSllama.cppunsloth UD-IQ4_XS
ik + IQ4_KSik_llama.cppubergarm MTP-IQ4_KS
ik + UD-IQ4_XSik_llama.cppunsloth UD-IQ4_XS
ik + UD-Q4_K_Mik_llama.cppunsloth UD-Q4_K_M

The duplicate UD-IQ4_XS row is deliberate. Running the same weights in both engines lets us distinguish an engine difference from a quantisation difference.

For the short-prompt benchmark we used club-3090's harness, which we did not write and did not modify: three warm-ups followed by five measured requests for both narrative and code workloads. Thinking was disabled and the harness used temperature 0.6. The 10K and 90K probes measure prompt processing rather than generation.

Short prompts: IQ4_KS wins decode, mainline wins prefill

Throughput with reasoning suppressed: four configurations on one RTX 3090 A four-row bar table of throughput measured with reasoning suppressed, using club-3090's bench.sh at three warmup plus five measured runs per category. Columns are narrative decode, code decode, prefill at 10,000 tokens, prefill at 90,000 tokens and free VRAM at peak. ik_llama.cpp with ubergarm IQ4_KS leads decode at 45.58 narrative and 63.97 code, ahead of mainline llama.cpp with unsloth UD-IQ4_XS at 44.73 and 61.96. Mainline leads prefill decisively at 876 tokens per second at 10K and 636 at 90K, against 715 to 731 and 494 to 508 for the three ik_llama.cpp configurations. Mainline also leaves the most headroom, 4.5GB, against 1.5GB for ubergarm. unsloth UD-Q4_K_M is last on decode at 34.08 narrative and 49.04 code. > THROUGHPUT, REASONING SUPPRESSED CASE: short prompts, long generations, reasoning off club-3090 bench.sh · 3 warmup + 5 measured · temp 0.6 · one RTX 3090 · 131,072 ctx · q8_0 KV ENGINE · FILE NARRATIVE CODE PF 10K PF 90K FREE llama.cpp · UD-IQ4_XS 44.73 61.96 876 636 4.5GB ik_llama.cpp · IQ4_KS 45.58 63.97 731 508 1.5GB ik_llama.cpp · UD-IQ4_XS 41.32 56.68 716 499 3.5GB ik_llama.cpp · UD-Q4_K_M 34.08 49.04 725 494 1.6GB Decode in tokens/s, prefill in prompt tokens/s. Free VRAM measured at peak during a request. Speed only. Nothing here measures whether the output is any good.
Throughput with reasoning suppressed. Three warm-up plus five measured runs per workload, one RTX 3090.
ConfigurationNarrativeCodePrefill 10KPrefill 90KPeak VRAM
mainline + UD-IQ4_XS44.73 tok/s61.96 tok/s876.38 tok/s636.30 tok/s19,944 MiB
ik + IQ4_KS45.58 tok/s63.97 tok/s730.97 tok/s507.56 tok/s23,052 MiB
ik + UD-IQ4_XS41.32 tok/s56.68 tok/s715.61 tok/s499.34 tok/s20,978 MiB
ik + UD-Q4_K_M34.08 tok/s49.04 tok/s724.61 tok/s493.58 tok/s22,922 MiB

The short-prompt decode winner is ik + IQ4_KS, ubergarm's IQK-format quant, but the lead over mainline is small: about 1.9% on narrative and 3.2% on code.

Prefill is different. Mainline processes the 10K probe about 20% faster than IQ4_KS and the 90K probe about 25% faster. That matters for code assistants, retrieval systems and agents because their prompts grow as files, tool results and conversation history accumulate.

The memory difference is also substantial. Peak VRAM is 19,944 MiB for mainline against 23,052 MiB for IQ4_KS, a gap of 3,108 MiB, or about 3.0 GiB.

These are the final figures from the shipped configuration. Mainline and IQ4_KS are means of two canonical runs; the other two rows are single canonical runs. Across the measured runs, decode was noisier than prefill, so the small short-prompt gap should be read as small rather than as a universal ranking.

Engine versus quantisation

The matched UD-IQ4_XS rows give the cleanest engine comparison:

Testik / mainline
Narrative decode0.92x
Code decode0.91x
10K prefill0.82x
90K prefill0.78x

With the same weights and draft depth, mainline was faster in every measured workload.

Inside ik_llama.cpp, however, switching from UD-IQ4_XS to IQ4_KS raised decode by about 1.10x on narrative and 1.13x on code. That explains why IQ4_KS can still top the short-prompt table even though the matched-weight engine comparison favours mainline.

This also helps explain club-3090's earlier Qwen3.6-27B result, where its single-card IQ4_KS recipe was documented at roughly 18 to 20% faster decode than its mainline Q4_K_M path. Our Qwen3.8 test does not reproduce that comparison directly because we did not run Q4_K_M on mainline. What it does show is that the engine alone is not enough to explain the short-prompt IQ4_KS lead: the matched UD-IQ4_XS comparison favours mainline, while changing the quant inside ik materially changes decode speed. club-3090 single-card comparison

The bits-per-weight data points in the same direction. Repeating-layer BPW was 4.140 for UD-IQ4_XS, 4.461 for IQ4_KS and 4.745 for UD-Q4_K_M, yet the middle file was the fastest of the three on ik. Fewer bits therefore did not map directly to higher throughput in this test. We did not isolate which kernel or layout differences caused that result.

At 106K context, the ranking changes

A configured context size is not the same as a tested working context, so we also ran a retrieval test through a 106,612-token prompt.

The deterministic haystack contained six unique codes distributed from early context through the middle, deep context, near the end and at the end. Every run used the same prompt, a cold cache, temperature 0 and reasoning at medium. The generator used a fixed seed, and scoring was exact string retrieval.

ConfigurationRetrievalPrefillDecodePeak VRAMWall time
mainline + UD-IQ4_XS6/6613.81 tok/s49.06 tok/s19,940 MiB3m06s
ik + IQ4_KS6/6503.31 tok/s38.26 tok/s23,150 MiB4m02s
ik + UD-IQ4_XS6/6484.07 tok/s37.36 tok/s21,074 MiB4m12s
ik + UD-Q4_K_M6/6482.68 tok/s34.16 tok/s23,020 MiB4m14s
Working context with reasoning on: can each configuration still reach what it holds The same four configurations tested at 106,612 prompt tokens with reasoning enabled at medium effort, one request each, cold cache. All four retrieved six of six codes spread from 12 percent to 100 percent of the prompt, so retrieval capability does not separate them. Speed does: mainline llama.cpp with unsloth UD-IQ4_XS prefills at 613.81 tokens per second and decodes at 49.06, while the three ik_llama.cpp configurations prefill at 482 to 503 and decode at only 34.16 to 38.26. Free VRAM at this length ranges from 4.5GB on mainline to 1.4GB on ubergarm IQ4_KS. Compared with the reasoning-suppressed figures, the ordering on decode reverses, but the two tests use different sampler and reasoning settings and are not directly comparable. > WORKING CONTEXT, REASONING ON CASE: one 106,612-token prompt, six needles, reasoning at medium single request · temp 0 · cold cache · same card and session as above ENGINE · FILE NEEDLES PREFILL DECODE FREE llama.cpp · UD-IQ4_XS 6/6 613.81 49.06 4.5GB ik_llama.cpp · IQ4_KS 6/6 503.31 38.26 1.4GB ik_llama.cpp · UD-IQ4_XS 6/6 484.07 37.36 3.4GB ik_llama.cpp · UD-Q4_K_M 6/6 482.68 34.16 1.5GB Six codes spread from 12% to 100% of prompt depth. Addressability, not retrieval quality. Different sampler and reasoning settings from the figure above. Not directly comparable.
At 106,612 prompt tokens with reasoning at medium: every configuration retrieves all six codes, and the speed ordering changes.

All four retrieved 6/6, so this test did not separate them on addressability. It did separate them on performance.

With the working context loaded, mainline decoded at 49.06 tok/s against 38.26 tok/s for IQ4_KS, about 28% faster, while also retaining the lower VRAM footprint. The complete request finished in 3m06s instead of 4m02s.

That reversal is the main reason our recommendation is not based on the highest short-prompt TPS number.

Why we stayed at 131K context

Qwen3.8-27B supports a native 262,144-token context, but our benchmark used 131,072 with q8_0 KV. That was a choice about serving quality, not a model limit.

club-3090 has separately demonstrated that the same single-card UD-IQ4_XS configuration can boot at the full 262,144 tokens by dropping K/V cache to q4_0. Its own profile labels that path experimental and below its serving KV floor; the q8_0 serving-grade override is 131,072 tokens. club-3090 q4_0 262K single-card profile

That is the trade-off in practical terms: q4_0 KV buys the full 262K window on one 3090; q8_0 KV keeps the higher-fidelity cache and tops out at 131K on this single-card setup. For this benchmark, and for our production default, we stay on q8_0. Our validated working-context result is therefore the narrower claim we actually exercised: 106,612 prompt tokens, 6/6 retrieval.

MTP check

MTP acceptance was 70.0% for IQ4_KS, 68.9% for UD-IQ4_XS and 68.6% for UD-Q4_K_M across the ik runs. Those values are too close to explain the larger throughput spread, so MTP acceptance is not the main differentiator here.

What should you run?

General-purpose single-card default: mainline llama.cpp + UD-IQ4_XS. It gives up only a few percent on short-prompt decode, while winning prompt processing, VRAM headroom and the 106K working-context performance test.

Short-context decode first: ik_llama.cpp + ubergarm IQ4_KS. It produced the highest narrative and code decode figures in the short-prompt benchmark, but with about 3.0 GiB more peak VRAM and slower large-context processing.

ik + UD-IQ4_XS: useful if you specifically need ik_llama.cpp features, but there is no throughput case for it in these measurements because the same file ran faster on mainline.

ik + UD-Q4_K_M: the slowest decode configuration we tested. Its prefill was close to the other ik rows, so this is not a claim that it lost every metric. It is also a throughput result, not a quality judgement.

What this benchmark does not answer

We did not measure quantisation fidelity. There is no perplexity, KL divergence or controlled behavioural evaluation here. The short benchmark follows the harness sampler used for these measured throughput runs; it is not a quality evaluation. The files span about 0.6 BPW across their repeating layers, so a slower file could still preserve model behaviour better. A quality comparison needs a separate experiment.

This is also not a vLLM versus llama.cpp benchmark. Qwen recommends vLLM among its production serving engines, and club-3090 discussion #1037 measured Qwen3.8-27B on two RTX 3090s at 84.56 tok/s narrative and 109.11 tok/s code with vLLM INT8, versus 44.22 and 52.91 tok/s for Q8 llama.cpp. That is useful context, but it uses different hardware and precision, so we do not project it onto this one-card GGUF test. club-3090 discussion #1037

Reproducibility and deployment notes

The final measurements used one RTX 3090 with 24,576 MiB VRAM, NVIDIA driver 610.57.04, 131,072-token context, q8_0 K/V cache and MTP draft depth four. The tested serving images were:

  • mainline: ghcr.io/mostlygeek/llama-swap:unified-cuda at sha256:5cb8cc8221b802a62dbb051d3534d4bd0e88f38fcef2c93d04e7733740fd978a
  • ik: ghcr.io/ikawrakow/ik-llama-cpp:cu13-swap-4819 at sha256:aab468c5e4976109966df4a4e4af1d5269688017459d1852db9f3b0c5f3bfc44

Two revisions of the harness were in use across the session, and the two differ in what they capture: the later one adds the cache-busted prefill probes, the PCIe and VRAM captures and the acceptance capture. Every figure published here came from the later revision. Both are hosted verbatim in the Configs library under club-3090's Apache 2.0 licence, with digests, so a given figure can be checked against the exact code that produced it. For the maintained harness, use club-3090 rather than our copies.

The weight files were SHA256-pinned in the test record, and all four configurations are published as launcher scripts in the Configs library so the measured setups can be reproduced without translating prose back into flags. For agentic use we used the corrected Qwen Fixed Chat Templates. A current llama.cpp build is recommended; release b10434 added reasoning_effort passthrough to chat templates. We do not claim an exact upstream llama.cpp revision for the benchmark container because its exposed version uses the distributor's build scheme.

Configured capacity and validated working context are separate claims: the server was configured for 131,072 tokens, while our long-context test actually exercised 106,612 prompt tokens and retrieved all six planted values.

Coming next: the largest quant that fits

A note on where this series goes, not a fifth result. Everyone wants to run the biggest file their card will take, and so do we. On 24GB you have to be realistic about what that means.

We have since run unsloth UD-Q4_K_XL on ik_llama.cpp through the same 106,612-token prompt at 131,072 configured context and q8_0 KV, on 2 September 2026. It completes: 6/6 codes, 486.28 tok/s prefill, 31.78 tok/s decode, cold cache confirmed. It also peaks at 24,068 MiB of the card's 24,576, which leaves 508 MiB free, and it is the slowest decode we have measured on this model. Roughly two percent of the card spare is a very different proposition from the headroom any configuration above leaves, and it is the sharpest illustration we have of what "it fits at 131K" is actually claiming: one measured peak, on one prompt.

That deserves its own article rather than a row in this one, because the question it raises is operational, not comparative: how you run a file that tight day to day, what it costs you, and what the alternatives are for going higher on quality still, including vLLM. That is the next piece in this series. The comparison above is unaffected by it and stands as measured.

Sources

All benchmark figures, BPW values, VRAM measurements, image digests and file hashes in this article are from the 30 August 2026 test record included with the pack. The UD-Q4_K_XL figures in the closing note are from a separate run on 2 September 2026 and are not part of the comparison.

Something wrong here? Corrections are welcome at hello@devfactory.mu and will be noted on the article.