AdaptFM

AdaptFM @ ICML 2026 · From the organizers

Efficient LLM Challenge Results

Model: Qwen3.5-4B Hardware: 1× NVIDIA A10G (24 GB) Metric: mean latency speedup over baseline, 3 prompt lengths
Overview

The Efficient LLM Challenge, run as part of the AdaptFM workshop at ICML 2026, asked one focused question: how fast can you serve Qwen3.5-4B on a single NVIDIA A10G without giving up quality? We fixed the model, the hardware, and a quality floor on MMLU-Pro [14], IFEval [15], and GPQA-Diamond [16], then ranked entries purely on latency across short (64-token), medium (2,048-token), and long (8,192-token) prompts. Around 176 teams took part, of which 43 landed scored submissions on the leaderboard, with reported speedups spanning 1.0× to 7.745× and a median of 2.89×. This post walks through what the leading submissions did and why it worked. The short version: every fast entry leaned on speculative decoding [1][2], most of them on a trained block drafter (DFlash [6]), and the differences that ordered the field came from serving overhead, long-context prefill, and how weights were quantized [10][11].

1The task

We built the challenge to remove the variables that usually muddy an inference comparison: one model (Qwen3.5-4B), one GPU (NVIDIA A10G, 24 GB), one objective (minimize latency), and one constraint (preserve quality). Quality is a pass/fail gate on three public benchmarks, so the ranking comes down to latency alone. Latency is the mean speedup over an unoptimized baseline across three prompt-length regimes.

Table 1. Quality gates and latency regimes
Quality gateThresholdLatency regimePrompt toksBaselinePrimary cost
MMLU-Pro [14]≥ 0.621Short642,582 msoverhead + decode
IFEval [15]≥ 0.814Medium2,0485,441 msprefill + decode
GPQA-Diamond [16]≥ 0.630Long8,1926,576 msprefill-bound, KV
Table 2. The field at a glance
MetricValueMetricValue
Teams participating~176≥ 2×32
Scored submissions43≥ 3×17
Valid (non-zero)41≥ 5×10
Faster than baseline39Median speedup2.89×
Scored 0.000×2Maximum speedup7.745×
Top-2 spread< 1%  

The two fastest submissions (7.745× and 7.708×) finished within one percent of each other. We have organized the rest of this post by technique rather than by team.

2Where the time goes

Most of the results follow from one property of the workload: a 4B model in fp16, decoding one token per step, is memory-bandwidth-bound rather than compute-bound, the regime Pope et al. analyze in roofline terms for transformer inference [13]. Each decode step reads the full weight set from HBM to produce a single token, and the A10G's arithmetic units sit largely idle. Three consequences shaped what the leaders built:

  • For short and medium prompts the dominant cost is decode, so the main lever is producing more tokens per weight read.
  • The long bucket is a separate regime: at 8,192 tokens, prefill compute and KV-cache reads dominate, which is why the long bucket produced the widest spread across teams.
  • Weight quantization lowers bandwidth but addresses that bound less directly than the decode-side technique in §3; we cover its effect in §5.

3Speculative decoding did the heavy lifting

Speculative decoding [1][2] (surveyed in [3]) uses a cheap drafter to propose K tokens that the target verifies in a single forward pass, keeping the longest prefix the target would itself have produced. Under greedy verification the output distribution is provably unchanged [1], so it cuts latency without touching the quality gates. The leaders relied on exactly this property: the top submission serves at temperature=0, and one team's config notes that its drafter-context cap is safe "because spec-decode at greedy is bit-exact regardless of draft quality."

Every leading submission used speculative decoding. Where they differed was the drafter, and that was the main axis of variation. Table 3 maps the drafting families to prior work and to where they landed.

Table 3. Drafting families among the leading submissions
FamilyMechanismPrior workTeams
DFlash (trained block drafter)small model predicts a block of K tokens from target features in one pass[6]; lineage: EAGLE [7], Medusa [8], block diffusion [9]7, incl. #1
Suffix decodingdrafts from a suffix tree over recent contextSuffixDecoding / Arctic Inference [5]1 (#2)
N-gram / prompt-lookupcopies spans from the promptprompt-lookup decoding [4]1 (#5)

Trained block drafters (DFlash [6]) accounted for seven of the leading entries. DFlash sits in the feature-conditioned drafting lineage of EAGLE [7] and Medusa [8] and the block-parallel prediction of block diffusion models [9]; its drafts come from a model trained on the target's hidden states rather than from surface patterns, which is why it accepts more tokens on generated text. The winning submission is among the simplest instances: a 4-bit (W4A16 GPTQ [10], group size 128) Qwen3.5-4B target with the lm_head untied and quantized to 8-bit, paired with a full-precision DFlash drafter at a fixed K=15 (a roughly 1 GB, 5-layer predictor tapping target layers [1,8,15,22,29]), in about 290 lines around vLLM [11].

The runner-up is the only non-DFlash entry near the top, and its main addition is a correctness fix. A patched suffix decoder [5] pads every draft to a constant K=20 (short drafts filled with -1 sentinels). Its comments trace this to two vLLM issues: #39273, where empty drafts left requests in stale Gated-DeltaNet [12] state and pushed GPQA as low as 0.27, and #39809, where variable-length drafts drift the hybrid KV-cache block count under prefix caching and crash the engine. The padding stays lossless because a padded position is accepted only when it matches the verifier's own greedy token. A good share of the engineering near the top of the board went into engine correctness under speculation rather than into the speculation algorithm itself.

N-gram drafting [4] (#5, 6.118×) uses no auxiliary model and is competitive on prompt-heavy inputs, but it accepts fewer tokens on generated output, which shows up as the highest long-bucket latency among its neighbors (2,891 ms). Team #6 switched methods mid-competition, from suffix decoding at K=128 to DFlash at K=15. That tracks the general pattern: draft cost and rejection both grow with K while acceptance does not, so a moderate K with a trained drafter tends to beat a large speculative window. Reported K values cluster in 12–23 and are fixed per submission rather than varied by prompt length; the winner serves a single K=15.

4The long bucket separated the field

The short bucket kept the leaders within 225–390 ms, since there is little prefill to optimize. The long bucket (8,192 tokens) is where the cost shifts from decode bandwidth to prefill compute and KV reads, and it produced the widest spread. Dynamic sparse-attention prefill [17] is an obvious lever here, though none of the leading entries reached for it. The two that mattered in practice were:

  • Prefix caching. The benchmark reuses a long shared prompt, so leaving prefix caching [11] on (as the runner-up did) cuts per-request prefill, which shows in its low medium (620 ms) and long (1,833 ms) latencies. The catch is that prefix caching interacts poorly with the hybrid GDN/Mamba KV cache [12] under speculation, so it paid off only after a patch (#2) or was turned off outright (#3, whose config comments report MMLU-Pro falling from 0.686 to 0.419 with it on).
  • Drafter KV windowing. A DFlash drafter's KV grows with prompt length, raising draft-step cost on long prompts. Team #3 bounds the drafter's attention to a 1,024-token sliding window [18], applied at run time through a vLLM plugin, which stays lossless at greedy.
What we sawLong-bucket latency tracks final rank closely: the strongest drafter (#1, 1,912 ms) is well ahead of n-gram drafting (#5, 2,891 ms), and that gap accounts for much of the distance between the 5× and 7× groups. This is a correlation across the field, not a controlled ablation.

5Quantization was common, but not the deciding lever

Each of the top three quantized the target to 4 bits, yet quantization did not by itself order the field. A half-precision entry (#4, bf16, 6.197×) finished ahead of several 4-bit submissions below it, and the two entries just under it (#7, #8) serve full-precision targets. Read it as a component that shows up near the top rather than the thing that separated the field, since both the fastest entry and several mid-pack ones are 4-bit. The choices we saw:

  • Target quantization. The #1 target is W4A16 GPTQ [10] (group size 128) with the lm_head untied and quantized to 8-bit; #2 serves a 4-bit AWQ [19] target through the awq_marlin [20] kernel; #3 an AWQ [19] W4A16 target that was fine-tuned with quantization-aware distillation to recover the accuracy lost to 4-bit weights, rather than applying quantization as an afterthought; #5 a W4A16 compressed-tensors target (group size 32, keeping the first two and last two layers and lm_head in fp16); and #9 an int8 W8A8 target with a W8A16 lm_head. Target quantization mainly helps the bandwidth-bound short and medium buckets.
  • Drafter quantization. Team #3 pairs its quantized target with a 4-bit (W4A16 GPTQ [10], group size 128, 277 MB) drafter that fits in cache, turning each draft step into an INT4 GEMM and speeding up the speculative path without changing the verifier that determines output. The winner, by contrast, keeps its drafter at full precision.

One detail worth calling out: 4-bit quantization is only free on the leaderboard if it clears the quality gate, and it does not always do so on its own. Team #3 treated this as a training problem rather than a serving flag, using quantization-aware distillation to fine-tune the W4A16 target and recover the accuracy that naive weight quantization would have lost. That is the difference between quantization as a bandwidth trick and quantization as a co-designed part of the model, and it is what let a fully 4-bit target hold the gate at 6.978×.

6The rest was overhead

The distance between the simplest competitive entry and the winner bounds what the remaining engineering was worth. Team #4 ran unmodified vLLM [11] with a DFlash drafter at K=12 and a bf16 target, in about 25 lines of configuration, and reported 6.197×. Closing the gap to 7.745× came from a mix of overhead reduction, a quantized target, and a larger K rather than from a new algorithm:

  • Lean serving paths. The winner adds a set of LOSSLESS_* engine flags (no-clone tensor handling, speculative and GDN metadata caches), each shaving per-step cost without changing outputs, alongside a 4-bit target and K=15 (versus #4's bf16 target and K=12). These factors are not cleanly separable across the two entries.
  • Kernel fusion. Team #7 (DFlash K=18) enables two fused Gated-DeltaNet [12] kernel flags, one for prefill convolution-prep and one for the speculative-decode step, inside a larger set of GDN and DFlash tuning flags; its long-bucket latency (2,198 ms) beats several teams ranked above it.
  • CUDA-graph capture. Teams #8 and #9 capture decode into CUDA graphs sized to the speculative block, trimming per-step launch latency, which helps the launch-bound short bucket most.

These are incremental gains that reorder the top tier rather than decide entry into it.

7Simple recipes went a long way

For buildersSome of the most ambitious engineering in the field came from teams that did not finish at the very top, and it deserves a callout. Team #9 built one of the deepest custom stacks of anyone (SGLang [21], DFlash K=16, an int8 W8A8 target with a W8A16 lm_head, piecewise CUDA graphs, and hand-written static-config Triton [22] kernels, with a W8A8/W8A16 phase-routing path held in reserve) and turned in a strong 5.056×. That it landed just behind a roughly 25-line stock-vLLM entry at #4 (6.197×) is one of the more instructive results of the challenge: on this particular workload a lean "engine plus trained drafter" recipe was hard to beat, since each extra component also adds launch latency and more chances for acceptance to slip. We would encourage builders to read this as an invitation rather than a caution: working bottom-up and trying things end to end is exactly how the next round of wins gets found, and several of these techniques should pay off on other hardware and workloads even when they did not top this particular leaderboard. This is an observation about this field, not a general law.

8The leading submissions

Table 4. The leading submissions: speedup, per-bucket median latency, and defining technique
#TeamMeanShortMedLongDefining technique
1AFM-e4e9muaf7.745×2256521,912W4A16 target (8-bit lm_head) + full-precision DFlash K=15, lean serving path
2AFM-6j6duhm67.708×2406201,833Suffix K=20, constant-K patch, AWQ target, prefix caching on
3AFM-k5984d3s6.978×2327822,313AWQ W4A16 target (QAD-tuned) + W4A16 DFlash K=15 drafter, 1,024-token drafter window
4AFM-v8ws998a6.197×2787942,691Stock vLLM + DFlash K=12, bf16 target (~25-line config)
5AFM-gv7e2ebx6.118×3107032,891N-gram K=23 + W4A16 target (compressed-tensors, group size 32)
6AFM-qd39x2e65.366×3458812,684Migrated suffix K=128 to DFlash K=15
7AFM-cpykb5cu5.090×3879722,198DFlash K=18 + fused Gated-DeltaNet kernels, fp16 target
8AFM-34qt8st65.084×3389883,128DFlash K=15, tuned CUDA-graph capture sizes
9AFM-v4we4h8u5.056×3899412,403SGLang + DFlash K=16, W8A8 target (W8A16 lm_head)

Latencies are median milliseconds per prompt bucket (Short 64-token / Medium 2,048-token / Long 8,192-token).

9What we took away

Ordering the techniques by how much they seemed to move the reported speedups:

  1. Speculative decoding with a trained drafter [1][6]. Every leading entry used it; DFlash drafters ranked highest, suffix decoding was comparable on repetitive prompts, and n-gram trailed on generated output.
  2. A lean, lossless serving path. This is most of the distance between the stock entry (6.2×) and the winner (7.7×): greedy verification and a handful of per-step overhead flags.
  3. Long-context prefill handling. Prefix caching and drafter KV windowing were the main levers on the 8K bucket, which tracks rank.
  4. Weight quantization [10][19]. A 4-bit target is common among the leaders (all of the top three) but not universal and does not track rank cleanly; quantizing the drafter too (lossless under greedy verification) is an option some teams took.
  5. Kernel fusion and CUDA graphs. Smaller gains that reorder the top tier.

The recurring lesson across the board: on a bandwidth-bound 4B model, a good trained drafter plus a clean serving path gets you most of the way, and the more advanced additions refine the ordering among the leaders. Both halves of that are worth pursuing, and we would love to see the more ambitious techniques pushed further on future workloads.

§References

  1. Leviathan, Kalman, Matias. "Fast Inference from Transformers via Speculative Decoding." ICML 2023. arXiv:2211.17192.
  2. Chen et al. "Accelerating Large Language Model Decoding with Speculative Sampling." 2023. arXiv:2302.01318.
  3. Xia et al. "Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding." ACL Findings 2024. arXiv:2401.07851.
  4. Saxena. "Prompt Lookup Decoding." 2023. github.com/apoorvumang/prompt-lookup-decoding.
  5. Oliaro et al. "SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications." 2024. arXiv:2411.04975. Model-free, suffix-tree based; open-sourced via Snowflake Arctic Inference, github.com/snowflakedb/ArcticInference.
  6. DFlash: "Block Diffusion for Flash Speculative Decoding." Reference implementation github.com/z-lab/dflash (MIT; draft models and vLLM/SGLang/Transformers integrations), with an accompanying developer technical blog.
  7. Li et al. "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty." ICML 2024. arXiv:2401.15077; github.com/SafeAILab/EAGLE.
  8. Cai et al. "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads." 2024. arXiv:2401.10774; github.com/FasterDecoding/Medusa.
  9. Arriola et al. "Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models." ICLR 2025. arXiv:2503.09573.
  10. Frantar et al. "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." ICLR 2023. arXiv:2210.17323; github.com/IST-DASLab/gptq.
  11. Kwon et al. "Efficient Memory Management for LLM Serving with PagedAttention" (vLLM). SOSP 2023. arXiv:2309.06180; github.com/vllm-project/vllm.
  12. Yang et al. "Gated Delta Networks: Improving Mamba2 with Delta Rule." 2024. arXiv:2412.06464. Mamba: Gu and Dao, arXiv:2312.00752.
  13. Pope et al. "Efficiently Scaling Transformer Inference." MLSys 2023. arXiv:2211.05102.
  14. Wang et al. "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark." NeurIPS 2024. arXiv:2406.01574.
  15. Zhou et al. "Instruction-Following Evaluation for Large Language Models (IFEval)." 2023. arXiv:2311.07911.
  16. Rein et al. "GPQA: A Graduate-Level Google-Proof Q&A Benchmark." COLM 2024. arXiv:2311.12022.
  17. Jiang et al. "MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention." NeurIPS 2024. arXiv:2407.02490.
  18. Beltagy et al. "Longformer: The Long-Document Transformer" (sliding-window attention). 2020. arXiv:2004.05150.
  19. Lin et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." MLSys 2024. arXiv:2306.00978; github.com/mit-han-lab/llm-awq.
  20. MARLIN ("Mixed Auto-Regressive Linear" kernel): an optimized FP16×INT4 (W4A16) GPU matmul kernel for LLM inference, IST-DASLab. github.com/IST-DASLab/marlin.
  21. Zheng et al. "SGLang: Efficient Execution of Structured Language Model Programs." NeurIPS 2024. arXiv:2312.07104; github.com/sgl-project/sglang.
  22. Tillet et al. "Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations." MAPL 2019; github.com/triton-lang/triton.