On this page
Abstract
I optimized the Vulkan decode path for Llama 2 7B Q4_0 on an AMD Radeon RX 6700 XT. The final configuration increased throughput from 75.23 to 85.71 tokens per second with flash attention disabled, a +13.93% gain with a 95% paired confidence interval of +13.66% to +14.19%. With flash attention enabled, throughput increased from 80.57 to 89.67 tokens per second, a +11.29% gain.
The implementation combines shape-specific quantized matrix-vector kernels, shared projection launches, fused activation preparation, fused post-QKV work, and attention kernels designed for Navi22. I measured the combined stack with three interleaved ABBA cycles and seven repetitions per invocation, producing 84 raw samples for each reported decode result. The repository contains the source patches, benchmark scripts, and raw measurements.
Focused scale · 70.00–90.00 tokens/s
- Control
- 75.23 tokens/s
- Optimized
- 85.71 tokens/s
Δ +10.48 tokens/s · gain +13.93% · 95% paired CI +13.66% to +14.19%
Motivation
I work in a lab where I train machine-learning models, and I wanted a local environment for smaller experiments and faster iteration. A local feedback loop is useful for testing an idea, inspecting model behavior, or debugging a pipeline before moving to a larger accelerator.
My local machine uses an RX 6700 XT. In my experience, running ML workloads on AMD consumer GPUs involves more tooling friction than using CUDA, and this card's Vulkan backend appeared to leave optimization headroom. Rather than treat that as fixed, I used the card as a systems optimization problem.
Autoregressive decode is a useful target because the model repeats a latency-sensitive graph for every generated token. Lowering the latency of that graph reduces total generation time across the response. The question was whether reorganizing llama.cpp's Vulkan execution for the RX 6700 XT's Navi22 architecture could improve that loop while preserving the model's output.
Approach & Methodology
The optimized candidate specialized the Q4_0 × Q8_1 matrix-vector path for Llama 2 7B decode. Its base Navi22 route uses wave64 integer-dot kernels with two output rows per workgroup, while the retained profile adds one-row specializations for the shared QKV group and the 4,096 × 11,008 down projection. The down-projection kernel preloads the next quantized weight block before accumulating the current block. For eligible graphs, the fused RMSNorm-plus-scale dispatch also writes the Q8_1 activation consumed by the next Q4_0 matrix-vector operation, eliminating the standalone Q8_1 quantization dispatch.
I then reduced graph-level overhead around the kernels. The candidate co-dispatches the three Q, K, and V projections that share one activation. It also evaluates the feed-forward gate and up projections together and writes their SwiGLU result directly. The recorded route audit for every candidate process reported 62 grouped launches—32 QKV and 30 feed-forward—covering 156 logical projections. A second fusion performs Q/K rotary-position encoding and the K/V cache writes in one dispatch for each layer.
Attention received separate shape-specific paths. With flash attention disabled, wave32 F16 kernels handle the exact QK and AV matrix-vector shapes used during single-token Llama 2 decode. With flash attention enabled, an exact-shape N=1, KV=256 masked-split route omits a fully masked split when applicable, performs the split reduction inline, and emits a Q8_1 sidecar for the next projection. It activated for 31 of 32 structural opportunities and belongs only to the separate +11.29% aggregate result, not the flash-attention-off headline result.
The retained optimization stack was benchmarked as a unit, so the +13.93% gain measures the combined implementation rather than the isolated contribution of one kernel. The routes default to off. On the validated RX 6700 XT/RADV target, nonmatching graph shapes retain the existing Vulkan paths; invalid flag combinations or unsupported devices are rejected rather than silently enabled. Before timing, the candidate passed graph, attention, route-coverage, token, logit, numerical, and shader-validation checks.
For performance measurement, A was the matched control and B was the optimized build. Each result used three independent cycles in fixed A1 B1 B2 A2 order, with seven repetitions inside every invocation. Interleaving the builds kept both versions in the same session and reduced sensitivity to temperature or clock drift. The paired gain is the mean of the three cycle ratios. Its 95% interval comes from 100,000 deterministic hierarchical-bootstrap replicates that resample cycles and then the observations within each invocation. Continuous benchmark values are displayed here to two decimal places, while the analysis uses the original unrounded measurements.
× 3 cycles
- reps / invocation
- 7
- samples / result
- 84
- thermal abort
- 95.00°C
- run power cap
- 174.00 W
A1 → B1 → B2 → A2 order, output checks, exclusive GPU lock, and automatic fan restoration after every run.
Results
With flash attention off, the matched control averaged 75.23 tokens per second and the optimized build averaged 85.71. The increase was +10.48 tokens per second, or +13.93%, with a 95% paired interval from +13.66% to +14.19%.
With flash attention on, throughput moved from 80.57 to 89.67 tokens per second. That is an increase of +9.10 tokens per second and a paired gain of +11.29%, with a 95% interval from +10.91% to +11.72%.
The earlier nine-patch profile measured +2.88% without flash attention and +3.97% with it. The later retained profile also included the shape-specific down-projection prefetch, fused feed-forward/SwiGLU execution, post-QKV fusion, and specialized attention work. Because these were separate full-stack campaigns rather than feature-by-feature ablations, the larger retained gains cannot be assigned to any one addition.
The appendix keeps the four decode rows in one compact table. The repository contains the full benchmark protocol, source patches, scripts, and raw data.
Benchmark data & reproducibility 4 decode rows
Each result contains 84 raw samples. The repository contains the full protocol, source, scripts, and raw measurements.
| profile | flash attention | control | optimized | gain / 95% CI |
|---|---|---|---|---|
| Retained a551 | off | 75.23 | 85.71 | +13.93% [+13.66%, +14.19%] |
| Retained a551 | on | 80.57 | 89.67 | +11.29% [+10.91%, +11.72%] |
| Nine-patch | off | 75.84 | 78.02 | +2.88% [+1.94%, +3.49%] |
| Nine-patch | on | 80.89 | 84.10 | +3.97% [+3.31%, +4.64%] |
Conclusion
This experiment began with a practical need for faster local model iteration and showed that reorganizing repeated decode work in llama.cpp could materially improve throughput on the RX 6700 XT. The retained stack combined shape-specific quantized matrix-vector kernels, grouped projections, and fused activation, cache, and attention-adjacent work; as a unit, it measured +13.93% faster decode without flash attention and +11.29% with it under the matched ABBA protocol.
The next step is to carry the experimental method, rather than the RX 6700 XT-specific code, to datacenter AI accelerators. I want to profile production-scale inference on NVIDIA H100 and AMD Instinct MI300X, the closest AMD datacenter-GPU counterpart to H100, then investigate low-precision kernels, KV-cache and HBM traffic, continuous batching, and multi-GPU communication. Component-level ablations would separate the contribution of each change before applying the most useful ideas to realistic model-serving workloads.