llama.cpp Cuts Redundant Scans, Lifting One Long-Context Test 51%

llama.cpp release b10707, published at 03:03 UTC on August 31, removes a surprisingly large amount of bookkeeping from one long-context generation path. The n-gram lookup used for speculative decoding walked the project’s maximum 256 sequence slots for every occupied KV-cache cell, even though a cell normally belongs to only one sequence. Pull request 28011 keeps track of the sequences actually present and stops as soon as all of them have been seen. The traversal order and callback behavior stay the same; the patch changes how soon the scan can finish, not the model’s answers.

The reported result is material and deliberately narrow. Using an RTX PRO 6000, flash attention and a UD-Q4_K_XL quantization of Qwen3.8-Flash-Next, the contributor measured generation at 74.3 tokens per second versus 56.3 at a 55,000-token context, a 32.0% increase. At 132,000 tokens, generation rose from 33.6 to 50.9 tokens per second, or 51.5%. Prompt processing did not change. The gain grows as more KV cells are occupied, and the author says it is effectively invisible on short prompts. These are warm project benchmarks on one GPU, model and quantization—not an independent fleet study and not a claim that every llama.cpp workload becomes 51% faster.

An earlier release in the same 48-hour cycle attacks prefill rather than generation. Release b10704 includes a CUDA change that lets any expert count use llama.cpp’s faster mm_ids_helper path. The old gate accepted expert counts that divide a 32-lane warp, plus a hard-coded exception for six; Qwen3.8-Flash-Next uses ten and therefore fell to a slower kernel. The patch pads the grouping width to the next power of two while still dispatching ten experts. In the contributor’s revised test on the same RTX PRO 6000 at a 55,000-token context, prompt processing increased from 2,139 to 2,351 tokens per second, or 9.9%. Generation was unaffected, and the implementation has only been exercised on CUDA’s 32-wide warps.

The two results should not be added together. One removes redundant sequence scanning during generation; the other changes expert routing during prompt processing. Both were measured on a named model and accelerator, and neither establishes energy use, concurrency under production load, tail latency or total serving cost. What they do establish is an operational possibility: software can raise useful output per installed GPU-hour without changing the accelerator. A capacity planner should reproduce the result on the exact model, context distribution and batch policy before treating it as available capacity.

China’s August purchasing managers’ survey, released by the National Bureau of Statistics on August 31, supplies the demand and financial counterweight. Manufacturing PMI rose 0.6 point to 49.8, still below the 50 expansion line. Production reached 50.4 and new orders 50.6, while new export orders were 50.1. Large manufacturers returned to expansion at 50.6, but medium firms were at 49.4 and small firms at 47.9. The improvement is broad enough to matter, yet not broad enough to call the factory sector expanding as a whole.

The price split is especially relevant to margins. The input-price index jumped 3.4 points to 56.6, while the factory-gate price index rose 2.6 points to 50.4. The bureau’s interpretation attributes part of that movement to higher crude-oil and non-ferrous-metal prices; in non-ferrous smelting and processing, both price measures exceeded 60. When input prices rise faster and from a higher level than selling prices, companies may not retain every efficiency gain. Better inference kernels can lower the compute needed for a task, but they do not neutralize energy, hardware, materials or financing costs elsewhere in the chain.

Technology-linked manufacturing was stronger than the aggregate. The official interpretation says production and new orders in electrical machinery and in computers, communications and electronic equipment were both above 53. Equipment manufacturing PMI was 51.4 and high-tech manufacturing 52.9, both in expansion. Consumer-goods manufacturing was 49.0 and energy-intensive manufacturing 47.9. That divergence supports a targeted technology-demand signal, not a claim that AI spending is lifting the entire economy. The PMI measures whether more respondents report improvement than deterioration; it does not measure revenue growth, accelerator purchases or model usage.

Services show the same uneven map. Overall non-manufacturing business activity held at 49.0; construction was 46.9 and services 49.3. Postal services, telecommunications and broadcasting, and internet, software and information-technology services each registered above 55, according to the bureau, while wholesale, retail and capital-market services were below 50. Non-manufacturing new orders fell to 44.1 and employment remained at 45.4. The composite output index improved 0.2 point to 49.5, leaving total surveyed activity below expansion despite the strength of selected digital categories.

Two fresh news reports help frame rather than replace the primary record. Xinhua highlighted that 16 of the 21 manufacturing industries improved from July and that procurement volume rose to 50.5. The Associated Press noted the headline reading was better than economists had expected, while still describing overall manufacturing as contracting. Neither observation changes what the survey can prove. The seasonally adjusted sample covers 3,200 manufacturing companies across 31 industries and 4,300 non-manufacturing companies across 43 industries; it captures activity breadth, not the dollar value of demand.

The operational and financial tests now divide cleanly. Engineering teams should reproduce both patches separately, measure prompt processing and generation by context bucket, and add power, memory, concurrent-request and p95 latency data. Capacity and finance teams should convert only verified production gains into GPU-hour or cash-cost assumptions, then ask whether higher utilization consumes the saving. In China’s data, the next confirmation would be manufacturing, services and the composite index crossing 50 together, with new orders and employment improving without a widening input-price squeeze.

The strongest conclusion is smaller than the headline number. llama.cpp has shown that long contexts can hide avoidable work, with a 51.5% generation gain in one 132,000-token test and a separate 9.9% prefill gain. China’s survey shows high-tech and digital services expanding inside an economy that has not regained broad-based growth, while input prices are moving faster than output prices. More useful work per GPU-hour is a credible engineering and margin target; proving that it survives production workloads and meets durable paying demand is the harder test ahead.