mirror of
https://github.com/ggml-org/whisper.cpp.git
synced 2026-10-18 21:55:18 +02:00
140e57a4e20af04e8e00930798474a4707f0ee47
281
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Georgi Gerganov
|
4dd48dde35 |
metal : add sparse FA (llama/28098)
* metal : support n_kv_max sparse mask hint in flash attention vec kernel
- add kernel_flash_attn_ext_vec_idx: compacts finite mask entries into
a per-row index list (Hillis-Steele scan, one threadgroup per row)
- extend vec FA kernel with optional sparse index gathering (FC slot 5)
- add host-side gate: sparse path when n_kv_max > 0, mask present,
supported head sizes / KV types, n_kv_max <= 4096
- new buffer region extra_idx for the index list
- pipeline getter extended with has_sparse param
- add test cases: head sizes, quant types, nb>1, nr23 variants,
sinks, ALiBi, softcap, permute, v_view_of_k, no-mask fallback
Note: multi-row (nb*nr23[1] > 1) cases still failing - rid mapping
in the store phase needs revisiting for the sparse path.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : fix sparse flash attention row addressing
- kernel_flash_attn_ext_vec_idx: mask param is half* but nb31 is a byte
stride, so the per-row mask offset was scaled by 2x; cast to char*
before applying the byte strides
- kernel_flash_attn_ext_vec: sparse pidx param is char* so the per-row
element offset was under-scaled by sizeof(int); scale it by sizeof(int)
to get the correct byte offset
- fixes the multi-row (nb*nr23[1] > 1) sparse flash attention failures
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* cont : use sparse vec FA for prefill
* metal : single-pass flash attention sparse index compaction
The idx kernel previously read the mask row twice: once to count the finite
entries (for the prefix scan) and again to recover their positions. Since the
kernel is memory-bound, this doubled the mask traffic.
Keep the finite positions in a per-thread register array during the count
pass and write them out directly, avoiding the second mask read. A dense
mask with more than NLOCAL finite entries in a slice falls back to re-reading
the mask to write the remaining positions.
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* tests : add perf cases for sparse flash attention prefill
Measure the sparse vec FA kernel across KV sizes, n_kv_max hints and batch
sizes. Run with:
./build/bin/test-backend-ops -b MTL0 -o FLASH_ATTN_EXT -p "n_kv_max=[1-9]" perf
Assisted-by: pi:llama.cpp/DeepSeek-v4-0731
* qwen4 : enable sparse attention
* cont : adjust nsg
* cont : sync test-backend-ops
* cont : disable Qwen4 for now
* cont : clean-up + tests
|
||
|
Georgi Gerganov
|
d55d345e69 |
metal : fix glu dispatch with ne00 = 1 (llama/28306)
* metal : fix glu dispatch with ne00 = 1 * tests : disable ill-defined tests |
||
|
Isaac
|
1bdda1e366 | metal : add fa-vec tunings for M3 (llama/28236) | ||
|
|
3a1c7d6b6f |
metal : fix memory query under low-memory conditions (llama/27701)
* metal: Fix memory query under low-memory conditions * Simply variable name Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Write it even shorter Co-authored-by: Niklas Wenzel <dev@nikwen.de> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Niklas Wenzel <dev@nikwen.de> |
||
|
Jhen-Jie Hong
|
fcc2feee2f | metal : add metallib build support for xcframework (llama/28163) | ||
|
Lukasz Stolcman
|
5f07f856c5 |
metal : add fa-vec tuning for M2 Pro (llama/28122)
* metal: add fa-vec tuning for M2 Pro * metal : update fa-vec tuning for M2 Pro with new dtypes |
||
|
Jhen-Jie Hong
|
8cca1a3616 | metal : add fa-vec tunings for A18 Pro (MacBook Neo) (llama/28152) | ||
|
Niklas WenzelandYiChen Lv
|
a245a8f48f |
metal : fix more leaks due to missing autoreleasepools (llama/27883)
* metal : fix more leaks due to missing autoreleasepools * metal : rename variable * metal : fix another missing pool warning Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> --------- Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> |
||
|
Georgi Gerganov
|
870db2afa1 |
metal : add fa-vec tuning for M2 Max (llama/28015)
Rows for M2 Max (30 GPU cores) collected with 'ggml-metal-tuning fa-vec --dtype f16,q8_0', pasted into fa_vec_tuned_table. ref: https://github.com/ggml-org/llama.cpp/discussions/27668#discussioncomment-18205786 Assisted-by: pi:llama.cpp/Qwen3.8-27B |
||
|
James Francis
|
5032008bc8 |
metal: enable Metal 4.0 tensor API on M5+/A19+ (llama/27461)
* metal : request Metal 4.0 language version for the tensor API * metal : load the tensor API kernels from a separate metallib * tests : add external-metallib tensor API regression test * metal : fix metallib build order for the tensor API kernels |
||
|
Buğra Özgürsoy
|
8e54c659b5 |
metal : add fa-vec tunings for M1 Ultra (llama/28088)
* metal : add fa-vec tunings for M1 Ultra * metal : move M1 Ultra tunings after M1 Max section * metal : remove duplicate blank line |
||
|
Georgi Gerganov
|
dbc40efce8 |
metal : add concat support for quantized types (llama/28116)
Assisted-by: pi:llama.cpp/DeepSeek-V4-Flash-0731 |
||
|
Georgi Gerganov
|
c6934d0fcf |
metal : add top-k radix implementation (llama/28073)
Assisted-by: DeepSeek-v4-Flash-0731 |
||
|
Niklas Wenzel
|
7614a4c139 | metal : add fa-vec tunings for M1 (llama/28078) | ||
|
codemonkey
|
e5c96ca45d | metal : add remaining Q4_1/Q5_0/Q5_1 fa-vec tunings for M2 (llama/28017) | ||
|
Aman Gupta
|
e9583f075a |
ggml: add SWIGLU_CLAMP (llama/27930)
* ggml: add SWIGLU_CLAMP * add vulkan shader |
||
|
Nils Gladitz
|
1e0f382573 | metal: add fa-vec tunings for M3 Ultra (llama/27999) | ||
|
Daya Adianto
|
b66593ef1b |
metal : Add fa-vec tuning for M3 Pro (llama/27963)
Related issue: #27668 |
||
|
QuintinShaw
|
c68f20555a |
metal : fix null-pipeline crash for F16 src1 mul_mat/mul_mat_id (llama/25648)
* metal : fail closed on mul_mat shapes with missing F16 kernels * metal : abort on nil pipeline in encoder_set_pipeline * metal : address review comments * metal : share mul_mat mm dispatch with supports_op |
||
|
Aman Gupta
|
c969c68b54 |
ggml: allow passing alloc dependencies in graph_optimize (llama/27301)
* ggml: allow passing alloc dependencies in graph_optimize * add alloc dep tests * add TODO about using flat array |
||
|
codemonkey
|
b33bbc5c4e | metal : add fa-vec tunings for M2 (llama/27940) | ||
|
Georgi Gerganov
|
285f1ffd99 |
metal : assert shared memory padding (llama/27951)
* metal : assert shared memory padding * cont : add ref |
||
|
Niklas Wenzel
|
308fa4f8a2 | metal : add remaining fa-vec tunings for M4 Pro (llama/27915) | ||
|
Jhen-Jie Hong
|
590fe18902 | metal : add fa-vec tunings for M1 Max (llama/27932) | ||
|
Tekin ErtekinandGeorgi Gerganov
|
caea96f6ce |
ggml : fix conv_transpose_2d for multiple batches (llama/26132)
* ggml : fix conv_transpose_2d for multiple batches ggml_compute_forward_conv_transpose_2d_impl only computed the first batch (ne[3] of the destination); every batch after the first was left as zero. Both the src1 permutation and the main compute loop now iterate over the batch dimension, and the work buffer size in ggml_graph_plan is scaled by the src1 batch count so the extra permuted batches fit. A multi-batch test case is added to test-backend-ops. Fixes ggml-org/ggml#1448 * metal : fix conv_transpose_2d for multiple batches The kernel only computed batch 0 of the input (src1->ne[3]); every output batch after the first was left as zero, so multi-batch conv_transpose_2d results diverged from the CPU reference. The grid now covers all batches (OW x OH x OC x N), the kernel decodes the batch from the grid z coordinate and offsets both the input and destination indices accordingly. nb3 is passed in the kernel args. Assisted-by: pi:llama.cpp/Qwen3.8-27B --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |
||
|
StrongtutandStrongtut
|
0a150873ef |
metal : add fa-vec tunings for M4 (llama/27875)
This adds fa_vec_tuned_table records for Apple M4 to ggml-metal-tuning.cpp. Includes F16, Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0. (M4, 10 GPU Cores) Co-authored-by: Strongtut <8432058+Strongtut@users.noreply.github.com> |
||
|
Georgi Gerganov
|
530e3f4834 |
metal : add fa-vec tunings for M3 Max, M5 and M5 Pro (llama/27863)
* metal : add fa-vec tunings for M5 This is a followup contribution to efeda76b948f59ee52ea20db640bc4cf3dfe8ac1 as requested in https://github.com/ggml-org/llama.cpp/discussions/27668 to add support for additional Apple GPUs. I generated this output using the provided instructions: ```sh git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build -DGGML_METAL=ON cmake --build build --target ggml-metal-tuning -j ./build/bin/ggml-metal-tuning fa-vec --dtype f16,q8_0 > fa_vec_rows.txt 2> fa_vec_sweep.log ``` This ran on a machine with Apple M5. Assisted-by: pi:llama.cpp/Qwen3.8-27B * metal : add fa-vec tunings for M5 Pro This adds fa_vec_tuned_table records for Apple M5 Pro to ggml-metal-tuning.cpp. Contributed by SerayaEryn in https://github.com/ggml-org/llama.cpp/discussions/27668#discussioncomment-18157544 (F16, Q4_0, Q8_0; M5 Pro, 20 GPU cores). Assisted-by: pi:llama.cpp/Qwen3.8-27B * metal : add fa-vec tunings for M3 Max This adds fa_vec_tuned_table records for Apple M3 Max to ggml-metal-tuning.cpp. Contributed by TeeAaTeeUu in https://github.com/ggml-org/llama.cpp/discussions/27668#discussioncomment-18175220 (F16, Q8_0; M3 Max, MacBook Pro 64GB, low power mode). Assisted-by: pi:llama.cpp/Qwen3.8-27B * cont : whitespaces |
||
|
Brad Smith
|
ff38b98e5a |
metal : add fa-vec tunings for M4 Pro (llama/27824)
This is a followup contribution to efeda76b948f59ee52ea20db640bc4cf3dfe8ac1 as requested in https://github.com/ggml-org/llama.cpp/discussions/27668 to add support for additional Apple GPUs. I generated this output using the provided instructions: ```sh git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build -DGGML_METAL=ON cmake --build build --target ggml-metal-tuning -j ./build/bin/ggml-metal-tuning fa-vec --dtype f16,q8_0 > fa_vec_rows.txt 2> fa_vec_sweep.log ``` This ran on a MacBook Pro (14-inch, Nov 2024) with Apple M4 Pro. The `ggml-metal-tuning` command completed successfully in 1h 13m 1s with no other notable load on the system. |
||
|
Niklas Wenzel
|
a5db1d66f0 | metal : fix memory leaks due to missing autoreleasepools (llama/27758) | ||
|
|
8c0adb05fe |
ggml-metal: add chunked SSD MMA for Mamba-2 prefill optimization (llama/26647)
* metal: WIP chunked SSD SSM_SCAN kernels for multi-token prefill * metal: drop scalar SSD path; MMA + sequential tail * drop WIP ssm scan test noise * remove state_from_dst and rename CS and NSG constants * remove unrelated added whitespace padding * added clarity to mma_tokens calculation * added clarity to use_mma bool checks * added comments to metal ssd op constants for clarity * reserve K tokens for sequential kernel rollback snapshots * reset concurrency between mma and seq tail * remove print args no longer used * fixed comment to no longer point to specific line * add FC_SSM_SCAN so seq path skips token offlset unless it's mma tail * added changes to new ssm.metal for rebase after ggml-metal.metal refactor * specialize ssm_scan tail with a template instead of a function constant --------- Co-authored-by: dpantaleoni <dominikpantaleoni@gmail.com> Co-authored-by: forforever73 <690105611@qq.com> |
||
|
Yuri KhrustalevandGeorgi Gerganov
|
be12d39a17 |
metal : null-check buffer alloc to fix OOM crash (llama/25371)
* metal : null-check ggml_metal_buffer_init result to avoid OOM crash ggml_backend_metal_buffer_type_alloc_buffer used the result of ggml_metal_buffer_init without checking for NULL. ggml_metal_buffer_init returns NULL when the underlying Metal allocation fails (e.g. an out-of-memory condition), and the following ggml_metal_buffer_is_shared(res) call dereferences it, turning a recoverable allocation failure into a hard crash (EXC_BAD_ACCESS). This is easy to hit on memory-constrained devices such as iOS when a model/context exceeds the available Metal budget. Log the failure using the existing GGML_LOG_ERROR convention and return NULL so the allocator surfaces a diagnosable error up the stack instead of crashing. * cont : fix log --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |
||
|
YiChen LvandGeorgi Gerganov
|
fa3b87c188 |
metal : per-device tuned (Q, NE) for flash-attn vec (llama/26570)
* metal : per-device tuned (Q, NE) for flash-attn vec (llama/25750) * rebase Q-generic FA vec body from 01dc93607 (llama/23114) * add 53 f16 (Q,NE) flash-attn vec instantiations (vec 80 -> 133) * add FA vec (Q,NE) tuning table + dispatch wiring + SMEM cap fallback * add FA vec (Q,NE) perf sweep * fill tuning result * fold family table into a per-family representative SKU * refactor tuning result format * extend FA vec tuning to quantized KV caches * sync fa vec tuner bucketing with runtime, use pointwise tuning regret * update tuned table * format and cleanup * prefix fa_vec tuning procs with ggml_backend_metal_tuning_, drop unused fa_vec_override_active * add device id -> token lookup for the offline tuning tool * add ggml-metal-tuning skeleton * add op-agnostic perf cell + median timing for the tuner * add FA-vec graph build + tensor init to the tuner * tools : add FA-vec (Q,NE) sweep, compression and table emit * cool down and re-measure the dirty window on thermal drift * test-backend-ops : replace the FA vec tune mode with a bounded (Q,NE) slice * tools : document the Metal tuner, point the table comment at it * abort on unknown KV type, single-source fa_vec_legal_ne * cleanup * honor -o in the FA vec (Q,NE) slice * retune FA-vec (Q, NE) under a pointwise no-harm gate * cont : add fa-vec tunings for M1 Pro, M2 Ultra, M5 Max --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |
||
|
YiChen Lv
|
b15d31d0fd | metal: per-op source split + parallel compile (llama/26561) | ||
|
Georgi Gerganov
|
aa25d33f58 |
ggml : shorten virtual device naming in CUDA and Metal (llama/27608)
* ggml : shorten virtual device naming in CUDA and Metal Assisted-by: llama.cpp:DeepSeek-V4-Flash-0731 * ggml-metal : build device description at init Assisted-by: llama.cpp:DeepSeek-V4-Flash-0731 * cont : naming |
||
|
Georgi Gerganov
|
d68216aa3e |
metal : clamp K extent in tensor API mat-mat kernel for K not a multiple of 32 (llama/27450)
The Tensor API mat-mat path of kernel_mul_mm (GGML_METAL_HAS_TENSOR) fed a static K=32 tile to the matmul2d op on every iteration. On the last, partial K tile (ne00 % 32 != 0) the src1 slice extends past the K extent of the tensor, and the op reads those out-of-bounds elements (undefined behavior per the MSL specification, section 2.22.2). Depending on stale memory contents, this corrupted the result or produced NaN. Make the matmul2d op use dynamic_extent for K, and clamp the K extent of both operand tensor views to the remaining valid K range (min(32, K - loop_k)) per iteration, so the op reads exactly the valid K range on every iteration (mirroring the tail handling of the MPP matmul2d examples). On K-aligned inputs the clamp degenerates to the full 32-wide tile: the only difference from the static-K op is that the dynamic-K op derives K from the operand extents and edge-checks the tile against the tensor extents (a handful of integer ops per iteration). Add test-backend-ops MUL_MAT cases with K not a multiple of 32 to exercise the unaligned K path. Assisted-by: pi:llama.cpp/Qwen3.8-27B |
||
|
Georgi Gerganov
|
60f14a5ecc | metal : dequant kv cache only for large batches (llama/27438) | ||
|
Georgi Gerganov
|
283775e0a5 |
metal : dequantize quantized KV to F16 before flash attention (llama/27390)
* metal: dequantize q8_0 KV to f16 before flash attention Add a preprocessing pass for GGML_OP_FLASH_ATTN_EXT on the Metal backend: when the KV cache is quantized (Q8_0 for now), dequantize K and V into a contiguous F16 scratch buffer and run the existing F16 flash attention kernels on it, instead of the in-kernel dequantization path. - new kernel kernel_flash_attn_ext_dequant_to_f16<block_t, QK, deq_t4x4>: one thread per quant block (K then V), stride-aware so permuted KV is supported; instantiated for Q8_0 (extending to Q4_0/Q4_1/Q5_0/Q5_1 is one instantiation + one gate case) - the gate is type-only: dequantize whenever the KV is quantized, regardless of head sizes, GQA ratio or n_kv; the attention kernels themselves are untouched - the F16 copies live in the op's own scratch allocation (ggml_metal_op_flash_attn_ext_extra_dequant_f16); the KV pad kernel reads the dequantized buffers when the path is active - the FA pipeline getters gain a use_f16_kv flag selecting the existing f16 kernels and contiguous strides - ref: https://github.com/ggml-org/llama.cpp/pull/25556 Verification (M2 Ultra): - test-backend-ops test -o FLASH_ATTN_EXT: 4798/4798 pass, including the new q8_0 eval cases (decode/prompt, permuted, sinks+ALiBi+softcap, kv=113 pad path, kv=16384) - llama-perplexity on Qwen2.5-0.5B with -ctk q8_0 -ctv q8_0 matches the f16 KV reference (PPL 1.0008 vs 1.0008) Assisted-by: pi:llama.cpp/Qwen3.8-27B * metal : launch the FA KV dequant kernel separately for K and V Simplify kernel_flash_attn_ext_dequant_to_f16: it now dequantizes a single tensor (its own ne/nb and dst) with no is_v branching, and the op dispatches it twice with the same pipeline - once for K and once for V. The kargs struct shrinks to a single ne/nb set plus nblocks. Assisted-by: pi:llama.cpp/Qwen3.8-27B * metal : dequantize q4_0, q4_1, q5_0 and q5_1 KV to f16 before flash attention The dequant pass now covers all quantized KV types supported by the Metal flash attention kernels. The dequant kernel, kargs, scratch allocation and dispatch are type-generic, so each type is one kernel instantiation plus one gate case. Assisted-by: pi:llama.cpp/Qwen3.8-27B * metal : skip the redundant V dequant when V is a view of K In MLA-based models, the V of the FA op is a view of K (the first ne20 elements of each K row); the dequantized V is then a view of the dequantized K, so skip the second dequant dispatch, do not reserve the V scratch region, and let the pad and attention kernels read V from the K F16 buffer with K's strides. The detection follows the CUDA backend: V->view_src && (V->view_src == K || (V->view_src == K->view_src && V->view_offs == K->view_offs)) Also fix the FA pipeline getters: ns10/ns20 are function constants baked into the kernels and must be the actual K/V row widths as seen by the kernel. The dispatch now passes them explicitly (nb11_attn/nb10_attn, nb21_attn/nb20_attn) instead of the getters assuming contiguous F16 KV (ns20 = dv), which was wrong when V is read from K with K's row pitch (e.g. 576 vs 512). New test cases: 576/512 q8_0 (MLA shape, V is a view of K) at kv=113 (KV pad), nb=1 (vec) and nb=64 (non-vec). Assisted-by: pi:llama.cpp/Qwen3.8-27B * test : remove backend-specific wording from test-backend-ops comments Assisted-by: pi:llama.cpp/Qwen3.8-27B * pi : avoid backend mentions in test-backend-ops comments Assisted-by: pi:llama.cpp/Qwen3.8-27B * metal : rename the FA dequant_f16 identifiers to kv_f16 Assisted-by: pi:llama.cpp/Qwen3.8-27B * cont : clean-up * cont : remove TODO |
||
|
Xuan-Son Nguyen
|
8442c74f56 |
ggml: add ggml_rope_set_offset (+ metal support) (llama/27120)
* add params * cpu kernel * metal kernel * add test backend ops * gate other backends * ggml: (cuda) support ggml_rope_set_offset (llama/27121) * rm cuda supports_op guard, fix webgpu clang-format * ggml: support ggml_rope_set_offset on vulkan (llama/27344) * ggml: support ggml_rope_set_offset on vulkan * remove inplace optimization |
||
|
Georgi Gerganov
|
d830bd220c | metal : dequantize q8_0 using packed types (llama/27370) | ||
|
|
b7ea8b19bf |
ggml : recurrent state rollback for ggml_ssm_scan (llama/26623)
* Initial changes for Recurrent state rollback for nemotron for cpu and cuda * Removing CPU RS rollback. Will enable it in subsequent PRs * addition of test case * Removing assert and calling runtime API to check if op is supported * removing extra API and updating the call sites for K * replace static cuda detection to runtime fused_op api * address review comments and fallback when SSM rollback not supprted * Adding changes for supporting RS-rollback in CPU. Also added test-backend-ops for cpu and cuda * removing memory manipulation as rs rollback is now supported in CPU * removing the static probe which is not needed now * correcting the format * address review comments * enabling test for all the backends, unsupported backends will fallback to CPU * Apply suggestions from code review Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * choose different graph based on the result of fused_ssm_op is supported or not and also handled memory->n_rs_seq >1 case incase of op is not supported * Support K > 1 in ssm_scan for all backends * Fix CI Issues --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> |
||
|
Georgi Gerganov
|
34009e8518 |
metal : add TQ2_0 support (llama/26980)
* metal: add TQ2_0 support Add support for the GGML_TYPE_TQ2_0 (ternary, 2 bits per element) type in the Metal backend. Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731 * cont : optimize mul_mv kernel - float ops over integer ops - precalculate sums - hoist coef out of the inner loop - contiguous y loads llama.cpp:DeepSeek-v4-Flash-0731 |
||
|
|
d9589688cb |
llama: add default load-mode auto, which avoids mmap on iGPUs (llama/26081)
* llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used * Update ggml/src/ggml-hexagon/ggml-hexagon.cpp Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> * set mmap_support to false on OpenCL backend * fix order of load modes * use -1 for auto * resolve load mode auto earlier to correctly pick gpu host or cpu memory * add load mode auto to llama-bench * bump virtgpu api version, regenerate docs --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |
||
|
Yash Raj Pandey
|
b3bc904638 |
ggml : require contiguous src for ROLL on CUDA and Metal (llama/25928)
ggml_roll only asserts nb[0] == ggml_type_size, so a permuted src is a valid input, but the CUDA and Metal roll kernels index by ne alone and never read the nb strides. A non-contiguous src therefore produced silently wrong results. Neither backend declared a contiguity requirement in supports_op, so the scheduler did not fall back to the CPU implementation, which does handle strides correctly. Add the requirement to both backends, matching the existing GGML_OP_ROPE guard, and add a permuted test_roll case. |
||
|
robertomeroni
|
8a5ba01c93 |
metal : fix NORM/RMS_NORM for row lengths that leave a partial simdgroup (llama/26708)
ggml_metal_op_norm sized the threadgroup with
`nth = std::min(nth, args.ne00_t)`, which can leave nth not a multiple of
the simdgroup size. The kernels finish their row reduction with a
cross-simdgroup step where each lane of the last simdgroup reads one
per-simdgroup partial sum out of shmem_f32:
if (tiisg == 0) { shmem_f32[sgitg] = sumf; }
threadgroup_barrier(mem_flags::mem_threadgroup);
sumf = shmem_f32[tiisg];
sumf = simd_sum(sumf);
When the last simdgroup is partial it has fewer lanes than the
threadgroup has simdgroups, so the tail of the partial sums is never
read and the row sum is too small. For ne00_t = 33 nth becomes 33: two
simdgroups, but only one lane in the second, so one of the two partial
sums is dropped. The mean and variance are then wrong for the whole row.
Round ne00_t up to a whole number of simdgroups instead. Rounding up
rather than dropping the clamp keeps the threadgroup as small as
possible: deleting the line would raise nth to the next power of two
(ne00_t = 544 -> 1024 instead of 544), which costs idle lanes on 26 row
lengths below 8192 that were already correct, including 1536 and 3584.
GGML_OP_NORM is affected as well as GGML_OP_RMS_NORM - both dispatch
through ggml_metal_op_norm.
No mainstream LLM hidden size hits this: ne00_t is ne00/4 on the
vectorized path, so 4096, 8192, 2048 and friends all give a multiple of
32. It is reachable from other norm shapes, e.g. 320-channel norms.
Add NORM and RMS_NORM cases for ne0 = 33, 132 and 260 across the
existing eps values. 33 exercises the scalar path and 132/260 the
vectorized one, since only those divide by 4.
Before, on M3 Pro:
test-backend-ops test -b MTL0 -o NORM 25/50
test-backend-ops test -b MTL0 -o RMS_NORM 26/51
After:
test-backend-ops test -b MTL0 -o NORM 50/50
test-backend-ops test -b MTL0 -o RMS_NORM 51/51
test-backend-ops test -b MTL0 13943/13943
|
||
|
JamePeng
|
5693378610 |
metal : avoid threadgroup matrix array instantiation in kernel_lightning_indexer (llama/26646)
- In MSL, declaring an array of matrix types like `threadgroup half4x4` causes a 'no matching constructor' compilation error because MSL matrix types do not have zero-argument default constructors and threadgroup variables cannot have initializers. - Fix this by declaring a POD `threadgroup half` array instead and casting to `threadgroup half4x4 *` for matrix indexing. Signed-off-by: JamePeng <jame_peng@sina.com> |
||
|
|
e5f6c74112 |
metal: implement DSv4 Lightning Indexer (llama/25893)
* metal: implement F16 Lightning Indexer - Implement GGML_OP_LIGHTNING_INDEXER for 128-dimensional, 64-head inputs with F32 queries and weights plus F16 keys and masks. - Add tiled and tail kernels and test KV lengths around 8- and 64-element boundaries. llama-bench (--mmap 1, -fa 1, -p 512, -n 128; d=0/10k/20k/30k): Before: - pp512: 153.73 ± 0.87 t/s - tg128: 8.91 ± 0.04 t/s - pp512 @ d10000: 73.90 ± 0.39 t/s - tg128 @ d10000: 8.66 ± 0.03 t/s - pp512 @ d20000: 45.83 ± 0.18 t/s - tg128 @ d20000: 8.26 ± 0.03 t/s - pp512 @ d30000: 33.40 ± 0.21 t/s - tg128 @ d30000: 7.94 ± 0.01 t/s After: - pp512: 155.19 ± 0.91 t/s - tg128: 8.95 ± 0.04 t/s - pp512 @ d10000: 86.95 ± 0.69 t/s - tg128 @ d10000: 9.00 ± 0.05 t/s - pp512 @ d20000: 62.01 ± 0.45 t/s - tg128 @ d20000: 8.68 ± 0.04 t/s - pp512 @ d30000: 49.18 ± 0.33 t/s - tg128 @ d30000: 8.60 ± 0.02 t/s Assisted-by: Codex * metal: stage Lightning Indexer K tiles - Stage and dequantize K in F16 threadgroup memory before simdgroup matrix loads. - Zero-fill partial tiles and guard stores so all KV segments use the same numerical path. - Support F32, F16, BF16, Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0 K caches. llama-bench (--mmap 1, -fa on, -p 512, -n 128; d=0/10k/20k): - pp512: 160.38 +/- 1.01 t/s - tg128: 9.08 +/- 0.03 t/s - pp512 @ d10000: 88.37 +/- 0.46 t/s - tg128 @ d10000: 9.07 +/- 0.04 t/s - pp512 @ d20000: 62.53 +/- 0.46 t/s - tg128 @ d20000: 8.84 +/- 0.03 t/s Assisted-by: Codex * dedup Lightning Indexer constants, fix flaky test * cont : fix whitespace --------- Co-authored-by: forforever73 <690105611@qq.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |
||
|
Talha Adnan
|
319e9a7082 |
metal : add SILU_BACK (llama/25982)
* feat(silu_back): implemented silu_back op for f32 * fix(silu_back): removed redundant asserts in ggml-metal-ops.cpp function ggml_metal_op_silu_back. |
||
|
Georgi Gerganov
|
4695615b6d | metal : add F16 support for bin ops (llama/26465) | ||
|
Georgi GerganovandThiago Padilha
|
e7ae4d718e |
metal: implement DeepSeek V4 hyper-connections (llama/26459)
- Implement GGML_OP_DSV4_HC_COMB, GGML_OP_DSV4_HC_PRE, and GGML_OP_DSV4_HC_POST with SIMDgroup register and shuffle optimized kernels. - Add Metal dispatch and support plumbing and test the production Sinkhorn iteration count and embedding width. Assisted-by: Codex Co-authored-by: Thiago Padilha <thiago@padilha.cc> |
||
|
+67Georgi GerganovAparna M PMax KrasnyanskyfairydreamingStanisław SzymczykJohannes GäßlerHongqiang WangMartin ChangVidasGianluca GuidaGianluca GuidaubergarmSaqibAkram-10xERehan QasimLi HeRaman ShindecphlipotRohit MaheshTodd MalsbaryFrosty40Claude Fable 5QuintinShawJeff BolzPasha KhosraviTitaniumtownCharles XuJusteLeoChyanhmsciderscientist3hmscidermaxiousFrancois DugastAndrew SmithNeo ZhangMichael LamothePascalleonardHONGDavid FriehsPranesh Gonegandlapraneshgoliminfei-amdAlexander HeislerAnav PrasadRuben OrtlamRajendra MatchaAman GuptaakleineGezahegneAaron TeoTodor BoinovskiPiotr WilkinMarkus EbnerWinston MaKamalesh VSWei Wangm1elshalinib-ibmOliver SimonsIlia Ilmeradgup-qtikumaalYongmin Yoo 유용민yzyyzyhhhBeinseziiNick LafleurYiChen LvmeatposesBhavik ShardaGaurav GargReese LevineGeramy LovelessKakaruKakaruHayateJonathan ClohessyNiklas WenzelDaniel Bevenius |
2ca53bb45e |
sync : ggml (#3962)
* hexagon: tiling, tracing and optimizations for unary ops (llama/25474) * hexagon: tile wide rows in pointwise unary ops to avoid VTCM overflow * unary: reject permuted tensors for now (not used by models) * hex-unary: replace divs with fastdiv * hex-unary: add vtcm layout and host computed kernel params * hex-unary: move fastdiv init into kernel params * hex-unary: add specialized thread functions to improve generated code * hex-unary: tracing instrumentation for unary ops * hex-unary: factor out hvx kernels, streamline and remove more duplication * ggml-hexagon: fix std::min collision with Windows min macro * hex-cmake: make lto build happy --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> * ggml : process data in smaller chunks in CUDA ggml_top_k() and ggml_argsort() to reduce temporary buffers memory usage (llama/24776) * ggml : process data in smaller chunks in CUDA ggml_top_k() implementation to reduce temporary buffers memory usage * ggml : allocate tmp_dst only only once before the loop * chore : whitespaces Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * ggml : use chunked processing in both CUDA CUB top-k and argsort implementations * chore : separate argsort_f32_i32_cuda_bitonic() call from return statement Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * chore : replace ternary operators with min/max --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * opencl: cluster-parallel decode FA for Adreno (llama/25473) * ggml-et: Initial ET backend (llama/24179) * ggml-et: Add performance logging * ggml-et: Quants helpers * ggml-et: Add MUL_MAT kernel * ggml-et: Add ROPE kernel * ggml-et: Add RMS_NORM kernel * ggml-et: Add GLU kernel * ggml-et: Add SOFT_MAX kernel * ggml-et: Add GET_ROWS kernel * ggml-et: Add CONT kernel * ggml-et: Add SET_ROWS kernel * ggml-et: Add MUL_MAT_ID kernel * ggml-et: Build et kernels as part of ggml * ggml-et: Embed kernels with fs fallback * ggml-et: Build fixes * ggml-et: Add MUL_MAT F32xF32 op * ggml_et: Add MUL_MAT_ID op * ggml-et: Disable offloading for debug * ggml-et: Refactor out block ops * ggml-et: ggml backend API changes * ggml-et: Add RESHAPE/TRANSPOSE to supported * ggml-et: Add CONT_F16 * ggml-et: Add supported ops doc * gglm-et: Initial doc * ggml-et: Remove runtime import hacks We can now import the runtime by a simple find_package(), so we can cleanup the CMakeLists.txt. * ggml-et: Fix GET_ROWS kernel Fix lost batch dimension. Also clean vibe-comments. * ggml-et: Fix SET_ROWS kernel Remove incorrect broadcasting guard. * ggml-et: Use custom instruction for fp32->fp16 * ggml-et: Vectorize set_rows fp32->fp16 * ggml-et: Fix ROPE kernel (yarn) ggml-et: fix et_logf WIP: Fix ramp WIP: fix ROPE! * ggml-et: Better sinf * ggml-et: Fix SOFT_MAX Add `max_bias` and `sink` support. * ggml-et: Fix CONT Reorder from contiguous write to read with atomic stores. * ggml-et: Fix elmap kernel Remainder handlin * ggml-et: Fix MUL_MAT MUL_MAT_ID remainders * ggml-et: Fix ET-SOC reference * ggml-et: Fix embed kernels scripts for old python This allows GGML-ET to build on pre-3.8 python. * Add sysemu support with compile time flag `-DGGML_ET_SYSEMU=ON` (llama/6) * Example using ET-Soc-1 emulator configuration Example usage: ```bash cmake -B build -DGGML_CUDA=OFF -DGGML_ET=ON -DLLAMA_CURL=OFF -DGGML_CCACHE=ON cmake --build build --config Release -j $(nproc) time ./build/bin/test-backend-ops ./build/bin/llama-server \ --model Qwen3-0.6B-Q8_0.gguf \ --alias Qwen3-0.6B-Q8_0 \ -fa 0 \ --ctx-size 1024 \ --no-warmup \ --host 127.0.0.1 \ --port 8080 ``` * build: proper dep tracking for kernels * support host using MOLD linker * initial multi core GET_ROW F32 implementation * vectorized q8 dequant * wip: cland warning clenaups and initial logging refactor * wip: message default message cleanup * chore: message cleanups * cmake cleanup * migrate to use platform provided functions * cmake back into subdir * support et_print() in kernels * fix: repair kernel building * perf: operations run async by default * debug: proper kernel dep tracking and error detection on kenrel launch * fix: kernel binary dep tracking and fixing get_rows_f32 erroring * perf: back to doing async kernel runs by default * perf: vectorize and parallel device memset * merge matmul work * misc: align allocation and enable all offload * misc: delete deadcode and respect memory limits * fix: repair tensor debug print * fix: loosen RMS_NORM op percision * feat: Q4_0 GET_ROWS * perf: FP32 MUL_MAT using TensorFMA * update limitations * perf: redue L1 load in compute_block_dot_product_q8_0 * feat: save kernel mapping (name to id) when profiling is enabled * chore: memops cleanup * perf: parallelize softmax by rows * perf: vectorize 2nd phase of softmax * perf: ban GET_ROWS from offloaded * perf: vectorize and non-atomic for eltwise ops and sub support * perf: vectorize normal rope * perf: glu runs in parallel * merge: manually merge saqib's work on kernel fixes * perf: more vectorized RoPE * perf: parallelize mul_mat_id * perf: parallelize set_rows_f32 * perf: vectorize softmax * feat: support kernel fusion and fuse RMS_NORM + MUL * fix: mostly resolve test-backend-ops failure in SOFT_MAX and ROPE * fix: bump max rope dims for gemma * feat: GeGLU and SCALE support to fully offload Gemma * perf: faster device memset * feat: get_rows supporting Q4_K and avoid cont cache coherent issues * better F32 MM * feat: NORM for ET backend * feat: SQR for ET backend * feat: UNARY on ET * feat: el_map support broadcasting for ET * feat: SUM_ROWS in ET backend * feat: more ops in ET backend * feat: WKV* operators in ET backend * perf: parallelize operators across cacheline instead of row * perf: parallelize get_rows on cacheline * wip: baseline FlashAttention for ET backend * wip: enough FA and CPY f32->f16 to run llama 3.1 fully offloaded with FA on * feat: f16 x f16 -> f32 MM using matrix engine * wip: f16 FlashAttention using matrix engine * wip: clean up * feat: barriers * perf: optimize FA_F16 in ET * perf: vectorize pack_k_for_transpose16 * perf: prefetch next loop matrix tile * perf: FlashAttention 2nd MM uses TensorFMA and optimizations * cleanup: flashattention reorg * perf: optimizations and fixes * feat: L2SCP API and make FlashAttention support DV = 256 for gemma * perf: parallelize norms beyond single row * feat: GATED_DELTA_NET support and relaxed L2_NORM requirment * feat: loosen RMS_NORM, NORM, ROPE contingous req too * feat: repeat supports brocasting on dim 0 and loosen cont check * feat: FILL and DIAG operator * feat: loosen UNARY support chcek * feat: TRI support * feat: SOLVE_TRI support * feat: basic SET support * feat: loosen CONT req * perf: fp16_to_fp32 use ASM * feat: IMROPE support * feat: PAD support * feat: global barrier * fix: view must live on the same backend as backing tensor * feat: relax CONCAT in ET backend * feat: dead simple CUMSUM implementation * feat: basic SSM_CONV support * feat: loosen CONCAT req * feat: relax GATED_DELTA_NET and add SET support proper * cleanup: cleanup LCM math * feat: SWIGLU single input * feat: SSM_SCAN support * feat: el_map supports non aligned tensors in best effort * feat: basic GROUP_NORM support * feat: loosen MUL_MAT capablities slightly * feat: loosen MUL_MAT and GET_ROWS and add IM2COL * feat: special case for softmax 1x1x1x1 * feat: loosen SOFT_MAX req in ET backend * fix: el_map unaligned acse fixes * perf: optimize zero_acc_vec in flash_attn_ext_f16_me * perf: use hart 1 for packing in MM and FA for FP16 * feat: kernel semaphore * perf: better instruction sequence in FlashAttention * fix: gated_delta_net with proper masking * perf: better parallelization for GATED_DELTA_NET * perf: parallelize SSM_CONV over nr * perf: vectorize SSM_CONV * perf: optimize MUL_MAT for q8 * feat: support Gemma 4 * fix: support multi-device * feat: broader GLU support * feat: unary ops supports view * fix: repair fp16 MM using matrix engine * perf: handle large N GEMV better * perf: better q8_0 MM * perf: better set_rows * add back deleted files * fix: repair after merge * feat: POC version of uberkernel * feat: RMS_NORM in uberkernel * feat: add more kernels into usage * chore: clean up uberkernel compilation * perf: faster flash attention * perf: opt flash attention for large seq length * feat: loosen op bounds. clamp and mean support * perf: vectorize ssm_scan * perf: slightly faster FA * perf: FlashAttention parallel MM and load * perf: fuse Q8 MM and ADD * feat: basic conv kernel for ET * softMAx_test * set_rows_f32 * get_rows and cont * testing * set_rows_exp * Junk addition * Narrowing the issue * Update flash_attn_ext_f16_me.c Focusing FA_ext_f16_me * test * Eviction updated * Detailed cache eviction debug * mulmat * removeal of `BUILD_FOR_UBERKERNEL` flag * cleaning... * fix: balance FCC0 count * feat: implement mul_mat and mul_mat_id for Q4_0 type * optimize uberkernel plan upload * add mul_mat q4 into uberkernel * enable gating flush to just uberkernel * update docs for ET * update op support for ET * et-backend: optimize Q4_0 and Q8_0 mul_mat_id row accumulations * et-backend: specialize mul_mat_id kernels for Q4_0 and Q8_0 * et-backend: fix RoPE YaRN corr_dim formula and handle degenerate inputs * test-backend-ops: add DeepSeek-V2-Lite RoPE test coverage * et-backend: add Q4_0 mul_mat matrix-engine kernel using TensorFMA32 * et-backend: vectorize Q4_0 matrix-engine dequantization * et-backend: support hybrid matrix/vector engine execution for Q4_0 mul_mat tail * et-backend: run partial-N tiles on matrix engine for Q4_0 mul_mat * et-backend: route Q4_0 mul_mat N < 53 to vecdot for better prefill latency * Update uberkernel.c * Update unary_f32.c * gemma 4 * bisect gemma4: enable scale_f32 only * bisect gemma4: +rms_norm_f32 * bisect gemma4: +rms_norm_mul_f32 * bisect gemma4: disable rms_norm_mul_f32 -- BREAKS OUTPUT * bisect gemma4: +rope_f32 (skip rms_norm_mul) * bisect gemma4: +el_map_f32 * bisect gemma4: +softmax_f32 * bisect gemma4: +get_rows_f32 * bisect gemma4: +glu_f32 * bisect gemma4: +mul_mat_f32 +mul_mat_f32_matrix_engine * bisect gemma4: +mul_mat_f16 +mul_mat_f16_matrix_engine * bisect gemma4: +mul_mat_Q8_0 +mul_mat_Q4_0 * bisect gemma4: +flash_attn_ext_f32 +flash_attn_ext_f16_me * bisect gemma4: +mul_mat_id_f32 * bisect gemma4: +sum_rows_f32 * bisect gemma4: +cont_f16 * bisect gemma4: +fill_f32 * bisect gemma4: +unary_f32 (all ops re-enabled except rms_norm_mul) * Update rms_norm_mul_f32.c * bisect2 gemma4 n64: +scale_f32 only * bisect2 gemma4 n64: +rms_norm_f32 +rope_f32 * bisect2 gemma4 n64: +rms_norm_mul_f32 (with ET_UBERKERNEL eviction fix) * bisect2 gemma4 n64: +el_map +get_rows +glu +softmax (skip rms_norm_mul) * bisect2 gemma4 n64: all ops enabled except rms_norm_mul * bisect2 n64: test unary+cont+fill+sum_rows (no mul_mat/flash_attn) * bisect2 n64: +mul_mat_f32 +mul_mat_f32_matrix_engine * bisect2 n64: +mul_mat_f16 +mul_mat_f16_matrix_engine * bisect2 n64: +mul_mat_Q8_0 +mul_mat_Q4_0 * bisect2 n64: +mul_mat_Q8_0 only (disable Q4_0) * bisect2 n64: +mul_mat_Q4_0 only (Q8_0 breaks) * bisect2 n64: +mul_mat_id +flash_attn_ext (skip Q8_0) * run-3: matmul + rms_norm_mul * run-4 * Revert "run-4" * run5 * changes after cleanup * cleanup before upstream * restrict changes into ET backend * move kernel embedding from Python to CMake * move uberkernel gen into CMake * apply clang format * update CMake style * update to match C and C++ style * use source ggml and quant headers instead of ET's * MROPE support * absorb view ops into same branch as none * fix bad rebase * add marty1885 to codeowners * oops * remove redundant newline * fix CI editor warnings --------- Co-authored-by: Vidas <vidas@nuolat.lt> Co-authored-by: Gianluca Guida <glguida@tlbflush.org> Co-authored-by: Gianluca Guida <gianluca@nekko.ai> Co-authored-by: ubergarm <leimgrub@gmail.com> Co-authored-by: SaqibAkram-10xE <saqib.akram@10xengineers.ai> Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai> * hexagon: improve ARGSORT performance for small tensors (llama/25512) * hex-sort: add efficient bitomic sort in hvx regs up to 1024 elements * hex-sort: fix inverted vrors * hex-sort: specialize sort functions for the common cases * hex-sort: add tracing and local context * opencl: add int8 dp4 dense and MoE prefill optimization for Adreno GPUs (llama/25537) * opencl: add int8 dp4 dense and moe GEMM * opencl: refactor --------- Co-authored-by: Li He <lih@qti.qualcomm.com> * Vulkan: route large matmuls to medium tile on Adreno (llama/24877) * [Vulkan] Fixes llama-cli breaking over longer promts sizes The llama-cli was breaking for longer promts sizes for q4_0 quantized networks. Causing due to insufficient shared memory. * Removed the un-used Adreno device * Updated matmul for small pipeline. * ggml : add GGML_OP_LIGHTNING_INDEXER that implements DeepSeek V3.2/V4 lightning indexer (llama/24231) * ggml : add GGML_OP_LIGHTNING_INDEXER that implements DeepSeek V3.2/V4 lightning indexer * ggml : remove scale parameters from lightning indexer OP, add f16 mask parameter * tests : add GGML_OP_LIGHTNING_INDEXER tests * ggml : bump RPC version * chore : check if lightning indexer input tensors are not transposed * tests : count flops instead of bandwidth in lightning indexer test * chore : add missing const * chore : whitespace * ggml : renamed variables in CPU lightning indexer implementation * ggml : fix lightning indexer mask broadcasting * tests : tests for lightning indexer mask broadcasting * chore : whitespace * llama : use GGML_OP_LIGHTNING_INDEXER in DeepSeek V3.2 and DeepSeek V4 models --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * cuda: Don't crash when querying memory on device with no free memory. (llama/25157) If a Cuda device has no or limited available memory, the actual call to cudaMemGetInfo() itself can cause a fatal crash due to a cuda out of memory error (there is not enough memory to actually query memory) This causes an issue because we query memory for all devices at startup even if the user isn't trying to use the device for inference. Fix this by making the error non-fatal and assigning zero total/free memory to the device. This will have the downstream effect of the fit algorithm not trying to put any layers on it, which is desired outcome vs hard crashing. this also prevents crashes in cuda enabled builds when user explicitly passes '-dev none' * gguf : reject empty metadata keys (llama/24917) * sycl: add Q2_K to DMMV reorder path (llama/25064) Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: add fused top-k MoE (llama/25217) * sycl: add fused top-k MoE * sycl: address review: GGML_SYCL_ENABLE_FUSION env, move fusion dispatch to topk-moe * sycl: print GGML_SYCL_ENABLE_FUSION at startup like other env vars Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * gguf : add tensor shape accessor (llama/24405) * gguf : add tensor shape accessors * gguf : return tensor shape as const int64_t * * gguf : remove n_dims accessor, keep only gguf_get_tensor_ne * vulkan: Use native e2m1 and e4m3 conversions for mxfp4/nvfp4 (llama/25338) This uses the new VK_EXT_shader_ocp_microscaling_types extension to do fp4 type promotions, and also uses the float8 extension to do ue4m3 promotions for nvfp4. It's reasonable to assume that an implementation that supports fp4 will also support fp8, so we don't need to handle all possible combinations of support. * CUDA: refactor MMQ kernel configuration (llama/24127) * CUDA: refactor MMQ kernel configuration * fix Blackwell config * remove legacy code * metal : add Q2_0 support (llama/25419) * sycl: set fattn_vec_nthreads to 256 for Battlemage (llama/25205) Currently detects lunarlake + battlemage / xe2 and sets the value to 256. Keeps default at 128, Intel's ARC Alchemist's prefered value. * kleidiai : add SME2 f32 kernel (llama/24414) * kleidiai : add SME2 f32 kernel * enable dynamic scheduling for SME2 f32 kernel * ggml: uniformize im2col dst_type for all conv ops (llama/23660) * ggml: uniformize im2col dst_type for all conv ops * Update ggml/src/ggml.c Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * ggml : uniformize im2col casting logic across all conv ops * fix : allow im2col_f16 to accept any kernel type --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * ggml : add a set of functions for checking contiguity of inner tensor dimensions (llama/25650) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * vulkan/cpu: Support f16 as SET_ROWS src. (llama/25432) * vulkan/cpu: Support f16 as SET_ROWS src. This adds full support for f16 SET_ROWS (equivalent to f32) to vulkan and CPU backends, and adds more backend tests. * Set DenormPreserve 16 when supported, to try to fix failures on Intel * tune error threshold * update metal supports_op * opencl: fix a dp4a bug for devices where cl_khr_integer_dot_product is unavailable (llama/25639) * opencl: do not fail backend init on devices without cl_khr_integer_dot_product * opencl: do not call dp4 kernels when dp is unavailable --------- Co-authored-by: Li He <lih@qti.qualcomm.com> * hexagon: fix hmx-queue signal enum-narrowing problem (llama/25677) * opencl: avoid the vec path in GEMV for unaligned row stride (llama/25671) The f16 GEMV kernels take a vectorized path for ne00 >= 128 that casts the row pointers to half4 or float4. When the row stride is not aligned, the wide load becomes misaligned. On devices that require natural alignment for vector loads, the kernel reads garbage. This is the case Intel GPUs and the kernels produce incorrect results there. Adreno happpens to be byte addressable and the kernels happen to work. * opencl: handle OOB write in noshuffle GEMV kernels (odd ne01) (llama/25640) * opencl: do not use `clCreateBufferWithProperties` when targeting CL 2.x (llama/25673) * Flash Attention with XMX engine via oneDNN (llama/25222) * [SYCL] F16 (default) Flash Attention with XMX engine via oneDNN graph API; Qwen3.6-27b-Q8_0 prefill speed up x1.21 at p=512 and x4.26 at p=80k * [SYCL] Address review on FA oneDNN path. Result: llama-bench---pp512; 32% increase with fa1; llama-perplexity---0.11% difference; tested model: mradermacher/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf * PR-25222 revision v2: addressed audits * [SYCL] flash-attn oneDNN SDPA KV F16 rev 3.0: add BMG gate + multi-device sync. Narrow the scrope of this PR to Battlemage only (bmg; Xe2). Other archs (e.g., alchemist) fall back to existing FA kernel. When device_count >1, apply stream -> wait_and_throw(), validated working path for multi-gpu sync fix by @maxious. Co-authored-by: maxious <81432+maxious@users.noreply.github.com> * updated comment on bmg gate, noted the issue --------- Co-authored-by: scientist3 <scientist.3@users.noreply.github.com> Co-authored-by: hmscider <hmscider@users.noreply.github.com> Co-authored-by: maxious <81432+maxious@users.noreply.github.com> * sycl: Increase minimum buffer size for USM system allocations (llama/25525) Raise the threshold for minimum buffer size from 1 GiB to 4 GiB, based on real-world experiments of overcommitting device memory with model weights larger than available VRAM, for example Qwen3.5-35B-A3B-Q8 running on a B70. Also add a debug message to better track USM system allocations. Signed-off-by: Francois Dugast <francois.dugast@intel.com> * sycl : implement xielu op (llama/25550) * sycl : support kernel type fp16 for conv2d_dw (llama/25653) * sycl : fix get_rows Q2_K, Q4_K, Q5_K (llama/25656) * ggml: add f16 out_prod support for CPU and out_prod op for Vulkan (llama/23997) * metal: fuse snake activation (mul, sin, sqr, mul, add) (llama/25459) * metal: fuse snake activation (mul, sin, sqr, mul, add) Mirror the CUDA, Vulkan and CPU snake fusion: same matcher on the naive 5-op chain, same F32 contract on a and inv_b, same F32/F16/BF16 kernel with F32 compute. Follows the Metal backend idioms: bf16 instantiation gated behind GGML_METAL_HAS_BF16 and concurrency ranges checked on the remaining chain nodes before encoding, as done by the bin fusion. Covered by the existing backend-agnostic SNAKE_FUSE tests. * metal: absorb snake fusion into ggml_metal_op_bin Extract the matcher to ggml_metal_op_can_fuse_snake, mirroring the Vulkan naming, and dispatch the fused path from ggml_metal_op_bin. The encode loop switch is back to a single call per case. Address review from ggerganov * metal: fix indentation in ggml_metal_op_can_fuse_snake * cuda : relax tensor contiguity requirements for quantized concat (llama/25678) * cuda : relax tensor contiguity requirements for quantized concat * tests : add test cases for non-contiguous quantized concat * ggml : relax contiguity requirements for quantized concat --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * CUDA: tighter MMQ src1 buffer size for native fp4 (llama/25613) * opencl: fix two issues on flash attention for Adreno a7x (llama/25697) * opencl: route `sub_group_shuffle_xor` to qcom ext when KHR ext is unavailable KHR `sub_group_shuffle_xor` is not defined by compiler when `cl_qcom_subgroup_shuffle` is present, causing certain FA kernels fail to build. Define the KHR shuffle_xor using the qcom extension. * opencl: skip FA kernels with mixed and quant types for A7x to avoid compiler crash * cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) (llama/25545) * cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) * chore : remove indentation of #pragma unroll * cuda : remove unnecessary kernel template declarations * cuda : add WARPS_PER_BLOCK and K_VECS_PER_BLOCK template parameters in lightning indexer kernels to avoid duplication of constants. * cuda : relax MMA architecture requirements to Turing in lightning indexer implementation * chore : renamed variables * chore : rename ggml_cuda_op_lightning_indexer() to ggml_cuda_lightning_indexer() * chore : TODO for AMD rocWMMA * chore : whitespace formatting * chore : another variable rename to fix problems caused by shadowing * chore : yet another rename, this time uppercased all constants * cuda : added alignment checks for Q and K tensors in lightning indexer implementation --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * opencl: exclude some moe kernels on Adreno a7x (llama/25698) * opencl: exclude Adreno A7x from using Adreno MoE kernels Some compilers for A7x devices miscompile the repack kernels, corrupting the weights and causing MoE models to generate garbage output * opencl: exclude A6x and unknown Adreno from MoE weights repack * cuda: extract Q1_0 elements via __byte_perm (llama/25628) * opencl: disable FA and MoE weights repack to work around compiler issues for Adreno 850 GPU (llama/25745) * opencl: workaround for A850 compiler compat * opencl: fix DX compiler version parsing and cleanup --------- Co-authored-by: Li He <lih@qti.qualcomm.com> * CUDA: dedup MoE gate/up activation quantization (llama/25441) * CUDA: dedup MoE gate/up activation quantization (fp4) For MoE gate/up projections the src1 activation is broadcast across the routed experts (ne11 == 1), so ids_src1 maps every one of a token's n_expert_used slots to the same physical row. The MMQ path therefore re-quantized each token's activation n_expert_used times. For fp4 (NVFP4/MXFP4) src0, quantize each unique token row once instead of once per expert. For NVFP4 a single quantize+scatter kernel (quantize_scatter_mmq_nvfp4) quantizes each token once and writes the resulting block_fp4_mmq straight to all n_expert_used slots, using an inverse token->compact-row map (build_tok2c). MXFP4, and GGML_CUDA_MOE_QUANT_GATHER=1, use a two-kernel variant: quantize unique rows then gather into the expert-sorted layout (gather_mmq_fp4_blocks). Both are bit-identical to the previous gather-then-quantize path (identical source data, deterministic per-block quantization), verified by test-backend-ops MUL_MAT_ID (type_a=nvfp4, broadcast b=1; 790/790 for the default, gather, and per-expert paths) and by coherent end-to-end generation. Set GGML_CUDA_NO_MOE_QUANT_DEDUP=1 to force the original per-expert path. Same-binary A/B on RTX 5090 (sm_120), Qwen3.6-35B-A3B-NVFP4 prefill @8192 (nsys, graphs-off; the unchanged mul_mat_q GEMM confirms stable clocks): activation-quant GPU-busy drops 61% (78.2 -> 30.4 ms) with the fused quantize+scatter, vs 33% (78.2 -> 52.8 ms) for the two-kernel gather. The fused path avoids materializing and re-reading the 8x compact buffer, writing the expert copies directly from registers. * CUDA: bounds-check token ids in build_tok2c_kernel Guard against malformed ids_src1: skip out-of-range token ids (t < 0 or t >= n_tokens) and drop entries beyond n_expert_used per token instead of writing past the token's tok2c region. No behavior change for valid MoE routing data; test-backend-ops MUL_MAT_ID 790/790. * Refactor the code based on review comments - Removed previously added kernels that were not necessary anymore\ - Added an inverse mapping from (token, slot) to compact row. Each token is quantized once and scattered to its compact rows. * Adding q8_1 support for dedup and addressing review comments * Add pragma unrolls * Remove redundant cudaMemsetAsync call * Removing follow up redundancies --------- Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com> * ggml-cuda : restore prop.integrated on HIP builds (llama/24233) PR #16308 set info.devices[id].integrated = false unconditionally for all CUDA/HIP devices as a workaround for corrupted output on Jetson Orin (#15034). On HIP/ROCm the device's real hipDeviceProp_t.integrated flag is needed: with the cached field forced to false, supports_buft() refuses CUDA host buffers on AMD APU/UMA parts, while get_type() already reads prop.integrated (#23007) — an inconsistency that breaks integrated-GPU host-buffer use on ROCm. Guard the workaround so it only applies to non-HIP (CUDA) builds and restore prop.integrated for HIP, keeping the Jetson workaround intact for CUDA. Fixes #23977 Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com> * Enable CUDA graphs on volta+turing (llama/25749) * CUDA: Support CUDA Virtual Devices (llama/25228) * support cuda virtual devices * disable NCCL path when virtual devices are used * label virtual devices in description; add GPUx2 server CI jobs * code refactor * vulkan: when using transfer queue for async copies, sync on event_wait to avoid race (llama/25229) * kleidiai: Add SME vs SME2 distinction in kernel dispatch (llama/25478) The current integration treats SME as a single capability (CPU_FEATURE_SME) with no distinction between SME(v1) and SME2. The kernels dispatched under CPU_FEATURE_SME use SME2-specific instructions, making dispatch incorrect on SME(v1)-only hardware. We introduce build-time and runtime distinction between SME and SME2, and wire SME(v1) and SME2 kernels based on actual hardware support. * hexagon: L2 cache handling rework (dirty bit tracking with lazy flushing) and more MUL_MAT updates (llama/25762) * hex-mm: fix artificial limit in the solver that restricted number of act-prep threads * hex-mm: fix warning * hex-prof: do not apply --top to the timeline report * hmx-mm: add suport for tiled act-processing to better distribute hvx work * hex-l2: add tracing for l2flush events * workqueue: redo the legacy workpool api to match hmx-queue and dma-queue * hmx-mm: fix f32 activation buffer alignmnet for nhvx=5,6,7 * hex-work: minor cleanup for work-queue apis * hex-work: further cleanup of the work-queue api * hex-l2: optimize l2flushes at the opbatch level * hex-work: remove unused mask * hex-work: no need to drop hvx ctx in the work-queue * hex-work: add explicit wakeup/suspend and make threads spin * hex-bufs: mark any non-weight tensor as compute * hex-dma: dma-queue support for alias queues and cached dma * hex-l2: track tensor aliases and delay or skip flushes as much as possible * hex-l2: simplify tensor alias handling * hex-l2: handle overlapping views as a circular list of aliases * hex-tens: add flags helper * hex-l2: add helper for marking tensors clearn/dirty * hex-l2: mark binary and rope outputs as l2-clean and keep the rest as is for now * hex-l2: proper support for handling all tensor overlap scenarios * hex-trace: instrument matmul init code and cleanup trace checks * hex-thread: introduce dedicated main thread with explicit stack and priority * hex-l2: track dirty state as bitmap and introduce threaded flush * hex-trace: remove redundant checks for ctx != null * hex-l2: allocate entire context as one buffer and l2fetch it after big flushes * hex-l2: disable tensor clearing in binary and rope for now seems to cause issues with fusion * hmx-mm: update act proc to use fastdivs and fix DMA overflow * hmx-mm: make MUL_MAT_ID kernels robust to multi-chunk cases (start_row>0) * hex-queue: remove obsolete queue interfaces and flush hmx-queue at the end of the op-batch * hex-queue: dont use early wakeup for small op-batches * hex-tensors: properly cap max_tensors in op-batches and dirty_map * hex-l2: make sure threaded l2flush does proper rounding * hex-l2: factor out htp_tensor_flush for reuse (if needed) * hex-l2: optimize tensor flushes by coalescing flush-all * hex-l2: optimize multi-threaded flush * hex-drv: futureproof version checks * hexagon: fix errors and warnings on windows * hex-main: update main thread to only use dspqueue_read, dspqueue_peek is not available on some platforms * hex-main: add fallback mode for dspqueue with callbacks * hex-main: introduce fallback mode for using dspqueue callbacks for full op processing * hex-main: remove early wakeup, not helping and seems to cause some errors with certain batch sizes * hex-l2: make sure to use invalidate version of flushall * hex-l2: dont try to trace early l2flush at the start of op-batch * hex-main: remove offset_ctx that must be zero anyway * hex-hmx: fix hmx_queue_depth to use idx_write - idx_read * hex-hmx: use atomic_load for idx_read/write * hex-main: add static assert to make sure n_threads are aligned * DeepseekV4: Add fused hyper-connection ops (llama/25585) * dsv4 hc-ops * add missing files; * add cparams * update rpc version * address review comments * address review comments * docs: added a note about using OpenCl with Adreno 810 (llama/25786) * opencl: loads quants as uint for q4_K and q5_K flat mv (optimization for Adreno A7x GPUs) (llama/25780) * opencl: load quant as uint in mv_q4_k_f32_flat helps older compilers (e.g., E031.41, boosts 2x), no impact on newer compilers (e.g., E031.45 or newer) * opencl: load quant as uint in mv_q5_K_f32_flat helps older compilers (e.g., E031.41) * opencl: format --------- Co-authored-by: Li He <lih@qti.qualcomm.com> * opencl: add ABS op (llama/25115) * sycl: fix row calculation when K_QUANTS_PER_ITERATION is 1 (llama/25690) * sycl: fix incorrect row calculation when K_QUANTS_PER_ITERATION=1 Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: use K_QUANTS_PER_ITERATION for non-reordered Q5_K kernel This is the only Q5_K kernel that was not using KQPI. Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: add missing second half processing to reordered q5_k Error found while running GGML_SYCL_PRIORITIZE_DMMV=1 \ build/bin/test-backend-ops test -o MUL_MAT Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: fix potential off-by-one error Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: fix missing row > nrows check Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> --------- Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * vulkan: Support Q2_0 (llama/25430) * vulkan: Support Q2_0 The backend perf tests for mat-vec-mul weren't very good at first (worse than q2_k), doubling the rows per workgroup made a big difference. * reorder * resolve merge conflict, adjust err threshold for f16->q2_0 set_rows * ggml-blas: default hadamard mul_mat to cpu routine (llama/25710) Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * ggml : bump version to 0.17.0 (ggml/1568) * opencl: transpose q4_K noshuffle scales for coalesced reads (llama/25805) * opencl: read/write MoE dp4a activation tiles to local memory as 128-bit (vectorized LD/ST perf opt) for Adreno GPUs (llama/25810) * opencl: read MoE dp4a activation tile as 128-bit local loads * opencl: vectorize MoE dp4a activation staging as 128-bit loads * opencl: load and use `kernel_gemm_moe_q6_k_f32_ns` from bin kernel lib (llama/25797) * opencl: Support broadcast for Adreno MUL_MAT and honor `view_offs` for Adreno Q8_0 MUL_MAT for llama-server multi-stream (llama/25910) * opencl: handle broadcast for adreno gemm/gemv_noshuffle * opencl: honor view_offs for adreno noshuffle gemm/gemv * opencl: general GEMM/GEMV support broadcast * opencl: remove unnecessary tests * opencl: remove unnecessary comments --------- Co-authored-by: Li He <lih@qti.qualcomm.com> * hexagon: add CLAMP op (llama/25934) * CUDA: vectorize same-type get_rows with int4 copy (llama/25929) k_get_rows_float did a scalar one-element-per-thread copy and recomputed the row-invariant work (index load, fast_div_modulo, src/dst row pointers) for every element. Hoist that out of the per-element loop, and add a vectorized path (k_get_rows_float_vec) that copies one int4 (16 B) per thread for the contiguous same-type (no-cast) case. The vectorized path is gated at compile time (is_same<src0_t, dst_t>) and at runtime on 16-byte alignment of the base pointers and all row strides and on ne00 % VEC == 0. Vectorizing divides the block count by VEC, so a small single-row gather can drop below the device CU count and regress; an occupancy gate keeps those on the block-rich scalar path. On Strix Halo (gfx1151) the DeltaNet recurrent-state gather (ne00=524288) drops 18.6us -> 13.0us (rocprofv3 HW timestamps), faster than the Vulkan backend, with no regression on the small conv-state gather; total get_rows -27%. test-backend-ops GET_ROWS passes (47/47). Assisted-by: Claude Opus 4.8 * ggml-openvino: Add GGML_BACKEND_DL_IMPL invocation for OpenVINO backend (llama/25795) This adds the missing `GGML_BACKEND_DL_IMPL()` macro invocation, that other backends have. Fixes #25586 for me * vulkan: Refactor vk_queue to use per-instance mutexes and unique handles (llama/23570) * Refactor vk_queue to use per-instance mutexes and unique handles * integrates VK_KHR_internally_synchronized_queues, abstracting the queue submission into a polymorphic interface that completely bypasses host-side mutex locking when driver-side synchronization is supported * fix compilation error * fix duplicate pNext chain for VkPhysicalDeviceInternallySynchronizedQueuesFeaturesKHR * add fallback defines for VK_KHR_internally_synchronized_queues * add null checks for queues in vk_device_struct destructor * use unique_ptr for outer queues to enforce exclusive ownership and optimize lifetime * use static constexpr for eInternallySynchronizedKHR * add lock guard to ggml_vk_create_aliased_queue for thread safety * initialize sync_query_features.internallySynchronizedQueues to VK_FALSE * reuse sync_query_features for internallySynchronizedQueues and simplify chaining * refactor internallySynchronizedQueues detection * fix internallySynchronizedQueues query guard * use eInternallySynchronizedKHR constant * fix self-referential alias for eInternallySynchronizedKHR * use macro for eInternallySynchronizedKHR fallback * fix internallySynchronizedQueues query timing in ggml-vulkan.cpp to prevent device creation mismatch * reset sync_query_features.pNext before reusing in device creation chain, also removed the redundant second probe call * refactor internally synchronized queues detection to use chained feature query and avoid redundant API calls * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * rename sync_enable_features to internally_synchronized_queues_features * queue_flags is still computed before has_internally_synchronized_queues is set * fix trailing whitespace * replace eInternallySynchronizedKHR macro with static constexpr * preserve source queue semantics in single-queue aliased transfer queue * vulkan: fix cmd_pool access via pointer for compute_queue unique_ptr * vulkan: lock queue during debug label emission when not internally synchronized --------- Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * kleidiai : warn once when a weight type has no KleidiAI kernel (llama/25701) * cuda: add sqrt_softplus in topk-moe for dsv4 (llama/25896) * hexagon: check tensor type when reusing descriptors (llama/25968) * cuda: GET_ROWS quants (llama/25962) * cuda: add k-quant support to GET_ROWS Device-side embedding lookups require GET_ROWS to handle the k-quants used by common GGUF recipes (Q4_K_M stores token_embd as q6_K). Without it the backend rejects the op and the scheduler falls back to the host, copying the full embedding matrix back on every token in single-device graphs. Factor the super-block dequantizers out of the dequantize_block kernels in convert.cu into shared device functions in dequantize.cuh and reuse them from a new k_get_rows_kq kernel : one thread block dequantizes one (dst row, super-block) pair with the existing thread layouts, 32 threads for q4_K and 64 for the other k-quants. Covers q2_K to q6_K in get_rows_cuda and supports_op. i-quants are left as a TODO. * cuda: add i-quant support to GET_ROWS Extends the shared super-block dequantizers to the nine i-quants and reuses them from k_get_rows_kq with the 32-thread layout of the matching convert.cu kernels. supports_op gates the k-quant and i-quant path on ne0 being a multiple of QK_K, which iq4_nl does not guarantee on its own (QK4_NL sub-blocks). mxfp4 is left as a TODO. * cuda: add mxfp4 support to GET_ROWS Moves the mxfp4 dequantizer into the shared super-block helpers and reuses it from k_get_rows_kq with the 32-thread layout of the matching convert.cu kernel. mxfp4 joins the ne0 % QK_K gate in supports_op since its 32-value sub-blocks do not guarantee QK_K-aligned rows on their own. This closes GET_ROWS type coverage on CUDA: every quantized GGML type now takes the direct device path. * cuda: gate the GET_ROWS row size only for 32-value sub-block types Address review from @pwilkin: the i-quant commit replaced the return shared by the whole supported type cascade, so f16/f32/bf16/i32 and the legacy quants also inherited the ne0 % QK_K == 0 gate and any row size that is not a multiple of 256 fell back to the scheduler. Split the cascade: unconditional support is restored everywhere, the gate stays only on iq4_nl and mxfp4 whose 32-value sub-blocks do not guarantee the QK_K super-blocks the kernel iterates on. * webgpu : add CONV_2D_DW (depthwise conv2d) kernel (llama/25847) * webgpu : add CONV_2D_DW (depthwise conv2d) kernel Implement GGML_OP_CONV_2D_DW for the WebGPU backend, ported from the Vulkan backend's conv2d_dw.comp. Assisted-by: Claude Opus-4.8 * Remove unnecessary comments in webgpu support * update supported ops tables, triggered by adding webgpu CONV_2D_DW * ggml: enable PowerPC backend variants on AIX (llama/25983) * ggml: enable PowerPC backend variants on AIX Allow the PowerPC CPU backend variants to be built on AIX by extending the platform check in the CMake configuration. This reuses the existing PowerPC backend implementations without changing their behavior. Also fix a missing semicolon in the PowerPC Q0 matmul implementation. * Fix missing semicolon in sgemm.cpp * hexagon: activation ops update (llama/25974) * hex-geglu: optimized all-in-one geglu microkernel * hex-geglu: enable non-contiguous src and strided DMA * hex-act: enable non-contiguous srs and strided DMA for rest of ACT ops * hex-act: generalize GLU per-thread functions via DEFINE_GLU_PER_THREAD macro * hexagon: move UNARY_SILU and UNARY_GELU to unary-ops * hex-act: replace the generic ops_context scratchpad usage with a local htp_vtcm_layout computation per act op. --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> * CUDA: Improve NVFP4 W4A4 activation quantization (llama/25730) * Squash history before conflict-resolution during rebase on master WIP commit Add 32-byte loads, restore per-block amax Use nvfp4x4 intrinsic when available Fuse per-channel amax and quantization kernels Do pointer arithmetic only once on x Remove unnecessary ternary in the load We assert on host side that ne00 is 64-aligned Add back scale-search, but optimize it with intrinsics Code cleanup Make scale in MMQ-epilogue NVFP4-specific/restrictive for now Remove unneeded include, add comment Fix trailing whitespace Guard __builtin_align__(32) struct to NVIDIA Seems like HIP doesn't have this available, see https://github.com/ggml-org/llama.cpp/actions/runs/29438651734/job/87431623001 * compiler massaging to avoid unnecessary LDCs * kvalues_mxfp4 -> kvalues_nvfp4 in quantize_mmq_nvfp4 * Always pass in src1_scale.ptr * Extract ggml_cuda_is_aligned helper * metal : add f16 type support to leaky relu (llama/25981) * CUDA: fix external compilation of q1_0 MMQ (llama/25778) * hexagon: fix Windows crash when op_poll is enabled (llama/26029) * hexagon: further improved pipeline of the core bits (L2, DMA, MM, FA) (llama/26049) * hex-l2: use dirty ranges for flushing * hex-l2: simplify range based flush logic * hex-l2: optimize dirty range scans * hex-hvx: support for reduce_max_i32 * hex-mm: optimize fused MUL_MAT+ADD to use vtcm for bias when it fits * hex-mmid: optimize mmid row-mapping generation * hex-mmid: optimize mmid row-mapping generation * hex-mmid: optimize mmid row-mapping generation (round2) * hmx-mm: optimize output proc by tiling (col-chunking) * hex-fa: start the next q dmas a bit earlier * hex-fa: prefetch Q even earlier * hvx-fa: optimize softmax to keep things in hvx registers * hex-fa: hoist const register init in softmax loop * hmx-fa: kick off next-qkv DMAs before o-proc * hmx-fa: hoist various checks out of the inner loop * hmx-fa: adjust the cost model to better balance softmax work across hvx threads * hmx-fa: overlap diag rescale build with last HMX task * hmx-fa: optimize idx update in output proc * hmx-fa: unroll the softmax loops for improved perf * hmx-fa: overlap qk-dot with softmax, double-buffer p and s tiles * hex-trace: double the default number of trace entries * hex-trace: add trace events for opbatch and buffer mgmt * hex-trace: overhaul tracing to simplify runtime event handling and support opbatch stats * hex-trace: replace ascii timeline diagram with pipeline bubbles detector * hex-trace: handle missing start/stop events * hex-dma: always log stop/start trace events even for dummy dmas * hex-scripts: fix flake warnings * opencl: do not treat NULL-mask flash attention as causal (llama/25771) * opencl: cache compiled cl_program binaries on disk (llama/26050) * HIP: remove rocWMMA FlashAttention (llama/26046) * Update ggml/src/gguf.cpp : Defined virtual keyword for destructor of gguf_writer_base (llama/25867) Without a virtual destructor, deleting a derived object through a base-class pointer only invokes the base destructor, skipping the derived one. * hexagon: partial im2col support (llama/26007) * hexagon: add IM2COL op Add Hexagon IM2COL support targeting only patch-embedding convolutions. * hexagon: im2col refactor and cleanup * hex-im2col: instrument and update im2col. * hex-im2col: add local htp_vtcm_layout computation. * opencl: fix fused RMS norm mul view offset (llama/26085) * ggml-cpu: Enable BF16 tiled gemm optimization on PowerPC (llama/26068) * ggml : adjust logic for offloading ops to weight's backend (llama/25832) * ggml : adjust logic for offloading ops to weight's backend * llama : dsv4 graph fixes * sycl(build): parallelize ocloc invocations (llama/25903) * Disable -ffast-math on HIP (llama/25495) * ggml-metal: FWHT kernel for metal backend (llama/25924) * metal fwht wip * shape guard and formatting * formatting * Formatting and typos Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> * fix narrowing issue Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> * cont : minor style --------- Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * sycl: fix use-after-return of the SDPA scale in the oneDNN flash-attention path (llama/25880) * sycl: fix use-after-return of the SDPA scale in the oneDNN flash-attention path The scale was uploaded with an async memcpy sourced from a stack local. On the in-order queue that copy is ordered behind the K/V staging kernels; once n_kv is large enough (>= ~26k observed on Arc Pro B70) the staging outlives the host stack frame and the copy reads recycled memory, feeding the SDPA a garbage scale. Output then collapses to a single repeated token and the KV cache is poisoned for the rest of the session. Short contexts win the race by accident, and test-backend-ops caps FLASH_ATTN_EXT at kv=1024, which is why CI never caught it. The previous device_count > 1 wait_and_throw() gate (and reverting it, PR #25741) fixes the symptom only by keeping the frame alive across the copy at the cost of a host sync on every FA call. Fix: cache one device scalar per (device, value) -- the scale is constant per model -- and upload it synchronously once. The single-device fast path (no per-call host sync) is then safe: every device-side hazard already serializes on the in-order queue. The multi-GPU conservative wait is kept unchanged. Also: - GGML_SYCL_FA_ONEDNN_MAX_KV env (0 = unlimited): optional n_kv ceiling that routes very long sequences to the native FA kernel. - test-backend-ops: FLASH_ATTN_EXT F16 cases up to kv=65536 (Qwen3.6-27B geometry hsk=hsv=256 GQA 6, and hsk=128 GQA 4), closing the kv=1024 blind spot. Note the race itself needs a live multi-op pipeline to reproduce; single-op runs pass even on broken builds. Verified on Arc Pro B70 (bmg_g31), Qwen3.6-27B Q4_K, -c 131072: output byte-identical at temp 0 to the native FA path through 32k-deep prefill, with prefill depth-flat at 820-840 t/s (vs 340-350 native at 32k depth). Assisted-by: Claude Fable 5 * sycl: handle GGML_SYCL_FA_ONEDNN_MAX_KV like the other runtime env vars and document it Review feedback on #25880: - read the variable once at backend init into g_ggml_sycl_fa_onednn_max_kv via ggml_sycl_get_env, and print it in the startup env listing (-lv 4 shows it) - document GGML_SYCL_FA_ONEDNN and GGML_SYCL_FA_ONEDNN_MAX_KV in the SYCL.md runtime table Also trim the added FLASH_ATTN_EXT cases to kv={4096,16384}: the 32768/65536 shapes exceed the legacy NMSE threshold on both the oneDNN and native kernels (long-sequence fp16 accumulation drift, present before this PR) and would fail CI for an unrelated reason. Assisted-by: Claude Fable 5 * sycl: clarify GGML_SYCL_FA_ONEDNN_MAX_KV default is disabled Assisted-by: Claude Fable 5 * sycl: state default behavior of GGML_SYCL_FA_ONEDNN_MAX_KV explicitly Assisted-by: Claude Fable 5 * Update ggml/src/ggml-sycl/fattn-onednn.cpp Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com> * sycl: write the SDPA scale from a kernel instead of caching it The per-(device, value) scale cache was a function-local static unordered_map with no synchronization, so concurrent backend instances could access and rehash it at the same time. Write the scalar with a single_task instead. The value is captured into the command, so no host memory has to outlive the call -- which is what the use-after-return fix needed in the first place. That removes the shared container, the leaked device allocation and the string key, and it also closes the remaining async-memcpy-from-a-stack-local on the first flash-attention call. Ordering does not rely on timing: the queue is created with sycl::property::queue::in_order and the dnnl stream wraps that same queue, so the write completes before the SDPA reads the scalar. The multi-GPU wait_and_throw() branch is unchanged. Also drop the <cstdlib> include, which is unused. Assisted-by: Claude Opus 5 --------- Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com> * ggml-cuda: add chunked SSD matmul for Mamba-2 prefill acceleration (llama/22675) * ggml-cuda: add chunked SSD matmul for Mamba-2 prefill acceleration * cuda: added SSD CICD fixes for CUDA / HIP / MUSA / MSVC. * ggml-cuda: review comments fixed. * ggml-cuda: Fuse M matrix materialization into pre_matmul kernel and enabled test. * ggml-cuda: test updates and fixes * ggml-cuda: test updates to remove hardcoding of tensor initialise data limits. * ggml-cuda: ssd minor review comment fixed. * ggml-cuda: ssd minor CICD fixed. * CUDA SSD: Fixes correctness by promoting s0_stride_seq to int64_t, improves memory coalescing in ssm_ssd_prepare_dt_kernel, and boosts efficiency by merging B_weighted and C_scaled; also addresses prior review comments. * cuda: fix sdata read-write race in prepare_dt fallback scan loop * vulkan: add iq4_nl support back to FA (llama/24585) * vulkan: add iq4_nl support back to FA I was originally concerned about wasting shared memory on the LUT, but it's small and unlikely to matter in practice. Also support q1_0 for non-coopmat2. Fixes #23681 * remove q1_0 FA support * ggml : set output of view src (llama/25729) * llama-graph: set_outputs to t->view_src * change set_output to GGML_ASSERT about views not being outputs * sampler : avoid views in outputs * cont : fix dist sampler * cont : consistent logits handling * ggml : set output of view src * graph : simplify set_outputs() * cont : cleanup Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> * opencl: skip the Adreno KQ/KQV image kernels for multi-stream batches (llama/26189) The Adreno KQ/KQV image1d kernels (ggml_cl_mul_mat_kq_kqv_adreno) ignore dim 3 entirely: the sub-buffer covers only nb02*ne02 bytes and the kernel receives no ne03/ne13/nb03/nb13 arguments. With the unified KV cache, multi-sequence batches (e.g. llama-perplexity with its default -b 2048, n_seq=4, or a multi-slot llama-server) present KQ/KQV as 4D tensors with ne3 = n_stream, so every stream past the first reads the first stream's K/V and produces garbage. Flash attention masks the bug where it is enabled; devices where FA is declined (e.g. Adreno 740) hit it with default settings. Route ne03/ne13 > 1 to the general path, which handles dim 3, and honor view_offs when creating the sub-buffers (currently always 0 for tensors reaching this function, but the function would silently misread any future view). Llama-3.2-1B-Instruct Q4_0, wiki.test.raw, 8 chunks, -ngl 99: - Adreno 740, default: PPL 1817.64 -> 15.61 - Adreno 740, -fa 0: PPL 1941.64 -> 15.61 - Adreno 840, -fa 0: PPL 1943.90 -> 15.50 - single-stream (-b 512) results unchanged (15.6090) - test-backend-ops -o MUL_MAT on 740: identical before/after (909 OK, 12 pre-existing q6_K failures) * ggml-webgpu: Fix some binding alias issues to support all archs, fix recurrent-state-rollback test (llama/25931) * Add overlap glu variant to support all archs, fix recurrent-state-rollback test * format * Fix all arch overlapped ranges * format * diagnose bus error on apple ci * More testing * more testing * more targeted testing * Fix bug in alignment for > 4gb buffer offsets * Fix bug in view offsets * Try avoiding multi_buffers * not fixed yet, more logging :( * Handle edge case in set_rows * Try looking at view source * Skip deepseek32 for now and clean up trace infrastructure * simplify skipping * last cleanup * actually final cleanup * update handling of overlap * format * try skipping other failing model * add rdna3.5, and 3 to mmq configs so they can be tuned independently. (llama/26199) * RPC: add tensor_memset (llama/25912) * sycl: contiguous fast path + 32-bit index math for unary elementwise ops (llama/25946) * sycl: contiguous fast path + 32-bit index math for unary elementwise ops * sycl: use fastdiv for elementwise index math * ggml-cuda : disable MMQ on devices with less than 48 KiB shared memory (llama/26141) ggml_cuda_should_use_mmq() selects MMQ purely from the quantization type. The current MMQ configurations are designed and maintained against a minimum of 48 KiB per-block shared memory, the limit provided by NVIDIA Pascal GPUs and later. On devices that report less, no supported MMQ tile fits and mul_mat_q_switch_J() aborts when every tile size exceeds the device's per-block shared memory budget. Disable MMQ when smpbo < 48 KiB so the caller falls back to the BLAS path instead of hitting GGML_ABORT. Some current MUSA QY1 devices report only 28 KiB and are covered by this guard. Reproduced on a Moore Threads MTT S70 (arch mp_21, 28 KiB shared memory per block) with an RWKV-7 0.1B Q8_0 model: $ llama-bench -m rwkv7-g1d-0.1b-Q8_0.gguf -p 128 -n 0 J_best=0 ggml/src/ggml-cuda/template-instances/../mmq.cuh:1521: fatal error (core dumped) Only prefill (batch > 1) is affected; token generation is fine. After the fix the same device falls back to the BLAS path: Q8_0 pp128 1470.7 t/s, tg8 55.3 t/s (was: abort) FP16 unchanged Q4_K_M unchanged This matches a -DGGML_CUDA_FORCE_CUBLAS=ON build (pp128 1464.2 t/s), which confirms the fallback path is the one being taken. This is not MUSA-specific: any device with less than 48 KiB per-block shared memory is affected. Co-authored-by: KakaruHayate <KakaruHayate@users.noreply.github.com> * ggml : Fix issue with kleidiai ci and stringop overflow warning (llama/26277) Signed-off-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com> * metal: fix memory unwire if model is freed without any GPU operations (llama/26082) * metal: fix memory leak if model is freed without any GPU operations * metal: run dummy work only if residency sets are used * metal: wrap function in #if defined * metal: measure system-wide wired memory in test * metal: always build regression test Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> --------- Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> * CUDA: add Q2_0 support (llama/25707) * ggml : bump version to 0.18.0 (ggml/1576) * sync : ggml * parakeet : update parakeet test models generation This commit updates the parakeet test models to ensure that the random values generated for mel filters are not negative. This will otherwise cause a debug assertion and fail the parakeet-test. Refs: https://github.com/ggml-org/whisper.cpp/actions/runs/30547589734/job/90887624319?pr=3962 --------- Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> Signed-off-by: Francois Dugast <francois.dugast@intel.com> Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com> Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> Signed-off-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com> Co-authored-by: Aparna M P <aparmp@qti.qualcomm.com> Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de> Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> Co-authored-by: Martin Chang <marty1885@users.noreply.github.com> Co-authored-by: Vidas <vidas@nuolat.lt> Co-authored-by: Gianluca Guida <glguida@tlbflush.org> Co-authored-by: Gianluca Guida <gianluca@nekko.ai> Co-authored-by: ubergarm <leimgrub@gmail.com> Co-authored-by: SaqibAkram-10xE <saqib.akram@10xengineers.ai> Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai> Co-authored-by: Li He <lih@qti.qualcomm.com> Co-authored-by: Raman Shinde <raman.shinde15@gmail.com> Co-authored-by: cphlipot <9103367+cphlipot@users.noreply.github.com> Co-authored-by: Rohit Mahesh <74331568+rohitmahesh1@users.noreply.github.com> Co-authored-by: Todd Malsbary <todd.malsbary@intel.com> Co-authored-by: Frosty40 <newjordan@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: QuintinShaw <yx6f20@soton.ac.uk> Co-authored-by: Jeff Bolz <jbolz@nvidia.com> Co-authored-by: Pasha Khosravi <khosravipasha@users.noreply.github.com> Co-authored-by: Titaniumtown <titaniumtown@proton.me> Co-authored-by: Charles Xu <charles.xu@arm.com> Co-authored-by: JusteLeo <leonard.adamo66@gmail.com> Co-authored-by: Chyan <163109379+chyan8@users.noreply.github.com> Co-authored-by: hmscider <201289679+hmscider@users.noreply.github.com> Co-authored-by: scientist3 <scientist.3@users.noreply.github.com> Co-authored-by: hmscider <hmscider@users.noreply.github.com> Co-authored-by: maxious <81432+maxious@users.noreply.github.com> Co-authored-by: Francois Dugast <francois.dugast@intel.com> Co-authored-by: Andrew Smith <atsmith19@comcast.net> Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com> Co-authored-by: Michael Lamothe <michael.lamothe@gmail.com> Co-authored-by: Pascal <admin@serveurperso.com> Co-authored-by: leonardHONG <2695316095@qq.com> Co-authored-by: David Friehs <david@friehs.info> Co-authored-by: Pranesh Gonegandla <pranesh.iitp@gmail.com> Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com> Co-authored-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com> Co-authored-by: Alexander Heisler <126129661+heislera763@users.noreply.github.com> Co-authored-by: Anav Prasad <anavp@nvidia.com> Co-authored-by: Ruben Ortlam <rortlam@redhat.com> Co-authored-by: Rajendra Matcha <matcraje@qti.qualcomm.com> Co-authored-by: Aman Gupta <amangupta052@gmail.com> Co-authored-by: akleine <alb.kleine@gmx.de> Co-authored-by: Gezahegne <gezahegne.yirefu@gmail.com> Co-authored-by: Aaron Teo <aaron.teo1@ibm.com> Co-authored-by: Todor Boinovski <todorb@qti.qualcomm.com> Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Markus Ebner <seijikun@users.noreply.github.com> Co-authored-by: Winston Ma <winstonma@ymail.com> Co-authored-by: Kamalesh VS <76260512+kkjjkamal123@users.noreply.github.com> Co-authored-by: Wei Wang <w10493wang@163.com> Co-authored-by: m1el <m1el@ya.ru> Co-authored-by: shalinib-ibm <Shalini.Salomi.Bodapati@ibm.com> Co-authored-by: Oliver Simons <osimons@nvidia.com> Co-authored-by: Ilia Ilmer <iliailmer@users.noreply.github.com> Co-authored-by: adgup-qti <adgup@qti.qualcomm.com> Co-authored-by: kumaal <44551860+kumaal@users.noreply.github.com> Co-authored-by: Yongmin Yoo 유용민 <yymin1022@gmail.com> Co-authored-by: yzyyzyhhh <96101183+happyyzy@users.noreply.github.com> Co-authored-by: Beinsezii <39478211+Beinsezii@users.noreply.github.com> Co-authored-by: Nick Lafleur <55208706+nicklafleur@users.noreply.github.com> Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> Co-authored-by: meatposes <computerdork@verizon.net> Co-authored-by: Bhavik Sharda <10757940+BLSharda@users.noreply.github.com> Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> Co-authored-by: Reese Levine <reeselevine1@gmail.com> Co-authored-by: Geramy Loveless <gloveless@jqluv.com> Co-authored-by: Kakaru <97896816+KakaruHayate@users.noreply.github.com> Co-authored-by: KakaruHayate <KakaruHayate@users.noreply.github.com> Co-authored-by: Jonathan Clohessy <jonathan.clohessy@arm.com> Co-authored-by: Niklas Wenzel <dev@nikwen.de> Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com> |