public_git
/
whisper.cpp
mirror of https://github.com/ggml-org/whisper.cpp.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
whisper.cpp/ggml/src/ggml-opencl/kernels
History
Hongqiang Wang 3a1387caf8 opencl: general flash attention decode performance optimizations (llama/25366) 
* opencl: vec flash-attention decode kernels for f16/q8_0/q4_0 KV

* opencl: improve non FA KQ mv kernels

* opencl: tweaks for multiquery FA

* opencl: some tweaks for FA q1 kernels

* opencl: FA with DK=DV=512 for gemma-4

* opencl: various fixes

* opencl: cleanup

* opencl: fix FA decode crash for DK=512 (gemma-4)

The DK=512 decode-only program does not create the f32_f16 prefill
kernel, so the compiled check in ensure_fa_variant never hit and
supports_op gave inconsistent answers for the same op. block_n is also
unset for DK=512 decode; guard it to avoid an out-of-range read at
dispatch.

* opencl: run DK=512 FA decode on CPU

DK=512 decode is bandwidth-bound and faster on the CPU than the GPU,
increasingly so with depth. Decline it in supports_op; prefill stays on the GPU.

* opencl: compile MQ_GQA=8 FA kernels in a minimal program

The full program compiled with -D MQ_GQA=8 runs the Adreno compiler out
of memory at DK>=256. Only the vec_mq kernels are used from this
program, so compile it with FA_MQ_ONLY, which excludes everything else.
Also include the program name in the compile error log.

* opencl: remove stray token in flash_attn_f32_f16.cl

A stray "." broke the f32_f16 program build.

* opencl: split f16-KV FA decode finer (FD_KV_PER_SPLIT_F16)

The 2048 default under-fills the GPU on single-query f16-KV decode;
use 512 for f16 KV to get more splits. Quantized KV keeps 2048.

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
 		2026-07-10 13:06:42 +03:00
..
add.cl opencl: allow mixed f16/f32 `add` (llama/15140) 2025-08-18 20:30:45 +03:00
add_id.cl opencl: add `swiglu_oai` and `add_id` (llama/15121) 2025-08-18 20:30:45 +03:00
argsort.cl opencl: add new ops - `argsort`, `div`, `sub`, `addrows`, `sigmoid`, `group_norm` (llama/13787) 2025-06-01 15:14:44 +03:00
clamp.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
concat.cl opencl: improve get_rows, cpy, concat and q6_k flat gemv (llama/24160) 2026-06-08 14:36:36 +03:00
conv2d.cl opencl: add conv2d kernel (llama/14403) 2025-07-28 13:02:32 +03:00
conv2d_f16_f32.cl opencl: add conv2d kernel (llama/14403) 2025-07-28 13:02:32 +03:00
cpy.cl opencl: improve get_rows, cpy, concat and q6_k flat gemv (llama/24160) 2026-06-08 14:36:36 +03:00
cumsum.cl opencl: add cumsum op (llama/18981) 2026-03-16 13:10:15 +02:00
cvt.cl opencl: initial q1_0 support (llama/25160) 2026-07-10 13:06:42 +03:00
diag.cl opencl: add neg, exp and diag (llama/20127) 2026-03-16 13:10:15 +02:00
diag_mask_inf.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
div.cl opencl: add f16 for `add`, `sub`, `mul`, `div` (llama/14984) 2025-08-18 20:30:45 +03:00
embed_kernel.py ggml : add opencl backend (skip) (llama/10693) 2025-01-14 10:38:01 +02:00
exp.cl opencl: add neg, exp and diag (llama/20127) 2026-03-16 13:10:15 +02:00
expm1.cl opencl: refactor expm1 and softplus (llama/19404) 2026-02-27 20:57:58 +02:00
fill.cl opencl: add FILL op support (llama/18682) 2026-01-14 09:11:59 +02:00
flash_attn_f16.cl opencl: general flash attention decode performance optimizations (llama/25366) 2026-07-10 13:06:42 +03:00
flash_attn_f32.cl opencl: general flash attention decode performance optimizations (llama/25366) 2026-07-10 13:06:42 +03:00
flash_attn_f32_f16.cl opencl: general flash attention decode performance optimizations (llama/25366) 2026-07-10 13:06:42 +03:00
flash_attn_f32_q4_0.cl opencl: general flash attention decode performance optimizations (llama/25366) 2026-07-10 13:06:42 +03:00
flash_attn_f32_q8_0.cl opencl: general flash attention decode performance optimizations (llama/25366) 2026-07-10 13:06:42 +03:00
flash_attn_pre_f16.cl opencl: flash attention improvement (llama/25069) 2026-07-10 13:06:42 +03:00
gated_delta_net.cl Remove padding and multiple D2D copies for MTP (llama/24086) 2026-06-15 10:33:53 +03:00
gelu.cl opencl: add GELU_ERF (llama/14476) 2025-07-12 19:23:56 +03:00
gemm_moe_mxfp4_f32.cl opencl: transposed gemm/gemv moe kernel with mxfp4,f32 (llama/16602) 2025-10-22 12:58:11 +03:00
gemm_moe_mxfp4_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemm_moe_q4_0_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemm_moe_q4_1_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemm_moe_q4_k_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemm_moe_q5_0_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemm_moe_q5_1_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemm_moe_q5_k_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemm_moe_q6_k_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemm_noshuffle_iq4_nl_f32.cl opencl: add iq4_nl support (llama/22272) 2026-04-30 11:29:19 +03:00
gemm_noshuffle_q1_0_f32.cl opencl: initial q1_0 support (llama/25160) 2026-07-10 13:06:42 +03:00
gemm_noshuffle_q4_0_f32.cl opencl: refactor Adreno q4_0 (llama/22335) 2026-05-14 21:26:48 +03:00
gemm_noshuffle_q4_1_f32.cl opencl: add optimized q4_1 mm kernel for adreno (llama/19840) 2026-03-16 13:10:15 +02:00
gemm_noshuffle_q4_k_f32.cl opencl: add q4_K gemm and gemv kernels for Adreno (llama/20919) 2026-04-30 11:28:57 +03:00
gemm_noshuffle_q5_0_f32.cl opencl: add q5_0/q5_1 gemm and gemv kernels for Adreno (llama/24319) 2026-06-15 10:33:53 +03:00
gemm_noshuffle_q5_1_f32.cl opencl: add q5_0/q5_1 gemm and gemv kernels for Adreno (llama/24319) 2026-06-15 10:33:53 +03:00
gemm_noshuffle_q5_k_f32.cl opencl: add q5_K gemm and gemv kernels for Adreno (llama/21595) 2026-04-30 11:29:11 +03:00
gemm_noshuffle_q6_k_f32.cl opencl: add q6_K gemm and gemv kernels for Adreno (llama/20089) 2026-03-29 15:04:36 +03:00
gemm_noshuffle_q8_0_f32.cl opencl: refactor Adreno q4_0 (llama/22335) 2026-05-14 21:26:48 +03:00
gemm_xmem_f16_f32_os8.cl opencl: add opt-in Adreno xmem F16xF32 GEMM for prefill (llama/22755) 2026-05-14 21:26:48 +03:00
gemv_moe_mxfp4_f32.cl opencl: transposed gemm/gemv moe kernel with mxfp4,f32 (llama/16602) 2025-10-22 12:58:11 +03:00
gemv_moe_mxfp4_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemv_moe_q4_0_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemv_moe_q4_1_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemv_moe_q4_k_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemv_moe_q5_0_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemv_moe_q5_1_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemv_moe_q5_k_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemv_moe_q6_k_f32_ns.cl opencl: generalize Adreno MoE kernels on M (llama/23449) 2026-05-25 12:26:07 +03:00
gemv_noshuffle_iq4_nl_f32.cl opencl: add iq4_nl support (llama/22272) 2026-04-30 11:29:19 +03:00
gemv_noshuffle_q1_0_f32.cl opencl: initial q1_0 support (llama/25160) 2026-07-10 13:06:42 +03:00
gemv_noshuffle_q4_0_f32.cl opencl: refactor Adreno q4_0 (llama/22335) 2026-05-14 21:26:48 +03:00
gemv_noshuffle_q4_0_f32_spec.cl opencl: refactor Adreno q4_0 (llama/22335) 2026-05-14 21:26:48 +03:00
gemv_noshuffle_q4_1_f32.cl opencl: add optimized q4_1 mm kernel for adreno (llama/19840) 2026-03-16 13:10:15 +02:00
gemv_noshuffle_q4_k_f32.cl opencl: add q4_K gemm and gemv kernels for Adreno (llama/20919) 2026-04-30 11:28:57 +03:00
gemv_noshuffle_q5_0_f32.cl opencl: add q5_0/q5_1 gemm and gemv kernels for Adreno (llama/24319) 2026-06-15 10:33:53 +03:00
gemv_noshuffle_q5_1_f32.cl opencl: add q5_0/q5_1 gemm and gemv kernels for Adreno (llama/24319) 2026-06-15 10:33:53 +03:00
gemv_noshuffle_q5_k_f32.cl opencl: add q5_K gemm and gemv kernels for Adreno (llama/21595) 2026-04-30 11:29:11 +03:00
gemv_noshuffle_q6_k_f32.cl opencl: add q6_K gemm and gemv kernels for Adreno (llama/20089) 2026-03-29 15:04:36 +03:00
gemv_noshuffle_q8_0_f32.cl opencl: q8_0 gemv precision improvement (llama/24923) 2026-06-26 16:03:57 +03:00
get_rows.cl opencl: improve get_rows, cpy, concat and q6_k flat gemv (llama/24160) 2026-06-08 14:36:36 +03:00
glu.cl opencl: add `swiglu_oai` and `add_id` (llama/15121) 2025-08-18 20:30:45 +03:00
group_norm.cl OpenCL: add fused group_norm/norm, mul, add (llama/15314) 2025-09-20 13:42:43 +03:00
im2col_f16.cl opencl: fix `im2col` when `KW!=KH` (llama/14803) 2025-07-28 13:02:32 +03:00
im2col_f32.cl opencl: fix `im2col` when `KW!=KH` (llama/14803) 2025-07-28 13:02:32 +03:00
l2_norm.cl opencl: fix l2_norm (llama/20480) 2026-03-16 13:10:15 +02:00
mean.cl opencl: optimize mean and sum_row kernels (llama/19614) 2026-02-27 20:57:58 +02:00
moe_reorder_b.cl opencl: Adreno optimization for MoE - MxFP4 (llama/22301) 2026-05-14 21:26:48 +03:00
moe_sort_by_expert.cl opencl: Adreno optimization for MoE - MxFP4 (llama/22301) 2026-05-14 21:26:48 +03:00
mul.cl opencl: add f16 for `add`, `sub`, `mul`, `div` (llama/14984) 2025-08-18 20:30:45 +03:00
mul_mat_f16_f32.cl opencl: add tiled mul_mat_f16_f32 (llama/14535) 2025-07-12 19:23:56 +03:00
mul_mm_f16_f32_kq_kqv.cl opencl: add kernel to handle mat mul in attention to improve encoding speed (llama/17181) 2025-11-17 21:05:46 +02:00
mul_mm_f16_f32_l4_lm.cl opencl: fix boundary handling for mul_mm (llama/16875) 2025-11-09 23:38:03 +02:00
mul_mm_f32_f32_l4_lm.cl opencl: fix boundary handling for mul_mm (llama/16875) 2025-11-09 23:38:03 +02:00
mul_mm_iq4_nl_f32_l4_lm.cl opencl: add iq4_nl support (llama/22272) 2026-04-30 11:29:19 +03:00
mul_mm_q1_0_f32_l4_lm.cl opencl: initial q1_0 support (llama/25160) 2026-07-10 13:06:42 +03:00
mul_mm_q4_0_f32_l4_lm.cl opencl: add basic support for q4_1 (llama/19534) 2026-02-15 21:44:37 +02:00
mul_mm_q4_1_f32_l4_lm.cl opencl: add basic support for q4_1 (llama/19534) 2026-02-15 21:44:37 +02:00
mul_mm_q4_k_f32_l4_lm.cl opencl: add flattened Q4_K mv and general Q4_K mm (llama/20773) 2026-03-29 15:04:36 +03:00
mul_mm_q5_0_f32_l4_lm.cl opencl: add basic support for q5_0 and q5_1 (llama/23548) 2026-06-08 14:36:36 +03:00
mul_mm_q5_1_f32_l4_lm.cl opencl: add basic support for q5_0 and q5_1 (llama/23548) 2026-06-08 14:36:36 +03:00
mul_mm_q5_k_f32_l4_lm.cl opencl: add basic support for q5_k (llama/21593) 2026-04-30 11:29:06 +03:00
mul_mm_q6_k_f32_l4_lm.cl opencl: add general Q6_K mm and Q4_K mv (llama/19347) 2026-02-15 21:44:37 +02:00
mul_mm_q8_0_f32_l4_lm.cl opencl: fix boundary handling for mul_mm (llama/16875) 2025-11-09 23:38:03 +02:00
mul_mv_f16_f16.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
mul_mv_f16_f32.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
mul_mv_f16_f32_1row.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
mul_mv_f16_f32_l4.cl opencl: general flash attention decode performance optimizations (llama/25366) 2026-07-10 13:06:42 +03:00
mul_mv_f32_f32.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
mul_mv_id_mxfp4_f32.cl opencl: add initial mxfp4 support via mv (llama/15270) 2025-08-18 20:30:45 +03:00
mul_mv_id_mxfp4_f32_flat.cl opencl: optimize mxfp4 kernels (llama/16037) 2025-09-20 13:46:39 +03:00
mul_mv_id_q4_0_f32_8x_flat.cl opencl: add `mul_mv_id_q4_0_f32_8x_flat` (llama/14003) 2025-06-18 12:40:34 +03:00
mul_mv_id_q8_0_f32.cl opencl: initial `q8_0` mv support (llama/15732) 2025-09-29 15:18:08 +03:00
mul_mv_id_q8_0_f32_flat.cl opencl: initial `q8_0` mv support (llama/15732) 2025-09-29 15:18:08 +03:00
mul_mv_iq4_nl_f32.cl opencl: add iq4_nl support (llama/22272) 2026-04-30 11:29:19 +03:00
mul_mv_iq4_nl_f32_flat.cl opencl: add iq4_nl support (llama/22272) 2026-04-30 11:29:19 +03:00
mul_mv_mxfp4_f32.cl opencl: add initial mxfp4 support via mv (llama/15270) 2025-08-18 20:30:45 +03:00
mul_mv_mxfp4_f32_flat.cl opencl: optimize mxfp4 kernels (llama/16037) 2025-09-20 13:46:39 +03:00
mul_mv_q1_0_f32.cl opencl: initial q1_0 support (llama/25160) 2026-07-10 13:06:42 +03:00
mul_mv_q1_0_f32_flat.cl opencl: initial q1_0 support (llama/25160) 2026-07-10 13:06:42 +03:00
mul_mv_q4_0_f32.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
mul_mv_q4_0_f32_1d_8x_flat.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
mul_mv_q4_0_f32_1d_16x_flat.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
mul_mv_q4_0_f32_8x_flat.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
mul_mv_q4_0_f32_v.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
mul_mv_q4_1_f32.cl opencl: add basic support for q4_1 (llama/19534) 2026-02-15 21:44:37 +02:00
mul_mv_q4_1_f32_flat.cl opencl: add basic support for q4_1 (llama/19534) 2026-02-15 21:44:37 +02:00
mul_mv_q4_k_f32.cl opencl: add general Q6_K mm and Q4_K mv (llama/19347) 2026-02-15 21:44:37 +02:00
mul_mv_q4_k_f32_flat.cl opencl: add flattened Q4_K mv and general Q4_K mm (llama/20773) 2026-03-29 15:04:36 +03:00
mul_mv_q5_0_f32.cl opencl: add basic support for q5_0 and q5_1 (llama/23548) 2026-06-08 14:36:36 +03:00
mul_mv_q5_0_f32_flat.cl opencl: add basic support for q5_0 and q5_1 (llama/23548) 2026-06-08 14:36:36 +03:00
mul_mv_q5_1_f32.cl opencl: add basic support for q5_0 and q5_1 (llama/23548) 2026-06-08 14:36:36 +03:00
mul_mv_q5_1_f32_flat.cl opencl: add basic support for q5_0 and q5_1 (llama/23548) 2026-06-08 14:36:36 +03:00
mul_mv_q5_k_f32.cl opencl: add basic support for q5_k (llama/21593) 2026-04-30 11:29:06 +03:00
mul_mv_q5_k_f32_flat.cl opencl: add basic support for q5_k (llama/21593) 2026-04-30 11:29:06 +03:00
mul_mv_q6_k_f32.cl opencl: add flattened q6_K mv (llama/19054) 2026-01-30 15:56:40 +02:00
mul_mv_q6_k_f32_flat.cl opencl: improve get_rows, cpy, concat and q6_k flat gemv (llama/24160) 2026-06-08 14:36:36 +03:00
mul_mv_q8_0_f32.cl opencl: initial `q8_0` mv support (llama/15732) 2025-09-29 15:18:08 +03:00
mul_mv_q8_0_f32_flat.cl opencl: initial `q8_0` mv support (llama/15732) 2025-09-29 15:18:08 +03:00
neg.cl opencl: add neg, exp and diag (llama/20127) 2026-03-16 13:10:15 +02:00
norm.cl opencl: support non-contig rows in norm (llama/24965) 2026-06-26 16:03:57 +03:00
pad.cl opencl: support pad_ext (llama/15888) 2025-10-12 11:16:23 +03:00
relu.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
repeat.cl opencl: refactor some ops, concat, repeat, tanh and scale (llama/19226) 2026-02-08 09:29:10 +02:00
rms_norm.cl opencl: fix rms_norm_mul (llama/17250) 2025-11-17 21:05:46 +02:00
rope.cl opencl: support imrope (llama/16914) 2025-11-09 23:38:03 +02:00
scale.cl opencl: refactor some ops, concat, repeat, tanh and scale (llama/19226) 2026-02-08 09:29:10 +02:00
set_rows.cl opencl: flash attention improvement (llama/25069) 2026-07-10 13:06:42 +03:00
sigmoid.cl opencl: add new ops - `argsort`, `div`, `sub`, `addrows`, `sigmoid`, `group_norm` (llama/13787) 2025-06-01 15:14:44 +03:00
silu.cl opencl: split ggml-opencl.cl into multiple files and cleanup (llama/12886) 2025-04-24 20:39:16 +03:00
softmax_4_f16.cl opencl: support sink in `soft_max` (attn sinks) (llama/15152) 2025-08-18 20:30:45 +03:00
softmax_4_f32.cl opencl: support sink in `soft_max` (attn sinks) (llama/15152) 2025-08-18 20:30:45 +03:00
softmax_f16.cl opencl: support sink in `soft_max` (attn sinks) (llama/15152) 2025-08-18 20:30:45 +03:00
softmax_f32.cl opencl: support sink in `soft_max` (attn sinks) (llama/15152) 2025-08-18 20:30:45 +03:00
softplus.cl opencl: refactor expm1 and softplus (llama/19404) 2026-02-27 20:57:58 +02:00
solve_tri.cl OpenCL: add SOLVE_TRI op support (llama/18846) 2026-01-30 15:56:40 +02:00
sqr.cl opencl: add sqr, sqrt, mean and ssm_conv (llama/17476) 2025-12-12 17:53:09 +02:00
sqrt.cl opencl: add sqr, sqrt, mean and ssm_conv (llama/17476) 2025-12-12 17:53:09 +02:00
ssm_conv.cl opencl: add sqr, sqrt, mean and ssm_conv (llama/17476) 2025-12-12 17:53:09 +02:00
sub.cl opencl: add f16 for `add`, `sub`, `mul`, `div` (llama/14984) 2025-08-18 20:30:45 +03:00
sum_rows.cl opencl: optimize mean and sum_row kernels (llama/19614) 2026-02-27 20:57:58 +02:00
tanh.cl opencl: refactor some ops, concat, repeat, tanh and scale (llama/19226) 2026-02-08 09:29:10 +02:00
transpose.cl opencl: add optimized q4_1 mm kernel for adreno (llama/19840) 2026-03-16 13:10:15 +02:00
tri.cl opencl: add TRI op support (llama/18979) 2026-01-30 15:56:40 +02:00
tsembd.cl ggml : fix padding in timestep embedding kernels (llama/15932) 2025-09-20 13:45:30 +03:00
upscale.cl opencl : update upscale to support align corners (llama/14488) 2025-07-12 19:23:56 +03:00