Piotr Wilkin (ilintar) 9a7d43df1e ggml-cuda : convert contiguous tensors four elements at a time (llama/29155) ...

convert_unary handles the contiguous case through the general strided kernel,
one element per thread: each lane reads 4 bytes and writes 2. Converting the
activations for a bf16 matrix multiplication that way moves 126 MB in 1021 us
on gfx1151, about 65% of what the memory system can do.

Give the contiguous path its own kernel that takes four elements per thread
through a vector type, so a warp loads 512 bytes at a time instead of 128. It
is used only when the element count is a multiple of four and both pointers
carry the alignment the vector type needs, and falls back to the strided
kernel otherwise.

Model level, Qwen3.8-Next-Flash IQ3_XXS on gfx1151, llama-bench -ub 2048 -r 6,
mean of the last 3 reps, ABBA counterbalanced:

    pp2048   688.0 680.0  ->  694.3 691.1   +1.26%
    tg128     24.8  24.8  ->   24.8  24.8   +0.14%

Every conversion in a prefill takes the new kernel (kernel trace: 1146
convert_unary_cont_vec4, no convert_unary). Output is bit identical; MUL_MAT,
MUL_MAT_ID, CPY, CONT, GET_ROWS and SET_ROWS pass.

Assisted-by: Claude Opus 5

2026-09-23 20:46:47 +03:00

..

cmake

CUDA: replace GGML_FA_ALL_QUANTS with GGML_FA_QUANTS, more control over what is compiled (llama/28079)

2026-09-14 20:45:06 +03:00

include

sycl : pinned memory use right device context instead of 0 (llama/28895)

2026-09-23 20:46:47 +03:00

src

ggml-cuda : convert contiguous tensors four elements at a time (llama/29155)

2026-09-23 20:46:47 +03:00

.gitignore

…

CMakeLists.txt

ggml : bump version to 0.24.0 (ggml/1627)

2026-09-14 20:45:06 +03:00