public_git/whisper.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/whisper.cpp.git synced 2026-10-12 01:05:43 +02:00
Code Issues Packages Projects Releases Wiki Activity
5,382 Commits 67 Branches 95 Tags
8d2a2cb471e8cd43f4a5a2e9b5e8741a8cf22460
Commit Graph
5382 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Trivikram Reddy 8d2a2cb471 hexagon: optimize concat op (llama/29673) 
* hex-concat: reduce pkts in gather/transpose hot loop

gather directly into dst buffer, use special instruction for gather sync

* hex-concat: use fastdiv

replace calls to sw divide with fastpath

* hex-concat: optimize DMA-HVX pipeline and add transpose helpers
 2026-10-06 10:38:10 +03:00
Pascal 45093fc8f0 CUDA: bitonic argsort handles rows wider than one block (llama/28957) 
Without CUB (HIP, MUSA) argsort ran the bitonic kernel with one thread
per padded column, so any row above 1024 entries launched an invalid
block configuration. Each thread now owns several columns, every stage
of the network runs all owned columns before the barrier, and the block
is capped at 1024 threads. Shared memory becomes the only bound, which
supports_op checks against the device instead of a fixed 1024.

Rows up to 1024 run the same work as before. Bit-exact with the CUB
path on rows of 2048.
 2026-10-06 10:38:10 +03:00
thelittlefiremanandCarl Philipp Klemm 482baacdad ggml-cuda: HIP: optimize packed byte subtraction (__vsubss4 -> __vsub4) (llama/29478) 
* ggml-cuda: HIP: optimize non-saturating packed byte subtraction (`__vsubss4`)

* CI: ignore 1 spilled vgpr in fattn_vec

---------

Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz>
 2026-10-06 10:38:10 +03:00
Aaron Teo 3b86cabd47 ci: add zdnn backend build but not test (llama/29541) 
* ci: add zdnn backend build but not test

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: attempt to run a ubuntu 26.04 container

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: set shell to bash

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ci: clean up comments

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* ggml-zdnn: fix compiler errors

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

* vendor: attempt to ignore warnings from vendored files

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>

---------

Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2026-10-06 10:38:10 +03:00
lhez 9f12c9fe6a opencl: fix get_tensor for q5_K adreno gemm_nonshuffle kernel (llama/29555) 2026-10-06 10:38:10 +03:00
Ruben Ortlam 5439f3d969 vulkan: Tune GDN kernel, fix Intel performance (llama/29476) 
* vulkan: tune GDN shader

* tune for Intel
 2026-10-06 10:38:09 +03:00
Matt Corallo 6c07d8218b vulkan : Load F32 A matrix 2 at a time when its 2-aligned (llama/29254) 
It turns out Intel doesn't particularly like loading F32s one at a
time and we already have the _2aliagned load logic in mul_mat_vec,
so here we use it.

While we do already check all the requirements to load elements 4
at a time across [B]F16 and F32, it turns out [B]F16 loading 4 at a
time is sometimes slower on very specific shapes on Intel BMG.
Loading 4 at a time is a bit faster on F32, but its not material
and I assume might be slower on other platforms.

Note that we also need to validate `a_offset` is 2-aligned in
`mul_mat_vec.comp`, which was missing in the original 2-way-load
patch.

Some selected speedups from `test-backend-ops perf` on a B60.

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1704 runs -   767.17 us/run - 117.44 MFLOP/run - 153.08 GFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    2556 runs -   529.81 us/run - 117.44 MFLOP/run - 221.66 GFLOPS

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1704 runs -   727.13 us/run - 234.88 MFLOP/run - 323.03 GFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    2130 runs -   528.84 us/run - 234.88 MFLOP/run - 444.15 GFLOPS

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1704 runs -   702.19 us/run - 352.32 MFLOP/run - 501.74 GFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=3,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1988 runs -   532.14 us/run - 352.32 MFLOP/run - 662.08 GFLOPS

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1278 runs -   919.50 us/run - 469.76 MFLOP/run - 510.89 GFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=4,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1917 runs -   543.69 us/run - 469.76 MFLOP/run - 864.03 GFLOPS

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1197 runs -   892.12 us/run - 587.20 MFLOP/run - 658.21 GFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=5,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1881 runs -   575.17 us/run - 587.20 MFLOP/run -   1.02 TFLOPS

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1498 runs -   716.40 us/run - 939.52 MFLOP/run -   1.31 TFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                    1819 runs -   576.36 us/run - 939.52 MFLOP/run -   1.63 TFLOPS

  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                   134 runs -  7467.09 us/run -  60.13 GFLOP/run -   8.05 TFLOPS
  MUL_MAT(type_a=f32,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1,src_overlap=0):                   134 runs -  7478.12 us/run -  60.13 GFLOP/run -   8.04 TFLOPS
 2026-10-06 10:38:09 +03:00
Ankit Khandelwal 8cc3ca48c3 vulkan: MOE aware mat_mul_id tile selection (llama/29182) 
mut_mul_id selected its matmul tile with total token count.
For MoE dispatch grid the true N per workgroup is per-expert rows.
At pp128 on Sarvam 30B that is 6, not 128, so the picker took the l-tile for ~6 live rows.
Most workers in each group had nothing to do.
This wasted time. The slow part was 55% of the whole job.
 2026-10-06 10:38:09 +03:00
XZiarGeorgi GerganovXZiar
 		deae6f3439 ggml : fix c++ odr by properly using GGML_COMMON_DECL_CPP (llama/29504) 
* fix c++ odr by properly using GGML_COMMON_DECL_CPP

* using actual field rather than macro

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: XZiar <xziar@xziar.xziar>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-10-06 10:38:09 +03:00
Adrien Gallouët b97d419389 ggml : accumulate f16 dot products in f32 on AVX512-FP16 (llama/29545) 
Supersedes #29530

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
 2026-10-06 10:38:09 +03:00
Georgi Gerganov 4d4817ed25 ggml : require input tensors to be GGML_OP_NONE (llama/29647) 2026-10-06 10:38:08 +03:00
Aparna M P 5e1914a735 hexagon: add FP32 GELU_ERF and GEGLU_ERF support (llama/29631) 
* hexagon: add FP32 GELU_ERF and GEGLU_ERF support

* hex-erf: reduce register pressure in kernels
 2026-10-06 10:38:08 +03:00
Georgi Gerganov 763b67fd85 ggml : collect all input tensors into graph_inputs (llama/29634) 
graph_inputs was populated while splitting the graph, so it only
contained the inputs that are used as srcs of some node. With pipeline
parallelism (n_copies > 1), each graph input contributes n_copies leafs
to graph_copy, so switching between batches that consume different
inputs (e.g. token batches that do not use the embeddings input vs
image batches that do) changed the graph composition. This shifted the
input copies in graph_copy, making the backend ids comparison report
spurious changes and forcing the scheduler to re-reserve. The
re-reserve could then record smaller input sizes (e.g. out_ids with
n_outputs = 0) and abort later on a graph with an unchanged size via
GGML_SCHED_DEBUG_REALLOC.

Collect the inputs after the split instead, from all input leafs of the
graph, so that the graph composition depends only on which inputs
exist, not on which inputs are used.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
 2026-10-06 10:38:08 +03:00
Aaron Teo 847812906f ggml-zdnn: fix 0-row tensor crash (llama/29636) 
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
 2026-10-06 10:38:08 +03:00
Adrien GallouëtandJohannes Gäßler f449de9463 ggml : speed up model loading (llama/29598) 
* ggml : speed up model loading

A crafted model could hang the server for a very long time, try with:

    llama-cli -hf angt/test-gguf-1Mkv -hff model.gguf

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Avoid empty keys

Signed-off-by: Adrien Gallouët <angt@huggingface.co>

* Fix

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

---------

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
 2026-10-06 10:38:08 +03:00
bri-prism b30ef05f7b metal: FWHT perf optimizations (llama/29602) 
Assisted-by: Claude Code
 2026-10-06 10:38:08 +03:00
Andrew Lee eecee67b76 vulkan : reuse descriptor sets when bindings are constant (llama/29280) 
* vulkan : reuse descriptor sets when bindings are constant

* vulkan : bump buffer_destroy_count before destroying the buffer
 2026-10-06 10:38:07 +03:00
Linux User 97690105c4 vulkan: include functional header (llama/29597) 
Fixes the compile error: `no template named 'function' in namespace 'std'`
 2026-10-06 10:38:07 +03:00
Ravi Panchumarthy 830dc664df ggml-openvino: mark unaligned batch-stride views unsupported (llama/29603) 2026-10-06 10:38:07 +03:00
jbooth 81b951e012 webgpu: Handle unaligned writes in ggml_backend_webgpu_buffer_set_tensor (llama/29471) 
* Fix:  Handle unaligned writes in ggml_backend_webgpu_buffer_set_tensor

* Clang formatting
 2026-10-06 10:38:07 +03:00
Georgi Gerganov ee86a150bd tests : refactor test-recurrent-state-rollback (llama/29426) 
* tests : use llama_context_ptr in test-recurrent-state-rollback

Replace raw llama_context pointers with llama_context_ptr and drop the
manual llama_free calls and cleanup lambda.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL

* tests : run test-recurrent-state-rollback over all dummy models

Add a --models DIR mode that mirrors test-save-load-state: iterate every
dummy model, report PASS/FAIL/SKIP in a table and fail only when a model
fails. Register a single ctest entry with ARGS --models instead of the four
per-model registrations.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL

* cont : fix typo

* metal : allow fusing 0-element nodes to keep graph packing shape-independent

The fusion packing in ggml_metal_fusion_max excluded 0-element tensors and
the topk_moe/moe_reduce checks rejected n_tokens == 0, so graphs decoding
batches with no outputs packed differently from the worst-case reserved
graph. The Metal optimizer then reordered the nodes differently and
ggml_gallocr_needs_realloc failed on the layout mismatch, forcing an
unexpected graph re-reserve (caught by GGML_SCHED_DEBUG_REALLOC).

Treat empty tensors like their non-empty counterparts: match them in the
pattern sequence and only reject genuinely malformed shapes. Fused kernels
dispatch zero threadgroups for empty graphs, which is a legal no-op.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL

* tests : run test_multi_seq_split_replay as a separate test

test_multi_seq_split_replay was invoked at the end of test_rollback,
so its result was folded into the rollback status and it only ran when
the rollback part passed.

Give it its own test_status return, run both tests independently over
both cache fills via a shared run_tests helper, and report them as
separate rollback / split replay columns in the --models table with
per-test summaries. The exit code fails when either test fails.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL

* tests : loosen the split replay nmse bound to 1e-4

test-generate-models seeds its weights from std::random_device, and some
generated lfm2 models drift up to ~1.7e-5 nmse on the split replay due to
rounding noise, tripping the previous 1e-5 bound. Raise the bound to 1e-4
so the random generations stop flaking.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL

* tests : reuse run_tests_for_model in single-model mode

The single-model path duplicated the model init and the non-recurrent
check from run_tests_for_model; route it through the shared helper
instead. Model load failures now return FAIL rather than SKIP so that
--model with a broken file still exits non-zero, and the helper loads
with model_only like the --models loop does since the tests create
their own contexts.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL
 2026-10-06 10:38:07 +03:00
SXX 83fd1554ca ggml-cpu: enable tiled flash attention for non-vector-multiple head dims on x86 (llama/29423) 
* ggml-cpu: enable tiled flash attention for non-vector-multiple head dims on x86

* add AVX2 support for masked loading and storing in simd_gemm_ukernel_tail

* ggml-cpu: fix FA softcap handling for padded KV tiles
 2026-10-06 10:38:07 +03:00
François-Xavier Gsell 65449dbc40 vulkan: fuse qwen4exp's SCALE -> SIGMOID -> SCALE -> hc_post chain (llama/29520) 2026-10-06 10:38:06 +03:00
uvos 0c17f7d6bb HIP: fix template skip for DKQ > 256 mfma kernels (llama/29559) 2026-10-06 10:38:06 +03:00
Pascal 0e537510d6 metal: support left and circular padding in GGML_OP_PAD (llama/29561) 
* metal: support left and circular padding in GGML_OP_PAD

Align Metal with CPU, CUDA and Vulkan: shift the source coordinates by
the left paddings, wrap them around with the same wrap_around when
circular, and read the source through nb00, which also fixes a right
padding of a permuted source. A test case covers it.

Drop the f32_4 kernel: its selection is disabled as slower, and it
fails two pad cases once enabled.

* metal: use a function constant for the circular pad variant

Address review from ggerganov: replace the bool template with FC_PAD,
as FC_upscale_aa does, so the pad kernel is compiled once and
specialized per pipeline.
 2026-10-06 10:38:06 +03:00
Sarah Wu b7b849500e Enables Windows ARM64 build with MSVC cl.exe (llama/28362) 
* can reproduce the issue vlad sees

* fix fma issue

* drop volatile

* fix volatile runtime task

* add arm flag if needed

* fix hsum compile error

* fix syntax in quants

* strengthen sve probing

* make the syntax fixes one liners

* remove debug code

* formatting

* remove macro for float

* drive down gcc instruction count

* support armec

* fix CI comments

address CI comments

fix cross compile issue

remove warning

fix style and fix fma probing

fix style

* add documentation

* update documentation
 2026-10-06 10:38:06 +03:00
Pascal 85f6926153 vulkan: fix wrong results when a mul_mat reads a slice of a larger cache (llama/28956) 
* vulkan: read the batch stride of an in place src0 from nb[2]

A dim01 contiguous tensor can still be a view whose batches are
strided by more than ne[1] rows, the first rows of a KV cache for
example. Both the mat-vec and the matrix paths read such a tensor in
place but passed ne00*ne01 as the batch stride, so every head past
the first read the wrong rows. The same applies to src1. The stride
now comes from nb[2] whenever the tensor is used in place; the value
is unchanged for a contiguous tensor.

test-backend-ops gets an m_v parameter on test_mul_mat, the number of
rows of a in memory, and two cases at the shapes of a decoder self
attention over a cache.

* vulkan: size the in place A and B ranges by their strided extent

The matrix path bound src0 and src1 to the shader with a range of
elements times type size, which ends before the batches of a strided
view. Pipelines with bounded access read zero past that range, so the
same view that the mat-vec path already handles gave wrong results
on Intel and on NVIDIA without coopmat2. The range now comes from
ggml_nbytes when the tensor is read in place.

* vulkan: address review from jeffbolznv

Bind the in place A and B of the matrix path with ggml_vk_subbuffer,
which spans to the end of the buffer, so a strided view is in range
without computing its extent.

mul_mat_id reads the batch stride of an in place src0 and src1 with
the same helper as mul_mat. test_mul_mat_id gets an m_v parameter,
the number of rows of as in memory, and a case whose experts are
strided by more rows than it uses.

* vulkan: read the batch stride of an in place src0 in mul_mat_vec_id

The single token path of mul_mat_id passed ne00*ne01 as the batch
stride of A, so a strided expert view read the wrong rows. The stride
now comes from ggml_vk_batch_stride like the other three paths, and
src1 follows the same rule.

test_mul_mat_id gets a single token case over the strided view.

* vulkan: address review from jeffbolznv

The batch stride of an in place tensor is taken from nb[2] as
nb[2] / type_size * block_size, which holds when nb[2] is padded and
not a multiple of nb[1]. A test_mul_mat case with a padded batch stride
covers it.

* vulkan: keep the A and B ranges exact in mul_mm

The quantized A loads of mul_mm carry no row bound and rely on the
descriptor range to read zeros past the last row of a partial tile.
Binding A and B up to the end of the buffer let those tiles read the
leftovers of a previous node and hung the NVFP4 mul_mm on NVIDIA
without coopmat2. The range is the strided extent of a tensor read in
place and the staged size otherwise.
 2026-10-06 10:38:06 +03:00
lhez 51db47553c opencl: refine bin kernel loading condition (llama/29503) 2026-10-06 10:38:05 +03:00
bri-prism 55a695da12 sycl: FWHT kernels for block widths above 512 (llama/29243) 
The SYCL FWHT covers 64 to 512 via the standard butterfly network, plus
384/640/768/1280 via the Kronecker/Paley construction added separately in
Hadamard hint can produce (1024, 2048, 4096, 8192); those still fall through
to the default case and run as a dense GEMM against the materialized
rotation tensor, correct but O(n^2) instead of O(n log n).

fwht_kernel_wide runs one row per work-group instead of per sub-group, so
each work-item keeps N/NT values rather than N/WARP_SIZE. Butterflies below
the sub-group width still shuffle; those up to the work-group width go
through work-group local memory; the rest stay in registers. Same butterfly
and sign convention as the existing narrow kernel.

ggml's SYCL backend registration (dpct::dev_mgr) unconditionally requires a
GPU-labeled platform to exist and throws before any op-level test can run,
so test-backend-ops could not be exercised on this box (a GPU-less pod) even
via the CPU device. Verified instead with a standalone harness: the same
kernel body run through a real SYCL CPU device (Intel oneAPI DPC++ 2026.1,
OpenCL CPU backend), checked against an independent recursive-doubling
Hadamard reference, cross-validated by first running the existing unmodified
narrow kernel through the identical harness and confirming it passes (rules
out a reference-convention bug before trusting a pass on the new code).
Random-input results for all four widths, single- and multi-row:

  N=1024 NT=256 rows=1  max_abs_err=1.7e-07  max_rel_err=4.9e-04  PASS
  N=2048 NT=256 rows=1  max_abs_err=1.9e-07  max_rel_err=2.0e-04  PASS
  N=4096 NT=256 rows=1  max_abs_err=2.0e-07  max_rel_err=1.4e-04  PASS
  N=8192 NT=256 rows=1  max_abs_err=2.5e-07  max_rel_err=3.8e-03  PASS
  N=1024 NT=256 rows=7  max_abs_err=2.4e-07  max_rel_err=1.0e-03  PASS
  N=2048 NT=256 rows=5  max_abs_err=3.0e-07  max_rel_err=9.4e-04  PASS
  N=4096 NT=256 rows=3  max_abs_err=2.7e-07  max_rel_err=1.7e-03  PASS
  N=8192 NT=256 rows=2  max_abs_err=2.5e-07  max_rel_err=1.9e-03  PASS

This covers the kernel algorithm itself; it does not exercise the ggml
dispatch/supports_op integration end to end, which needs a real GPU (or a
SYCL GPU plugin) to get past backend registration. test-backend-ops build
is verified: fwht.cpp recompiles with zero warnings as part of ggml-sycl.
 2026-10-06 10:38:05 +03:00
Animesh 846af52607 CUDA: tune fp16 tile FlashAttention configs for head sizes 40-112 (llama/26289) 2026-10-06 10:38:05 +03:00
uvos 743f1ad286 HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch sizes (llama/28907) 
* HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch sizes

* CI: hip-quality-check: ignore spills for very large mfma mma kernels
 2026-10-06 10:38:05 +03:00
Ruben Ortlam 14d1aa70e6 vulkan: fix argsort kernel selection for Adreno (llama/29469) 2026-10-06 10:38:05 +03:00
Aman Gupta 24cf265ba6 RPC: use RDMA completion channel to not spin (llama/29440) 
* RPC: use RDMA completion queue to not spin

* add TODO for apple RDMA
 2026-10-06 10:38:05 +03:00
kurquharandMax Krasnyansky 7eea51881e hexagon: support tiled Q4_0 and Q8_0 GET_ROWS (llama/29511) 
* hexagon: support tiled Q4_0 and Q8_0 GET_ROWS

* hex-get-rows: fix macros

* hex-get-rows: use tiled HVX dequantization

Assisted-by: OpenCode

* hex-get-rows: fix register spills and clean up checks for unsupported ops

* hex-get-rows: improve dma pipeline

* hex-get-rows: improve/simplify kernel selection logic

* hex-build: reenable vectorizer, didnt notice the regression earlier in the sampler update

---------

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
 2026-10-06 10:38:04 +03:00
Max Krasnyansky fe06027d27 hexagon: support for backend sampler (llama/29502) 
* hex-topk: trying to improve/cleanup the pipeline

* hex-sampling: add STEP op

* hex-sampler: add SUM op

* hex-sampler: update CPY to support sampling cases

* hex-binary: add support for chunking to handle large logits

* hex-argmax: super basic version of ARGMAX

* hex-binary: support for scalars in extended buffers

* hex-binary: fix wrong indexing for dim 1 broadcasts across dim 2 slices

* hex-argsort: fix missing header

* hex-sampler: cleanup dma usage in the sampler related ops, and binary

* hex-build: disable autovectorizer, it is better to use explicit hints for critical loops

* hex-binary: fix perf regression due to is_1d fallback

* hex-ops: update supported ops
 2026-10-06 10:38:04 +03:00
Anav Prasad fc1ebfabf2 cuda: support Nemotron 3 Puzzle state size 96 for ssm scan (llama/28717) 2026-10-06 10:38:04 +03:00
bri-prism dfe8fbd7e7 cuda: add F16 input to the FWHT (llama/29096) 
* cuda: add F16 input to the FWHT

The CUDA FWHT accepts F32 input only. This makes the source type a template
parameter, so the kernel reads an F16 source directly instead of requiring a
converted copy. The F32 path is unchanged.

supports_op accepts an F16 src1 against an F32 src0 for the Hadamard hint.
Every other F16 src1 against a non-F16 src0 is still refused.

ggml_cuda_op_mul_mat_use_fwht is the single predicate both supports_op and
the dispatch call now share, checking contiguity and same-shape(src1, dst)
in addition to the type/hint conditions above. Without a shared predicate,
supports_op could admit an op that ggml_cuda_op_fwht then rejects only after
the unconditional same-shape assert has already fired; that gap predates
this change (it applies to the existing F32 path too) but this PR is what
touches supports_op, so it closes it here.

test-backend-ops on an A10 (lambdalabs): MUL_MAT 1297/1297, including all
24 Hadamard cases (18 existing F32, 6 new F16).

* cuda: use ggml_cuda_cast in the FWHT load, drop the comment
 2026-10-06 10:38:04 +03:00
jboothandGeorgi Gerganov 447a77510d ggml-cpu: tiled mul_mat for k-quants (llama/27851) 
* Added tiled mul_mat.

For each mul_mat_one_chunk, quants are unpacked into (max) 256x256 tiles of int8,
one routine per quent.  Then microkernel computes 16x16 tiles before writing out
256x256 float reults to main memory.

Tests/benches in tests/test-tiled-mulmat.cpp.  3-6x speed improvement
for large matmul, break even at 4096x64 * 64x4096, 80% performance (net
loss) for GEMV.  Error rates trivial (order of 1-e04 max, 1-e05 rmse).

* Fixes for ARM/windows builds

* more windows fixes, ggml-cpu.h isn't visible in MSVC for some reason

* unified iqp + tiled on the Q5_K, IQ4_XS set for benchmarking, updated benchmark

* Fixed accidental removal of llama_build_and_test(test-backend-ops.cpp)

* First integration of iqp code

Co-authored-by Bartowski <3266127+bartowski1182@users.noreply.github.com>

* Cleaning up declaration of iq unpacking helpers to align with the bit unpackers

* Removed iqp path

* Fix cross-platform warnings

* Disabling benchmarks unless explicitly enabled

* Fix backend_init for DLL-based builds, add self and bartowski to CODEOWNERS for tiled

* Put benchmarks behind a flag

* kernel fix for AVX2, iq quants

* Fix for asan, leaking memory in test-tiled-mulmat and avoid stack use after return

* guarding env flags with std::call_once

* Simplified repacking for VNNI to a single call per macrotile

* No threadlocals anymore, aligned wdata access

* Doing aligned reads since we ensure alignment with padding in wdata

* Eliminated per-thread gather of Q8_K rows in mul_mat_id, we now gather/repack in a single pass.  Repack method now takes pointer array to support both dense/normal and mmid paths.  Interface with ggml-cpu.c simplified as a result

* Unified/simplified dispatch and support checks.  Put details on wdata needed inside the kernel.h body, simplified interactions with ggml-cpu.c.

* Cleanup includes and whitespace, update src1_repack to return false if we don't need a special repack, so the common case is handled by driver

* Better detection of win32 and additional whitespace fixes

* Gating fuzz tests behind a parameter and some extra prints to try and fix slow CI hosts

* Optimized AVX2 kernel

* Changed interleave format and added ability to interleave in-place after dequant

* Repacks now happen in-place, 16x64 microtiles are independent of each other

* Only repack rows in groups of 16 as they're needed.  Save work in low n_rows cases and optimize L1 usage in other cases

* Use long panels for memory-bound regime (M <= 16), reintroduce IQP path for benchmarks

* Fix unused warnings and cleanup.  Improved IQ dequantization speed.

* Removed separate process benchmarks

* Revert "Removed separate process benchmarks"

This reverts commit 0688cf43d5e37ec7a52828c5576df563514fe760.

* AVX2 optimizations and guards for tests on windows

* Removed temp perf harness

* Remove perf-mulmat from build

* Removed IQP path, simplified tests to not use sub processes

* Cleaning up alignment of wdata

* Whitespace fixes and aligning L2 workspace to clean 512kb boundaries

* Update ggml/src/ggml-cpu/tiled/tiled-kernel.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* Cleanup merge-duplicated declaration of test-backend-ops target

* Undo accidental line deletion in ggml.c

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-10-06 10:38:04 +03:00
shaofeiqi bf1d787a7c opencl: add A8 Q8_0 non-MoE dp4a binary kernel (llama/29439) 2026-10-06 10:38:04 +03:00
Trivikram Reddy 9731f6fdbf hexagon: find software divide calls using binary inspection tool (llama/29449) 
* hex-scripts: fix table alignment

* hex-scripts: find sw div calls using binary inspection tool
 2026-10-06 10:38:03 +03:00
shaofeiqiandLi He 2d91497ae9 opencl: add bin kernel kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a8_bin, kernel_gemm_noshuffle_q5_k_q8_1_dp4a_ila_a8_bin (llama/29401) 
* opencl: add A8 Q5_K non-MoE non dp4a + dp4a binary kernel

* opencl: fix s transpose - s only transposed for bin kernels

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
 2026-10-06 10:38:03 +03:00
sliu39 669188ef02 Fixing the vulkan build issue of legacy GLSLC version that has no cooperativeMatrix API support (https://github.com/ggml-org/llama.cpp/issues/29373) (llama/29409) 
* vulkan : fix build issue of legacy glslc version by adding GGML_VULKAN_COOPMAT_GLSLC_SUPPORT macro check for Intel FA shader compiling

* vulkan : add preprocess condition to filter out unsupported FA 2 phases kernels before creation.

* vulkan : move lock_guard for Intel FA shader pointer creation under CM1 compiling preprocessor
 2026-10-06 10:38:03 +03:00
bri-prismandGeorgi Gerganov 9a919424e4 metal: FWHT kernels for block widths above 512 (llama/29095) 
* metal: FWHT kernels for block widths above 512

The Metal FWHT covers widths 64 to 512, one row per simdgroup with N/32 values
per lane. Wider blocks need more registers per lane than that layout allows.

kernel_fwht_tg runs one row per threadgroup with 256 threads, so each thread
keeps N/256 values. Butterflies below the simdgroup width still shuffle, those
up to the threadgroup width go through threadgroup memory, and the rest stay in
registers. Same butterfly and sign convention as the simdgroup kernel.

Widths 64 to 512 keep the simdgroup kernel. 1024 through 8192 use the new one,
for both F32 and F16 sources.

The wide kernels allocate float[N] of threadgroup memory, 32 KB at 8192, so the
size check takes the device limit and reports those widths as unsupported where
they would not fit. Without that a device with less threadgroup memory would
accept the op and then abort on a nil pipeline.

test-backend-ops on M5 Pro: MUL_MAT_HADAMARD 26/26, MUL_MAT 1265/1265.

* cont : add TODOs

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-10-06 10:38:03 +03:00
Georgi Gerganov 331d3c8aac sync : ggml 2026-10-06 10:37:30 +03:00
Georgi Gerganov 4b0998ada3 metal : split fa kernels into per-dtype libraries (llama/29329) 2026-10-06 10:36:44 +03:00
Georgi Gerganov bad9b54625 sync : ggml 2026-10-06 10:34:37 +03:00
ynankaniGeorgi GerganovJohannes Gäßler
 		afc10c26a4 llama : add llama_prec_policy + model-driven W4A4 path (llama/24364) 
* Rebase and update based on #26675

Signed-off-by: ynankani <ynankani@nvidia.com>

* CI failure fix(launh_bounds overload on HIP) and cleanup

Signed-off-by: ynankani <ynankani@nvidia.com>

* Address review comments

Signed-off-by: ynankani <ynankani@nvidia.com>

* Use ggml tensor instead of name in act policy map

Signed-off-by: ynankani <ynankani@nvidia.com>

* Address review comments and cleanup

Signed-off-by: ynankani <ynankani@nvidia.com>

* Address review comments

Signed-off-by: ynankani <ynankani@nvidia.com>

* Rename changes

Signed-off-by: ynankani <ynankani@nvidia.com>

* Update ggml/src/ggml-cuda/mmq.cu

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* MXFP4 dispatch changes for higher src prec

Signed-off-by: ynankani <ynankani@nvidia.com>

* Refactor and address review comments

Signed-off-by: ynankani <ynankani@nvidia.com>

* Updates based on review comments

Signed-off-by: ynankani <ynankani@nvidia.com>

* Apply batched suggestions from code review

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* Address review comments

Signed-off-by: ynankani <ynankani@nvidia.com>

* Apply patch from review

Signed-off-by: ynankani <ynankani@nvidia.com>

---------

Signed-off-by: ynankani <ynankani@nvidia.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
 2026-10-06 10:34:34 +03:00
uvos d7e83b18cb HIP: bump HIP_VERSION requried for fp8 to avoid missing __hip_fp8_e4m3 support in 6.2 (llama/29231) 2026-10-06 10:34:34 +03:00
Jess SullivanandGeorgi Gerganov 26bfdc66ba rpc: include nb in the get_alloc_size cache key and floor the result at ggml_nbytes (llama/29283) 
* rpc : include nb in the get_alloc_size cache key and floor the result at ggml_nbytes

* cont : remove redundant comment

* cont : add TODO

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
 2026-10-06 10:34:34 +03:00
Neo Zhang 93c4308ae9 support sparse FA (llama/28796) 
* fix conflict

* fix format issue

* rm unused code
 2026-10-06 10:34:34 +03:00