The Tensor API mat-mat path of kernel_mul_mm (GGML_METAL_HAS_TENSOR) fed a
static K=32 tile to the matmul2d op on every iteration. On the last, partial
K tile (ne00 % 32 != 0) the src1 slice extends past the K extent of the
tensor, and the op reads those out-of-bounds elements (undefined behavior per
the MSL specification, section 2.22.2). Depending on stale memory contents,
this corrupted the result or produced NaN.
Make the matmul2d op use dynamic_extent for K, and clamp the K extent of both
operand tensor views to the remaining valid K range (min(32, K - loop_k)) per
iteration, so the op reads exactly the valid K range on every iteration
(mirroring the tail handling of the MPP matmul2d examples). On K-aligned
inputs the clamp degenerates to the full 32-wide tile: the only difference
from the static-K op is that the dynamic-K op derives K from the operand
extents and edge-checks the tile against the tensor extents (a handful of
integer ops per iteration).
Add test-backend-ops MUL_MAT cases with K not a multiple of 32 to exercise
the unaligned K path.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* opencl: decline KV-convert flash_attn variants on Adreno A7X (compiler SIGSEGV)
The Adreno 740 (A7X) compiler E031.41 crashes inside clBuildProgram when
building the flash_attn programs whose KV path is mixed-type or dequantized:
flash_attn_f32_f16, flash_attn_f32_q8_0, flash_attn_f32_q4_0. It is a driver
crash rather than a compile-error return, so build_program_from_source_ex()
cannot catch it. The uniform f32 and f16 programs build correctly.
Decline the three KV-convert variants on the A7X in supports_op so they never
lazy-compile; those attention layers run on the CPU backend instead. Same
idiom as the existing Intel DK=512 and X1E carve-outs.
test-backend-ops FLASH_ATTN_EXT on the 740: 226 OK / 0 FAIL, previously exit
139. Other parts are unaffected - the gate is dead code there.
* opencl: fix q6_K flat mul_mat on older Adreno E031 compilers, gated
kernel_mul_mv_q6_K_f32_flat produces ~10x-wrong output on the older Adreno
E031 compilers while q4_K and q5_K are correct. Four codegen defects, each
confirmed on-device against the CPU reference:
1. 64-bit ulong arithmetic is miscompiled, so every weight and scale read
hit the wrong address - the primary cause, and why q5_K (int offsets)
was unaffected. The block index is computed in int and widened only
inside the pointer expression.
2. The vectorized dequant (int4/float4 bit-ops, convert_*4, dot()) is
miscompiled; the 6-bit weights are reconstructed and the dot done
scalar.
3. vload4 of the f32 activations is miscompiled; replaced by a
scalar-indexed load.
4. The accumulation is miscompiled unless a side effect forces the partial
sums to materialize. A printf under a guard the compiler cannot prove
false acts as a zero-cost optimizer barrier; its placement is
load-bearing.
The defect tracks the compiler, not the GPU generation: it reproduces on
E031.38 (Adreno 642L) and E031.41 (Adreno 740) and is fixed by E031.45
(Adreno 619), so the workarounds are gated on the compiler version. Where
they are not needed they cost real throughput - 42.4 -> 35.1 GFLOPS on an
Adreno 840 q6_K GEMV. The explicit compiler-type check is required, not
redundant: newer_than_or_same() is false for every non-E031 compiler, so
negating it alone would enable the workarounds on E17 and DX.
test-backend-ops MUL_MAT is 919/919 on the Adreno 740, 642L, 619, 840 and
850; the 740 and 642L were 909/919 before. The 642L additionally needs the
A6X per-kernel-program support to reach these tests at all.
* CI: Use LLVM's OpenMP over MSFT_DEBUG_non_redist on Windows
Currently, we ship the non-redist debug version of microsoft's libomp.
This PR changes this to official LLVM's release, also packaging
the license as needed.
* Remove LLVM SHA from job name to increase legibility
* Add temp validations to CI
* Revert "Add temp validations to CI"
This reverts commit eef97c88b5bac280803ebb3c3b7bb09f89b0fd88.
* Build OpenMP in CI
* Make OpenMP fetch self-contained in cmake and cache in CI
* Robustify Licens-packaging
1. Ship OpenMP license, not LLVM's.
2. Invalidate cache also on checksum of the license
* Remove stale reference in docs/build.md
* No longer package base license in release
This was scope-creep
* Add explanatory comment to OpenMP license
* Remove arm64 smoke
Forgot this during conflict resolution during rebase of
c54c0e9cf6030a5a54ce8bdd81b3e146d9787d42
* Remove GGML_OPENMP_FETCH_CACHE_DIR as requested by @CISC
* whitespace changes
* CUDA: runtime GGML_CUDA_MMVQ_MAX to tune the mvq->MMQ decode crossover
Add a runtime override of the mul_mat_vec_q -> MMQ batch crossover
(default MMVQ_MAX_BATCH_SIZE). Lowering it routes batches above the
threshold from the CUDA-core vector kernel to the int8 MMQ tensor-core
path, which is faster once quantized decode becomes compute-bound at
B>1 (measured +23-41% at B=8 on RTX 5090 for Q4_K dense, no low-batch loss).
The value is parsed once and clamped to [1, MMVQ_MAX_BATCH_SIZE], since
mul_mat_vec_q asserts ncols_dst <= that; invalid input warns and falls
back to the default. The override is applied consistently in both the
mul_mat_vec_q and MUL_MAT_ID dispatch paths. Default behavior unchanged.
* Added Blackwell specific switch point, to reduce dependence on runtime env var.
* Add per-HW switch point values for DGX Spark and removing runtime env var
* Adding switch points for Ada, tested on RTX 4090
* Modifying DGX Spark numbers based on latest run and adding some comments and small functional changes relating to MoE
* Reverting an unnecessary conditional
* Update ggml/src/ggml-cuda/mmvq.cu
---------
Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com>
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* metal: dequantize q8_0 KV to f16 before flash attention
Add a preprocessing pass for GGML_OP_FLASH_ATTN_EXT on the Metal backend:
when the KV cache is quantized (Q8_0 for now), dequantize K and V into a
contiguous F16 scratch buffer and run the existing F16 flash attention
kernels on it, instead of the in-kernel dequantization path.
- new kernel kernel_flash_attn_ext_dequant_to_f16<block_t, QK, deq_t4x4>:
one thread per quant block (K then V), stride-aware so permuted KV is
supported; instantiated for Q8_0 (extending to Q4_0/Q4_1/Q5_0/Q5_1 is
one instantiation + one gate case)
- the gate is type-only: dequantize whenever the KV is quantized,
regardless of head sizes, GQA ratio or n_kv; the attention kernels
themselves are untouched
- the F16 copies live in the op's own scratch allocation
(ggml_metal_op_flash_attn_ext_extra_dequant_f16); the KV pad kernel
reads the dequantized buffers when the path is active
- the FA pipeline getters gain a use_f16_kv flag selecting the existing
f16 kernels and contiguous strides
- ref: https://github.com/ggml-org/llama.cpp/pull/25556
Verification (M2 Ultra):
- test-backend-ops test -o FLASH_ATTN_EXT: 4798/4798 pass, including the
new q8_0 eval cases (decode/prompt, permuted, sinks+ALiBi+softcap,
kv=113 pad path, kv=16384)
- llama-perplexity on Qwen2.5-0.5B with -ctk q8_0 -ctv q8_0 matches the
f16 KV reference (PPL 1.0008 vs 1.0008)
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : launch the FA KV dequant kernel separately for K and V
Simplify kernel_flash_attn_ext_dequant_to_f16: it now dequantizes a single
tensor (its own ne/nb and dst) with no is_v branching, and the op dispatches
it twice with the same pipeline - once for K and once for V. The kargs
struct shrinks to a single ne/nb set plus nblocks.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : dequantize q4_0, q4_1, q5_0 and q5_1 KV to f16 before flash attention
The dequant pass now covers all quantized KV types supported by the Metal
flash attention kernels. The dequant kernel, kargs, scratch allocation and
dispatch are type-generic, so each type is one kernel instantiation plus one
gate case.
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : skip the redundant V dequant when V is a view of K
In MLA-based models, the V of the FA op is a view of K (the first ne20
elements of each K row); the dequantized V is then a view of the dequantized
K, so skip the second dequant dispatch, do not reserve the V scratch region,
and let the pad and attention kernels read V from the K F16 buffer with K's
strides. The detection follows the CUDA backend:
V->view_src && (V->view_src == K || (V->view_src == K->view_src && V->view_offs == K->view_offs))
Also fix the FA pipeline getters: ns10/ns20 are function constants baked into
the kernels and must be the actual K/V row widths as seen by the kernel. The
dispatch now passes them explicitly (nb11_attn/nb10_attn, nb21_attn/nb20_attn)
instead of the getters assuming contiguous F16 KV (ns20 = dv), which was wrong
when V is read from K with K's row pitch (e.g. 576 vs 512).
New test cases: 576/512 q8_0 (MLA shape, V is a view of K) at kv=113 (KV pad),
nb=1 (vec) and nb=64 (non-vec).
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* test : remove backend-specific wording from test-backend-ops comments
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* pi : avoid backend mentions in test-backend-ops comments
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* metal : rename the FA dequant_f16 identifiers to kv_f16
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : clean-up
* cont : remove TODO
* ggml: fix backend split scheduler race condition
splits without input were running concurrently with other splits, while potentially reusing memory the other split is accessing
* only sync when split has no inputs
* provide static workspace for cuBLAS handles
* account for concurrent streams when using GGML_CUDA_GRAPH_OPT
* drop cublas_handle overloads and remove direct cublasSetStream calls
* Update ggml/src/ggml-cuda/common.cuh
---------
Co-authored-by: Oliver Simons <osimons@nvidia.com>
* backend: propagate buffer usage in meta backend
* ggml-meta: make sure to call init_tensor for all new tensors
* meta: remove explicit check for meta backend in ggml_backend_meta_get_split_state
I can't seem to reproduce the original failure in the latest code.
* hexagon: fix FA HMX queue ordering in the pipelined path
* hexagon: double buffer D matrix, store diagonal tile only
* format code
* align the indentation
* opencl: port fused ssm_scan kernel (Mamba-2, d_state in {128, 256})
Fold the fused per-token SSM_SCAN recurrent step from opencl/gdn-qwen36-35b
onto the unified base. Previously SSM_SCAN fell back to CPU here; now scalar-A
Mamba-2 with d_state in {128,256}, all-f32, runs on GPU. Other shapes (incl.
Mamba-1 element-wise A) still fall back. test-backend-ops -o SSM_SCAN passes on
Adreno X2-90. opt-out via GGML_OPENCL_DISABLE_SSM_SCAN=1.
* opencl: cleanup
* opencl: require K == 1
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
* ggml-cpu: gate __fp16 on __ARM_FP16_FORMAT_IEEE
__ARM_NEON only signals NEON availability. The __fp16 type also needs
the IEEE half format, implied on AArch64 but selected with
-mfp16-format=ieee on 32 bit Arm, where the compiler otherwise rejects
the type.
The guard keeps every toolchain that provides the type on the same code
and sends that one configuration to the generic lookup path.
* ggml-cpu: gate the NEON+FMA block on __ARM_FP16_FORMAT_IEEE
Both halves of the F16 section dereference __fp16, so armv7 with
neon-vfpv4 hits the same unknown type error. Without the IEEE
format the configuration now falls back to the scalar path.
Address review from @JonathanC-ARM
* vulkan : dequant q8_0 KV once in coopmat1
Assisted-by: Claude (Opus 4.8)
* vulkan : fall back instead of aborting when FA scratch exceeds maxStorageBufferRange
* vulkan : require KV-cache layout in FA dequant path
Assisted-by: Claude (Opus 4.8)
* vulkan : skip FA dequant path on coopmat2
Assisted-by: Claude (Opus 4.8)
* tests : add contiguously-allocated quant K/V FA tests
Assisted-by: Claude (Opus 4.8)
* vulkan : trim comments
* vulkan : tighten permutation checks for FA path
* vulkan : set prealloc_x_need_sync after the FA dispatch
* vulkan : exclude Intel Xe1 from FA dequant path
* add params
* cpu kernel
* metal kernel
* add test backend ops
* gate other backends
* ggml: (cuda) support ggml_rope_set_offset (llama/27121)
* rm cuda supports_op guard, fix webgpu clang-format
* ggml: support ggml_rope_set_offset on vulkan (llama/27344)
* ggml: support ggml_rope_set_offset on vulkan
* remove inplace optimization
* vulkan: tiled transpose for 0<->2 permuted CONT
-ggml_vk_get_cpy_pipeline only routed to the tiled shared-memory transpose
shader when dim1 was the innermost dimension, i.e. ggml_transpose (a 0<->1
swap). A 0<->2 swap -- ggml_cont(ggml_permute(x, 2, 1, 0, 3)) -- fell back to
the generic per-element strided copy, whose source reads stride by ne0*ne1
elements: one cache line per lane.
-DeepSeek-V4's lightning indexer performs exactly that permute on a
[n_kv, n_tokens, n_head] tensor. On Vulkan/RADV gfx1151 it ran at ~1-9 GB/s of
a ~200 GB/s part and accounted for 43% of total prefill time.
-Add copy_transpose_02.comp, mirroring copy_transpose.comp but tiling over dst
dims (0, 2) with dims 1 and 3 as the batch, so reads walk src dim2 and writes
walk dst dim0 -- both contiguous. The selection condition additionally requires
a non-contiguous source and a contiguous destination so it cannot take cases
the contiguous-copy shader already handles.
-test-backend-ops only exercised ggml_transpose for CONT, so the strided path
was untested. Add test_cont_permute covering (2,1,0,3), (1,2,0,3) and (0,2,1,3)
over f32/f16 at tile-aligned, tile-unaligned and large shapes. The large shapes
are in the eval set rather than only in perf because perf mode does not verify
results.
-Measured on gfx1151, ne=[n_kv,64,64,1], perm=(2,1,0,3), f32:
n_kv=1024: 9.08 -> 579.85 GB/s
n_kv=1280: 20.03 -> 153.71 GB/s
n_kv=2048: 7.11 -> 91.68 GB/s
n_kv=2304: 16.24 -> 86.49 GB/s
-The ~2.2x penalty previously seen at power-of-two n_kv (destination-stride
aliasing) is gone. End to end, DeepSeek-V4-Flash IQ3_XXS prefill on a 9k-token
prompt goes from 56.33 t/s to 103.74 t/s (+84%).
-Note: at n_tokens=512 a single slow-path dispatch takes ~273 ms and looping it
in perf mode can trip the GPU watchdog, so the perf cases use n_tokens=64.
* tests: fold test_cont_permute into test_cont, add L2-exceeding perf shapes
Review feedback: test_cont gains a permute parameter ({0,0,0,0} = none),
matching test_mul_mat's pattern, and the separate struct is gone. Perf
adds [n_kv, 512, 64, 1] variants (~0.5 GB per run) that exceed GPU L2,
since the 64-token shapes fit in cache on large parts and read above
memory bandwidth.
* tests: trim perf-case comment to the two-line summary
* vulkan: trim comments on the 0<->2 transpose path
Drop the shader file header, the read/write block comments and the
rationale prose in the CONT test cases. Keep the tile-shape and
bank-conflict notes and the permute parameter documentation.
---------
Co-authored-by: Kevin Hopper <no-reply@maestro.press>
Kernel is a port of `ggml-cuda/fwht.cu`
(us/run, median):
```
m x n x k GEMM FWHT speedup
64 x 1 x 64 10.20 2.93 3.48x
64 x 2048 x 64 10.75 2.71 3.97x
128 x 1 x 128 10.33 2.88 3.59x
128 x 32 x 128 9.20 2.77 3.33x
128 x 2048 x 128 16.46 2.76 5.95x
256 x 1 x 256 10.19 2.77 3.68x
256 x 2048 x 256 16.69 3.41 4.89x
512 x 2048 x 512 54.16 12.89 4.20x
```
* CUDA: MMVQ nwarps=8 for bs=1 for dense models on DGX Spark
Signed-off-by: ynankani <ynankani@nvidia.com>
* skip moe experts and allow others based on k geometry (allow only small idle tail)
Signed-off-by: ynankani <ynankani@nvidia.com>
* rename MMVQ DGX Spark params to GB10 and fix MSVC constexpr lambda capture
Signed-off-by: ynankani <ynankani@nvidia.com>
---------
Signed-off-by: ynankani <ynankani@nvidia.com>
Adjusts the thread/block count to be proportional to the size
of the quant, reducing under/over subscription.
Largest perf improvement is the q4_0 -> f32 path, with, on
a Arc 70, throughput goes from 20.21 GB/s to 158.19 GB/s
The rest of the quants are flat in performance uplift.
* Initial changes for Recurrent state rollback for nemotron for cpu and cuda
* Removing CPU RS rollback. Will enable it in subsequent PRs
* addition of test case
* Removing assert and calling runtime API to check if op is supported
* removing extra API and updating the call sites for K
* replace static cuda detection to runtime fused_op api
* address review comments and fallback when SSM rollback not supprted
* Adding changes for supporting RS-rollback in CPU. Also added test-backend-ops for cpu and cuda
* removing memory manipulation as rs rollback is now supported in CPU
* removing the static probe which is not needed now
* correcting the format
* address review comments
* enabling test for all the backends, unsupported backends will fallback to CPU
* Apply suggestions from code review
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* choose different graph based on the result of fused_ssm_op is supported or not and also handled memory->n_rs_seq >1 case incase of op is not supported
* Support K > 1 in ssm_scan for all backends
* Fix CI Issues
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Co-authored-by: Gaurav Garg <gaugarg@nvidia.com>
* OpenVINO backend: 1) enable gpt-oss moe on OV bk; 2) enable mxfp4 support
* OpenVINO backend: disable TOPK_MOE op test
* OpenVINO Backend: Add op FILL support
* OpenVINO backend: enable set rows with multi dims
* fix the name missmatch in setrow + view
* OpenVINO backend: enable op GGML_UNARY_OP_SIGMOID
* OpenVINO Backend: enable SQR & SQRT
* OpenVINO backend: 1) ensure unique node names for OpenVINO; 2) add org_src to recorde the src ggml tensor for OpenVINO dynamic shape infer
* OpenVINO backend: enable fallback for openVINO to CPU backend
* OpenVINO backend: fix accurace issue in gemma3n arch test
* fix mpt failed case
* OpenVINO backend: clean nodeinfo
* OpenVINO Backend: enable zero-size copy for view
* add concat ssm_conv in compute_dynamic_dim
enable qwen35
Fix after rebase
remove logging
* OpenVINO backend: disable EXP with FP32, which failed in op test. Root reason: the backend test initializes unary op inputs over a wide range, [-150, 150]. For FP32, exp(x) overflows around x ~= 88.7, so this test can randomly generate values right in or beyond the overflow region
* OpenVINO backend: fix CPY op test failed issue
* OpenVINO backend: fix GATED_DELTA_NET op test failed issue
* handle in-place op, handle qwen35 dynamic clearing of cache in cgraph
* handle qwen35 dynamic clearing of cache correctly
* Enable qwen35 dense multi seq
* Fix qwen35 9b gqa
* Fix after rebase
* Disable SOLVE_TRI
* openvino: fix NEOX RoPE accuracy on GPU stateful (mixed-rank Multiply)
In stateful mode the NEOX RoPE branch fed rank-3 data ([S, n_heads,
head_size]) into the Multiply against the rank-4 cos/sin tables
([1, S, 1, n_dims/2]). That mixed-rank broadcast is miscomputed by the
OpenVINO GPU plugin, corrupting the rotated Q/K and producing garbage
output (e.g. Phi-3-mini). Lift the data to rank-4 before the split/
Multiply so the operands are equal-rank, matching what the TYPE_NORMAL
branch already does. CPU and stateless paths are unaffected.
Phi-3-mini-Q4_K_M, wiki.test perplexity, GPU stateful:
before: PPL = 27120.43
after: PPL = 6.2263 (CPU reference: 6.2251)
* OpenVINO backend: 1) remove the unique name in llama.cpp; 2) add new ov name in ov bk; 3) fix issue in arch test & op test with latest code update
* OpenVINO Backenb: remove changes in llama.cpp
* Doc change (use x64 Native Tools Command Prompt for VS)
* Cleaner sentence
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* OpenVINO Backend: cache key upgrade includes all src name
* OpenVINO Backend: enable llama arch test on ci
* OpenVINO Backend: move parameter node creating from decoder into translate
* OpenVINO Backend: create extra input ov node move from decoder to translate
* fix for op regression due to is_model_splitted
* openvino: fix CPY writeback for recurrent state rollback
Detect the rollback conv/gdn state writeback CPY nodes structurally
instead of by tensor name, since the rollback path in
build_conv_state does not call cb() and left the nodes unnamed. Add
per-node runtime offsets (rs_slot_begin_*, rs_src_begin_*) so the
cached IR handles any kv head, sequence count and snapshot slot for
both the conv state and the GDN state writeback.
Assisted-by: GitHub Copilot
* qwen35 moe
* optimize MoE expert aggregation with ReduceSum
* Skip GET_ROWS inaccurate test
* openvino: fallback dynamic MUL_MAT_ID shapes
* OpenVINO Backend: fix error in arch test model mpt
* fix error caused by cpy in arch test model kimi-linear
* OpenVINO Backend: fix error in arch test model minimax-m3
* openvino: fix GPU mul_mat_id op tests
* ggml-openvino: add GGML_OPENVINO_RELEASE_WEIGHTS to reclaim host weight RSS on GPU
The OpenVINO weight Constants are zero-copy views into host buffers
allocated by the backend (ggml_aligned_malloc, anonymous memory). On GPU
the plugin holds its own device copy after compile_model, so these host
pages are dead weight for inference. For a 1B Q4_K_M model this leaves
~850 MB of host RSS resident that the GPU path never reads again.
Add an opt-in GGML_OPENVINO_RELEASE_WEIGHTS mode that madvise(MADV_DONTNEED)s
the registered host weight buffers once the model is compiled, dropping
their resident pages while keeping the mappings valid (ggml still owns the
lifetime; tensors still point in). Measured steady-state RSS drops from
~1555 MB to ~710 MB on Llama-3.2-1B-Q4_K_M (Arc iGPU) with unchanged
throughput and correct output.
The GPU backend uses a single dynamic-shape model for both prefill and
decode, so a graph is compiled once and reused; the only event that forces
a recompile is clear_caches() on backend teardown. The change therefore:
- releases on the first cache-hit (model compiled, plugin has its copy);
- pins the compiled-model cache across backend teardown so a later
context reuses it instead of recompiling against the dropped pages;
- fails loud (GGML_ABORT) on a cache-miss recompile or on a second model
load, both of which would otherwise read zeroed weights or silently
reuse the wrong compiled graph.
Scope/limitations (all fail loud, never silently wrong): GPU only (the CPU
plugin reads the host Constants at inference time), one model per process,
and stable graph shapes. This reduces steady-state RSS, not the transient
compile-time peak. All changes are confined to the OpenVINO backend.
* ggml-openvino: stream weight requantization to cut the compile-time RSS peak
requantize_to_buffers() dequantized the entire tensor to a temporary
std::vector<float> of n_elements before requantizing. For token_embd.weight
(128256 x 2048) that transient is ~1 GB (1B model) / ~2 GB (8B), and it is
the single largest contributor to the OpenVINO compile-time memory peak --
it also fires twice for token_embd (once at load, once at graph build,
because token_embd is loaded via a CPU/mmap buffer and not cached as an OV
weight extra).
Stream the dequant instead: process a fixed window of complete rows
(CHUNK_ROWS=256) into a small scratch buffer and quantize/convert each chunk
straight into the output buffers. The transient F32 footprint is now
CHUNK_ROWS*ne0 floats regardless of tensor size.
quantize_q8_0/q8_1 gain an optional block_offset arg (default 0) so a chunk
writes its weights/scales/zp at the correct block. Streaming is applied to
the Q8_0_C / Q8_1_C / F16 targets (the large requant cases); the u4 (Q4_0)
path keeps the whole-array call because it packs two weights per byte with
running zp ORs, and a fallback handles any future target whose block size
does not divide a row.
Measured peak RSS (cold compile, GPU): 1B 2868 -> 1809 MB (-1.06 GB);
8B 11618 -> 9608 MB (-2.0 GB). Output verified unchanged
("capital of France is Paris"); throughput unchanged. Unlike
GGML_OPENVINO_RELEASE_WEIGHTS this reduces the transient peak, not just
steady-state, and needs no env flag. All changes confined to the OpenVINO
backend.
* ggml-openvino: avoid redundant token_embd requantization at compile
token_embd.weight is referenced twice in the graph path: as the GET_ROWS
embedding (a CPU/mmap-buffer tensor) it was re-extracted/re-requantized on
every weight-node build, and is_model_splitted() built a full (naive) set of
weight nodes just to test name membership — each requant is a ~1-2 GB F32
dequant of the 262M-element embedding.
Two changes:
- Add collect_weight_names(): a name-only collector for topology checks.
is_model_splitted() now uses it instead of create_weight_nodes(cgraph,
true), so the splitted-check no longer triggers any weight extraction.
- Memoize weight nodes built from non-OpenVINO buffers in a process-lifetime
cache keyed by tensor->data. These tensors have no OV buffer context to own
a cached extra, so without this they were rebuilt on every (re)compile;
prefill and decode graphs now share one build (verified: 2nd graph hits the
cache instead of re-requantizing).
Peak RSS is unchanged (the streaming-requant commit already removed the F32
transient); this removes redundant compile-time work. Output verified
unchanged ("capital of France is Paris"). Confined to the OpenVINO backend.
* ggml-openvino: gate compile-memory optimizations behind GGML_OPENVINO_REDUCE_COMPILE_MEM
The streaming requantization and the non-OpenVINO-buffer weight-node cache
(plus the name-only is_model_splitted path that pairs with it) are now opt-in
via GGML_OPENVINO_REDUCE_COMPILE_MEM. When unset, requantize_to_buffers()
fully materializes the F32 buffer and weights are rebuilt per compile exactly
as before; when set, the streaming path and the cross-compile weight cache
are used.
Default off keeps behavior identical to upstream unless explicitly enabled.
Verified: flag off -> peak RSS 2800 MB (original), flag on -> 1810 MB; output
"capital of France is Paris" in both modes. (GGML_OPENVINO_RELEASE_WEIGHTS,
added earlier, remains a separate opt-in for the steady-state release.)
* ggml-openvino: add frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR)
The plugin-level ov::cache_dir caches the compiled blob keyed by the OV
model, but producing that model still runs the full frontend every time:
weight requantization (incl. the large token_embd F32 transient) and the
ggml->OV graph conversion. This adds an opt-in frontend cache keyed off a
fingerprint computed directly from the ggml cgraph, so a hit imports a
previously exported CompiledModel and skips requant + convert + compile
entirely.
Key (model-cache.{h,cpp}) = 64-bit FNV-1a of: graph topology (n_nodes + per
node op/name), a sampled per-weight fingerprint (name/shape/type + bounded
head+tail byte sample), and blob-affecting config (device, flash-attn, rope
params, REDUCE_COMPILE_MEM/stateful flags, OpenVINO version). A sidecar
manifest stores every weight's fingerprint and is re-verified on load, so a
sampled-hash collision cannot cause a wrong-model hit (verified: two
different quantizations of the same model produce distinct cache entries).
Flow (dynamic single-model path only; split models defer to ov::cache_dir):
on a verified hit, core.import_model() restores the CompiledModel and a
lightweight decoder is built with a names-only weight map (membership is all
the decoder needs for I/O mapping; weights live in the imported model). On a
miss, compile as usual then export the blob (atomic temp+rename, manifest
written first). The frontend cache supersedes ov::cache_dir, so CACHE_DIR/
CACHE_MODE are stripped from the config used for the cached compile and the
import — a blob compiled with cache_dir set cannot be re-imported.
Measured 8B Q4_K_M (GPU): full requant+convert+compile 15.3s -> import 6.3s
(~2.4x faster compile phase). Output verified unchanged on cold and warm,
standalone and combined with REDUCE_COMPILE_MEM + RELEASE_WEIGHTS. Default
off; confined to the OpenVINO backend.
* ggml-openvino: harden frontend model cache correctness
The frontend model cache imports a previously exported CompiledModel keyed by a fingerprint of the ggml graph, weights, and blob-affecting config. The original key covered device, stateful execution, REDUCE_COMPILE_MEM, RoPE params, OpenVINO version, topology, and sampled weights, but missed runtime/frontend toggles that can change the lowered graph or the I/O binding contract. That made it possible to reuse a blob produced under a different OpenVINO backend configuration.
Add a small extra-config helper for the dynamic model-cache path and fold in the effective values of GGML_OPENVINO_DISABLE_KV_SLICE and GGML_OPENVINO_MANUAL_GQA_ATTN. MANUAL_GQA_ATTN is keyed by the behavior that actually takes effect: an explicit env value wins, otherwise GPU defaults to enabled and other devices default to disabled. This matches flash_attn_ext lowering and avoids unnecessary cache splits for equivalent configurations while separating genuinely different attention graphs.
DISABLE_KV_SLICE is also included because it changes the KV-cache tensor shape/output binding strategy used around imported models. Even when weights and graph topology are identical, switching this flag should not inherit a CompiledModel cache entry created for a different binding mode.
Also make cache artifact publication cleaner: write manifest.tmp and blob.tmp, publish the blob first, and publish the manifest last. Cache hits already require both blob and a verified manifest, so making the manifest the final visible artifact avoids leaving an apparently complete manifest for a failed or interrupted blob export. Temporary files are removed on the handled failure paths.
While touching this path, fix the indentation of the non-imported compile branch so the cache miss flow is easier to review. Behavior is otherwise unchanged: verified hits still import, misses still create weights, convert, compile, export, and create the infer request normally.
* ggml-openvino: add memory optimization umbrella switch
Add GGML_OPENVINO_MEMORY_OPTIMIZE as a single opt-in switch for the OpenVINO backend memory-saving paths. The existing fine-grained GGML_OPENVINO_REDUCE_COMPILE_MEM and GGML_OPENVINO_RELEASE_WEIGHTS variables remain supported and explicitly override the umbrella switch when set, so users can still bisect or disable one side of the optimization independently.
Centralize the policy in ggml_openvino_reduce_compile_mem_enabled() and ggml_openvino_release_weights_enabled(device). The umbrella switch enables compile-memory reductions everywhere REDUCE_COMPILE_MEM is used today: streaming requantization, non-OV weight-node caching, split-model weight-name collection, and the frontend model-cache fingerprint. On GPU it also enables host weight-buffer release unless GGML_OPENVINO_RELEASE_WEIGHTS is explicitly set.
Keep host weight release GPU-only because it relies on the plugin holding its own device copy after compile_model. Update the fail-fast diagnostic and comments to mention GGML_OPENVINO_MEMORY_OPTIMIZE, so users who enable the umbrella switch get accurate guidance if a later cache-miss recompile would read released host weight pages.
* ggml-openvino: rename compiled model cache env
Rename the frontend export/import cache environment variable from GGML_OPENVINO_MODEL_CACHE_DIR to GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR. The cache stores blobs produced by ov::CompiledModel::export_model() and restores them with core.import_model(), so the new name distinguishes it from GGML_OPENVINO_CACHE_DIR, which configures OpenVINO plugin-level ov::cache_dir.
Update the registered env var, the cache-directory lookup, and comments around the frontend compiled-model cache. The old GGML_OPENVINO_MODEL_CACHE_DIR name is removed rather than kept as a fallback so there is a single spelling for the new option.
* docs: document OpenVINO memory optimization env vars
Add runtime configuration entries for the newly recognized OpenVINO environment variables.
Document GGML_OPENVINO_COMPILED_MODEL_CACHE_DIR as the frontend compiled-model cache used to export and import compiled blobs for matching single-graph models.
Document GGML_OPENVINO_MEMORY_OPTIMIZE as the umbrella switch, including how GGML_OPENVINO_REDUCE_COMPILE_MEM and the GPU-only GGML_OPENVINO_RELEASE_WEIGHTS override or inherit from it.
* ggml-openvino: fix Qwen3VL crash and deepstack correctness bug
1. GGML_OP_PAD was missing from compute_node_dynamic_dims(), causing a crash
on decode for models that pad the token embedding (n_embd -> n_embd_inp).
PAD never reorders/merges dims, so it keeps the same dynamic dim index as
its source.
2. process_view_input_new() chained VIEW inputs through src[0] (the
immediate op-graph parent) using offsets treated as relative to that
parent. But ggml_tensor::view_offs is always absolute from the true root
allocation (ggml collapses VIEW-of-VIEW chains internally). For the
per-layer deepstack view ("embd (view)", whose src[0] is "embd" - itself
an already-narrowed, zero-offset VIEW of the padded root, with the SAME
ggml shape as the deepstack view but a different absolute offset), this
caused an out-of-bounds re-slice that silently fell back to returning the
wrong (already-resolved sibling) tensor. In practice every deepstack ADD
ended up adding the real base token embedding into the residual stream
instead of zero, corrupting generation ("Hello my name is 1000000..."
instead of coherent text). Fixed by detecting this pattern (same shape as
the immediate src, different absolute offset) and re-slicing directly
from the untouched root tensor using the innermost view's absolute
offset.
Also adds a GGML_OPENVINO_DEBUG_NODE=<name1>,<name2>,... env var that attaches
extra debug Result nodes for arbitrary intermediate tensors, without binding
them to any ggml buffer (avoiding the risk of reading a ggml buffer that has
since been overwritten by a later in-place op). This was instrumental in
diagnosing bug #2 above and is left in as a general-purpose debugging aid.
* ggml-openvino: fix IMROPE inp_pos padding for NPU static shapes
IMROPE's inp_pos tensor packs 4 stacked t/h/w/e position planes into
ne[0] = 4*n_tokens instead of one value per token. On NPU's static-shape
path, inp_pos was padded/shaped as if it held a single plane, which
interleaved padding across the 4 planes and desynced later reshapes
from the rest of the (chunk_size-wide) graph.
- add GgmlOvDecoder::get_inp_pos_n_planes() to detect IMROPE's 4-plane layout
- get_graph_input_shape(): size inp_pos as n_planes * chunk_size (prefill)
or n_planes (decode) instead of assuming 1 value per token
- get_ov_input_tensor_static_prefill(): pad each plane to chunk_size
independently instead of one flat block
- get_ov_input_tensor_static_decode(): copy n_planes contiguous values
instead of asserting/copying a single scalar
* disable test-llama-archs tests.
* openvino: gate fallback with env var
* Revert changes in test-llama-archs
* Apply editor config
* reject CPY with quantized destination as unsupported
---------
Co-authored-by: Xuejun <Xuejun.Zhai@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
Co-authored-by: Mustafa Cavus <mustafacavus@intel.com>
Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
* metal: add TQ2_0 support
Add support for the GGML_TYPE_TQ2_0 (ternary, 2 bits per element) type in
the Metal backend.
Assisted-by: llama.cpp:DeepSeek-v4-Flash-0731
* cont : optimize mul_mv kernel
- float ops over integer ops
- precalculate sums
- hoist coef out of the inner loop
- contiguous y loads
llama.cpp:DeepSeek-v4-Flash-0731
* Add DMMV Q4_K and Q6_K ESIMD kernels
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Refactor ESIMD kernels to share common code
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Move control of ESIMD from compile to runtime
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Use ESIMD by default when available
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Fix possible error when using ESIMD by default
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add explicit unroll to ESIMD kernels
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Tidy up ESIMD kernels a bit
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add DMMV Q3_K ESIMD kernel
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
---------
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
Xuan-Son Nguyen
Georgi Gerganov
Hongqiang Wang
lhez
Jeff Bolz
Oliver Simons
Pranesh Gonegandla
Ruben Ortlam
Alexander Heisler
Markus Tavenrath
Max Krasnyansky
Yiwei Shao
Pascal
Nathanw1014
Jetson Tan
Titaniumtown
Kevin Hopper
Masashi Yoshimura
Aman Gupta
ynankani
Mario Limonciello
Neo Zhang
Jiang, Fish
Masato Nakasaka
lnigam
Mendy Berger
Zijun Yu
jinzihao
Ian Faust
Todd Malsbary