* metal: WIP chunked SSD SSM_SCAN kernels for multi-token prefill
* metal: drop scalar SSD path; MMA + sequential tail
* drop WIP ssm scan test noise
* remove state_from_dst and rename CS and NSG constants
* remove unrelated added whitespace padding
* added clarity to mma_tokens calculation
* added clarity to use_mma bool checks
* added comments to metal ssd op constants for clarity
* reserve K tokens for sequential kernel rollback snapshots
* reset concurrency between mma and seq tail
* remove print args no longer used
* fixed comment to no longer point to specific line
* add FC_SSM_SCAN so seq path skips token offlset unless it's mma tail
* added changes to new ssm.metal for rebase after ggml-metal.metal refactor
* specialize ssm_scan tail with a template instead of a function constant
---------
Co-authored-by: dpantaleoni <dominikpantaleoni@gmail.com>
Co-authored-by: forforever73 <690105611@qq.com>
* rpc: support apple RDMA as an RPC transport
* remove set_tensor micro optimization, rpc socket pinning per CR
* remove transparent reconnect
* trigger apple builds on RPC changes
---------
Co-authored-by: Ryan Churaman <rschu@meta.com>
* metal : null-check ggml_metal_buffer_init result to avoid OOM crash
ggml_backend_metal_buffer_type_alloc_buffer used the result of
ggml_metal_buffer_init without checking for NULL. ggml_metal_buffer_init
returns NULL when the underlying Metal allocation fails (e.g. an
out-of-memory condition), and the following ggml_metal_buffer_is_shared(res)
call dereferences it, turning a recoverable allocation failure into a hard
crash (EXC_BAD_ACCESS). This is easy to hit on memory-constrained devices
such as iOS when a model/context exceeds the available Metal budget.
Log the failure using the existing GGML_LOG_ERROR convention and return
NULL so the allocator surfaces a diagnosable error up the stack instead of
crashing.
* cont : fix log
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Bump version to 1.3.9
* Fix JFKReader's format
* Add Whisper::Context#free
* Add test for Whisper::Context#free
* Add Whisper::Context#free to RBS
* Add Whisper::VAD::Context#free
* Add test for Whisper::VAD::Context#free
* Add Whisper::VAD::Context#free to RBS
* Add Parakeet::Context#free
* Add test for Parakeet::Context#free
* Add Parakeet::Context#free to RBS
* Check MemoryView format more strictly
* Fallback MemoryView flags to SIMPLE when MemoryView not available
* Add changelog entries
* Update document on callbacks
* Remove whitespace
Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
---------
Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
whisper_full()/whisper_full_with_state() only compute the mel spectrogram when
n_samples > 0. For n_samples == 0 on a freshly allocated state the mel is never
touched, but struct whisper_mel had no member initializers, so n_len / n_len_org
/ n_mel were indeterminate heap garbage (the state is allocated with
new whisper_state). seek_end is derived from that garbage and, depending on it,
the call either quietly returns 0 or runs the encoder with garbage dimensions
over an empty (NULL) mel buffer, dereferencing address 0 in the mel copy loop.
Give whisper_mel default member initializers so a never-computed mel reads as
0 frames and the n_samples == 0 case deterministically takes the existing
too-short path.
Fixes#3978
* metal : per-device tuned (Q, NE) for flash-attn vec (llama/25750)
* rebase Q-generic FA vec body from 01dc93607 (llama/23114)
* add 53 f16 (Q,NE) flash-attn vec instantiations (vec 80 -> 133)
* add FA vec (Q,NE) tuning table + dispatch wiring + SMEM cap fallback
* add FA vec (Q,NE) perf sweep
* fill tuning result
* fold family table into a per-family representative SKU
* refactor tuning result format
* extend FA vec tuning to quantized KV caches
* sync fa vec tuner bucketing with runtime, use pointwise tuning regret
* update tuned table
* format and cleanup
* prefix fa_vec tuning procs with ggml_backend_metal_tuning_, drop unused fa_vec_override_active
* add device id -> token lookup for the offline tuning tool
* add ggml-metal-tuning skeleton
* add op-agnostic perf cell + median timing for the tuner
* add FA-vec graph build + tensor init to the tuner
* tools : add FA-vec (Q,NE) sweep, compression and table emit
* cool down and re-measure the dirty window on thermal drift
* test-backend-ops : replace the FA vec tune mode with a bounded (Q,NE) slice
* tools : document the Metal tuner, point the table comment at it
* abort on unknown KV type, single-source fa_vec_legal_ne
* cleanup
* honor -o in the FA vec (Q,NE) slice
* retune FA-vec (Q, NE) under a pointwise no-harm gate
* cont : add fa-vec tunings for M1 Pro, M2 Ultra, M5 Max
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* DSV4: sm tensor
* set coarser granularity for head splits
* fix dspark
* add model saving for dsv4 + allow dflash to return on specific device
* add comment about dsv4 seq_rm
* simplify
* add shared expert delayed allreduce
* remove special test for dsv4
* opencl: fold the gpt-oss MoE bias adds into swiglu_oai
Default on, opt out with GGML_OPENCL_FUSE_MOE_BIAS_GLU=0.
* opencl: fold the MoE down-projection bias into the combine
Default on, opt out with GGML_OPENCL_FUSE_MOE_BIAS_COMBINE=0.
* whisper : guard null source in buffer loader read callback
whisper_init_from_buffer_with_params_no_state installs a read callback that
copies from buf->buffer + current_offset. When the buffer is exhausted (or the
supplied buffer is empty), size_to_copy is 0 and the source pointer can be null;
passing a null pointer to memcpy is undefined behavior even for a zero-length
copy (UBSan: 'null pointer passed as argument 2' at the memcpy). Loading a
crafted/short model through the buffer loader could hit this.
Skip the memcpy when there is nothing to copy. Loading from a null/empty or
truncated buffer now fails gracefully (returns NULL) with no UB.
This addresses bug 1 of #3879. Bug 2 (integer overflow when sizing the mel
filter buffer) is covered by the open PR #3780.
* fixup! whisper : guard null source in buffer loader read callback
---------
Co-authored-by: Ben Younes <2910651+ousamabenyounes@users.noreply.github.com>
This commit updates the OpenVINO model conversion script and its
dependencies as they were currently not working.
The first issue was an import that needed updating for fix the following
error:
```console
(openvino_conv_env) $ python convert-whisper-to-openvino.py --model base.en
Traceback (most recent call last):
File "/home/danbev/work/ai/whisper-work/models/convert-whisper-to-openvino.py", line 6, in <module>
from openvino.runtime import serialize
ModuleNotFoundError: No module named 'openvino.runtime'
```
And after that there was a missing dependency:
```console
ModuleNotFoundError: No module named 'onnxscript'
```
With the changes in this commit I was able to successfully convert the model
and run the inference using OpenVINO.
* scripts : add release.sh script [no ci]
This commit adds a release.sh script similar to what ggml and llama.cpp
have to prepare a release.
* scripts : use sed_inplace [no ci]
This commit updates the README.md to include a release badge.
The motivation for this is that we have been manually updating the
Stable link in this document for releases. Removing this means that we
don't have to touch this file and only update CMakeLists.txt.
I also removed the link to the Roadmap was it felt out of place and it
has not been kept up to date so I hope that is alright.
I'll follow up with a PR to introduce a release script that will help
assist the creation of releases (updateing the version and creating the
release PR etc).
This commit adds the --parallel flag to the cmake build command to
speed up compilation. This will hopefully help a little with CI runs
even though the are often limited to 2 cores.
* Add DMMV Q4_K and Q6_K ESIMD kernels
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Refactor ESIMD kernels to share common code
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Move control of ESIMD from compile to runtime
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Use ESIMD by default when available
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Fix possible error when using ESIMD by default
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add explicit unroll to ESIMD kernels
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Tidy up ESIMD kernels a bit
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add a reordered Q2_K MMVQ kernel
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add DMMV Q2_K ESIMD kernel
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
---------
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add DMMV Q4_K and Q6_K ESIMD kernels
Configure cmake build with -DGGML_SYCL_ESIMD=ON to enable.
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Refactor ESIMD kernels to share common code
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Move control of ESIMD from compile to runtime
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Use ESIMD by default when available
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Fix possible error when using ESIMD by default
While not an issue in the current version, this will become an
issue when additional QK ESIMD kernels are added (such as Q2_K).
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add explicit unroll to ESIMD kernels
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Tidy up ESIMD kernels a bit
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Add DMMV Q5_K ESIMD kernel
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* Remove redundant copyright notice
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
---------
Signed-off-by: Todd Malsbary <todd.malsbary@intel.com>
* opencl: keep the vocab-scale K-quant lm_head on the CPU on the Adreno A7X
* opencl: revise comments
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
David Friehs