The cpu and cuda backends use fp16 for the VKQ accumulator type, this change
does the same for vulkan. This helps particularly with large head sizes which
are very register-limited.
I tried this for the coopmat1 path and it slowed down a bit. I didn't try for
scalar.
I applied the softmax bias that the cuda backend uses to avoid overflow,
although I was not able to reproduce the original bug without it.
Write out a 2-bit code per block and avoid loading the mask when it
matches these two common cases.
Apply this optimization when the mask is relatively large (i.e. prompt
processing).
* vulkan: fix GPU deduplication logic.
As reported in https://github.com/ggml-org/llama.cpp/issues/19221, the
(same uuid, same driver) logic is problematic for windows+intel igpu.
Let's just avoid filtering for MoltenVK which is apple-specific, and
keep the logic the same as before 88d23ad5 - just dedup based on UUID.
Verified that MacOS + 4xVega still reports 4 GPUs with this version.
* vulkan: only skip dedup when both drivers are moltenVk
* ggml-virtgpu: regenerate_remoting.py: add the ability to deprecate a function
* ggml-virtgpu: deprecate buffer_type is_host remoting
not necessary
* ggml-virtgpu: stop using static vars as cache
The static init isn't thread safe.
* ggml-virtgpu: protect the use of the shared memory to transfer data
* ggml-virtgpu: make the remote calls thread-safe
* ggml-virtgpu: backend: don't continue if couldn't allocate the tensor memory
* ggml-virtgpu: add a cleanup function for consistency
* ggml-virtgpu: backend: don't crash if buft->iface.get_max_size is missing
* fix style and ordering
* Remove the static variable in apir_device_get_count
* ggml-virtgpu: improve the logging
* fix review minor formatting changes
* CUDA: use mmvq for mul-mat-id for small batch sizes
* add mmvq too
* Fix perf issue on ampere. Use mmvf mm-id only for non-nvidia GPUs
* templatize multi_token_path
Hangs were reported on Jetson Orin AGX if we set CUDA_SCALE_LAUNCH_QUEUES=4x. Reverting the previous PR (#19042) and updating the document to consider setting CUDA_SCALE_LAUNCH_QUEUES=4x for faster throughput on multi-GPU systems.
* Remove mutex for pipeline caches, since they are now per-thread.
* Add comment
* Run clang-format
* Cleanup
* Run CI again
* Run CI once more
* Run clang-format
* wip
* ggml-hexagon: add vectorized dot product function for FP32 and FP16 accumulation
* ggml-hexagon: optimize dot product functions for FP16 and FP32 with new vectorized implementations
* wip
* ggml-hexagon: optimize hvx_vec_dump_f32_n and hvx_vec_reduce_sum_qf32x2 functions for improved performance
* ggml-hexagon: refactor dot product functions to use a common loading function for improved readability
* optimize vector dot product functions to use unified reduction for improved performance
* wip
* ggml-hexagon: add vectorized dot product function for FP32 and FP16 accumulation
* ggml-hexagon: optimize dot product functions for FP16 and FP32 with new vectorized implementations
* wip
* ggml-hexagon: optimize hvx_vec_dump_f32_n and hvx_vec_reduce_sum_qf32x2 functions for improved performance
* ggml-hexagon: refactor dot product functions to use a common loading function for improved readability
* optimize vector dot product functions to use unified reduction for improved performance
* hexagon: optimize reduce-sum for v75+
* hexagon: always keep row_sums in sf/fp32
* ggml-hexagon: enhance directory checks for HEXAGON_SDK_ROOT and HEXAGON_TOOLS_ROOT
* fix compiling error after rebase
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
* Add Q8_0 OpenCL kernel
Co-authored-by: yunjie <yunjie@qti.qualcomm.com>
* opencl: fix build for non-adreno
* opencl: refactor q8_0
* opencl: enforce subgroup size of 64 for adreno for q8_0
* For A750 and older generations, subgroup size can be 64 or 128.
This kernel assumes subgroup size 64.
* opencl: suppress warning when adreno kernels are disabled
---------
Co-authored-by: yunjie <yunjie@qti.qualcomm.com>
Co-authored-by: Li He <lih@qti.qualcomm.com>
* Don't convert to temporary VALUE
* Define Whisper::Context::Params
* Add test for Whisper::Context::Params
* Implement Whisper::Context::Params
* Add tests for Context::Params
* Fix Whisper::Token memory management
* Add test for token_timestamps
* Make Context accept Context::Params
* Make Context::Params.new accept keyword args
* Add test for Context::Params.new with keyword args
* Add signature of Context::Params
* Add example for Whisper::Token
* Fix typos
* Revert "Don't convert to temporary VALUE"
This reverts commit dee66e7384.
* Hold Token#text as Ruby objectd
* Don't use pointer for ruby_whisper_context_params.params
* Use RUBY_DEFAULT_FREE instead of custom function
* Update bindings/ruby/README.md
Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
* Add document for Whisper::Context::Params
---------
Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
* ruby : Bump version to 1.3.6
* Fix code in example
* Add sample code to transcribe from MemoryView
* Define GetVADContext macro
* Use GetVADContext
* Extract parse_full_args function
* Use parse_full_args in ruby_whisper_full_parallel
* Free samples after use
* Check return value of parse_full_args()
* Define GetVADParams macro
* Add VAD::Context#segments_from_samples
* Add tests for VAD::Context#segments_from_samples
* Add signature for VAD::Context#segments_from_samples
* Add sample code for VAD::Context#segments_from_samples
* Add test for Whisper::Context#transcribe with Pathname
* Make Whisper::Context#transcribe and Whisper::VAD::Context#detect accept Pathname
* Update signature of Whisper::Context#transcribe
* Fix variable name
* Don't free memory view
* Make parse_full_args return struct
* Fallback when failed to get MemoryView
* Add num of samples when too long
* Check members of MemoryView
* Fix a typo
* Remove unnecessary include
* Fix a typo
* Fix a typo
* Care the case of MemoryView doesn't fit spec
* Add TODO comment
* Add optimazation option to compiler flags
* Use ALLOC_N instead of malloc
* Add description to sample code
* Rename and change args: parse_full_args -> parse_samples
* Free samples when exception raised
* Assign type check result to a variable
* Define wrapper function of whisper_full
* Change signature of parse_samples for rb_ensure
* Ensure release MemoryView
* Extract fill_samples function
* Free samples memory when filling it failed
* Free samples memory when transcription failed
* Prepare transcription in wrapper funciton
* Change function name
* Simplify function boundary
* sycl: add softplus unary op implementation
* sycl: add softplus unary op implementation
* docs(ops): mark SYCL SOFTPLUS as supported
* docs: update SYCL status for SOFTPLUS
Nechama Krashinski
Jeff Bolz
Georgi Gerganov
Oleksandr Kuvshynov
will-lms
Kevin Pouget
Aman Gupta
Ruben Ortlam
Oliver Simons
George
Gaurav Garg
lhez
Neo Zhang
Tamar
Johannes Gäßler
Christian Kastner
Nikhil Jain
Max Krasnyansky
nullname
shaofeiqi
Simon Redman
KITAITI Makoto
Frieder Bluemle
bssrdf
s8322
RachelMantel
Zheyuan Chen
Todor Boinovski
yulo
Vishal Singh