mirror of
https://github.com/ggml-org/whisper.cpp.git
synced 2026-10-12 01:05:43 +02:00
jboothandGeorgi Gerganov
447a77510d
* Added tiled mul_mat. For each mul_mat_one_chunk, quants are unpacked into (max) 256x256 tiles of int8, one routine per quent. Then microkernel computes 16x16 tiles before writing out 256x256 float reults to main memory. Tests/benches in tests/test-tiled-mulmat.cpp. 3-6x speed improvement for large matmul, break even at 4096x64 * 64x4096, 80% performance (net loss) for GEMV. Error rates trivial (order of 1-e04 max, 1-e05 rmse). * Fixes for ARM/windows builds * more windows fixes, ggml-cpu.h isn't visible in MSVC for some reason * unified iqp + tiled on the Q5_K, IQ4_XS set for benchmarking, updated benchmark * Fixed accidental removal of llama_build_and_test(test-backend-ops.cpp) * First integration of iqp code Co-authored-by Bartowski <3266127+bartowski1182@users.noreply.github.com> * Cleaning up declaration of iq unpacking helpers to align with the bit unpackers * Removed iqp path * Fix cross-platform warnings * Disabling benchmarks unless explicitly enabled * Fix backend_init for DLL-based builds, add self and bartowski to CODEOWNERS for tiled * Put benchmarks behind a flag * kernel fix for AVX2, iq quants * Fix for asan, leaking memory in test-tiled-mulmat and avoid stack use after return * guarding env flags with std::call_once * Simplified repacking for VNNI to a single call per macrotile * No threadlocals anymore, aligned wdata access * Doing aligned reads since we ensure alignment with padding in wdata * Eliminated per-thread gather of Q8_K rows in mul_mat_id, we now gather/repack in a single pass. Repack method now takes pointer array to support both dense/normal and mmid paths. Interface with ggml-cpu.c simplified as a result * Unified/simplified dispatch and support checks. Put details on wdata needed inside the kernel.h body, simplified interactions with ggml-cpu.c. * Cleanup includes and whitespace, update src1_repack to return false if we don't need a special repack, so the common case is handled by driver * Better detection of win32 and additional whitespace fixes * Gating fuzz tests behind a parameter and some extra prints to try and fix slow CI hosts * Optimized AVX2 kernel * Changed interleave format and added ability to interleave in-place after dequant * Repacks now happen in-place, 16x64 microtiles are independent of each other * Only repack rows in groups of 16 as they're needed. Save work in low n_rows cases and optimize L1 usage in other cases * Use long panels for memory-bound regime (M <= 16), reintroduce IQP path for benchmarks * Fix unused warnings and cleanup. Improved IQ dequantization speed. * Removed separate process benchmarks * Revert "Removed separate process benchmarks" This reverts commit 0688cf43d5e37ec7a52828c5576df563514fe760. * AVX2 optimizations and guards for tests on windows * Removed temp perf harness * Remove perf-mulmat from build * Removed IQP path, simplified tests to not use sub processes * Cleaning up alignment of wdata * Whitespace fixes and aligning L2 workspace to clean 512kb boundaries * Update ggml/src/ggml-cpu/tiled/tiled-kernel.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Cleanup merge-duplicated declaration of test-backend-ops target * Undo accidental line deletion in ggml.c --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>