amritahs-ibm 0001ec075f llamafile : ppc64le GEMV forwarding for FP32. (llama/12594) ... This patch enables usage of MMA when one of the dimensions of the matrix(ie either M or N) is 1. This is useful in case of token generation where N < 2. The concept of 'GEMV Forwarding' is used where when one of the matrix has a single row/column, the elements are broadcasted, instead of using packing routine to prepack the matrix elements. This change results in 5% - 15% improvement in total speed(ie all tokens/total time), across various batch sizes. This is in comparision with the corresponding dot product implementation. The patch is tested with FP32 models of Meta-Lllama-3-8B, Mistral-7B, Llama-2-7B-chat-hf on a IBM POWER10 machine. Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>		2025-03-28 21:47:42 +02:00
..
cmake	ggml : sync/merge cmake,riscv,powerpc, add common.cmake (ggml/0)	2025-03-27 11:06:03 +02:00
include	rpc : send hash when tensor data is above some fixed threshold (llama/12496)	2025-03-28 21:47:42 +02:00
src	llamafile : ppc64le GEMV forwarding for FP32. (llama/12594)	2025-03-28 21:47:42 +02:00
.gitignore	whisper : reorganize source code + improve CMake (#2256)	2024-06-26 19:34:09 +03:00
CMakeLists.txt	ggml : sync/merge cmake,riscv,powerpc, add common.cmake (ggml/0)	2025-03-27 11:06:03 +02:00