ggml-cuda : disable MMQ on devices with less than 48 KiB shared memory (llama/26141)
ggml_cuda_should_use_mmq() selects MMQ purely from the quantization type. The current MMQ configurations are designed and maintained against a minimum of 48 KiB per-block shared memory, the limit provided by NVIDIA Pascal GPUs and later. On devices that report less, no supported MMQ tile fits and mul_mat_q_switch_J() aborts when every tile size exceeds the device's per-block shared memory budget. Disable MMQ when smpbo < 48 KiB so the caller falls back to the BLAS path instead of hitting GGML_ABORT. Some current MUSA QY1 devices report only 28 KiB and are covered by this guard. Reproduced on a Moore Threads MTT S70 (arch mp_21, 28 KiB shared memory per block) with an RWKV-7 0.1B Q8_0 model: $ llama-bench -m rwkv7-g1d-0.1b-Q8_0.gguf -p 128 -n 0 J_best=0 ggml/src/ggml-cuda/template-instances/../mmq.cuh:1521: fatal error (core dumped) Only prefill (batch > 1) is affected; token generation is fine. After the fix the same device falls back to the BLAS path: Q8_0 pp128 1470.7 t/s, tg8 55.3 t/s (was: abort) FP16 unchanged Q4_K_M unchanged This matches a -DGGML_CUDA_FORCE_CUBLAS=ON build (pp128 1464.2 t/s), which confirms the fallback path is the one being taken. This is not MUSA-specific: any device with less than 48 KiB per-block shared memory is affected. Co-authored-by: KakaruHayate <KakaruHayate@users.noreply.github.com>
This commit is contained in:
parent
e103c72182
commit
a816f4064b
|
|
@ -296,6 +296,15 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
|
|||
return false;
|
||||
}
|
||||
|
||||
// MMQ tiles require at least 48 KiB per-block shared memory; fall back to BLAS otherwise.
|
||||
{
|
||||
const int id = ggml_cuda_get_device();
|
||||
const size_t smpbo = ggml_cuda_info().devices[id].smpbo;
|
||||
if (smpbo < 48 * 1024) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
if (turing_mma_available(cc)) {
|
||||
return true;
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue