whisper.cpp/tests
Ben Younes 6a27b12f27 cli : drop invalid UTF-8 bytes from JSON output
whisper-cli --output-json-full can emit a lone UTF-8 lead byte as a whole
segment's "text" value, because byte-level decoder tokens (large-v3-turbo)
can leave an incomplete UTF-8 sequence at a segment boundary. The resulting
.json file is not valid UTF-8 and breaks json.load / jq / iconv, while
whisper-cli still exits 0.

Add utf8_sanitize() next to the existing utf8_trailing_bytes_needed() helper in
common-whisper, dropping every byte that is not part of a valid, complete UTF-8
sequence (orphan continuation bytes, truncated sequences, overlong/surrogate
encodings). The JSON string escaper now sanitizes before escaping, so every
string written to the JSON output is well-formed UTF-8.

Fixes #3760
2026-08-16 16:07:34 +00:00
..
earnings21 vad : Silero VAD v6.2.0 (#3524) 2025-11-17 22:26:17 +09:00
librispeech musa: correct MUSA SDK rc4.0.1 download URL (#3217) 2025-06-03 06:02:12 +02:00
librispeech-parakeet parakeet : add support for NVIDIA Parakeet (#3735) 2026-06-16 20:44:10 +02:00
.gitignore tests : adding transcription tests 2022-12-02 21:40:02 +02:00
CMakeLists.txt cli : drop invalid UTF-8 bytes from JSON output 2026-08-16 16:07:34 +00:00
en-0-ref.txt tests : adding transcription tests 2022-12-02 21:40:02 +02:00
en-1-ref.txt tests : adding transcription tests 2022-12-02 21:40:02 +02:00
en-2-ref.txt tests : adding transcription tests 2022-12-02 21:40:02 +02:00
es-0-ref.txt tests : adding transcription tests 2022-12-02 21:40:02 +02:00
parakeet-expected-diffusion-output.txt parakeet : add support for NVIDIA Parakeet (#3735) 2026-06-16 20:44:10 +02:00
parakeet-expected-gb1-output.txt parakeet : add support for NVIDIA Parakeet (#3735) 2026-06-16 20:44:10 +02:00
parakeet-expected-jfk-output.txt parakeet : add support for NVIDIA Parakeet (#3735) 2026-06-16 20:44:10 +02:00
parakeet-verification.h parakeet : add support for NVIDIA Parakeet (#3735) 2026-06-16 20:44:10 +02:00
run-tests.sh parakeet : add support for NVIDIA Parakeet (#3735) 2026-06-16 20:44:10 +02:00
test-c.c whisper : reorganize source code + improve CMake (#2256) 2024-06-26 19:34:09 +03:00
test-common-utf8.cpp cli : drop invalid UTF-8 bytes from JSON output 2026-08-16 16:07:34 +00:00
test-parakeet-full.cpp parakeet : add support for NVIDIA Parakeet (#3735) 2026-06-16 20:44:10 +02:00
test-parakeet.cpp parakeet : verify hparams loaded from parakeet model bin file (#3950) 2026-07-30 06:59:23 +02:00
test-vad-full.cpp whisper : expose internal VAD speech segments (#3916) 2026-07-01 13:10:16 +02:00
test-vad.cpp tests : update VAD tests to use Silero V6.2.0 (#3534) 2025-12-06 10:58:58 +01:00
test-whisper.js Node.js package (#260) 2022-12-12 20:17:27 +02:00