whisper-cli --output-json-full can emit a lone UTF-8 lead byte as a whole
segment's "text" value, because byte-level decoder tokens (large-v3-turbo)
can leave an incomplete UTF-8 sequence at a segment boundary. The resulting
.json file is not valid UTF-8 and breaks json.load / jq / iconv, while
whisper-cli still exits 0.
Add utf8_sanitize() next to the existing utf8_trailing_bytes_needed() helper in
common-whisper, dropping every byte that is not part of a valid, complete UTF-8
sequence (orphan continuation bytes, truncated sequences, overlong/surrogate
encodings). The JSON string escaper now sanitizes before escaping, so every
string written to the JSON output is well-formed UTF-8.
Fixes#3760
* common: add memory buffer overload of read_audio_data
whisper-server /inference without --convert passed the uploaded file
bytes to read_audio_data as a filename, so ma_decoder_init_file tried
to open a path starting with "RIFF" and failed. every request returned
HTTP 400 "Invalid request" on builds without WHISPER_FFMPEG, which is
the default.
factor the PCM extraction into a shared helper and add an overload that
decodes straight from a memory buffer via ma_decoder_init_memory, which
the function already used for the stdin path. server now calls it with
the upload content. the filename overload behavior is unchanged.
Ben Younes
Noah Lyons
Pascal
Georgi Gerganov