27101c01dc
* cli : merge tokens split across UTF-8 boundaries in JSON output When a multi-byte UTF-8 codepoint (most commonly a CJK character, 3 bytes) is split across multiple whisper tokens, the -ojf/--output-json-full writer emitted each token's partial bytes as its own JSON string, producing invalid UTF-8 that chokes downstream parsers. Merge adjacent tokens in output_json whenever the accumulated text still ends on an incomplete UTF-8 sequence. The merged entry keeps the first token's id/p/t_dtw and extends t1 to the last absorbed token, which matches how segment text is assembled elsewhere. Refs #1798 * fix: address review — add braces for consistency, use full issue URL - Add braces to if/else chain for codebase consistency - Use full URL for issue #1798 reference Review: @danbev --------- Co-authored-by: texasich <texasich@users.noreply.github.com> Co-authored-by: texasich <texasich@gmail.com> |
||
|---|---|---|
| .. | ||
| addon.node | ||
| bench | ||
| bench.wasm | ||
| cli | ||
| command | ||
| command.wasm | ||
| deprecation-warning | ||
| lsp | ||
| python | ||
| quantize | ||
| server | ||
| stream | ||
| stream.wasm | ||
| sycl | ||
| talk-llama | ||
| vad-speech-segments | ||
| wchess | ||
| whisper.android | ||
| whisper.android.java | ||
| whisper.nvim | ||
| whisper.objc | ||
| whisper.swiftui | ||
| whisper.wasm | ||
| CMakeLists.txt | ||
| coi-serviceworker.js | ||
| common-ggml.cpp | ||
| common-ggml.h | ||
| common-sdl.cpp | ||
| common-sdl.h | ||
| common-whisper.cpp | ||
| common-whisper.h | ||
| common.cpp | ||
| common.h | ||
| ffmpeg-transcode.cpp | ||
| generate-karaoke.sh | ||
| grammar-parser.cpp | ||
| grammar-parser.h | ||
| helpers.js | ||
| json.hpp | ||
| livestream.sh | ||
| miniaudio.h | ||
| server.py | ||
| stb_vorbis.c | ||
| twitch.sh | ||
| yt-wsp.sh | ||