Files
bambuddy/backend/tests/unit/test_mqtt_debug_on_change.py
maziggy ce807fb1cc fix(queue): upload to printers in parallel, cap wedge retries, make debug logs survive a farm
The reporter's 19-printer farm started prints "one by one", up to an hour apart.
check_queue awaited each dispatch inline, and a dispatch includes the FTP upload,
so every printer queued behind every other printer's transfer despite being an
independent machine. His logs give the arithmetic: 40978500 bytes in 254.1s,
157 KB/s - a Bambu printer's SD write, not the network, is the bottleneck. Nineteen
of those in series is ~80 minutes, and the next upload started 131 ms after the
previous one finished. The delay is linear in fleet size, which is why it got worse
the more printers he selected.

Dispatch is now collected during the (still sequential) selection loop and run
concurrently afterwards, capped by queue_max_concurrent_uploads - Settings ->
Workflow -> Queue & Dispatch, default 4, 1 restores the old behaviour. Every gate
is untouched; only the transfers overlap. The pass still awaits its uploads before
returning: _start_print flips the row pending -> printing only after the upload,
so an early return would let the next tick re-dispatch the same rows.

FTP work moves to its own thread pool. It was on asyncio's default executor -
min(32, cpu+4), six threads on a 2-core NAS, shared with everything else - which
was survivable only while uploads were serial.

Two problems the same bundle exposed:

A printer that accepts project_file but never starts (#1678) was retried forever:
270s watchdog, revert to pending, re-upload the whole file, repeat. Hence his
"printer who, since the morning, still not launch" - and on a farm each lap also
eats an upload slot the other printers are waiting on. Attempts are now counted on
the queue item; after three it fails with a message pointing at the printer instead
of queueing a fourth re-upload.

The debug bundle we asked him for held 4m49s of history. The push_status dumps fired
on every frame rather than on change - several while their own comment claimed
otherwise - which is 27,727 of the bundle's 29,830 lines and rolls 5 MB in under five
minutes on 19 printers. They now log transitions only. The bundle also read just the
live log while three rotated backups sat next to it, under a byte budget four times
larger than the file it was reading.

Migration verified on SQLite and Postgres: idempotent, backfills legacy NULLs
(dispatch_attempts + 1 is NULL for a NULL row, which would silently disable the cap).

Tests: 6 on concurrent dispatch (overlap, cap honoured, 1 == serial, default applies
with no settings row, a failed printer does not cancel its siblings, no early return),
4 on the retry budget, 6 on the bundle's rotated-log span, 7 on the debug gating.
Each verified to fail against the unfixed code - the first end-to-end log assertion I
wrote passed without the fix and had to be tightened.
2026-07-14 10:35:58 +02:00

184 lines
8.7 KiB
Python

"""Per-frame debug dumps must log transitions, not every frame (#2555).
The state dumps in the push_status handler fired whenever their field was
*present* in the frame. A full push_status carries every field, so they fired on
every frame regardless of whether anything had changed — several while their own
comment claimed to log "when X changes".
On one printer that is ~1.5 lines/s and nobody noticed. On a 19-printer farm it
is ~100 lines/s: the reporter turned on debug logging as asked, and the 5 MB log
rolled over in under five minutes. 27,727 of the 29,830 lines in the support
bundle were these dumps, and the queue problem we were chasing was nowhere in the
window.
"""
import logging
from unittest.mock import MagicMock, patch
from backend.app.services.bambu_mqtt import BambuMQTTClient
def _client() -> BambuMQTTClient:
return BambuMQTTClient(ip_address="10.0.0.1", serial_number="SERIAL", access_code="code", model="A1")
class TestDebugOnChange:
def test_repeated_identical_values_log_once(self):
client = _client()
with patch("backend.app.services.bambu_mqtt.logger") as log:
for _ in range(50):
client._debug_on_change("wifi_signal", -52, "[%s] wifi_signal: %s", "SERIAL", -52)
assert log.debug.call_count == 1, (
f"50 identical frames produced {log.debug.call_count} log lines — this is the flood"
)
def test_each_change_is_logged(self):
"""Suppressing repeats must not suppress transitions — the transitions are
the entire reason anyone reads these lines."""
client = _client()
with patch("backend.app.services.bambu_mqtt.logger") as log:
for value in (-52, -52, -60, -60, -52):
client._debug_on_change("wifi_signal", value, "[%s] wifi_signal: %s", "SERIAL", value)
assert log.debug.call_count == 3
assert [c.args[-1] for c in log.debug.call_args_list] == [-52, -60, -52]
def test_keys_are_tracked_independently(self):
client = _client()
with patch("backend.app.services.bambu_mqtt.logger") as log:
client._debug_on_change("tray_now", 1, "tray_now: %s", 1)
client._debug_on_change("ams_status", 1, "ams_status: %s", 1)
client._debug_on_change("tray_now", 1, "tray_now: %s", 1) # repeat, suppressed
assert log.debug.call_count == 2, "same value under a different key must not be swallowed"
def test_printers_are_tracked_independently(self):
"""State is per-client. Two printers reporting the same value must each
get their own line — a farm is exactly where this matters."""
a, b = _client(), _client()
with patch("backend.app.services.bambu_mqtt.logger") as log:
a._debug_on_change("tray_now", 3, "tray_now: %s", 3)
b._debug_on_change("tray_now", 3, "tray_now: %s", 3)
assert log.debug.call_count == 2
def test_composite_values_detect_a_change_in_any_field(self):
"""Messages that render several fields must pass all of them, or a change
in the unwatched field is silently dropped."""
client = _client()
with patch("backend.app.services.bambu_mqtt.logger") as log:
client._debug_on_change("chamber", (40.0, 0.0, False), "chamber %s %s %s", 40.0, 0.0, False)
client._debug_on_change("chamber", (40.0, 60.0, True), "chamber %s %s %s", 40.0, 60.0, True)
assert log.debug.call_count == 2, "target/heating changed while current stayed 40.0 — must still log"
def test_dict_values_compare_by_content(self):
"""The AMS dict dump is the biggest line by volume; it is a fresh dict every
frame, so identity comparison would never suppress anything."""
client = _client()
with patch("backend.app.services.bambu_mqtt.logger") as log:
for _ in range(10):
client._debug_on_change("ams", {"tray_now": "0", "bits": "7000000"}, "ams: %s", {})
client._debug_on_change("ams", {"tray_now": "1", "bits": "7000000"}, "ams: %s", {})
assert log.debug.call_count == 2
class TestRuntimeDebugToggle:
"""Debug logging is turned on at RUNTIME (POST /support/debug-logging) and these
clients outlive the toggle — which is the whole workflow this change serves:
"enable debug logging, reproduce, send the bundle".
So the cache must not be warmed while running at INFO. If it were, the operator
would enable debug, and every steady-state value would already be "seen" — an
idle printer's bundle would contain none of these lines at all, which is worse
than the flood it replaced.
"""
def test_enabling_debug_at_runtime_still_dumps_a_baseline(self):
client = _client()
mqtt_logger = logging.getLogger("backend.app.services.bambu_mqtt")
original = mqtt_logger.level
try:
# Steady state at INFO: the app has been running for hours.
mqtt_logger.setLevel(logging.INFO)
with patch("backend.app.services.bambu_mqtt.logger", wraps=mqtt_logger) as log:
for _ in range(200):
client._debug_on_change("wifi_signal", -52, "wifi %s", -52)
assert log.debug.call_count == 0, "nothing should be emitted at INFO"
# Operator flips debug on. The value has NOT changed — but they turned
# this on to see the printer's state, so the very next frame must dump it.
mqtt_logger.setLevel(logging.DEBUG)
with patch("backend.app.services.bambu_mqtt.logger", wraps=mqtt_logger) as log:
client._debug_on_change("wifi_signal", -52, "wifi %s", -52)
assert log.debug.call_count == 1, (
"no baseline after enabling debug — the cache was warmed while at "
"INFO, so the operator sees nothing until the value happens to change"
)
# ...and it still dedups from there.
for _ in range(50):
client._debug_on_change("wifi_signal", -52, "wifi %s", -52)
assert log.debug.call_count == 1
finally:
mqtt_logger.setLevel(original)
def test_disabling_debug_drops_the_cache(self):
"""Off -> on must be as cold as a fresh process, not just first-ever-on."""
client = _client()
mqtt_logger = logging.getLogger("backend.app.services.bambu_mqtt")
original = mqtt_logger.level
try:
mqtt_logger.setLevel(logging.DEBUG)
client._debug_on_change("tray_now", 2, "tray_now %s", 2)
assert client._debug_last
mqtt_logger.setLevel(logging.INFO)
client._debug_on_change("tray_now", 2, "tray_now %s", 2)
assert client._debug_last == {}, "cache must be dropped while debug is off"
finally:
mqtt_logger.setLevel(original)
class TestRealDumpSitesAreGated:
"""End-to-end: feed the same push_status frame twice and count the lines."""
def test_identical_push_status_frames_do_not_re_dump_state(self):
# Deliberately the client's real PrinterState, not a mock: a MagicMock
# state would return the same stub object for every attribute read, so
# the values would compare equal and the test would pass even with the
# gating removed.
client = _client()
assert not isinstance(client.state, MagicMock)
frame = {
"print": {
"ams": {
"ams": [],
"ams_exist_bits": "1",
"tray_exist_bits": "f",
"tray_now": "0",
},
"wifi_signal": "-52dBm",
"ipcam": {"ipcam_record": "enable"},
}
}
logging.getLogger("backend.app.services.bambu_mqtt").setLevel(logging.DEBUG)
with patch("backend.app.services.bambu_mqtt.logger") as log:
log.isEnabledFor.return_value = True
client._process_message(dict(frame))
first = [c.args[0] for c in log.debug.call_args_list]
log.debug.reset_mock()
client._process_message(dict(frame))
second = [c.args[0] for c in log.debug.call_args_list]
# Frame 1 must still dump — the point is to log transitions, not to go quiet.
assert first, "the first frame stopped dumping state entirely — the logs are now useless"
# Frame 2 is byte-identical, so it must produce NOTHING. Asserting merely
# "fewer than frame 1" is not enough: a couple of these sites happen to be
# naturally one-shot, so an ungated build still measures 3 < 5 and the
# assertion passes while every real dump keeps firing on every frame.
assert second == [], f"an identical push_status frame re-dumped {len(second)} line(s): {second}"