A bundle described everything except the process it runs in. So a report of
memory climbing over days until the OOM killer fires arrives with no way to
act on it: the numbers that name the mechanism only exist while it is
happening, and by the time anyone asks, the container has been restarted.
The new `process` section carries what actually separates the candidates.
Resident against virtual memory: 650MB RSS with 12.9GB VMS is address
space — thread stacks or allocator arenas — not a heap full of live data,
and that reading is the opposite of the one the reporter drew from the same
figures. Thread count and child-process count then split those two apart,
and a census of live objects by type names what a growing heap is filling
up with. Open files, sockets and uptime round it out.
Three constraints worth keeping:
The heap census is skipped above 2GB. gc.get_objects() materialises every
tracked object, so it costs most on exactly the process that can least
afford it — a bundle generated to diagnose runaway memory must not be the
allocation that tips the host over. Everything else is still collected, and
the skip is recorded with its reason rather than silently omitted.
Children are recorded by executable name only. An ffmpeg command line
carries the camera URL, and with it the camera's password.
Collection runs off the event loop and every metric is independently
best-effort. psutil raises on hardened kernels and in restricted
containers, and the bundle is how someone reports a problem in the first
place — it has to be produced even when half the numbers are unavailable.
This does not fix#2734, and nothing here should be read as having found
its cause. The bundle's own evidence contradicts both proposed causes: the
orphan janitor ran 7 times in 26 days over 725 stream-ends and killed no
orphaned ffmpeg, which is not the #776 signature; and the 5 "database is
locked" errors all fall between two OOM kills, making them a symptom of the
memory pressure rather than a source of it.
A heavy MakerWorld model — one Bambu Studio also takes a long time over —
failed after five minutes with "Slicer sidecar unreachable". The sidecar
was reachable the whole time and still slicing when we hung up on it.
SlicerApiService carried a hardcoded 300s timeout, passed to httpx as a
bare float so it covered connect, read, write and pool alike. On a single
long request that is not a health check, it is a cap on how long a model is
allowed to take. And because httpx.ReadTimeout subclasses RequestError,
expiry landed in the same handler as a refused connection and was reported
as an unreachable sidecar — so the reporter went and updated their sidecar
container, which was never the problem.
The information to do better was already being collected. _poll_progress
polls /slice/progress/{id} once a second alongside the blocking POST to
drive the live progress toast, so at minute five Bambuddy had fresh
evidence the slicer was working. It killed the request anyway.
So the read timeout comes off the HTTP call and the poller supervises
instead: the deadline moves forward on every progress update, and only
genuine silence ends the wait. A model that keeps reporting runs to
completion however long it takes. Connect and pool keep short timeouts —
a sidecar that will not accept a connection is unreachable and should
still say so quickly.
Only a *changed* progress payload counts as alive. The sidecar re-serves
its last snapshot on every poll, so counting repeats would leave the
watchdog unable to detect a stall at all.
The window is floored at three poll intervals: liveness can only be
observed as fast as the poller ticks, so anything shorter would expire in
the gap between two polls and fail every slice instantly.
New setting slicer_stall_timeout_minutes (Settings > Workflow > Slicer),
default 15, range 1-240, alongside the sidecar URL and gated on
use_slicer_api like its neighbours. Sidecars too old to report progress
have no liveness signal, so for those the same number bounds total elapsed
time — the old behaviour, configurable and no longer 300s flat. The
message says which case applies and where to change it.
SlicerTimeoutError is its own type and maps to 504, not 502: the sidecar
answered throughout, we stopped waiting. Connection failures keep
SlicerApiUnavailableError. The preview slice path gets the same treatment.
Deleting a print that belonged to a project left it on the project page as
a card with a missing thumbnail, and there was no way to remove it.
Deleting a print is a soft delete by default (#1343): the files go from
disk, the row stays so global Quick Stats keeps counting its filament,
time and cost. Every other consumer filters those rows out. The projects
module filtered none of them — the only deleted_at check in the whole file
was for LibraryFile — so a deleted print kept its project_id and kept
being listed, pointing at a thumbnail that no longer existed. The same
broken previews appeared on the overview cards, and in the timeline, where
the entry links to an archive that no longer opens. Unassigning was
impossible because the only UI that can change a print's project lives on
the Archives page, which correctly hides deleted prints: visible on the
project, unreachable from anywhere.
All eight project-scoped archive queries now filter, counts included. That
last part is a deliberate divergence from #1343, where the whole point of
the soft delete is that the contribution survives: a project is a piece of
work with a definite membership, not a lifetime total, so a project that
lists eleven prints must not claim twelve. The reasoning is recorded at
the constant so nobody later "fixes" it back.
remove_archives_from_project keeps working on hidden rows on purpose — it
is the repair path for links written before this. The BOM print_name
lookups are left alone; naming a since-deleted print is still correct.
Two more consumers had the same gap. The CSV/Excel export handed back rows
the interface says are gone — filtered at the base query, since the export
is the list you are looking at saved to a file. Per-project failure
analysis measured a failure rate against prints deleted from the project,
and disagreed with the project's own numbers; only the project-scoped
branch filters, global analysis still counts every run including orphans
as #1390 established.
Finally, the project page needed a manual reload to catch up. staleTime is
60s and the delete mutations invalidated only ['archives'], so a project
visited within the minute served its cached copy, print still there. The
project-assign mutations had the mirror-image bug: ['projects'] refreshed
the overview cards but never ['project', id]. Both now go through one
shared helper covering every project-derived key, as bare prefixes so all
cached project ids are matched.
The reporter's printer lost its session to a keep-alive timeout at 02:19
and did not come back until 11:24 — nine hours offline, with the web UI
open throughout.
check_staleness() was never going to catch it. Its first line is
`if self.state.connected and self.is_stale()`, so it only ever handles the
half-broken session that is still connected but has gone quiet. This
client had connected=False from 02:19:42 (the offline notification fired a
minute later), so every call returned immediately, and paho's own retry was
the only thing left watching. When that stopped making progress nothing
noticed.
Adds a sweep every 60s that rebuilds a client when all four hold: it is
disconnected, it had a working session before, it has been silent for five
minutes, and its MQTT port still answers. The port check is what keeps this
from becoming a nuisance — a switched-off printer is left to paho, so a
farm powering down overnight causes no client churn and no log spam. The
five-minute grace sits well past the 60s stale timeout and the 30s max
reconnect backoff, so a session recovering on its own is never interrupted.
The rebuild goes through force_reconnect_stale_session from async context,
which takes the hard-reset path: fresh client_id and paho's QoS 1 queue
dropped, so a project_file left unacked on the dead session cannot replay
into the new one and trip 0500_4003 (#1136). Rate-limited per printer,
cooldown cleared when the printer returns, and the sweep continues past a
client that throws rather than abandoning the rest of the farm. The log
line names how long the printer was gone and the last connect error, so a
session that dies repeatedly leaves a trail.
check_port gains a public alias in printer_diagnostic rather than having
the watchdog reach for the private name.
Also corrects the Developer Mode path added in the previous commit: the
wiki documents it under Settings > Network, not Settings > General. The
menu path is dropped from the translated string entirely, since it varies
by model and firmware and the wiki carries the detail.
A P1S on firmware 01.10.00.00 rejected every control command and said so:
HMS 0500-0500-0001-0007, "MQTT command verification failed". Bambuddy
received that, dropped it, and reported a healthy printer instead.
The frontend filtered it out. This code's meaning lives in attr's low half
(0500) and code's high half (0001), both of which the MMMM_EEEE short form
discards, so it collapsed to "0500_0007" — no catalog entry, no firmware
actions, and filterKnownHMSErrors drops uncatalogued action-less errors.
Catalog lookups now try full_code first, in both the description and the
filter, and errors matched that way display the four-group code the
printer's own screen shows. The remedy line is ours, not Bambu's: their
wiki says to update Studio or Handy, which does not apply to a print sent
from Bambuddy.
The developer-mode probe made it worse. It read anything that was not an
explicit refusal as confirmation, and this firmware answers the probe with
an empty result while refusing everything else — so an inference drawn
from a non-answer became "developer_mode: pass" in the support bundle of a
printer that had not accepted a command all day. The probe now has three
outcomes: explicit success enables, explicit verify-failure disables,
anything else stays unknown and the diagnostic reports skip.
The HMS is authoritative over that inference in both directions. It forces
developer_mode False when present, and clears back to unknown when the
printer stops reporting it, so enabling Developer Mode and restarting the
printer is picked up without restarting Bambuddy.
Dispatch no longer treats a refusal as a wedge. The watchdog latches the
HMS across both phases and fails the item on the first attempt naming the
code and the fix, rather than spending three uploads and 270s a lap to
arrive at a message about SD cards. The check runs after the active-state
exit in both phases, so a lingering HMS can never abort a print that is
visibly running.
Also: the "wrong or mis-cased serial number" hint no longer fires in the
moment after a reconnect. _report_messages_since_connect is reset by
_on_connect, so a reconnect landing microseconds before the staleness
check leaves it at 0 for reasons that have nothing to do with the serial —
this reporter's healthy printer was told to go check its serial 1 ms after
reconnecting.
Obico's ml_api container takes an optional ML_API_TOKEN environment variable.
With it set, ml_api/auth.py answers a bare 401 to any request whose
Authorization header isn't "Bearer <token>"; with it unset it ignores the
header entirely. Bambuddy never sent one, so pointing it at a protected server
meant deleting the token there — which the reporter had set for their Home
Assistant integration and did not want to undo.
Settings -> Failure Detection gains an ML API Token field. When it is empty no
header is sent, so an unconfigured install's request stays byte-identical to
what shipped before the setting existed.
This failed in the worst possible way, and that is the more important half of
the change. Obico decorates /p/ with token_required but leaves /hc/ open. Test
Connection pinged /hc/, so it reported success against a server that was
rejecting every real detection call, the settings looked right, and detection
silently never ran. The only symptom was a generic "ML API call failed" buried
in the status card.
So the test now proves what it claims. After health passes it probes GET /p/
with no img parameter: the auth decorator runs before the handler, so 401 means
the token was rejected and 422 ("Invalid request params") means it was
accepted. No inference work is done either way. A probe that itself errors
reports the token as unknown rather than as working — the UI says it could not
be checked instead of claiming success.
The detection loop checks for 401 before raise_for_status, so a rejected token
is reported as a rejected token, naming the setting and the environment
variable, instead of surfacing "401 Unauthorized" with no hint of what to do.
The message never contains the token; a test pins that.
The setting name carries "token", so the support bundle's keyword redactor
masks it with no new rule. Resolving "field omitted" to the saved token is the
route's job, keeping test_connection a pure outbound call with no database
access.
Second fix, same issue: support bundles misreported which printers Obico
watches. The bundle split obico_enabled_printers on commas and read an empty
value as "no printers". The settings UI writes a JSON array, and empty means
*all* printers — the default — so a working Obico setup showed obico_enabled
false against every printer in its own bundle. That is the reporter's bundle
exactly, and it points anyone reading it at the wrong subsystem. The bundle now
parses the setting the way ObicoDetectionService does, keeps a comma fallback
for any install that stored the legacy shape, and factors in the global switch.
The reporter's A1 mini has nine Flow Dynamics calibrations, all of them saved
under Generic PLA and named after the spool's colour — "Dark Brown", "Glow",
"Marble". Bambu Studio lists all nine for that slot. Configure AMS Slot offered
one: the profile already bound to the slot. After a slot reset it offered none,
leaving the slicer as the only way to assign a K value.
Two independent faults, both tripped by picking a built-in generic preset.
The filament-id match discarded Bambu's generic GFx99 ids as too broad. But the
comparison already requires both sides to carry the same id, so that exclusion
could only ever fire when the selected preset was itself the generic one —
precisely the case where the match is right. The printer keeps one calibration
table per filament id, so a slot on Generic PLA should offer everything
calibrated under Generic PLA. Equal ids now match, generic or not.
The name fallback was dead for the same presets: parsePresetName reads the
leading "Generic" in "Generic PLA" as a manufacturer, which put the matcher into
brand-gated mode and demanded the word GENERIC appear in the profile name. No
real profile has it. "Generic" is no longer treated as a brand, so profiles still
match on material when a printer reports no filament_id with its calibrations.
The one profile that did appear came from the #1689 safety net that always
surfaces the slot's active cali_idx — which is also why a reset slot, having no
active profile, showed an empty list.
Neither fix can be complete on its own, because profile names are free text and
nothing ties "Marble" to a material. The picker now also lists every remaining
profile on the printer under "Other K profiles on this printer", so a profile
that exists can always be selected. Applying one from that group needs no new
backend work: configure_ams_slot already realigns the slot's filament context to
the chosen profile's, which is what makes the cali_idx stick.
Options are keyed by name+k_value rather than the bare name, so two profiles
sharing a name are no longer indistinguishable in the select. Both render blocks
carry the change — the modal duplicates the picker for its full-screen variant.
isMatchingCalibration gets the same generic-id rule for the spool form's PA
suggester, with two guards. A new generic-id-to-material table means a PETG spool
can never claim GFL99 profiles just because both sides stored a generic id
(Nylon and PA compare as one material). And a spool that names its own brand
keeps the stricter name path, so its suggestions stay brand-specific rather than
becoming the printer's whole generic table.
The photo fired the moment layer_num reached total_layer_num. That edge is
where the printer *starts* its final layer, not where it finishes it: the
reporter's H2C capture shows it arriving at 92% with mc_remaining_time=2,
three minutes and seventeen seconds and one filament change before the print
actually ended, so the frame caught the toolhead mid-print over the model.
The trigger also latched _finish_photo_captured, which locked out both the
stage-22 and FINISH triggers for the rest of the print — so on firmware that
never reports an end-of-print filament unload (H2C and A1 Mini confirmed)
nothing could replace the bad frame.
Remove the last-layer trigger. The photo is now taken at the FINISH-state
trigger, which every model sends and which lands after the toolhead parks.
Since Bambu's end G-code drops the plate ~100mm just before that, restore the
framing before capturing: absolute G90/G1 Z to max_z_height + 10mm clearance,
settle, capture, then drop it back so the print is as reachable as the printer
left it. Absolute is the safety argument — that Z is a height the toolhead
occupied seconds earlier, so it is inside the travel limits by construction and
leaves the nozzle above the part, and it is unambiguous across model families
because Z is the nozzle-to-bed gap whether the bed moves or the toolhead does.
M211 is never touched (#2579). This is what #1145, #1397 and #1565 asked for.
The height is only trusted when two independent sources agree: the archive is
matched by the finished print's subtask_name by equality (not LIKE, so "Cube"
cannot resolve to "Cube v2"), and its layer count from the 3MF must match the
layer count the printer reported over MQTT. Matching on "most recent archive
for this printer" was not safe — on_print_complete pops the _active_prints
binding concurrently, and a print Bambuddy failed to archive would have
resolved to its predecessor. A wrong height is the one failure that could drive
the nozzle into the model.
The move is additionally skipped when the print height is unknown, when a queue
item is pending for the printer, when the printer has left FINISH, and when the
new finish_photo_restore_plate setting is off.
for every FINISH-state capture — which is what shipped the mid-print photo —
the bank is used only when the dispatcher recorded that it injected End G-code
into this print, since a SwapMod snippet may have ejected the plate. The flag is
handed over in two steps (mark_pending at dispatch, adopt at print start) so it
can never outlive its print: a job started from the slicer or SD card adopts
False rather than inheriting its predecessor's answer. Those prints also skip
the plate move outright, bank or no bank.
The bank now refreshes on mc_percent advances as well as layer changes, via a
new on_print_progress callback. Layer changes stop the instant the final layer
begins, which left the #1867 fallback frame stale by the whole length of that
layer; progress keeps ticking there and freezes before the End G-code runs, so
a swapped plate still cannot reach the bank. The last-layer throttle exemption
is dropped, since it would now fire a grab on every percent tick.
On the timelapse path the moment producer returns early, so the consumer does
the restore itself before its live-grab fallback — the documented usual outcome
on P1-series, where the video has not transferred by the time the notification
goes out and the shipped photo was of an already-dropped plate. The two waits
are now derived from the settle window and the video poll timeout rather than
hardcoded; at the old flat 75s that fallback was guaranteed to be cut off
mid-settle.
extract_max_z_height_from_3mf reads only a bounded prefix of the plate G-code,
since a sliced plate is routinely tens of megabytes and the header is ~40 lines.
It returns None for missing, unparseable, zero and negative values so callers
must treat "don't know" as such rather than defaulting.
Review follow-ups on applying camera_rotation to finish photos and
layer-timelapse frames.
Rotating the frame popped from _stage22_finish_frames rotated one of its
sources twice. The cache has two kinds of feeder: live grabs, which are raw,
and the #1867 in-print bank, whose bytes come from
_capture_snapshot_for_notification and have already been rotated on the way
in. The consumer cannot tell them apart, so on the finish_state trigger - the
path the bank exists to serve, on firmware that never emits stg_cur=22 - a 180
degree rotation cancelled itself out and the photo was upside-down again,
which is the reported symptom exactly; 90 and 270 landed 180 out. Rotation now
happens where each frame is captured, so every entry in the cache carries one
rotation whatever produced it, and the invariant is stated both where the
cache is declared and where it is consumed.
Two finish-photo sources were still writing unrotated files: the built-in
camera's own capture_finish_photo, and the still extracted from a
printer-recorded timelapse - which is the *preferred* source for a built-in
camera print, so a user with a rotation set got a correctly oriented photo or
not depending on which source happened to win. Neither ever holds the frame as
bytes; ffmpeg writes the file and they return a filename. apply_camera_rotation_to_file
handles that case and is best-effort - a failed rotate leaves the unrotated
file rather than losing a delivered photo. The archived video itself is the
printer's own file and is not re-encoded, so it still plays at the camera's
native orientation; the CHANGELOG says so rather than leaving it to be
discovered.
apply_camera_rotation logs at debug, not info. It was on a path that runs once
per layer, where a tall print would have put hundreds of lines in the log for
something the surrounding capture already reports at debug.
The moved rotation logic had no test of its own - every existing test patches
it out and asserts the call, so a flipped sign or a dropped expand=True would
have shipped green. test_camera_rotation.py drives the real round trip: a
corner marker pins which way it turns, the dimensions pin that the frame is
not cropped, and an undecodable frame comes back by identity because a capture
path must not lose a frame to a failed rotate.
Tests for the fix itself sit on both sides of the cache. The producer half is
driven directly; the consumer half is a closure nested inside on_print_complete
with nothing able to reach it, so it is pinned by an AST guard - checked
against the source because the alternative is no check at all. Reverting
main.py to the pre-fix shape fails three of the five, the guard among them.
The three new tests used Path("/tmp/test") for a patched base_dir, which Bandit
flagged (B108); they take tmp_path now.
Review follow-ups on the orphaned timelapse session cleanup.
The sweep's own docstring said min_age_seconds made it safe to call mid-run.
It did not. on_print_complete drops the session from _active_sessions before
handing frames_dir to ffmpeg, so for the length of a stitch the directory
matches no active session, and its mtime is the last layer's frame write -
which on a tall print's final layer is easily older than the margin. The
default margin is 300s and the stitch timeout is also 300s, so the two were
tied with no headroom at all: a sweep landing in that window deleted ffmpeg's
input from under it. _finalizing_sessions now covers the stitch, set as the
session leaves _active_sessions and cleared in a finally so a failed stitch
cannot leak the marker and make that printer's leftovers permanently
un-sweepable. The docstring names all three guards and which gap each covers,
including that the margin does have real headroom for the two cases it suits -
a session mid-creation, and the freshly written .mp4 awaiting attach.
The file branch now requires the timelapse_<session_id>.mp4 shape its own
comment describes. It previously deleted any file under
timelapse_frames/<printer_id>/ past the margin; nothing else writes there
today, but age alone is not a reason to delete a file this feature did not
create.
Dropped ignore_errors=True from the rmtree. It made the surrounding
except OSError unreachable, so a read-only mount or a permissions problem was
counted and logged as a successful removal - and that log is the only evidence
an operator has of what was deleted.
Tests 5 -> 9: sparing a session mid-stitch, the finalizing marker cleared even
when the stitch raises, unrelated files left alone, and a failed removal not
counted. The failure test's rmtree stub honours the real contract and returns
silently when ignore_errors=True, because that silent no-op is exactly what the
old call could never observe; a stub that raised unconditionally would have
passed against both versions and proved nothing.
main.py is unchanged: it has no module-level logger, and the inline
logging.getLogger(__name__) the sweep uses is the idiom throughout lifespan.
Review follow-ups on the external-camera capture coalescing.
The coalescing was transplanted from camera.py, which is keyed by printer IP
and so has nothing to hide in a log line. These keys carry the camera URL, and
an RTSP camera URL routinely embeds user:pass@ - so the five new log lines
printed the password, one of them at warning level, where it reaches support
bundles. All five now go through _log_key(), which redacts before truncating:
slicing first can cut the URL short of the @ the pattern anchors on and leave
the password intact, which is why every other URL log in the module already
does it in that order.
_capture_frame_uncoalesced gained the blanket catch its camera.py counterpart
has. That is load-bearing once captures are shared: the wrapper hands one
task's outcome to every caller waiting on it and can only give a follower its
own turn for an outcome it recognises, so an escaping exception reached all of
them at once and none retried - one caller's failure becoming N. The per-type
helpers catch narrowly (aiohttp.ClientError / OSError / timeouts), so the
guarantee belongs here rather than resting on their coverage. CancelledError
is re-raised ahead of it, since the wrapper distinguishes a cancelled leader
from a failed one.
test_connection reports whether it shared a capture. It reaches capture_frame
like any other consumer, so a test landing while Obico is polling got that
frame back and answered "connected" for a connection it never made - the one
answer a connection test must not give silently. It still shares rather than
forcing its own capture, because forcing one would open the second handle to a
single-reader device that this whole mechanism exists to prevent. The response
carries `coalesced`, which also gives capture_in_flight() the consumer its
camera.py counterpart has in the Diagnose tool, and the Test button says
"shared with a capture already running" instead of a bare success.
Tests 12 -> 20: an unexpected error reported as a failed capture, a raising
leader whose follower still gets a frame, the three coalesced states, and
redaction on each log line that can carry a URL. The raising-leader test
patches _capture_rtsp_frame rather than _capture_frame_uncoalesced, since a
stand-in installed in the latter's place sits above the catch and would test
the wrapper against a shape it can no longer be handed.
Same gap as the finish-photo fix: camera_rotation was only ever wired
into the notification-snapshot path, so a layer-timelapse video came
out upside-down whenever a rotation was configured - every frame
(fresh or reused from the live view's buffer) was written to disk raw.
Extracts the rotation logic out of main.py into a shared
apply_camera_rotation(image_data, rotation, logger) in services/camera.py
(taking the rotation value directly rather than a printer object, so
both main.py's printer-shaped callers and layer_timelapse's plain int
field can use it). main.py's _apply_camera_rotation becomes a thin
compatibility wrapper so its existing call sites are unchanged.
Threads a `rotation` field through TimelapseSession/start_session,
set from printer.camera_rotation in _maybe_start_layer_timelapse, and
applies it (via asyncio.to_thread, since PIL rotation is CPU-bound) in
capture_layer before each frame is written - after #2707's
live_frame_for_capture() resolves the frame, regardless of whether it
came fresh or from the live view's buffer.
Rebased onto #2707's landed implementation: capture_layer now calls
live_frame_for_capture() instead of the older is_stream_active/
try_get_active_buffered_frame pair this was originally written
against, so the layer-timelapse tests are updated to match, and the
now-redundant TestCaptureLayerAvoidsCompetingWithLiveViewer class
(superseded by #2707's own test_external_camera_live_frame_reuse.py)
is dropped.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_apply_camera_rotation was only ever wired into
_capture_snapshot_for_notification (print-start photo, in-print frame
bank) - the finish-photo pipeline (_background_finish_photo's three
branches: stage-22 pre-captured frame, external-camera fallback,
built-in buffered-frame fallback) saved frames straight to disk with
no rotation, so a finish photo came out upside-down whenever
camera_rotation was configured. Pre-existing gap, unrelated to
yesterday's collision fixes - just more visible now that a finish
photo reliably gets captured at all.
Built-in camera's final capture_finish_photo() fallback (writes its
own file, returns only a filename) is left unrotated - out of scope
here since Carl uses an external camera; would need its own change to
services/camera.py.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_active_sessions is in-memory only, so a process restart mid-print
loses track of any active layer-timelapse session without ever calling
cancel_session()/cleanup() - the frames directory (and, if stitching
had already produced output before the restart, a stray
timelapse_<session_id>.mp4) are then orphaned on disk permanently, with
no equivalent to the ffmpeg orphan janitor to reap them.
Confirmed live: 38MB of exactly this leftover on the OrangePi after
several restarts during this week's testing, including two corrupt
48-byte .mp4s from stitches that got interrupted mid-write.
Adds cleanup_orphaned_timelapse_sessions(), run once at startup: for
each printer_id under timelapse_frames/, remove any frame directory or
stitched-output file that doesn't match that printer's current active
session (if any) and is older than a defensive margin (5 min default).
A restart-recovered print never gets a new timelapse session either
(#1353's _maybe_start_layer_timelapse only fires on fresh PRINT_START
events), so nothing orphaned here can ever be resumed - safe to always
remove once it's old enough not to be a startup race.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
#2705 fixed simultaneous captures colliding on the built-in camera path,
keyed by printer IP through capture_camera_frame_bytes(). External
cameras reach the same collision through a different function -
external_camera.capture_frame() - that #2705 didn't touch, and a V4L2
USB device allows exactly one open handle just like Bambu's own RTSP
limit.
Nothing coalesced two one-shot capturers here either: Obico polling,
the in-print frame bank, the finish-photo moment, plate detection and
the notification snapshot could each open their own connection to the
same USB camera and collide - is_stream_active() only stops a
capturer from competing with an attached viewer, not with another
capturer (that's what #2707 fixed).
capture_frame() is now a single-flight coalescing wrapper (actual
dispatch moved to _capture_frame_uncoalesced), keyed by (url,
camera_type, snapshot_url) - snapshot_url is part of the key since
#1177's override routes to a completely different endpoint. Mirrors
#2705's shape: coalesces, doesn't cache (a call after the previous one
finishes always captures fresh); each caller keeps its own timeout via
wait_for(shield(...)) rather than inheriting the leader's; a follower
whose leader fails takes its own turn instead of inheriting a failure
it never had a chance to avoid, bounded at two rounds; cancellation is
disambiguated via leader.cancelled() so a follower's own cancellation
still propagates while a cancelled leader is treated as a failed one.
12 tests mirroring test_camera_capture_coalescing.py's structure.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
device.airduct is pushed field by field - the modeCur handler reads it with
an "in" check for that reason - so a frame can carry parts without carrying
every fan. Absence in that list is what tells us a kit is not fitted, and
taken from a truncated frame it made both accessory badges vanish mid-print
and started rejecting fan=aux2 on a printer that has the fan.
A parts list now counts as a full inventory only when it carries ids 1 (part
cooling) and 2 (aux). Neither is optional on a machine that reports an
airduct at all, and both appear in every layout in the support-package
archive - P2S base 1,2 / P2S+kit 1,2,3 / X2D 1,2,3,10 / H2C,H2D,H2S 1,2,3,6.
Anything narrower is a diff frame: its speeds are applied, presence is left
alone. Presence can still be added from a partial frame; only retraction
needs the full list, so a kit that really is removed still disappears.
Also compose showChamberFan from both model lists rather than branching
between them, so the P2S/X2D entries in MODELS_WITH_CHAMBER_FAN stay
reachable instead of reading as dead, and note in the fan-speed docstring
that the aux2 gate also rejects between connect and the first airduct push.
The filament list is positional from the modal down to the CLI's
filament_N.json parts, but for a source that already carries slice_info the
requirements endpoint returns only the slots the plate consumes. A
MakerWorld model declaring four filaments and painting with slot 4 alone
therefore showed one dropdown, whose PETG pick the CLI bound to slot 1 —
slot 4 sliced with the profile baked into the source, and the print came out
PLA.
The endpoint now takes full_slots, which widens that answer to every
project slot with used_in_plate flags, and only the slice modal passes it.
Print-time AMS matching shares the endpoint and keeps the used-only list, so
it still asks for exactly the spools the job needs.
setInterval does not await an async callback. Slicing a large project
blocks the backend for seconds, so poll ticks piled up behind one stalled
request, each holding a snapshot taken while the job was still active.
They resolved together, and every one of them ran the completion path —
one toast and two query invalidations each. A 20s stall against the 1.5s
interval produced 13 "Sliced X" toasts from a single slice.
Only one poll round is now in flight at a time, which also stops queueing
requests against a backend that is already saturated. Completion is
recorded once per job id, and a round still awaiting a response when the
effect tears down now returns instead of acting.
feat(db): warn when the connection pool can outgrow the PostgreSQL server
fix(mqtt): an unusable layer_num must not drop the printer connection
test: patch settings.base_dir via monkeypatch so it unwinds on error
test: restore the config module after reloading it
On a printer with an external camera, watching the live view while a print
ran meant the layer timelapse recorded almost nothing and the finish photo
went out with no image. The reporter measured 0 of 87 layer captures on one
print and 0 of 105 on another, both watched throughout. A USB camera allows
one V4L2 handle, so a capture during a live view fails outright.
The built-in camera has had this rule since #1348 and #1271: reuse the
viewer's buffered frame rather than opening a second connection. It was
never extended to the external paths, and it could not have been -- the
buffer it depends on was only ever populated by the built-in paths.
generate_mjpeg_stream yields multipart-wrapped chunks, so the route layer
could not recover the JPEG, and a guarded caller would have found an empty
buffer and skipped every time.
So the stream now publishes each raw frame through a new on_frame callback
(parallel to on_process from #2675), and the six one-shot consumers reuse
it: layer timelapse, the finish-photo moment and its background fallback,
the notification snapshot, Obico polling, and the plate check. A viewer
attached with nothing buffered yet skips that one attempt rather than
competing -- kicking the viewer off is worse than missing a frame.
on_frame exceptions are logged and swallowed, like iter_subscriber's
on_unsubscribe: buffering is a side effect and must never be able to take
the live stream down with it. The external stream's teardown now releases
the buffered frame too, ownership-checked so a concurrent viewer of the
same printer keeps its own.
Two side effects on paths not touched here, both improvements: the snapshot
endpoint and the finish-photo fallback chain consult get_buffered_frame and
can now serve an external camera's live frame. plate_detection's docstring
already claimed this behaviour while implementing it only for the built-in
fallback; that drift is resolved.
ffmpeg is spawned with stderr=PIPE and it was only read on the error
paths, so for the life of a working stream nobody read that pipe. ffmpeg
writes its banner, the input analysis, then a progress line at a steady
rate; a 64 KiB pipe fills eventually, ffmpeg blocks writing to it, frames
stop, and the stream's own 30s timeout fires -- logged as "RTSP read
timeout" with no hint that we starved it ourselves.
How long that takes is unmeasured and evidently long: one H2D upstream
ran 21m36s without stalling, and an earlier 512 B/s extrapolation of mine
was mostly the one-off startup banner. So this is a bounded resource being
treated as unbounded, not a fault anyone has reported.
_FfmpegStderrTail drains the pipe continuously and keeps a 16 KiB rolling
tail. That tail is what the error paths now report, which is better
material than before: it holds what ffmpeg said as things went wrong,
where the on-demand read returned whatever was printed first -- usually
the banner, which the summariser strips anyway.
Three readers wanted this one pipe, and asyncio rejects concurrent reads
on a StreamReader, so the collector is authoritative: it registers by pid,
_read_ffmpeg_stderr returns its tail when present and otherwise reads the
pipe unchanged, and _terminate_ffmpeg skips its own stderr drain when the
collector owns it (the collector keeps draining through teardown, which is
all wait() needs). The generator starts it after the immediate-failure
check, which reads the pipe directly because the process is already dead,
and releases it after _terminate_ffmpeg.
text() goes through _summarize_ffmpeg_stderr like every other stderr log
here, so the access code ffmpeg echoes in its input URL stays masked.
aclose() awaits the cancelled pump rather than firing and forgetting, so
no pending task survives into loop teardown.
Closing a camera view and reopening it immediately could leave the new
stream unregistered while it was running and delivering frames. The
damage was all indirect: is_stream_active() reported no viewer, so Obico
polling and snapshots opened a second camera connection against the live
view (the thing #1348 and #1271 exist to prevent); the janitor's /proc
scan found an ffmpeg missing from _active_streams and killed the live
stream as an orphan; and /camera/stop reported "Stopped 0" with a stream
running.
The fan-out stream id was f"{printer_id}-fanout" -- constant per printer,
so every successive stream shared one registry key, and the departing
generator's finally popped whatever was under it, including its
successor's entry. The same finally also cleared the per-printer frame
buffer unconditionally, discarding the new stream's frame. It needed the
two streams to overlap, which the 4s teardown made easy.
Each stream now gets its own key via _new_fanout_stream_id(), so a
generator can only clean up after itself -- the external-camera path
already does this (#2675) and this brings the fan-out path in line. The
per-printer dicts are released through _release_printer_frame_state(),
which checks that no other stream for the printer is still running; both
the RTSP and chamber-image cleanups had the same unconditional pop.
Also hoisted time and uuid to module level and dropped four
function-local `import time` statements. A local import shadows the name
for the whole function, so any use on a branch that doesn't reach the
import raises UnboundLocalError -- a real hazard in camera_stream, whose
external-camera branch imported both while the RTSP path needs them too.
A test pins camera_stream as free of function-local imports.
Closing a camera view logged "ffmpeg didn't terminate gracefully,
killing" followed by "ffmpeg did not exit within 2.0s of SIGKILL;
abandoning wait", on every single close. Both waits expired every time,
so teardown took a fixed 4.00s -- and since the firmware allows one
camera connection, that was 4s in which nothing else could use it.
ffmpeg is spawned with stdout and stderr as pipes and the teardown paths
have stopped reading them, so it sits blocked in write() on a full 64 KiB
pipe. SIGTERM cannot be acted on there: the handler only sets a flag that
the main loop polls, and the loop never gets back to the check. SIGKILL
does kill it, but asyncio resolves Process.wait()'s waiter through
_try_finish(), which requires every pipe transport to report
disconnected; paused, unread pipes never reach EOF, so wait() blocks with
returncode already set. A negative-control test shows returncode=-9 at
the instant the abandon fires.
Draining both pipes while stopping the process fixes both halves: 4.00s
becomes ~0.15s. The signal ladder and its bounds stay as backstops, so a
genuinely wedged process still cannot hang a stream, a Stop request or
the janitor.
This corrects _FFMPEG_KILL_TIMEOUT's premise and #2580's conclusion. That
12-hour hang was the unbounded form of this same self-inflicted stall, not
an ffmpeg stuck in uninterruptible I/O -- the process observed doing it
was in state S, which cannot survive a delivered SIGKILL. Bounding the
wait capped the symptom without removing the cause.
Bambu firmware allows exactly one camera connection. The existing guards
(is_stream_active / try_get_active_buffered_frame, #1271 and #1348) only
stop a one-shot capturer from competing with the fan-out broadcaster.
Nothing coordinated the capturers with each other, so with no viewer
attached every consumer correctly concluded it was not competing with a
viewer and then collided with the others. On the reporter's P2S an Obico
poll and a snapshot opened two RTSP sockets 207 ms apart, which knocked
over the fan-out stream feeding the camera wall; it was then reaped for
having received no frames for 58s.
capture_camera_frame_bytes() now coalesces: the first caller opens the
connection, callers arriving while it is in flight await the same result.
Eight paths reach that function independently - Obico polling, the
snapshot route, the finish-photo moment and its disk-writing sibling,
plate detection, the camera test and the diagnose tool - so the
single-flight sits at the bottom of the stack and no call site changes.
Keyed by IP, since that is what the firmware's limit applies to and the
function never sees a printer_id. The key excludes the timeout on
purpose: the call sites disagree about it, from 10s to 30s, so keying on
it would mean the Obico-vs-snapshot pair from the report never coalesced
at all.
It coalesces, it does not cache. A call arriving after the previous
capture finished still captures fresh, because plate detection and the
finish-photo path judge a running print from these frames and a stale one
there is worse than a slow one - #1397 was a finish photo taken seconds
late showing the bed already lowered.
Each caller waits on its own deadline rather than inheriting whichever
one happened to open the connection, and shield() means giving up leaves
the capture running for whoever else is still waiting. A follower whose
leader fails takes a turn of its own instead of inheriting a failure it
never had a chance to avoid; the leader has finished by then, so there is
nothing left to compete with. Bounded at two rounds. That also covers the
follower whose timeout is longer than the leader's, which coalescing
alone cannot. Cancellation is disambiguated via leader.cancelled(), so a
follower's own cancellation propagates while a cancelled leader is
treated as a failed one.
The leader is deliberately not wrapped in a second wait_for: the
implementation already enforces the timeout internally, where it can also
kill the ffmpeg process, and an outer deadline would abandon the
subprocess instead of killing it.
The diagnose tool now marks a stage whose frame came from a capture
already in flight as coalesced_capture. The pass is real evidence the
camera works, but duration_ms is then mostly time spent queueing, and a
diagnostic must not report a connection it never opened - the same reason
that file declares its live_stream_active shortcut instead of quietly
passing. Failures are not annotated, since a follower whose leader fails
goes on to capture on its own.
The Filament Mapping panel reported "(Ready)" with a green tick for a slot
where the slice wanted dark red and the auto-matched tray held dark green.
Manually picking that same tray reported the mismatch correctly, which is
what made it obvious something was inconsistent.
Auto-match ranks candidates by tray_info_idx first, and a uniquely-matching
preset was accepted as definitive on the premise "same preset = same spool =
same colour". The preset names the variant, not the spool: GFA00 is PLA
Basic, GFA01 PLA Matte, GFA17 PLA Translucent, in every colour Bambu sells.
The reporter's own bundle has eight GFA00 trays in eight colours. With one
Matte spool loaded, every Matte requirement idx-matched it and the colour
comparison was never reached - which is why this surfaced on PLA Matte and
not on Basic, where several spools are usually loaded and the match falls
through to the branch that does compare colours.
The verdict now comes from the tray that was selected rather than from which
rule selected it, and both branches share one comparison so they cannot
drift apart again. Selection is unchanged - the right variant still wins per
mismatch and the slot stays selected.
A requirement with no colour at all is treated as satisfied rather than
mismatched; 3MFs that omit it parse to "" and there is nothing to disagree
with. That also affects the manual branch, which used to flag it.
No dispatch change: _get_missing_force_color_slots already required an exact
colour, so force colour match was gated correctly throughout.
Follow-up to #2680: the calendar toggle only put dates on the file pane, so
the folder tree had no way to show the timestamp it was already sorting on.
FolderTreeItem now takes showModified and renders latest_activity_at under
the folder name, threaded through the recursive call so nested folders get it
too. No backend change - the field was already on the wire from the sort fix.
Folders are labelled "last activity", not "last modified", and get their own
i18n key. The value is the newest timestamp among the folder, its files and
everything below it, so a folder can read as newer than its own directory
mtime - calling that "modified" would look like a fresh instance of the
ls -lt mismatch the issue was originally about. Folders with no activity
render nothing rather than an Invalid Date placeholder.
The name span moved into a flex column so the second line does not disturb
the row's link badge, file count or kebab menu. That broke a folder-delete
test that reached the row via parentElement, now fixed to use closest().
Separately, the #996 collapse describe left an implementation on the
module-global localStorage.getItem mock, which silently collapsed the folder
tree for every describe after it. It resets in afterEach now; without that,
any later test asserting on nested folders fails for reasons unrelated to
what it is testing.
mqtt_relay reads state.left_aux_fan_speed, but the SimpleNamespace fixture in
test_plate_clear_mqtt_notification enumerates its fields explicitly, so the two
status-payload tests raised AttributeError. I updated the equivalent fixture in
test_printer_manager_status_broadcast and missed this one — running only the
touched suites is what hid it.
Also addresses the round-2 review notes:
- exhaust_fan_present: documented that the H2 series reports part 3 too, so the
flag is not model-specific despite the name.
- Mask the part id after shifting, matching get_flag_bits(id, 4, 8), for
consistency with the state decode. No behaviour change for any observed id.
- Noted the unmapped H2 id 6 beside the id branches.
- Reject fan=aux2 when the printer reports no left_aux_fan_speed, so a POST
against an A1 no longer sends M106 P10 for absent hardware. The UI already
hid the badge; this closes the same hole on the API.
- Added a test asserting EXHAUST_FAN_LABEL_MODELS and the frontend's
MODELS_WITH_EXHAUST_LABEL cannot drift apart.
The hook is mounted globally in WebSocketProvider, so refetchInterval on its
per-printer status queries added one request per printer every 30s on every
page. The Printers page already runs that fallback on the same query key, and
useWebSocket writes ['printerStatus', id] straight into the cache, so the poll
bought nothing outside the Printers page and cost a request per printer per
tab everywhere else.
Also captures document.title at mount instead of restoring to a hardcoded
'Bambuddy', so the default no longer has to be kept in sync with index.html.
jsdom has no canvas backend, so getContext('2d') logged a "Not implemented"
jsdomError with a full React stack on every run of the hook's tests, and the
favicon branch bailed on the null context and went untested. Stubbing
getContext/toDataURL removes the noise and lets the ring code run, so the
favicon swap and the restore-on-toggle-off path are now asserted.
Telegram groups with Topics enabled always received notifications in the
General topic, since only Bot Token and Chat ID were configurable. Splitting
notifications per printer meant running a separate chat for each one.
The Telegram provider now takes an optional Forum Topic ID - the last number
in a topic's link, t.me/c/1234567890/25 - and routes its messages there. Left
empty, nothing changes.
The value is coerced to an int once in _send_telegram and attached to both the
sendMessage JSON body and the sendPhoto form data. That ordering matters:
Telegram rejects a string message_thread_id in the JSON body while accepting
one in the multipart call, so passing the raw form value through would have
worked for thumbnail notifications and 400'd for plain-text ones. A
non-numeric value is rejected in the form and again server-side before any
request goes out.
No migration - provider config is a JSON blob.
Adds Forum Topic ID plus help text to the Telegram section of the provider
dialog, translated in all 13 locales. Backend tests cover omitted / blank /
int-typed / non-numeric values and both send paths; frontend tests cover the
field being optional, absent for other providers, round-tripping on save, and
blocking save on a bad value.
Optional and off by default, toggled under Settings -> Appearance. When enabled, the browser tab shows the soonest-finishing print's percentage plus a green progress-ring favicon, updated live over the existing WebSocket. The preference is stored per-browser in localStorage.
Adds the usePrintProgressTitle hook (with tests), a ThemeContext preference, the Settings toggle, i18n strings for all locales, and a README entry.
The uk locale was inserted before ru in i18n/index.ts - in the import block,
the resources map and availableLanguages. Locales are appended to those lists
as they land (tr, then ru on 2026-07-19, then uk on 2026-07-28), and
SettingsPage renders availableLanguages in array order, so the picker showed
Ukrainian above Russian while every other entry followed the order it was
added.
Moves uk to the end of all three lists. SUPPORTED_LNGS is sorted
alphabetically and already had uk in the correct position, so it is unchanged.
Frontend-only, no behaviour change beyond the picker's row order.
Subprocess output and user-supplied URLs are scrubbed of credentials
before they reach the application log. Adds a shared redaction helper in
core/logging_filters and routes the existing support-bundle sanitizer
through the same pattern.
maziggy