Decide migration idempotency by SQLSTATE, not by English error text (issue #2949)

PostgreSQL renders its messages in the server's lc_messages locale. _safe_execute
    recognised an already-applied statement by searching the error text for
    "already exists", so a Russian-locale server -- which says "уже существует" --
    re-raised it and aborted startup.

    The column already existing is the expected outcome: create_all() builds the
    tables from the models before the migration list runs, so on a fresh database
    essentially every ADD COLUMN in that list is a duplicate by design, and all 382
    of them relied on that recognition. No PostgreSQL server outside an English
    locale could start Bambuddy at all, fresh install or upgrade.

    Classify on SQLSTATE instead -- 42701, 42P07, 42710, 23505 -- which PostgreSQL
    never translates. The existing narrowing is kept and now rests on a code rather
    than a phrase: a missing column counts as already-applied only for RENAME COLUMN,
    so a missing column during ADD COLUMN or CREATE INDEX still aborts rather than
    hiding a corrupt schema. SQLite keeps the text match; its driver publishes no
    SQLSTATE and it does not localise. The OIDC auto-link constraint read message
    text the same way and gets the same treatment.

    Verified against PostgreSQL 15 under ru_RU, en_US and C: init_db() completes on
    a fresh database and on a re-run in all three, and the schema the Russian server
    ends up with is byte-identical to the English one.
This commit is contained in:
maziggy
2026-08-26 10:17:03 +02:00
3 changed files with 218 additions and 27 deletions
+1
View File
@@ -21,6 +21,7 @@ All notable changes to Bambuddy will be documented in this file.
- **Generated thumbnails are lit, so one model no longer looks like the next (#2816, requested by @NaegeliJ, contributed by @sadontsev in #2861)** — Both renderers that draw a model themselves — the File Manager's STL/3MF thumbnails and the plate cards — handed matplotlib a mesh with no light source, and without one every triangle is filled with the identical green whichever way it faces. The result was a flat silhouette, so two models of similar outline were the same picture. The mesh is now shaded from a fixed light whose angle is pinned to the camera angle rather than chosen freely: the two are a pair, and a light aimed at the far side of the model gives both visible faces the same brightness and no contrast at all. Lighting then exposed two faults a flat render had hidden. A face wound the wrong way shades as though it faced away, so an STL with inconsistent winding came out patchy like camouflage; winding is now repaired before the render — outward rather than merely consistent, and only for the meshes that need it, since the check is milliseconds where the repair is seconds. And a mesh whose facets are all zero-area or collinear — stub or truncated STLs, 3MFs with an empty triangle list — would have failed outright once lit, so those are detected and still render flat instead of counting as a failure in a folder-wide batch. Both renderers share one copy of the light, the camera and the repair, so a plate card and a library thumbnail of the same model cannot drift apart. Passing the faces as an array rather than a list of lists also cut the collection build on an 82k-face mesh from ~0.19s to ~0.007s, which speeds up the unlit path too.
### Fixed
- **Bambuddy could not start against a PostgreSQL server whose messages are not in English (#2949, reported and diagnosed by @dvb6666)** — @dvb6666's Debian PostgreSQL 15 answered the first migration statement with `столбец "parent_run_id" отношения "pipeline_runs" уже существует` and startup aborted. The column already existing is the *expected* outcome there: `create_all()` builds the tables from the models before the migration list runs, so on a fresh database essentially every `ADD COLUMN` in that list is a duplicate by design, and all 382 of them relied on the runner recognising "already applied" and moving on. It recognised it by searching the error text for `already exists` — but PostgreSQL renders its messages in the server's own `lc_messages` locale, and a Russian server does not say that. So the very first statement was re-raised as a fatal error. This was never specific to `parent_run_id` or to one PostgreSQL version, which is why @dvb6666 got the identical failure after moving from 15 to 18: no PostgreSQL server outside an English locale could start Bambuddy at all, fresh install or upgrade, and the failure surfaced as a schema error rather than as anything pointing at the language. Idempotency is now decided by SQLSTATE, the five-character code PostgreSQL never translates — `42701` duplicate column, `42P07` duplicate table or index, `42710` duplicate constraint or trigger, `23505` duplicate key. The narrowing that was already there is kept and now rests on a code rather than a phrase: a missing column counts as "already applied" only for `RENAME COLUMN`, so a column missing during `ADD COLUMN` or `CREATE INDEX` still aborts startup instead of quietly hiding a corrupt schema — as do a missing table, an unknown column type, and a syntax error, each verified against a live server. SQLite keeps the text match, since its driver publishes no SQLSTATE and it never localises its messages. The same treatment went to the second place that read message text, the OIDC auto-link safety constraint, where a mistranslated "already exists" would have failed startup on the same servers. Measured rather than reasoned about: against PostgreSQL 15 under `ru_RU`, `en_US` and `C`, a full `init_db()` completes on both a fresh database and a re-run, and the schema the Russian server ends up with is byte-identical to the English one — nothing is being silently skipped to make startup succeed.
- **A test in the plug-energy suite failed for 31 minutes of every day (#2938, reported and measured by @ojimpo)** — `test_nothing_derivable_before_the_first_midnight` passed for 23.5 hours a day and failed for the other half hour, which is the shape that reads as ordinary flakiness and gets re-run rather than fixed. @ojimpo hit it running the full suite for an unrelated PR at 22:10 UTC, stashed his branch to confirm it reproduced on clean `dev`, then measured the window minute by minute against the real day boundary instead of guessing at it. **The test's premise, not the product.** It asserts that nothing can be derived when a plug's only snapshot was taken *after* this local midnight, and it placed that snapshot at a raw wall-clock offset — `now - 30 minutes`. The comment beside it, "taken this morning, after midnight", is the premise, and it is only true away from the boundary: for the first half hour of each local day, `now - 30 minutes` lands *before* local midnight, where it is a perfectly good baseline. `_counter_at` finds it, and `today` comes back `103.5 - 102.0 = 1.5` instead of `None`. The window is local 00:00–00:30, which is 22:00–22:30 UTC while CEST is in effect and 23:00–23:30 under CET — it moves with DST, because the module pins `Europe/Berlin` in an autouse fixture and an outer `TZ` makes no difference to it. **`derive_today_yesterday` is right and unchanged.** A snapshot from before local midnight genuinely *is* a valid baseline for today, and the production code treating it as one is the documented behaviour, so nothing here reaches a running install. The snapshot is now anchored to `local_day_start(now) + 30 minutes`, which is the idiom the file's other nine snapshot writes already use and the reason none of *them* could drift — this was the only one that offset from the wall clock, and the only one that failed. Verified by replaying 5760 minutes across four days including both DST switch days: the old expression fails 31 minutes per day, the new one fails none. **Scope.** One test file. No product code, no DB migration, no schema change, no permission, no i18n change.
- **The slice dialog took settings from the file with "Use the file's built-in settings" switched off (#2942, reported by @zevulos)** — Two separate features read as one. The checkbox slices a 3MF the way its designer set it up, ignoring the picked profiles; the per-option "from file" ticks beside each setting carry the designer's individual deviations onto the profile you picked (#2622), and those arrived pre-ticked whatever the checkbox said. So a slice run deliberately *without* the file's settings still took sixteen values out of it — the reporter's own log names them, `enable_support` and `support_type` among them, landing on a process preset they had chosen on purpose. The ticks now follow the checkbox: off, nothing comes out of the file until it is asked for by name; on, every setting the file changed shows ticked, because on that path the file really does drive the whole slice. Taking the designer's work in bulk is still one click, from a line at the top of the settings panel that says how many settings the file changed — and it still leaves the machine-tuned ones and the two that *are* the picked preset for a per-key decision, which is the classification #2622 made and this does not widen. Two things underneath had to change for the checkbox to mean what it says. The panel greys out options the slicer's own rules switch off, and it was evaluating those rules against what the user had typed alone, falling back to the compiled-in schema defaults for the rest — so a preset with supports on read as `enable_support: false` and greyed out the whole Support page while the slice ran supports. A greyed row greyed its tick too, which is how the reporter's screenshot shows a support type marked "from file", applied to the slice, and impossible to clear. The rules now see what the slice will actually run with: the preset's values, the file's values for the keys that are on, and anything typed on top. And the tick is no longer gated on the slicer's rules at all, because it answers a different question — not whether an option is in play, but where its value comes from. Separately, the support carry-over from #1881 ran underneath the ticks entirely, lifting four support keys out of any 3MF that had supports on, with nothing on screen to decline. It now stands down for the keys that were offered and turned down, which the request can say for the first time: an empty `design_overrides` list means the caller was shown the file's settings and took none, where no list at all is a caller that predates the choice and keeps #1881 whole — as do sources that record no deviations to tick, an OrcaSlicer export among them. Covered by backend and frontend tests, and measured against the reporter's own sixteen keys. Note the practical change: a MakerWorld file with supports enabled no longer switches supports on for you — tick **Enable support** in the panel, or the checkbox above it, if that is what you want.
- **A colour mismatch was reported between two filaments the app itself called "Blue" (#2941)** — The print dialog compared a slicer profile's near-pure `#0028FF` against the Bambu navy `#0A2989` loaded in A4 and correctly said they differ: 118 apart in the blue channel alone, a CIEDE2000 distance of 15 where 1 is a just-noticeable difference. Nothing in the dialog said so. A hex that misses the colour catalogue is named by a coarse family bucket, so both sides resolved to the name "Blue", and the warning sat between two identical labels with no way to tell what it was objecting to — the reporter read it, reasonably, as the matcher being broken. Where both sides of a mismatch carry the same name they are now qualified by their hex, and the tooltip names them together: "Same type, different color: needs Blue (#0028FF), slot has Blue (#0A2989)". Names that already differ are left alone, since the hex is noise once the words separate them. The comparison itself is untouched — it was right, and its tolerance is not the kind of thing to widen on a single report: admitting a difference that large would start matching navy to cyan, and the eligibility rule is shared with the queue scheduler, which would then dispatch on it. The panel's own status line, the required-filament tooltip, the auto-matched/manual marker, the slot placeholder and the type-not-found message were all hardcoded English; they are translated now, in all thirteen locales, and two of them turned out to have had translations sitting unused in every locale file the whole time.
+68 -27
View File
@@ -594,16 +594,64 @@ async def _migrate_encrypt_legacy_secrets() -> None:
)
# PostgreSQL SQLSTATE codes meaning "this DDL statement has already been applied".
# We classify on these rather than on the error text because the server renders
# messages in its own ``lc_messages`` locale: a Russian-locale server answers a
# duplicate ADD COLUMN with "уже существует", which no English substring check can
# recognise. That made Bambuddy unstartable on every non-English PostgreSQL server,
# fresh or existing — create_all() runs before run_migrations(), so on a new database
# essentially every ADD COLUMN below is expected to come back as a duplicate (#2949).
_PG_ALREADY_APPLIED = frozenset(
{
"42701", # duplicate_column — ALTER TABLE ADD COLUMN
"42P07", # duplicate_table — CREATE TABLE, CREATE INDEX
"42710", # duplicate_object — ADD CONSTRAINT, CREATE TRIGGER
"23505", # unique_violation — duplicate key
}
)
# undefined_column. Idempotency only for RENAME COLUMN (the rename already ran);
# on any other statement a missing column means a broken schema, not a re-run.
_PG_UNDEFINED_COLUMN = "42703"
def _sqlstate(exc) -> str | None:
"""Return the PostgreSQL SQLSTATE behind a SQLAlchemy error, or None.
None on SQLite, whose DBAPI exceptions carry no such code — and which never
localises its messages, so the text match below stays correct there.
"""
orig = getattr(exc, "orig", None)
for attr in ("sqlstate", "pgcode"):
code = getattr(orig, attr, None)
if code:
return str(code)
return None
def _is_already_applied(exc, sql: str) -> bool:
"""Return True if a failed DDL statement had simply already been applied."""
is_rename = "rename column" in sql.lower()
state = _sqlstate(exc)
if state is not None:
return state in _PG_ALREADY_APPLIED or (state == _PG_UNDEFINED_COLUMN and is_rename)
msg = str(exc).lower()
if any(k in msg for k in ("already exists", "duplicate key", "duplicate column name", "no such column")):
return True
return is_rename and "column" in msg and "does not exist" in msg
async def _safe_execute(conn, sql):
"""Execute a DDL migration statement, silently ignoring idempotency errors.
'already exists', 'duplicate column name' (SQLite ADD COLUMN), 'no such column'
(SQLite RENAME COLUMN), 'duplicate key', and the compound
'column … does not exist' (PostgreSQL RENAME COLUMN idempotency) are swallowed
so that re-running DDL migrations is safe. The compound check additionally
requires the SQL to be a RENAME COLUMN statement so that "does not exist" errors
from ADD COLUMN or CREATE INDEX (which would indicate schema corruption, not
idempotency) are never silently swallowed.
Statements that had already been applied are swallowed so that re-running DDL
migrations is safe — see :func:`_is_already_applied` for how that is decided
(SQLSTATE on PostgreSQL, message text on SQLite). Idempotency for a missing
column is narrowed to RENAME COLUMN, so a missing column on ADD COLUMN or
CREATE INDEX — which would indicate schema corruption, not a re-run — is never
silently swallowed.
Any other error is logged and re-raised — callers must not assume silent
recovery, as a failure will abort the migration sequence and prevent
application startup.
@@ -621,14 +669,7 @@ async def _safe_execute(conn, sql):
async with conn.begin_nested():
await conn.execute(text(sql))
except (OperationalError, ProgrammingError) as exc:
msg = str(exc).lower()
# Only swallow "column … does not exist" for RENAME COLUMN — not for ADD COLUMN
# or CREATE INDEX where it would indicate schema corruption, not idempotency.
column_not_exists = "rename column" in sql.lower() and "column" in msg and "does not exist" in msg
if (
not any(k in msg for k in ("already exists", "duplicate key", "duplicate column name", "no such column"))
and not column_not_exists
):
if not _is_already_applied(exc, sql):
logger.error("Migration statement failed: %s | SQL: %.200s", exc, sql)
raise
@@ -1303,8 +1344,8 @@ async def run_migrations(conn):
# Migration: Add source_archive_id column to pipeline_runs (#1425 PR B follow-up).
# Allows a pipeline run to source from an archive's source 3MF in addition
# to a library file. Idempotent — _safe_execute swallows the "already exists"
# case on both SQLite and Postgres.
# to a library file. Idempotent — _safe_execute swallows an already-applied
# statement on both SQLite and Postgres.
await _safe_execute(
conn,
"ALTER TABLE pipeline_runs ADD COLUMN source_archive_id INTEGER REFERENCES print_archives(id) ON DELETE SET NULL",
@@ -2065,7 +2106,7 @@ async def run_migrations(conn):
# timestamp; the type differs by dialect (SQLite DATETIME vs Postgres
# TIMESTAMP) so an existing-DB upgrade doesn't hit "type datetime does not
# exist" on Postgres. On a fresh DB create_all() already built the column, so
# the ALTER is swallowed as "already exists".
# the ALTER is swallowed as already applied.
#
# Placed AFTER the print_queue_new2 table-recreate above: that recreate
# (SQLite-only, and only on ancient DBs whose archive_id is still NOT NULL)
@@ -3259,17 +3300,17 @@ async def run_migrations(conn):
# SQLite does not support ALTER TABLE ADD CONSTRAINT — handled by __table_args__ at creation.
# Runs AFTER the backfill so Fall B rows don't fail constraint validation.
if not is_sqlite():
add_constraint = (
"ALTER TABLE oidc_providers ADD CONSTRAINT ck_auto_link_requires_verified_email_claim "
"CHECK (auto_link_existing_accounts = FALSE OR email_claim != 'email' OR require_email_verified = TRUE)"
)
try:
async with conn.begin_nested():
await conn.execute(
text(
"ALTER TABLE oidc_providers ADD CONSTRAINT ck_auto_link_requires_verified_email_claim "
"CHECK (auto_link_existing_accounts = FALSE OR email_claim != 'email' OR require_email_verified = TRUE)"
)
)
await conn.execute(text(add_constraint))
except (OperationalError, ProgrammingError) as exc:
msg = str(exc).lower()
if "already exists" not in msg:
# Classified by SQLSTATE, not by message text: a non-English server
# reports the constraint as already present in its own language (#2949).
if not _is_already_applied(exc, add_constraint):
logger.error(
"Security constraint migration FAILED — auto_link safety constraint may not be enforced: %s",
exc,
@@ -4336,7 +4377,7 @@ async def run_migrations(conn):
# ordering was arbitrary. Nullable; the timestamp type differs by dialect
# (SQLite DATETIME vs Postgres TIMESTAMP) so an existing-DB upgrade doesn't hit
# "type datetime does not exist" on Postgres. On a fresh DB create_all() already
# built the column, so the ALTER is swallowed as "already exists".
# built the column, so the ALTER is swallowed as already applied.
if is_sqlite():
await _safe_execute(conn, "ALTER TABLE library_files ADD COLUMN fs_modified_at DATETIME")
await _safe_execute(conn, "ALTER TABLE library_folders ADD COLUMN fs_modified_at DATETIME")
@@ -0,0 +1,149 @@
"""Migration idempotency must not depend on the server's message language (#2949).
``_safe_execute`` used to decide whether a failed DDL statement had simply
already been applied by looking for ``"already exists"`` in the error text.
PostgreSQL renders its messages in the server's ``lc_messages`` locale, so a
Russian-locale server answered a duplicate ``ADD COLUMN`` with
``столбец … уже существует`` — no English substring, so the statement was
re-raised and startup aborted.
That was not a corner case: ``create_all()`` runs before ``run_migrations()``,
so on a fresh database essentially every ``ADD COLUMN`` in the migration list is
*expected* to come back as a duplicate. The reporter's install died on the very
first one, and no PostgreSQL server outside an English locale could start at all.
The classifier now reads the SQLSTATE, which PostgreSQL never translates. These
tests pin both halves: the codes it accepts, the codes it must still let through,
and the SQLite fallback for a DBAPI that has no SQLSTATE to offer.
"""
from __future__ import annotations
import pytest
from sqlalchemy import text
from sqlalchemy.exc import OperationalError, ProgrammingError
from backend.app.core.database import _is_already_applied, _safe_execute, _sqlstate
# Verbatim from a PostgreSQL 15 server running lc_messages=ru_RU.utf8 — the exact
# text the reporter pasted into the issue. Nothing in the classifier may read it.
RU_DUPLICATE_COLUMN = 'столбец "parent_run_id" отношения "pipeline_runs" уже существует'
RU_DUPLICATE_TABLE = 'отношение "t_dup" уже существует'
RU_DUPLICATE_OBJECT = 'ограничение-проверка "ck_a" для отношения "t_ck" уже существует'
RU_UNDEFINED_COLUMN = 'столбец "nope" не существует'
RU_UNDEFINED_TABLE = 'отношение "no_such_table" не существует'
ADD_COLUMN = (
"ALTER TABLE pipeline_runs ADD COLUMN parent_run_id INTEGER REFERENCES pipeline_runs(id) ON DELETE SET NULL"
)
RENAME_COLUMN = "ALTER TABLE t_rn RENAME COLUMN nope TO other"
CREATE_INDEX = "CREATE INDEX ix_t ON t (nope)"
class _FakeOrig(Exception):
"""Stand-in for asyncpg's DBAPI wrapper, which exposes ``sqlstate``."""
def __init__(self, sqlstate: str, message: str):
super().__init__(message)
self.sqlstate = sqlstate
def _pg_error(sqlstate: str, message: str, sql: str) -> ProgrammingError:
return ProgrammingError(sql, {}, _FakeOrig(sqlstate, message))
class TestSqlstateIsPreferredOverTheMessage:
@pytest.mark.parametrize(
"sqlstate,message,sql",
[
("42701", RU_DUPLICATE_COLUMN, ADD_COLUMN), # duplicate_column — the reported failure
("42P07", RU_DUPLICATE_TABLE, "CREATE TABLE t_dup (id INTEGER)"), # duplicate_table
("42P07", RU_DUPLICATE_TABLE, "CREATE INDEX ix_t_idx ON t_idx (a)"), # duplicate index
("42710", RU_DUPLICATE_OBJECT, "ALTER TABLE t_ck ADD CONSTRAINT ck_a CHECK (a > 0)"), # duplicate_object
],
)
def test_a_russian_language_duplicate_is_recognised(self, sqlstate, message, sql):
assert _is_already_applied(_pg_error(sqlstate, message, sql), sql) is True
def test_undefined_column_is_idempotency_only_for_rename(self):
"""The rename already ran. On any other statement a missing column means
the schema is broken, and swallowing it would hide the corruption."""
rename = _pg_error("42703", RU_UNDEFINED_COLUMN, RENAME_COLUMN)
assert _is_already_applied(rename, RENAME_COLUMN) is True
index = _pg_error("42703", RU_UNDEFINED_COLUMN, CREATE_INDEX)
assert _is_already_applied(index, CREATE_INDEX) is False
@pytest.mark.parametrize(
"sqlstate,message",
[
("42P01", RU_UNDEFINED_TABLE), # undefined_table — migrating a table that isn't there
("42704", 'тип "notatype" не существует'), # undefined_object — a typo'd column type
("42601", "ошибка синтаксиса"), # syntax_error
],
)
def test_real_failures_still_abort_startup(self, sqlstate, message):
exc = _pg_error(sqlstate, message, ADD_COLUMN)
assert _is_already_applied(exc, ADD_COLUMN) is False
def test_an_english_message_saying_already_exists_cannot_rescue_a_fatal_sqlstate(self):
"""Once a SQLSTATE is present it is the whole answer. A server whose text
happens to contain the old keyword must not talk us out of a real error.
"""
exc = _pg_error("42P01", 'relation "printers" does not exist, table already exists', ADD_COLUMN)
assert _is_already_applied(exc, ADD_COLUMN) is False
class TestSqlstateExtraction:
def test_reads_sqlstate_from_the_dbapi_error(self):
assert _sqlstate(_pg_error("42701", RU_DUPLICATE_COLUMN, ADD_COLUMN)) == "42701"
def test_falls_back_to_pgcode(self):
"""psycopg spells the same value ``pgcode``."""
class _Psycopg(Exception):
pgcode = "42P07"
exc = ProgrammingError("CREATE TABLE t (a INTEGER)", {}, _Psycopg())
assert _sqlstate(exc) == "42P07"
def test_none_when_the_driver_offers_no_code(self):
"""SQLite. Callers fall back to matching the message, which SQLite —
unlike PostgreSQL — never translates."""
exc = OperationalError("ALTER TABLE t ADD COLUMN a INTEGER", {}, Exception("duplicate column name: a"))
assert _sqlstate(exc) is None
class TestSqliteFallback:
"""No SQLSTATE, so the message keywords still decide — unchanged behaviour."""
@pytest.mark.parametrize(
"message,sql,expected",
[
("duplicate column name: a", ADD_COLUMN, True),
("table t already exists", "CREATE TABLE t (a INTEGER)", True),
("no such column: nope", RENAME_COLUMN, True),
("no such table: printers", ADD_COLUMN, False),
('near "GARBAGE": syntax error', ADD_COLUMN, False),
],
)
def test_message_keywords(self, message, sql, expected):
exc = OperationalError(sql, {}, Exception(message))
assert _is_already_applied(exc, sql) is expected
@pytest.mark.asyncio
class TestSafeExecuteAgainstRealSqlite:
async def test_a_duplicate_add_column_is_swallowed(self, db_session):
conn = await db_session.connection()
await conn.execute(text("CREATE TABLE t_2949 (id INTEGER PRIMARY KEY)"))
await _safe_execute(conn, "ALTER TABLE t_2949 ADD COLUMN extra INTEGER")
await _safe_execute(conn, "ALTER TABLE t_2949 ADD COLUMN extra INTEGER")
cols = {row[1] for row in await conn.execute(text("PRAGMA table_info(t_2949)"))}
assert cols == {"id", "extra"}
async def test_a_genuine_failure_is_re_raised(self, db_session):
conn = await db_session.connection()
with pytest.raises(OperationalError):
await _safe_execute(conn, "ALTER TABLE table_that_does_not_exist ADD COLUMN x INTEGER")