refactor(models): break the schema cycle that backup and restore sort through

print_archives.library_file_id -> library_files.folder_id ->
    library_folders.archive_id -> print_archives. Three nullable SET NULL
    links, each reasonable alone, that together made a loop
    metadata.sorted_tables could not sort: it dropped those edges, warned on
    every backup and every restore, and could return an order placing a
    child before its parent -- which once imported library_files ahead of
    library_folders and killed a restore on a ForeignKeyViolation.

    The restore no longer depends on that order (it strips every foreign key
    before importing and adds them back after), but the backup export sorts
    the same way, and the warning ends with "may raise an error in a future
    release" -- which would break backup and restore on one upgrade.

    Marking one edge use_alter removes it from the sort graph, not from the
    database: PostgreSQL emits it as ALTER TABLE ADD CONSTRAINT, as it
    already did for every constraint on these three tables, and SQLite
    inlines it into CREATE TABLE, so ON DELETE SET NULL holds on both.
    Verified against PostgreSQL 16 and SQLite.
This commit is contained in:
maziggy
2026-09-24 14:51:59 +02:00
10 changed files with 794 additions and 13 deletions
+2
View File
@@ -5,10 +5,12 @@ All notable changes to Bambuddy will be documented in this file.
## [1.2.5.6] - 2026-09-21
### Changed
- **The schema no longer contains a cycle that `metadata.sorted_tables` cannot sort** — `print_archives.library_file_id` → `library_files.folder_id` → `library_folders.archive_id` → back to `print_archives`: three nullable `SET NULL` links, each reasonable alone, that together made a loop SQLAlchemy answered with a sort it could not make and a warning on every backup and every restore. The order it returned could place a child before its parent, which is what once imported `library_files` ahead of `library_folders` and killed a restore on a foreign key violation — the restore has since stopped depending on that order by stripping every foreign key before importing and adding them back afterwards, but the backup export sorts the same way, and the warning ends with "may raise an error in a future release", which would break backup and restore on the same upgrade. One edge of the loop is now marked `use_alter`, which removes it from the sort graph without removing the constraint from the database: PostgreSQL emits it as `ALTER TABLE ... ADD CONSTRAINT`, as it already did for every constraint on those three tables, and SQLite — which reports no ALTER support — inlines it into `CREATE TABLE` exactly as before, so `ON DELETE SET NULL` keeps working on both. `use_alter` requires a name, so new databases carry `fk_library_folders_archive_id` where existing ones keep PostgreSQL's generated `library_folders_archive_id_fkey`; nothing refers to either by name. Pinned by a test that fails if a new foreign key closes a loop again, and by one asserting the key still exists on SQLite — if that inlining ever stopped, deleting an archive would leave a dangling `library_folders.archive_id` instead of nulling it, silently.
- **The MakerWorld integration is now a model-provider package, so a second model site is an implementation rather than a second copy of the feature ([PR #2845](https://github.com/maziggy/bambuddy/pull/2845) by @pascalheidmann)** — `services/makerworld.py` was a single 635-line module that owned everything from URL parsing to the presigned S3 download, and the routes called it by name, so Thingiverse or Printables would have meant a second route file beside the first. It is now an interface: a `ModelProvider` descriptor carrying identity, host patterns, the credentials it needs, its library folder, its permissions and its SSRF allowlists, plus a per-request `ProviderService` that does the transport — with MakerWorld as the first implementation and a registry that hands a pasted URL to whichever provider claims it. Nothing about the MakerWorld flow changes: the endpoints and their request and response shapes are unchanged, the new `source_type` field defaults to `makerworld` so existing callers send exactly what they sent before, and the transport moved verbatim — thirteen of the eighteen functions are byte-identical by AST comparison, including the CDN allowlist, the 200 MB download cap and the certifi-pinned TLS context that keeps S3 downloads working on Windows. Four things did change. Bambu Cloud credential storage moved out of `routes/cloud.py` into `services/bambu_cloud_credentials.py`, because a feature package that has to import a route module to read a stored token is a dependency pointing the wrong way; the cloud endpoints themselves did not move. An import naming a provider that is not registered is now refused before anything happens, where it used to create the destination folder on its way to failing. "Your Bambu sign-in expired" is now an explicit flag the provider sets rather than something inferred from whether an error message happened to be present, so a network failure or a rate limit can no longer be reported to you as an expired sign-in. And each request is authorised against the permission of the provider it actually named instead of MakerWorld's — latent while MakerWorld is the only one registered, wrong on the day it is not.
- **Every FTP session Bambuddy opens now records how it closed (#3009, reported by @grengojbo)** — the report traced a print completion that opened two FTP connections to the printer, deleted one file and then, as far as the log showed, did nothing else until the printer was powered off 21 minutes later, and concluded the connections were being left open. They were not: the post-print SD-card cleanup opens one connection per candidate filename and closes each in a `finally`, which a run against a real FTPS server confirms at the server end for both the delete and the 550 not-here case. The trouble is that nothing in the log could have said so. Neither the clean close nor the hard socket drop logged anything at any level, so a session closed properly and a socket genuinely abandoned produced the same output — none — and the only way to tell them apart was to read the source. Both now log one DEBUG line naming the printer, whether QUIT was acknowledged or the socket had to be dropped without it, why, and how long the session was held. Every connect in a debug log is now paired with a close, so the next person suspecting a leaked FTP connection can settle it from a support bundle rather than by inference. Nothing about the connection handling itself changed, and at default log level nothing new is printed. This does not explain the SD-card read/write error in that report or in #645; it only removes one theory from the list by making it checkable.
### Fixed
- **Restoring a backup into a different version of Bambuddy failed, and the failure took the live database with it** — A backup carries the schema of the install that made it, and the PostgreSQL restore path does not use that schema: it drops every table, recreates them from the running process's ORM, and inserts the backup's columns into the result. Any NOT NULL column the running version has and the backup does not then arrives with nothing to put in it, and a model-side `default=` cannot help, because SQLAlchemy applies one to ORM and Core inserts and never to the raw `text()` SQL this import builds — `create_all` emits no DDL default for it either. A backup made after `user_wallets.currency` was dropped in #3123, restored onto an install that still had the column, died on `null value in column "currency"` with the drop already committed: the restore's first transaction wipes the database, so the install was left with an empty schema, the previous data gone, and the MFA key file already replaced by the backup's — whatever had survived would have been encrypted under a key that no longer matched. The import now fills such a column from the model's default (evaluating a callable one), leaves a column with a server default to the database rather than second-guessing a timestamp it is better placed to produce, and ignores a nullable one exactly as before. A column that genuinely cannot be filled is refused **before anything is touched** — before services are paused, before the key file is written, before the first `DROP` — and the same applies to a `bambuddy.db` that is not a readable SQLite database at all, which previously was discovered only by the import that had already dropped the tables. Backups now carry a `manifest.json` naming the version that wrote them, so the refusal reads "this backup was made by X, this install runs Y" rather than a list of column names; a backup taken before the manifest existed restores exactly as it did, with the version simply unknown. The refusal is an HTTP 400, and the frontend was dropping its body on the floor — a refused restore is an `HTTPException`, so its `{detail}` is not the `{success, message}` a successful one returns, and the modal raised an empty error toast — so the reason now reaches the operator who has to act on it. A SQLite install is unaffected throughout: it restores by copying the backup's pages, schema included, and migrates forward afterwards.
- **Slicing a plate of many copies of one part could take the server down (#3135, reported by @TheUltimateC0der)** — the reporter sliced a plate of 25 small Gridfinity bins and Bambuddy was OOM-killed straight after the slice finished, taking the slice job with it. After every slice Bambuddy draws the plate thumbnail itself, because the slicer's command-line mode leaves it out, and it read the model with trimesh. trimesh mishandles the layout Bambu Studio and OrcaSlicer save projects in, where each copy is a reference into a shared model file: it read that file again for every reference and kept every copy of its triangles, so 25 copies of a part arrived as 625, and 25 bins of 10,000 triangles each became 6.4 million. Drawing that took 54 seconds and 8.4 GB, and it ran on the server's main loop, so the web UI and every printer connection stalled for as long. A project whose model file holds several parts had a second symptom: each placed part pulled in every other part from the same file, so its thumbnail drew parts in places they were never put. The plate is now read directly, each model file once and each part once however many times it is placed, and a large plate is simplified per part to a fixed triangle budget before its copies are laid out. The reporter's plate now renders in about 2.4 seconds at under 400 MB, and the heaviest real projects measured went from 48 seconds and 9.3 GB to 20 seconds and 0.8 GB. The thumbnail is drawn off the main loop, and a plate still too large after simplifying is left without one instead of risking the server.
- **Moving a queued job from "Any P2S" to one P2S threw away its filament colour (#3133, reported by @bgrr74)** — a job queued for any printer of a model can print in a different filament from the one it was sliced with: brown in the 3MF, Bone White on the spool. Editing that job and switching it to a specific printer of the same model reset the choice, and the job went back to brown. The print dialog cleared overrides whenever the target model changed, and "Specific Printer" empties the target model, so the switch counted as a change. Clearing would not have been the whole problem anyway: in specific-printer mode the dialog matched trays against the 3MF's colours, never sent the override, and left the old one on the queue item, where it contradicted the trays the job would actually use. Now the override is kept through the switch, and the chosen printer's trays are matched against the requested filament, so Bone White picks the Bone White spool, and a printer without one shows the usual colour-mismatch warning for Bone White instead of quietly settling on brown. The job keeps the override on the queue item, so if the dialog could not resolve every tray, the scheduler's match at dispatch looks for the same colour. Moving the job to a different printer model, or picking a different plate, still clears the override, as before. A printer's queue card and badge no longer hide a job assigned to that printer because its override colour is not loaded. That check is for picking among printers of a model, which a job for one printer has already done. The same goes for a virtual printer's force-colour jobs sent to a specific printer, which the card used to hide the same way even though the scheduler dispatched them. Saving a virtual-printer job in the print dialog now keeps the PLA-variant pin (Basic, Matte or Silk) its force-colour overrides carry. Editing one queued for any printer of a model used to strip it.
- **Asking for 25 copies across two printer models queued exactly one (#3101, reported by @Leander-Vh)** — the reporter sliced one project for two printer types, selected both files in the File Manager, set the count and pressed Print. One item printed, the queue was empty behind it, and the Batches tab held nothing. Two features had met badly. Cross-model alternatives (#671) put several sliced files behind one queue slot, and a multi-plate file (#342) moves the run count out of the modal's single Quantity field onto a stepper beside each plate. The cross-model submit reads the field, not the steppers — and on a multi-plate file that field is not rendered at all, so it stayed at its initial 1 with nothing on screen able to change it. The modal read "19 runs in total" directly above a button that queued one. Cross-model no longer borrows the per-plate steppers: such a job picks its plate per candidate, down in the alternatives list, so there is only ever one number to give and the ordinary Quantity field is back. The plate selector above it is gone in that mode for the same reason — its choice never reached the request. It only decided which plate the filament panel described, which is why picking plate 3 for a candidate while plate 1 stayed ticked above it produced overrides computed from a plate the job would not print; that panel now follows the first candidate's own plate. The backend needed nothing for the count itself — it has always created one item per copy and given each its own candidate rows, so the same request now produces 25 jobs, each free to land on whichever of the two models frees up first. It did need a name: a cross-model job carries neither an archive nor a library file, because the candidates are the files, so both branches that name a batch missed and the order the reporter went looking for would have appeared in the Batches tab as "Batch". It is now named after its first candidate, the same file the print dialog names the job after.
+183 -5
View File
@@ -17,7 +17,7 @@ from backend.app.core.auth import (
require_auth_if_enabled,
require_energy_cost_update,
)
from backend.app.core.config import settings as app_settings
from backend.app.core.config import APP_VERSION, settings as app_settings
from backend.app.core.database import get_db
from backend.app.core.permissions import Permission
from backend.app.models.settings import Settings
@@ -792,6 +792,20 @@ async def create_backup_zip(output_path: Path | None = None) -> tuple[Path, str]
except PermissionError as e:
logger.warning("Permission denied copying %s: %s", name, e)
# Say which version made this, so a restore that cannot import it can
# name the versions rather than a list of columns. Backups from before
# this existed simply have no manifest, and restore treats the version
# as unknown.
import json as _json
manifest = {
"format": 1,
"app_version": APP_VERSION,
"created_at": datetime.now().isoformat(timespec="seconds"),
"database": "sqlite" if is_sqlite() else "postgresql",
}
(temp_path / "manifest.json").write_text(_json.dumps(manifest, indent=2) + "\n")
# Include the MFA encryption key as a ZIP top-level entry alongside
# bambuddy.db. Without it, encrypted client_secret / TOTP secret rows
# would be unrecoverable after restore on a host without MFA_ENCRYPTION_KEY set.
@@ -853,6 +867,119 @@ async def create_backup(
)
class BackupSchemaIncompatible(Exception):
"""The backup has no value for a column this version requires.
A backup carries the schema of the install that made it. Restoring it into
a different version means the destination can have NOT NULL columns the
backup never heard of -- either because that version is older and still has
a column since removed (``user_wallets.currency``, dropped in #3123), or
because it is newer and has added one. Most such columns have a default and
can simply be filled. The ones that cannot are what this reports, and it has
to be reported BEFORE the restore drops anything: the Postgres import wipes
every table in the first transaction, so a failure halfway leaves the
install with an empty schema and the previous data gone.
"""
def _missing_required_columns(pg_table, src_columns: set[str]):
"""Split the destination's NOT NULL columns that the backup lacks.
Returns ``(injectable, db_filled, unfillable)``:
* ``injectable`` -- ``{name: value}`` from the model's Python-side default.
These are invisible to the import's raw SQL: SQLAlchemy applies a
``default=`` on ORM and Core inserts, never on ``text()``, and
``create_all`` emits no DDL default for one. So a column like
``currency VARCHAR(3) NOT NULL`` with ``default="EUR"`` arrives with
nothing to put in it unless we put it there.
* ``db_filled`` -- has a server default or is the autoincrement key; the
database fills it when the column is left out of the INSERT.
* ``unfillable`` -- nothing can supply a value. The backup is incompatible.
"""
injectable: dict = {}
db_filled: list[str] = []
unfillable: list[str] = []
for col in pg_table.columns:
if col.nullable or col.name in src_columns:
continue
if col.default is not None:
arg = col.default.arg
injectable[col.name] = arg(None) if callable(arg) else arg
elif col.server_default is not None or col.primary_key:
db_filled.append(col.name)
else:
unfillable.append(col.name)
return injectable, db_filled, unfillable
def check_backup_schema_compatible(sqlite_path: Path, backup_version: str | None = None) -> None:
"""Raise if this version cannot import that backup. Touches nothing.
Only the cross-engine path needs this. A SQLite install restores by copying
the backup's pages, schema included, and `init_db()` migrates it forward
afterwards; the Postgres import instead recreates the schema from THIS
process's ORM and then inserts the backup's columns into it.
"""
import sqlite3
from backend.app.core.database import Base
src = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True)
try:
src_tables = {
row[0]
for row in src.execute(
"SELECT name FROM sqlite_master WHERE type='table' "
"AND name NOT LIKE 'sqlite_%' AND name NOT LIKE 'archive_fts%'"
)
}
problems: list[str] = []
# metadata.tables, not sorted_tables: the latter warns about the
# library_files/library_folders/print_archives cycle, and nothing here
# depends on the order.
for name, pg_table in Base.metadata.tables.items():
if name not in src_tables:
continue
# An empty table inserts nothing, so a column it cannot supply
# cannot fail. Refusing a restore over one would be a false alarm.
if src.execute(f'SELECT 1 FROM "{name}" LIMIT 1').fetchone() is None: # noqa: S608 # nosec B608 — name comes from ORM metadata
continue
src_columns = {row[1] for row in src.execute(f'PRAGMA table_info("{name}")')}
_, _, unfillable = _missing_required_columns(pg_table, src_columns)
problems.extend(f"{name}.{col}" for col in unfillable)
finally:
src.close()
if not problems:
return
made_by = f"The backup was made by Bambuddy {backup_version}, " if backup_version else "The backup "
raise BackupSchemaIncompatible(
"This backup cannot be restored by this version of Bambuddy. It carries no value for "
f"{len(problems)} column(s) this version requires and cannot default: {', '.join(sorted(problems))}. "
f"{made_by}and this install runs {APP_VERSION}. Restore it on the version that made it, or "
"upgrade this install to that version. Nothing has been changed."
)
def _read_backup_manifest(temp_path: Path) -> dict:
"""The backup's manifest.json, or {} for a backup made before it existed."""
import json
path = temp_path / "manifest.json"
if not path.is_file():
return {}
try:
data = json.loads(path.read_text())
except (OSError, ValueError) as exc:
logger.warning("Ignoring unreadable backup manifest: %s", exc)
return {}
return data if isinstance(data, dict) else {}
async def _import_sqlite_to_postgres(sqlite_path: Path, postgres_url: str):
"""Import data from a SQLite database file into the current PostgreSQL database.
@@ -865,6 +992,11 @@ async def _import_sqlite_to_postgres(sqlite_path: Path, postgres_url: str):
from backend.app.core.database import Base, _create_engine
# Before anything is dropped. The route checks this too, earlier and with
# the backup's version in the message; this call is what makes the guarantee
# a property of the import itself rather than of one caller.
check_backup_schema_compatible(sqlite_path)
# Create a temporary engine for the import (current engine was disposed)
pg_engine = _create_engine()
@@ -973,8 +1105,24 @@ async def _import_sqlite_to_postgres(sqlite_path: Path, postgres_url: str):
if not columns:
continue
col_list = ", ".join(columns)
param_list = ", ".join(f":{c}" for c in columns)
# Columns this schema requires that the backup does not have at
# all. The block below handles a column PRESENT in the backup
# with a NULL in it; one the backup never had is not in
# `columns` and so never reached it -- which is how a backup
# from an install without `user_wallets.currency` died on
# NotNullViolationError against a version that still had it.
injected, _db_filled, _unfillable = _missing_required_columns(pg_table, set(src_columns))
if injected:
logger.info(
"Filling %s column(s) absent from the backup in %s: %s",
len(injected),
table_name,
", ".join(sorted(injected)),
)
insert_columns = columns + list(injected)
col_list = ", ".join(insert_columns)
param_list = ", ".join(f":{c}" for c in insert_columns)
# ON CONFLICT DO NOTHING handles duplicate rows from SQLite (which doesn't enforce unique constraints)
insert_sql = text(f"INSERT INTO {table_name} ({col_list}) VALUES ({param_list}) ON CONFLICT DO NOTHING") # noqa: S608 # nosec B608
@@ -1015,9 +1163,15 @@ async def _import_sqlite_to_postgres(sqlite_path: Path, postgres_url: str):
now = dt.now()
def _convert_row(
row, cols=columns, bools=bool_columns, dts=datetime_columns, nn_defaults=not_null_defaults, _now=now
row,
cols=columns,
bools=bool_columns,
dts=datetime_columns,
nn_defaults=not_null_defaults,
_now=now,
inject=injected,
):
result = {}
result = dict(inject)
for c in cols:
val = row[c]
if val is None and c in nn_defaults:
@@ -1149,6 +1303,30 @@ async def restore_backup(
if not backup_db.exists():
raise HTTPException(400, "Invalid backup: missing bambuddy.db")
# 2b. Can this version import this backup at all?
#
# Deliberately here: everything below has a side effect. The virtual
# printer stops, background services stop, the MFA key file is
# overwritten with the backup's -- and then the Postgres import drops
# every table in its first transaction. A backup rejected at the INSERT
# took the install's data with it and left the encrypted secrets under a
# key that no longer matches. Nothing above this line has touched
# anything.
import sqlite3
manifest = _read_backup_manifest(temp_path)
backup_version = manifest.get("app_version")
if backup_version:
logger.info("Backup was created by Bambuddy %s; this install runs %s", backup_version, APP_VERSION)
if not is_sqlite():
try:
check_backup_schema_compatible(backup_db, backup_version)
except BackupSchemaIncompatible as exc:
logger.error("Refusing backup: %s", exc)
raise HTTPException(400, str(exc)) from exc
except sqlite3.DatabaseError as exc:
raise HTTPException(400, f"Invalid backup: bambuddy.db is not readable ({exc})") from exc
try:
import asyncio
+23 -1
View File
@@ -25,7 +25,29 @@ class LibraryFolder(Base):
# Link to project or archive
project_id: Mapped[int | None] = mapped_column(ForeignKey("projects.id", ondelete="SET NULL"), nullable=True)
archive_id: Mapped[int | None] = mapped_column(ForeignKey("print_archives.id", ondelete="SET NULL"), nullable=True)
# use_alter breaks a dependency cycle in the schema, and is not about this
# link being special: print_archives.library_file_id -> library_files,
# library_files.folder_id -> library_folders, and this column back to
# print_archives. Each is reasonable alone and together they are a loop
# SQLAlchemy cannot topologically sort, so metadata.sorted_tables dropped
# those edges, warned on every backup and restore, and could hand back an
# order placing a child before its parent -- which once imported
# library_files ahead of library_folders and killed a restore on a foreign
# key violation. Marking ONE edge for ALTER removes it from the sort graph
# and the other two order correctly. The constraint is still created and
# still enforced: PostgreSQL emits it as ALTER TABLE ADD CONSTRAINT (as it
# already does for every constraint on these three tables), and SQLite,
# which reports no ALTER support, inlines it into CREATE TABLE as before.
# It has to be named, because an unnamed constraint cannot be ALTERed in.
archive_id: Mapped[int | None] = mapped_column(
ForeignKey(
"print_archives.id",
ondelete="SET NULL",
use_alter=True,
name="fk_library_folders_archive_id",
),
nullable=True,
)
# Timestamps
created_at: Mapped[datetime] = mapped_column(DateTime, server_default=func.now())
@@ -0,0 +1,89 @@
"""The backup says which version made it, and restore refuses one it cannot import.
Restoring a backup into a different version of Bambuddy is ordinary -- an
upgrade, a rebuild, a move to another host. What is not ordinary is the Postgres
restore path, which throws the backup's schema away and rebuilds from the
running ORM. A NOT NULL column the running version has and the backup does not
then has nothing to put in it, and the import fails on the INSERT: after the
drop, with the install's data already gone.
So the incompatibility has to be found before any of that, and it has to say
something an operator can act on. A column name does not; a pair of version
numbers does, which is what the manifest is for.
"""
from __future__ import annotations
import io
import json
import sqlite3
import zipfile
from pathlib import Path
from unittest.mock import patch
import pytest
from backend.app.core.config import APP_VERSION, settings as app_settings
@pytest.mark.asyncio
@pytest.mark.integration
async def test_the_backup_records_the_version_that_made_it(async_client, monkeypatch, tmp_path):
from backend.app.api.routes.settings import create_backup_zip
monkeypatch.setenv("DATA_DIR", str(tmp_path))
monkeypatch.setattr(app_settings, "base_dir", tmp_path)
zip_path, _filename = await create_backup_zip(output_path=tmp_path)
try:
with zipfile.ZipFile(zip_path) as zf:
assert "manifest.json" in zf.namelist()
manifest = json.loads(zf.read("manifest.json"))
finally:
zip_path.unlink(missing_ok=True)
assert manifest["app_version"] == APP_VERSION
assert manifest["format"] == 1
assert manifest["database"] in ("sqlite", "postgresql")
assert manifest["created_at"]
def _incompatible_backup(tmp_path: Path, *, version: str) -> bytes:
"""A backup whose `cost_centers` has no `name` -- NOT NULL, no default."""
db = tmp_path / "bambuddy.db"
conn = sqlite3.connect(db)
conn.execute("CREATE TABLE cost_centers (id INTEGER PRIMARY KEY, code TEXT)")
conn.execute("INSERT INTO cost_centers (id, code) VALUES (1, 'abc')")
conn.commit()
conn.close()
buffer = io.BytesIO()
with zipfile.ZipFile(buffer, "w") as zf:
zf.write(db, "bambuddy.db")
zf.writestr("manifest.json", json.dumps({"format": 1, "app_version": version}))
return buffer.getvalue()
@pytest.mark.asyncio
@pytest.mark.integration
async def test_an_unimportable_backup_is_refused_with_both_versions(async_client, tmp_path):
"""A 400 naming the two versions, and -- the point -- nothing touched.
is_sqlite is patched false because this is the PostgreSQL path: a SQLite
install restores by copying the backup's pages, schema and all, and has
never had this problem.
"""
payload = _incompatible_backup(tmp_path, version="99.9.9")
with patch("backend.app.core.db_dialect.is_sqlite", return_value=False):
response = await async_client.post(
"/api/v1/settings/restore",
files={"file": ("backup.zip", payload, "application/zip")},
)
assert response.status_code == 400, response.text
detail = response.json()["detail"]
assert "cost_centers.name" in detail
assert "99.9.9" in detail
assert APP_VERSION in detail
assert "Nothing has been changed" in detail
+25 -4
View File
@@ -1912,6 +1912,27 @@ class TestEncryptionRoundtrip:
# ============================================================================
def _minimal_sqlite_backup() -> bytes:
"""A real, if empty, SQLite database to stand in for a backup's bambuddy.db.
Restore now refuses a bambuddy.db it cannot open, and refuses it before it
stops services or overwrites the MFA key file -- a corrupt or truncated
backup used to be found only by the Postgres import, which by then had
dropped every table in the live database. These tests are about the key
handling around the swap, so they need a file that opens; what is in it does
not matter.
"""
import sqlite3
conn = sqlite3.connect(":memory:")
try:
conn.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, username TEXT)")
conn.commit()
return conn.serialize()
finally:
conn.close()
class TestBackupKeyFiles:
@pytest.mark.asyncio
@pytest.mark.integration
@@ -1980,7 +2001,7 @@ class TestBackupKeyFiles:
# Build a minimal ZIP with a stub DB and the key file.
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w") as zf:
zf.writestr("bambuddy.db", b"SQLite format 3")
zf.writestr("bambuddy.db", _minimal_sqlite_backup())
zf.writestr(".mfa_encryption_key", "test-restored-key")
buf.seek(0)
@@ -2020,7 +2041,7 @@ class TestBackupKeyFiles:
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w") as zf:
zf.writestr("bambuddy.db", b"SQLite format 3")
zf.writestr("bambuddy.db", _minimal_sqlite_backup())
# Intentionally no .mfa_encryption_key entry.
buf.seek(0)
@@ -2061,7 +2082,7 @@ class TestBackupKeyFiles:
# Build ZIP with a key file that we will fail to write to DATA_DIR.
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w") as zf:
zf.writestr("bambuddy.db", b"SQLite format 3 backup data")
zf.writestr("bambuddy.db", _minimal_sqlite_backup())
zf.writestr(".mfa_encryption_key", "backup-key-content")
buf.seek(0)
@@ -2141,7 +2162,7 @@ class TestBackupKeyFiles:
assert new_key != old_key
buf = io.BytesIO()
with zipfile.ZipFile(buf, "w") as zf:
zf.writestr("bambuddy.db", b"SQLite format 3 backup data")
zf.writestr("bambuddy.db", _minimal_sqlite_backup())
zf.writestr(".mfa_encryption_key", new_key)
buf.seek(0)
+86
View File
@@ -0,0 +1,86 @@
"""The schema has to be sortable, because backup and restore sort it.
`metadata.sorted_tables` is asked for the order in three places: the backup
export, the restore's import loop, and the loop that puts foreign keys back
afterwards. Three nullable SET NULL links used to close a loop --
print_archives.library_file_id -> library_files.folder_id ->
library_folders.archive_id -> print_archives -- and SQLAlchemy answered a
sort it could not make, with a warning on every backup and every restore:
Cannot correctly sort tables; there are unresolvable cycles between
tables "library_files, library_folders, print_archives" ... this warning
may raise an error in a future release.
Two things were wrong with living on that. The order it returns can place a
child before its parent, which is what once imported library_files ahead of
library_folders and killed a restore on a ForeignKeyViolation. And the
sentence at the end is a promise: if it ever becomes an error, backup and
restore break on the same upgrade.
One edge of the loop is marked use_alter, which takes it out of the sort
graph without taking the constraint out of the database.
"""
from __future__ import annotations
import importlib
import pkgutil
import warnings
from sqlalchemy import create_engine, inspect
from backend.app.core.database import Base
def _all_models_imported() -> None:
"""Base.metadata is filled by imports, so a partial import means a partial
schema -- and a cycle in a table nobody imported would not be found here."""
import backend.app.models as models
for module in pkgutil.iter_modules(models.__path__):
importlib.import_module(f"backend.app.models.{module.name}")
def test_the_schema_sorts_without_a_cycle_warning():
_all_models_imported()
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
assert Base.metadata.sorted_tables
cycles = [str(w.message) for w in caught if "cycles" in str(w.message)]
assert not cycles, (
f"a new foreign key has closed a loop in the schema; backup and restore sort these tables: {cycles}"
)
def test_the_tables_that_used_to_cycle_sort_parents_first():
"""The property the warning took away. Order is what the restore's import
loop follows, and a child ahead of its parent is a FK violation."""
_all_models_imported()
order = [t.name for t in Base.metadata.sorted_tables]
assert order.index("library_folders") < order.index("library_files"), (
"library_files.folder_id points at library_folders"
)
assert order.index("library_files") < order.index("print_archives"), (
"print_archives.library_file_id points at library_files"
)
def test_the_altered_constraint_still_exists_on_sqlite():
"""use_alter asks for ALTER TABLE ADD CONSTRAINT, and SQLite has no such
statement. It inlines the key into CREATE TABLE instead -- but if that ever
stopped being true, deleting an archive would leave a dangling
library_folders.archive_id rather than nulling it, silently."""
_all_models_imported()
engine = create_engine("sqlite://")
Base.metadata.create_all(engine)
keys = inspect(engine).get_foreign_keys("library_folders")
archive_link = [k for k in keys if k["referred_table"] == "print_archives"]
assert archive_link, f"library_folders lost its archive key: {keys}"
assert archive_link[0]["constrained_columns"] == ["archive_id"]
assert archive_link[0]["options"].get("ondelete") == "SET NULL"
@@ -0,0 +1,306 @@
"""Restoring a backup made by a different version of Bambuddy.
A backup carries the schema of the install that made it, and the Postgres
restore path does NOT use that schema: it drops every table, recreates them
from the running process's ORM, and inserts the backup's columns into the
result. So any NOT NULL column the running version has and the backup does not
arrives with nothing to put in it.
That is how a 2026-09-23 backup failed to restore on the published image:
``user_wallets.currency`` was dropped from the model in #3123, the backup
therefore had no such column, the older image still declared it NOT NULL, and
the import died on::
null value in column "currency" of relation "user_wallets"
Its ``default="EUR"`` could not help -- SQLAlchemy applies a Python-side default
to ORM and Core inserts, never to the raw ``text()`` SQL this import builds, and
``create_all`` emits no DDL default for one.
Two things have to hold, and the second is the serious one:
1. A column with a default is filled rather than refused.
2. A column that cannot be filled is refused BEFORE the restore drops anything.
The drop is the first thing the import does, in its own transaction, so a
refusal at the INSERT means the install's data is already gone -- and the
restore has by then also overwritten the MFA key file, leaving whatever
survives encrypted under a key that no longer matches.
"""
from __future__ import annotations
import sqlite3
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from sqlalchemy import Column, DateTime, Integer, MetaData, Numeric, String, Table, func
from backend.app.api.routes.settings import (
BackupSchemaIncompatible,
_missing_required_columns,
_read_backup_manifest,
check_backup_schema_compatible,
)
def _wallets_table(metadata: MetaData) -> Table:
"""`user_wallets` as the published image still declares it."""
return Table(
"user_wallets",
metadata,
Column("id", Integer, primary_key=True),
Column("user_id", Integer, nullable=False),
Column("balance", Numeric(14, 2), nullable=False, default=0.0),
Column("currency", String(3), nullable=False, default="EUR"),
Column("updated_at", DateTime, nullable=False, server_default=func.now()),
)
# ---------------------------------------------------------------------------
# Which missing columns are a problem
# ---------------------------------------------------------------------------
def test_a_column_with_a_model_default_is_filled_not_refused():
"""The exact case from the incident, with the values it would have used."""
table = _wallets_table(MetaData())
injectable, db_filled, unfillable = _missing_required_columns(table, {"id", "user_id", "balance", "updated_at"})
assert injectable == {"currency": "EUR"}
assert unfillable == []
assert db_filled == []
def test_a_column_with_a_server_default_is_left_to_the_database():
"""Omitting it from the INSERT is right: Postgres fills it. Sending the
ORM's idea of the default instead would overwrite a timestamp the database
is better placed to produce."""
table = _wallets_table(MetaData())
injectable, db_filled, unfillable = _missing_required_columns(table, {"id", "user_id", "balance", "currency"})
assert db_filled == ["updated_at"]
assert injectable == {}
assert unfillable == []
def test_a_callable_default_is_evaluated():
table = Table(
"cost_centers",
MetaData(),
Column("id", Integer, primary_key=True),
Column("code", String(32), nullable=False, default=lambda: "generated"),
)
injectable, _, unfillable = _missing_required_columns(table, {"id"})
assert injectable == {"code": "generated"}
assert unfillable == []
def test_a_required_column_with_no_default_is_unfillable():
table = Table(
"cost_centers",
MetaData(),
Column("id", Integer, primary_key=True),
Column("name", String(150), nullable=False),
)
injectable, _, unfillable = _missing_required_columns(table, {"id"})
assert unfillable == ["name"]
assert injectable == {}
def test_a_nullable_column_the_backup_lacks_is_not_a_problem():
"""Most schema drift is this, and it has always worked: the column is
simply omitted and the row gets NULL."""
table = Table(
"printers",
MetaData(),
Column("id", Integer, primary_key=True),
Column("nickname", String(50), nullable=True),
)
injectable, db_filled, unfillable = _missing_required_columns(table, {"id"})
assert (injectable, db_filled, unfillable) == ({}, [], [])
# ---------------------------------------------------------------------------
# The preflight, against a real backup file
# ---------------------------------------------------------------------------
def _source(tmp_path: Path, ddl: str, rows: list[str]) -> Path:
path = tmp_path / "bambuddy.db"
conn = sqlite3.connect(path)
conn.execute(ddl)
for row in rows:
conn.execute(row)
conn.commit()
conn.close()
return path
def test_a_backup_this_version_can_import_is_accepted(tmp_path):
"""`users` as the ORM has it, minus columns that are nullable or defaulted."""
path = _source(
tmp_path,
"CREATE TABLE users (id INTEGER PRIMARY KEY, username TEXT)",
["INSERT INTO users (id, username) VALUES (1, 'alice')"],
)
check_backup_schema_compatible(path) # does not raise
def test_a_backup_missing_a_required_column_is_refused(tmp_path):
"""`cost_centers.name` is NOT NULL with no default anywhere."""
path = _source(
tmp_path,
"CREATE TABLE cost_centers (id INTEGER PRIMARY KEY, code TEXT)",
["INSERT INTO cost_centers (id, code) VALUES (1, 'abc')"],
)
with pytest.raises(BackupSchemaIncompatible) as exc:
check_backup_schema_compatible(path)
assert "cost_centers.name" in str(exc.value)
assert "Nothing has been changed" in str(exc.value)
def test_the_refusal_names_the_version_that_made_the_backup(tmp_path):
"""A column name tells an operator nothing about what to do. Two version
numbers tell them which install to restore on."""
path = _source(
tmp_path,
"CREATE TABLE cost_centers (id INTEGER PRIMARY KEY, code TEXT)",
["INSERT INTO cost_centers (id, code) VALUES (1, 'abc')"],
)
with pytest.raises(BackupSchemaIncompatible) as exc:
check_backup_schema_compatible(path, backup_version="1.2.7")
assert "1.2.7" in str(exc.value)
def test_an_empty_table_is_not_a_reason_to_refuse(tmp_path):
"""No rows, no INSERT, no violation. Refusing here would block a restore
over a feature the backup's install never used."""
path = _source(tmp_path, "CREATE TABLE cost_centers (id INTEGER PRIMARY KEY, code TEXT)", [])
check_backup_schema_compatible(path) # does not raise
def test_a_table_the_orm_does_not_know_is_ignored(tmp_path):
"""Backups carry tables from features this version has removed. The import
skips them (it only imports source ∩ ORM), so the check must not judge them
either -- a column requirement that no longer exists cannot fail an INSERT
that will never be made."""
from backend.app.core.database import Base
assert "legacy_removed_feature" not in Base.metadata.tables
path = _source(
tmp_path,
"CREATE TABLE legacy_removed_feature (id INTEGER PRIMARY KEY, whatever TEXT)",
["INSERT INTO legacy_removed_feature (id, whatever) VALUES (1, 'x')"],
)
check_backup_schema_compatible(path) # does not raise
# ---------------------------------------------------------------------------
# The import itself
# ---------------------------------------------------------------------------
def _mock_engine():
"""An engine that records every statement and parameter set."""
executed: list[tuple[str, object]] = []
conn = MagicMock()
conn.execute = AsyncMock(
side_effect=lambda stmt, *a, **k: executed.append((getattr(stmt, "text", str(stmt)), a[0] if a else None))
)
conn.run_sync = AsyncMock()
begin_cm = MagicMock()
begin_cm.__aenter__ = AsyncMock(return_value=conn)
begin_cm.__aexit__ = AsyncMock(return_value=False)
engine = MagicMock()
engine.begin = MagicMock(return_value=begin_cm)
engine.dispose = AsyncMock()
return engine, executed
@pytest.mark.asyncio
async def test_the_import_refuses_before_it_drops_anything(tmp_path):
"""The whole point. The drop is the import's first act and it is not
reversible: by the time an INSERT fails, the install is empty."""
path = _source(
tmp_path,
"CREATE TABLE cost_centers (id INTEGER PRIMARY KEY, code TEXT)",
["INSERT INTO cost_centers (id, code) VALUES (1, 'abc')"],
)
from backend.app.api.routes import settings as settings_module
engine, executed = _mock_engine()
create_engine = MagicMock(return_value=engine)
with (
patch("backend.app.core.database._create_engine", new=create_engine),
pytest.raises(BackupSchemaIncompatible),
):
await settings_module._import_sqlite_to_postgres(path, "postgresql+asyncpg://test/test")
assert executed == [], f"SQL ran against the destination before the refusal: {executed}"
create_engine.assert_not_called()
@pytest.mark.asyncio
async def test_a_missing_defaulted_column_is_inserted_with_its_default(tmp_path):
"""What would have rescued the failed restore: the column absent from the
backup joins the INSERT carrying the model's default."""
path = _source(
tmp_path,
"CREATE TABLE cost_centers (id INTEGER PRIMARY KEY, code TEXT, name TEXT, created_at TEXT, updated_at TEXT)",
[
"INSERT INTO cost_centers (id, code, name, created_at, updated_at) "
"VALUES (1, 'abc', 'Lab', '2026-09-07 10:44:42', '2026-09-07 10:44:42')"
],
)
from backend.app.api.routes import settings as settings_module
engine, executed = _mock_engine()
with patch("backend.app.core.database._create_engine", new=MagicMock(return_value=engine)):
await settings_module._import_sqlite_to_postgres(path, "postgresql+asyncpg://test/test")
inserts = [(sql, params) for sql, params in executed if sql.startswith("INSERT INTO cost_centers")]
assert inserts, f"no INSERT was emitted: {[sql[:60] for sql, _ in executed]}"
sql, params = inserts[0]
# is_active is NOT NULL with default=True in the model and absent above.
assert "is_active" in sql
assert params[0]["is_active"] is True
assert params[0]["code"] == "abc", "the backup's own values must survive the injection"
# ---------------------------------------------------------------------------
# The manifest
# ---------------------------------------------------------------------------
def test_a_backup_without_a_manifest_reads_as_unknown(tmp_path):
"""Every backup taken before the manifest existed. It must restore exactly
as it did, with the version simply unknown."""
assert _read_backup_manifest(tmp_path) == {}
def test_an_unreadable_manifest_does_not_break_the_restore(tmp_path):
(tmp_path / "manifest.json").write_text("{ this is not json")
assert _read_backup_manifest(tmp_path) == {}
def test_the_manifest_is_read(tmp_path):
(tmp_path / "manifest.json").write_text('{"format": 1, "app_version": "1.2.7"}')
assert _read_backup_manifest(tmp_path)["app_version"] == "1.2.7"
@@ -0,0 +1,64 @@
/**
* Tests for api.importBackup's handling of a refused restore.
*
* A restore the server declines is an HTTPException, so its body is
* `{detail}` — not the `{success, message}` a successful restore returns.
* importBackup used to hand that body straight back, which left `success`
* undefined (falsy, so the UI took the failure branch) and `message`
* undefined with it: the operator got an empty error toast and no idea why.
*
* It matters most for exactly the case it was found in — a backup this version
* cannot import. That refusal names the columns and both version numbers, and
* all of it was being dropped on the floor.
*/
import { describe, it, expect, beforeAll, afterEach, afterAll } from 'vitest';
import { http, HttpResponse } from 'msw';
import { setupServer } from 'msw/node';
import { api } from '../../api/client';
const server = setupServer();
beforeAll(() => server.listen({ onUnhandledRequest: 'bypass' }));
afterEach(() => server.resetHandlers());
afterAll(() => server.close());
const backupFile = () => new File(['(a backup)'], 'backup.zip', { type: 'application/zip' });
describe('api.importBackup', () => {
it('passes a successful restore through unchanged', async () => {
server.use(
http.post('*/settings/restore', () =>
HttpResponse.json({ success: true, message: 'Backup restored successfully.' })
)
);
const result = await api.importBackup(backupFile());
expect(result.success).toBe(true);
expect(result.message).toBe('Backup restored successfully.');
});
it('turns a refusal into a failure carrying the reason', async () => {
const detail =
'This backup cannot be restored by this version of Bambuddy. It carries no value for ' +
'1 column(s) this version requires and cannot default: cost_centers.name.';
server.use(http.post('*/settings/restore', () => HttpResponse.json({ detail }, { status: 400 })));
const result = await api.importBackup(backupFile());
expect(result.success).toBe(false);
expect(result.message).toBe(detail);
});
it('reports a failure even when the error body is not JSON', async () => {
server.use(
http.post('*/settings/restore', () => new HttpResponse('upstream exploded', { status: 502 }))
);
const result = await api.importBackup(backupFile());
expect(result.success).toBe(false);
expect(result.message).toBe('');
});
});
+15 -2
View File
@@ -5759,10 +5759,23 @@ export const api = {
headers,
body: formData,
});
return response.json() as Promise<{
const data = (await response.json().catch(() => null)) as {
success?: boolean;
message?: string;
detail?: string;
} | null;
// A refused restore is an HTTPException, so the body is {detail}, not
// {success, message}. Returning it unmapped made `success` undefined and
// `message` undefined too — the modal then raised an empty error toast,
// which is the one case where the reason matters most (e.g. a backup this
// version cannot import names the columns and both versions).
if (!response.ok) {
return { success: false, message: data?.detail ?? data?.message ?? '' };
}
return (data ?? { success: false, message: '' }) as {
success: boolean;
message: string;
}>;
};
},
checkFfmpeg: () =>
request<{ installed: boolean; path: string | null }>('/settings/check-ffmpeg'),
@@ -1532,7 +1532,7 @@ export function GitHubBackupSettings() {
if (result.success) {
showToast(t('backup.backupRestoredRestart'), 'success');
} else {
showToast(result.message, 'error');
showToast(result.message || t('backup.failedToRestore'), 'error');
}
} catch (e) {
const message = e instanceof Error ? e.message : t('backup.failedToRestore');