fix(backup): preserve NOT NULL/DEFAULT/FK/UNIQUE in Postgres→SQLite backup (#2526)

On a PostgreSQL install, create_backup_zip() exports a portable SQLite copy
so backups move between engines. It rebuilt each table with only column name
+ type + PK, dropping NOT NULL, server_default/DEFAULT, foreign keys, and
unique constraints. Restore onto SQLite page-copies that schema straight onto
the live database, and post-restore init_db() can't repair it (create_all is
CREATE TABLE IF NOT EXISTS). So server_default columns like
spoolbuddy_devices.created_at (server_default=func.now()) ended up with no
DEFAULT: SQLAlchemy omits them on INSERT, the DB wrote NULL, and the next read
500'd on Pydantic validation. Every server_default column was exposed the same
way; the FK/unique loss followed from the same simplified CREATE TABLE.

Build the portable schema with Base.metadata.create_all() against a SQLite
engine instead of the hand-rolled loop, so it emits the exact DDL a native
SQLite install gets (NOT NULL, DEFAULT func.now() -> CURRENT_TIMESTAMP, FKs,
unique constraints, indexes). The data-export insert path is unchanged, and
the #1333 OIDC-icon guard is preserved automatically (LargeBinary -> BLOB),
which lets the now-redundant _sqlalchemy_type_to_sqlite_type() helper be
removed. Fixes newly-created backups; a backup from an older build still
carries the degraded schema, so re-take backups after upgrading.

Replace the #1333 type-mapping unit tests with three that inspect the real
backup schema via metadata.create_all + PRAGMA table_info: icon_data is BLOB,
created_at keeps its CURRENT_TIMESTAMP DEFAULT, a NOT NULL non-PK column stays
NOT NULL.
This commit is contained in:
maziggy
2026-07-09 09:02:34 +02:00
parent 6127e30abf
commit 9e7f6cafd9
3 changed files with 85 additions and 73 deletions
+20 -40
View File
@@ -35,32 +35,6 @@ _SENSITIVE_FIELDS_FOR_API_KEY = (
)
def _sqlalchemy_type_to_sqlite_type(type_repr: str) -> str:
"""Map a SQLAlchemy column type's ``str()`` to a SQLite-native column type.
Used by ``create_backup_zip`` to reconstruct a portable SQLite database
file from PostgreSQL data. Falling through to TEXT for binary columns
corrupts non-UTF8 bytes — the BLOB branch is the #1333 regression guard
for OIDC icon BLOBs.
Extracted as a pure helper so it can be unit-tested without spinning up
the full FastAPI app + backup pipeline.
"""
type_str = type_repr.upper()
if "INT" in type_str:
return "INTEGER"
if "FLOAT" in type_str or "REAL" in type_str or "NUMERIC" in type_str:
return "REAL"
if "BOOL" in type_str:
return "BOOLEAN"
if "BLOB" in type_str or "BYTEA" in type_str or "BINARY" in type_str:
# OIDC icon BLOB column (#1333) — without this branch the column
# was created as TEXT and non-UTF8 bytes were corrupted during the
# PG→SQLite-ZIP backup round trip.
return "BLOB"
return "TEXT"
async def get_setting(db: AsyncSession, key: str) -> str | None:
"""Get a single setting value by key."""
result = await db.execute(select(Settings).where(Settings.key == key))
@@ -582,25 +556,31 @@ async def create_backup_zip(output_path: Path | None = None) -> tuple[Path, str]
import json
import sqlite3
from sqlalchemy import create_engine as create_sync_engine
from backend.app.core.database import Base, engine
backup_db_path = temp_path / "bambuddy.db"
dst = sqlite3.connect(str(backup_db_path))
metadata = Base.metadata
# Create tables in SQLite backup (simplified — just column names and types)
for table in metadata.sorted_tables:
cols = []
pk_cols = [col.name for col in table.columns if col.primary_key]
for col in table.columns:
col_type = _sqlalchemy_type_to_sqlite_type(str(col.type))
# Only inline PRIMARY KEY for single-column PKs
pk = " PRIMARY KEY" if col.primary_key and len(pk_cols) == 1 else ""
cols.append(f"{col.name} {col_type}{pk}")
# Add composite primary key constraint if needed
if len(pk_cols) > 1:
cols.append(f"PRIMARY KEY ({', '.join(pk_cols)})")
dst.execute(f"CREATE TABLE IF NOT EXISTS {table.name} ({', '.join(cols)})") # noqa: S608
# Build the portable SQLite schema with SQLAlchemy's own DDL rather
# than a hand-rolled CREATE TABLE. metadata.create_all() emits the
# exact schema a native SQLite install gets — NOT NULL, DEFAULT
# (server_default=func.now() → CURRENT_TIMESTAMP), foreign keys,
# unique constraints and indexes. The previous name+type-only
# rebuild dropped all of these, so a Postgres→SQLite restore left
# server_default columns (e.g. spoolbuddy_devices.created_at) with
# no DEFAULT — SQLAlchemy omits such columns on INSERT and the DB
# then wrote NULL, which 500'd on the next read (#2526). Using the
# real DDL also keeps the #1333 BLOB guard: LargeBinary still
# renders as BLOB, so OIDC icon bytes survive the round trip.
schema_engine = create_sync_engine(f"sqlite:///{backup_db_path}")
try:
metadata.create_all(schema_engine)
finally:
schema_engine.dispose()
dst = sqlite3.connect(str(backup_db_path))
# Export data from Postgres to SQLite
async with engine.connect() as conn: