public_git/bambuddy
1
0
Fork 0
mirror of https://github.com/maziggy/bambuddy.git synced 2026-10-21 01:57:41 +02:00
Code Issues Packages Projects Releases Wiki Activity

Drop the restore's foreign keys in the database, not in the ORM metadata

Browse Source 
Restoring a SQLite backup into PostgreSQL died part-way with

  insert or update on table "library_files" violates foreign key
  constraint "library_files_folder_id_fkey"
  DETAIL: Key (folder_id)=(1) is not present in table "library_folders".

The import recreates the schema and is supposed to create every table
without foreign keys, so the order rows arrive in cannot matter; the
constraints are added back once the data has landed. Phase 1 did that by
discarding each ForeignKeyConstraint from table.constraints before
create_all -- which only suppresses the inline REFERENCES clause.
Table.foreign_key_constraints is derived from the columns' ForeignKey
objects and was never touched, and when create_all meets a dependency
cycle it cannot sort, it falls back to emitting those tables' keys as
separate ALTER TABLE ... ADD FOREIGN KEY statements read from exactly
that property.

library_files, library_folders and print_archives form such a cycle, so
twelve constraints survived across the three of them -- measured against
a real PostgreSQL by running the old phase verbatim. The same cycle also
costs those tables their place in sorted_tables, so they were imported
alphabetically, putting library_files ahead of the library_folders rows
its folder_id references.

Phase 1 now creates the tables normally and drops every foreign key from
pg_constraint afterwards, in the same transaction, scoped to contype 'f'
in the public schema. That is indifferent to how create_all chose to
emit them, so a future cycle between other tables cannot bring this
back. Phase 3 is unchanged.

This also removes a second fault: the keys were stripped from the
process-wide Base.metadata and only restored after the drop/create
transaction, so a failure in between left the running app without them
until restart. The metadata is no longer modified at all.

Verified end to end against a real PostgreSQL -- a backup whose child
rows import before their parents restores cleanly, with all 90
constraints back afterwards. Four regression tests added.
This commit is contained in:
maziggy
2026-08-10 16:48:10 +02:00
parent 140ce1a593
commit fd3f5331f3
3 changed files with 241 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/app/api/routes/settings.py
+55 -18
View File
@@ -815,15 +815,8 @@ async def _import_sqlite_to_postgres(sqlite_path: Path, postgres_url: str):
sorted_tables = [t.name for t in metadata.sorted_tables if t.name in tables_to_import]
# Phase 1: Drop all tables and recreate WITHOUT foreign keys.
# This avoids all FK ordering/orphan issues during import.
saved_fks = {}
for table in metadata.sorted_tables:
fks = list(table.foreign_key_constraints)
if fks:
saved_fks[table.name] = fks
for fk in fks:
table.constraints.discard(fk)
# This avoids all FK ordering/orphan issues during import; the
# constraints go back on at the end, once every row has landed.
async with pg_engine.begin() as conn:
# Cap how long DROP TABLE will wait for AccessExclusiveLock so
# any residual concurrent writer (per-printer MQTT clients
@@ -856,11 +849,38 @@ async def _import_sqlite_to_postgres(sqlite_path: Path, postgres_url: str):
)
await conn.run_sync(metadata.create_all)
# Restore FK definitions in metadata (needed for re-adding later)
for table_name, fks in saved_fks.items():
table_obj = metadata.tables[table_name]
for fk in fks:
table_obj.constraints.add(fk)
# Now strip the foreign keys, at the database level.
#
# This used to be done by discarding each ForeignKeyConstraint
# from `table.constraints` before `create_all`. That only
# suppresses the inline REFERENCES clause inside CREATE TABLE:
# `Table.foreign_key_constraints` is derived from the *columns'*
# ForeignKey objects, which the discard never touched. When
# `create_all` meets a dependency cycle it can't sort -- and
# library_files / library_folders / print_archives are exactly
# such a cycle -- it falls back to emitting those tables' keys
# as separate ALTER TABLE ... ADD FOREIGN KEY statements read
# straight from that property. Twelve constraints survived,
# including library_files.folder_id, and because the same cycle
# also drops the ordering edge from `sorted_tables` the child
# table was imported before its parent and the restore died on
# a ForeignKeyViolationError.
#
# Dropping them from pg_constraint instead is indifferent to how
# create_all chose to emit them, so a future model cycle cannot
# reintroduce this. It also keeps the app's global Base.metadata
# untouched: the old code only put the constraints back *after*
# the transaction, so a failure in here left the running process
# with an FK-less metadata until restart.
await conn.execute(
text(
"DO $$ DECLARE r RECORD; BEGIN "
"FOR r IN (SELECT conrelid::regclass AS tbl, conname FROM pg_constraint "
"WHERE contype = 'f' AND connamespace = 'public'::regnamespace) LOOP "
"EXECUTE 'ALTER TABLE ' || r.tbl || ' DROP CONSTRAINT ' || quote_ident(r.conname); "
"END LOOP; END $$;"
)
)
# Phase 2: Import data (no FKs to worry about)
async with pg_engine.begin() as conn:
@@ -958,7 +978,7 @@ async def _import_sqlite_to_postgres(sqlite_path: Path, postgres_url: str):
src.close()
logger.info("Cross-database import complete: %d tables imported", len(tables_to_import))
# Recreate FK constraints from ORM metadata (not from saved definitions).
# Recreate FK constraints from ORM metadata, which Phase 1 left intact.
# Use individual transactions so orphaned SQLite data doesn't block valid FKs.
from sqlalchemy.schema import AddConstraint
@@ -968,11 +988,28 @@ async def _import_sqlite_to_postgres(sqlite_path: Path, postgres_url: str):
try:
async with pg_engine.begin() as fk_conn:
await fk_conn.execute(AddConstraint(fk))
except Exception:
failed_fks.append(f"{table.name}.{fk.name}")
except Exception as e:
# Name the constraint by what it links, not by `fk.name`:
# these are unnamed in the ORM, so that field is None and
# the warning used to read "print_archives.None" for every
# one of the five keys on that table -- unusable for
# working out which rows to go and look at.
cols = ", ".join(c.name for c in fk.columns)
target = fk.elements[0].target_fullname if fk.elements else "unknown"
failed_fks.append(f"{table.name}({cols}) -> {target}")
# Postgres puts the offending key in a DETAIL line; it
# names the exact orphan value, which is the one thing
# that turns this into an actionable report.
detail = next(
(ln.strip() for ln in str(e).splitlines() if ln.startswith("DETAIL:")),
str(e).splitlines()[0] if str(e) else e.__class__.__name__,
)
logger.info("FK %s(%s) -> %s not restored: %s", table.name, cols, target, detail)
if failed_fks:
logger.warning(
"Could not restore %d FK constraints (orphaned data in SQLite): %s",
"Could not restore %d FK constraints (orphaned data in the backup): %s. "
"The data is restored and usable; those columns are simply no longer "
"enforced. See the INFO lines above for the offending key in each case.",
len(failed_fks),
", ".join(failed_fks),
)