`if not isinstance(total, int) or seen >= total or not entries: return blobs, ""`
— the first arm short-circuited the page loop into a **success** holding page 1
only. Gitea clamps `per_page` to `MAX_RESPONSE_ITEMS` (default 50), so that is
50 entries of an arbitrarily large tree returned as a complete listing.
The restore then reports genuinely-present categories as "Not present in this
backup commit". That silent skip is the exact failure this override exists to
prevent, and the same class as E7 and G2 — G2 fixed the arithmetic here and
left the shape. GitHub and GitLab both hard-fail in the equivalent spot; only
Gitea guessed, and it guessed in the one direction that loses data quietly.
Whether Gitea always sends `total_count` on this route is beside the point: the
code was defending against a response shape it did not trust, and then trusting
it.
Now a missing or non-int `total_count` means "page until a short or empty
page". A page shorter than the first one is the last one, floored at Gitea's
default clamp so a genuinely small tree still costs exactly one request — the
reason G2 rejected paging-until-short in the `total_count`-present case, which
is unchanged and still stops on the count. The existing `page <= 50` ceiling
gives the correct hard failure for a tree that really is over cap, so this
cannot truncate.
Residual, and deliberately not widened into a `return None` on the first
ambiguous response — that would break single-page trees, the common case: an
instance whose `MAX_RESPONSE_ITEMS` is set *below* 50 *and* which omits
`total_count` would still stop at page 1. Both halves have to be true.
Tests: +6 (paged to the end with no count, on both Gitea and Forgejo; a short
page ends it; a non-int count is treated as no count; the page ceiling still
fails). Fail-pre-fix 5, control that passes either way 1 (a small tree is one
request). These don't match `-k github`, so 274 -> 280 across the three restore
files but `-k github` is unmoved.
The pager computed `seen = (page - 1) * 1000 + len(entries)`, taking the
requested `per_page` as fact. Gitea clamps `per_page` to
`MAX_RESPONSE_ITEMS`, which defaults to 50. So on a default install page 1
returns 50 entries and sets `seen` to 50, then page 2 sets it to 1050 —
which clears any `total_count` under 1050. The loop returns `success: true`
holding the first 100 entries of a much larger tree.
The restore then reads every missing path as "category not present in this
commit" and skips it silently, which is precisely the failure this override
was written to prevent. Same class as the GitLab pager fix, in the one
direction that got left behind.
Fix: accumulate `seen += len(entries)`. A genuinely over-cap tree still
hard-fails rather than truncating; the cap is a page count, not a file
count, because the page size is the server's choice.
Test: a 120-entry tree served 50 at a time reaches its last entry, in three
requests. Confirmed failing against the pre-fix backend — it stopped after
two pages and reported 100 entries as the whole tree.
The three remaining review items, all in the read path.
E1 — four provider calls where two would do. preview() called list_commits
twice: once inside _resolve_ref to turn HEAD into a SHA, once more at limit=20
purely to find the entry describing that same SHA. And list_tree's recursive
tree GET was thrown away, so fetch_files immediately fetched the identical tree
again to map path -> blob SHA. _resolve_ref now returns the entry it already
has, and list_tree returns its blob_shas map for fetch_files to take as an
optional argument. GitLab reads files by path and ignores it.
E2 — `commit: null` for a ref outside the 20 most recent. Two causes, and the
second is the one that actually bit: REF_PATTERN accepts a 7-character ref while
providers return the full 40, so the exact `==` in the scan never matched an
abbreviated SHA *even when the commit was in the window*. Fixed by prefix
comparison, plus a get_commit(ref) on the GitHub and GitLab backends for the
genuinely-outside-the-window case. Gitea and Forgejo inherit GitHub's. Still
best-effort: it is a subject line and a date, so a failed lookup renders the
preview without them rather than failing it.
E7 — the two tree readers disagreed, and each was wrong in the other's
direction. GitHub's recursive trees endpoint is not paginated and signals
overflow with truncated=true, which _blob_shas_at hard-fails on. Gitea and
Forgejo *do* page that endpoint, and inherited that single GET unchanged — so a
large backup repo returned only the first page and every category beyond it
looked absent from the commit. GiteaBackend now has its own paging
_blob_shas_at. GitLab had the mirror-image bug the review did not name: at its
50-page cap it exited through the while condition and returned success: True
with a silently partial path list. Both now fail loudly, which is what the
GitHub version was always doing.
Both halves of E7 are the same failure the module already refuses to allow: a
restore that skips categories and calls it "not present in this backup commit".
24 new or changed tests, all failing against this commit's parent.
Gitea/Forgejo served under a ROOT_URL path prefix (e.g. https://host/gitea)
place repos at /<prefix>/owner/repo. The Gitea backend assumed a host-root
layout: parse_repo_url required exactly two path segments (so subpath URLs
raised "Cannot parse repository URL") and get_api_base dropped the prefix,
yielding https://host/api/v1 instead of https://host/gitea/api/v1.
Treat the final two path segments as owner/repo and keep leading segments as
a base-path prefix; derive the API base as {scheme}://{host}{prefix}/api/v1.
Root-hosted instances are unaffected. Forgejo inherits the fix; GitHub/GitLab
are untouched.
feat(#1239): first cut at Gitea backups silently failing after 1st run
feat(#1239): Added Token Scope for Forgejo edge case. Also included: test coverage for fixes
Subsequent backups against Gitea 1.24+ failed with the opaque
"Backup failed: 'tree'" message after the initial-backup fix landed in
7ee89b56. Root cause: Gitea's GET /repos/{owner}/{repo}/git/commits/{sha}
returns the wrapped Commit schema where the tree lives at
data["commit"]["tree"]["sha"], whereas GitHub's same-named Git Database
endpoint returns the unwrapped GitCommit schema with tree at the top
level. The bare commit_response.json()["tree"]["sha"] lookup at
gitea.py:109 raised KeyError: 'tree' and the broad except in push_files
surfaced it as the opaque "Backup failed: 'tree'" string — masking the
real shape mismatch.
Adds a _commit_tree_sha() helper that tries the flat shape first
(GitHub-compatible / older Gitea) and falls back to the wrapped shape
(Gitea 1.24+, Forgejo). Returns None on truly malformed responses;
push_files maps that to a clear "Failed to extract tree SHA from commit
response" instead of leaking a KeyError repr. Keeps the existing-files
diff working on both shapes so subsequent backups don't re-upload every
blob — preferred over the .get()-and-skip approach which would have
required also dropping base_tree from the tree POST and re-uploading
unchanged files on every backup.
Two interacting bugs in the Gitea/Forgejo backend, both inherited from
GitHubBackend because PR #1160 assumed Gitea's Git Data API was fully
GitHub-compatible. It isn't, on two specific points:
1. List-shaped ref response. Gitea/Forgejo's
GET /api/v1/repos/{owner}/{repo}/git/refs/heads/{branch} returns a
GET /api/v1/repos/{owner}/{repo}/git/refs/heads/{branch} returns a
list of matching refs even when only one matches; GitHub returns a
single object. The inherited push paths did
ref_response.json()["object"]["sha"] and crashed with
"list indices must be integers or slices, not str" against any
populated Gitea repo.
2. Empty-repo writes refused. GitHub accepts blob/tree/commit POSTs
against a brand-new empty repo and creates the initial commit
implicitly. Gitea refuses every blob POST with 404 until the repo
has at least one commit, so _create_initial_commit silently failed:
blobs returned 404, tree_items stayed empty, the tree POST then
also 404'd ("Failed to create tree").
Fix lives entirely in GiteaBackend — github.py is untouched so the
proven GitHub path takes zero risk. GiteaBackend now overrides
push_files, _create_branch_and_push, and _create_initial_commit:
- _ref_sha() helper accepts both list and dict shapes; called at the
two SHA extraction sites in push_files and _create_branch_and_push.
- _create_initial_commit posts to Gitea's Contents API
(POST /api/v1/repos/{owner}/{repo}/contents with a files array plus
branch + new_branch) which seeds the initial commit + branch in
one transaction and is documented to work on empty repos.
ForgejoBackend extends GiteaBackend with no overrides and inherits
both fixes; tests pin that.
jmoore-skild