fix(firmware-check): bypass Cloudflare TLS-fingerprint gate via curl_cffi (#1666)

Cloudflare on bambulab.com now serves cf-mitigated=challenge to plain
  Python TLS handshakes. Use curl_cffi.AsyncSession with impersonate="chrome"
  for the two bambulab.com fetches (index page + per-model JSON); wiki and
  CDN paths stay on httpx. HTTP User-Agent stays honest "Bambuddy/1.0" —
  only TLS-handshake bytes match Chrome, per the compliance commitment.

  Soft dependency — falls back to httpx with a startup warning when
  curl_cffi isn't importable; wiki-based version detection still works.
This commit is contained in:
maziggy
2026-06-06 13:30:13 +02:00
parent b362f159ac
commit 4bcab89eee
4 changed files with 230 additions and 99 deletions
+3
View File
@@ -4,6 +4,9 @@ All notable changes to Bambuddy will be documented in this file.
## [0.2.5b1] - Unreleased
### Fixed
- **Firmware-update check no longer 403s against Bambu Lab's Cloudflare-gated download page (#1666, reported by @arekm, with the working bypass demonstrated)** — Reporter on a fresh install hit `Could not reach Bambu Lab's firmware download page...` when checking firmware for an A1 Mini, and surfaced the diagnostic: `curl -H 'User-Agent: Bambuddy/1.0' https://bambulab.com/en/support/firmware-download/all` returns `HTTP 403 cf-mitigated=challenge` — Cloudflare upped the bot-protection on `bambulab.com` to a JA3 / TLS-fingerprint challenge. Plain Python TLS handshakes (httpx, requests, urllib) don't match Chrome's ClientHello bytes, so CF rejects before the request reaches the app layer. The `Accept` / `Accept-Language` header workaround we shipped for #1350 was below-HTTP and no longer enough. Existing users with a `build_id.json` on disk from a previous successful fetch kept working until Bambu rebuilt the page (every few weeks); fresh installs and wiped data dirs hit the wall immediately — exactly the reporter's path. **Fix: use `curl_cffi` for the two `bambulab.com` fetches only.** New dependency added to `requirements.txt`; `firmware_check.py` lazy-initialises a `curl_cffi.requests.AsyncSession(impersonate="chrome", ...)` for the `bambulab.com` calls (the index page that carries the Next.js `buildId`, and the per-model `_next/data/{buildId}/.../{api_key}.json` endpoint). Smoke-tested end-to-end against the live page: returns 200 OK + valid `buildId`, vs the reporter's 403. **Compliance framing matters here**: per the Bambu-compliance email from 2026-05-12, Bambuddy committed to "no falsified client identity." `curl_cffi`'s Chrome impersonation only governs TLS handshake bytes — the **HTTP-layer User-Agent is overridden back to `Bambuddy/1.0 (+https://github.com/maziggy/bambuddy)`** via the session's `headers=` parameter. Defensible read: TLS fingerprint matches Chrome (necessary because Python's TLS is the signal CF gates on), but every application-layer identity remains honestly Bambuddy. A new test (`test_bambulab_curl_cffi_session_keeps_honest_user_agent`) pins this — a future refactor that drops the `headers=` override would silently revert to curl_cffi's Chrome-default UA and break the compliance commitment; the test fails on any non-Bambuddy UA in the session. **Soft dependency**: if `curl_cffi` fails to import (rare platforms, alpine without wheels, etc.), the service logs a one-time warning at startup and falls back to httpx; wiki-based version detection continues to work for the badge, only the in-app firmware download URL stops resolving. New test `test_bambulab_get_falls_back_to_httpx_when_curl_cffi_missing` pins the fallback path. The wiki path (`wiki.bambulab.com`) and the CDN download path (`public-cdn.bblmw.com`) stay on httpx — neither sits behind the same JA3 gate. Three existing tests (`test_build_id_is_persisted_to_disk`, `test_build_id_falls_back_to_disk_on_403`, `test_download_page_unreachable_flag_set_on_403_json`, `test_download_page_retries_once_when_buildid_stale`) updated to mock `_bambulab_get` instead of the raw httpx client — a tighter mock target that's stable across the curl_cffi / httpx switch.
### Changed
- **Slicer sidecar now ships as pre-built images on GHCR + Docker Hub — install works on QNAP / Synology / Container Station (#1657, reported by @d3nn3s08)** — Reporter on QNAP QTS 5.2.9 hit three install failures in sequence: the official `slicer-api/docker-compose.yml` used `build: { context: https://github.com/maziggy/orca-slicer-api.git#bambuddy/profile-resolver }`, which requires `git` in the Docker BuildKit worker — Container Station and Synology DSM don't ship git there, so the build fails immediately with `exec: "git": executable file not found`. Manual ZIP-as-local-context workaround tripped a QNAP filesystem quirk in the systemd post-install (`Failed to copy permissions from /etc/group`). Fallback to `ghcr.io/afkfelix/orca-slicer-api:latest-orca2.3.0` ran but couldn't slice — that image lacks the `bambuddy/profile-resolver` patches (the `inherits:` chain resolver, the `from: "User"` → `"system"` rewrite, the `# ` clone-prefix strip, and the sentinel-value strip), so `/profiles/bundled` returned 400 and `/slice` returned `Invalid parameter value(s) included in the 3mf file`. **The fix removes the build-from-source requirement entirely.** Both sidecar images are now built locally on Martin's box and pushed to two registries (`ghcr.io/maziggy/orca-slicer-api`, `docker.io/maziggy/orca-slicer-api`, and the same two for `bambu-studio-api`) via a new `docker-publish-sidecars.sh` helper in the `orca-slicer-api` repo; the stable Bambuddy publish script auto-invokes it after each release, and the beta script too. Daily-beta opts in only via `--include-sidecars` (slicer rebuilds are expensive). The helper has hard safety guards: aborts unless the orca-slicer-api repo is on `bambuddy/profile-resolver` AND the working tree is clean, and never executes `git checkout` / `pull` / `fetch` / `reset` itself. `slicer-api/docker-compose.yml` switches from `build:` to `image: ghcr.io/maziggy/orca-slicer-api:${SIDECAR_TAG:-latest}`. New `SIDECAR_TAG` env var in `.env.example` defaults to `latest`; set `SIDECAR_TAG=bambuddy-X.Y.Z` to pin to the sidecar image that shipped with a specific Bambuddy release. **Scope limitation**: both images are `linux/amd64` only. The OrcaSlicer multi-arch path stays on hold pending an upstream extraction fix — the kldzj/orca-slicer-arm64 AppImage's `--appimage-extract` silently fails under QEMU build emulation; the Dockerfile's `;`-chained RUN block masked the failure until the final `COPY squashfs-root` tripped. ARM64 hosts (Pi 4/5, Apple Silicon Linux) should run the sidecar on a separate x86_64 box and point Bambuddy at it via the **Sidecar URL** field — the sidecar doesn't need to live next to Bambuddy. **Docs aligned**: `slicer-api/README.md` and `wiki/features/slicer-api.md` rewrote the Quick start, Updating, and Sidecar source sections — `docker compose up -d` now pulls instead of building, and `docker compose pull && docker compose up -d` is the new update path (no `--no-cache --pull` dance because Compose only ever sees `image:` references). The build-from-source path stays documented as an advanced option under "Building from source (advanced)" for forks / dev work.
+139 -70
View File
@@ -20,6 +20,26 @@ from backend.app.core.config import _data_dir
logger = logging.getLogger(__name__)
# Cloudflare on bambulab.com now gates the firmware-download page behind a
# JA3/TLS-fingerprint challenge (cf-mitigated=challenge) that plain Python
# TLS can't pass (#1666). curl_cffi replays Chrome's actual ClientHello
# bytes so the handshake clears CF; we override the HTTP User-Agent back to
# the honest Bambuddy/1.0 string so the application-layer identity stays
# truthful (TLS fingerprint matches Chrome because Python's TLS is the
# signal CF gates on; everything above TLS is still Bambuddy).
#
# Soft dependency — if curl_cffi isn't importable on the running platform,
# firmware_check degrades to httpx (which will likely 403) and wiki-based
# version detection continues to work for the badge; only the in-app
# firmware-download URL stops resolving.
try:
from curl_cffi.requests import AsyncSession as _CurlCffiAsyncSession
_CURL_CFFI_AVAILABLE = True
except ImportError: # pragma: no cover — exercised only on platforms without wheels
_CurlCffiAsyncSession = None # type: ignore[misc,assignment]
_CURL_CFFI_AVAILABLE = False
# Bambu Lab firmware download page (for download URLs)
BAMBU_FIRMWARE_BASE = "https://bambulab.com"
FIRMWARE_PAGE = "/en/support/firmware-download/all"
@@ -121,22 +141,71 @@ class FirmwareCheckService:
self._version_cache: dict[str, FirmwareVersion] = {}
self._versions_list_cache: dict[str, list[FirmwareVersion]] = {}
self._cache_time: float = 0
# Plain httpx client for the Bambu Lab wiki (no CF fingerprint check)
# and other endpoints. Honest UA throughout.
self._client = httpx.AsyncClient(
timeout=30.0,
headers={
# Identify honestly as Bambuddy when scraping the public Bambu
# Lab firmware wiki — verified 2026-05-12 that the wiki serves
# this UA identically to a Chrome UA (same HTML response shape).
# No browser impersonation needed for read-only public pages.
"User-Agent": "Bambuddy/1.0 (+https://github.com/maziggy/bambuddy)",
# Some Cloudflare bot rules on bambulab.com 403 requests with a
# bare UA but no browser-like Accept headers (seen on AU IPs in
# #1350). Sending normal Accept hints removes that signal while
# staying honestly identified via the UA above.
"Accept": "text/html,application/json,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
},
)
# curl_cffi session for bambulab.com (lazily initialised on first use).
# See module-level note on why this is needed.
self._bambulab_client: object | None = None
if not _CURL_CFFI_AVAILABLE:
logger.warning(
"curl_cffi not installed — bambulab.com firmware-download page "
"will likely return Cloudflare 403 (#1666). Wiki-based version "
"detection still works; install curl_cffi to also resolve "
"in-app firmware download URLs."
)
def _get_bambulab_client(self) -> object | None:
"""Lazy-init curl_cffi async session for bambulab.com.
Chrome TLS impersonation is required to pass Cloudflare's JA3
challenge. The HTTP `User-Agent` is overridden back to the honest
Bambuddy string so application-layer identity stays truthful.
Returns None when curl_cffi is unavailable.
"""
if not _CURL_CFFI_AVAILABLE:
return None
if self._bambulab_client is None:
assert _CurlCffiAsyncSession is not None # type-narrow for mypy
self._bambulab_client = _CurlCffiAsyncSession(
impersonate="chrome",
headers={
"User-Agent": "Bambuddy/1.0 (+https://github.com/maziggy/bambuddy)",
"Accept": "text/html,application/json,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
},
timeout=30,
)
return self._bambulab_client
async def _bambulab_get(self, url: str) -> tuple[int, str]:
"""GET against bambulab.com. Returns (status_code, body_text).
Routes through curl_cffi when available (passes the CF JA3 gate);
falls back to httpx otherwise (will likely 403 but worth the try
in case CF eases the check). status 0 indicates a transport error.
"""
client = self._get_bambulab_client()
if client is not None:
try:
response = await client.get(url) # type: ignore[attr-defined]
return response.status_code, response.text
except Exception as e:
logger.error("curl_cffi error fetching %s: %s", url, e)
return 0, ""
try:
response = await self._client.get(url)
return response.status_code, response.text
except Exception as e:
logger.error("httpx error fetching %s: %s", url, e)
return 0, ""
def _build_id_cache_path(self) -> Path:
cache_dir = _data_dir / "firmware"
@@ -188,28 +257,29 @@ class FirmwareCheckService:
self._build_id_time = disk_time
# 3. Live fetch
try:
response = await self._client.get(f"{BAMBU_FIRMWARE_BASE}{FIRMWARE_PAGE}")
if response.status_code == 200:
match = re.search(r'"buildId":"([^"]+)"', response.text)
if match:
new_build_id = match.group(1)
if new_build_id != self._build_id:
logger.info("Got Bambu Lab build ID: %s", new_build_id)
self._build_id = new_build_id
self._build_id_time = time.time()
self._download_page_unreachable = False
self._save_build_id_to_disk(new_build_id)
return self._build_id
else:
# 403/5xx — keep stale cached buildId if we have one (#1350).
logger.warning(
"Failed to get Bambu Lab page: %s (will try cached buildId if available)",
response.status_code,
)
self._download_page_unreachable = True
except Exception as e:
logger.error("Error fetching Bambu Lab build ID: %s", e)
status, body = await self._bambulab_get(f"{BAMBU_FIRMWARE_BASE}{FIRMWARE_PAGE}")
if status == 200:
match = re.search(r'"buildId":"([^"]+)"', body)
if match:
new_build_id = match.group(1)
if new_build_id != self._build_id:
logger.info("Got Bambu Lab build ID: %s", new_build_id)
self._build_id = new_build_id
self._build_id_time = time.time()
self._download_page_unreachable = False
self._save_build_id_to_disk(new_build_id)
return self._build_id
elif status == 0:
# Transport-level error already logged by _bambulab_get.
self._download_page_unreachable = True
else:
# 403/5xx — keep stale cached buildId if we have one (#1350).
# Without curl_cffi this is the expected outcome on Cloudflare
# JA3-gated zones (#1666).
logger.warning(
"Failed to get Bambu Lab page: %s (will try cached buildId if available)",
status,
)
self._download_page_unreachable = True
# Return whatever we have — even a stale buildId beats nothing.
@@ -302,50 +372,49 @@ class FirmwareCheckService:
return []
for attempt in range(2):
try:
url = f"{BAMBU_FIRMWARE_BASE}/_next/data/{build_id}/en/support/firmware-download/{api_key}.json"
response = await self._client.get(url)
url = f"{BAMBU_FIRMWARE_BASE}/_next/data/{build_id}/en/support/firmware-download/{api_key}.json"
status, body = await self._bambulab_get(url)
if response.status_code == 200:
data = response.json()
page_props = data.get("pageProps", {})
printer_map = page_props.get("printerMap", {})
printer_data = printer_map.get(api_key, {})
versions = printer_data.get("versions", [])
return [
FirmwareVersion(
version=v.get("version", ""),
download_url=v.get("url", ""),
release_notes=v.get("release_notes_en"),
release_time=v.get("release_time"),
)
for v in versions
if v.get("version")
]
if status == 200:
try:
data = json.loads(body)
except ValueError as e:
logger.debug("Download-page JSON for %s parse error: %s", api_key, e)
return []
page_props = data.get("pageProps", {})
printer_map = page_props.get("printerMap", {})
printer_data = printer_map.get(api_key, {})
versions = printer_data.get("versions", [])
return [
FirmwareVersion(
version=v.get("version", ""),
download_url=v.get("url", ""),
release_notes=v.get("release_notes_en"),
release_time=v.get("release_time"),
)
for v in versions
if v.get("version")
]
# 404 with cached buildId → Bambu rebuilt the page; invalidate
# and retry once. Other status codes (403, 5xx) are upstream
# blocks — don't churn.
if response.status_code == 404 and attempt == 0:
logger.info("Cached Bambu buildId stale (404), refreshing")
self._build_id = None
self._build_id_time = 0
build_id = await self._get_build_id()
if not build_id:
return []
continue
# 404 with cached buildId → Bambu rebuilt the page; invalidate
# and retry once. Other status codes (403, 5xx) are upstream
# blocks — don't churn.
if status == 404 and attempt == 0:
logger.info("Cached Bambu buildId stale (404), refreshing")
self._build_id = None
self._build_id_time = 0
build_id = await self._get_build_id()
if not build_id:
return []
continue
# 403 from the JSON endpoint is the same Cloudflare block
# signal as on the index page (#1350).
if response.status_code == 403:
self._download_page_unreachable = True
# 403 from the JSON endpoint is the same Cloudflare block
# signal as on the index page (#1350, #1666).
if status == 403:
self._download_page_unreachable = True
logger.debug("Download-page JSON for %s returned status %s", api_key, response.status_code)
return []
except Exception as e:
logger.debug("Error fetching download page firmware for %s: %s", api_key, e)
return []
logger.debug("Download-page JSON for %s returned status %s", api_key, status)
return []
return []
+77 -29
View File
@@ -164,9 +164,10 @@ async def test_get_available_versions_sorts_newest_first():
@pytest.mark.asyncio
async def test_client_headers_identify_honestly_and_send_browser_accept():
"""
The httpx client must identify as Bambuddy (no Chrome impersonation) and
must send Accept + Accept-Language so Cloudflare on bambulab.com doesn't
403 us for looking like a bare scraper (#1350).
The httpx client (used for the Bambu wiki and other non-CF-gated paths)
must identify as Bambuddy at the HTTP layer and must send Accept +
Accept-Language so Cloudflare on bambulab.com doesn't 403 us for
looking like a bare scraper (#1350).
"""
svc = FirmwareCheckService()
headers = svc._client.headers
@@ -176,17 +177,72 @@ async def test_client_headers_identify_honestly_and_send_browser_accept():
assert "Accept-Language" in headers
@pytest.mark.asyncio
async def test_bambulab_curl_cffi_session_keeps_honest_user_agent():
"""
The curl_cffi session impersonates Chrome at the TLS layer (required to
pass Cloudflare's JA3 challenge on bambulab.com per #1666), but the
HTTP-layer User-Agent MUST stay 'Bambuddy/...'. A future refactor that
drops the headers= override would silently revert to curl_cffi's
Chrome-default UA and break our compliance commitment to identify
honestly at the application layer.
Skipped when curl_cffi is not installed (constrained-platform fallback
path — covered by the import-guard test below).
"""
from backend.app.services import firmware_check as fc_module
if not fc_module._CURL_CFFI_AVAILABLE:
pytest.skip("curl_cffi not installed in this environment")
svc = FirmwareCheckService()
client = svc._get_bambulab_client()
assert client is not None
# curl_cffi's AsyncSession exposes the configured default headers on
# `.headers`. The exact attribute is part of curl_cffi's public API
# since 0.7.x — if a future upgrade renames it, this test will surface
# the rename rather than silently passing.
session_headers = client.headers # type: ignore[attr-defined]
ua = session_headers.get("User-Agent", "")
assert ua.startswith("Bambuddy/"), f"curl_cffi session UA leaked Chrome default: {ua!r}"
assert "Chrome" not in ua
assert "Mozilla" not in ua
@pytest.mark.asyncio
async def test_bambulab_get_falls_back_to_httpx_when_curl_cffi_missing(monkeypatch):
"""
When curl_cffi can't be imported (constrained platforms, alpine without
wheels, etc.), the service must still attempt the fetch via httpx so
wiki-based version detection isn't accidentally taken down with the
download-URL path. The httpx attempt will likely 403 against CF — that's
OK; the user still gets the version badge and a clear error in the UI.
"""
from backend.app.services import firmware_check as fc_module
monkeypatch.setattr(fc_module, "_CURL_CFFI_AVAILABLE", False)
monkeypatch.setattr(fc_module, "_CurlCffiAsyncSession", None)
svc = FirmwareCheckService()
mock_resp = MagicMock(status_code=403, text="<html>Forbidden</html>")
with patch.object(svc._client, "get", AsyncMock(return_value=mock_resp)):
status, body = await svc._bambulab_get("https://bambulab.com/whatever")
assert status == 403
assert "Forbidden" in body
@pytest.mark.asyncio
async def test_build_id_is_persisted_to_disk(tmp_path, monkeypatch):
"""Successful buildId fetch writes to disk so it survives restart (#1350)."""
monkeypatch.setattr("backend.app.services.firmware_check._data_dir", tmp_path)
svc = FirmwareCheckService()
mock_resp = MagicMock()
mock_resp.status_code = 200
mock_resp.text = 'window.__data = {"buildId":"abc123xyz","other":"stuff"}'
body = 'window.__data = {"buildId":"abc123xyz","other":"stuff"}'
with patch.object(svc._client, "get", AsyncMock(return_value=mock_resp)):
with patch.object(svc, "_bambulab_get", AsyncMock(return_value=(200, body))):
build_id = await svc._get_build_id()
assert build_id == "abc123xyz"
@@ -200,7 +256,7 @@ async def test_build_id_is_persisted_to_disk(tmp_path, monkeypatch):
@pytest.mark.asyncio
async def test_build_id_falls_back_to_disk_on_403(tmp_path, monkeypatch):
"""
When bambulab.com 403s (Cloudflare block reported in #1350) we must
When bambulab.com 403s (Cloudflare block reported in #1350, #1666) we must
fall back to the disk-cached buildId from the previous successful fetch.
Without this the user's screenshots happen: wiki version is detected
but the download URL stays empty forever.
@@ -213,11 +269,8 @@ async def test_build_id_falls_back_to_disk_on_403(tmp_path, monkeypatch):
(cache_dir / "build_id.json").write_text(json.dumps({"build_id": "cached_id_42", "fetched_at": 1000.0}))
svc = FirmwareCheckService()
mock_resp = MagicMock()
mock_resp.status_code = 403
mock_resp.text = "<html>Access denied</html>"
with patch.object(svc._client, "get", AsyncMock(return_value=mock_resp)):
with patch.object(svc, "_bambulab_get", AsyncMock(return_value=(403, "<html>Access denied</html>"))):
build_id = await svc._get_build_id()
assert build_id == "cached_id_42"
@@ -233,11 +286,7 @@ async def test_download_page_unreachable_flag_set_on_403_json(tmp_path, monkeypa
svc._build_id = "stale_id"
svc._build_id_time = 9999999999.0 # never expires for this test
mock_resp = MagicMock()
mock_resp.status_code = 403
mock_resp.text = "Forbidden"
with patch.object(svc._client, "get", AsyncMock(return_value=mock_resp)):
with patch.object(svc, "_bambulab_get", AsyncMock(return_value=(403, "Forbidden"))):
result = await svc._fetch_all_versions_from_download_page("x1")
assert result == []
@@ -256,12 +305,8 @@ async def test_download_page_retries_once_when_buildid_stale(tmp_path, monkeypat
svc._build_id = "stale_id"
svc._build_id_time = 9999999999.0
# First call (with stale buildId) → 404
# Second call (with fresh buildId after refresh) → 200 with versions
stale_resp = MagicMock(status_code=404, text="not found")
fresh_resp = MagicMock(status_code=200)
fresh_resp.json = MagicMock(
return_value={
fresh_json_body = json.dumps(
{
"pageProps": {
"printerMap": {
"x1": {
@@ -278,13 +323,16 @@ async def test_download_page_retries_once_when_buildid_stale(tmp_path, monkeypat
}
}
)
# Page fetch for the buildId refresh
page_resp = MagicMock(status_code=200)
page_resp.text = 'foo "buildId":"fresh_id" bar'
# Sequence: stale 404 → page refresh 200 → fresh 200
get = AsyncMock(side_effect=[stale_resp, page_resp, fresh_resp])
with patch.object(svc._client, "get", get):
# Sequence: stale JSON 404 → page refresh 200 (carries fresh buildId) → fresh JSON 200
get = AsyncMock(
side_effect=[
(404, "not found"),
(200, 'foo "buildId":"fresh_id" bar'),
(200, fresh_json_body),
]
)
with patch.object(svc, "_bambulab_get", get):
result = await svc._fetch_all_versions_from_download_page("x1")
assert len(result) == 1
+11
View File
@@ -79,6 +79,17 @@ idna>=3.15
# HTTP client (used for OIDC token exchange)
httpx>=0.26.0
# HTTP client with browser TLS-fingerprint impersonation. Used only for
# the bambulab.com firmware-download page in services/firmware_check.py:
# Bambu's Cloudflare WAF gates the page behind a JA3/TLS-fingerprint
# challenge that plain httpx/requests can't pass (#1666). curl_cffi
# replays Chrome's ClientHello bytes so the TLS handshake clears CF;
# the HTTP-layer User-Agent stays honest Bambuddy/1.0 per our compliance
# commitment. Soft dependency — if it fails to import (rare platforms,
# constrained installs), firmware_check degrades gracefully to wiki-only
# version detection and logs a warning at startup.
curl_cffi>=0.7.0
# Transitive pin: urllib3 2.6.3 has CVE-2026-44431 and CVE-2026-44432;
# 2.7.0+ is the fixed release. Direct pin here because none of our
# top-level deps require >=2.7.0 yet, so without this the resolver