mirror of
https://github.com/maziggy/bambuddy.git
synced 2026-09-30 03:01:21 +02:00
ci: balance the backend test shards by measured time, not test count
Backend Tests (shard 1/4) timed out after 10 minutes on e8b901f54
("Updated BACKERS"), a docs-only commit. The step was not hung: it
reached 98%, every test passing, and was killed about 12 seconds short
of finishing.
pytest-split balances by duration only when it has a durations file,
and there was none. Without one it splits by test COUNT -- 2926 / 2926
/ 2926 / 2925, exactly 25% each, which is what the comment here claimed
was good enough. Count is not time. Measured over the full suite
(11703 tests, 1045.6s):
shard 1 2926 tests 658.2s
shard 2 2926 tests 173.1s
shard 3 2926 tests 113.3s
shard 4 2925 tests 101.1s
Shard 1 was carrying 63% of the suite's runtime -- a 6.5x spread -- and
had been walking toward the cap for weeks: 475s, 404s, 445s, then 587s
on 08-30, thirteen seconds under it, and 613s here. CI printed the
reason in its own log on every run: "[pytest-split] No test durations
found."
Commit tests/.test_durations and pass --durations-path explicitly:
shard 1 1273 tests 261.6s
shard 2 1070 tests 261.5s
shard 3 1522 tests 262.0s
shard 4 7838 tests 260.6s
A 1.01x spread. The lopsided test counts are the point: shard 4 takes
thousands of fast unit tests, shard 1 keeps the slow integration ones.
The four groups still partition the suite exactly -- union is 11703
with nothing dropped or duplicated, and all four pass.
--durations-path has to be explicit because pytest-split defaults it to
$CWD/.test_durations, and the two matrices run from different
directories: the native job from backend/, the Docker job from the
image root. Left implicit, the Docker one finds nothing and silently
falls back to the count split. Node IDs match across both because
backend/tests/pytest.ini pins rootdir to backend/tests either way, so a
single file serves them both; verified the file clears .dockerignore
and lands at /app/backend/tests/.test_durations at full size.
timeout-minutes 10 -> 15 for headroom, since the file goes stale as
tests are added. Staleness degrades slowly rather than breaking --
unknown tests are treated as average.
This commit is contained in:
@@ -124,21 +124,35 @@ jobs:
|
||||
pip install -r requirements-dev.txt
|
||||
|
||||
- name: Run tests (shard ${{ matrix.shard }}/4)
|
||||
timeout-minutes: 10
|
||||
timeout-minutes: 15
|
||||
run: |
|
||||
cd backend
|
||||
# -v dropped: 5300+ "PASSED foo::bar" lines per worker eat 30-60s
|
||||
# of stdout I/O time on 2-vCPU runners. --tb=short is enough.
|
||||
# --splits 4 --group N uses pytest-split to slice the collected
|
||||
# test set roughly evenly across the 4 matrix shards; first run
|
||||
# is name-hash-based, subsequent runs improve via .test_durations
|
||||
# if you ever commit one (we don't — even the naive hash split
|
||||
# gets us ≈25% per shard given the test mix here).
|
||||
#
|
||||
# --splits 4 --group N slices the suite across the 4 matrix shards.
|
||||
# Without a durations file pytest-split balances by test COUNT, which
|
||||
# is not the same thing as time: measured 2026-08-31, that put 658s of
|
||||
# the suite's 1046s into shard 1 and 101s into shard 4 -- a 6.5x spread
|
||||
# that walked shard 1 into the old 10-minute timeout on a docs-only
|
||||
# commit. tests/.test_durations is committed so the split is balanced
|
||||
# by measured time instead (1.01x spread). Shard test counts are
|
||||
# lopsided by design once it is used: shard 4 takes thousands of fast
|
||||
# unit tests, shard 1 keeps the slow integration ones.
|
||||
#
|
||||
# --durations-path must be explicit: pytest-split defaults it to
|
||||
# $CWD/.test_durations, and this job runs from backend/ while the
|
||||
# Docker job below runs from the image root. Left implicit, one of
|
||||
# them silently finds no file and falls back to the count split.
|
||||
# Regenerate after adding slow tests:
|
||||
# cd backend && python -m pytest tests/ --store-durations \
|
||||
# --durations-path tests/.test_durations
|
||||
python -m pytest tests/ \
|
||||
--tb=short \
|
||||
--timeout=60 --timeout-method=thread \
|
||||
-n auto \
|
||||
--splits 4 --group ${{ matrix.shard }}
|
||||
--splits 4 --group ${{ matrix.shard }} \
|
||||
--durations-path tests/.test_durations
|
||||
|
||||
# ============================================================================
|
||||
# Frontend Checks
|
||||
@@ -365,7 +379,8 @@ jobs:
|
||||
--timeout=60 --timeout-method=thread \
|
||||
-p no:cacheprovider \
|
||||
-n auto \
|
||||
--splits 4 --group ${{ matrix.shard }}
|
||||
--splits 4 --group ${{ matrix.shard }} \
|
||||
--durations-path backend/tests/.test_durations
|
||||
|
||||
docker-test:
|
||||
name: Docker Build
|
||||
|
||||
+11
-1
@@ -31,7 +31,17 @@ repos:
|
||||
args: ['--maxkb=1000']
|
||||
# CHANGELOG.md is intentionally large (detailed per-release entries over
|
||||
# many versions); exempt it while keeping the 1 MB guard for everything else.
|
||||
exclude: ^(static/assets/|CHANGELOG\.md$)
|
||||
#
|
||||
# backend/tests/.test_durations is ~1.5 MB and cannot be made smaller:
|
||||
# its 11703 pytest node IDs are 1160 kB of keys before a single duration
|
||||
# value is written, so no encoding fits under the guard (compact JSON
|
||||
# with 3-decimal values still measures 1263 kB). Pruning fast tests is
|
||||
# worse than useless -- pytest-split gives a test missing from the file
|
||||
# the AVERAGE of the entries that remain, so dropped 1 ms tests come back
|
||||
# weighted at ~89 ms and the balance it exists to provide degrades.
|
||||
# It compresses to ~314 kB in the object store. Regenerate with
|
||||
# --store-durations; see the backend test job in .github/workflows/ci.yml.
|
||||
exclude: ^(static/assets/|CHANGELOG\.md$|backend/tests/\.test_durations$)
|
||||
- id: check-merge-conflict
|
||||
- id: debug-statements
|
||||
- id: detect-private-key
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user