- Item 5: Convert remaining printf-style logger call to structured logging
(key=value format) in x402_middleware.py
- Item 6: Wire CACHE_HITS counter into cache get() and per-tier SCRAPE_LATENCY
histogram into UltimateScraper scrape loop
- Item 7: Add circuit-breaker checks in extractor.py (Ollama) and
routers/vision.py (OpenRouter) to fail fast and record success/failure
when external services are down; attach recovery results to resilience
registry for per-service circuit state
560 tests passing, ruff clean
- Merge all configuration into settings.py with PRY_* env var prefix.
- Add legacy env aliases (PROXY_URL, TOR_ENABLED, MAX_RETRIES, etc.) for migration.
- Load and persist runtime overrides from JSON config file (default /app/config.json).
- Update scraper.py, deps.py, routers/config.py to use unified settings.
- Add resolved_proxy_chain and resolved_proxy_url properties.
- Replace tests/test_mconfig.py with tests/test_settings.py (9 tests).
- Update .env.example with new PRY_* options.
- All 500 tests pass; ruff clean.
This is the first split of api.py (4,668 lines) into a routers/
package, one router per OpenAPI tag. This commit demonstrates the
pattern by splitting just the Auth tag (6 endpoints, 127 lines in
the new file). The remaining 51 tags can be split in subsequent
commits, one router per commit.
The full split is a multi-hour refactor; this commit sets up the
infrastructure (routers/__init__.py, the include_router pattern,
the SPDX headers) so future splits are mechanical.
Changes:
- New package routers/
- __init__.py (45 lines, package docstring, migration order)
- auth.py (127 lines, 6 endpoints, all behavior identical to
the inline versions that were in api.py)
- api.py: removed the 6 inline Auth endpoint definitions, replaced
with a single `app.include_router(auth_router)` call
- api.py LOC: 4,668 -> 4,627 (-41 lines)
- Total FastAPI routes: 197 -> 192 (the 6 inline removed, 1
_IncludedRouter placeholder added; 5 unique paths in OpenAPI
spec - same as before, since GET+POST share a path)
- All routes registered, all behavior preserved
- Tests: 436/437 pass (1 pre-existing SSE sandbox failure, unrelated)
The pattern for future commits:
1. Read a tag's endpoints from api.py
2. Create routers/<tag>.py with the same code, but using a
local `router = APIRouter(tags=["<Tag>"])` instead of
`@app.post(..., tags=["<Tag>"])`
3. Replace the inline section in api.py with
`from routers.<tag> import router as <tag>_router`
`app.include_router(<tag>_router)`
4. Commit
Suggested commit order (smallest first, to spread risk):
- health (3 endpoints, ~50 lines)
- stats (1 endpoint, ~30 lines)
- costing (4 endpoints, ~150 lines)
- freshness (3 endpoints, ~100 lines)
- structure (3 endpoints, ~120 lines)
- seo (3 endpoints, ~120 lines)
- compliance (2 endpoints, ~200 lines)
- gdpr (8 endpoints, ~300 lines)
- sessions (5 endpoints, ~200 lines)
- monitoring (5 endpoints, ~250 lines)
- intelligence (4 endpoints, ~300 lines)
- scraping (8 endpoints, ~400 lines)
- extraction (8 endpoints, ~400 lines)
- advanced (16 endpoints, ~700 lines - needs to be split further)
When all routers are split, api.py will be ~500 lines (the
lifespan, models, helpers, app definition, and include_router
calls), well under the 500-line per-file rule.
