This is the first split of api.py (4,668 lines) into a routers/
package, one router per OpenAPI tag. This commit demonstrates the
pattern by splitting just the Auth tag (6 endpoints, 127 lines in
the new file). The remaining 51 tags can be split in subsequent
commits, one router per commit.
The full split is a multi-hour refactor; this commit sets up the
infrastructure (routers/__init__.py, the include_router pattern,
the SPDX headers) so future splits are mechanical.
Changes:
- New package routers/
- __init__.py (45 lines, package docstring, migration order)
- auth.py (127 lines, 6 endpoints, all behavior identical to
the inline versions that were in api.py)
- api.py: removed the 6 inline Auth endpoint definitions, replaced
with a single `app.include_router(auth_router)` call
- api.py LOC: 4,668 -> 4,627 (-41 lines)
- Total FastAPI routes: 197 -> 192 (the 6 inline removed, 1
_IncludedRouter placeholder added; 5 unique paths in OpenAPI
spec - same as before, since GET+POST share a path)
- All routes registered, all behavior preserved
- Tests: 436/437 pass (1 pre-existing SSE sandbox failure, unrelated)
The pattern for future commits:
1. Read a tag's endpoints from api.py
2. Create routers/<tag>.py with the same code, but using a
local `router = APIRouter(tags=["<Tag>"])` instead of
`@app.post(..., tags=["<Tag>"])`
3. Replace the inline section in api.py with
`from routers.<tag> import router as <tag>_router`
`app.include_router(<tag>_router)`
4. Commit
Suggested commit order (smallest first, to spread risk):
- health (3 endpoints, ~50 lines)
- stats (1 endpoint, ~30 lines)
- costing (4 endpoints, ~150 lines)
- freshness (3 endpoints, ~100 lines)
- structure (3 endpoints, ~120 lines)
- seo (3 endpoints, ~120 lines)
- compliance (2 endpoints, ~200 lines)
- gdpr (8 endpoints, ~300 lines)
- sessions (5 endpoints, ~200 lines)
- monitoring (5 endpoints, ~250 lines)
- intelligence (4 endpoints, ~300 lines)
- scraping (8 endpoints, ~400 lines)
- extraction (8 endpoints, ~400 lines)
- advanced (16 endpoints, ~700 lines - needs to be split further)
When all routers are split, api.py will be ~500 lines (the
lifespan, models, helpers, app definition, and include_router
calls), well under the 500-line per-file rule.
Per CONVENTIONS.md Part 2 ("Never bare except") and CONVENTIONS.md
Part 7 (pre-commit hooks: ruff), blind `except Exception` is now a
lint failure. Pre-existing sites are marked `# noqa: BLE001` for
later manual review; new code must use specific exception types.
Changes:
- pyproject.toml: added "BLE" to ruff lint select. BLE001 is now enforced
- 103 of 166 `except Exception` sites were auto-converted to specific
types based on context (httpx, json, OSError, subprocess, etc.)
- 62 remaining sites marked with `# noqa: BLE001` for later review
(mostly generic try/except wrappers that legitimately need broad catch
for graceful degradation: e.g. compliance LLM fallback must catch
any error to preserve the regex result)
- 1 manual fix: reverted compliance.py LLM fallback to broad except
with explicit "must catch all errors" comment + noqa
- 2 files (commerce_sync.py, crm_sync.py) needed `import httpx` added
so the auto-converted exception references would resolve
- 5 source files (agency, monitor, pipelines, auth_connector,
llm_providers/registry) renamed "name" -> "<scope>_name" in
extra={...} dicts because "name" is a reserved LogRecord field
Test impact:
- 14 failing tests -> 1 (the SSE subprocess test is a sandbox limitation,
pre-existing and unrelated)
- New `test_ble_temp.py` verifies BLE001 catches new violations
Follow-up:
- Each `# noqa: BLE001` site should be reviewed and replaced with a
specific exception type where possible. The most common legitimate
broad-catch case is the LLM fallback path; everything else probably
can be narrowed.
Pry logs are now JSON objects with the required fields (timestamp,
level, service, event, plus key-value pairs). This is the standard
required by CONVENTIONS.md Part 5 and is what makes the service
operable in production (Loki, ELK, etc. can index the structured
records).
New module logging_config.py:
setup_logging(level, fmt) - configure once at process startup
get_logger(name) - get a structlog logger; falls back to stdlib
is_configured() - diagnostic for /health
Configuration via env vars:
PRY_LOG_FORMAT=json|console (default json)
PRY_LOG_LEVEL=DEBUG|INFO|... (default INFO)
PRY_LOG_STRICT_EXTRAS=1 (default unset = lenient)
Backward compatibility:
- stdlib logging.getLogger(__name__) calls still work
- setup_logging bridges stdlib through structlog's formatter
- In lenient mode, extra={...} keys that collide with reserved
LogRecord names (e.g. 'name') are moved to an `extra` sub-dict
so existing code doesn't crash
Wired in:
api.py: setup_logging() at module import time; lifespan log uses
structlog style (logger.info("event", key="value") without
the `extra={...}` wrapper)
pyproject.toml: structlog>=24.0.0 dep added
Fixed source files that used reserved LogRecord keys in extra={...}:
agency.py: "name" -> "agency_name"
auth_connector.py: "name" -> "credential_name"
monitor.py: "name" -> "monitor_name"
pipelines.py: "name" -> "pipeline_name"
llm_providers/registry.py: "name" -> "provider_name"
These would have crashed with KeyError "Attempt to overwrite 'name' in
LogRecord" the moment a real log handler was attached.
Tests: 8/8 in test_logging_config.py pass. Full test suite went from
14 failures -> 2 (one is the SSE subprocess test that doesn't work in
this sandbox; one was the openapi title test that I also fixed in
this commit).
Documentation: DEVELOPMENT.md now has a full "Logging" section with
quick-start, config, and the reserved-key gotcha.
The deploy at /srv/pry/ had a thin proxy_referrals.py with 5 curated
proxy provider affiliate entries (Bright Data, Oxylabs, Smartproxy,
IPRoyal, Webshare). The repo was missing this file, so deploy and repo
were out of sync.
Changes:
- Add proxy_referrals.py (MIT) with the 5-provider curated catalog
- proxy_manager.py: import PROVIDER_REFERRALS, add 4 helper methods:
get_proxy_referral(tag)
get_proxy_referral_url(tag)
list_proxy_referrals()
get_proxy_referral_summary() - per-tier breakdown
- api.py: expose 2 new endpoints
GET /v1/proxy/referrals - full catalog + summary
GET /v1/proxy/referrals/{tag} - single provider
- 12/12 existing proxy_manager tests still pass
- Total routes: 195 -> 197
Squashed from chore/license-relicense. Full message preserved in the
original branch commit bb77eb5. See ADR-0002 for the decision rationale.
Refs: ADR-0002, commit bb77eb5
