Commit graph

10 commits

Author SHA1 Message Date
5a21133b1d feat(observability): structured logging, metrics wiring, graceful degradation
- Item 5: Convert remaining printf-style logger call to structured logging
  (key=value format) in x402_middleware.py
- Item 6: Wire CACHE_HITS counter into cache get() and per-tier SCRAPE_LATENCY
  histogram into UltimateScraper scrape loop
- Item 7: Add circuit-breaker checks in extractor.py (Ollama) and
  routers/vision.py (OpenRouter) to fail fast and record success/failure
  when external services are down; attach recovery results to resilience
  registry for per-service circuit state

560 tests passing, ruff clean
2026-07-03 12:13:03 +02:00
2970c15dbb feat(url_guard): SSRF and path traversal guard for scraper inputs
- Add url_guard.py with validate_url() that rejects:
  - Non-HTTP schemes (file://, ftp://, javascript:)
  - Private/reserved IP ranges (RFC 1918, loopback, CGNAT, link-local)
  - Internal hostnames (localhost, kubernetes, docker, metadata, etc.)
  - Embedded credentials in URLs
  - Path traversal sequences (.., %2e%2e%2f, %%2e%%2e%%2f)
- Integrate into scraper.scrape() and UltimateScraper.scrape()
- Use typed InvalidRequestError for structured API error responses
- Add 24 tests covering all rejection cases and valid URLs
- 560 tests passing, ruff clean
2026-07-03 12:00:40 +02:00
da31a1f9e7 feat(resilience): circuit breakers and per-domain tier memory 2026-07-03 11:49:33 +02:00
775f5412bf feat(scraper): wire orphaned anti-detection modules into fallback chain
- Integrate TLSScraper (curl_cffi TLS fingerprint impersonation) as Tier 2.
- Integrate CamoufoxBrowser as Tier 6 stealth Firefox alternative.
- Inject StealthEngine.generate_all() into Playwright contexts in both ultimate_scraper.py and scraper.py.
- Update UltimateScraper docstring to 12-tier fallback chain.
- Add 18 tests covering TLS, Camoufox, full fallthrough, and tier success paths.
- All 518 tests pass; ruff clean.
2026-07-03 11:28:29 +02:00
8b52f14774 feat(pry): phase 0 — split routers, add tests, apify schema, pry api key (#5)
Some checks failed
CI / Secret scan (gitleaks) (push) Successful in 34s
CI / Security audit (bandit) (push) Successful in 35s
CI / test (push) Successful in 1m23s
CI / lint (push) Failing after 49s
CI / typecheck (push) Successful in 55s
2026-07-03 03:43:02 +02:00
98eebe62bf fix(lint): resolve remaining ruff errors and unblock MCP SSE test (#1)
Some checks failed
CI / lint (push) Failing after 2s
CI / typecheck (push) Failing after 1s
CI / test (push) Failing after 1s
CI / Secret scan (gitleaks) (push) Failing after 2s
CI / Security audit (bandit) (push) Failing after 2s
2026-07-02 23:18:40 +02:00
a7c30b12cd chore(lint): auto-fix 253 of 283 ruff issues (F401, I001, E402, RUF100, UP037, SIM105)
Some checks failed
CI / lint (push) Failing after 2s
CI / typecheck (push) Failing after 2s
CI / test (push) Failing after 2s
CI / Secret scan (gitleaks) (push) Failing after 1s
CI / Security audit (bandit) (push) Failing after 2s
Mass ruff auto-fix:
  - ruff check --fix: 109 issues fixed (F401 unused imports,
    I001 unsorted imports, UP037 quoted annotations, SIM105
    suppressible exception, RUF100 unused-noqa)
  - ruff check --fix --unsafe-fixes: 22 additional issues
  - ruff format: 70 files reformatted
  - Manual pass: fix 16 misplaced import httpx lines
  - Manual pass: fix remaining E402 (import-after-docstring)

Result: 283 errors -> 30 errors.

The remaining 30 are real issues that need manual review:
  5 F401 unused-import (likely auto-generated stubs)
  5 F821 undefined-name (real bugs in code that references
    redis/pydantic/LLMRegistry without imports)
  3 BLE001 (the compliance LLM fallback is intentional; the
    other two are real)
  3 RUF012 mutable-class-default
  3 SIM105, 3 SIM117, 2 E722, 2 E741
  1 B007, 1 B025, 1 E402, 1 RUF200 (pyproject.toml issue)

Tests: 436/437 pass (1 pre-existing SSE sandbox failure).
format check + import sort: now clean.
make ci: still gated on the 30 remaining real issues.
Follow-up: triage the 30 issues file-by-file.
2026-07-02 21:51:25 +02:00
0200bf3e16 refactor(exceptions): add ruff BLE001; convert 103 broad except Exception
Per CONVENTIONS.md Part 2 ("Never bare except") and CONVENTIONS.md
Part 7 (pre-commit hooks: ruff), blind `except Exception` is now a
lint failure. Pre-existing sites are marked `# noqa: BLE001` for
later manual review; new code must use specific exception types.

Changes:
- pyproject.toml: added "BLE" to ruff lint select. BLE001 is now enforced
- 103 of 166 `except Exception` sites were auto-converted to specific
  types based on context (httpx, json, OSError, subprocess, etc.)
- 62 remaining sites marked with `# noqa: BLE001` for later review
  (mostly generic try/except wrappers that legitimately need broad catch
  for graceful degradation: e.g. compliance LLM fallback must catch
  any error to preserve the regex result)
- 1 manual fix: reverted compliance.py LLM fallback to broad except
  with explicit "must catch all errors" comment + noqa
- 2 files (commerce_sync.py, crm_sync.py) needed `import httpx` added
  so the auto-converted exception references would resolve
- 5 source files (agency, monitor, pipelines, auth_connector,
  llm_providers/registry) renamed "name" -> "<scope>_name" in
  extra={...} dicts because "name" is a reserved LogRecord field

Test impact:
- 14 failing tests -> 1 (the SSE subprocess test is a sandbox limitation,
  pre-existing and unrelated)
- New `test_ble_temp.py` verifies BLE001 catches new violations

Follow-up:
- Each `# noqa: BLE001` site should be reviewed and replaced with a
  specific exception type where possible. The most common legitimate
  broad-catch case is the LLM fallback path; everything else probably
  can be narrowed.
2026-07-02 21:04:53 +02:00
8d25702eca chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth)
Squashed from chore/license-relicense. Full message preserved in the
original branch commit bb77eb5. See ADR-0002 for the decision rationale.

Refs: ADR-0002, commit bb77eb5
2026-07-02 19:59:18 +02:00
47ba268131 docs: apply fleet-template (16-artifact scaffold)
Adds missing standard artifacts:
- README.md (if missing)
- AGENTS.md (AI agent contract)
- PLAN.md (current sprint)
- STATUS.md (where we are)
- DEVELOPMENT.md (dev workflow)
- DEPLOYMENT.md (deploy procedure)
- TESTING.md (test strategy)
- DECISIONS.md (ADR index + templates)
- .github/CODEOWNERS
- .github/workflows/ci.yml

Preserves all existing artifacts.

Refs: RugMunchMedia/fleet-template
2026-07-02 02:07:13 +07:00