Commit graph

9 commits

Author SHA1 Message Date
345cd79bc9 feat(pry): production readiness pass — apify actor, async db, retry wiring, tests, observability, mypy
- Pin Dockerfile --workers 1

- Wire retry.py + circuit breakers into ultimate_scraper tiers

- Add Apify actor (apify_actor.py, Dockerfile.apify, .actor/actor.json)

- Add async SQLAlchemy support alongside sync db.py

- Add 31 HTTP integration tests (tests/test_api_integration.py + test_api_mcp.py)

- Add OTLP exporter support in observability.py

- Re-enable mypy var-annotated error code; fix annotations

- Improve CI workflow (pip cache, install -e .[dev], gitleaks, commitlint, 40% coverage gate)
2026-07-03 14:41:41 +02:00
2970c15dbb feat(url_guard): SSRF and path traversal guard for scraper inputs
- Add url_guard.py with validate_url() that rejects:
  - Non-HTTP schemes (file://, ftp://, javascript:)
  - Private/reserved IP ranges (RFC 1918, loopback, CGNAT, link-local)
  - Internal hostnames (localhost, kubernetes, docker, metadata, etc.)
  - Embedded credentials in URLs
  - Path traversal sequences (.., %2e%2e%2f, %%2e%%2e%%2f)
- Integrate into scraper.scrape() and UltimateScraper.scrape()
- Use typed InvalidRequestError for structured API error responses
- Add 24 tests covering all rejection cases and valid URLs
- 560 tests passing, ruff clean
2026-07-03 12:00:40 +02:00
775f5412bf feat(scraper): wire orphaned anti-detection modules into fallback chain
- Integrate TLSScraper (curl_cffi TLS fingerprint impersonation) as Tier 2.
- Integrate CamoufoxBrowser as Tier 6 stealth Firefox alternative.
- Inject StealthEngine.generate_all() into Playwright contexts in both ultimate_scraper.py and scraper.py.
- Update UltimateScraper docstring to 12-tier fallback chain.
- Add 18 tests covering TLS, Camoufox, full fallthrough, and tier success paths.
- All 518 tests pass; ruff clean.
2026-07-03 11:28:29 +02:00
1f9e71d294 refactor(settings): unify config.py, mconfig.py, and settings.py into single PrySettings
- Merge all configuration into settings.py with PRY_* env var prefix.
- Add legacy env aliases (PROXY_URL, TOR_ENABLED, MAX_RETRIES, etc.) for migration.
- Load and persist runtime overrides from JSON config file (default /app/config.json).
- Update scraper.py, deps.py, routers/config.py to use unified settings.
- Add resolved_proxy_chain and resolved_proxy_url properties.
- Replace tests/test_mconfig.py with tests/test_settings.py (9 tests).
- Update .env.example with new PRY_* options.
- All 500 tests pass; ruff clean.
2026-07-03 05:04:29 +02:00
98eebe62bf fix(lint): resolve remaining ruff errors and unblock MCP SSE test (#1)
Some checks failed
CI / lint (push) Failing after 2s
CI / typecheck (push) Failing after 1s
CI / test (push) Failing after 1s
CI / Secret scan (gitleaks) (push) Failing after 2s
CI / Security audit (bandit) (push) Failing after 2s
2026-07-02 23:18:40 +02:00
a7c30b12cd chore(lint): auto-fix 253 of 283 ruff issues (F401, I001, E402, RUF100, UP037, SIM105)
Some checks failed
CI / lint (push) Failing after 2s
CI / typecheck (push) Failing after 2s
CI / test (push) Failing after 2s
CI / Secret scan (gitleaks) (push) Failing after 1s
CI / Security audit (bandit) (push) Failing after 2s
Mass ruff auto-fix:
  - ruff check --fix: 109 issues fixed (F401 unused imports,
    I001 unsorted imports, UP037 quoted annotations, SIM105
    suppressible exception, RUF100 unused-noqa)
  - ruff check --fix --unsafe-fixes: 22 additional issues
  - ruff format: 70 files reformatted
  - Manual pass: fix 16 misplaced import httpx lines
  - Manual pass: fix remaining E402 (import-after-docstring)

Result: 283 errors -> 30 errors.

The remaining 30 are real issues that need manual review:
  5 F401 unused-import (likely auto-generated stubs)
  5 F821 undefined-name (real bugs in code that references
    redis/pydantic/LLMRegistry without imports)
  3 BLE001 (the compliance LLM fallback is intentional; the
    other two are real)
  3 RUF012 mutable-class-default
  3 SIM105, 3 SIM117, 2 E722, 2 E741
  1 B007, 1 B025, 1 E402, 1 RUF200 (pyproject.toml issue)

Tests: 436/437 pass (1 pre-existing SSE sandbox failure).
format check + import sort: now clean.
make ci: still gated on the 30 remaining real issues.
Follow-up: triage the 30 issues file-by-file.
2026-07-02 21:51:25 +02:00
0200bf3e16 refactor(exceptions): add ruff BLE001; convert 103 broad except Exception
Per CONVENTIONS.md Part 2 ("Never bare except") and CONVENTIONS.md
Part 7 (pre-commit hooks: ruff), blind `except Exception` is now a
lint failure. Pre-existing sites are marked `# noqa: BLE001` for
later manual review; new code must use specific exception types.

Changes:
- pyproject.toml: added "BLE" to ruff lint select. BLE001 is now enforced
- 103 of 166 `except Exception` sites were auto-converted to specific
  types based on context (httpx, json, OSError, subprocess, etc.)
- 62 remaining sites marked with `# noqa: BLE001` for later review
  (mostly generic try/except wrappers that legitimately need broad catch
  for graceful degradation: e.g. compliance LLM fallback must catch
  any error to preserve the regex result)
- 1 manual fix: reverted compliance.py LLM fallback to broad except
  with explicit "must catch all errors" comment + noqa
- 2 files (commerce_sync.py, crm_sync.py) needed `import httpx` added
  so the auto-converted exception references would resolve
- 5 source files (agency, monitor, pipelines, auth_connector,
  llm_providers/registry) renamed "name" -> "<scope>_name" in
  extra={...} dicts because "name" is a reserved LogRecord field

Test impact:
- 14 failing tests -> 1 (the SSE subprocess test is a sandbox limitation,
  pre-existing and unrelated)
- New `test_ble_temp.py` verifies BLE001 catches new violations

Follow-up:
- Each `# noqa: BLE001` site should be reviewed and replaced with a
  specific exception type where possible. The most common legitimate
  broad-catch case is the LLM fallback path; everything else probably
  can be narrowed.
2026-07-02 21:04:53 +02:00
8d25702eca chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth)
Squashed from chore/license-relicense. Full message preserved in the
original branch commit bb77eb5. See ADR-0002 for the decision rationale.

Refs: ADR-0002, commit bb77eb5
2026-07-02 19:59:18 +02:00
47ba268131 docs: apply fleet-template (16-artifact scaffold)
Adds missing standard artifacts:
- README.md (if missing)
- AGENTS.md (AI agent contract)
- PLAN.md (current sprint)
- STATUS.md (where we are)
- DEVELOPMENT.md (dev workflow)
- DEPLOYMENT.md (deploy procedure)
- TESTING.md (test strategy)
- DECISIONS.md (ADR index + templates)
- .github/CODEOWNERS
- .github/workflows/ci.yml

Preserves all existing artifacts.

Refs: RugMunchMedia/fleet-template
2026-07-02 02:07:13 +07:00