RugMunchMedia/pryscraper
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

refactor(mcp): split mcp_production into pry_mcp sub-package #11

Merged
cryptorugmunch merged 11 commits from refactor/mcp-x402-split into main 2026-07-03 15:55:07 +02:00
Conversation 0 Commits 11 Files changed 51 +4820 -1577

11 commits

Author SHA1 Message Date
cryptorugmunch
55528f157d chore(ci): re-trigger Forgejo Actions workflow
Some checks are pending
CI / lint (pull_request) Waiting to run
Details
CI / typecheck (pull_request) Waiting to run
Details
CI / test (pull_request) Waiting to run
Details
CI / Secret scan (gitleaks) (pull_request) Waiting to run
Details
CI / Security audit (bandit) (pull_request) Waiting to run
Details
2026-07-03 15:51:10 +02:00
cryptorugmunch
a948639eca style(tests): fix ruff lint errors in new snapshot/fallback/retry tests
Some checks are pending
CI / lint (pull_request) Waiting to run
Details
CI / typecheck (pull_request) Waiting to run
Details
CI / test (pull_request) Waiting to run
Details
CI / Secret scan (gitleaks) (pull_request) Waiting to run
Details
CI / Security audit (bandit) (pull_request) Waiting to run
Details
2026-07-03 15:36:59 +02:00
cryptorugmunch
4d2603cdd5 refactor(mcp): modernize imports to use pry_mcp sub-package directly 
- api.py, cli.py, mcp_sse.py now import from pry_mcp

- tests import from pry_mcp (snapshot tests still verify mcp_production shim)
 2026-07-03 15:18:11 +02:00
cryptorugmunch
f7f9fa0c88 refactor(mcp): split mcp_production.py into pry_mcp/ sub-package 
- pry_mcp/constants.py: protocol constants

- pry_mcp/notifications.py: observer pattern, logging handler, notification helpers

- pry_mcp/tools.py: PRY_TOOLS definitions

- pry_mcp/resources.py: PRY_RESOURCES and PRY_PROMPTS

- pry_mcp/fallback_server.py: FallbackMCPServer implementation

- pry_mcp/sdk_server.py: official MCP SDK path

- mcp_production.py: backward-compatible re-export shim

- Avoid namespace collision with installed mcp SDK by using pry_mcp/
 2026-07-03 14:52:59 +02:00
cryptorugmunch
e0c7b67b2d chore(mcp,x402): add API-surface snapshot tests before file split 
Inventory of public symbols in mcp_production.py and x402.py

Snapshot tests ensure backward-compatible re-export shims after split
 2026-07-03 14:44:17 +02:00
cryptorugmunch
345cd79bc9 feat(pry): production readiness pass — apify actor, async db, retry wiring, tests, observability, mypy 
- Pin Dockerfile --workers 1

- Wire retry.py + circuit breakers into ultimate_scraper tiers

- Add Apify actor (apify_actor.py, Dockerfile.apify, .actor/actor.json)

- Add async SQLAlchemy support alongside sync db.py

- Add 31 HTTP integration tests (tests/test_api_integration.py + test_api_mcp.py)

- Add OTLP exporter support in observability.py

- Re-enable mypy var-annotated error code; fix annotations

- Improve CI workflow (pip cache, install -e .[dev], gitleaks, commitlint, 40% coverage gate)
 2026-07-03 14:41:41 +02:00
cryptorugmunch
5a21133b1d feat(observability): structured logging, metrics wiring, graceful degradation 
- Item 5: Convert remaining printf-style logger call to structured logging
  (key=value format) in x402_middleware.py
- Item 6: Wire CACHE_HITS counter into cache get() and per-tier SCRAPE_LATENCY
  histogram into UltimateScraper scrape loop
- Item 7: Add circuit-breaker checks in extractor.py (Ollama) and
  routers/vision.py (OpenRouter) to fail fast and record success/failure
  when external services are down; attach recovery results to resilience
  registry for per-service circuit state

560 tests passing, ruff clean
 2026-07-03 12:13:03 +02:00
cryptorugmunch
2970c15dbb feat(url_guard): SSRF and path traversal guard for scraper inputs 
- Add url_guard.py with validate_url() that rejects:
  - Non-HTTP schemes (file://, ftp://, javascript:)
  - Private/reserved IP ranges (RFC 1918, loopback, CGNAT, link-local)
  - Internal hostnames (localhost, kubernetes, docker, metadata, etc.)
  - Embedded credentials in URLs
  - Path traversal sequences (.., %2e%2e%2f, %%2e%%2e%%2f)
- Integrate into scraper.scrape() and UltimateScraper.scrape()
- Use typed InvalidRequestError for structured API error responses
- Add 24 tests covering all rejection cases and valid URLs
- 560 tests passing, ruff clean
 2026-07-03 12:00:40 +02:00
cryptorugmunch
da31a1f9e7 feat(resilience): circuit breakers and per-domain tier memory 2026-07-03 11:49:33 +02:00
cryptorugmunch
775f5412bf feat(scraper): wire orphaned anti-detection modules into fallback chain 
- Integrate TLSScraper (curl_cffi TLS fingerprint impersonation) as Tier 2.
- Integrate CamoufoxBrowser as Tier 6 stealth Firefox alternative.
- Inject StealthEngine.generate_all() into Playwright contexts in both ultimate_scraper.py and scraper.py.
- Update UltimateScraper docstring to 12-tier fallback chain.
- Add 18 tests covering TLS, Camoufox, full fallthrough, and tier success paths.
- All 518 tests pass; ruff clean.
 2026-07-03 11:28:29 +02:00
cryptorugmunch
1f9e71d294 refactor(settings): unify config.py, mconfig.py, and settings.py into single PrySettings 
- Merge all configuration into settings.py with PRY_* env var prefix.
- Add legacy env aliases (PROXY_URL, TOR_ENABLED, MAX_RETRIES, etc.) for migration.
- Load and persist runtime overrides from JSON config file (default /app/config.json).
- Update scraper.py, deps.py, routers/config.py to use unified settings.
- Add resolved_proxy_chain and resolved_proxy_url properties.
- Replace tests/test_mconfig.py with tests/test_settings.py (9 tests).
- Update .env.example with new PRY_* options.
- All 500 tests pass; ruff clean.
 2026-07-03 05:04:29 +02:00