refactor(mcp): split mcp_production into pry_mcp sub-package #11

Merged

cryptorugmunch merged 11 commits from refactor/mcp-x402-split into main 2026-07-03 15:55:07 +02:00

Conversation 0 Commits 11 Files changed 51 +4820 -1577

cryptorugmunch commented 2026-07-03 15:40:18 +02:00

Owner

Copy link

Summary

Splits the monolithic mcp_production.py into a maintainable pry_mcp/ sub-package while keeping mcp_production.py as a thin backward-compatibility shim.

Changes

• Created pry_mcp/ sub-package (avoids collision with the installed mcp SDK)
• mcp_production.py is now a thin re-export shim
• Updated api.py, cli.py, mcp_sse.py, and tests to import from pry_mcp directly
• Added API-surface snapshot tests to guard against accidental breakage
• Style fixes in new tests (ruff clean)

Verification

• uv run pytest tests/ -q → 621 passed, 1 skipped
• uv run ruff check . → clean
• uv run python -m mypy . → success (no issues found in 229 source files)

Deferred

The x402.py split is intentionally left for a focused follow-up branch. It has tightly coupled globals (X402_WALLET, EIP7702_RPC, _batch_lock, _batch_payments, etc.) that need careful extraction.

## Summary Splits the monolithic `mcp_production.py` into a maintainable `pry_mcp/` sub-package while keeping `mcp_production.py` as a thin backward-compatibility shim. ## Changes - Created `pry_mcp/` sub-package (avoids collision with the installed `mcp` SDK) - `mcp_production.py` is now a thin re-export shim - Updated `api.py`, `cli.py`, `mcp_sse.py`, and tests to import from `pry_mcp` directly - Added API-surface snapshot tests to guard against accidental breakage - Style fixes in new tests (ruff clean) ## Verification - `uv run pytest tests/ -q` → 621 passed, 1 skipped - `uv run ruff check .` → clean - `uv run python -m mypy .` → success (no issues found in 229 source files) ## Deferred The `x402.py` split is intentionally left for a focused follow-up branch. It has tightly coupled globals (`X402_WALLET`, `EIP7702_RPC`, `_batch_lock`, `_batch_payments`, etc.) that need careful extraction.

cryptorugmunch added 10 commits 2026-07-03 15:40:19 +02:00

refactor(settings): unify config.py, mconfig.py, and settings.py into single PrySettings ... 1f9e71d294

- Merge all configuration into settings.py with PRY_* env var prefix.
- Add legacy env aliases (PROXY_URL, TOR_ENABLED, MAX_RETRIES, etc.) for migration.
- Load and persist runtime overrides from JSON config file (default /app/config.json).
- Update scraper.py, deps.py, routers/config.py to use unified settings.
- Add resolved_proxy_chain and resolved_proxy_url properties.
- Replace tests/test_mconfig.py with tests/test_settings.py (9 tests).
- Update .env.example with new PRY_* options.
- All 500 tests pass; ruff clean.

feat(scraper): wire orphaned anti-detection modules into fallback chain ... 775f5412bf

- Integrate TLSScraper (curl_cffi TLS fingerprint impersonation) as Tier 2.
- Integrate CamoufoxBrowser as Tier 6 stealth Firefox alternative.
- Inject StealthEngine.generate_all() into Playwright contexts in both ultimate_scraper.py and scraper.py.
- Update UltimateScraper docstring to 12-tier fallback chain.
- Add 18 tests covering TLS, Camoufox, full fallthrough, and tier success paths.
- All 518 tests pass; ruff clean.

feat(resilience): circuit breakers and per-domain tier memory da31a1f9e7

feat(url_guard): SSRF and path traversal guard for scraper inputs ... 2970c15dbb

- Add url_guard.py with validate_url() that rejects:
  - Non-HTTP schemes (file://, ftp://, javascript:)
  - Private/reserved IP ranges (RFC 1918, loopback, CGNAT, link-local)
  - Internal hostnames (localhost, kubernetes, docker, metadata, etc.)
  - Embedded credentials in URLs
  - Path traversal sequences (.., %2e%2e%2f, %%2e%%2e%%2f)
- Integrate into scraper.scrape() and UltimateScraper.scrape()
- Use typed InvalidRequestError for structured API error responses
- Add 24 tests covering all rejection cases and valid URLs
- 560 tests passing, ruff clean

feat(observability): structured logging, metrics wiring, graceful degradation ... 5a21133b1d

- Item 5: Convert remaining printf-style logger call to structured logging
  (key=value format) in x402_middleware.py
- Item 6: Wire CACHE_HITS counter into cache get() and per-tier SCRAPE_LATENCY
  histogram into UltimateScraper scrape loop
- Item 7: Add circuit-breaker checks in extractor.py (Ollama) and
  routers/vision.py (OpenRouter) to fail fast and record success/failure
  when external services are down; attach recovery results to resilience
  registry for per-service circuit state

560 tests passing, ruff clean

feat(pry): production readiness pass — apify actor, async db, retry wiring, tests, observability, mypy ... 345cd79bc9

- Pin Dockerfile --workers 1

- Wire retry.py + circuit breakers into ultimate_scraper tiers

- Add Apify actor (apify_actor.py, Dockerfile.apify, .actor/actor.json)

- Add async SQLAlchemy support alongside sync db.py

- Add 31 HTTP integration tests (tests/test_api_integration.py + test_api_mcp.py)

- Add OTLP exporter support in observability.py

- Re-enable mypy var-annotated error code; fix annotations

- Improve CI workflow (pip cache, install -e .[dev], gitleaks, commitlint, 40% coverage gate)

chore(mcp,x402): add API-surface snapshot tests before file split ... e0c7b67b2d

Inventory of public symbols in mcp_production.py and x402.py

Snapshot tests ensure backward-compatible re-export shims after split

refactor(mcp): split mcp_production.py into pry_mcp/ sub-package ... f7f9fa0c88

- pry_mcp/constants.py: protocol constants

- pry_mcp/notifications.py: observer pattern, logging handler, notification helpers

- pry_mcp/tools.py: PRY_TOOLS definitions

- pry_mcp/resources.py: PRY_RESOURCES and PRY_PROMPTS

- pry_mcp/fallback_server.py: FallbackMCPServer implementation

- pry_mcp/sdk_server.py: official MCP SDK path

- mcp_production.py: backward-compatible re-export shim

- Avoid namespace collision with installed mcp SDK by using pry_mcp/

refactor(mcp): modernize imports to use pry_mcp sub-package directly ... 4d2603cdd5

- api.py, cli.py, mcp_sse.py now import from pry_mcp

- tests import from pry_mcp (snapshot tests still verify mcp_production shim)

style(tests): fix ruff lint errors in new snapshot/fallback/retry tests a948639eca

cryptorugmunch added 1 commit 2026-07-03 15:51:13 +02:00

chore(ci): re-trigger Forgejo Actions workflow 55528f157d

cryptorugmunch merged commit 090f284c37 into main 2026-07-03 15:55:07 +02:00

cryptorugmunch referenced this pull request from a commit 2026-07-03 15:55:08 +02:00

Merge pull request 'refactor(mcp): split mcp_production into pry_mcp sub-package' (#11) from refactor/mcp-x402-split into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: RugMunchMedia/pryscraper#11

No description provided.