pryscraper/README.md

//: # (Copyright (c) 2026 Rug Munch Media LLC)

Pry — Open any website

Self-hosted web scraping + browser automation API. Cloudflare bypass, document parsing, AI summarization. No API keys needed.

Quickstart

# Install
pip install pry

# Start the server
pry serve

# Scrape a URL
pry open https://example.com

# With JSON extraction
pry open https://store.com/product --json --schema product.json

# Crawl a site
pry crawl https://docs.com --max-pages 20 -o data.json

Docker

docker compose up -d
# Pry on :8002, FlareSolverr on :8191

CLI Reference

Command	Description
pry open <url>	Scrape URL to clean markdown
pry watch <url>	Monitor page for changes
pry crawl <url>	Crawl multiple pages
pry batch <file>	Batch scrape URLs from file
pry parse <url>	Parse PDF/DOCX/image
pry ss <url>	Take screenshot
pry run [pry.yml]	Execute job file
pry serve	Start API server

API

POST /v1/scrape

{"url": "https://example.com", "bypassCloudflare": true, "formats": ["markdown"]}

POST /v1/crawl

{"url": "https://docs.com", "maxPages": 10, "maxDepth": 2}

POST /v1/automate

{"steps": [{"action": "navigate", "url": "https://...", {"action": "click", "selector": "#btn"}]]}

POST /v1/vision

Analyze images with free OpenRouter vision models. Auto-fallback across 5 models.

{"url": "https://...", "question": "What is shown?"}

POST /v1/extract

Extract structured data with JSON schema + optional AI fallback.

POST /v1/batch

Scrape up to 50 URLs in parallel.

POST /v1/compare

Diff two URLs.

POST /v1/summarize

AI summarization via local Ollama (free, private).

GET /health

Service health + cache stats + active sessions.

Python SDK

    from pry_sdk import PryCrawl

mc = PryCrawl("http://localhost:8002")
result = await mc.scrape("https://example.com")
print(result["data"]["markdown"])

    from pry_sdk import PryCrawlSync

mc = PryCrawlSync("http://localhost:8002")
result = mc.scrape("https://example.com")

SDK Methods

Method	Description
scrape(url, **opts)	Scrape to markdown
scrape_json(url, schema, **opts)	Scrape with JSON extraction
crawl(url, max_pages, **opts)	Crawl site
map(url, limit)	Discover URLs
parse(url)	Parse document
automate(steps, **opts)	Browser automation
screenshot(url)	Screenshot page
health()	Service health

MCP (AI Agent Integration)

POST /mcp/call
{"name": "pry_scrape", "arguments": {"url": "https://..."}}

Compatible with Claude, Hermes, Cursor, and any MCP client.

Features

• 4-tier anti-detection: Direct → FlareSolverr → Playwright → Googlebot
• Cloudflare bypass: Automatic via FlareSolverr
• Document parsing: PDF, DOCX, images (OCR), CSV, JSON
• Browser automation: Login flows, form filling, sessions
• Vision AI: Free OpenRouter vision models with auto-fallback
• Diff tracking: Page change monitoring with webhooks
• Batch processing: Parallel scrape with templates
• SEO analysis: Title, meta, headings, keywords, readability
• Schema extraction: JSON-LD, Open Graph, microdata
• Export formats: JSON, CSV, RSS, TXT, SQL
• Rate limiting: Per-IP token bucket (default 120 RPM)
• Caching: LRU + Redis with TTL-based invalidation
• WebSocket streaming: Real-time job progress
• Circuit breaker: Per-domain backoff on failures

Architecture

┌─────────────┐  ┌──────────────┐  ┌──────────┐
│   Client    │→ │  Pry API     │→ │ Scraper  │
│ (CLI/SDK)   │  │  (FastAPI)   │  │  Engine  │
└─────────────┘  └──────┬───────┘  └────┬─────┘
                        │               ├─ Direct HTTP
                        │               ├─ FlareSolverr
                        │               ├─ Playwright
                        │               └─ Googlebot
                        │
                   ┌────┴────┐
                   │ Cache   │
                   │ Redis   │
                   │ Rate    │
                   └─────────┘

Deployment

Docker Compose (recommended)

docker compose up -d

Bare metal

pip install pry
playwright install chromium
# Optional: docker run -d -p 8191:8191 ghcr.io/flaresolverr/flaresolverr
pry serve

Environment

Variable	Default	Description
PRY_URL	http://localhost:8005	API endpoint for CLI
TOR_ENABLED	false	Enable Tor routing
PROXY_URL	—	HTTP/SOCKS proxy URL
RATE_LIMIT_RPM	120	Requests per minute
OPENROUTER_API_KEY	—	For vision endpoint

Development

make install      # Install with dev deps
make dev          # Start hot-reload server
make lint         # ruff check
make format       # ruff format
make typecheck    # mypy
make test         # pytest
make check        # Full audit

License

Pry is dual-licensed to balance community openness with the protection of its advanced anti-detection and stealth capabilities.

• Most code (core functionality, utilities, and standard modules) is licensed under the MIT License. This permissive license encourages community contributions and wide adoption. See LICENSE.

• The anti-detection / stealth module is licensed under the BSL 1.1 (Business Source License). This module is free to use for non-production purposes; a commercial license is required for any production use. See LICENSE-BSL-STEALTH.

Additional Use Grant

Under the BSL 1.1, an Additional Use Grant permits free use of the stealth module for:

• Personal, non-commercial projects
• Non-production testing and evaluation
• Academic research
• Non-commercial open source software development
• A 90-day evaluation period for any other purpose

For commercial licensing (production use, enterprise deployment, or embedding in proprietary products), please contact: enterprise@rugmunch.io