cryptorugmunch 47ba268131 docs: apply fleet-template (16-artifact scaffold)

Adds missing standard artifacts:
- README.md (if missing)
- AGENTS.md (AI agent contract)
- PLAN.md (current sprint)
- STATUS.md (where we are)
- DEVELOPMENT.md (dev workflow)
- DEPLOYMENT.md (deploy procedure)
- TESTING.md (test strategy)
- DECISIONS.md (ADR index + templates)
- .github/CODEOWNERS
- .github/workflows/ci.yml

Preserves all existing artifacts.

Refs: RugMunchMedia/fleet-template

2026-07-02 02:07:13 +07:00

5.2 KiB

Raw Blame History

Pry — Open any website

Self-hosted web scraping + browser automation API. Cloudflare bypass, document parsing, AI summarization. No API keys needed.

Quickstart

# Install
pip install pry

# Start the server
pry serve

# Scrape a URL
pry open https://example.com

# With JSON extraction
pry open https://store.com/product --json --schema product.json

# Crawl a site
pry crawl https://docs.com --max-pages 20 -o data.json

Docker

docker compose up -d
# Pry on :8002, FlareSolverr on :8191

CLI Reference

Command	Description
`pry open <url>`	Scrape URL to clean markdown
`pry watch <url>`	Monitor page for changes
`pry crawl <url>`	Crawl multiple pages
`pry batch <file>`	Batch scrape URLs from file
`pry parse <url>`	Parse PDF/DOCX/image
`pry ss <url>`	Take screenshot
`pry run [pry.yml]`	Execute job file
`pry serve`	Start API server

API

`POST /v1/scrape`

{"url": "https://example.com", "bypassCloudflare": true, "formats": ["markdown"]}

`POST /v1/crawl`

{"url": "https://docs.com", "maxPages": 10, "maxDepth": 2}

`POST /v1/automate`

{"steps": [{"action": "navigate", "url": "https://...", {"action": "click", "selector": "#btn"}]]}

`POST /v1/vision`

Analyze images with free OpenRouter vision models. Auto-fallback across 5 models.

{"url": "https://...", "question": "What is shown?"}

`POST /v1/extract`

Extract structured data with JSON schema + optional AI fallback.

`POST /v1/batch`

Scrape up to 50 URLs in parallel.

`POST /v1/compare`

Diff two URLs.

`POST /v1/summarize`

AI summarization via local Ollama (free, private).

`GET /health`

Service health + cache stats + active sessions.

Python SDK

    from pry_sdk import PryCrawl

mc = PryCrawl("http://localhost:8002")
result = await mc.scrape("https://example.com")
print(result["data"]["markdown"])

    from pry_sdk import PryCrawlSync

mc = PryCrawlSync("http://localhost:8002")
result = mc.scrape("https://example.com")

SDK Methods

Method	Description
`scrape(url, **opts)`	Scrape to markdown
`scrape_json(url, schema, **opts)`	Scrape with JSON extraction
`crawl(url, max_pages, **opts)`	Crawl site
`map(url, limit)`	Discover URLs
`parse(url)`	Parse document
`automate(steps, **opts)`	Browser automation
`screenshot(url)`	Screenshot page
`health()`	Service health

MCP (AI Agent Integration)

POST /mcp/call
{"name": "pry_scrape", "arguments": {"url": "https://..."}}

Compatible with Claude, Hermes, Cursor, and any MCP client.

Features

4-tier anti-detection: Direct → FlareSolverr → Playwright → Googlebot
Cloudflare bypass: Automatic via FlareSolverr
Document parsing: PDF, DOCX, images (OCR), CSV, JSON
Browser automation: Login flows, form filling, sessions
Vision AI: Free OpenRouter vision models with auto-fallback
Diff tracking: Page change monitoring with webhooks
Batch processing: Parallel scrape with templates
SEO analysis: Title, meta, headings, keywords, readability
Schema extraction: JSON-LD, Open Graph, microdata
Export formats: JSON, CSV, RSS, TXT, SQL
Rate limiting: Per-IP token bucket (default 120 RPM)
Caching: LRU + Redis with TTL-based invalidation
WebSocket streaming: Real-time job progress
Circuit breaker: Per-domain backoff on failures

Architecture

┌─────────────┐  ┌──────────────┐  ┌──────────┐
│   Client    │→ │  Pry API     │→ │ Scraper  │
│ (CLI/SDK)   │  │  (FastAPI)   │  │  Engine  │
└─────────────┘  └──────┬───────┘  └────┬─────┘
                        │               ├─ Direct HTTP
                        │               ├─ FlareSolverr
                        │               ├─ Playwright
                        │               └─ Googlebot
                        │
                   ┌────┴────┐
                   │ Cache   │
                   │ Redis   │
                   │ Rate    │
                   └─────────┘

Deployment

Docker Compose (recommended)

docker compose up -d

Bare metal

pip install pry
playwright install chromium
# Optional: docker run -d -p 8191:8191 ghcr.io/flaresolverr/flaresolverr
pry serve

Environment

Variable	Default	Description
`PRY_URL`	`http://localhost:8005`	API endpoint for CLI
`TOR_ENABLED`	`false`	Enable Tor routing
`PROXY_URL`	—	HTTP/SOCKS proxy URL
`RATE_LIMIT_RPM`	`120`	Requests per minute
`OPENROUTER_API_KEY`	—	For vision endpoint

Development

make install      # Install with dev deps
make dev          # Start hot-reload server
make lint         # ruff check
make format       # ruff format
make typecheck    # mypy
make test         # pytest
make check        # Full audit

License

MIT — Rug Munch Media LLC

5.2 KiB Raw Blame History