RugMunchMedia/pryscraper
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
pryscraper/ROADMAP.md
cryptorugmunch 47ba268131 docs: apply fleet-template (16-artifact scaffold) 
Adds missing standard artifacts:
- README.md (if missing)
- AGENTS.md (AI agent contract)
- PLAN.md (current sprint)
- STATUS.md (where we are)
- DEVELOPMENT.md (dev workflow)
- DEPLOYMENT.md (deploy procedure)
- TESTING.md (test strategy)
- DECISIONS.md (ADR index + templates)
- .github/CODEOWNERS
- .github/workflows/ci.yml

Preserves all existing artifacts.

Refs: RugMunchMedia/fleet-template
2026-07-02 02:07:13 +07:00

5.4 KiB
Raw Blame History

title status owner last_updated
Pry Roadmap & Status canonical Rug Munch Media LLC Engineering 2026-06-30

Pry Roadmap

Current Status: v3.0.0 — Active Development

Pry is a mature, feature-complete self-hosted web scraping platform. This roadmap tracks remaining gaps and planned improvements.

What's Built (✅ Complete)

Core Scraping Engine

  • ✅ 10-tier fallback scraper (direct → FlareSolverr → Playwright → Tor → cache)
  • ✅ Block detection with adaptive strategy rotation
  • ✅ Document parsing (PDF, DOCX, OCR, CSV, JSON)
  • ✅ Shadow DOM extraction
  • ✅ Lazy load / infinite scroll handling

Anti-Detection

  • ✅ Cloudflare bypass via FlareSolverr
  • ✅ undetected-chromedriver integration
  • ✅ Stealth engine (6 anti-detection JS scripts)
  • ✅ User-agent rotation
  • ✅ Tor proxy routing
  • ✅ SOCKS5 proxy support
  • ✅ Browser pool with pre-warming

API (115 endpoints)

  • ✅ REST API with 46 endpoint groups
  • ✅ Health probes (liveness, readiness)
  • ✅ Prometheus metrics
  • ✅ Rate limiting (token bucket)
  • ✅ Circuit breaker (per-domain)
  • ✅ WebSocket streaming
  • ✅ Async job queue
  • ✅ MCP server (Claude/Hermes/Cursor compatible)

Auth & Automation

  • ✅ CAPTCHA solver (6 providers, auto-fallback)
  • ✅ Credential vault (encrypted)
  • ✅ SSO login script generation
  • ✅ Session management (create, save, restore, close)
  • ✅ Browser action recorder
  • ✅ Signup automation
  • ✅ Account pool management

Extraction & Analysis

  • ✅ JSON schema extraction
  • ✅ LLM extraction with chunking strategies
  • ✅ Schema.org/JSON-LD extraction
  • ✅ Email extraction (Gmail, Outlook, raw)
  • ✅ AI summarization (Ollama)
  • ✅ AI categorization
  • ✅ Vision AI (OpenRouter, 5-model fallback)
  • ✅ SEO analysis (title, meta, headings, keywords, readability)
  • ✅ Tech stack detection
  • ✅ Field suggestion (AI)

Monitoring & Alerts

  • ✅ Page change detection (content fingerprinting)
  • ✅ Diff tracking between versions
  • ✅ Scheduled monitors (cron-based)
  • ✅ Content freshness dashboard
  • ✅ Structure change monitoring
  • ✅ Multi-channel alerts (webhook, Slack, email, SMS)

Data Pipeline

  • ✅ Pipeline definition, validation, and execution
  • ✅ Hook registration and execution
  • ✅ Data transformation (JSON, CSV, RSS, TXT, SQL)
  • ✅ Multi-format export
  • ✅ Webhook delivery, S3, GCS, SFTP

Quality & Review

  • ✅ Content quality scoring (completeness, accuracy, freshness)
  • ✅ Human review queue (submit, approve, reject)
  • ✅ Quality history tracking

Commerce & CRM

  • ✅ WooCommerce product sync
  • ✅ Shopify product sync (with Shopify app backend)
  • ✅ Salesforce CRM sync
  • ✅ HubSpot CRM sync
  • ✅ Zoho CRM sync

Compliance

  • ✅ GDPR consent management
  • ✅ Data retention policies
  • ✅ Data deletion (right to be forgotten)
  • ✅ Data portability (export)
  • ✅ Compliance audit log
  • ✅ Sensitive data detection
  • ✅ PII/copyright stripping for training data

Agency

  • ✅ White-label agency profiles
  • ✅ Client sub-account management
  • ✅ Usage analytics
  • ✅ Quota enforcement

Training Data

  • ✅ Dataset generation from scraped content
  • ✅ PII and copyright stripping
  • ✅ License classification
  • ✅ Dataset CRUD

Templates (110+)

  • ✅ Pre-built scraper templates for major sites
  • ✅ Template engine with parameterization
  • ✅ Template CRUD

Intelligence

  • ✅ Competitive intelligence snapshots
  • ✅ Competitor comparison/diff
  • ✅ Intel report generation

Integrations

  • ✅ Webhook support
  • ✅ Zapier integration
  • ✅ Make (formerly Integromat) integration
  • ✅ Browser extension (Chrome)
  • ✅ WordPress plugin
  • ✅ Shopify app
  • ✅ GPT Action manifest
  • ✅ MCP server config
  • ✅ CLI with autocomplete (bash, zsh, fish)

Infrastructure

  • ✅ Docker Compose deployment
  • ✅ Dockerfile
  • ✅ GitHub Actions CI/CD
  • ✅ Pre-commit hooks (ruff, mypy, bandit, gitleaks)
  • ✅ Makefile with standardized targets
  • ✅ pyproject.toml with ruff/mypy/pytest config

What's Remaining (🔜 Planned)

High Priority

  • Comprehensive test coverage > 80% (current: ~266 tests, ~40% coverage)
  • End-to-end integration tests with FlareSolverr + Playwright
  • OpenAPI specification cleanup and versioning
  • Type annotations across all modules (mypy strict)

Medium Priority

  • Redis-based distributed rate limiting
  • PostgreSQL-backed job queue (beyond filesystem)
  • Web dashboard (beyond current /dashboard health view)
  • API key authentication for multi-user deployments
  • OpenTelemetry tracing instrumentation
  • Prometheus metrics for all modules (currently only in api.py)

Low Priority

  • GraphQL API alternative
  • Webhook retry with exponential backoff
  • Template marketplace / sharing
  • i18n for multi-language content extraction
  • Mobile app (React Native) companion
  • Desktop app (Electron) companion

Audit Metrics (2026-06-30)

Metric Value
Version 3.0.0
Python source files 101
Total lines of Python 18,567
Test files 43
Total test functions 266
API endpoints 115
Endpoint groups (tags) 46
JSON scraper templates 110
HTML templates 2
Stealth JS scripts 6
Browser extension files 5
Shopify app files 3
WordPress plugin files 1
Config/support files 8
Total functions (all modules) 455
Total classes 48