RugMunchMedia/pryscraper
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
pryscraper/ROADMAP.md
cryptorugmunch bb77eb5f35 chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 
Re-license Pry from full Proprietary to a dual-license model:

- Core engine, extraction, templates (80+), MCP server, x402 payment rail,
  CLI, SDK, browser extension, WordPress plugin, Shopify app, and
  llm_providers: MIT (see LICENSE)
- Anti-detection / stealth subset (15 files): BSL 1.1 with Change Date
  2029-01-01 (see LICENSE-BSL-STEALTH)

BSL files (anti-detection moat):
  ultimate_scraper.py, stealth_engine.py, stealth_scripts/*.js (6),
  camoufox_integration.py, tls_fingerprint.py, cookie_warmer.py,
  behavioral_biometrics.py, adaptive.py, browser_pool.py, network.py,
  captcha_solver.py, shadow_dom.py, lazy_load.py, signup_automator.py,
  auth_connector.py

This enables community contributions to the core engine (templates,
integrations, MCP tools) while protecting the anti-detection techniques
that constitute the actual competitive moat. BSL Additional Use Grant
permits free non-production use; production deployment requires a
commercial license from enterprise@rugmunch.io.

Changes:
- Replace proprietary LICENSE with MIT LICENSE + new LICENSE-BSL-STEALTH
- Add SPDX-License-Identifier headers to 300+ source files
- Add docs/adr/0002-dual-licensing.md (ADR documenting the decision)
- Update README.md: new License section with BSL Additional Use Grant
- Update LICENSING_PRICING_STRATEGY.md: Section 3 (PryScraper) for dual license
- Update AGENTS.md: license line in header + new rule 8 (PRs touching BSL rejected)
- Update pyproject.toml: license = "MIT AND BSL-1.1" + classifiers + license-files
- Update DECISIONS.md index with ADR-0002
- Update STATUS.md (2026-07-03) and PLAN.md sprint goals

Refs: ADR-0002
2026-07-02 19:49:21 +02:00

5.5 KiB
Raw Blame History

//: # (Copyright (c) 2026 Rug Munch Media LLC)

title: Pry Roadmap & Status status: canonical owner: Rug Munch Media LLC Engineering last_updated: 2026-06-30

Pry Roadmap

Current Status: v3.0.0 — Active Development

Pry is a mature, feature-complete self-hosted web scraping platform. This roadmap tracks remaining gaps and planned improvements.

What's Built (✅ Complete)

Core Scraping Engine

  • ✅ 10-tier fallback scraper (direct → FlareSolverr → Playwright → Tor → cache)
  • ✅ Block detection with adaptive strategy rotation
  • ✅ Document parsing (PDF, DOCX, OCR, CSV, JSON)
  • ✅ Shadow DOM extraction
  • ✅ Lazy load / infinite scroll handling

Anti-Detection

  • ✅ Cloudflare bypass via FlareSolverr
  • ✅ undetected-chromedriver integration
  • ✅ Stealth engine (6 anti-detection JS scripts)
  • ✅ User-agent rotation
  • ✅ Tor proxy routing
  • ✅ SOCKS5 proxy support
  • ✅ Browser pool with pre-warming

API (115 endpoints)

  • ✅ REST API with 46 endpoint groups
  • ✅ Health probes (liveness, readiness)
  • ✅ Prometheus metrics
  • ✅ Rate limiting (token bucket)
  • ✅ Circuit breaker (per-domain)
  • ✅ WebSocket streaming
  • ✅ Async job queue
  • ✅ MCP server (Claude/Hermes/Cursor compatible)

Auth & Automation

  • ✅ CAPTCHA solver (6 providers, auto-fallback)
  • ✅ Credential vault (encrypted)
  • ✅ SSO login script generation
  • ✅ Session management (create, save, restore, close)
  • ✅ Browser action recorder
  • ✅ Signup automation
  • ✅ Account pool management

Extraction & Analysis

  • ✅ JSON schema extraction
  • ✅ LLM extraction with chunking strategies
  • ✅ Schema.org/JSON-LD extraction
  • ✅ Email extraction (Gmail, Outlook, raw)
  • ✅ AI summarization (Ollama)
  • ✅ AI categorization
  • ✅ Vision AI (OpenRouter, 5-model fallback)
  • ✅ SEO analysis (title, meta, headings, keywords, readability)
  • ✅ Tech stack detection
  • ✅ Field suggestion (AI)

Monitoring & Alerts

  • ✅ Page change detection (content fingerprinting)
  • ✅ Diff tracking between versions
  • ✅ Scheduled monitors (cron-based)
  • ✅ Content freshness dashboard
  • ✅ Structure change monitoring
  • ✅ Multi-channel alerts (webhook, Slack, email, SMS)

Data Pipeline

  • ✅ Pipeline definition, validation, and execution
  • ✅ Hook registration and execution
  • ✅ Data transformation (JSON, CSV, RSS, TXT, SQL)
  • ✅ Multi-format export
  • ✅ Webhook delivery, S3, GCS, SFTP

Quality & Review

  • ✅ Content quality scoring (completeness, accuracy, freshness)
  • ✅ Human review queue (submit, approve, reject)
  • ✅ Quality history tracking

Commerce & CRM

  • ✅ WooCommerce product sync
  • ✅ Shopify product sync (with Shopify app backend)
  • ✅ Salesforce CRM sync
  • ✅ HubSpot CRM sync
  • ✅ Zoho CRM sync

Compliance

  • ✅ GDPR consent management
  • ✅ Data retention policies
  • ✅ Data deletion (right to be forgotten)
  • ✅ Data portability (export)
  • ✅ Compliance audit log
  • ✅ Sensitive data detection
  • ✅ PII/copyright stripping for training data

Agency

  • ✅ White-label agency profiles
  • ✅ Client sub-account management
  • ✅ Usage analytics
  • ✅ Quota enforcement

Training Data

  • ✅ Dataset generation from scraped content
  • ✅ PII and copyright stripping
  • ✅ License classification
  • ✅ Dataset CRUD

Templates (110+)

  • ✅ Pre-built scraper templates for major sites
  • ✅ Template engine with parameterization
  • ✅ Template CRUD

Intelligence

  • ✅ Competitive intelligence snapshots
  • ✅ Competitor comparison/diff
  • ✅ Intel report generation

Integrations

  • ✅ Webhook support
  • ✅ Zapier integration
  • ✅ Make (formerly Integromat) integration
  • ✅ Browser extension (Chrome)
  • ✅ WordPress plugin
  • ✅ Shopify app
  • ✅ GPT Action manifest
  • ✅ MCP server config
  • ✅ CLI with autocomplete (bash, zsh, fish)

Infrastructure

  • ✅ Docker Compose deployment
  • ✅ Dockerfile
  • ✅ GitHub Actions CI/CD
  • ✅ Pre-commit hooks (ruff, mypy, bandit, gitleaks)
  • ✅ Makefile with standardized targets
  • ✅ pyproject.toml with ruff/mypy/pytest config

What's Remaining (🔜 Planned)

High Priority

  • Comprehensive test coverage > 80% (current: ~266 tests, ~40% coverage)
  • End-to-end integration tests with FlareSolverr + Playwright
  • OpenAPI specification cleanup and versioning
  • Type annotations across all modules (mypy strict)

Medium Priority

  • Redis-based distributed rate limiting
  • PostgreSQL-backed job queue (beyond filesystem)
  • Web dashboard (beyond current /dashboard health view)
  • API key authentication for multi-user deployments
  • OpenTelemetry tracing instrumentation
  • Prometheus metrics for all modules (currently only in api.py)

Low Priority

  • GraphQL API alternative
  • Webhook retry with exponential backoff
  • Template marketplace / sharing
  • i18n for multi-language content extraction
  • Mobile app (React Native) companion
  • Desktop app (Electron) companion

Audit Metrics (2026-06-30)

Metric Value
Version 3.0.0
Python source files 101
Total lines of Python 18,567
Test files 43
Total test functions 266
API endpoints 115
Endpoint groups (tags) 46
JSON scraper templates 110
HTML templates 2
Stealth JS scripts 6
Browser extension files 5
Shopify app files 3
WordPress plugin files 1
Config/support files 8
Total functions (all modules) 455
Total classes 48