cryptorugmunch bb77eb5f35 chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth)

Re-license Pry from full Proprietary to a dual-license model:

- Core engine, extraction, templates (80+), MCP server, x402 payment rail,
  CLI, SDK, browser extension, WordPress plugin, Shopify app, and
  llm_providers: MIT (see LICENSE)
- Anti-detection / stealth subset (15 files): BSL 1.1 with Change Date
  2029-01-01 (see LICENSE-BSL-STEALTH)

BSL files (anti-detection moat):
  ultimate_scraper.py, stealth_engine.py, stealth_scripts/*.js (6),
  camoufox_integration.py, tls_fingerprint.py, cookie_warmer.py,
  behavioral_biometrics.py, adaptive.py, browser_pool.py, network.py,
  captcha_solver.py, shadow_dom.py, lazy_load.py, signup_automator.py,
  auth_connector.py

This enables community contributions to the core engine (templates,
integrations, MCP tools) while protecting the anti-detection techniques
that constitute the actual competitive moat. BSL Additional Use Grant
permits free non-production use; production deployment requires a
commercial license from enterprise@rugmunch.io.

Changes:
- Replace proprietary LICENSE with MIT LICENSE + new LICENSE-BSL-STEALTH
- Add SPDX-License-Identifier headers to 300+ source files
- Add docs/adr/0002-dual-licensing.md (ADR documenting the decision)
- Update README.md: new License section with BSL Additional Use Grant
- Update LICENSING_PRICING_STRATEGY.md: Section 3 (PryScraper) for dual license
- Update AGENTS.md: license line in header + new rule 8 (PRs touching BSL rejected)
- Update pyproject.toml: license = "MIT AND BSL-1.1" + classifiers + license-files
- Update DECISIONS.md index with ADR-0002
- Update STATUS.md (2026-07-03) and PLAN.md sprint goals

Refs: ADR-0002

2026-07-02 19:49:21 +02:00

11 KiB

Raw Blame History

//: # (Copyright (c) 2026 Rug Munch Media LLC)

title: Pry Feature Catalog status: canonical owner: Rug Munch Media LLC Engineering last_updated: 2026-06-30

Pry Feature Catalog

Scraping & Crawling

Feature	Module	Endpoint	Status
Single URL scrape	`scraper.py`	`POST /v1/scrape`	✅
Multi-page crawl	`scraper.py`	`POST /v1/crawl`	✅
Batch scrape (parallel)	`advanced.py`	`POST /v1/batch`	✅
Batch from file + template	`advanced.py`	`POST /v1/batch-file`	✅
URL discovery (sitemap)	`scraper.py`	`POST /v1/map`	✅
Link analysis	`scraper.py`	`POST /v1/links`	✅

Anti-Detection & Bypass

Feature	Module	Endpoint	Status
10-tier fallback scraper	`ultimate_scraper.py`	Internal	✅
Block detection	`adaptive.py`	`POST /v1/detect-block`	✅
Adaptive strategy rotation	`adaptive.py`	Internal	✅
Browser pool + pre-warming	`browser_pool.py`	Internal	✅
Stealth engine (6 JS scripts)	`stealth_engine.py`	Internal	✅
Stealth scripts	`stealth_scripts/`	Injected	✅
Tor proxy routing	`network.py`	`POST /v1/config/profile/tor`	✅
SOCKS5 proxy support	`network.py`	Config	✅
User-agent rotation	`scraper.py`	Internal	✅

Document Parsing

Feature	Module	Endpoint	Status
PDF parsing	`parser.py`	`POST /v1/parse`	✅
DOCX parsing	`parser.py`	`POST /v1/parse`	✅
Image OCR	`parser.py`	`POST /v1/parse`	✅
CSV parsing	`parser.py`	`POST /v1/parse`	✅
JSON parsing	`parser.py`	`POST /v1/parse`	✅
Shadow DOM extraction	`shadow_dom.py`	`POST /v1/shadow-dom`	✅
Lazy load / infinite scroll	`lazy_load.py`	`POST /v1/capture/lazy`	✅

Extraction & Structured Data

Feature	Module	Endpoint	Status
JSON schema extraction	`extractor.py`	`POST /v1/extract`	✅
LLM extraction + chunking	`extraction.py`	`POST /v1/extract/llm`	✅
Schema.org / JSON-LD	`extractor.py`	`POST /v1/schema`	✅
Email address extraction	`extractor.py`	`POST /v1/emails`	✅
Structured field extraction	`extraction.py`	`POST /v1/extract/fields`	✅
Field suggestion (AI)	`advanced.py`	`POST /v1/suggest`	✅

AI & Vision

Feature	Module	Endpoint	Status
Vision AI (OpenRouter, 5-model fallback)	`advanced.py`	`POST /v1/vision`	✅
AI summarization (Ollama)	`advanced.py`	`POST /v1/summarize`	✅
AI categorization	`advanced.py`	`POST /v1/categorize`	✅
GPT Action manifest	`ai_plugin.py`	`GET /v1/ai/gpt-manifest`	✅
MCP server config	`ai_plugin.py`	`GET /v1/ai/mcp-config`	✅
MCP tool discovery	`mcp_server.py`	`GET /mcp/tools`	✅
MCP tool execution	`mcp_server.py`	`POST /mcp/call`	✅

Browser Automation

Feature	Module	Endpoint	Status
Step-based automation	`automator.py`	`POST /v1/automate`	✅
Screenshot capture	`automator.py`	`POST /v1/screenshot`	✅
Persistent browser sessions	`sessions.py`	`POST /v1/session/create`	✅
Session list	`sessions.py`	`GET /v1/sessions`	✅
Session save/restore	`sessions.py`	`POST /v1/session/save`	✅
Session close	`sessions.py`	`POST /v1/session/close`	✅
Action recorder	`advanced.py`	`POST /v1/record/start`	✅

CAPTCHA Solving

Feature	Module	Endpoint	Status
CAPTCHA solving (6 providers)	`captcha_solver.py`	`POST /v1/auth/captcha`	✅
Auto-fallback across providers	`captcha_solver.py`	Internal	✅

Authentication

Feature	Module	Endpoint	Status
Credential vault (encrypted)	`auth_connector.py`	`POST /v1/auth/credentials`	✅
SSO login script generation	`auth_connector.py`	`POST /v1/auth/sso`	✅
Session health check	`auth_connector.py`	`POST /v1/auth/session/health`	✅
Signup automation	`signup_automator.py`	Internal	✅
Account pool management	`account_manager.py`	Internal	✅

Monitoring & Change Detection

Feature	Module	Endpoint	Status
Page watch / change detection	`freshness.py`	`POST /v1/watch`	✅
Diff tracking	`advanced.py`	`POST /v1/diff`	✅
Content freshness dashboard	`freshness.py`	`GET /v1/freshness/dashboard`	✅
Scheduled monitors (cron)	`monitor.py`	`POST /v1/monitor`	✅
Monitor run on-demand	`monitor.py`	`POST /v1/monitor/run`	✅
Monitor list / delete	`monitor.py`	`GET /v1/monitors`	✅
Structure monitoring	`structure_monitor.py`	`POST /v1/structure/check`	✅

Alerts & Notifications

Feature	Module	Endpoint	Status
Multi-channel alerts	`alerter.py`	`POST /v1/alert/send`	✅
Supported channel listing	`alerter.py`	`GET /v1/alert/channels`	✅
Webhook support	`destinations.py`	Config	✅
Slack integration	`destinations.py`	Config	✅

Quality & Review

Feature	Module	Endpoint	Status
Content quality scoring	`quality.py`	`POST /v1/quality/check`	✅
Quality history	`quality.py`	`GET /v1/quality/history`	✅
Human review queue	`review.py`	`POST /v1/review/submit`	✅
Review approve/reject	`review.py`	`POST /v1/review/{id}/approve`	✅
Review queue listing	`review.py`	`GET /v1/reviews`	✅

Data Export & Integration

Feature	Module	Endpoint	Status
Multi-format export (JSON, CSV, RSS, TXT, SQL)	`destinations.py`	`POST /v1/export`	✅
Webhook delivery	`destinations.py`	Config	✅
S3 upload	`destinations.py`	Config	✅
GCS upload	`destinations.py`	Config	✅
SFTP upload	`destinations.py`	Config	✅
Data transformation	`advanced.py`	`POST /v1/transform`	✅

Commerce & CRM

Feature	Module	Endpoint	Status
WooCommerce product sync	`commerce_sync.py`	`POST /v1/commerce/sync`	✅
Shopify product sync	`commerce_sync.py`	`POST /v1/commerce/sync`	✅
Shopify app backend	`shopify-app/`	External	✅
Salesforce CRM sync	`crm_sync.py`	`POST /v1/crm/sync`	✅
HubSpot CRM sync	`crm_sync.py`	`POST /v1/crm/sync`	✅
Zoho CRM sync	`crm_sync.py`	`POST /v1/crm/sync`	✅

Pipelines

Feature	Module	Endpoint	Status
Pipeline definition validation	`pipelines.py`	`POST /v1/pipelines/validate`	✅
Pipeline execution	`pipelines.py`	`POST /v1/pipelines/run`	✅
Pipeline CRUD	`pipelines.py`	CRUD endpoints	✅
Hook registration	`pipeline.py`	`POST /v1/pipeline/hook`	✅
Hook execution	`pipeline.py`	`POST /v1/pipeline/run`	✅
Schema pipeline	`pipelines.py`	`POST /v1/pipelines/schema`	✅

Intelligence & Competitive Analysis

Feature	Module	Endpoint	Status
Competitor snapshot	`intelligence.py`	`POST /v1/intel/snapshot`	✅
Intel comparison (diff)	`intelligence.py`	`POST /v1/intel/compare`	✅
Intel report generation	`intelligence.py`	`GET /v1/intel/report`	✅
Tech stack detection	`enrichment.py`	`POST /v1/enrichment/analyze`	✅

Feature	Module	Endpoint	Status
Compliance check (URL)	`compliance.py`	`POST /v1/compliance/check`	✅
Consent recording	`gdpr.py`	`POST /v1/gdpr/consent`	✅
Consent check / revoke	`gdpr.py`	`GET /v1/gdpr/consent/{id}`	✅
Data deletion request	`gdpr.py`	`POST /v1/gdpr/deletion/request`	✅
Deletion execution	`gdpr.py`	`POST /v1/gdpr/deletion/execute`	✅
Data retention policy	`gdpr.py`	`GET /v1/gdpr/retention`	✅
Data portability (export)	`gdpr.py`	`POST /v1/gdpr/export`	✅
Compliance audit log	`gdpr.py`	`GET /v1/gdpr/audit`	✅

Training Data

Feature	Module	Endpoint	Status
PII/copyright stripping	`training_data.py`	`POST /v1/training/clean`	✅
Dataset generation	`training_data.py`	`POST /v1/training/dataset`	✅
Dataset CRUD	`training_data.py`	CRUD endpoints	✅
License classification	`training_data.py`	Internal	✅

Agency / White-Label

Feature	Module	Endpoint	Status
Agency profile creation	`agency.py`	`POST /v1/agency/create`	✅
Agency branding	`agency.py`	`PUT /v1/agency/{id}/branding`	✅
Client sub-account management	`agency.py`	`POST /v1/agency/{id}/clients`	✅
Usage analytics	`agency.py`	`GET /v1/agency/{id}/analytics`	✅
Client quota enforcement	`agency.py`	`GET /v1/client/{id}/quota`	✅

Reporting

Feature	Module	Endpoint	Status
Report generation	`reports.py`	`POST /v1/reports/generate`	✅
Report list / get	`reports.py`	`GET /v1/reports`	✅

Templates

Feature	Module	Endpoint	Status
Template listing	`template_engine.py`	`GET /v1/templates`	✅
Template detail	`template_engine.py`	`GET /v1/templates/{id}`	✅
Template generation	`template_engine.py`	`POST /v1/templates/generate`	✅
Pre-built templates	110 JSON files	`templates/`	✅

SEO

Feature	Module	Endpoint	Status
SEO analysis	`seo_monitor.py`	`POST /v1/seo/analyze`	✅
SEO change tracking	`seo_monitor.py`	`POST /v1/seo/track`	✅
Keyword analysis	`seo_monitor.py`	`POST /v1/seo/keywords`	✅

Infrastructure

Feature	Module	Status
Health probes (liveness/readiness)	`api.py`	✅
Prometheus metrics	`api.py`	✅
Rate limiting (token bucket)	`ratelimit.py`	✅
LRU cache + Redis	`cache.py`	✅
Circuit breaker per-domain	`advanced.py`	✅
WebSocket streaming	`pryextras.py`	✅
Async job queue	`jobqueue.py`	✅
Docker Compose deployment	`docker-compose.yml`	✅
CI/CD (GitHub Actions)	`.github/workflows/ci.yml`	✅
Pre-commit hooks	`.pre-commit-config.yaml`	✅
Browser extension (Chrome)	`browser-extension/`	✅
WordPress plugin	`wordpress-plugin/`	✅

11 KiB Raw Blame History

//: # (Copyright (c) 2026 Rug Munch Media LLC)

title: Pry Feature Catalog status: canonical owner: Rug Munch Media LLC Engineering last_updated: 2026-06-30

Pry Feature Catalog

Scraping & Crawling

Anti-Detection & Bypass

Document Parsing

Extraction & Structured Data

AI & Vision

Browser Automation

CAPTCHA Solving

Authentication

Monitoring & Change Detection

Alerts & Notifications

Quality & Review

Data Export & Integration

Commerce & CRM

Pipelines

Intelligence & Competitive Analysis

Compliance & GDPR

Training Data

Agency / White-Label

Reporting

Templates

SEO

Infrastructure

11 KiB

Raw Blame History