docs: apply fleet-template (16-artifact scaffold)
Adds missing standard artifacts: - README.md (if missing) - AGENTS.md (AI agent contract) - PLAN.md (current sprint) - STATUS.md (where we are) - DEVELOPMENT.md (dev workflow) - DEPLOYMENT.md (deploy procedure) - TESTING.md (test strategy) - DECISIONS.md (ADR index + templates) - .github/CODEOWNERS - .github/workflows/ci.yml Preserves all existing artifacts. Refs: RugMunchMedia/fleet-template
This commit is contained in:
commit
47ba268131
310 changed files with 38429 additions and 0 deletions
183
ROADMAP.md
Normal file
183
ROADMAP.md
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
---
|
||||
title: Pry Roadmap & Status
|
||||
status: canonical
|
||||
owner: Rug Munch Media LLC Engineering
|
||||
last_updated: 2026-06-30
|
||||
---
|
||||
|
||||
# Pry Roadmap
|
||||
|
||||
## Current Status: v3.0.0 — Active Development
|
||||
|
||||
Pry is a mature, feature-complete self-hosted web scraping platform. This roadmap tracks remaining gaps and planned improvements.
|
||||
|
||||
## What's Built (✅ Complete)
|
||||
|
||||
### Core Scraping Engine
|
||||
- ✅ 10-tier fallback scraper (direct → FlareSolverr → Playwright → Tor → cache)
|
||||
- ✅ Block detection with adaptive strategy rotation
|
||||
- ✅ Document parsing (PDF, DOCX, OCR, CSV, JSON)
|
||||
- ✅ Shadow DOM extraction
|
||||
- ✅ Lazy load / infinite scroll handling
|
||||
|
||||
### Anti-Detection
|
||||
- ✅ Cloudflare bypass via FlareSolverr
|
||||
- ✅ undetected-chromedriver integration
|
||||
- ✅ Stealth engine (6 anti-detection JS scripts)
|
||||
- ✅ User-agent rotation
|
||||
- ✅ Tor proxy routing
|
||||
- ✅ SOCKS5 proxy support
|
||||
- ✅ Browser pool with pre-warming
|
||||
|
||||
### API (115 endpoints)
|
||||
- ✅ REST API with 46 endpoint groups
|
||||
- ✅ Health probes (liveness, readiness)
|
||||
- ✅ Prometheus metrics
|
||||
- ✅ Rate limiting (token bucket)
|
||||
- ✅ Circuit breaker (per-domain)
|
||||
- ✅ WebSocket streaming
|
||||
- ✅ Async job queue
|
||||
- ✅ MCP server (Claude/Hermes/Cursor compatible)
|
||||
|
||||
### Auth & Automation
|
||||
- ✅ CAPTCHA solver (6 providers, auto-fallback)
|
||||
- ✅ Credential vault (encrypted)
|
||||
- ✅ SSO login script generation
|
||||
- ✅ Session management (create, save, restore, close)
|
||||
- ✅ Browser action recorder
|
||||
- ✅ Signup automation
|
||||
- ✅ Account pool management
|
||||
|
||||
### Extraction & Analysis
|
||||
- ✅ JSON schema extraction
|
||||
- ✅ LLM extraction with chunking strategies
|
||||
- ✅ Schema.org/JSON-LD extraction
|
||||
- ✅ Email extraction (Gmail, Outlook, raw)
|
||||
- ✅ AI summarization (Ollama)
|
||||
- ✅ AI categorization
|
||||
- ✅ Vision AI (OpenRouter, 5-model fallback)
|
||||
- ✅ SEO analysis (title, meta, headings, keywords, readability)
|
||||
- ✅ Tech stack detection
|
||||
- ✅ Field suggestion (AI)
|
||||
|
||||
### Monitoring & Alerts
|
||||
- ✅ Page change detection (content fingerprinting)
|
||||
- ✅ Diff tracking between versions
|
||||
- ✅ Scheduled monitors (cron-based)
|
||||
- ✅ Content freshness dashboard
|
||||
- ✅ Structure change monitoring
|
||||
- ✅ Multi-channel alerts (webhook, Slack, email, SMS)
|
||||
|
||||
### Data Pipeline
|
||||
- ✅ Pipeline definition, validation, and execution
|
||||
- ✅ Hook registration and execution
|
||||
- ✅ Data transformation (JSON, CSV, RSS, TXT, SQL)
|
||||
- ✅ Multi-format export
|
||||
- ✅ Webhook delivery, S3, GCS, SFTP
|
||||
|
||||
### Quality & Review
|
||||
- ✅ Content quality scoring (completeness, accuracy, freshness)
|
||||
- ✅ Human review queue (submit, approve, reject)
|
||||
- ✅ Quality history tracking
|
||||
|
||||
### Commerce & CRM
|
||||
- ✅ WooCommerce product sync
|
||||
- ✅ Shopify product sync (with Shopify app backend)
|
||||
- ✅ Salesforce CRM sync
|
||||
- ✅ HubSpot CRM sync
|
||||
- ✅ Zoho CRM sync
|
||||
|
||||
### Compliance
|
||||
- ✅ GDPR consent management
|
||||
- ✅ Data retention policies
|
||||
- ✅ Data deletion (right to be forgotten)
|
||||
- ✅ Data portability (export)
|
||||
- ✅ Compliance audit log
|
||||
- ✅ Sensitive data detection
|
||||
- ✅ PII/copyright stripping for training data
|
||||
|
||||
### Agency
|
||||
- ✅ White-label agency profiles
|
||||
- ✅ Client sub-account management
|
||||
- ✅ Usage analytics
|
||||
- ✅ Quota enforcement
|
||||
|
||||
### Training Data
|
||||
- ✅ Dataset generation from scraped content
|
||||
- ✅ PII and copyright stripping
|
||||
- ✅ License classification
|
||||
- ✅ Dataset CRUD
|
||||
|
||||
### Templates (110+)
|
||||
- ✅ Pre-built scraper templates for major sites
|
||||
- ✅ Template engine with parameterization
|
||||
- ✅ Template CRUD
|
||||
|
||||
### Intelligence
|
||||
- ✅ Competitive intelligence snapshots
|
||||
- ✅ Competitor comparison/diff
|
||||
- ✅ Intel report generation
|
||||
|
||||
### Integrations
|
||||
- ✅ Webhook support
|
||||
- ✅ Zapier integration
|
||||
- ✅ Make (formerly Integromat) integration
|
||||
- ✅ Browser extension (Chrome)
|
||||
- ✅ WordPress plugin
|
||||
- ✅ Shopify app
|
||||
- ✅ GPT Action manifest
|
||||
- ✅ MCP server config
|
||||
- ✅ CLI with autocomplete (bash, zsh, fish)
|
||||
|
||||
### Infrastructure
|
||||
- ✅ Docker Compose deployment
|
||||
- ✅ Dockerfile
|
||||
- ✅ GitHub Actions CI/CD
|
||||
- ✅ Pre-commit hooks (ruff, mypy, bandit, gitleaks)
|
||||
- ✅ Makefile with standardized targets
|
||||
- ✅ pyproject.toml with ruff/mypy/pytest config
|
||||
|
||||
## What's Remaining (🔜 Planned)
|
||||
|
||||
### High Priority
|
||||
- [ ] Comprehensive test coverage > 80% (current: ~266 tests, ~40% coverage)
|
||||
- [ ] End-to-end integration tests with FlareSolverr + Playwright
|
||||
- [ ] OpenAPI specification cleanup and versioning
|
||||
- [ ] Type annotations across all modules (mypy strict)
|
||||
|
||||
### Medium Priority
|
||||
- [ ] Redis-based distributed rate limiting
|
||||
- [ ] PostgreSQL-backed job queue (beyond filesystem)
|
||||
- [ ] Web dashboard (beyond current `/dashboard` health view)
|
||||
- [ ] API key authentication for multi-user deployments
|
||||
- [ ] OpenTelemetry tracing instrumentation
|
||||
- [ ] Prometheus metrics for all modules (currently only in `api.py`)
|
||||
|
||||
### Low Priority
|
||||
- [ ] GraphQL API alternative
|
||||
- [ ] Webhook retry with exponential backoff
|
||||
- [ ] Template marketplace / sharing
|
||||
- [ ] i18n for multi-language content extraction
|
||||
- [ ] Mobile app (React Native) companion
|
||||
- [ ] Desktop app (Electron) companion
|
||||
|
||||
## Audit Metrics (2026-06-30)
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Version | 3.0.0 |
|
||||
| Python source files | 101 |
|
||||
| Total lines of Python | 18,567 |
|
||||
| Test files | 43 |
|
||||
| Total test functions | 266 |
|
||||
| API endpoints | 115 |
|
||||
| Endpoint groups (tags) | 46 |
|
||||
| JSON scraper templates | 110 |
|
||||
| HTML templates | 2 |
|
||||
| Stealth JS scripts | 6 |
|
||||
| Browser extension files | 5 |
|
||||
| Shopify app files | 3 |
|
||||
| WordPress plugin files | 1 |
|
||||
| Config/support files | 8 |
|
||||
| Total functions (all modules) | 455 |
|
||||
| Total classes | 48 |
|
||||
Loading…
Add table
Add a link
Reference in a new issue