cf376ab3a8
AUDIT-2026-Q3.md §Hallucinated features flagged that FEATURES.md claimed
Zoho CRM sync but no code existed. The codebase now ships Salesforce,
HubSpot, Pipedrive, and Close.com in crm_sync.py — no Zoho.
This commit enforces doc-vs-code parity:
- Remove the Zoho row from FEATURES.md
- Strip '~~Zoho~~' from USAGE.md and ARCHITECTURE.md
- Move ROADMAP.md from '✅ Zoho CRM sync' to '⏸ deferred'
- Add crm_sync_zoho.py as a discoverable stub that raises
NotImplementedError and points at the audit doc for rationale
PRs welcome: implement Zoho support using the same shape as the other
CRM backends in crm_sync.py.
246 lines
11 KiB
Markdown
246 lines
11 KiB
Markdown
[//]: # (SPDX-License-Identifier: MIT)
|
||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||
---
|
||
title: Pry Feature Catalog
|
||
status: canonical
|
||
owner: Rug Munch Media LLC Engineering
|
||
last_updated: 2026-06-30
|
||
---
|
||
|
||
# Pry Feature Catalog
|
||
|
||
## Scraping & Crawling
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Single URL scrape | `scraper.py` | `POST /v1/scrape` | ✅ |
|
||
| Multi-page crawl | `scraper.py` | `POST /v1/crawl` | ✅ |
|
||
| Batch scrape (parallel) | `advanced.py` | `POST /v1/batch` | ✅ |
|
||
| Batch from file + template | `advanced.py` | `POST /v1/batch-file` | ✅ |
|
||
| URL discovery (sitemap) | `scraper.py` | `POST /v1/map` | ✅ |
|
||
| Link analysis | `scraper.py` | `POST /v1/links` | ✅ |
|
||
|
||
## Anti-Detection & Bypass
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| 10-tier fallback scraper | `ultimate_scraper.py` | Internal | ✅ |
|
||
| Block detection | `adaptive.py` | `POST /v1/detect-block` | ✅ |
|
||
| Adaptive strategy rotation | `adaptive.py` | Internal | ✅ |
|
||
| Browser pool + pre-warming | `browser_pool.py` | Internal | ✅ |
|
||
| Stealth engine (6 JS scripts) | `stealth_engine.py` | Internal | ✅ |
|
||
| Stealth scripts | `stealth_scripts/` | Injected | ✅ |
|
||
| Tor proxy routing (tier documented but disabled) | `network.py` | `POST /v1/config/profile/tor` | ⚠️ `aiohttp-socks` missing |
|
||
| SOCKS5 proxy support | `network.py` | Config | ✅ |
|
||
| User-agent rotation | `scraper.py` | Internal | ✅ |
|
||
|
||
## Document Parsing
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| PDF parsing | `parser.py` | `POST /v1/parse` | ✅ |
|
||
| DOCX parsing | `parser.py` | `POST /v1/parse` | ✅ |
|
||
| Image OCR | `parser.py` | `POST /v1/parse` | ✅ |
|
||
| CSV parsing | `parser.py` | `POST /v1/parse` | ✅ |
|
||
| JSON parsing | `parser.py` | `POST /v1/parse` | ✅ |
|
||
| Shadow DOM extraction | `shadow_dom.py` | `POST /v1/shadow-dom` | ✅ |
|
||
| Lazy load / infinite scroll | `lazy_load.py` | `POST /v1/capture/lazy` | ✅ |
|
||
|
||
## Extraction & Structured Data
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| JSON schema extraction | `extractor.py` | `POST /v1/extract` | ✅ |
|
||
| LLM extraction + chunking | `extraction.py` | `POST /v1/extract/llm` | ✅ |
|
||
| Schema.org / JSON-LD | `extractor.py` | `POST /v1/schema` | ✅ |
|
||
| Email address extraction | `extractor.py` | `POST /v1/emails` | ✅ |
|
||
| Structured field extraction | `extraction.py` | `POST /v1/extract/fields` | ✅ |
|
||
| Field suggestion (AI) | `advanced.py` | `POST /v1/suggest` | ✅ |
|
||
|
||
## AI & Vision
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Vision AI (OpenRouter, 5-model fallback) | `advanced.py` | `POST /v1/vision` | ✅ |
|
||
| AI summarization (Ollama) | `advanced.py` | `POST /v1/summarize` | ✅ |
|
||
| AI categorization | `advanced.py` | `POST /v1/categorize` | ✅ |
|
||
| GPT Action manifest | `ai_plugin.py` | `GET /v1/ai/gpt-manifest` | ✅ |
|
||
| MCP server config | `ai_plugin.py` | `GET /v1/ai/mcp-config` | ✅ |
|
||
| ~~MCP tool discovery~~ | ~~`mcp_server.py`~~ | ~~`GET /mcp/tools`~~ | ⚠️ not in `api.py` |
|
||
| ~~MCP tool execution~~ | ~~`mcp_server.py`~~ | ~~`POST /mcp/call`~~ | ⚠️ not in `api.py` |
|
||
|
||
> `GET /mcp/tools` and `POST /mcp/call` exist only in `mcp_production.py`, which is **not deployed**. Use `pry mcp serve` (stdio transport) for MCP access — see AUDIT-2026-Q3.md.
|
||
|
||
## Browser Automation
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Step-based automation | `automator.py` | `POST /v1/automate` | ✅ |
|
||
| Screenshot capture | `automator.py` | `POST /v1/screenshot` | ✅ |
|
||
| Persistent browser sessions | `sessions.py` | `POST /v1/session/create` | ✅ |
|
||
| Session list | `sessions.py` | `GET /v1/sessions` | ✅ |
|
||
| Session save/restore | `sessions.py` | `POST /v1/session/save` | ✅ |
|
||
| Session close | `sessions.py` | `POST /v1/session/close` | ✅ |
|
||
| Action recorder | `advanced.py` | `POST /v1/record/start` | ✅ |
|
||
|
||
## CAPTCHA Solving
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| CAPTCHA solving (6 providers) | `captcha_solver.py` | `POST /v1/auth/captcha` | ✅ |
|
||
| Auto-fallback across providers | `captcha_solver.py` | Internal | ✅ |
|
||
|
||
## Authentication
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Credential vault (encrypted) | `auth_connector.py` | `POST /v1/auth/credentials` | ✅ |
|
||
| SSO login script generation | `auth_connector.py` | `POST /v1/auth/sso` | ✅ |
|
||
| Session health check | `auth_connector.py` | `POST /v1/auth/session/health` | ✅ |
|
||
| Signup automation | `signup_automator.py` | Internal | ✅ |
|
||
| Account pool management | `account_manager.py` | Internal | ✅ |
|
||
|
||
## Monitoring & Change Detection
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Page watch / change detection | `freshness.py` | `POST /v1/watch` | ✅ |
|
||
| Diff tracking | `advanced.py` | `POST /v1/diff` | ✅ |
|
||
| Content freshness dashboard | `freshness.py` | `GET /v1/freshness/dashboard` | ✅ |
|
||
| Scheduled monitors (cron) | `monitor.py` | `POST /v1/monitor` | ✅ |
|
||
| Monitor run on-demand | `monitor.py` | `POST /v1/monitor/run` | ✅ |
|
||
| Monitor list / delete | `monitor.py` | `GET /v1/monitors` | ✅ |
|
||
| Structure monitoring | `structure_monitor.py` | `POST /v1/structure/check` | ✅ |
|
||
|
||
## Alerts & Notifications
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Multi-channel alerts | `alerter.py` | `POST /v1/alert/send` | ✅ |
|
||
| Supported channel listing | `alerter.py` | `GET /v1/alert/channels` | ✅ |
|
||
| Webhook support | `destinations.py` | Config | ✅ |
|
||
| Slack integration | `destinations.py` | Config | ✅ |
|
||
|
||
## Quality & Review
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Content quality scoring | `quality.py` | `POST /v1/quality/check` | ✅ |
|
||
| Quality history | `quality.py` | `GET /v1/quality/history` | ✅ |
|
||
| Human review queue | `review.py` | `POST /v1/review/submit` | ✅ |
|
||
| Review approve/reject | `review.py` | `POST /v1/review/{id}/approve` | ✅ |
|
||
| Review queue listing | `review.py` | `GET /v1/reviews` | ✅ |
|
||
|
||
## Data Export & Integration
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| ~~Multi-format export (JSON, CSV, RSS, TXT, SQL)~~ | `destinations.py` | `POST /v1/export` | ⚠️ `pryfile.py` only does `json`/`csv`; RSS/TXT/SQL not implemented |
|
||
| Webhook delivery | `destinations.py` | Config | ✅ |
|
||
| S3 upload | `destinations.py` | Config | ✅ |
|
||
| GCS upload | `destinations.py` | Config | ✅ |
|
||
| SFTP upload | `destinations.py` | Config | ✅ |
|
||
| Data transformation | `advanced.py` | `POST /v1/transform` | ✅ |
|
||
|
||
## Commerce & CRM
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| WooCommerce product sync | `commerce_sync.py` | `POST /v1/commerce/sync` | ✅ |
|
||
| Shopify product sync | `commerce_sync.py` | `POST /v1/commerce/sync` | ✅ |
|
||
| Shopify app backend | `shopify-app/` | External | ✅ |
|
||
| Salesforce CRM sync | `crm_sync.py` | `POST /v1/crm/sync` | ✅ |
|
||
| HubSpot CRM sync | `crm_sync.py` | `POST /v1/crm/sync` | ✅ |
|
||
|
||
## Pipelines
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Pipeline definition validation | `pipelines.py` | `POST /v1/pipelines/validate` | ✅ |
|
||
| Pipeline execution | `pipelines.py` | `POST /v1/pipelines/run` | ✅ |
|
||
| Pipeline CRUD | `pipelines.py` | CRUD endpoints | ✅ |
|
||
| Hook registration | `pipeline.py` | `POST /v1/pipeline/hook` | ✅ |
|
||
| Hook execution | `pipeline.py` | `POST /v1/pipeline/run` | ✅ |
|
||
| Schema pipeline | `pipelines.py` | `POST /v1/pipelines/schema` | ✅ |
|
||
|
||
## Intelligence & Competitive Analysis
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Competitor snapshot | `intelligence.py` | `POST /v1/intel/snapshot` | ✅ |
|
||
| Intel comparison (diff) | `intelligence.py` | `POST /v1/intel/compare` | ✅ |
|
||
| Intel report generation | `intelligence.py` | `GET /v1/intel/report` | ✅ |
|
||
| Tech stack detection | `enrichment.py` | `POST /v1/enrichment/analyze` | ✅ |
|
||
|
||
## Compliance & GDPR
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Compliance check (URL) | `compliance.py` | `POST /v1/compliance/check` | ✅ |
|
||
| Consent recording | `gdpr.py` | `POST /v1/gdpr/consent` | ✅ |
|
||
| Consent check / revoke | `gdpr.py` | `GET /v1/gdpr/consent/{id}` | ✅ |
|
||
| Data deletion request | `gdpr.py` | `POST /v1/gdpr/deletion/request` | ✅ |
|
||
| Deletion execution | `gdpr.py` | `POST /v1/gdpr/deletion/execute` | ✅ |
|
||
| Data retention policy | `gdpr.py` | `GET /v1/gdpr/retention` | ✅ |
|
||
| Data portability (export) | `gdpr.py` | `POST /v1/gdpr/export` | ✅ |
|
||
| Compliance audit log | `gdpr.py` | `GET /v1/gdpr/audit` | ✅ |
|
||
|
||
## Training Data
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| PII/copyright stripping | `training_data.py` | `POST /v1/training/clean` | ✅ |
|
||
| Dataset generation | `training_data.py` | `POST /v1/training/dataset` | ✅ |
|
||
| Dataset CRUD | `training_data.py` | CRUD endpoints | ✅ |
|
||
| License classification | `training_data.py` | Internal | ✅ |
|
||
|
||
## Agency / White-Label
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Agency profile creation | `agency.py` | `POST /v1/agency/create` | ✅ |
|
||
| Agency branding | `agency.py` | `PUT /v1/agency/{id}/branding` | ✅ |
|
||
| Client sub-account management | `agency.py` | `POST /v1/agency/{id}/clients` | ✅ |
|
||
| Usage analytics | `agency.py` | `GET /v1/agency/{id}/analytics` | ✅ |
|
||
| Client quota enforcement | `agency.py` | `GET /v1/client/{id}/quota` | ✅ |
|
||
|
||
## Reporting
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Report generation | `reports.py` | `POST /v1/reports/generate` | ✅ |
|
||
| Report list / get | `reports.py` | `GET /v1/reports` | ✅ |
|
||
|
||
## Templates
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| Template listing | `template_engine.py` | `GET /v1/templates` | ✅ |
|
||
| Template detail | `template_engine.py` | `GET /v1/templates/{id}` | ✅ |
|
||
| ~~Template generation~~ | ~~`template_engine.py`~~ | ~~`POST /v1/templates/generate`~~ | ❌ endpoint not registered in `api.py` |
|
||
| Pre-built templates | 110 JSON files | `templates/` | ⚠️ ~21/110 pass structural validation; ~30–40% work end-to-end (see AUDIT-2026-Q3.md) |
|
||
|
||
## SEO
|
||
|
||
| Feature | Module | Endpoint | Status |
|
||
|---------|--------|----------|--------|
|
||
| SEO analysis | `seo_monitor.py` | `POST /v1/seo/analyze` | ✅ |
|
||
| SEO change tracking | `seo_monitor.py` | `POST /v1/seo/track` | ✅ |
|
||
| Keyword analysis | `seo_monitor.py` | `POST /v1/seo/keywords` | ✅ |
|
||
|
||
## Infrastructure
|
||
|
||
| Feature | Module | Status |
|
||
|---------|--------|--------|
|
||
| Health probes (liveness/readiness) | `api.py` | ✅ |
|
||
| Prometheus metrics | `api.py` | ✅ |
|
||
| Rate limiting (token bucket) | `ratelimit.py` | ✅ |
|
||
| LRU cache + Redis | `cache.py` | ✅ |
|
||
| Circuit breaker per-domain | `advanced.py` | ✅ |
|
||
| WebSocket streaming | `pryextras.py` | ✅ |
|
||
| Async job queue | `jobqueue.py` | ✅ |
|
||
| Docker Compose deployment | `docker-compose.yml` | ✅ |
|
||
| CI/CD (GitHub Actions) | `.github/workflows/ci.yml` | ✅ |
|
||
| Pre-commit hooks | `.pre-commit-config.yaml` | ✅ |
|
||
| Browser extension (Chrome) | `browser-extension/` | ✅ |
|
||
| WordPress plugin | `wordpress-plugin/` | ✅ |
|