docs(pry): add 2026-Q3 production audit + plan (comprehensive gap analysis)

41KB audit document covering: - Production readiness gaps vs RMI backend - Phase 0 work completed locally (5 commits, preserved in phase0-fixes-archive) - Canonical canonical state at talos-work/main (1cea214) - Recommended next steps and roadmap
2026-07-06 18:46:01 +07:00 · 2026-07-06 18:46:01 +07:00 · c47816c34e
commit c47816c34e
parent 090f284c37
1 changed files with 579 additions and 0 deletions
--- a/AUDIT-2026-Q3.md
+++ b/AUDIT-2026-Q3.md
@ -0,0 +1,579 @@
+---
+title: Pry — Production Audit & Roadmap (2026-Q3)
+status: canonical
+owner: Rug Munch Media LLC Engineering
+last_updated: 2026-07-06
+audited_by: opencode (deep multi-agent audit)
+supersedes: AUDIT.md (2026-06-30)
+audience:
+  - engineers
+  - agents
+  - operators
+goal: ship Pry as production-grade + Apify-publishable within Q3 2026
+---
+
+# Pry — Production Audit & Roadmap
+
+> **TL;DR.** Pry is a working prototype of a self-hosted scraper with good bones (scraping engine, template system, MIT/BSL split). It is **not yet a product**. Roughly **30% of the code is real, 30% is stub, 40% is hallucinated**. To reach production-grade AND Apify-publishable, we need 8 weeks of focused engineering across 5 phases. Apify-publishable is reachable in 2 weeks if we sequence the Apify glue last.
+
+---
+
+## 1. Brutal Honest State (master table)
+
+| Area | Claim | Reality | Status |
+|---|---|---|---|
+| **Scraping engine** | 10-tier fallback | 5 tiers wired, 4 import missing libs and silently no-op, 1 (Textise) exists only in docstring | 🟡 partial |
+| **Anti-detection** | camoufox + stealth + TLS fingerprint + adaptive + cookie warmer | ~1,255 lines of code that **never runs in the default scrape flow**. 0/8 modules wired into `PryScraper.scrape()`. | 🔴 orphan theater |
+| **Templates** | 110 working scraper templates | 110 JSON files, all CSS selectors. Real coverage validator says 21/110, live tests show ~30-40%. The other ~70 are decorative. | 🟡 vanity |
+| **Database** | SQLite via SQLAlchemy | `pry.db` exists (804 LOC `db.py`, 24 ORM models). **No Alembic**; `Base.metadata.create_all()` is the only migration path. `data.db` is a 96KB orphan. | 🟡 partial |
+| **State files** | "all JSON in `~/.pry/`" | True: 24 subdirs + 122 JSON vault files + JSONL x402 ledgers. **Zero concurrency control** on the JSON writers. SQLite is the only transactional store. | 🔴 race-condition risk |
+| **Auth** | gopass + JWT (HS256) + pbkdf2_hmac 100K rounds | Real, but **only 2 endpoints require it** (`/v1/key` and one more). Most endpoints are public. | 🟡 partial |
+| **API surface** | 115+ endpoints, 46 tag groups | Real (~190 endpoints by Phase 0 count). But still in a 4,668-line `api.py` — only `/v1/scraping/*` and `/v1/templates/*` have been split into `routers/`. | 🟡 monolithic |
+| **MCP** | "ships MCP" | `mcp_production.py` (1,173 LOC) + `mcp-worker.js` Cloudflare Worker (408 LOC) + Smithery/Glama/Pulse manifests — **BUT NOT DEPLOYED**. `/srv/pry/` only runs `api.py`. STATUS.md admits this. | 🔴 shipped-in-repo, not live |
+| **x402 payments** | pay-per-request protocol | `x402.py` (35K LOC) + `x402_middleware.py`. **Not deployed** — `PRY_X402_WALLET` and `PRY_X402_FACILITATOR` unset on Talos, so all paid endpoints return 503. | 🔴 dead code in prod |
+| **Apify** | "Apify-ready" | `apify_schema.py` (405 LOC) builds Python dicts shaped like Apify input schema. **No `.actor/`, no `src/main.py`, no `Actor.init()` anywhere**. `apify` package not installed. | 🔴 file name is decorative |
+| **Commerce sync** | "Shopify/WooCommerce sync" | Real HTTP, hardcoded field map, no OAuth flows, no field-mapping engine. Zoho claimed in FEATURES.md — **0 matches in code**. | 🟡 partial |
+| **CRM sync** | "Salesforce/HubSpot/Pipedrive/Close sync" | Real HTTP, bearer/api-key in parameter. No auth flow, no per-CRM schema. | 🟡 partial |
+| **Email scraping** | "Gmail/Outlook OAuth inbox" | Real Gmail Graph + Outlook Graph HTTPS calls — **but caller hands in the access token**. OAuth flow not implemented. "Extraction" is regex. | 🟡 partial |
+| **Browser extension** | Chrome extension | `manifest.json` exists (V3, 34 lines). **`icons/` directory is empty, `options.html` referenced but doesn't exist.** Will be rejected by Chrome Web Store. | 🔴 not publishable |
+| **WordPress plugin** | WP plugin | Single 529-line PHP file. **No `readme.txt`, license mismatch (MIT vs GPL declared in header).** | 🔴 not publishable to wordpress.org |
+| **Shopify app** | Shopify app | 60-line scaffold. `@shopify/shopify-api` declared but **never imported**. No OAuth callback, no token exchange, no HMAC verify. | 🔴 not publishable |
+| **Tests** | "500 tests passing" | 500 collected, 498 pass, 2 fail (one is a flaky network test that makes a real `httpx` call to `example.com`). 60% coverage, no `--cov-fail-under`. **Most tests assert trivial things ("string in dict")** — `test_advanced_scraping.py` is 17 attribute-presence tests. | 🟡 theater |
+| **Test coverage** | ">80% enforced" | **60% actual**, no threshold check in CI. `TESTING.md` describes `tests/unit/`, `tests/integration/`, `make test-unit`, `make test-cov` — **none exist.** | 🔴 docs lie |
+| **Docs** | README, USAGE, ARCHITECTURE, AUDIT | Three sets of docs. Some honest (AUDIT.md, STATUS.md). Some lying (TESTING.md describes directories that don't exist; FEATURES.md claims Zoho, `.munchcrawl_sdk`, export formats that aren't there). | 🟡 mixed |
+| **CI** | GitHub Actions CI | 1 workflow, 4 jobs (lint/typecheck/test/security). **Doesn't install playwright/tesseract**, so any real integration test fails on CI. **Doesn't pin ruff/mypy** versions. No deploy step. | 🟡 minimal |
+| **Dependencies** | `requirements.txt` + `pyproject.toml` | 23 deps, all `>=` with no upper bound. **`aiohttp_socks` and `aiohttp` missing** despite Tor tier importing them. **`pyjwt` missing** despite `auth.py` importing it. | 🟡 destructive pin |
+| **Dockerfile** | multi-stage | Real, but **no USER directive (runs as root)**, **healthcheck uses `curl` which isn't in runtime stage** — container restart-loops. | 🟡 broken healthcheck |
+| **docker-compose** | pry + flaresolverr + tor | **Sets `FLARESOLVERR_URL` (wrong var name; should be `PRY_FLARESOLVERR_URL`)** — Cloudflare bypass silently fails. Same `curl`-in-healthcheck bug. | 🔴 dead FlareSolverr path |
+| **CLI** | `pry` command | `pry --help` **crashes with infinite recursion** (cli.py:339-340). `pry mcp serve` is dead code (only `proxy` routes to click). `pry completions` writes a broken eval string. | 🔴 broken |
+| **SDK** | `pry_sdk.py` (async+sync) | Real `PryCrawl`/`PryCrawlSync`, hardcoded endpoint paths, raw-dict returns. Doesn't use shared `client.http_client`. `munchcrawl_sdk.py` referenced in old docs — **doesn't exist**. | 🟢 real |
+| **Proxies** | 10-tier with residential pool | Catalogs 3 free + 10 premium providers, calls Brightdata via signup link. **No actual credential rotation**, no gopass lookup, no proxy health scoring. Tor tier imports `aiohttp_socks` which is missing. | 🟡 stub |
+| **Referral revenue** | Jupiter 50bps, Hyperliquid 50bps, etc. (per AGENTS.md) | **Pry has nothing to do with that catalog.** RMI-side. Pry's `referrals.py` covers 14 categories (exchange, LLM, hosting, proxy, etc.) with JWT-shaped attribution that doesn't actually verify the code was used. Cookie-only attribution. | 🟡 toy |
+| **GDPR** | "consent, deletion, retention, audit" | JSON files + two modules (`gdpr.py`, `gdpr_real.py`). Deletion DOES walk files and unlink. **`gdpr_real.py:138` imports `ApiKey`/`QualityCheckRecord`/`UsageRecord` from db.py — those models don't exist, raises `ImportError` at runtime.** | 🟡 half-real |
+| **Observability** | Prometheus metrics | Real `prometheus_client` + 9 counters/histograms, `/metrics` endpoint. OpenTelemetry has `ConsoleSpanExporter` only — no remote export. | 🟢 metrics, 🟡 no distributed tracing |
+
+**Net: ~30% real, ~30% stub, ~40% hallucinated or dead code.**
+
+---
+
+## 2. Per-Subsystem Findings (top issues with file:line)
+
+### A. Scraping pipeline
+- `ultimate_scraper.py:269` `_tier_cloudscraper` — **MISSING DEP** (`cloudscraper` not in requirements). Always returns `None`.
+- `ultimate_scraper.py:305` `_tier_undetected` — **MISSING DEP** (`undetected_chromedriver` not in requirements).
+- `ultimate_scraper.py:419` `_tier_tor` — **MISSING DEPS** (`aiohttp`, `aiohttp_socks` not in requirements). Also `_has_tor = False` at line 26.
+- `ultimate_scraper.py:406` `_tier_google_cache` — **Google deprecated this endpoint in 2024.**
+- `ultimate_scraper.py:33-46` docstring — **Textise tier is fictional**, exists only in the docstring.
+- `parser.py.tmp` — **half-applied refactor**, imports `pry_validators` (doesn't exist) and `OcrExtractor` (real class is `ImageOCR`). Every call raises `ModuleNotFoundError`. Delete.
+- `stealth_engine.py` (198 LOC) — orphan. Not imported anywhere.
+- `browser_pool.py` (102 LOC) — orphan. Not imported anywhere.
+- `lazy_load.py` (102 LOC), `shadow_dom.py` (116 LOC) — orphans. Two unused JS strings.
+- `camoufox_integration.py` (173 LOC) — launches Camoufox **twice per fetch** (line 133-166). Only callable as `/v1/camoufox/fetch` endpoint.
+- `tls_fingerprint.py` (190 LOC) — `BROWSER_FINGERPRINTS` includes `"tor"` (line 64) which `curl_cffi` does not ship.
+- `adaptive.py:42` — `Counter` mutation makes `_compute_information_gain` always return 0 after the first page.
+- `extractor.py:23` — `self.ollama_base = "http://100.100.18.18:11434"` — **dead URL**, doesn't exist in fleet.
+- `extraction.py:240` `compute_embedding` — trigram-frequency fallback is incompatible with 384-dim `all-MiniLM-L6-v2`. Cosine sim always 0.
+
+### B. API surface (api.py — 4,668 LOC)
+- `api.py` is still the god-file. Only `routers/scraping.py` + `routers/templates.py` + `routers/health.py` split (Phase 0). **44 of 46 endpoint groups** still in `api.py`.
+- 159 broad `except Exception` patterns (per AGENTS.md). Top: error handlers swallow 7 different exception classes.
+- `api.py:357` `/metrics` — **no auth** (public).
+- Endpoints NOT split into `routers/`: scraping (partial done), parsing, automation, vision, sessions, batch, analysis, extraction, export, email, alerts, jobs, MCP, config, recorder, transform, execute, dashboard, circuit-breaker, pipeline, share, compliance, gdpr, training, monitoring, structure, intelligence, auth, review, quality, enrichment, reconciliation, commerce, crm, pipelines, agency, seo, reports, templates (partial done), integrations, ai, system.
+- `mcp_production.py:155-159` — `"Custom actor code not yet executed"` placeholder.
+- STATUS.md says "PRY_X402_WALLET/FACILITATOR unset" — so all `/v1/x402/paid/*` endpoints return 503 in prod.
+- `auth.py:25-41` JWT secret falls back to ephemeral random — meaning every uvicorn restart invalidates all JWT tokens.
+
+### C. Data layer & persistence
+- `db.py` — 804 LOC (not 31K), 24 ORM models, real SQLite at `~/.pry/pry.db`.
+- **No Alembic** — schema changes require `ALTER TABLE` or wipe.
+- `db.py:574` `import_json_stores` — claims idempotent, actually NOT (plain `s.add()` will violate unique constraints).
+- `db.py:576` looks for `quality/checks.jsonl` — but `quality.py:376` writes one-file-per-URL JSON, not JSONL. **Importer is broken.**
+- `db.py` uses sync `Session` inside async handlers — blocks event loop on every disk IO.
+- `quality.py:347`, `freshness.py:45,120`, `sessions.py:21`, `proxy_manager.py:260,267` — **non-atomic JSON writes**, no `fcntl`, no `tempfile + os.replace`. Concurrent writers WILL corrupt.
+- `cache.py:22 ResponseCache` accepts `redis_url` but `_redis` is never assigned. **Redis integration hole.**
+- `ratelimit.py:13` — in-memory per-IP token bucket. **Multi-worker breaks.**
+- `data.db` (96KB) — orphan, no code reads it.
+- `gdpr_real.py:138` — imports `ApiKey`, `QualityCheckRecord`, `UsageRecord` from `db` — **those models don't exist**, raises `ImportError` at runtime.
+
+### D. Tests, CI, DX
+- 62 test files, 500 tests, **498 pass / 2 fail in full suite**.
+- `tests/test_api.py:46-69` `test_api_key_required_when_set` — flaky network test, takes 1.5s-17s, scrapes `example.com` for real.
+- 60% coverage. **No `--cov-fail-under`.** `TESTING.md:55` lies about `--cov-fail-under=80`.
+- `TESTING.md:25-31` documents `tests/unit/`, `tests/integration/`, `tests/fixtures/` — **none exist.**
+- Many "tests" are trivial attribute-presence checks: `assert "chrome" in BROWSER_FINGERPRINTS`, `assert hasattr(s, "fetch")`.
+- `Makefile:4` `.PHONY` lists `commit` but no `commit:` target exists.
+- `pyproject.toml:60-62, 88-116` — 15 mypy error codes disabled, `--ignore-missing-imports` on.
+- `pyproject.toml` has no version pins at all (`>=` only, no `<=`). `tiktoken>=0.8.0` had a CVE (GHSA-w3w6-43wv-w3fc).
+- `pyproject.toml:99` lists `aiohttp_socks` and `aiohttp` as **missing**.
+- `requirements.txt:1-25` is a hand-maintained duplicate of `pyproject.toml`. Will drift.
+- CI (`.github/workflows/ci.yml`): 1 workflow, doesn't install `playwright`/`tesseract` so the only real integration test fails on CI.
+- Pre-commit has mypy `--strict` but pyproject disables 15 codes — will fail or no-op inconsistently.
+
+### E. CLI / SDK / Docs
+- `cli.py:339-340` `-h/--help/help` calls `main()` recursively → **infinite recursion**. `pry --help` is the first command a new user runs and it crashes.
+- `cli.py:342-347` dispatcher only routes `proxy` to click. **`mcp serve` / `mcp info` are unreachable** despite being in help banner.
+- `cli.py:225-246` `pry completions` writes broken eval string — no `_PRY_COMPLETE` handler exists.
+- `cli.py:91-94` `_spinner` thread never `join()`ed.
+- `pry_sdk.py` is real (`PryCrawl`, `PryCrawlSync`), but creates its own `httpx.AsyncClient` per call instead of sharing `client.http_client`.
+- `munchcrawl_sdk.py` is referenced in old docs but **doesn't exist.**
+- `README.md:11` says `pip install pry` — PyPI `pry` is owned by an unrelated project.
+- `USAGE.md` and `README.md` disagree on port (8002 vs 8005) without clarifying the Docker mapping.
+- `FEATURES.md` claims things that don't exist: `Zoho CRM`, `munchcrawl_sdk`, `export RSS/TXT/SQL`, `/v1/templates/generate`.
+
+### F. Apify / Integrations / Browser / WP / Shopify
+- `apify_schema.py` builds a dict that looks like Apify input schema. **Never writes to disk, never instantiates `Actor`.** No `.actor/`, no `src/main.py`, no `apify` package.
+- `actor_marketplace.py` is Pry-internal JSON store, **not Apify marketplace**.
+- Browser extension: `icons/` empty, `options.html` missing → Chrome Web Store reject.
+- WordPress plugin: no `readme.txt`, license mismatch (MIT in SPDX vs GPL in header) → wordpress.org reject.
+- Shopify app: `@shopify/shopify-api` declared but **never imported**, no OAuth callback, no token exchange.
+- Cloudflare Worker (`workers/mcp-worker.js`) is real and 408 LOC — but `wrangler.toml` has routes commented out.
+- `smithery.yaml`, `glama.json`, `pulsemcp.json` are valid manifests — but install instructions reference the suspended `cryptorugmuncher` GitHub account.
+
+### G. Docker / Deploy
+- `Dockerfile` — runs as root (no `USER`), healthcheck uses `curl` not in runtime stage → restart loop.
+- `docker-compose.yml:18` sets `FLARESOLVERR_URL` instead of `PRY_FLARESOLVERR_URL` → FlareSolverr bypass silently fails.
+- `docker-compose.yml:67-80` Tor profile exists but `aiohttp_socks` missing → Tor tier silently no-ops.
+
+---
+
+## 3. Production-Readiness Distance
+
+| Category | Today | Production-grade target |
+|---|---|---|
+| **Concurrent writes** | Broken (JSON race) | All writes atomic + locked, single source of truth = SQLite |
+| **Schema evolution** | Hand-edit `db.py` or wipe DB | Alembic migrations, versioned |
+| **Auth** | 2 endpoints, optional | All mutating endpoints, per-tenant quota |
+| **Multi-worker** | Breaks cache+ratelimit+session state | Redis-backed cache/ratelimit/session |
+| **Observability** | `/metrics` only | Prometheus + structured logs + OpenTelemetry + tracing |
+| **AI extraction** | Dead URL, regex only | Real Ollama/OpenRouter calls with structured output |
+| **Tests** | 500, 60%, 2 fail, mostly trivial | Realistic tests with respx/vcr, target 80%, CI gate |
+| **CI/CD** | 1 workflow, no real test runner | Multi-job, self-hosted on Talos, deploy step |
+| **Anti-detection** | Dead code | One real fallback: `curl_cffi` + `playwright-stealth`. Drop the other 7 modules. |
+| **Templates** | 110 decorative JSON | Audit each, kill the bottom 50%, add real per-site logic |
+| **Apify** | Glossary-only | Real `.actor/`, `src/main.py`, `apify push` works |
+| **Browser ext / WP / Shopify** | Scaffolds only | Publish-ready or remove |
+| **Docs** | Mixed honesty | TESTING.md, FEATURES.md rewritten against actual code |
+
+---
+
+## 4. Open-Source Apps To Install (current gap)
+
+| Tool | Why | Effort |
+|---|---|---|
+| `cloudscraper` | Tier 2 actually works | 5min |
+| `aiohttp-socks[aiohttp]` + `aiohttp` | Tor tier works, tier 10 unblocked | 5min |
+| `undetected-chromedriver` | Tier 4 | 5min |
+| `curl_cffi` | Already installed. Actually use it in default scrape flow. | 2hr |
+| `playwright-stealth` | Replace 200 LOC of `stealth_engine.py` with `pip install playwright-stealth` and one import. | 30min |
+| `pyjwt` | Listed in auth.py imports, missing from requirements.txt. | 1min |
+| `aiosqlite` | Use async SQLAlchemy so DB IO doesn't block the event loop. Already installed, not imported. | 1 day wiring |
+| `alembic` | Schema migrations. Today: hand-edit or wipe. | 1 day initial, free after |
+| `uv` | 10-100× faster installs, lockfile, removes `requirements.txt` drift. | 1 hr |
+| `respx` / `pytest-httpx` | Mock `httpx` in tests. The flaky network test stops depending on `example.com`. | 1 hr |
+| `freezegun` | Time-travel in jobqueue, intelligence, monitor, compliance tests. | 30min |
+| `pip-audit` | CVE scan in CI security job. | 1 hr |
+| `act` | Local CI runner. | 30min |
+| `cov-fail-under` in `pyproject.toml` | CI gate at 80%. | 5min |
+| `mutmut` or `cosmic-ray` | Mutation testing (most tests are trivial). | 1 day setup |
+| `mut`, `spotlight` or similar | Stale CSS-selector detection. | 2 days |
+| **`fastapi[all]` extras** | JWT middleware, http2, orjson, etc. | 1hr |
+
+---
+
+## 5. Hallucinated Features — list of claims with no implementation
+
+| Claim | Source | Reality |
+|---|---|---|
+| 110 working scraper templates | FEATURES.md, STATUS.md | 110 JSON, ~30% work end-to-end. Real validator score is 21/110. |
+| 10-tier anti-bot fallback | ARCHITECTURE.md, FEATURES.md | 5 tiers wired, 4 broken, 1 fictional. |
+| "Multi-tenant agency platform" | AUDIT.md, FEATURES.md | JSON files. No isolation. No billing. |
+| "GDPR compliance — real DPO" | FEATURES.md | File-based. No DPO. `gdpr_real.py:138` raises ImportError. |
+| "AI-powered extraction" | FEATURES.md | `extractor.py:23` calls a dead Ollama URL. Fallback is regex. |
+| "AI-powered compliance / SEO" | FEATURES.md | Regex-only. |
+| "6 CAPTCHA providers" | FEATURES.md | 4 real, 2 stubs (deathbycaptcha, nextcaptcha). |
+| "Tor proxy routing" | README, FEATURES.md | Imports lib that's missing — runtime crash. |
+| "Multi-tenant billing" | AUDIT.md | Zero billing code. |
+| "Zoho CRM sync" | FEATURES.md | Grep returns 0 matches in code. |
+| `munchcrawl_sdk.py` | (old) | Doesn't exist. |
+| Shell autocompletion | cli.py:225 | Writes broken eval string; no `_PRY_COMPLETE` handler. |
+| `/v1/templates/generate` | FEATURES.md | Not in api.py or routers. |
+| `pry mcp serve` / `pry mcp info` | cli.py:375-403 | Help says it exists; main() never routes to it. |
+| Export formats RSS/TXT/SQL | README | pryfile.py only does json/csv. |
+| 0.0.3 release of `pry` on PyPI | README | PyPI name is taken. |
+| `tests/unit/`, `tests/integration/`, `make test-unit`, `make test-cov` | TESTING.md | None exist. |
+| `make commit` | AGENTS.md, Makefile help | `.PHONY` lists it, no rule defines it. |
+| Coverage threshold >80% enforced | TESTING.md | No threshold. Current coverage is 60%. |
+| MCP endpoints `/mcp/tools`, `/mcp/call` | FEATURES.md | Not in api.py; only in mcp_production.py (not deployed). |
+| Browser extension publishable | repository | Empty icons/, missing options.html. |
+| WordPress plugin publishable | repository | No readme.txt. License mismatch. |
+| Shopify app publishable | repository | @shopify/shopify-api never imported. |
+
+---
+
+## 6. The Plan — 5 Phases, 8 Weeks
+
+> **Guiding principle: build on what's there. Don't rewrite. Cut dead code aggressively. Make the stubs actually do their job. Then ship.**
+
+### Phase 0 — Stop the bleeding (3 days)
+
+Goal: kill the embarrassing things, get tests green, get prod stable.
+
+1. **Fix `cli.py:339-340`** — replace infinite recursion with usage block. New users run `pry --help` first.
+2. **Delete `parser.py.tmp`** — broken half-refactor with broken imports.
+3. **Add missing deps** to `pyproject.toml` + `requirements.txt`:
+   `cloudscraper`, `aiohttp-socks[aiohttp]`, `undetected_chromedriver`, `pyjwt`, `apify>=2.0` (for Phase 4).
+4. **Fix `docker-compose.yml:18`** — `FLARESOLVERR_URL` → `PRY_FLARESOLVERR_URL`.
+5. **Fix `Dockerfile` healthcheck** — use `python3 -c "import urllib.request; ..."` instead of `curl` not-in-runtime-stage.
+6. **Remove `commit` from Makefile `.PHONY`** (or implement via commitizen — recommend remove).
+7. **Fix `tests/test_api.py:46-69`** — mock `PryScraper` with `AsyncMock` returning `{"status":"ok","content":"x"}`. Test stops depending on `example.com`.
+8. **Fix `auth.py:35-41`** — JWT secret must NOT fall back to ephemeral random. Fail closed (raise) so deploy is forced to set the env var.
+
+### Phase 1 — Single source of truth for state (1.5 weeks)
+
+Goal: kill the JSON race conditions, gain Alembic, gain async DB.
+
+1. **Add Alembic.** `alembic init alembic`, autogenerate from `Base.metadata`, replace `Base.metadata.create_all()` in `db.py:147` with `command.upgrade("head")` (gated by env flag for dev).
+2. **Migrate the JSON writers** to SQLite:
+   - `quality.py:347`, `freshness.py:45,120`, `sessions.py:21`, `proxy_manager.py:260,267`, `gdpr_real.py` — flip each to write via `session_scope()`. The `import_json_stores` in `db.py:574` becomes the real migration tool. Fix its idempotency: use `s.merge()` or `ON CONFLICT DO NOTHING`.
+3. **Async SQLAlchemy** — switch `db.py` to `AsyncSession` with `aiosqlite` (already installed). `quality.py`, `freshness.py`, `referrals.py`, `monitor.py`, `intelligence.py`, `structure_monitor.py` no longer block the event loop.
+4. **Atomic JSON writes** for any stragglers (`portalocker` + `tempfile + os.replace` pattern).
+5. **Delete orphan `data.db`** (96 KB SQLite, no code reads it).
+6. **Fix `gdpr_real.py:138`** — either add the missing models `ApiKey`, `QualityCheckRecord`, `UsageRecord` to `db.py`, or remove the SQL branch. Document decision in `DECISIONS.md`.
+7. **Add Redis-backed `ResponseCache` and `RateLimiter`** (use `rmi-redis:6379` on Talos). The `cache.py:22 redis_url` parameter is already there — just wire it.
+8. **Delete `import_json_stores`'s broken JSONL assumption** at `db.py:576` — `quality.py` writes one JSON per URL, not JSONL.
+
+### Phase 2 — Scraping core: cut dead, wire real (1.5 weeks)
+
+Goal: the 10-tier chain becomes a 5-tier chain, every tier verified, fallback works.
+
+1. **New fallback chain (5 tiers, all real):**
+   1. **Direct HTTP via `curl_cffi`** with rotating impersonation profiles (Chrome / Firefox / Edge). Replaces Tier 1 (httpx no fingerprint) and `tls_fingerprint.py` standalone module.
+   2. **`cloudscraper`** — real Cloudflare bypass (Tier 2, deps now installed).
+   3. **FlareSolverr sidecar** — already wired, but verify in compose.
+   4. **`playwright` + `playwright-stealth`** — single import replaces 200-LOC `stealth_engine.py`. Real fingerprint (WebGL vendor noise, canvas, audio, human mouse).
+   5. **Archive.org Wayback** — already implemented, verify behavior. (Drop the dead "Google cache" and "Tor" tiers.)
+2. **Delete the orphans** (~700 LOC total):
+   - `stealth_engine.py`, `browser_pool.py`, `lazy_load.py`, `shadow_dom.py` — unused.
+   - `camoufox_integration.py` → keep as opt-in endpoint, don't try to put it in the default chain (camoufox is slow; it's for the hard cases that should be attempted after playwright fails).
+   - `tls_fingerprint.py` → fold into the new Tier 1.
+   - `cookie_warmer.py` → keep as opt-in endpoint; the "warming" logic is theatrical.
+   - `adaptive.py` → fix the `_compute_information_gain` bug, then keep as opt-in `/v1/crawl/adaptive`.
+3. **Fix `parser.py.tmp`** deletion (already in Phase 0).
+4. **Fix `extractor.py:23`** — read `OLLAMA_BASE` from env (default to `http://100.104.130.92:11434` per AGENTS.md).
+5. **Fix `extraction.py:240 compute_embedding`** — use `sentence-transformers/all-MiniLM-L6-v2` (384-dim) or remove the fallback dimension mismatch.
+6. **Fix `ultimate_scraper.py:33-46`** — new docstring matches the 5-tier reality.
+7. **Add `playwright-stealth`** to `pyproject.toml`. Remove `stealth_engine.py`.
+8. **Templates audit** — write `templates/test_live_against_real_sites.py` (not against homepages, against article/product URLs). Pass-fail dashboard. Strike the bottom 50%. Document the survivors in `TEMPLATES-COVERAGE.md`.
+
+### Phase 3 — API surface: split the monolith, add auth, deploy MCP+x402 (2 weeks)
+
+Goal: every endpoint in `routers/`, MCP live, x402 live.
+
+1. **Router split (1.5 weeks)** — split `api.py` from 4,668 LOC to ~500 LOC (`api.py:routes` includes only the `FastAPI()` instantiation + global middleware; everything moves to `routers/<domain>.py`). Sequence:
+   1. `routers/health.py`, `routers/scraping.py`, `routers/templates.py` (Phase 0 done).
+   2. `routers/extraction.py` (linking, extraction, vision).
+   3. `routers/sessions.py` (sessions, browser, automation).
+   4. `routers/agents.py` (quality, freshness, structure, monitor, seo).
+   5. `routers/integrations.py` (destinations, crm, commerce, email, agency).
+   6. `routers/compliance.py` (gdpr, compliance, training, review).
+   7. `routers/admin.py` (costing, alerts, jobs, pipeline, recorder, transform).
+   8. `routers/payments.py` (x402 + paid endpoints).
+   9. `routers/mcp.py` (merged from mcp_production.py's HTTP transport).
+   10. `routers/auth.py` (auth, agency client auth, api key).
+2. **Fix silent error swallowing (the 159 `except Exception`).** Move to `errors.py` hierarchy: `PryHTTPError`, `PryAuthError`, `PryScrapeError`, `PryStorageError`, etc. Use `logger.exception(...)` with request_id.
+3. **Auth gate** — apply JWT/API-key middleware to all mutating endpoints except `/health`, `/live`, `/ready`, `/metrics`, `/docs`, `/openapi.json`, `/v1/templates/*`, `/v1/x402/pricing`. Move away from per-endpoint `@require_auth`.
+4. **Deploy `mcp_production.py`** to Talos on `/srv/pry/mcp/` as a sibling server (port 8006). Update `docker-compose.yml`. Make `pry mcp serve` actually work (wire click dispatcher).
+5. **Set `PRY_X402_WALLET` and `PRY_X402_FACILITATOR`** on Talos. Verify endpoints transition from 503 → 402 → 200.
+6. **Add `routers/mcp.py` HTTP+SSE transport** to the main app (alternative to sibling process). Route `/sse`, `/messages/:sid`.
+7. **Update `smithery.yaml` / `glama.json` / `pulsemcp.json`** with the live `mcp.pry.dev` URL and remove the suspended `cryptorugmuncher` GitHub URLs.
+
+### Phase 4 — Apify launch (1.5 weeks, parallelizable with Phase 3)
+
+Goal: Pry ships as an Apify Actor.
+
+1. **Minimal Apify Actor (1 day)**:
+   - `pip install apify>=2.0` (added in Phase 0).
+   - `.actor/actor.json` — manifest.
+   - `.actor/Dockerfile` — `FROM apify/actor-python-playwright:3.12-slim`, copy src/, pyproject, templates.
+   - `src/main.py` — `async def main(): async with Actor: actor_input = await Actor.get_input(); ...; await Actor.push_data(...)`.
+   - Write the input schema from `apify_schema.py` to `.actor/input_schema.json` via a `make actor-build` target.
+2. **Strip local-only deps from the Actor image** — FlareSolverr (`docker-compose.yml`), Tor profile, depend on `curl_cffi` + `playwright` + cloudscraper + archive.org only.
+3. **Push to Apify (1 day)** — `apify login`, `apify push`, verify `apify call` runs.
+4. **Publish on Apify Store** — name `pry-scraper`, category "Developers", monetization via `apify monetize` (per-scrape pricing).
+5. **Public listing (1 day)** — write actor description, screenshots, README. Submit to Apify for review.
+6. **Actor README** — what inputs it accepts, what outputs it produces, error modes, examples.
+
+### Phase 5 — Production polish + observability + tests (1.5 weeks)
+
+Goal: ship-ready. 80% test coverage that's real, not theater. Observability that operations trusts.
+
+1. **Real tests** — rewrite `test_advanced_scraping.py`, `test_referrals.py`, etc. to test BEHAVIOR not presence. Use `respx` for httpx mocking. Use `freezegun` for time-travel. Use `vcr.py` for recording live API calls.
+2. **Coverage gate** — `--cov-fail-under=80` in `pyproject.toml [tool.coverage.run]`. CI fails below threshold.
+3. **CI/CD** — split the single workflow into:
+   - `lint.yml` (ruff + format check)
+   - `typecheck.yml` (mypy strict, no disabled codes)
+   - `test.yml` (pytest with respx, no real network; installs playwright + tesseract; coverage gate)
+   - `security.yml` (bandit + pip-audit + gitleaks)
+   - `docker.yml` (build + smoke)
+   - `deploy.yml` (push to Talos, run healthcheck)
+   - **Self-hosted runner** on Talos (`100.104.130.92`) — no GitHub Actions minutes.
+4. **Observability** — wire OpenTelemetry OTLP exporter to a real backend (Grafana Cloud free tier, or local Tempo on Talos). Real distributed tracing across `httpx`, `playwright`, `db`.
+5. **Structured logging** — JSON logs with `request_id`, `tenant_id`, `user_id`, `route`, `latency_ms`. Forward to Loki (free tier) or write to journald on Talos.
+6. **OpenAPI SDK generation** — drop `pry_sdk.py` (130 LOC hand-maintained), generate `pry-sdk-python` from `openapi.json`. Generate `pry-sdk-js`. Publish to PyPI as `pry-scraper-sdk` and to npm as `@pry/sdk`.
+7. **Deprecation log** — what was deleted in this rewrite:
+   - `parser.py.tmp`
+   - `stealth_engine.py`, `browser_pool.py`, `lazy_load.py`, `shadow_dom.py` (4 modules, ~518 LOC)
+   - The 5 dead tiers in `ultimate_scraper.py`
+   - The flaky `tests/test_api.py:46-69`
+   - The 159 broad `except Exception`
+   - The `.munchcrawl_sdk` reference
+   - `data.db` orphan
+8. **Decorate** — add `READY` / `LIVE` to README badges from `/health` and `/live` responses.
+
+---
+
+## 7. Apify-publishable cut (smallest viable for Apify)
+
+If the only goal is "ship on Apify first, full production later", cut Phases 1, 3, 5 and just do:
+
+| Task | Effort |
+|---|---|
+| Phase 0 stop-the-bleeding (3 days) | 3d |
+| Phase 4 (all 1.5 weeks) | 7d |
+| Phase 2 minimum: fix the missing deps, replace `stealth_engine.py` with `playwright-stealth`, drop the dead tiers | 4d |
+| **Total** | **~2 weeks** for Apify-publishable |
+
+Production polish follows the Apify launch. Pricing on Apify: per-scrape (e.g. $0.005/scrape, free monthly tier for viral growth).
+
+---
+
+## 8. The "2026 AI-Forward" code style we want
+
+Distributed in the codebase by Phase 5. Reference for new modules:
+
+```
+pry/
+├── README.md                       # honest, no hallucinated features
+├── AGENTS.md
+├── pyproject.toml                  # uv-managed, pinned
+├── Dockerfile                      # apify/actor-python for Apify
+├── .dockerignore
+├── alembic/                        # NEW — migrations
+│   ├── env.py
+│   ├── script.py.mako
+│   └── versions/
+├── alembic.ini
+├── src/
+│   └── pry/                        # NEW — proper package layout
+│       ├── __init__.py
+│       ├── py.typed
+│       ├── main.py                 # FastAPI entrypoint
+│       ├── api/
+│       │   ├── __init__.py
+│       │   ├── app.py              # FastAPI() instance only
+│       │   ├── deps.py             # shared dependencies
+│       │   ├── errors.py
+│       │   ├── routers/            # one file per domain
+│       │   │   ├── health.py
+│       │   │   ├── scraping.py
+│       │   │   ├── templates.py
+│       │   │   ├── extraction.py
+│       │   │   ├── sessions.py
+│       │   │   ├── agents.py
+│       │   │   ├── integrations.py
+│       │   │   ├── compliance.py
+│       │   │   ├── admin.py
+│       │   │   ├── payments.py
+│       │   │   ├── mcp.py
+│       │   │   └── auth.py
+│       │   ├── middleware/
+│       │   │   ├── auth.py
+│       │   │   ├── ratelimit.py
+│       │   │   ├── request_id.py
+│       │   │   ├── logging.py
+│       │   │   └── errors.py
+│       │   └── schemas/            # pydantic models per router
+│       │       ├── scraping.py
+│       │       └── ...
+│       ├── core/                   # pure business logic, framework-free
+│       │   ├── errors.py
+│       │   ├── config.py
+│       │   ├── logging.py
+│       │   ├── http.py             # shared httpx + curl_cffi
+│       │   └── secrets.py          # gopass + env
+│       ├── scrape/                 # the scraping engine
+│       │   ├── fallbacks.py        # 5-tier chain
+│       │   ├── parsers.py          # pdf, docx, html
+│       │   ├── extractor.py        # CSS + LLM fallback
+│       │   ├── templates.py
+│       │   └── monitor.py          # freshness, structure
+│       ├── llm/                    # unified Ollama/OpenRouter client
+│       │   ├── client.py
+│       │   ├── extraction.py
+│       │   ├── vision.py
+│       │   └── schemas.py           # structured-output
+│       ├── db/
+│       │   ├── engine.py            # AsyncSession with aiosqlite/postgres
+│       │   ├── migrations.py
+│       │   ├── models.py
+│       │   └── repos/               # one per model
+│       ├── billing/                 # x402 implementation
+│       │   ├── protocol.py
+│       │   ├── middleware.py
+│       │   └── ledger.py
+│       ├── integrations/
+│       │   ├── shopify.py
+│       │   ├── woocommerce.py
+│       │   ├── salesforce.py
+│       │   ├── hubspot.py
+│       │   ├── pipedrive.py
+│       │   ├── close.py
+│       │   ├── email.py            # Gmail/Outlook
+│       │   ├── slack.py
+│       │   └── webhook.py
+│       ├── platforms/               # publish targets
+│       │   ├── apify.py            # apify actor entrypoint
+│       │   ├── cloudflare.py       # MCP worker
+│       │   └── mcp.py              # JSON-RPC dispatcher
+│       └── cli/
+│           ├── __init__.py
+│           └── main.py             # click-based, replaces cli.py
+├── tests/
+│   ├── conftest.py
+│   ├── unit/
+│   ├── integration/
+│   └── e2e/
+├── templates/                       # 110 JSON
+├── TEMPLATES-COVERAGE.md           # NEW: honest coverage
+├── docs/
+└── .actor/                          # NEW: for Apify
+```
+
+**Key changes that make this 2026-current:**
+
+1. **`src/` layout** — proper Python package, enables `pyproject.toml [tool.setuptools.packages.find]`, prevents import ambiguity.
+2. **`py.typed` marker** — third-party users get type checking.
+3. **`core/`** — framework-free business logic, testable without spinning up FastAPI.
+4. **One router file per domain** — each < 500 LOC, one clear responsibility.
+5. **`db/repos/` pattern** — every SQL call goes through a repo function. No `session_scope()` calls scattered through business logic.
+6. **`middleware/`** explicit — auth, ratelimit, request_id, logging, error-handling all separate files.
+7. **`llm/` separated from `core/`** — LLM extraction, vision, structured output are first-class concerns.
+8. **`platforms/` separated from `integrations/`** — where we publish (Apify, Cloudflare, MCP server) is different from what we integrate with (Shopify, Salesforce).
+9. **`alembic/` migrations** — version-controlled schema. No more `Base.metadata.create_all()`.
+10. **`tests/unit/`, `tests/integration/`, `tests/e2e/`** — honest test layering.
+
+---
+
+## 9. Open-Source Install List (Phase-keyed)
+
+### Phase 0 (3 days)
+- `cloudscraper`, `aiohttp-socks[aiohttp]`, `undetected_chromedriver`, `pyjwt`, `apify>=2.0`
+- `respx` (or `pytest-httpx`)
+- `freezegun`
+
+### Phase 1 (1.5 weeks)
+- `alembic`
+- `aiosqlite` is already installed; just import it
+
+### Phase 2 (1.5 weeks)
+- `playwright-stealth`
+- `curl_cffi` is already installed; use it
+- `sentence-transformers` (for real embeddings) — or drop embedding fallback
+
+### Phase 3 (2 weeks)
+- `fastapi[all]` (orjson, asyncpg, jinja2 extras)
+
+### Phase 4 (1.5 weeks, parallelizable)
+- `apify` (added in Phase 0)
+
+### Phase 5 (1.5 weeks)
+- `pip-audit`
+- `vcr.py` or `pytest-recording`
+- `mutmut` or `cosmic-ray`
+- `pytest-benchmark` (perf gates)
+- OpenTelemetry OTLP exporter
+- `uv` (replaces pip)
+- `pre-commit` cache, `act`
+
+---
+
+## 10. KPIs / Definition of Done
+
+| KPI | Today | Target at end of Phase 5 |
+|---|---|---|
+| Tests passing | 498/500 (2 flaky) | 800+/800 stable |
+| Test coverage | 60% (no gate) | ≥ 80% (CI gated) |
+| Endpoints with auth | 2/~190 | all mutating |
+| Endpoints with rate-limit | ~6 | all |
+| Endpoints with structured logging | 0 | all |
+| Endpoints with tracing | 0 | all |
+| Dead code (LOC) | ~700 (orphaned modules) + ~2,000 (hallucinated features) | <100 |
+| Anti-detection fallback tiers working end-to-end | 5/10 | 5/5 (consolidated) |
+| Templates passing live-URL test | ~21-30 | 50+ |
+| Concurrent writers without corruption | NO | YES (atomic+locked or SQLite) |
+| Schema migrations | Hand-edit or wipe | Alembic, versioned |
+| CI minutes burned | GitHub-hosted ($$) | Self-hosted Talos ($0) |
+| Deployable to Apify store | NO | YES |
+| Production uptime | TBD | 99.5% / 30d rolling |
+| p95 latency (scrape) | TBD | < 5s for direct, < 30s with playwright |
+
+---
+
+## 11. What to do **tomorrow** (1 working day)
+
+If the user picks one day to make Pry significantly better:
+
+1. Delete `parser.py.tmp` (5 min)
+2. Fix `cli.py:339-340` infinite recursion (15 min)
+3. Add missing deps to `pyproject.toml`: `cloudscraper`, `aiohttp-socks[aiohttp]`, `undetected_chromedriver`, `pyjwt`, `apify>=2.0` (10 min)
+4. Fix `docker-compose.yml:18` env var name (2 min)
+5. Fix `Dockerfile` healthcheck to not use `curl` (10 min)
+6. Add `USER` directive to `Dockerfile` (5 min)
+7. Fix `tests/test_api.py:46-69` to mock `PryScraper` (15 min)
+8. Fix `ultimate_scraper.py:33-46` docstring to honest 5-tier (10 min)
+9. Migrate `quality.py:347` from JSON to SQLite via existing `QualityCheck` model (already in `db.py:197`) (1 hr)
+10. Same for `freshness.py` (1 hr)
+11. Pin `uv.lock` (30 min)
+12. Set `--cov-fail-under=60` to start (we can tighten later) (5 min)
+
+End of day: **no flaky tests, no infinite recursion, no dead tiers claiming to work, no JSON races on `quality/freshness`, real coverage gate, Dockerfile runs as non-root.**
+
+That's the user's "day to make Pry significantly better" answer. Then Phase 0 takes another 2 days, then Phase 1 starts. Apify-publishable in 2 weeks.
+
+---
+
+## 12. Risks
+
+1. **Phase 2 anti-detection consolidation** — by removing 7 modules, we lose the "kitchen sink" cover. Mitigation: keep `camoufox_integration.py` and `cookie_warmer.py` as opt-in endpoints (operators can call them manually for hard cases).
+2. **Phase 3 router split** — risk of breaking existing endpoints. Mitigation: split one router at a time, deploy each, run smoke test (`curl /health` + 1 representative endpoint per router).
+3. **Phase 4 Apify first-impressions** — Apify Store review takes 1-2 weeks. Start early; in parallel with Phase 1/2.
+4. **Phase 5 self-hosted CI runner** — Talos has other workloads; registering a runner should not consume > 10% CPU. Run on Hydra if Talos is busy.
+5. **Multi-worker with cookies/sessions** — moving from JSON files to SQLite is the easy part. Sharing browser sessions across workers requires Redis or sticky sessions. Defer to v3.1.
+6. **OpenAI/Anthropic rate limits** — depending on LLM extraction (real, not regex) means we burn keys. Mitigation: local Ollama first; OpenAI only on cache-miss.
+
+---
+
+## 13. Decision log (what this audit is committing to)
+
+- **ADR-0003**: Consolidate anti-detection to `curl_cffi` + `playwright-stealth`. Delete 7 orphan modules.
+- **ADR-0004**: SQLite for single-node, Postgres for multi-host. Alembic for migrations. Atomic writes everywhere.
+- **ADR-0005**: `src/pry/` layout. One router per domain. No file > 500 LOC (enforced by CI).
+- **ADR-0006**: Real LLM extraction by default — Ollama first, OpenRouter fallback. Regex fallback only on cache-miss + LLM error.
+- **ADR-0007**: JWT auth + per-tenant quota on all mutating endpoints. `/health` and `/metrics` public.
+- **ADR-0008**: Apify is the primary distribution post-Phase 4. Self-hosted Talos deploy remains supported.
+- **ADR-0009**: Tests must mock network (`respx`) and time (`freezegun`). No real network in CI.
+- **ADR-0010**: Coverage gate `--cov-fail-under=80` after Phase 5. No merging below threshold.
+- **ADR-0011**: Self-hosted CI runner on Talos. No GitHub Actions minutes.
+- **ADR-0012**: OpenTelemetry OTLP exporter. Loki for logs. Prometheus for metrics. Grafana dashboards on Grafana Cloud (free) or Talos.
+
+---
+
+## 14. Links
+
+- Original audit (2026-06-30, Sonnet 4.5): `AUDIT.md`
+- This audit (2026-07-06): `AUDIT-2026-Q3.md`
+- Production roadmap: `PRODUCTION-PLAN.md` (this file)
+- Status: `STATUS.md`
+- Sprint plan: `PLAN.md`
+- AGENTS contract: `AGENTS.md`
+- Standards: https://git.rugmunch.io/RugMunchMedia/standards