pryscraper/AUDIT-2026-Q3.md

---
title: Pry — Production Audit & Roadmap (2026-Q3)
status: canonical
owner: Rug Munch Media LLC Engineering
last_updated: 2026-07-06
audited_by: opencode (deep multi-agent audit)
supersedes: AUDIT.md (2026-06-30)
audience:
  - engineers
  - agents
  - operators
goal: ship Pry as production-grade + Apify-publishable within Q3 2026
---

# Pry — Production Audit & Roadmap

> **TL;DR.** Pry is a working prototype of a self-hosted scraper with good bones (scraping engine, template system, MIT/BSL split). It is **not yet a product**. Roughly **30% of the code is real, 30% is stub, 40% is hallucinated**. To reach production-grade AND Apify-publishable, we need 8 weeks of focused engineering across 5 phases. Apify-publishable is reachable in 2 weeks if we sequence the Apify glue last.

---

## 1. Brutal Honest State (master table)

| Area | Claim | Reality | Status |
|---|---|---|---|
| **Scraping engine** | 10-tier fallback | 5 tiers wired, 4 import missing libs and silently no-op, 1 (Textise) exists only in docstring | 🟡 partial |
| **Anti-detection** | camoufox + stealth + TLS fingerprint + adaptive + cookie warmer | ~1,255 lines of code that **never runs in the default scrape flow**. 0/8 modules wired into `PryScraper.scrape()`. | 🔴 orphan theater |
| **Templates** | 110 working scraper templates | 110 JSON files, all CSS selectors. Real coverage validator says 21/110, live tests show ~30-40%. The other ~70 are decorative. | 🟡 vanity |
| **Database** | SQLite via SQLAlchemy | `pry.db` exists (804 LOC `db.py`, 24 ORM models). **No Alembic**; `Base.metadata.create_all()` is the only migration path. `data.db` is a 96KB orphan. | 🟡 partial |
| **State files** | "all JSON in `~/.pry/`" | True: 24 subdirs + 122 JSON vault files + JSONL x402 ledgers. **Zero concurrency control** on the JSON writers. SQLite is the only transactional store. | 🔴 race-condition risk |
| **Auth** | gopass + JWT (HS256) + pbkdf2_hmac 100K rounds | Real, but **only 2 endpoints require it** (`/v1/key` and one more). Most endpoints are public. | 🟡 partial |
| **API surface** | 115+ endpoints, 46 tag groups | Real (~190 endpoints by Phase 0 count). But still in a 4,668-line `api.py` — only `/v1/scraping/*` and `/v1/templates/*` have been split into `routers/`. | 🟡 monolithic |
| **MCP** | "ships MCP" | `mcp_production.py` (1,173 LOC) + `mcp-worker.js` Cloudflare Worker (408 LOC) + Smithery/Glama/Pulse manifests — **BUT NOT DEPLOYED**. `/srv/pry/` only runs `api.py`. STATUS.md admits this. | 🔴 shipped-in-repo, not live |
| **x402 payments** | pay-per-request protocol | `x402.py` (35K LOC) + `x402_middleware.py`. **Not deployed** — `PRY_X402_WALLET` and `PRY_X402_FACILITATOR` unset on Talos, so all paid endpoints return 503. | 🔴 dead code in prod |
| **Apify** | "Apify-ready" | `apify_schema.py` (405 LOC) builds Python dicts shaped like Apify input schema. **No `.actor/`, no `src/main.py`, no `Actor.init()` anywhere**. `apify` package not installed. | 🔴 file name is decorative |
| **Commerce sync** | "Shopify/WooCommerce sync" | Real HTTP, hardcoded field map, no OAuth flows, no field-mapping engine. Zoho claimed in FEATURES.md — **0 matches in code**. | 🟡 partial |
| **CRM sync** | "Salesforce/HubSpot/Pipedrive/Close sync" | Real HTTP, bearer/api-key in parameter. No auth flow, no per-CRM schema. | 🟡 partial |
| **Email scraping** | "Gmail/Outlook OAuth inbox" | Real Gmail Graph + Outlook Graph HTTPS calls — **but caller hands in the access token**. OAuth flow not implemented. "Extraction" is regex. | 🟡 partial |
| **Browser extension** | Chrome extension | `manifest.json` exists (V3, 34 lines). **`icons/` directory is empty, `options.html` referenced but doesn't exist.** Will be rejected by Chrome Web Store. | 🔴 not publishable |
| **WordPress plugin** | WP plugin | Single 529-line PHP file. **No `readme.txt`, license mismatch (MIT vs GPL declared in header).** | 🔴 not publishable to wordpress.org |
| **Shopify app** | Shopify app | 60-line scaffold. `@shopify/shopify-api` declared but **never imported**. No OAuth callback, no token exchange, no HMAC verify. | 🔴 not publishable |
| **Tests** | "500 tests passing" | 500 collected, 498 pass, 2 fail (one is a flaky network test that makes a real `httpx` call to `example.com`). 60% coverage, no `--cov-fail-under`. **Most tests assert trivial things ("string in dict")** — `test_advanced_scraping.py` is 17 attribute-presence tests. | 🟡 theater |
| **Test coverage** | ">80% enforced" | **60% actual**, no threshold check in CI. `TESTING.md` describes `tests/unit/`, `tests/integration/`, `make test-unit`, `make test-cov` — **none exist.** | 🔴 docs lie |
| **Docs** | README, USAGE, ARCHITECTURE, AUDIT | Three sets of docs. Some honest (AUDIT.md, STATUS.md). Some lying (TESTING.md describes directories that don't exist; FEATURES.md claims Zoho, `.pry_sdk`, export formats that aren't there). | 🟡 mixed |
| **CI** | GitHub Actions CI | 1 workflow, 4 jobs (lint/typecheck/test/security). **Doesn't install playwright/tesseract**, so any real integration test fails on CI. **Doesn't pin ruff/mypy** versions. No deploy step. | 🟡 minimal |
| **Dependencies** | `requirements.txt` + `pyproject.toml` | 23 deps, all `>=` with no upper bound. **`aiohttp_socks` and `aiohttp` missing** despite Tor tier importing them. **`pyjwt` missing** despite `auth.py` importing it. | 🟡 destructive pin |
| **Dockerfile** | multi-stage | Real, but **no USER directive (runs as root)**, **healthcheck uses `curl` which isn't in runtime stage** — container restart-loops. | 🟡 broken healthcheck |
| **docker-compose** | pry + flaresolverr + tor | **Sets `FLARESOLVERR_URL` (wrong var name; should be `PRY_FLARESOLVERR_URL`)** — Cloudflare bypass silently fails. Same `curl`-in-healthcheck bug. | 🔴 dead FlareSolverr path |
| **CLI** | `pry` command | `pry --help` **crashes with infinite recursion** (cli.py:339-340). `pry mcp serve` is dead code (only `proxy` routes to click). `pry completions` writes a broken eval string. | 🔴 broken |
| **SDK** | `pry_sdk.py` (async+sync) | Real `PryCrawl`/`PryCrawlSync`, hardcoded endpoint paths, raw-dict returns. Doesn't use shared `client.http_client`. `pry_sdk.py` referenced in old docs — **doesn't exist**. | 🟢 real |
| **Proxies** | 10-tier with residential pool | Catalogs 3 free + 10 premium providers, calls Brightdata via signup link. **No actual credential rotation**, no gopass lookup, no proxy health scoring. Tor tier imports `aiohttp_socks` which is missing. | 🟡 stub |
| **Referral revenue** | Jupiter 50bps, Hyperliquid 50bps, etc. (per AGENTS.md) | **Pry has nothing to do with that catalog.** RMI-side. Pry's `referrals.py` covers 14 categories (exchange, LLM, hosting, proxy, etc.) with JWT-shaped attribution that doesn't actually verify the code was used. Cookie-only attribution. | 🟡 toy |
| **GDPR** | "consent, deletion, retention, audit" | JSON files + two modules (`gdpr.py`, `gdpr_real.py`). Deletion DOES walk files and unlink. **`gdpr_real.py:138` imports `ApiKey`/`QualityCheckRecord`/`UsageRecord` from db.py — those models don't exist, raises `ImportError` at runtime.** | 🟡 half-real |
| **Observability** | Prometheus metrics | Real `prometheus_client` + 9 counters/histograms, `/metrics` endpoint. OpenTelemetry has `ConsoleSpanExporter` only — no remote export. | 🟢 metrics, 🟡 no distributed tracing |

**Net: ~30% real, ~30% stub, ~40% hallucinated or dead code.**

---

## 2. Per-Subsystem Findings (top issues with file:line)

### A. Scraping pipeline
- `ultimate_scraper.py:269` `_tier_cloudscraper` — **MISSING DEP** (`cloudscraper` not in requirements). Always returns `None`.
- `ultimate_scraper.py:305` `_tier_undetected` — **MISSING DEP** (`undetected_chromedriver` not in requirements).
- `ultimate_scraper.py:419` `_tier_tor` — **MISSING DEPS** (`aiohttp`, `aiohttp_socks` not in requirements). Also `_has_tor = False` at line 26.
- `ultimate_scraper.py:406` `_tier_google_cache` — **Google deprecated this endpoint in 2024.**
- `ultimate_scraper.py:33-46` docstring — **Textise tier is fictional**, exists only in the docstring.
- `parser.py.tmp` — **half-applied refactor**, imports `pry_validators` (doesn't exist) and `OcrExtractor` (real class is `ImageOCR`). Every call raises `ModuleNotFoundError`. Delete.
- `stealth_engine.py` (198 LOC) — orphan. Not imported anywhere.
- `browser_pool.py` (102 LOC) — orphan. Not imported anywhere.
- `lazy_load.py` (102 LOC), `shadow_dom.py` (116 LOC) — orphans. Two unused JS strings.
- `camoufox_integration.py` (173 LOC) — launches Camoufox **twice per fetch** (line 133-166). Only callable as `/v1/camoufox/fetch` endpoint.
- `tls_fingerprint.py` (190 LOC) — `BROWSER_FINGERPRINTS` includes `"tor"` (line 64) which `curl_cffi` does not ship.
- `adaptive.py:42` — `Counter` mutation makes `_compute_information_gain` always return 0 after the first page.
- `extractor.py:23` — `self.ollama_base = "http://100.100.18.18:11434"` — **dead URL**, doesn't exist in fleet.
- `extraction.py:240` `compute_embedding` — trigram-frequency fallback is incompatible with 384-dim `all-MiniLM-L6-v2`. Cosine sim always 0.

### B. API surface (api.py — 4,668 LOC)
- `api.py` is still the god-file. Only `routers/scraping.py` + `routers/templates.py` + `routers/health.py` split (Phase 0). **44 of 46 endpoint groups** still in `api.py`.
- 159 broad `except Exception` patterns (per AGENTS.md). Top: error handlers swallow 7 different exception classes.
- `api.py:357` `/metrics` — **no auth** (public).
- Endpoints NOT split into `routers/`: scraping (partial done), parsing, automation, vision, sessions, batch, analysis, extraction, export, email, alerts, jobs, MCP, config, recorder, transform, execute, dashboard, circuit-breaker, pipeline, share, compliance, gdpr, training, monitoring, structure, intelligence, auth, review, quality, enrichment, reconciliation, commerce, crm, pipelines, agency, seo, reports, templates (partial done), integrations, ai, system.
- `mcp_production.py:155-159` — `"Custom actor code not yet executed"` placeholder.
- STATUS.md says "PRY_X402_WALLET/FACILITATOR unset" — so all `/v1/x402/paid/*` endpoints return 503 in prod.
- `auth.py:25-41` JWT secret falls back to ephemeral random — meaning every uvicorn restart invalidates all JWT tokens.

### C. Data layer & persistence
- `db.py` — 804 LOC (not 31K), 24 ORM models, real SQLite at `~/.pry/pry.db`.
- **No Alembic** — schema changes require `ALTER TABLE` or wipe.
- `db.py:574` `import_json_stores` — claims idempotent, actually NOT (plain `s.add()` will violate unique constraints).
- `db.py:576` looks for `quality/checks.jsonl` — but `quality.py:376` writes one-file-per-URL JSON, not JSONL. **Importer is broken.**
- `db.py` uses sync `Session` inside async handlers — blocks event loop on every disk IO.
- `quality.py:347`, `freshness.py:45,120`, `sessions.py:21`, `proxy_manager.py:260,267` — **non-atomic JSON writes**, no `fcntl`, no `tempfile + os.replace`. Concurrent writers WILL corrupt.
- `cache.py:22 ResponseCache` accepts `redis_url` but `_redis` is never assigned. **Redis integration hole.**
- `ratelimit.py:13` — in-memory per-IP token bucket. **Multi-worker breaks.**
- `data.db` (96KB) — orphan, no code reads it.
- `gdpr_real.py:138` — imports `ApiKey`, `QualityCheckRecord`, `UsageRecord` from `db` — **those models don't exist**, raises `ImportError` at runtime.

### D. Tests, CI, DX
- 62 test files, 500 tests, **498 pass / 2 fail in full suite**.
- `tests/test_api.py:46-69` `test_api_key_required_when_set` — flaky network test, takes 1.5s-17s, scrapes `example.com` for real.
- 60% coverage. **No `--cov-fail-under`.** `TESTING.md:55` lies about `--cov-fail-under=80`.
- `TESTING.md:25-31` documents `tests/unit/`, `tests/integration/`, `tests/fixtures/` — **none exist.**
- Many "tests" are trivial attribute-presence checks: `assert "chrome" in BROWSER_FINGERPRINTS`, `assert hasattr(s, "fetch")`.
- `Makefile:4` `.PHONY` lists `commit` but no `commit:` target exists.
- `pyproject.toml:60-62, 88-116` — 15 mypy error codes disabled, `--ignore-missing-imports` on.
- `pyproject.toml` has no version pins at all (`>=` only, no `<=`). `tiktoken>=0.8.0` had a CVE (GHSA-w3w6-43wv-w3fc).
- `pyproject.toml:99` lists `aiohttp_socks` and `aiohttp` as **missing**.
- `requirements.txt:1-25` is a hand-maintained duplicate of `pyproject.toml`. Will drift.
- CI (`.github/workflows/ci.yml`): 1 workflow, doesn't install `playwright`/`tesseract` so the only real integration test fails on CI.
- Pre-commit has mypy `--strict` but pyproject disables 15 codes — will fail or no-op inconsistently.

### E. CLI / SDK / Docs
- `cli.py:339-340` `-h/--help/help` calls `main()` recursively → **infinite recursion**. `pry --help` is the first command a new user runs and it crashes.
- `cli.py:342-347` dispatcher only routes `proxy` to click. **`mcp serve` / `mcp info` are unreachable** despite being in help banner.
- `cli.py:225-246` `pry completions` writes broken eval string — no `_PRY_COMPLETE` handler exists.
- `cli.py:91-94` `_spinner` thread never `join()`ed.
- `pry_sdk.py` is real (`PryCrawl`, `PryCrawlSync`), but creates its own `httpx.AsyncClient` per call instead of sharing `client.http_client`.
- `pry_sdk.py` is referenced in old docs but **doesn't exist.**
- `README.md:11` says `pip install pry` — PyPI `pry` is owned by an unrelated project.
- `USAGE.md` and `README.md` disagree on port (8002 vs 8005) without clarifying the Docker mapping.
- `FEATURES.md` claims things that don't exist: `Zoho CRM`, `pry_sdk.py`, `export RSS/TXT/SQL`, `/v1/templates/generate`.

### F. Apify / Integrations / Browser / WP / Shopify
- `apify_schema.py` builds a dict that looks like Apify input schema. **Never writes to disk, never instantiates `Actor`.** No `.actor/`, no `src/main.py`, no `apify` package.
- `actor_marketplace.py` is Pry-internal JSON store, **not Apify marketplace**.
- Browser extension: `icons/` empty, `options.html` missing → Chrome Web Store reject.
- WordPress plugin: no `readme.txt`, license mismatch (MIT in SPDX vs GPL in header) → wordpress.org reject.
- Shopify app: `@shopify/shopify-api` declared but **never imported**, no OAuth callback, no token exchange.
- Cloudflare Worker (`workers/mcp-worker.js`) is real and 408 LOC — but `wrangler.toml` has routes commented out.
- `smithery.yaml`, `glama.json`, `pulsemcp.json` are valid manifests — but install instructions reference the suspended `cryptorugmuncher` GitHub account.

### G. Docker / Deploy
- `Dockerfile` — runs as root (no `USER`), healthcheck uses `curl` not in runtime stage → restart loop.
- `docker-compose.yml:18` sets `FLARESOLVERR_URL` instead of `PRY_FLARESOLVERR_URL` → FlareSolverr bypass silently fails.
- `docker-compose.yml:67-80` Tor profile exists but `aiohttp_socks` missing → Tor tier silently no-ops.

---

## 3. Production-Readiness Distance

| Category | Today | Production-grade target |
|---|---|---|
| **Concurrent writes** | Broken (JSON race) | All writes atomic + locked, single source of truth = SQLite |
| **Schema evolution** | Hand-edit `db.py` or wipe DB | Alembic migrations, versioned |
| **Auth** | 2 endpoints, optional | All mutating endpoints, per-tenant quota |
| **Multi-worker** | Breaks cache+ratelimit+session state | Redis-backed cache/ratelimit/session |
| **Observability** | `/metrics` only | Prometheus + structured logs + OpenTelemetry + tracing |
| **AI extraction** | Dead URL, regex only | Real Ollama/OpenRouter calls with structured output |
| **Tests** | 500, 60%, 2 fail, mostly trivial | Realistic tests with respx/vcr, target 80%, CI gate |
| **CI/CD** | 1 workflow, no real test runner | Multi-job, self-hosted on Talos, deploy step |
| **Anti-detection** | Dead code | One real fallback: `curl_cffi` + `playwright-stealth`. Drop the other 7 modules. |
| **Templates** | 110 decorative JSON | Audit each, kill the bottom 50%, add real per-site logic |
| **Apify** | Glossary-only | Real `.actor/`, `src/main.py`, `apify push` works |
| **Browser ext / WP / Shopify** | Scaffolds only | Publish-ready or remove |
| **Docs** | Mixed honesty | TESTING.md, FEATURES.md rewritten against actual code |

---

## 4. Open-Source Apps To Install (current gap)

| Tool | Why | Effort |
|---|---|---|
| `cloudscraper` | Tier 2 actually works | 5min |
| `aiohttp-socks[aiohttp]` + `aiohttp` | Tor tier works, tier 10 unblocked | 5min |
| `undetected-chromedriver` | Tier 4 | 5min |
| `curl_cffi` | Already installed. Actually use it in default scrape flow. | 2hr |
| `playwright-stealth` | Replace 200 LOC of `stealth_engine.py` with `pip install playwright-stealth` and one import. | 30min |
| `pyjwt` | Listed in auth.py imports, missing from requirements.txt. | 1min |
| `aiosqlite` | Use async SQLAlchemy so DB IO doesn't block the event loop. Already installed, not imported. | 1 day wiring |
| `alembic` | Schema migrations. Today: hand-edit or wipe. | 1 day initial, free after |
| `uv` | 10-100× faster installs, lockfile, removes `requirements.txt` drift. | 1 hr |
| `respx` / `pytest-httpx` | Mock `httpx` in tests. The flaky network test stops depending on `example.com`. | 1 hr |
| `freezegun` | Time-travel in jobqueue, intelligence, monitor, compliance tests. | 30min |
| `pip-audit` | CVE scan in CI security job. | 1 hr |
| `act` | Local CI runner. | 30min |
| `cov-fail-under` in `pyproject.toml` | CI gate at 80%. | 5min |
| `mutmut` or `cosmic-ray` | Mutation testing (most tests are trivial). | 1 day setup |
| `mut`, `spotlight` or similar | Stale CSS-selector detection. | 2 days |
| **`fastapi[all]` extras** | JWT middleware, http2, orjson, etc. | 1hr |

---

## 5. Hallucinated Features — list of claims with no implementation

| Claim | Source | Reality |
|---|---|---|
| 110 working scraper templates | FEATURES.md, STATUS.md | 110 JSON, ~30% work end-to-end. Real validator score is 21/110. |
| 10-tier anti-bot fallback | ARCHITECTURE.md, FEATURES.md | 5 tiers wired, 4 broken, 1 fictional. |
| "Multi-tenant agency platform" | AUDIT.md, FEATURES.md | JSON files. No isolation. No billing. |
| "GDPR compliance — real DPO" | FEATURES.md | File-based. No DPO. `gdpr_real.py:138` raises ImportError. |
| "AI-powered extraction" | FEATURES.md | `extractor.py:23` calls a dead Ollama URL. Fallback is regex. |
| "AI-powered compliance / SEO" | FEATURES.md | Regex-only. |
| "6 CAPTCHA providers" | FEATURES.md | 4 real, 2 stubs (deathbycaptcha, nextcaptcha). |
| "Tor proxy routing" | README, FEATURES.md | Imports lib that's missing — runtime crash. |
| "Multi-tenant billing" | AUDIT.md | Zero billing code. |
| "Zoho CRM sync" | FEATURES.md | Grep returns 0 matches in code. |
| `pry_sdk.py` | (old) | Doesn't exist. |
| Shell autocompletion | cli.py:225 | Writes broken eval string; no `_PRY_COMPLETE` handler. |
| `/v1/templates/generate` | FEATURES.md | Not in api.py or routers. |
| `pry mcp serve` / `pry mcp info` | cli.py:375-403 | Help says it exists; main() never routes to it. |
| Export formats RSS/TXT/SQL | README | pryfile.py only does json/csv. |
| 0.0.3 release of `pry` on PyPI | README | PyPI name is taken. |
| `tests/unit/`, `tests/integration/`, `make test-unit`, `make test-cov` | TESTING.md | None exist. |
| `make commit` | AGENTS.md, Makefile help | `.PHONY` lists it, no rule defines it. |
| Coverage threshold >80% enforced | TESTING.md | No threshold. Current coverage is 60%. |
| MCP endpoints `/mcp/tools`, `/mcp/call` | FEATURES.md | Not in api.py; only in mcp_production.py (not deployed). |
| Browser extension publishable | repository | Empty icons/, missing options.html. |
| WordPress plugin publishable | repository | No readme.txt. License mismatch. |
| Shopify app publishable | repository | @shopify/shopify-api never imported. |

---

## 6. The Plan — 5 Phases, 8 Weeks

> **Guiding principle: build on what's there. Don't rewrite. Cut dead code aggressively. Make the stubs actually do their job. Then ship.**

### Phase 0 — Stop the bleeding (3 days)

Goal: kill the embarrassing things, get tests green, get prod stable.

1. **Fix `cli.py:339-340`** — replace infinite recursion with usage block. New users run `pry --help` first.
2. **Delete `parser.py.tmp`** — broken half-refactor with broken imports.
3. **Add missing deps** to `pyproject.toml` + `requirements.txt`:
   `cloudscraper`, `aiohttp-socks[aiohttp]`, `undetected_chromedriver`, `pyjwt`, `apify>=2.0` (for Phase 4).
4. **Fix `docker-compose.yml:18`** — `FLARESOLVERR_URL` → `PRY_FLARESOLVERR_URL`.
5. **Fix `Dockerfile` healthcheck** — use `python3 -c "import urllib.request; ..."` instead of `curl` not-in-runtime-stage.
6. **Remove `commit` from Makefile `.PHONY`** (or implement via commitizen — recommend remove).
7. **Fix `tests/test_api.py:46-69`** — mock `PryScraper` with `AsyncMock` returning `{"status":"ok","content":"x"}`. Test stops depending on `example.com`.
8. **Fix `auth.py:35-41`** — JWT secret must NOT fall back to ephemeral random. Fail closed (raise) so deploy is forced to set the env var.

### Phase 1 — Single source of truth for state (1.5 weeks)

Goal: kill the JSON race conditions, gain Alembic, gain async DB.

1. **Add Alembic.** `alembic init alembic`, autogenerate from `Base.metadata`, replace `Base.metadata.create_all()` in `db.py:147` with `command.upgrade("head")` (gated by env flag for dev).
2. **Migrate the JSON writers** to SQLite:
   - `quality.py:347`, `freshness.py:45,120`, `sessions.py:21`, `proxy_manager.py:260,267`, `gdpr_real.py` — flip each to write via `session_scope()`. The `import_json_stores` in `db.py:574` becomes the real migration tool. Fix its idempotency: use `s.merge()` or `ON CONFLICT DO NOTHING`.
3. **Async SQLAlchemy** — switch `db.py` to `AsyncSession` with `aiosqlite` (already installed). `quality.py`, `freshness.py`, `referrals.py`, `monitor.py`, `intelligence.py`, `structure_monitor.py` no longer block the event loop.
4. **Atomic JSON writes** for any stragglers (`portalocker` + `tempfile + os.replace` pattern).
5. **Delete orphan `data.db`** (96 KB SQLite, no code reads it).
6. **Fix `gdpr_real.py:138`** — either add the missing models `ApiKey`, `QualityCheckRecord`, `UsageRecord` to `db.py`, or remove the SQL branch. Document decision in `DECISIONS.md`.
7. **Add Redis-backed `ResponseCache` and `RateLimiter`** (use `rmi-redis:6379` on Talos). The `cache.py:22 redis_url` parameter is already there — just wire it.
8. **Delete `import_json_stores`'s broken JSONL assumption** at `db.py:576` — `quality.py` writes one JSON per URL, not JSONL.

### Phase 2 — Scraping core: cut dead, wire real (1.5 weeks)

Goal: the 10-tier chain becomes a 5-tier chain, every tier verified, fallback works.

1. **New fallback chain (5 tiers, all real):**
   1. **Direct HTTP via `curl_cffi`** with rotating impersonation profiles (Chrome / Firefox / Edge). Replaces Tier 1 (httpx no fingerprint) and `tls_fingerprint.py` standalone module.
   2. **`cloudscraper`** — real Cloudflare bypass (Tier 2, deps now installed).
   3. **FlareSolverr sidecar** — already wired, but verify in compose.
   4. **`playwright` + `playwright-stealth`** — single import replaces 200-LOC `stealth_engine.py`. Real fingerprint (WebGL vendor noise, canvas, audio, human mouse).
   5. **Archive.org Wayback** — already implemented, verify behavior. (Drop the dead "Google cache" and "Tor" tiers.)
2. **Delete the orphans** (~700 LOC total):
   - `stealth_engine.py`, `browser_pool.py`, `lazy_load.py`, `shadow_dom.py` — unused.
   - `camoufox_integration.py` → keep as opt-in endpoint, don't try to put it in the default chain (camoufox is slow; it's for the hard cases that should be attempted after playwright fails).
   - `tls_fingerprint.py` → fold into the new Tier 1.
   - `cookie_warmer.py` → keep as opt-in endpoint; the "warming" logic is theatrical.
   - `adaptive.py` → fix the `_compute_information_gain` bug, then keep as opt-in `/v1/crawl/adaptive`.
3. **Fix `parser.py.tmp`** deletion (already in Phase 0).
4. **Fix `extractor.py:23`** — read `OLLAMA_BASE` from env (default to `http://100.104.130.92:11434` per AGENTS.md).
5. **Fix `extraction.py:240 compute_embedding`** — use `sentence-transformers/all-MiniLM-L6-v2` (384-dim) or remove the fallback dimension mismatch.
6. **Fix `ultimate_scraper.py:33-46`** — new docstring matches the 5-tier reality.
7. **Add `playwright-stealth`** to `pyproject.toml`. Remove `stealth_engine.py`.
8. **Templates audit** — write `templates/test_live_against_real_sites.py` (not against homepages, against article/product URLs). Pass-fail dashboard. Strike the bottom 50%. Document the survivors in `TEMPLATES-COVERAGE.md`.

### Phase 3 — API surface: split the monolith, add auth, deploy MCP+x402 (2 weeks)

Goal: every endpoint in `routers/`, MCP live, x402 live.

1. **Router split (1.5 weeks)** — split `api.py` from 4,668 LOC to ~500 LOC (`api.py:routes` includes only the `FastAPI()` instantiation + global middleware; everything moves to `routers/<domain>.py`). Sequence:
   1. `routers/health.py`, `routers/scraping.py`, `routers/templates.py` (Phase 0 done).
   2. `routers/extraction.py` (linking, extraction, vision).
   3. `routers/sessions.py` (sessions, browser, automation).
   4. `routers/agents.py` (quality, freshness, structure, monitor, seo).
   5. `routers/integrations.py` (destinations, crm, commerce, email, agency).
   6. `routers/compliance.py` (gdpr, compliance, training, review).
   7. `routers/admin.py` (costing, alerts, jobs, pipeline, recorder, transform).
   8. `routers/payments.py` (x402 + paid endpoints).
   9. `routers/mcp.py` (merged from mcp_production.py's HTTP transport).
   10. `routers/auth.py` (auth, agency client auth, api key).
2. **Fix silent error swallowing (the 159 `except Exception`).** Move to `errors.py` hierarchy: `PryHTTPError`, `PryAuthError`, `PryScrapeError`, `PryStorageError`, etc. Use `logger.exception(...)` with request_id.
3. **Auth gate** — apply JWT/API-key middleware to all mutating endpoints except `/health`, `/live`, `/ready`, `/metrics`, `/docs`, `/openapi.json`, `/v1/templates/*`, `/v1/x402/pricing`. Move away from per-endpoint `@require_auth`.
4. **Deploy `mcp_production.py`** to Talos on `/srv/pry/mcp/` as a sibling server (port 8006). Update `docker-compose.yml`. Make `pry mcp serve` actually work (wire click dispatcher).
5. **Set `PRY_X402_WALLET` and `PRY_X402_FACILITATOR`** on Talos. Verify endpoints transition from 503 → 402 → 200.
6. **Add `routers/mcp.py` HTTP+SSE transport** to the main app (alternative to sibling process). Route `/sse`, `/messages/:sid`.
7. **Update `smithery.yaml` / `glama.json` / `pulsemcp.json`** with the live `mcp.pry.dev` URL and remove the suspended `cryptorugmuncher` GitHub URLs.

### Phase 4 — Apify launch (1.5 weeks, parallelizable with Phase 3)

Goal: Pry ships as an Apify Actor.

1. **Minimal Apify Actor (1 day)**:
   - `pip install apify>=2.0` (added in Phase 0).
   - `.actor/actor.json` — manifest.
   - `.actor/Dockerfile` — `FROM apify/actor-python-playwright:3.12-slim`, copy src/, pyproject, templates.
   - `src/main.py` — `async def main(): async with Actor: actor_input = await Actor.get_input(); ...; await Actor.push_data(...)`.
   - Write the input schema from `apify_schema.py` to `.actor/input_schema.json` via a `make actor-build` target.
2. **Strip local-only deps from the Actor image** — FlareSolverr (`docker-compose.yml`), Tor profile, depend on `curl_cffi` + `playwright` + cloudscraper + archive.org only.
3. **Push to Apify (1 day)** — `apify login`, `apify push`, verify `apify call` runs.
4. **Publish on Apify Store** — name `pry-scraper`, category "Developers", monetization via `apify monetize` (per-scrape pricing).
5. **Public listing (1 day)** — write actor description, screenshots, README. Submit to Apify for review.
6. **Actor README** — what inputs it accepts, what outputs it produces, error modes, examples.

### Phase 5 — Production polish + observability + tests (1.5 weeks)

Goal: ship-ready. 80% test coverage that's real, not theater. Observability that operations trusts.

1. **Real tests** — rewrite `test_advanced_scraping.py`, `test_referrals.py`, etc. to test BEHAVIOR not presence. Use `respx` for httpx mocking. Use `freezegun` for time-travel. Use `vcr.py` for recording live API calls.
2. **Coverage gate** — `--cov-fail-under=80` in `pyproject.toml [tool.coverage.run]`. CI fails below threshold.
3. **CI/CD** — split the single workflow into:
   - `lint.yml` (ruff + format check)
   - `typecheck.yml` (mypy strict, no disabled codes)
   - `test.yml` (pytest with respx, no real network; installs playwright + tesseract; coverage gate)
   - `security.yml` (bandit + pip-audit + gitleaks)
   - `docker.yml` (build + smoke)
   - `deploy.yml` (push to Talos, run healthcheck)
   - **Self-hosted runner** on Talos (`100.104.130.92`) — no GitHub Actions minutes.
4. **Observability** — wire OpenTelemetry OTLP exporter to a real backend (Grafana Cloud free tier, or local Tempo on Talos). Real distributed tracing across `httpx`, `playwright`, `db`.
5. **Structured logging** — JSON logs with `request_id`, `tenant_id`, `user_id`, `route`, `latency_ms`. Forward to Loki (free tier) or write to journald on Talos.
6. **OpenAPI SDK generation** — drop `pry_sdk.py` (130 LOC hand-maintained), generate `pry-sdk-python` from `openapi.json`. Generate `pry-sdk-js`. Publish to PyPI as `pry-scraper-sdk` and to npm as `@pry/sdk`.
7. **Deprecation log** — what was deleted in this rewrite:
   - `parser.py.tmp`
   - `stealth_engine.py`, `browser_pool.py`, `lazy_load.py`, `shadow_dom.py` (4 modules, ~518 LOC)
   - The 5 dead tiers in `ultimate_scraper.py`
   - The flaky `tests/test_api.py:46-69`
   - The 159 broad `except Exception`
   - The `.pry_sdk` reference
   - `data.db` orphan
8. **Decorate** — add `READY` / `LIVE` to README badges from `/health` and `/live` responses.

---

## 7. Apify-publishable cut (smallest viable for Apify)

If the only goal is "ship on Apify first, full production later", cut Phases 1, 3, 5 and just do:

| Task | Effort |
|---|---|
| Phase 0 stop-the-bleeding (3 days) | 3d |
| Phase 4 (all 1.5 weeks) | 7d |
| Phase 2 minimum: fix the missing deps, replace `stealth_engine.py` with `playwright-stealth`, drop the dead tiers | 4d |
| **Total** | **~2 weeks** for Apify-publishable |

Production polish follows the Apify launch. Pricing on Apify: per-scrape (e.g. $0.005/scrape, free monthly tier for viral growth).

---

## 8. The "2026 AI-Forward" code style we want

Distributed in the codebase by Phase 5. Reference for new modules:

```
pry/
├── README.md                       # honest, no hallucinated features
├── AGENTS.md
├── pyproject.toml                  # uv-managed, pinned
├── Dockerfile                      # apify/actor-python for Apify
├── .dockerignore
├── alembic/                        # NEW — migrations
│   ├── env.py
│   ├── script.py.mako
│   └── versions/
├── alembic.ini
├── src/
│   └── pry/                        # NEW — proper package layout
│       ├── __init__.py
│       ├── py.typed
│       ├── main.py                 # FastAPI entrypoint
│       ├── api/
│       │   ├── __init__.py
│       │   ├── app.py              # FastAPI() instance only
│       │   ├── deps.py             # shared dependencies
│       │   ├── errors.py
│       │   ├── routers/            # one file per domain
│       │   │   ├── health.py
│       │   │   ├── scraping.py
│       │   │   ├── templates.py
│       │   │   ├── extraction.py
│       │   │   ├── sessions.py
│       │   │   ├── agents.py
│       │   │   ├── integrations.py
│       │   │   ├── compliance.py
│       │   │   ├── admin.py
│       │   │   ├── payments.py
│       │   │   ├── mcp.py
│       │   │   └── auth.py
│       │   ├── middleware/
│       │   │   ├── auth.py
│       │   │   ├── ratelimit.py
│       │   │   ├── request_id.py
│       │   │   ├── logging.py
│       │   │   └── errors.py
│       │   └── schemas/            # pydantic models per router
│       │       ├── scraping.py
│       │       └── ...
│       ├── core/                   # pure business logic, framework-free
│       │   ├── errors.py
│       │   ├── config.py
│       │   ├── logging.py
│       │   ├── http.py             # shared httpx + curl_cffi
│       │   └── secrets.py          # gopass + env
│       ├── scrape/                 # the scraping engine
│       │   ├── fallbacks.py        # 5-tier chain
│       │   ├── parsers.py          # pdf, docx, html
│       │   ├── extractor.py        # CSS + LLM fallback
│       │   ├── templates.py
│       │   └── monitor.py          # freshness, structure
│       ├── llm/                    # unified Ollama/OpenRouter client
│       │   ├── client.py
│       │   ├── extraction.py
│       │   ├── vision.py
│       │   └── schemas.py           # structured-output
│       ├── db/
│       │   ├── engine.py            # AsyncSession with aiosqlite/postgres
│       │   ├── migrations.py
│       │   ├── models.py
│       │   └── repos/               # one per model
│       ├── billing/                 # x402 implementation
│       │   ├── protocol.py
│       │   ├── middleware.py
│       │   └── ledger.py
│       ├── integrations/
│       │   ├── shopify.py
│       │   ├── woocommerce.py
│       │   ├── salesforce.py
│       │   ├── hubspot.py
│       │   ├── pipedrive.py
│       │   ├── close.py
│       │   ├── email.py            # Gmail/Outlook
│       │   ├── slack.py
│       │   └── webhook.py
│       ├── platforms/               # publish targets
│       │   ├── apify.py            # apify actor entrypoint
│       │   ├── cloudflare.py       # MCP worker
│       │   └── mcp.py              # JSON-RPC dispatcher
│       └── cli/
│           ├── __init__.py
│           └── main.py             # click-based, replaces cli.py
├── tests/
│   ├── conftest.py
│   ├── unit/
│   ├── integration/
│   └── e2e/
├── templates/                       # 110 JSON
├── TEMPLATES-COVERAGE.md           # NEW: honest coverage
├── docs/
└── .actor/                          # NEW: for Apify
```

**Key changes that make this 2026-current:**

1. **`src/` layout** — proper Python package, enables `pyproject.toml [tool.setuptools.packages.find]`, prevents import ambiguity.
2. **`py.typed` marker** — third-party users get type checking.
3. **`core/`** — framework-free business logic, testable without spinning up FastAPI.
4. **One router file per domain** — each < 500 LOC, one clear responsibility.
5. **`db/repos/` pattern** — every SQL call goes through a repo function. No `session_scope()` calls scattered through business logic.
6. **`middleware/`** explicit — auth, ratelimit, request_id, logging, error-handling all separate files.
7. **`llm/` separated from `core/`** — LLM extraction, vision, structured output are first-class concerns.
8. **`platforms/` separated from `integrations/`** — where we publish (Apify, Cloudflare, MCP server) is different from what we integrate with (Shopify, Salesforce).
9. **`alembic/` migrations** — version-controlled schema. No more `Base.metadata.create_all()`.
10. **`tests/unit/`, `tests/integration/`, `tests/e2e/`** — honest test layering.

---

## 9. Open-Source Install List (Phase-keyed)

### Phase 0 (3 days)
- `cloudscraper`, `aiohttp-socks[aiohttp]`, `undetected_chromedriver`, `pyjwt`, `apify>=2.0`
- `respx` (or `pytest-httpx`)
- `freezegun`

### Phase 1 (1.5 weeks)
- `alembic`
- `aiosqlite` is already installed; just import it

### Phase 2 (1.5 weeks)
- `playwright-stealth`
- `curl_cffi` is already installed; use it
- `sentence-transformers` (for real embeddings) — or drop embedding fallback

### Phase 3 (2 weeks)
- `fastapi[all]` (orjson, asyncpg, jinja2 extras)

### Phase 4 (1.5 weeks, parallelizable)
- `apify` (added in Phase 0)

### Phase 5 (1.5 weeks)
- `pip-audit`
- `vcr.py` or `pytest-recording`
- `mutmut` or `cosmic-ray`
- `pytest-benchmark` (perf gates)
- OpenTelemetry OTLP exporter
- `uv` (replaces pip)
- `pre-commit` cache, `act`

---

## 10. KPIs / Definition of Done

| KPI | Today | Target at end of Phase 5 |
|---|---|---|
| Tests passing | 498/500 (2 flaky) | 800+/800 stable |
| Test coverage | 60% (no gate) | ≥ 80% (CI gated) |
| Endpoints with auth | 2/~190 | all mutating |
| Endpoints with rate-limit | ~6 | all |
| Endpoints with structured logging | 0 | all |
| Endpoints with tracing | 0 | all |
| Dead code (LOC) | ~700 (orphaned modules) + ~2,000 (hallucinated features) | <100 |
| Anti-detection fallback tiers working end-to-end | 5/10 | 5/5 (consolidated) |
| Templates passing live-URL test | ~21-30 | 50+ |
| Concurrent writers without corruption | NO | YES (atomic+locked or SQLite) |
| Schema migrations | Hand-edit or wipe | Alembic, versioned |
| CI minutes burned | GitHub-hosted ($$) | Self-hosted Talos ($0) |
| Deployable to Apify store | NO | YES |
| Production uptime | TBD | 99.5% / 30d rolling |
| p95 latency (scrape) | TBD | < 5s for direct, < 30s with playwright |

---

## 11. What to do **tomorrow** (1 working day)

If the user picks one day to make Pry significantly better:

1. Delete `parser.py.tmp` (5 min)
2. Fix `cli.py:339-340` infinite recursion (15 min)
3. Add missing deps to `pyproject.toml`: `cloudscraper`, `aiohttp-socks[aiohttp]`, `undetected_chromedriver`, `pyjwt`, `apify>=2.0` (10 min)
4. Fix `docker-compose.yml:18` env var name (2 min)
5. Fix `Dockerfile` healthcheck to not use `curl` (10 min)
6. Add `USER` directive to `Dockerfile` (5 min)
7. Fix `tests/test_api.py:46-69` to mock `PryScraper` (15 min)
8. Fix `ultimate_scraper.py:33-46` docstring to honest 5-tier (10 min)
9. Migrate `quality.py:347` from JSON to SQLite via existing `QualityCheck` model (already in `db.py:197`) (1 hr)
10. Same for `freshness.py` (1 hr)
11. Pin `uv.lock` (30 min)
12. Set `--cov-fail-under=60` to start (we can tighten later) (5 min)

End of day: **no flaky tests, no infinite recursion, no dead tiers claiming to work, no JSON races on `quality/freshness`, real coverage gate, Dockerfile runs as non-root.**

That's the user's "day to make Pry significantly better" answer. Then Phase 0 takes another 2 days, then Phase 1 starts. Apify-publishable in 2 weeks.

---

## 12. Risks

1. **Phase 2 anti-detection consolidation** — by removing 7 modules, we lose the "kitchen sink" cover. Mitigation: keep `camoufox_integration.py` and `cookie_warmer.py` as opt-in endpoints (operators can call them manually for hard cases).
2. **Phase 3 router split** — risk of breaking existing endpoints. Mitigation: split one router at a time, deploy each, run smoke test (`curl /health` + 1 representative endpoint per router).
3. **Phase 4 Apify first-impressions** — Apify Store review takes 1-2 weeks. Start early; in parallel with Phase 1/2.
4. **Phase 5 self-hosted CI runner** — Talos has other workloads; registering a runner should not consume > 10% CPU. Run on Hydra if Talos is busy.
5. **Multi-worker with cookies/sessions** — moving from JSON files to SQLite is the easy part. Sharing browser sessions across workers requires Redis or sticky sessions. Defer to v3.1.
6. **OpenAI/Anthropic rate limits** — depending on LLM extraction (real, not regex) means we burn keys. Mitigation: local Ollama first; OpenAI only on cache-miss.

---

## 13. Decision log (what this audit is committing to)

- **ADR-0003**: Consolidate anti-detection to `curl_cffi` + `playwright-stealth`. Delete 7 orphan modules.
- **ADR-0004**: SQLite for single-node, Postgres for multi-host. Alembic for migrations. Atomic writes everywhere.
- **ADR-0005**: `src/pry/` layout. One router per domain. No file > 500 LOC (enforced by CI).
- **ADR-0006**: Real LLM extraction by default — Ollama first, OpenRouter fallback. Regex fallback only on cache-miss + LLM error.
- **ADR-0007**: JWT auth + per-tenant quota on all mutating endpoints. `/health` and `/metrics` public.
- **ADR-0008**: Apify is the primary distribution post-Phase 4. Self-hosted Talos deploy remains supported.
- **ADR-0009**: Tests must mock network (`respx`) and time (`freezegun`). No real network in CI.
- **ADR-0010**: Coverage gate `--cov-fail-under=80` after Phase 5. No merging below threshold.
- **ADR-0011**: Self-hosted CI runner on Talos. No GitHub Actions minutes.
- **ADR-0012**: OpenTelemetry OTLP exporter. Loki for logs. Prometheus for metrics. Grafana dashboards on Grafana Cloud (free) or Talos.

---

## 14. Links

- Original audit (2026-06-30, Sonnet 4.5): `AUDIT.md`
- This audit (2026-07-06): `AUDIT-2026-Q3.md`
- Production roadmap: `PRODUCTION-PLAN.md` (this file)
- Status: `STATUS.md`
- Sprint plan: `PLAN.md`
- AGENTS contract: `AGENTS.md`
- Standards: https://git.rugmunch.io/RugMunchMedia/standards