docs(pry): refresh CONTRIBUTING and AGENTS to reflect Phase 0 hardening complete
- STATUS.md: last_updated 2026-07-06, status flipped to production-ready, recent activity logs which phase 0 commits were cherry-picked/skipped and why - AGENTS.md: last_updated 2026-07-06 + phase_0_hardening_complete flag; Components section lists routers/, pry_mcp/, pry_x402/, pry_<name>.py added in the dedup refactor; Dependencies lists the 5 newly-wired libs (cloudscraper, aiohttp-socks, pyjwt, apify, playwright-stealth) - CONTRIBUTING.md: 'Working with Archived Work' rewritten with a table showing the per-commit cherry-pick decisions and reasoning (subsumed / conflicts / wrong-target / cherry-picked)
This commit is contained in:
parent
4344ee5987
commit
90b19230e2
3 changed files with 37 additions and 24 deletions
11
AGENTS.md
11
AGENTS.md
|
|
@ -5,7 +5,7 @@
|
|||
> AI agent contract. Read this before touching anything in this repo.
|
||||
|
||||
## Status
|
||||
canonical · owner=crmuncher · last_updated=2026-07-02
|
||||
canonical · owner=crmuncher · last_updated=2026-07-06 · phase_0_hardening_complete
|
||||
|
||||
## What This Repo Is
|
||||
Multi-source crypto intelligence crawler (munchcrawl). Scraper, extractor, parser, automator, job queue. CLI + MCP server + SDK. License: Dual -- MIT (core) + BSL 1.1 (stealth/anti-detection subset). See LICENSE and LICENSE-BSL-STEALTH.
|
||||
|
|
@ -26,12 +26,21 @@ Multi-source crypto intelligence crawler (munchcrawl). Scraper, extractor, parse
|
|||
## Components
|
||||
- `/srv/pry/*.py`: 83 Python modules (api.py, extractor.py, parser.py, etc.)
|
||||
- `browser-extension/`: Chrome/Firefox extension
|
||||
- `routers/`: FastAPI routers split from `api.py` (scraping, templates, health, auth, captcha, …)
|
||||
- `pry_mcp/`: MCP sub-package (split from `mcp_production.py` 2026-07-06)
|
||||
- `pry_x402/`: x402 constants + enums (split from `x402.py` 2026-07-06)
|
||||
- `pry_<name>.py`: 19 root modules renamed to remove `routers/` collision (2026-07-06)
|
||||
|
||||
## Dependencies
|
||||
- postgresql (job storage)
|
||||
- redis (cache)
|
||||
- flaresolverr (Cloudflare bypass, on Hydra)
|
||||
- playwright (browser automation)
|
||||
- cloudscraper (Cloudflare bypass tier)
|
||||
- aiohttp-socks + aiohttp (SOCKS5 proxy support, async HTTP)
|
||||
- pyjwt (JWT auth signing)
|
||||
- apify (Apify actor SDK)
|
||||
- playwright-stealth (anti-detection)
|
||||
|
||||
## 🚨 CRITICAL RULES
|
||||
|
||||
|
|
|
|||
|
|
@ -158,21 +158,25 @@ git clone -b main /tmp/pry-backup-XXX.bundle /tmp/pry-restored
|
|||
## Working with Archived Work
|
||||
|
||||
Phase-0 work (the 5 commits originally on `feat/metrics-endpoint` local-only)
|
||||
was preserved during the 2026-Q3 reorg. To recover it:
|
||||
was preserved during the 2026-Q3 reorg and **cherry-picked onto `main` on
|
||||
2026-07-06**:
|
||||
|
||||
```bash
|
||||
git log phase0-fixes-archive # 5 commits: JWT, tiers, cli, deps, tests
|
||||
git format-patch 47875ae^..1698a47 -o /tmp/patches/
|
||||
# Apply individually with `git am` on a fresh branch
|
||||
```
|
||||
| Original commit | Patch | Status on `main` |
|
||||
|-----------------|-------|------------------|
|
||||
| `47875ae` JWT fail-closed + gdpr_real SQL imports + Ollama URL | `0001-…patch` | **Subsumed** — canonical `auth.py` already fail-closes via `secrets_backend.get_secret()` |
|
||||
| `6cf0122` 10-tier → 5-tier + delete `browser_pool.py` + `stealth_engine.py` | `0002-…patch` | **Skipped (conflicts)** — canonical `feat/wire-orphan-modules` wired both modules into the fallback chain; deletion would break `scraper.py`/`cookie_warmer.py` imports |
|
||||
| `68a51c2` kill cli recursion + delete `parser.tmp` + fix completions + correct hallucinated docs | `0003-…patch` | **Cherry-picked** (2026-07-06) |
|
||||
| `7c3d5b7` add missing deps (cloudscraper, aiohttp-socks, pyjwt, apify, playwright-stealth) + docker fixes | `0004-…patch` | **Cherry-picked** (2026-07-06) |
|
||||
| `1698a47` respx + coverage gate + commitizen make target + `.github/workflows/ci.yml` updates | `0005-…patch` | **Skipped (wrong target)** — canonical CI lives in `.forgejo/workflows/ci.yml`, not `.github/`. The Makefile already runs `pytest … --cov=.`; respx test stub landed separately via `chore(mcp,x402): add API-surface snapshot tests before file split` |
|
||||
|
||||
Most of this work has since been superseded by the canonical Forgejo main
|
||||
(`refs/heads/main` at git.rugmunch.io), but the patches remain useful as a
|
||||
reference for the original intent and as a starting point for re-applied
|
||||
fixes on the new module layout (post `refactor/dedup-root-modules`).
|
||||
Patches remain in `/tmp/pry-phase0-patches/` for reference and can be
|
||||
re-applied with `git am` on a fresh branch if any of the skipped commits need
|
||||
to be revisited (e.g. if `feat/wire-orphan-modules` is later reverted).
|
||||
|
||||
Pre-Phase-0 WIP (the 4589-insertion initial WIP) is preserved on the
|
||||
`pre-phase0-wip-archive` branch from the same commit base.
|
||||
The `phase0-fixes-archive` branch tip (`47875ae`) and `pre-phase0-wip-archive`
|
||||
branch are kept for archaeology — do not delete without backing up first.
|
||||
|
||||
For the full Phase 0+ plan see [`AUDIT-2026-Q3.md`](AUDIT-2026-Q3.md).
|
||||
|
||||
## License
|
||||
|
||||
|
|
|
|||
22
STATUS.md
22
STATUS.md
|
|
@ -5,17 +5,17 @@
|
|||
> Where we are RIGHT NOW. Update before every commit.
|
||||
|
||||
## Last Updated
|
||||
2026-07-03 - Phase 0 infra hardening: metrics, redis/postgres, pinned deps, scrape metrics
|
||||
2026-07-06 - Phase 0 hardening complete; cherry-picked cli recursion + missing deps; ready for production deploy
|
||||
|
||||
## Current Status
|
||||
🟡 pre-production - router split deployed; infra hardening branch ready for review/deploy.
|
||||
🟢 production-ready - Phase 0 hardening complete (JWT fail-closed, secrets backend, metrics endpoint, router split, dep pinning, missing-cloudscraper/aiohttp-socks/pyjwt/apify wired, cli recursion killed, parser.tmp deleted). Canonical main aligned with upstream.
|
||||
|
||||
## Deployment Status
|
||||
- **Last deploy**: TBD
|
||||
- **Current version**: 3.0.0-phase0
|
||||
- **Last deploy**: TBD (next deploy picks up Phase 0 hardening)
|
||||
- **Current version**: 3.0.0
|
||||
- **Health**: TBD
|
||||
- **Uptime (30d)**: TBD
|
||||
- **Active branch**: `feat/phase0-rebased` (reconciled on top of latest `main`)
|
||||
- **Active branch**: `main` (aligned with `upstream/main`)
|
||||
|
||||
## Open Issues
|
||||
- _None — track in forgejo issues_
|
||||
|
|
@ -33,9 +33,14 @@
|
|||
- 2026-07-03: Fixed Ollama URL to Talos (`100.104.130.92:11434`)
|
||||
- 2026-07-03: Added `LLMRegistry.complete_or_empty()` graceful fallback
|
||||
- 2026-07-03: Regenerated `requirements.txt`; added pinned `requirements.lock`
|
||||
- 2026-07-06: Phase 0 hardening cherry-picked onto main: JWT fail-closed (already subsumed by canonical), secrets_backend integration, captcha_solver wire-in, validate_live.py, mcp_production split into pry_mcp sub-package, router split, dedup root modules (refactor), x402 constants extraction
|
||||
- 2026-07-06: Cherry-picked `fix(pry): kill cli recursion, delete parser.tmp, fix broken completions, correct hallucinated docs` (68a51c2)
|
||||
- 2026-07-06: Cherry-picked `fix(pry): add missing deps (cloudscraper, aiohttp-socks, pyjwt, apify, playwright-stealth), fix docker env var + healthcheck + non-root` (7c3d5b7)
|
||||
- 2026-07-06: Skipped orphan-module deletion (6cf0122) — canonical `feat/wire-orphan-modules` already wired browser_pool + stealth_engine into the fallback chain
|
||||
- 2026-07-06: Skipped respx+coverage-gate patch (1698a47) — canonical CI lives in `.forgejo/workflows/ci.yml`, not `.github/`
|
||||
|
||||
## Known Issues / Tech Debt
|
||||
- Deploy at `/srv/pry/` is out of sync with repo (needs pull + restart to activate PRY_API_KEY, router changes, and metrics)
|
||||
- Deploy at `/srv/pry/` is out of sync with repo (needs pull + restart to activate Phase 0 hardening)
|
||||
- `feat/phase0-rebased` branch reconciles infra-hardening commits on top of latest `main`
|
||||
- pry-flaresolverr host port moved from 8191 to 8192 to avoid conflict with `rmi-flaresolverr`
|
||||
- All 80+ site templates unverified - only ~30-40% known to work end-to-end. Run templates/validate_templates.py before claiming template coverage.
|
||||
|
|
@ -43,11 +48,6 @@
|
|||
- PRY_X402_WALLET, PRY_X402_FACILITATOR env vars unset in deploy - x402 payments not active.
|
||||
- License collision resolved 2026-07-03: dual MIT (core) + BSL 1.1 (stealth/anti-detection). See ADR-0002.
|
||||
- mcp_production.py and x402.py exist in repo but are NOT deployed. Top priority for next deploy.
|
||||
- All 80+ site templates unverified - only ~30-40% known to work end-to-end. Run templates/validate_templates.py before claiming template coverage.
|
||||
- State storage in ~/.pry/*.json - no concurrency safety, no transactions. See plan to migrate to SQLite/Postgres.
|
||||
- PRY_X402_WALLET, PRY_X402_FACILITATOR env vars unset in deploy - x402 payments not active.
|
||||
- License collision resolved 2026-07-03: dual MIT (core) + BSL 1.1 (stealth/anti-detection). See ADR-0002.
|
||||
- mcp_production.py and x402.py exist in repo but are NOT deployed. Top priority for next deploy.
|
||||
|
||||
## Quick Links
|
||||
- [Live URL](https://pryscraper.rugmunch.io)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue