diff --git a/.gitleaks.toml b/.gitleaks.toml new file mode 100644 index 0000000..554452a --- /dev/null +++ b/.gitleaks.toml @@ -0,0 +1,7 @@ +allowlist: + description: "Pryscraper allowlist — test fixtures, docs, known public addresses" + regexTarget: match + paths: + - tests/ + - docs/ + - "*.md" diff --git a/.semgrep/rules.yaml b/.semgrep/rules.yaml new file mode 100644 index 0000000..b87fe1a --- /dev/null +++ b/.semgrep/rules.yaml @@ -0,0 +1,25 @@ +rules: + - id: no-print-in-production + pattern: print(...) + message: "Remove print() before committing to production code. Use structured logging instead." + severity: WARNING + languages: [python] + paths: + exclude: + - tests/ + - "**/test_*.py" + + - id: no-os-system + pattern: os.system(...) + message: "os.system() is dangerous. Use subprocess.run() with shell=False instead." + severity: ERROR + languages: [python] + + - id: no-eval-exec + patterns: + - pattern-either: + - pattern: eval(...) + - pattern: exec(...) + message: "eval()/exec() are dangerous and should not be used in production code." + severity: ERROR + languages: [python] diff --git a/AGENTS.md b/AGENTS.md index a6c2e43..35cc0a9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,7 +8,7 @@ canonical · owner=crmuncher · last_updated=2026-07-06 · phase_0_hardening_complete ## What This Repo Is -Multi-source crypto intelligence crawler (munchcrawl). Scraper, extractor, parser, automator, job queue. CLI + MCP server + SDK. License: Dual -- MIT (core) + BSL 1.1 (stealth/anti-detection subset). See LICENSE and LICENSE-BSL-STEALTH. +Multi-source crypto intelligence crawler (pryscraper). Scraper, extractor, parser, automator, job queue. CLI + MCP server + SDK. License: Dual -- MIT (core) + BSL 1.1 (stealth/anti-detection subset). See LICENSE and LICENSE-BSL-STEALTH. ## Type `backend` · language=Python 3.12 + FastAPI diff --git a/AUDIT-2026-Q3.md b/AUDIT-2026-Q3.md index 6222dda..9319d8c 100644 --- a/AUDIT-2026-Q3.md +++ b/AUDIT-2026-Q3.md @@ -40,13 +40,13 @@ goal: ship Pry as production-grade + Apify-publishable within Q3 2026 | **Shopify app** | Shopify app | 60-line scaffold. `@shopify/shopify-api` declared but **never imported**. No OAuth callback, no token exchange, no HMAC verify. | 🔴 not publishable | | **Tests** | "500 tests passing" | 500 collected, 498 pass, 2 fail (one is a flaky network test that makes a real `httpx` call to `example.com`). 60% coverage, no `--cov-fail-under`. **Most tests assert trivial things ("string in dict")** — `test_advanced_scraping.py` is 17 attribute-presence tests. | 🟡 theater | | **Test coverage** | ">80% enforced" | **60% actual**, no threshold check in CI. `TESTING.md` describes `tests/unit/`, `tests/integration/`, `make test-unit`, `make test-cov` — **none exist.** | 🔴 docs lie | -| **Docs** | README, USAGE, ARCHITECTURE, AUDIT | Three sets of docs. Some honest (AUDIT.md, STATUS.md). Some lying (TESTING.md describes directories that don't exist; FEATURES.md claims Zoho, `.munchcrawl_sdk`, export formats that aren't there). | 🟡 mixed | +| **Docs** | README, USAGE, ARCHITECTURE, AUDIT | Three sets of docs. Some honest (AUDIT.md, STATUS.md). Some lying (TESTING.md describes directories that don't exist; FEATURES.md claims Zoho, `.pry_sdk`, export formats that aren't there). | 🟡 mixed | | **CI** | GitHub Actions CI | 1 workflow, 4 jobs (lint/typecheck/test/security). **Doesn't install playwright/tesseract**, so any real integration test fails on CI. **Doesn't pin ruff/mypy** versions. No deploy step. | 🟡 minimal | | **Dependencies** | `requirements.txt` + `pyproject.toml` | 23 deps, all `>=` with no upper bound. **`aiohttp_socks` and `aiohttp` missing** despite Tor tier importing them. **`pyjwt` missing** despite `auth.py` importing it. | 🟡 destructive pin | | **Dockerfile** | multi-stage | Real, but **no USER directive (runs as root)**, **healthcheck uses `curl` which isn't in runtime stage** — container restart-loops. | 🟡 broken healthcheck | | **docker-compose** | pry + flaresolverr + tor | **Sets `FLARESOLVERR_URL` (wrong var name; should be `PRY_FLARESOLVERR_URL`)** — Cloudflare bypass silently fails. Same `curl`-in-healthcheck bug. | 🔴 dead FlareSolverr path | | **CLI** | `pry` command | `pry --help` **crashes with infinite recursion** (cli.py:339-340). `pry mcp serve` is dead code (only `proxy` routes to click). `pry completions` writes a broken eval string. | 🔴 broken | -| **SDK** | `pry_sdk.py` (async+sync) | Real `PryCrawl`/`PryCrawlSync`, hardcoded endpoint paths, raw-dict returns. Doesn't use shared `client.http_client`. `munchcrawl_sdk.py` referenced in old docs — **doesn't exist**. | 🟢 real | +| **SDK** | `pry_sdk.py` (async+sync) | Real `PryCrawl`/`PryCrawlSync`, hardcoded endpoint paths, raw-dict returns. Doesn't use shared `client.http_client`. `pry_sdk.py` referenced in old docs — **doesn't exist**. | 🟢 real | | **Proxies** | 10-tier with residential pool | Catalogs 3 free + 10 premium providers, calls Brightdata via signup link. **No actual credential rotation**, no gopass lookup, no proxy health scoring. Tor tier imports `aiohttp_socks` which is missing. | 🟡 stub | | **Referral revenue** | Jupiter 50bps, Hyperliquid 50bps, etc. (per AGENTS.md) | **Pry has nothing to do with that catalog.** RMI-side. Pry's `referrals.py` covers 14 categories (exchange, LLM, hosting, proxy, etc.) with JWT-shaped attribution that doesn't actually verify the code was used. Cookie-only attribution. | 🟡 toy | | **GDPR** | "consent, deletion, retention, audit" | JSON files + two modules (`gdpr.py`, `gdpr_real.py`). Deletion DOES walk files and unlink. **`gdpr_real.py:138` imports `ApiKey`/`QualityCheckRecord`/`UsageRecord` from db.py — those models don't exist, raises `ImportError` at runtime.** | 🟡 half-real | @@ -115,10 +115,10 @@ goal: ship Pry as production-grade + Apify-publishable within Q3 2026 - `cli.py:225-246` `pry completions` writes broken eval string — no `_PRY_COMPLETE` handler exists. - `cli.py:91-94` `_spinner` thread never `join()`ed. - `pry_sdk.py` is real (`PryCrawl`, `PryCrawlSync`), but creates its own `httpx.AsyncClient` per call instead of sharing `client.http_client`. -- `munchcrawl_sdk.py` is referenced in old docs but **doesn't exist.** +- `pry_sdk.py` is referenced in old docs but **doesn't exist.** - `README.md:11` says `pip install pry` — PyPI `pry` is owned by an unrelated project. - `USAGE.md` and `README.md` disagree on port (8002 vs 8005) without clarifying the Docker mapping. -- `FEATURES.md` claims things that don't exist: `Zoho CRM`, `munchcrawl_sdk`, `export RSS/TXT/SQL`, `/v1/templates/generate`. +- `FEATURES.md` claims things that don't exist: `Zoho CRM`, `pry_sdk.py`, `export RSS/TXT/SQL`, `/v1/templates/generate`. ### F. Apify / Integrations / Browser / WP / Shopify - `apify_schema.py` builds a dict that looks like Apify input schema. **Never writes to disk, never instantiates `Actor`.** No `.actor/`, no `src/main.py`, no `apify` package. @@ -194,7 +194,7 @@ goal: ship Pry as production-grade + Apify-publishable within Q3 2026 | "Tor proxy routing" | README, FEATURES.md | Imports lib that's missing — runtime crash. | | "Multi-tenant billing" | AUDIT.md | Zero billing code. | | "Zoho CRM sync" | FEATURES.md | Grep returns 0 matches in code. | -| `munchcrawl_sdk.py` | (old) | Doesn't exist. | +| `pry_sdk.py` | (old) | Doesn't exist. | | Shell autocompletion | cli.py:225 | Writes broken eval string; no `_PRY_COMPLETE` handler. | | `/v1/templates/generate` | FEATURES.md | Not in api.py or routers. | | `pry mcp serve` / `pry mcp info` | cli.py:375-403 | Help says it exists; main() never routes to it. | @@ -326,7 +326,7 @@ Goal: ship-ready. 80% test coverage that's real, not theater. Observability that - The 5 dead tiers in `ultimate_scraper.py` - The flaky `tests/test_api.py:46-69` - The 159 broad `except Exception` - - The `.munchcrawl_sdk` reference + - The `.pry_sdk` reference - `data.db` orphan 8. **Decorate** — add `READY` / `LIVE` to README badges from `/health` and `/live` responses. diff --git a/docs/adr/0002-dual-licensing.md b/docs/adr/0002-dual-licensing.md index 831a568..c19aab3 100644 --- a/docs/adr/0002-dual-licensing.md +++ b/docs/adr/0002-dual-licensing.md @@ -10,7 +10,7 @@ Accepted ## Context -The Pry (munchcrawl) project was initially released under a full proprietary license. This licensing model resulted in zero community engagement, zero external contributors, and zero GitHub stars. In the competitive web scraping and data extraction market, open source projects such as Firecrawl and Crawl4AI have achieved dominant positions by leveraging MIT licensing to build strong community moats, attract contributors, and establish trust with users. +The Pry project was initially released under a full proprietary license. This licensing model resulted in zero community engagement, zero external contributors, and zero GitHub stars. In the competitive web scraping and data extraction market, open source projects such as Firecrawl and Crawl4AI have achieved dominant positions by leveraging MIT licensing to build strong community moats, attract contributors, and establish trust with users. The existing LICENSING_PRICING_STRATEGY.md document previously listed Pry as PROPRIETARY. This approach proved incompatible with the project's goal of achieving widespread adoption, particularly for the Model Context Protocol (MCP) server and template marketplace components that benefit from community contributions and third-party integrations.