RugMunchMedia/pryscraper
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(pry): kill cli recursion, delete parser.tmp, fix broken completions, correct hallucinated docs

Browse source
This commit is contained in:
Crypto Rug Munch 2026-07-06 18:26:03 +07:00
parent 68f1690ede
commit c6194ca444
5 changed files with 93 additions and 73 deletions
Download patch file Download diff file Expand all files Collapse all files

16
FEATURES.md
View file

@ -30,7 +30,7 @@ last_updated: 2026-06-30
| Browser pool + pre-warming | `browser_pool.py` | Internal | ✅ |
| Stealth engine (6 JS scripts) | `stealth_engine.py` | Internal | ✅ |
| Stealth scripts | `stealth_scripts/` | Injected | ✅ |
| Tor proxy routing | `network.py` | `POST /v1/config/profile/tor` | ✅ |
| Tor proxy routing (tier documented but disabled) | `network.py` | `POST /v1/config/profile/tor` | ⚠️ `aiohttp-socks` missing |
| SOCKS5 proxy support | `network.py` | Config | ✅ |
| User-agent rotation | `scraper.py` | Internal | ✅ |
@ -66,8 +66,10 @@ last_updated: 2026-06-30
| AI categorization | `advanced.py` | `POST /v1/categorize` | ✅ |
| GPT Action manifest | `ai_plugin.py` | `GET /v1/ai/gpt-manifest` | ✅ |
| MCP server config | `ai_plugin.py` | `GET /v1/ai/mcp-config` | ✅ |
| MCP tool discovery | `mcp_server.py` | `GET /mcp/tools` | ✅ |
| MCP tool execution | `mcp_server.py` | `POST /mcp/call` | ✅ |
| ~~MCP tool discovery~~ | ~~`mcp_server.py`~~ | ~~`GET /mcp/tools`~~ | ⚠️ not in `api.py` |
| ~~MCP tool execution~~ | ~~`mcp_server.py`~~ | ~~`POST /mcp/call`~~ | ⚠️ not in `api.py` |
> `GET /mcp/tools` and `POST /mcp/call` exist only in `mcp_production.py`, which is **not deployed**. Use `pry mcp serve` (stdio transport) for MCP access — see AUDIT-2026-Q3.md.
## Browser Automation
@ -133,7 +135,7 @@ last_updated: 2026-06-30
| Feature | Module | Endpoint | Status |
|---------|--------|----------|--------|
| Multi-format export (JSON, CSV, RSS, TXT, SQL) | `destinations.py` | `POST /v1/export` | ✅ |
| ~~Multi-format export (JSON, CSV, RSS, TXT, SQL)~~ | `destinations.py` | `POST /v1/export` | ⚠️ `pryfile.py` only does `json`/`csv`; RSS/TXT/SQL not implemented |
| Webhook delivery | `destinations.py` | Config | ✅ |
| S3 upload | `destinations.py` | Config | ✅ |
| GCS upload | `destinations.py` | Config | ✅ |
@ -149,7 +151,7 @@ last_updated: 2026-06-30
| Shopify app backend | `shopify-app/` | External | ✅ |
| Salesforce CRM sync | `crm_sync.py` | `POST /v1/crm/sync` | ✅ |
| HubSpot CRM sync | `crm_sync.py` | `POST /v1/crm/sync` | ✅ |
| Zoho CRM sync | `crm_sync.py` | `POST /v1/crm/sync` | ✅ |
| ~~Zoho CRM sync~~ | ~~`crm_sync.py`~~ | ~~`POST /v1/crm/sync`~~ | ❌ not implemented (grep: 0 matches in code) |
## Pipelines
@ -216,8 +218,8 @@ last_updated: 2026-06-30
|---------|--------|----------|--------|
| Template listing | `template_engine.py` | `GET /v1/templates` | ✅ |
| Template detail | `template_engine.py` | `GET /v1/templates/{id}` | ✅ |
| Template generation | `template_engine.py` | `POST /v1/templates/generate` | ✅ |
| Pre-built templates | 110 JSON files | `templates/` | ✅ |
| ~~Template generation~~ | ~~`template_engine.py`~~ | ~~`POST /v1/templates/generate`~~ | ❌ endpoint not registered in `api.py` |
| Pre-built templates | 110 JSON files | `templates/` | ⚠️ ~21/110 pass structural validation; ~30–40% work end-to-end (see AUDIT-2026-Q3.md) |
## SEO

2
Makefile
View file

@ -1,7 +1,7 @@
SHELL := /bin/bash
PYTHON := python3
.PHONY: help install dev lint format typecheck test security check clean commit precommit ci
.PHONY: help install dev lint format typecheck test security check clean precommit ci
help:
@echo "Pry Makefile"

18
README.md
View file

@ -7,8 +7,10 @@ Self-hosted web scraping + browser automation API. Cloudflare bypass, document p
## Quickstart
```bash
# Install
pip install pry
# Install (PyPI 'pry' is taken — install from this repo)
pip install -e .
# or:
# pip install "pry-scraper @ git+https://git.rugmunch.io/RugMunchMedia/pryscraper.git"
# Start the server
pry serve
@ -27,7 +29,7 @@ pry crawl https://docs.com --max-pages 20 -o data.json
```bash
docker compose up -d
# Pry on :8002, FlareSolverr on :8191
# Pry on :8005 (host) → :8002 (container), FlareSolverr on :8192 (host) → :8191 (container)
```
## CLI Reference
@ -96,7 +98,7 @@ Service health + cache stats + active sessions.
```python
from pry_sdk import PryCrawl
mc = PryCrawl("http://localhost:8002")
mc = PryCrawl("http://localhost:8005")
result = await mc.scrape("https://example.com")
print(result["data"]["markdown"])
```
@ -104,7 +106,7 @@ print(result["data"]["markdown"])
```python
from pry_sdk import PryCrawlSync
mc = PryCrawlSync("http://localhost:8002")
mc = PryCrawlSync("http://localhost:8005")
result = mc.scrape("https://example.com")
```
@ -141,7 +143,7 @@ Compatible with Claude, Hermes, Cursor, and any MCP client.
- **Batch processing**: Parallel scrape with templates
- **SEO analysis**: Title, meta, headings, keywords, readability
- **Schema extraction**: JSON-LD, Open Graph, microdata
- **Export formats**: JSON, CSV, RSS, TXT, SQL
- **Export formats**: JSON, CSV (RSS/TXT/SQL not yet implemented — see FEATURES.md)
- **Rate limiting**: Per-IP token bucket (default 120 RPM)
- **Caching**: LRU + Redis with TTL-based invalidation
- **WebSocket streaming**: Real-time job progress
@ -177,9 +179,9 @@ docker compose up -d
### Bare metal
```bash
pip install pry
pip install -e .
playwright install chromium
# Optional: docker run -d -p 8191:8191 ghcr.io/flaresolverr/flaresolverr
# Optional: docker run -d -p 8192:8191 ghcr.io/flaresolverr/flaresolverr
pry serve
```

20
USAGE.md
View file

@ -12,8 +12,10 @@ last_updated: 2026-06-30
## Quickstart
```bash
# Install
pip install pry
# Install (PyPI 'pry' is taken — install from this repo)
pip install -e .
# or:
# pip install "pry-scraper @ git+https://git.rugmunch.io/RugMunchMedia/pryscraper.git"
# Start the server
pry serve
@ -32,9 +34,11 @@ pry crawl https://docs.com --max-pages 20 -o data.json
```bash
docker compose up -d
# Pry on :8002, FlareSolverr on :8191
# Pry on :8005 (host) → :8002 (container), FlareSolverr on :8192 (host) → :8191 (container)
```
> **Port clarification:** The API listens on container port `8002` and is published to host port `8005` (see `docker-compose.yml`). From outside the container (host machine, SDK, CLI), use `http://localhost:8005`. Use `8002` only if you `docker exec` into the running container.
## CLI Reference
| Command | Description |
@ -47,7 +51,7 @@ docker compose up -d
| `pry ss <url>` | Take screenshot |
| `pry run [pry.yml]` | Execute job file |
| `pry serve` | Start API server |
| `pry completions` | Install shell autocomplete |
| `pry completions [bash\|zsh\|fish]` | Print shell-autocomplete install instructions |
### CLI Options
@ -226,7 +230,7 @@ Export scraped content in multiple formats.
}
```
Supported formats: `json`, `csv`, `rss`, `txt`, `sql`
Supported formats: `json`, `csv` (`rss`, `txt`, `sql` listed in FEATURES.md but not yet implemented)
### Alert Endpoints
@ -292,7 +296,7 @@ Execute a pipeline definition.
```python
from pry_sdk import PryCrawl
mc = PryCrawl("http://localhost:8002")
mc = PryCrawl("http://localhost:8005")
result = await mc.scrape("https://example.com")
print(result["data"]["markdown"])
```
@ -302,7 +306,7 @@ print(result["data"]["markdown"])
```python
from pry_sdk import PryCrawlSync
mc = PryCrawlSync("http://localhost:8002")
mc = PryCrawlSync("http://localhost:8005")
result = mc.scrape("https://example.com")
```
@ -383,7 +387,7 @@ POST /v1/commerce/sync
}
```
### Salesforce / HubSpot / Zoho
### Salesforce / HubSpot / ~~Zoho~~
```json
POST /v1/crm/sync

74
cli.py
View file

@ -11,7 +11,6 @@ Usage:
pry transform <data> Convert data format
pry run [pry.yml] Execute jobs from pry.yml
pry serve Start the API server
pry completions Install shell autocomplete
pry proxy ... Manage proxy providers and signup
"""
@ -231,35 +230,8 @@ def cmd_serve(host="0.0.0.0", port=8005): # nosec B104
os.execvp("uvicorn", ["uvicorn", "api:app", "--host", host, "--port", str(port)])
def cmd_completions(shell="bash"):
"""Install shell autocomplete."""
script = {
"bash": 'eval "$(_PRY_COMPLETE=bash_source pry)"',
"zsh": 'eval "$(_PRY_COMPLETE=zsh_source pry)"',
"fish": 'eval "$(_PRY_COMPLETE=fish_source pry)"',
}.get(shell, "")
if shell == "bash":
rc = os.path.expanduser("~/.bashrc")
elif shell == "zsh":
rc = os.path.expanduser("~/.zshrc")
elif shell == "fish":
rc = os.path.expanduser("~/.config/fish/config.fish")
else:
print(f"{YELLOW}Unknown shell: {shell}. Supported: bash, zsh, fish{NC}")
return
dirname = os.path.dirname(rc)
os.makedirs(dirname, exist_ok=True)
with open(rc, "a") as f:
f.write(f"\n# Pry autocomplete\n{script}\n")
print(f"{GREEN}✓{NC} Autocomplete installed for {shell}")
print(f" Restart your shell or run: source {rc}")
def main():
if len(sys.argv) >= 2 and sys.argv[1] == "proxy" and click is not None:
cli.main(standalone_mode=False)
return
if len(sys.argv) < 2:
def _print_usage() -> None:
"""Print the standard Pry usage block (no exit)."""
print(f"{BOLD}Pry v{VERSION}{NC} — Pry open any website. {CYAN}pry.dev{NC}")
print()
print(f" {BOLD}Usage:{NC}")
@ -288,6 +260,46 @@ def main():
print()
print(f" {BOLD}Settings:{NC}")
print(" PRY_URL=http://localhost:8005 (default)")
def cmd_completions(shell="bash"):
"""Print shell-autocomplete installation instructions.
Pry does not currently ship a `_PRY_COMPLETE` env-var handler, so we do
not silently write a broken eval line into the user's shell rc file.
Print copy/paste instructions instead.
"""
if shell not in ("bash", "zsh", "fish"):
print(f"{YELLOW}Unknown shell: {shell}. Supported: bash, zsh, fish{NC}")
return
print(f"{BOLD}Pry shell autocomplete ({shell}){NC}")
print()
if shell == "bash":
print(" Add the following line to your ~/.bashrc:")
print(f" {CYAN}eval \"$(register-python-argcomplete pry)\"{NC}")
print()
print(" install dependency: pip install argcomplete")
elif shell == "zsh":
print(" Add the following line to your ~/.zshrc:")
print(f" {CYAN}eval \"$(register-python-argcomplete --shell zsh pry)\"{NC}")
print()
print(" install dependency: pip install argcomplete")
else:
print(" Add the following line to ~/.config/fish/config.fish:")
print(f" {CYAN}register-python-argcomplete --shell fish pry | source{NC}")
print()
print(" install dependency: pip install argcomplete")
print()
print(f"{YELLOW}Note:{NC} Pry does not yet wire `_PRY_COMPLETE` itself; once it")
print(" does, this command will switch to a one-shot installer.")
def main():
if len(sys.argv) >= 2 and sys.argv[1] == "proxy" and click is not None:
cli.main(standalone_mode=False)
return
if len(sys.argv) < 2:
_print_usage()
return
cmd = sys.argv[1]
@ -350,7 +362,7 @@ def main():
print(f"Pry v{VERSION}")
elif cmd in ("-h", "--help", "help"):
main()
_print_usage()
elif cmd == "proxy":
if click is None: