chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth)

Squashed from chore/license-relicense. Full message preserved in the
original branch commit bb77eb5. See ADR-0002 for the decision rationale.

Refs: ADR-0002, commit bb77eb5
This commit is contained in:
Crypto Rug Munch 2026-07-02 19:59:18 +02:00
parent 72b225f548
commit 8d25702eca
305 changed files with 2423 additions and 340 deletions

View file

@ -1,3 +1,8 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
# Part of Pry - https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
# ── Pry Configuration ──
# Copy this to .env and adjust values.
# All PRY_* vars are auto-loaded by PrySettings.

View file

@ -1,3 +1,7 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
name: Bug Report
description: Report a defect or unexpected behavior
title: "[Bug]: "

View file

@ -1,3 +1,7 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
blank_issues_enabled: false
contact_links:
- name: Q&A

View file

@ -1,3 +1,7 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
name: Feature Request
description: Propose a new feature or improvement
title: "[Feature]: "

View file

@ -1,3 +1,7 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
name: CI
on:

View file

@ -1,3 +1,7 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
name: CI
on:

View file

@ -1,3 +1,7 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.11.0

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# AGENTS.md — PryScraper
> AI agent contract. Read this before touching anything in this repo.
@ -6,7 +8,7 @@
canonical · owner=crmuncher · last_updated=2026-07-02
## What This Repo Is
Multi-source crypto intelligence crawler (munchcrawl). Scraper, extractor, parser, automator, job queue. CLI + MCP server + SDK.
Multi-source crypto intelligence crawler (munchcrawl). Scraper, extractor, parser, automator, job queue. CLI + MCP server + SDK. License: Dual -- MIT (core) + BSL 1.1 (stealth/anti-detection subset). See LICENSE and LICENSE-BSL-STEALTH.
## Type
`backend` · language=Python 3.12 + FastAPI
@ -40,6 +42,7 @@ Multi-source crypto intelligence crawler (munchcrawl). Scraper, extractor, parse
5. **No duplicate scaffolds.** If a `backend/` or `frontend/` subdir exists at root, it's either THE app or bloat.
6. **Update STATUS.md before committing.** Builders must know where we are.
7. **Read DECISIONS.md before architectural changes.** Don't repeat rejected designs.
8. **Respect dual licensing.** When contributing, the core/ files are MIT (any PR welcome). The stealth/ files (listed in LICENSE-BSL-STEALTH) are BSL 1.1 -- only Rug Munch Media LLC can modify them. PRs touching BSL files will be rejected.
## Commands
```bash

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
---
title: Pry Architecture & Design
status: canonical

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
---
title: Pry — Brutal Honest Audit & Action Plan
status: canonical

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# DECISIONS.md — PryScraper
> Architecture Decision Records (ADRs). Read before architectural changes.
@ -21,6 +23,7 @@ A short document capturing an important architectural decision:
| Number | Title | Status | Date |
|--------|-------|--------|------|
| [0001](docs/adr/0001-initial-architecture.md) | Initial architecture | Accepted | 2026-07-02 |
| [0002](docs/adr/0002-dual-licensing.md) | Dual licensing: MIT (core) + BSL 1.1 (stealth) | Accepted | 2026-07-03 |
## Fleet-wide ADRs (in standards repo)
- [ADR-0001: Tailscale mesh as the only perimeter](https://git.rugmunch.io/RugMunchMedia/standards/raw/branch/main/MCP-ARCHITECTURE.md)

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# DEPLOYMENT.md — PryScraper
> How to deploy. Build, push, restart, rollback.

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# DEVELOPMENT.md — PryScraper
> Dev workflow. Install, code, test, commit, PR.

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
---
title: Pry Feature Catalog
status: canonical

57
LICENSE
View file

@ -1,17 +1,52 @@
# PROPRIETARY LICENSE — Rug Munch Media LLC
MIT License
Copyright (c) 2026 Rug Munch Media LLC. All rights reserved.
Copyright (c) 2026 Rug Munch Media LLC
This software and associated documentation files (the "Software") are
PROPRIETARY and CONFIDENTIAL to Rug Munch Media LLC.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the Software), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
UNAUTHORIZED USE, COPYING, MODIFICATION, MERGING, PUBLISHING,
DISTRIBUTING, SUBLICENSING, AND/OR SELLING COPIES OF THE SOFTWARE ARE
STRICTLY PROHIBITED WITHOUT PRIOR WRITTEN CONSENT FROM RUG MUNCH MEDIA LLC.
For licensing inquiries: licensing@rugmunch.io
For commercial licensing: enterprise@rugmunch.io
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================================================
DUAL LICENSING NOTICE
================================================================================
Pry is dual-licensed. Most of the codebase (the core scraping engine, extraction,
templates, MCP server, x402 payment rail, CLI, and SDK) is MIT-licensed under
the terms above. The following anti-detection / stealth components are licensed
under the Business Source License 1.1 — see LICENSE-BSL-STEALTH:
- ultimate_scraper.py (10-tier fallback engine)
- stealth_engine.py (stealth orchestration)
- stealth_scripts/*.js (6 anti-detection JS injectors)
- camoufox_integration.py (Firefox anti-detection)
- tls_fingerprint.py (TLS fingerprint randomization)
- cookie_warmer.py (cookie pre-aging)
- behavioral_biometrics.py (behavioral mimicry)
- adaptive.py (adaptive strategy rotation)
- browser_pool.py (browser pre-warming for stealth)
- network.py (Tor / SOCKS5 routing)
- captcha_solver.py (CAPTCHA bypass)
- shadow_dom.py (stealth DOM extraction)
- lazy_load.py (human-like scroll)
- signup_automator.py (human-like signup)
- auth_connector.py (human-like login)
Each of those files is individually marked with:
SPDX-License-Identifier: BSL-1.1
The BSL terms apply only to those files. All other files (including all
templates/, browser-extension/, wordpress-plugin/, shopify-app/, llm_providers/,
and the bulk of api.py, mcp_production.py, x402.py) are MIT-licensed.
================================================================================

68
LICENSE-BSL-STEALTH Normal file
View file

@ -0,0 +1,68 @@
Business Source License 1.1
Licensor: Rug Munch Media LLC
Licensed Work: Pry — Stealth / Anti-Detection Module
(the specific files listed in the DUAL LICENSING NOTICE
in LICENSE and individually marked with
SPDX-License-Identifier: BSL-1.1)
Change Date: 2029-01-01
Change License: MIT License (see LICENSE)
Additional Use Grant:
You may use the Licensed Work, without a commercial license from Rug Munch
Media LLC, for any of the following purposes:
1. Personal use (e.g., a developer running Pry on their own machine to
scrape sites for personal projects).
2. Non-production testing and development (e.g., a company running Pry
in CI to test their own scrapers against anti-bot targets).
3. Academic research and education.
4. Development of non-commercial open source software (any project whose
source code is publicly available under an OSI-approved license and
which is not sold, licensed, or distributed for a fee).
5. Internal evaluation by a potential customer considering a
commercial license (capped at 90 days).
All other use — including any production deployment, any use by a
for-profit entity in the course of its business operations, any SaaS /
PaaS offering, any consulting engagement, and any redistribution as
part of a commercial product or service — requires a separate written
commercial license from Rug Munch Media LLC.
For commercial licensing, contact: enterprise@rugmunch.io
For full canonical text of the Business Source License 1.1, see:
https://mariadb.com/bsl11/
The text reproduced below is the official license template; on the Change
Date (2029-01-01) the Licensed Work converts to the MIT License (see
LICENSE). Until that date the following terms govern.
------------------------------------------------------------------------------
Business Source License 1.1
Licensor: Rug Munch Media LLC
Licensed Work: Pry — Stealth / Anti-Detection Module
Change Date: 2029-01-01
Change License: MIT License
Additional Use Grant:
Free for non-production personal use, non-production testing, academic
research, and development of non-commercial open source software. See
LICENSE for the full Additional Use Grant and the list of covered files.
Production use and any use by a for-profit entity in the course of its
business requires a separate commercial license from Rug Munch Media LLC.
Contact enterprise@rugmunch.io.
The Licensor hereby grants you the right to copy, modify, create derivative
works, redistribute, and make non-production use of the Licensed Work. The
Licensor may make an Additional Use Grant, above, permitting limited
production use. Any other use, including production use, is prohibited.
All other rights are reserved by the Licensor.
THE LICENSED WORK IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED. THE LICENSOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
LICENSED WORK, INCLUDING ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# RUG MUNCH MEDIA LLC — LICENSING & PRICING STRATEGY
> **Complete decisions for every system.**
@ -124,19 +126,20 @@ WalletPress is **THREE products** that need different strategies:
---
## 3. PRYSCRAPER DETAILED PROFIT MODEL
## 3. PRYSCRAPER - DETAILED PROFIT MODEL
### 3.1 License: PROPRIETARY (CONFIRMED)
### 3.1 License: DUAL: MIT (core) + BSL 1.1 (stealth)
**Why:**
- Competitive moat is our stealth browser, anti-detection, and bypass techniques
- If competitors see our code, they can replicate in days
- Crypto scrapers are a race — whoever has the best stealth wins
- MIT for the core scraping engine builds community, adoption, trust, stars, contributions
- BSL 1.1 protects the anti-detection/stealth moat from commercial cloning
- 15 stealth files are BSL 1.1: ultimate_scraper.py, stealth_engine.py, stealth_scripts/*.js, camoufox_integration.py, tls_fingerprint.py, cookie_warmer.py, behavioral_biometrics.py, adaptive.py, browser_pool.py, network.py, captcha_solver.py, shadow_dom.py, lazy_load.py, signup_automator.py, auth_connector.py
- The rest (100+ files) are MIT
### 3.2 Pricing Three Models
### 3.2 Pricing - Three Models
**Model A: SaaS (Primary)**
- Hosted API at `pry.dev`
- Hosted API at pry.dev
- Pay-per-request with x402 micropayments
- Free tier: 1,000 requests/month
- Pro: $49/mo for 100K requests
@ -156,11 +159,11 @@ WalletPress is **THREE products** that need different strategies:
| Endpoint | Price |
|----------|-------|
| `/scrape` | $0.005/call |
| `/crawl` | $0.02/call |
| `/extract` | $0.01/call |
| `/screenshot` | $0.003/call |
| `/stealth_browser` | $0.05/minute |
| /scrape | $0.005/call |
| /crawl | $0.02/call |
| /extract | $0.01/call |
| /screenshot | $0.003/call |
| /stealth_browser | $0.05/minute |
**Model C: White-label Enterprise**
- Deploy PryScraper on your infrastructure
@ -182,8 +185,42 @@ WalletPress is **THREE products** that need different strategies:
| White-label Enterprise | $200K | $600K | $1.2M |
| **TOTAL** | **$310K** | **$1.02M** | **$2.2M** |
---
*Plus community template contributions which increase the moat over time (unquantified but real).*
### 3.4 Stealth BSL Specifics
**Which 15 files are BSL-covered:**
- ultimate_scraper.py
- stealth_engine.py
- stealth_scripts/*.js
- camoufox_integration.py
- tls_fingerprint.py
- cookie_warmer.py
- behavioral_biometrics.py
- adaptive.py
- browser_pool.py
- network.py
- captcha_solver.py
- shadow_dom.py
- lazy_load.py
- signup_automator.py
- auth_connector.py
**What counts as production use:**
- Live application serving real users
- Business internal operations
- Deploying as a service for third parties
**What does NOT count:**
- Personal use
- Non-production testing
- Academic research
- Non-commercial OSS development
- 90-day evaluation
**Change Date:** 2029-01-01, converts to MIT
**Commercial license:** enterprise@rugmunch.io
## 4. RMI (RUG MUNCH INTELLIGENCE) — DETAILED PROFIT MODEL
### 4.1 License: OPEN CORE (CONFIRMED)

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
---
title: Pry MCP + x402 Audit & 20-Win Improvement Plan
status: canonical

22
PLAN.md
View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# PLAN.md — PryScraper
> Current sprint. 21-day cycles per ADR-0012.
@ -6,13 +8,27 @@
current_sprint=2026-Q3-S1 · started=2026-07-01 · ends=2026-07-21
## Sprint Goals
1. _TODO: top 1-3 goals for this sprint_
1. DONE Re-license to dual MIT + BSL 1.1 (ADR-0002) - done 2026-07-03
2. WIP Deploy repo to /srv/pry/ - wire the new api.py (4,668 lines, 190 endpoints) so MCP and x402 actually serve
3. WIP Wire auth - JWT/API-key middleware on the live deploy so we can monetize
## In Progress
- [ ] TODO: in-progress items
- [x] Re-license the codebase: dual MIT (core) + BSL 1.1 (stealth)
- [x] Add SPDX headers to all 300+ files
- [x] Write ADR-0002 documenting the decision
- [ ] Deploy updated api.py with MCP and x402 endpoints to /srv/pry/
- [ ] Set PRY_X402_WALLET and PRY_X402_FACILITATOR env vars on Talos
- [ ] Run templates/validate_templates.py end-to-end, publish honest success rate
## Backlog (this sprint)
- [ ] TODO: planned items
- [ ] Migrate ~/.pry/* JSON storage to SQLite (concurrency + multi-tenant)
- [ ] Split api.py (4,668 lines) into router modules (cap 500 lines each)
- [ ] Replace 159 broad except Exception with specific types + add ruff BLE001
- [ ] Surface proxy_referrals.py and referrals.py in /v1/referrals API
- [ ] Publish browser extension (Chrome Web Store) and WordPress plugin (wordpress.org)
- [ ] Deploy Cloudflare Worker mcp-worker.js so MCP server is publicly hosted
- [ ] Register on Smithery, Glama, PulseMCP MCP directories
- [ ] Build pry.dev landing page with comparison table (Pry vs Firecrawl vs ScrapingBee)
## Risks
- TODO: anything blocking progress

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# Pry — Open any website
Self-hosted web scraping + browser automation API. Cloudflare bypass, document parsing, AI summarization. No API keys needed.
@ -205,4 +207,19 @@ make check # Full audit
## License
MIT — Rug Munch Media LLC
Pry is dual-licensed to balance community openness with the protection of its advanced anti-detection and stealth capabilities.
- **Most code** (core functionality, utilities, and standard modules) is licensed under the **MIT License**. This permissive license encourages community contributions and wide adoption. See [LICENSE](./LICENSE).
- **The anti-detection / stealth module** is licensed under the **BSL 1.1 (Business Source License)**. This module is free to use for non-production purposes; a commercial license is required for any production use. See [LICENSE-BSL-STEALTH](./LICENSE-BSL-STEALTH).
### Additional Use Grant
Under the BSL 1.1, an Additional Use Grant permits free use of the stealth module for:
- Personal, non-commercial projects
- Non-production testing and evaluation
- Academic research
- Non-commercial open source software development
- A 90-day evaluation period for any other purpose
For commercial licensing (production use, enterprise deployment, or embedding in proprietary products), please contact: **enterprise@rugmunch.io**

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
---
title: Pry Roadmap & Status
status: canonical

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# SECURITY.md — PryScraper
> Threat model, secrets, auth, rate limiting, vuln disclosure.

View file

@ -1,9 +1,14 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# STATUS.md — PryScraper
> Where we are RIGHT NOW. Update before every commit.
## Last Updated
2026-07-02 (template, replace on first commit)
2026-07-03 - Dual-licensing relicense complete (MIT core + BSL 1.1 stealth)
## Current Status
🟡 pre-production - repo scaffolded, re-license landed, deploy out of sync. See AUDIT.md.
## Deployment Status
- **Last deploy**: TBD
@ -19,7 +24,12 @@
- 2026-07-02: Repository created from `fleet-template`
## Known Issues / Tech Debt
- _None yet_
- Repo and deploy at /srv/pry/ are out of sync. Deploy runs a 789-line api.py; repo has 4,668-line api.py with 190 endpoints (MCP, x402, auth, etc.). MCP and x402 are written but not deployed. See AUDIT.md.
- All 80+ site templates unverified - only ~30-40% known to work end-to-end. Run templates/validate_templates.py before claiming template coverage.
- State storage in ~/.pry/*.json - no concurrency safety, no transactions. See plan to migrate to SQLite.
- PRY_API_KEY, PRY_X402_WALLET, PRY_X402_FACILITATOR env vars unset in deploy - no auth, no x402, no payments.
- License collision resolved 2026-07-03: dual MIT (core) + BSL 1.1 (stealth/anti-detection). See ADR-0002.
- mcp_production.py and x402.py exist in repo but are NOT deployed. Top priority for next deploy.
## Quick Links
- [Live URL](https://pryscraper.rugmunch.io)

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# TESTING.md — PryScraper
> Test strategy. Unit, integration, e2e, fixtures.

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
---
title: Pry Usage Guide
status: canonical

View file

@ -1,3 +1,8 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
"""Pry — Account pool management, session persistence, proxy scoring."""
import json

View file

@ -2,6 +2,12 @@
Pre-built scrapers that can be subscribed to and run on schedule.
Users can publish their own actors. We earn from usage."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import logging
import os

View file

@ -1,6 +1,13 @@
"""Pry — adaptive crawling with information foraging.
Intelligently decides when to stop crawling based on content relevance."""
# SPDX-License-Identifier: BSL-1.1
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — Stealth / Anti-Detection Module
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
# Change Date: 2029-01-01 (converts to MIT).
import logging
import re
from collections import Counter

View file

@ -3,6 +3,12 @@ Diff tracking, LLM summarization, RSS generation, change monitoring,
schema.org extraction, email finder, and more.
All powered by local Ollama no API keys needed."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import difflib
import hashlib
import json

View file

@ -1,6 +1,12 @@
"""Pry — White-Label Agency Dashboard.
Multi-tenant reseller platform: custom branding, client management, sub-accounts."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import logging
import os

View file

@ -1,6 +1,12 @@
"""Pry — AI Agent Plugin.
OpenAPI spec server, MCP server wrapper, and GPT Action helper."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import logging
from pathlib import Path

View file

@ -1,6 +1,12 @@
"""Pry — Multi-Channel Alerting System.
SMS, Email, Microsoft Teams, Discord, Telegram alerts."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import logging
from typing import Any

View file

@ -1,3 +1,8 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
"""Pry — Real anomaly detection. Multi-field, time-series aware, with seasonality support."""
import logging

6
api.py
View file

@ -2,6 +2,12 @@
Scrape + Crawl + Automate + Parse + Extract + MCP
Self-hosted, free, better than Firecrawl."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import asyncio
import base64
import difflib

View file

@ -1,3 +1,8 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
"""Pry — Authentication with JWT and API keys, per-key rate limiting."""
import hashlib

View file

@ -1,6 +1,13 @@
"""Pry — Enterprise SSO / Auth Connector System.
Credential vault, session persistence, SSO flow automation, CAPTCHA integration."""
# SPDX-License-Identifier: BSL-1.1
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — Stealth / Anti-Detection Module
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
# Change Date: 2029-01-01 (converts to MIT).
import asyncio
import json
import logging

View file

@ -6,6 +6,12 @@ Audited: session cleanup runs automatically, no injection vectors,
all user inputs validated before passing to Playwright.
"""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import asyncio
import base64
import json

View file

@ -3,6 +3,13 @@ Real human behavior simulation: hesitation, scroll-back, mouse drift, reading ti
Modern anti-bot systems detect 'too perfect' behavior. This module makes
behavior more realistic by adding human imperfections."""
# SPDX-License-Identifier: BSL-1.1
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — Stealth / Anti-Detection Module
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
# Change Date: 2029-01-01 (converts to MIT).
import logging
import math
import random

View file

@ -1,3 +1,7 @@
// SPDX-License-Identifier: MIT
// Copyright (c) 2026 Rug Munch Media LLC
// Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
// Licensed under MIT. See LICENSE.
// Pry Browser Extension — Background Service Worker
chrome.runtime.onInstalled.addListener(function() {
chrome.storage.sync.get(['pryServerUrl'], function(result) {

View file

@ -1,3 +1,5 @@
<!-- SPDX-License-Identifier: MIT
<!-- Copyright (c) 2026 Rug Munch Media LLC -->
<!DOCTYPE html>
<html>
<head>

View file

@ -1,3 +1,7 @@
// SPDX-License-Identifier: MIT
// Copyright (c) 2026 Rug Munch Media LLC
// Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
// Licensed under MIT. See LICENSE.
// Pry Browser Extension — Popup Script
document.addEventListener('DOMContentLoaded', function() {
loadSettings();

View file

@ -1,6 +1,13 @@
"""Pry — shared Playwright browser pool.
Reuses browser instances across requests instead of launching per scrape."""
# SPDX-License-Identifier: BSL-1.1
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — Stealth / Anti-Detection Module
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
# Change Date: 2029-01-01 (converts to MIT).
import asyncio
import logging
from typing import Any

View file

@ -1,6 +1,12 @@
"""Pry — in-memory LRU cache with Redis persistence.
Prevents re-scraping the same URL within TTL window."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import hashlib
import json
import time

View file

@ -3,6 +3,13 @@ Camoufox patches Firefox at the source level to bypass fingerprinting
more effectively than Playwright/Puppeteer. It's a drop-in alternative
for Playwright that focuses on stealth."""
# SPDX-License-Identifier: BSL-1.1
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — Stealth / Anti-Detection Module
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
# Change Date: 2029-01-01 (converts to MIT).
import logging
import random
import time

View file

@ -2,6 +2,13 @@
Supports: Capsolver -> 2Captcha -> Anti-Captcha -> CapMonster -> DeathByCaptcha -> NextCaptcha
Also handles Turnstile, reCAPTCHA v2/v3/invisible, and hCaptcha."""
# SPDX-License-Identifier: BSL-1.1
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — Stealth / Anti-Detection Module
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
# Change Date: 2029-01-01 (converts to MIT).
import asyncio
import json
import logging

6
cli.py
View file

@ -15,6 +15,12 @@ Usage:
pry proxy ... Manage proxy providers and signup
"""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import base64
import json
import os

View file

@ -1,6 +1,12 @@
"""Pry — shared httpx client pool with connection reuse.
All modules import `http_client` instead of creating new clients per request."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import logging
import httpx

View file

@ -1,6 +1,12 @@
"""Pry — Commerce Platform Sync Engine.
Unified interface for WooCommerce, Shopify, and generic API sync."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import logging
import os
from pathlib import Path

View file

@ -1,6 +1,12 @@
"""Pry — Legal Compliance Engine.
Per-source compliance scorecard: robots.txt, ToS, GDPR/CCPA, jurisdiction."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import logging
import re
from datetime import UTC, datetime

View file

@ -3,6 +3,12 @@
Loads settings from environment variables (via Pydantic Settings).
All secrets come from gopass, never from .env files committed to git.
"""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
from __future__ import annotations
import os

View file

@ -3,6 +3,13 @@ Pre-age cookies by browsing legitimate pages first. Aged cookies with
realistic browsing history bypass anti-bot detection that checks for
'fresh' cookies. This is a technique used by professional scraping services."""
# SPDX-License-Identifier: BSL-1.1
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — Stealth / Anti-Detection Module
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
# Change Date: 2029-01-01 (converts to MIT).
import asyncio
import json
import logging

View file

@ -1,6 +1,12 @@
"""Pry — Cost Analytics Engine.
Tracks usage costs, cache hit rates, projected burn, and smart scheduling."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import logging
import os

View file

@ -1,6 +1,12 @@
"""Pry — Reverse ETL to CRM.
Sync scraped data to Salesforce, HubSpot, Pipedrive, and Close.com."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import logging
from typing import Any

6
db.py
View file

@ -1,6 +1,12 @@
"""Pry — Database layer using SQLAlchemy (SQLite/PostgreSQL).
Replaces JSON file storage for production safety."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import logging
import os
from collections.abc import Generator

View file

@ -2,6 +2,12 @@
Detects near-duplicate content (e.g., 95% similar pages) using SimHash.
Useful for crawl deduplication, change detection, and content grouping."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import hashlib
import logging
import re

View file

@ -1,6 +1,12 @@
"""Pry — One-Click Business Integrations.
Native write destinations: Google Sheets, Slack, Airtable, email."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import logging
from datetime import UTC, datetime

View file

@ -1,3 +1,7 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
services:
pry:
build: .

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# ADR-NNNN: <Short Title>
## Status

View file

@ -1,3 +1,5 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# ADR-0001: Initial Architecture
## Status

View file

@ -0,0 +1,55 @@
[//]: # (SPDX-License-Identifier: MIT)
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
# ADR-0002: Dual Licensing (MIT + BSL 1.1)
## Status
Accepted
## Date
2026-07-03
## Context
The Pry (munchcrawl) project was initially released under a full proprietary license. This licensing model resulted in zero community engagement, zero external contributors, and zero GitHub stars. In the competitive web scraping and data extraction market, open source projects such as Firecrawl and Crawl4AI have achieved dominant positions by leveraging MIT licensing to build strong community moats, attract contributors, and establish trust with users.
The existing LICENSING_PRICING_STRATEGY.md document previously listed Pry as PROPRIETARY. This approach proved incompatible with the project's goal of achieving widespread adoption, particularly for the Model Context Protocol (MCP) server and template marketplace components that benefit from community contributions and third-party integrations.
The core challenge is that a fully proprietary license prevents the project from competing on community moat. Without open source licensing, potential users cannot inspect the codebase for security vulnerabilities, cannot contribute improvements, and cannot build trust in the scraping engine's reliability. The project requires a licensing strategy that balances community growth with protection of proprietary anti-detection techniques that constitute the project's competitive advantage.
## Decision
Pry will adopt a dual licensing model consisting of MIT for the core scraping engine and Business Source License 1.1 (BSL 1.1) for the anti-detection and stealth components.
The MIT license will apply to the core scraping engine, comprising over 100 files including: the scraper module, extractor module, parser module, all 80+ templates, MCP server, x402 protocol implementation, command-line interface, software development kit, browser extension, WordPress plugin, Shopify app, and all LLM provider integrations. The MIT license permits free use, modification, and redistribution of these files by any party.
The BSL 1.1 license will apply to 15 anti-detection and stealth files: ultimate_scraper.py, stealth_engine.py, all stealth_scripts/*.js files, camoufox_integration.py, tls_fingerprint.py, cookie_warmer.py, behavioral_biometrics.py, adaptive.py, browser_pool.py, network.py, captcha_solver.py, shadow_dom.py, lazy_load.py, signup_automator.py, and auth_connector.py. The BSL 1.1 includes an Additional Use Grant that permits personal use, non-production testing, academic research, non-commercial open source software development, and 90-day evaluation. The BSL Change License is MIT, not GPLv2. The Change Date is set to 2029-01-01, after which all BSL-licensed files automatically convert to MIT.
## Consequences
Positive: The MIT-licensed core engine will enable community growth, attract GitHub stars, build user trust through code transparency, and drive adoption of the MCP server and template marketplace. External contributors can submit improvements to the scraping engine, parser, and integrations.
Positive: Users and auditors can inspect the core scraping engine for security vulnerabilities, data handling practices, and code quality. This transparency is essential for enterprise adoption and regulatory compliance.
Positive: The BSL 1.1 protection on anti-detection files preserves the project's competitive moat. Competitors can see and copy the core engine code under MIT, but cannot legally copy the anti-detection techniques that differentiate Pry from other scraping tools.
Negative: Competitors can freely use, modify, and redistribute the core scraping engine code. This may lead to derivative projects that compete with Pry's commercial offerings, though they will lack the protected anti-detection capabilities.
Negative: The dual licensing model introduces complexity in license compliance. Contributors and users must understand which files are under MIT versus BSL 1.1, and the BSL 1.1 Additional Use Grant limitations must be clearly communicated.
## Alternatives
Full MIT license: Rejected because it would expose all anti-detection techniques to competitors, eliminating the project's primary competitive advantage. Without protection of these techniques, the project would have no sustainable differentiation in the market.
Full proprietary license: Rejected because it resulted in zero community engagement, zero contributors, and zero adoption. This model proved incompatible with building the community moat necessary to compete with Firecrawl and Crawl4AI.
AGPL version 3 plus commercial license: Rejected because the AGPL's viral copyleft nature would deter many potential users and contributors who prefer permissive licensing. The AGPL also creates compatibility issues with the MIT-licensed ecosystem that the project depends upon.
BSL 1.1 on all files: Rejected because applying BSL to the entire codebase would prevent the community adoption and trust-building that only a permissive license can provide. Users would be unable to freely use the core engine without evaluating the BSL's Additional Use Grant limitations.
## References
- LICENSE (MIT) - Core scraping engine license file
- LICENSE-BSL-STEALTH (BSL 1.1) - Anti-detection components license file
- https://mariadb.com/bsl11/ - Business Source License 1.1 official documentation
- ADR-0000-template.md - Architecture Decision Record template used for this document
- LICENSING_PRICING_STRATEGY.md - Previous licensing strategy document (superseded by this ADR)

View file

@ -1,6 +1,12 @@
"""Pry — Email Inbox Scraping.
Connect Gmail/Outlook, extract structured data from emails."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import base64
import logging
import re

View file

@ -1,6 +1,12 @@
"""Pry — Data Enrichment Pipeline.
Enrich scraped data with company info, social profiles, tech stack detection."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import logging
import re
from typing import Any

View file

@ -1,6 +1,12 @@
"""Pry — typed error hierarchy.
Every API error flows through here for consistent JSON responses."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
from typing import Any

View file

@ -1,6 +1,12 @@
"""Pry — structured extraction strategies.
CSS/XPath-based extraction (no LLM needed) + chunking strategies for LLM extraction."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import logging
import math
import re

View file

@ -4,6 +4,12 @@ LLM failures fall back gracefully to pattern mode.
No hallucination: JSON output is always parsed and validated.
"""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import re
from typing import Any

View file

@ -1,6 +1,12 @@
"""Pry — Adaptive Freshness Scheduling.
Conditional scraping, content fingerprinting, staleness dashboard, adaptive frequency."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import hashlib
import json
import logging

View file

@ -1,6 +1,12 @@
"""Pry — GDPR Compliance Portal.
Data deletion API, consent management, retention policies, audit log."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import hashlib
import json
import logging

View file

@ -1,3 +1,8 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
"""Pry — Real GDPR compliance: data subject access requests, data portability, real audit log."""
import contextlib

View file

@ -1,7 +1,7 @@
{
"$schema": "https://glama.ai/mcp/schema.json",
"name": "pry",
"title": "Pry Web Scraping & Browser Automation MCP",
"title": "Pry \u2014 Web Scraping & Browser Automation MCP",
"description": "Self-hosted web intelligence platform: scrape, crawl, extract, automate, and parse any website. Exposed as a Model Context Protocol server.",
"version": "3.0.0",
"license": "MIT",
@ -27,14 +27,38 @@
}
],
"tools": [
{ "name": "pry_scrape", "description": "Scrape a URL to clean markdown" },
{ "name": "pry_crawl", "description": "Crawl a website from a starting URL" },
{ "name": "pry_extract", "description": "Extract structured data with CSS selectors" },
{ "name": "pry_template", "description": "Execute a pre-built scraper template" },
{ "name": "pry_search_templates", "description": "Search scraper templates" },
{ "name": "pry_enrich", "description": "Enrich a URL with company and tech intelligence" },
{ "name": "pry_x402_pricing", "description": "Get x402 pay-per-call pricing" },
{ "name": "pry_referrals", "description": "Get the referral catalog" }
{
"name": "pry_scrape",
"description": "Scrape a URL to clean markdown"
},
{
"name": "pry_crawl",
"description": "Crawl a website from a starting URL"
},
{
"name": "pry_extract",
"description": "Extract structured data with CSS selectors"
},
{
"name": "pry_template",
"description": "Execute a pre-built scraper template"
},
{
"name": "pry_search_templates",
"description": "Search scraper templates"
},
{
"name": "pry_enrich",
"description": "Enrich a URL with company and tech intelligence"
},
{
"name": "pry_x402_pricing",
"description": "Get x402 pay-per-call pricing"
},
{
"name": "pry_referrals",
"description": "Get the referral catalog"
}
],
"tags": [
"web-scraping",
@ -43,5 +67,7 @@
"x402",
"self-hosted",
"mcp"
]
],
"_license": "MIT",
"_copyright": "Copyright (c) 2026 Rug Munch Media LLC"
}

View file

@ -3,6 +3,12 @@ Detects GraphQL endpoints, runs introspection queries, generates optimized queri
Many modern sites (Shopify, GitHub, Twitter/X, etc.) have GraphQL APIs that are
10-100x more efficient than scraping HTML."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import logging
import re

View file

@ -1,6 +1,12 @@
"""Pry — Competitive Intelligence Engine.
Historical snapshots, anomaly detection, natural-language alerts, weekly reports."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import logging
import os

View file

@ -1,6 +1,12 @@
"""Pry — batch job queue with Redis and webhook callbacks.
Async processing engine inspired by Firecrawl's webhook system."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import hashlib
import hmac
import json

View file

@ -1,3 +1,9 @@
# SPDX-License-Identifier: BSL-1.1
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — Stealth / Anti-Detection Module
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
# Change Date: 2029-01-01 (converts to MIT).
"""Pry — lazy load and infinite scroll handling."""
import logging

View file

@ -1,6 +1,12 @@
"""Pry — Real LLM-powered features. Replaces regex stubs with actual LLM calls.
Used by compliance, SEO, entity reconciliation, PII redaction, and other AI features."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import logging
from typing import Any

View file

@ -0,0 +1,5 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.

View file

@ -2,6 +2,12 @@
Supports pluggable providers: OpenAI, Anthropic, Google, Cohere, Mistral, Ollama, OpenRouter.
Includes referral/affiliate link tracking for revenue sharing."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import logging
from abc import ABC, abstractmethod
from dataclasses import dataclass, field

View file

@ -1,3 +1,8 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
"""Concrete LLM provider implementations."""
import logging

View file

@ -1,3 +1,8 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
"""LLM provider registry with fallback chain and cost tracking."""
import json

View file

@ -1,6 +1,12 @@
"""Pry — markdown generation strategies with content filtering.
Raw, Fit (pruning), and Fit+BM25 (query-relevant) markdown generators."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import logging
import math
import re

View file

@ -1,6 +1,12 @@
"""Pry Config — runtime configuration for proxy, Tor, VPN, and anti-detection.
Users can configure at startup (env vars) or at runtime (API calls)."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import os

View file

@ -14,6 +14,12 @@ Standard MCP features implemented:
Run: python -m mcp_production
"""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
from __future__ import annotations
import asyncio

View file

@ -1,6 +1,12 @@
"""Pry MCP Server — expose scrape, crawl, automate as MCP tools.
Enables any MCP-compatible AI agent (Claude, Hermes, Cursor) to use Pry."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import httpx

View file

@ -10,6 +10,12 @@ Session state is kept in memory. For production multi-worker deployments, use a
Redis-backed queue or sticky sessions.
"""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
from __future__ import annotations
import asyncio

View file

@ -1,6 +1,12 @@
"""Pry — scheduled monitors with AI change detection.
Cron-based monitors that diff content and judge meaningful changes."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import difflib
import json
import logging

View file

@ -1,6 +1,13 @@
"""Pry — network capture and lazy load handling.
Captures XHR/fetch requests from JS-heavy SPAs and handles lazy content."""
# SPDX-License-Identifier: BSL-1.1
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — Stealth / Anti-Detection Module
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
# Change Date: 2029-01-01 (converts to MIT).
import json
import logging
import re

View file

@ -1,3 +1,8 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
"""Pry — Observability: Prometheus metrics, OpenTelemetry tracing, structured logging."""
import logging

View file

@ -1,6 +1,12 @@
"""Pry — Image OCR using Tesseract.
Extract text from images on web pages. Uses pytesseract + Pillow."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import io
import logging
import os

View file

@ -3,6 +3,12 @@ Uses asyncio subprocess for CPU-bound OCR to avoid blocking the event loop.
Temp files are always cleaned up in finally blocks.
"""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import asyncio
import io
import os

View file

@ -2,6 +2,12 @@
Extracts structured tables from PDF documents (financial reports, invoices, etc.)
Uses pdfplumber, camelot, and pdfminer as fallback methods."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import io
import logging
import os

View file

@ -1,6 +1,12 @@
"""Pry — pipeline hook system for the scraping workflow.
Allows plugins and custom transformations at every stage."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import logging
from collections.abc import Awaitable, Callable
from typing import Any, cast

View file

@ -4,6 +4,12 @@ the engine executes them sequentially with branching and error handling.
A UI can render these steps as drag-and-drop blocks."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import logging
import os

View file

@ -2,6 +2,12 @@
Tries free proxies first (Tor, public rotating). When premium proxy is needed,
prompts user to sign up via affiliate links. Pre-wired to 5+ providers."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import logging
import os

View file

@ -5,6 +5,12 @@ Usage:
result = await mc.scrape("https://example.com")
"""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import asyncio
import logging
from typing import Any

View file

@ -1,6 +1,12 @@
"""Pry — WebSocket streaming, scheduler, batch-file, recorder, transforms.
New capabilities that make Pry unbeatable."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import csv
import io
import json

View file

@ -17,6 +17,12 @@ jobs:
"""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import json
import os

View file

@ -1,14 +1,18 @@
{
"$schema": "https://pulsemcp.com/schema.json",
"name": "pry",
"display_name": "Pry Web Scraping & Browser Automation",
"display_name": "Pry \u2014 Web Scraping & Browser Automation",
"description": "Scrape, crawl, extract, and automate any website. Self-hosted, x402 pay-per-call, MCP-compatible.",
"version": "3.0.0",
"license": "MIT",
"author": "Rug Munch Media LLC",
"homepage": "https://pry.dev",
"repository": "https://github.com/cryptorugmuncher/pry",
"categories": ["Web Scraping", "Browser Automation", "Data Extraction"],
"categories": [
"Web Scraping",
"Browser Automation",
"Data Extraction"
],
"install": {
"type": "pip",
"package": "pry",
@ -32,5 +36,13 @@
"pry_x402_pricing",
"pry_referrals"
],
"tags": ["scraping", "automation", "extraction", "x402", "self-hosted"]
"tags": [
"scraping",
"automation",
"extraction",
"x402",
"self-hosted"
],
"_license": "MIT",
"_copyright": "Copyright (c) 2026 Rug Munch Media LLC"
}

View file

@ -1,3 +1,7 @@
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
[build-system]
requires = ["setuptools>=69.0"]
build-backend = "setuptools.build_meta"
@ -8,8 +12,23 @@ version = "3.0.0"
description = "Free web scraping + browser automation API — self-hosted, no API keys needed"
readme = "README.md"
requires-python = ">=3.11"
license = { text = "Proprietary" }
license = { text = "MIT AND BSL-1.1" }
authors = [{ name = "Rug Munch Media LLC" }]
keywords = ["scraping", "crawler", "browser-automation", "mcp", "x402", "ai-agents"]
classifiers = [
"Development Status :: 4 - Beta",
"License :: OSI Approved :: MIT License",
"License :: Other/Proprietary License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Topic :: Internet :: WWW/HTTP :: Browsers",
"Topic :: Software Development :: Libraries :: Python Modules",
"Typing :: Typed",
]
[project.license-files]
paths = ["LICENSE", "LICENSE-BSL-STEALTH"]
dependencies = [
"fastapi>=0.115.0",
"uvicorn[standard]>=0.32.0",

View file

@ -1,6 +1,12 @@
"""Pry — Data Quality SLA Dashboard.
Per-extraction quality metrics, anomaly detection, freshness tracking."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import difflib
import hashlib
import json

View file

@ -1,6 +1,12 @@
"""Pry — token-bucket rate limiter per IP.
Prevents abuse while keeping the service responsive."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import time

View file

@ -1,6 +1,12 @@
"""Pry — Multi-Source Entity Reconciliation.
Cross-source entity matching, unified schema mapping, diff dashboard."""
# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Rug Munch Media LLC
#
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
# Licensed under MIT. See LICENSE.
import difflib
import hashlib
import logging

Some files were not shown because too many files have changed in this diff Show more