chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth)
Re-license Pry from full Proprietary to a dual-license model: - Core engine, extraction, templates (80+), MCP server, x402 payment rail, CLI, SDK, browser extension, WordPress plugin, Shopify app, and llm_providers: MIT (see LICENSE) - Anti-detection / stealth subset (15 files): BSL 1.1 with Change Date 2029-01-01 (see LICENSE-BSL-STEALTH) BSL files (anti-detection moat): ultimate_scraper.py, stealth_engine.py, stealth_scripts/*.js (6), camoufox_integration.py, tls_fingerprint.py, cookie_warmer.py, behavioral_biometrics.py, adaptive.py, browser_pool.py, network.py, captcha_solver.py, shadow_dom.py, lazy_load.py, signup_automator.py, auth_connector.py This enables community contributions to the core engine (templates, integrations, MCP tools) while protecting the anti-detection techniques that constitute the actual competitive moat. BSL Additional Use Grant permits free non-production use; production deployment requires a commercial license from enterprise@rugmunch.io. Changes: - Replace proprietary LICENSE with MIT LICENSE + new LICENSE-BSL-STEALTH - Add SPDX-License-Identifier headers to 300+ source files - Add docs/adr/0002-dual-licensing.md (ADR documenting the decision) - Update README.md: new License section with BSL Additional Use Grant - Update LICENSING_PRICING_STRATEGY.md: Section 3 (PryScraper) for dual license - Update AGENTS.md: license line in header + new rule 8 (PRs touching BSL rejected) - Update pyproject.toml: license = "MIT AND BSL-1.1" + classifiers + license-files - Update DECISIONS.md index with ADR-0002 - Update STATUS.md (2026-07-03) and PLAN.md sprint goals Refs: ADR-0002
This commit is contained in:
parent
979eae37d4
commit
bb77eb5f35
305 changed files with 2423 additions and 340 deletions
|
|
@ -1,3 +1,8 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
# Part of Pry - https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
# ── Pry Configuration ──
|
||||
# Copy this to .env and adjust values.
|
||||
# All PRY_* vars are auto-loaded by PrySettings.
|
||||
|
|
|
|||
|
|
@ -1,3 +1,7 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
name: Bug Report
|
||||
description: Report a defect or unexpected behavior
|
||||
title: "[Bug]: "
|
||||
|
|
|
|||
|
|
@ -1,3 +1,7 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
blank_issues_enabled: false
|
||||
contact_links:
|
||||
- name: Q&A
|
||||
|
|
|
|||
|
|
@ -1,3 +1,7 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
name: Feature Request
|
||||
description: Propose a new feature or improvement
|
||||
title: "[Feature]: "
|
||||
|
|
|
|||
|
|
@ -1,3 +1,7 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
name: CI
|
||||
|
||||
on:
|
||||
|
|
|
|||
4
.github/workflows/ci.yml
vendored
4
.github/workflows/ci.yml
vendored
|
|
@ -1,3 +1,7 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
name: CI
|
||||
|
||||
on:
|
||||
|
|
|
|||
|
|
@ -1,3 +1,7 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
repos:
|
||||
- repo: https://github.com/astral-sh/ruff-pre-commit
|
||||
rev: v0.11.0
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# AGENTS.md — PryScraper
|
||||
|
||||
> AI agent contract. Read this before touching anything in this repo.
|
||||
|
|
@ -6,7 +8,7 @@
|
|||
canonical · owner=crmuncher · last_updated=2026-07-02
|
||||
|
||||
## What This Repo Is
|
||||
Multi-source crypto intelligence crawler (munchcrawl). Scraper, extractor, parser, automator, job queue. CLI + MCP server + SDK.
|
||||
Multi-source crypto intelligence crawler (munchcrawl). Scraper, extractor, parser, automator, job queue. CLI + MCP server + SDK. License: Dual -- MIT (core) + BSL 1.1 (stealth/anti-detection subset). See LICENSE and LICENSE-BSL-STEALTH.
|
||||
|
||||
## Type
|
||||
`backend` · language=Python 3.12 + FastAPI
|
||||
|
|
@ -40,6 +42,7 @@ Multi-source crypto intelligence crawler (munchcrawl). Scraper, extractor, parse
|
|||
5. **No duplicate scaffolds.** If a `backend/` or `frontend/` subdir exists at root, it's either THE app or bloat.
|
||||
6. **Update STATUS.md before committing.** Builders must know where we are.
|
||||
7. **Read DECISIONS.md before architectural changes.** Don't repeat rejected designs.
|
||||
8. **Respect dual licensing.** When contributing, the core/ files are MIT (any PR welcome). The stealth/ files (listed in LICENSE-BSL-STEALTH) are BSL 1.1 -- only Rug Munch Media LLC can modify them. PRs touching BSL files will be rejected.
|
||||
|
||||
## Commands
|
||||
```bash
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
---
|
||||
title: Pry Architecture & Design
|
||||
status: canonical
|
||||
|
|
|
|||
2
AUDIT.md
2
AUDIT.md
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
---
|
||||
title: Pry — Brutal Honest Audit & Action Plan
|
||||
status: canonical
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# DECISIONS.md — PryScraper
|
||||
|
||||
> Architecture Decision Records (ADRs). Read before architectural changes.
|
||||
|
|
@ -21,6 +23,7 @@ A short document capturing an important architectural decision:
|
|||
| Number | Title | Status | Date |
|
||||
|--------|-------|--------|------|
|
||||
| [0001](docs/adr/0001-initial-architecture.md) | Initial architecture | Accepted | 2026-07-02 |
|
||||
| [0002](docs/adr/0002-dual-licensing.md) | Dual licensing: MIT (core) + BSL 1.1 (stealth) | Accepted | 2026-07-03 |
|
||||
|
||||
## Fleet-wide ADRs (in standards repo)
|
||||
- [ADR-0001: Tailscale mesh as the only perimeter](https://git.rugmunch.io/RugMunchMedia/standards/raw/branch/main/MCP-ARCHITECTURE.md)
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# DEPLOYMENT.md — PryScraper
|
||||
|
||||
> How to deploy. Build, push, restart, rollback.
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# DEVELOPMENT.md — PryScraper
|
||||
|
||||
> Dev workflow. Install, code, test, commit, PR.
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
---
|
||||
title: Pry Feature Catalog
|
||||
status: canonical
|
||||
|
|
|
|||
57
LICENSE
57
LICENSE
|
|
@ -1,17 +1,52 @@
|
|||
# PROPRIETARY LICENSE — Rug Munch Media LLC
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Rug Munch Media LLC. All rights reserved.
|
||||
Copyright (c) 2026 Rug Munch Media LLC
|
||||
|
||||
This software and associated documentation files (the "Software") are
|
||||
PROPRIETARY and CONFIDENTIAL to Rug Munch Media LLC.
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the Software), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
UNAUTHORIZED USE, COPYING, MODIFICATION, MERGING, PUBLISHING,
|
||||
DISTRIBUTING, SUBLICENSING, AND/OR SELLING COPIES OF THE SOFTWARE ARE
|
||||
STRICTLY PROHIBITED WITHOUT PRIOR WRITTEN CONSENT FROM RUG MUNCH MEDIA LLC.
|
||||
|
||||
For licensing inquiries: licensing@rugmunch.io
|
||||
For commercial licensing: enterprise@rugmunch.io
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
||||
================================================================================
|
||||
DUAL LICENSING NOTICE
|
||||
================================================================================
|
||||
Pry is dual-licensed. Most of the codebase (the core scraping engine, extraction,
|
||||
templates, MCP server, x402 payment rail, CLI, and SDK) is MIT-licensed under
|
||||
the terms above. The following anti-detection / stealth components are licensed
|
||||
under the Business Source License 1.1 — see LICENSE-BSL-STEALTH:
|
||||
|
||||
- ultimate_scraper.py (10-tier fallback engine)
|
||||
- stealth_engine.py (stealth orchestration)
|
||||
- stealth_scripts/*.js (6 anti-detection JS injectors)
|
||||
- camoufox_integration.py (Firefox anti-detection)
|
||||
- tls_fingerprint.py (TLS fingerprint randomization)
|
||||
- cookie_warmer.py (cookie pre-aging)
|
||||
- behavioral_biometrics.py (behavioral mimicry)
|
||||
- adaptive.py (adaptive strategy rotation)
|
||||
- browser_pool.py (browser pre-warming for stealth)
|
||||
- network.py (Tor / SOCKS5 routing)
|
||||
- captcha_solver.py (CAPTCHA bypass)
|
||||
- shadow_dom.py (stealth DOM extraction)
|
||||
- lazy_load.py (human-like scroll)
|
||||
- signup_automator.py (human-like signup)
|
||||
- auth_connector.py (human-like login)
|
||||
|
||||
Each of those files is individually marked with:
|
||||
SPDX-License-Identifier: BSL-1.1
|
||||
The BSL terms apply only to those files. All other files (including all
|
||||
templates/, browser-extension/, wordpress-plugin/, shopify-app/, llm_providers/,
|
||||
and the bulk of api.py, mcp_production.py, x402.py) are MIT-licensed.
|
||||
================================================================================
|
||||
|
|
|
|||
68
LICENSE-BSL-STEALTH
Normal file
68
LICENSE-BSL-STEALTH
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
Business Source License 1.1
|
||||
|
||||
Licensor: Rug Munch Media LLC
|
||||
Licensed Work: Pry — Stealth / Anti-Detection Module
|
||||
(the specific files listed in the DUAL LICENSING NOTICE
|
||||
in LICENSE and individually marked with
|
||||
SPDX-License-Identifier: BSL-1.1)
|
||||
Change Date: 2029-01-01
|
||||
Change License: MIT License (see LICENSE)
|
||||
|
||||
Additional Use Grant:
|
||||
You may use the Licensed Work, without a commercial license from Rug Munch
|
||||
Media LLC, for any of the following purposes:
|
||||
|
||||
1. Personal use (e.g., a developer running Pry on their own machine to
|
||||
scrape sites for personal projects).
|
||||
2. Non-production testing and development (e.g., a company running Pry
|
||||
in CI to test their own scrapers against anti-bot targets).
|
||||
3. Academic research and education.
|
||||
4. Development of non-commercial open source software (any project whose
|
||||
source code is publicly available under an OSI-approved license and
|
||||
which is not sold, licensed, or distributed for a fee).
|
||||
5. Internal evaluation by a potential customer considering a
|
||||
commercial license (capped at 90 days).
|
||||
|
||||
All other use — including any production deployment, any use by a
|
||||
for-profit entity in the course of its business operations, any SaaS /
|
||||
PaaS offering, any consulting engagement, and any redistribution as
|
||||
part of a commercial product or service — requires a separate written
|
||||
commercial license from Rug Munch Media LLC.
|
||||
|
||||
For commercial licensing, contact: enterprise@rugmunch.io
|
||||
|
||||
For full canonical text of the Business Source License 1.1, see:
|
||||
https://mariadb.com/bsl11/
|
||||
|
||||
The text reproduced below is the official license template; on the Change
|
||||
Date (2029-01-01) the Licensed Work converts to the MIT License (see
|
||||
LICENSE). Until that date the following terms govern.
|
||||
|
||||
------------------------------------------------------------------------------
|
||||
Business Source License 1.1
|
||||
|
||||
Licensor: Rug Munch Media LLC
|
||||
Licensed Work: Pry — Stealth / Anti-Detection Module
|
||||
Change Date: 2029-01-01
|
||||
Change License: MIT License
|
||||
|
||||
Additional Use Grant:
|
||||
Free for non-production personal use, non-production testing, academic
|
||||
research, and development of non-commercial open source software. See
|
||||
LICENSE for the full Additional Use Grant and the list of covered files.
|
||||
|
||||
Production use and any use by a for-profit entity in the course of its
|
||||
business requires a separate commercial license from Rug Munch Media LLC.
|
||||
Contact enterprise@rugmunch.io.
|
||||
|
||||
The Licensor hereby grants you the right to copy, modify, create derivative
|
||||
works, redistribute, and make non-production use of the Licensed Work. The
|
||||
Licensor may make an Additional Use Grant, above, permitting limited
|
||||
production use. Any other use, including production use, is prohibited.
|
||||
|
||||
All other rights are reserved by the Licensor.
|
||||
|
||||
THE LICENSED WORK IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EXPRESS
|
||||
OR IMPLIED. THE LICENSOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS
|
||||
LICENSED WORK, INCLUDING ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.
|
||||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# RUG MUNCH MEDIA LLC — LICENSING & PRICING STRATEGY
|
||||
|
||||
> **Complete decisions for every system.**
|
||||
|
|
@ -124,19 +126,20 @@ WalletPress is **THREE products** that need different strategies:
|
|||
|
||||
---
|
||||
|
||||
## 3. PRYSCRAPER — DETAILED PROFIT MODEL
|
||||
## 3. PRYSCRAPER - DETAILED PROFIT MODEL
|
||||
|
||||
### 3.1 License: PROPRIETARY (CONFIRMED)
|
||||
### 3.1 License: DUAL: MIT (core) + BSL 1.1 (stealth)
|
||||
|
||||
**Why:**
|
||||
- Competitive moat is our stealth browser, anti-detection, and bypass techniques
|
||||
- If competitors see our code, they can replicate in days
|
||||
- Crypto scrapers are a race — whoever has the best stealth wins
|
||||
- MIT for the core scraping engine builds community, adoption, trust, stars, contributions
|
||||
- BSL 1.1 protects the anti-detection/stealth moat from commercial cloning
|
||||
- 15 stealth files are BSL 1.1: ultimate_scraper.py, stealth_engine.py, stealth_scripts/*.js, camoufox_integration.py, tls_fingerprint.py, cookie_warmer.py, behavioral_biometrics.py, adaptive.py, browser_pool.py, network.py, captcha_solver.py, shadow_dom.py, lazy_load.py, signup_automator.py, auth_connector.py
|
||||
- The rest (100+ files) are MIT
|
||||
|
||||
### 3.2 Pricing — Three Models
|
||||
### 3.2 Pricing - Three Models
|
||||
|
||||
**Model A: SaaS (Primary)**
|
||||
- Hosted API at `pry.dev`
|
||||
- Hosted API at pry.dev
|
||||
- Pay-per-request with x402 micropayments
|
||||
- Free tier: 1,000 requests/month
|
||||
- Pro: $49/mo for 100K requests
|
||||
|
|
@ -156,11 +159,11 @@ WalletPress is **THREE products** that need different strategies:
|
|||
|
||||
| Endpoint | Price |
|
||||
|----------|-------|
|
||||
| `/scrape` | $0.005/call |
|
||||
| `/crawl` | $0.02/call |
|
||||
| `/extract` | $0.01/call |
|
||||
| `/screenshot` | $0.003/call |
|
||||
| `/stealth_browser` | $0.05/minute |
|
||||
| /scrape | $0.005/call |
|
||||
| /crawl | $0.02/call |
|
||||
| /extract | $0.01/call |
|
||||
| /screenshot | $0.003/call |
|
||||
| /stealth_browser | $0.05/minute |
|
||||
|
||||
**Model C: White-label Enterprise**
|
||||
- Deploy PryScraper on your infrastructure
|
||||
|
|
@ -182,8 +185,42 @@ WalletPress is **THREE products** that need different strategies:
|
|||
| White-label Enterprise | $200K | $600K | $1.2M |
|
||||
| **TOTAL** | **$310K** | **$1.02M** | **$2.2M** |
|
||||
|
||||
---
|
||||
*Plus community template contributions which increase the moat over time (unquantified but real).*
|
||||
|
||||
### 3.4 Stealth BSL Specifics
|
||||
|
||||
**Which 15 files are BSL-covered:**
|
||||
- ultimate_scraper.py
|
||||
- stealth_engine.py
|
||||
- stealth_scripts/*.js
|
||||
- camoufox_integration.py
|
||||
- tls_fingerprint.py
|
||||
- cookie_warmer.py
|
||||
- behavioral_biometrics.py
|
||||
- adaptive.py
|
||||
- browser_pool.py
|
||||
- network.py
|
||||
- captcha_solver.py
|
||||
- shadow_dom.py
|
||||
- lazy_load.py
|
||||
- signup_automator.py
|
||||
- auth_connector.py
|
||||
|
||||
**What counts as production use:**
|
||||
- Live application serving real users
|
||||
- Business internal operations
|
||||
- Deploying as a service for third parties
|
||||
|
||||
**What does NOT count:**
|
||||
- Personal use
|
||||
- Non-production testing
|
||||
- Academic research
|
||||
- Non-commercial OSS development
|
||||
- 90-day evaluation
|
||||
|
||||
**Change Date:** 2029-01-01, converts to MIT
|
||||
|
||||
**Commercial license:** enterprise@rugmunch.io
|
||||
## 4. RMI (RUG MUNCH INTELLIGENCE) — DETAILED PROFIT MODEL
|
||||
|
||||
### 4.1 License: OPEN CORE (CONFIRMED)
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
---
|
||||
title: Pry MCP + x402 Audit & 20-Win Improvement Plan
|
||||
status: canonical
|
||||
|
|
|
|||
22
PLAN.md
22
PLAN.md
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# PLAN.md — PryScraper
|
||||
|
||||
> Current sprint. 21-day cycles per ADR-0012.
|
||||
|
|
@ -6,13 +8,27 @@
|
|||
current_sprint=2026-Q3-S1 · started=2026-07-01 · ends=2026-07-21
|
||||
|
||||
## Sprint Goals
|
||||
1. _TODO: top 1-3 goals for this sprint_
|
||||
1. DONE Re-license to dual MIT + BSL 1.1 (ADR-0002) - done 2026-07-03
|
||||
2. WIP Deploy repo to /srv/pry/ - wire the new api.py (4,668 lines, 190 endpoints) so MCP and x402 actually serve
|
||||
3. WIP Wire auth - JWT/API-key middleware on the live deploy so we can monetize
|
||||
|
||||
## In Progress
|
||||
- [ ] TODO: in-progress items
|
||||
- [x] Re-license the codebase: dual MIT (core) + BSL 1.1 (stealth)
|
||||
- [x] Add SPDX headers to all 300+ files
|
||||
- [x] Write ADR-0002 documenting the decision
|
||||
- [ ] Deploy updated api.py with MCP and x402 endpoints to /srv/pry/
|
||||
- [ ] Set PRY_X402_WALLET and PRY_X402_FACILITATOR env vars on Talos
|
||||
- [ ] Run templates/validate_templates.py end-to-end, publish honest success rate
|
||||
|
||||
## Backlog (this sprint)
|
||||
- [ ] TODO: planned items
|
||||
- [ ] Migrate ~/.pry/* JSON storage to SQLite (concurrency + multi-tenant)
|
||||
- [ ] Split api.py (4,668 lines) into router modules (cap 500 lines each)
|
||||
- [ ] Replace 159 broad except Exception with specific types + add ruff BLE001
|
||||
- [ ] Surface proxy_referrals.py and referrals.py in /v1/referrals API
|
||||
- [ ] Publish browser extension (Chrome Web Store) and WordPress plugin (wordpress.org)
|
||||
- [ ] Deploy Cloudflare Worker mcp-worker.js so MCP server is publicly hosted
|
||||
- [ ] Register on Smithery, Glama, PulseMCP MCP directories
|
||||
- [ ] Build pry.dev landing page with comparison table (Pry vs Firecrawl vs ScrapingBee)
|
||||
|
||||
## Risks
|
||||
- TODO: anything blocking progress
|
||||
|
|
|
|||
19
README.md
19
README.md
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# Pry — Open any website
|
||||
|
||||
Self-hosted web scraping + browser automation API. Cloudflare bypass, document parsing, AI summarization. No API keys needed.
|
||||
|
|
@ -205,4 +207,19 @@ make check # Full audit
|
|||
|
||||
## License
|
||||
|
||||
MIT — Rug Munch Media LLC
|
||||
Pry is dual-licensed to balance community openness with the protection of its advanced anti-detection and stealth capabilities.
|
||||
|
||||
- **Most code** (core functionality, utilities, and standard modules) is licensed under the **MIT License**. This permissive license encourages community contributions and wide adoption. See [LICENSE](./LICENSE).
|
||||
|
||||
- **The anti-detection / stealth module** is licensed under the **BSL 1.1 (Business Source License)**. This module is free to use for non-production purposes; a commercial license is required for any production use. See [LICENSE-BSL-STEALTH](./LICENSE-BSL-STEALTH).
|
||||
|
||||
### Additional Use Grant
|
||||
|
||||
Under the BSL 1.1, an Additional Use Grant permits free use of the stealth module for:
|
||||
- Personal, non-commercial projects
|
||||
- Non-production testing and evaluation
|
||||
- Academic research
|
||||
- Non-commercial open source software development
|
||||
- A 90-day evaluation period for any other purpose
|
||||
|
||||
For commercial licensing (production use, enterprise deployment, or embedding in proprietary products), please contact: **enterprise@rugmunch.io**
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
---
|
||||
title: Pry Roadmap & Status
|
||||
status: canonical
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# SECURITY.md — PryScraper
|
||||
|
||||
> Threat model, secrets, auth, rate limiting, vuln disclosure.
|
||||
|
|
|
|||
14
STATUS.md
14
STATUS.md
|
|
@ -1,9 +1,14 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# STATUS.md — PryScraper
|
||||
|
||||
> Where we are RIGHT NOW. Update before every commit.
|
||||
|
||||
## Last Updated
|
||||
2026-07-02 (template, replace on first commit)
|
||||
2026-07-03 - Dual-licensing relicense complete (MIT core + BSL 1.1 stealth)
|
||||
|
||||
## Current Status
|
||||
🟡 pre-production - repo scaffolded, re-license landed, deploy out of sync. See AUDIT.md.
|
||||
|
||||
## Deployment Status
|
||||
- **Last deploy**: TBD
|
||||
|
|
@ -19,7 +24,12 @@
|
|||
- 2026-07-02: Repository created from `fleet-template`
|
||||
|
||||
## Known Issues / Tech Debt
|
||||
- _None yet_
|
||||
- Repo and deploy at /srv/pry/ are out of sync. Deploy runs a 789-line api.py; repo has 4,668-line api.py with 190 endpoints (MCP, x402, auth, etc.). MCP and x402 are written but not deployed. See AUDIT.md.
|
||||
- All 80+ site templates unverified - only ~30-40% known to work end-to-end. Run templates/validate_templates.py before claiming template coverage.
|
||||
- State storage in ~/.pry/*.json - no concurrency safety, no transactions. See plan to migrate to SQLite.
|
||||
- PRY_API_KEY, PRY_X402_WALLET, PRY_X402_FACILITATOR env vars unset in deploy - no auth, no x402, no payments.
|
||||
- License collision resolved 2026-07-03: dual MIT (core) + BSL 1.1 (stealth/anti-detection). See ADR-0002.
|
||||
- mcp_production.py and x402.py exist in repo but are NOT deployed. Top priority for next deploy.
|
||||
|
||||
## Quick Links
|
||||
- [Live URL](https://pryscraper.rugmunch.io)
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# TESTING.md — PryScraper
|
||||
|
||||
> Test strategy. Unit, integration, e2e, fixtures.
|
||||
|
|
|
|||
2
USAGE.md
2
USAGE.md
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
---
|
||||
title: Pry Usage Guide
|
||||
status: canonical
|
||||
|
|
|
|||
|
|
@ -1,3 +1,8 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
"""Pry — Account pool management, session persistence, proxy scoring."""
|
||||
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -2,6 +2,12 @@
|
|||
Pre-built scrapers that can be subscribed to and run on schedule.
|
||||
Users can publish their own actors. We earn from usage."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
|
|
|
|||
|
|
@ -1,6 +1,13 @@
|
|||
"""Pry — adaptive crawling with information foraging.
|
||||
Intelligently decides when to stop crawling based on content relevance."""
|
||||
|
||||
# SPDX-License-Identifier: BSL-1.1
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — Stealth / Anti-Detection Module
|
||||
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
|
||||
# Change Date: 2029-01-01 (converts to MIT).
|
||||
|
||||
import logging
|
||||
import re
|
||||
from collections import Counter
|
||||
|
|
|
|||
|
|
@ -3,6 +3,12 @@ Diff tracking, LLM summarization, RSS generation, change monitoring,
|
|||
schema.org extraction, email finder, and more.
|
||||
All powered by local Ollama — no API keys needed."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import difflib
|
||||
import hashlib
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — White-Label Agency Dashboard.
|
||||
Multi-tenant reseller platform: custom branding, client management, sub-accounts."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — AI Agent Plugin.
|
||||
OpenAPI spec server, MCP server wrapper, and GPT Action helper."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Multi-Channel Alerting System.
|
||||
SMS, Email, Microsoft Teams, Discord, Telegram alerts."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,8 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
"""Pry — Real anomaly detection. Multi-field, time-series aware, with seasonality support."""
|
||||
|
||||
import logging
|
||||
|
|
|
|||
6
api.py
6
api.py
|
|
@ -2,6 +2,12 @@
|
|||
Scrape + Crawl + Automate + Parse + Extract + MCP
|
||||
Self-hosted, free, better than Firecrawl."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import difflib
|
||||
|
|
|
|||
5
auth.py
5
auth.py
|
|
@ -1,3 +1,8 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
"""Pry — Authentication with JWT and API keys, per-key rate limiting."""
|
||||
|
||||
import hashlib
|
||||
|
|
|
|||
|
|
@ -1,6 +1,13 @@
|
|||
"""Pry — Enterprise SSO / Auth Connector System.
|
||||
Credential vault, session persistence, SSO flow automation, CAPTCHA integration."""
|
||||
|
||||
# SPDX-License-Identifier: BSL-1.1
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — Stealth / Anti-Detection Module
|
||||
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
|
||||
# Change Date: 2029-01-01 (converts to MIT).
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
|
|
|
|||
|
|
@ -6,6 +6,12 @@ Audited: session cleanup runs automatically, no injection vectors,
|
|||
all user inputs validated before passing to Playwright.
|
||||
"""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -3,6 +3,13 @@ Real human behavior simulation: hesitation, scroll-back, mouse drift, reading ti
|
|||
Modern anti-bot systems detect 'too perfect' behavior. This module makes
|
||||
behavior more realistic by adding human imperfections."""
|
||||
|
||||
# SPDX-License-Identifier: BSL-1.1
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — Stealth / Anti-Detection Module
|
||||
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
|
||||
# Change Date: 2029-01-01 (converts to MIT).
|
||||
|
||||
import logging
|
||||
import math
|
||||
import random
|
||||
|
|
|
|||
|
|
@ -1,3 +1,7 @@
|
|||
// SPDX-License-Identifier: MIT
|
||||
// Copyright (c) 2026 Rug Munch Media LLC
|
||||
// Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
// Licensed under MIT. See LICENSE.
|
||||
// Pry Browser Extension — Background Service Worker
|
||||
chrome.runtime.onInstalled.addListener(function() {
|
||||
chrome.storage.sync.get(['pryServerUrl'], function(result) {
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
<!-- SPDX-License-Identifier: MIT
|
||||
<!-- Copyright (c) 2026 Rug Munch Media LLC -->
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
|
|
|
|||
|
|
@ -1,3 +1,7 @@
|
|||
// SPDX-License-Identifier: MIT
|
||||
// Copyright (c) 2026 Rug Munch Media LLC
|
||||
// Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
// Licensed under MIT. See LICENSE.
|
||||
// Pry Browser Extension — Popup Script
|
||||
document.addEventListener('DOMContentLoaded', function() {
|
||||
loadSettings();
|
||||
|
|
|
|||
|
|
@ -1,6 +1,13 @@
|
|||
"""Pry — shared Playwright browser pool.
|
||||
Reuses browser instances across requests instead of launching per scrape."""
|
||||
|
||||
# SPDX-License-Identifier: BSL-1.1
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — Stealth / Anti-Detection Module
|
||||
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
|
||||
# Change Date: 2029-01-01 (converts to MIT).
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any
|
||||
|
|
|
|||
6
cache.py
6
cache.py
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — in-memory LRU cache with Redis persistence.
|
||||
Prevents re-scraping the same URL within TTL window."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
|
|
|
|||
|
|
@ -3,6 +3,13 @@ Camoufox patches Firefox at the source level to bypass fingerprinting
|
|||
more effectively than Playwright/Puppeteer. It's a drop-in alternative
|
||||
for Playwright that focuses on stealth."""
|
||||
|
||||
# SPDX-License-Identifier: BSL-1.1
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — Stealth / Anti-Detection Module
|
||||
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
|
||||
# Change Date: 2029-01-01 (converts to MIT).
|
||||
|
||||
import logging
|
||||
import random
|
||||
import time
|
||||
|
|
|
|||
|
|
@ -2,6 +2,13 @@
|
|||
Supports: Capsolver -> 2Captcha -> Anti-Captcha -> CapMonster -> DeathByCaptcha -> NextCaptcha
|
||||
Also handles Turnstile, reCAPTCHA v2/v3/invisible, and hCaptcha."""
|
||||
|
||||
# SPDX-License-Identifier: BSL-1.1
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — Stealth / Anti-Detection Module
|
||||
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
|
||||
# Change Date: 2029-01-01 (converts to MIT).
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
|
|
|
|||
6
cli.py
6
cli.py
|
|
@ -15,6 +15,12 @@ Usage:
|
|||
pry proxy ... Manage proxy providers and signup
|
||||
"""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — shared httpx client pool with connection reuse.
|
||||
All modules import `http_client` instead of creating new clients per request."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import logging
|
||||
|
||||
import httpx
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Commerce Platform Sync Engine.
|
||||
Unified interface for WooCommerce, Shopify, and generic API sync."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Legal Compliance Engine.
|
||||
Per-source compliance scorecard: robots.txt, ToS, GDPR/CCPA, jurisdiction."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import logging
|
||||
import re
|
||||
from datetime import UTC, datetime
|
||||
|
|
|
|||
|
|
@ -3,6 +3,12 @@
|
|||
Loads settings from environment variables (via Pydantic Settings).
|
||||
All secrets come from gopass, never from .env files committed to git.
|
||||
"""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
|
|
|||
|
|
@ -3,6 +3,13 @@ Pre-age cookies by browsing legitimate pages first. Aged cookies with
|
|||
realistic browsing history bypass anti-bot detection that checks for
|
||||
'fresh' cookies. This is a technique used by professional scraping services."""
|
||||
|
||||
# SPDX-License-Identifier: BSL-1.1
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — Stealth / Anti-Detection Module
|
||||
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
|
||||
# Change Date: 2029-01-01 (converts to MIT).
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Cost Analytics Engine.
|
||||
Tracks usage costs, cache hit rates, projected burn, and smart scheduling."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Reverse ETL to CRM.
|
||||
Sync scraped data to Salesforce, HubSpot, Pipedrive, and Close.com."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
|
|
|
|||
6
db.py
6
db.py
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Database layer using SQLAlchemy (SQLite/PostgreSQL).
|
||||
Replaces JSON file storage for production safety."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import Generator
|
||||
|
|
|
|||
6
dedup.py
6
dedup.py
|
|
@ -2,6 +2,12 @@
|
|||
Detects near-duplicate content (e.g., 95% similar pages) using SimHash.
|
||||
Useful for crawl deduplication, change detection, and content grouping."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — One-Click Business Integrations.
|
||||
Native write destinations: Google Sheets, Slack, Airtable, email."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
|
|
|
|||
|
|
@ -1,3 +1,7 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
services:
|
||||
pry:
|
||||
build: .
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# ADR-NNNN: <Short Title>
|
||||
|
||||
## Status
|
||||
|
|
|
|||
|
|
@ -1,3 +1,5 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# ADR-0001: Initial Architecture
|
||||
|
||||
## Status
|
||||
|
|
|
|||
55
docs/adr/0002-dual-licensing.md
Normal file
55
docs/adr/0002-dual-licensing.md
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
[//]: # (SPDX-License-Identifier: MIT)
|
||||
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
||||
# ADR-0002: Dual Licensing (MIT + BSL 1.1)
|
||||
|
||||
## Status
|
||||
Accepted
|
||||
|
||||
## Date
|
||||
2026-07-03
|
||||
|
||||
## Context
|
||||
|
||||
The Pry (munchcrawl) project was initially released under a full proprietary license. This licensing model resulted in zero community engagement, zero external contributors, and zero GitHub stars. In the competitive web scraping and data extraction market, open source projects such as Firecrawl and Crawl4AI have achieved dominant positions by leveraging MIT licensing to build strong community moats, attract contributors, and establish trust with users.
|
||||
|
||||
The existing LICENSING_PRICING_STRATEGY.md document previously listed Pry as PROPRIETARY. This approach proved incompatible with the project's goal of achieving widespread adoption, particularly for the Model Context Protocol (MCP) server and template marketplace components that benefit from community contributions and third-party integrations.
|
||||
|
||||
The core challenge is that a fully proprietary license prevents the project from competing on community moat. Without open source licensing, potential users cannot inspect the codebase for security vulnerabilities, cannot contribute improvements, and cannot build trust in the scraping engine's reliability. The project requires a licensing strategy that balances community growth with protection of proprietary anti-detection techniques that constitute the project's competitive advantage.
|
||||
|
||||
## Decision
|
||||
|
||||
Pry will adopt a dual licensing model consisting of MIT for the core scraping engine and Business Source License 1.1 (BSL 1.1) for the anti-detection and stealth components.
|
||||
|
||||
The MIT license will apply to the core scraping engine, comprising over 100 files including: the scraper module, extractor module, parser module, all 80+ templates, MCP server, x402 protocol implementation, command-line interface, software development kit, browser extension, WordPress plugin, Shopify app, and all LLM provider integrations. The MIT license permits free use, modification, and redistribution of these files by any party.
|
||||
|
||||
The BSL 1.1 license will apply to 15 anti-detection and stealth files: ultimate_scraper.py, stealth_engine.py, all stealth_scripts/*.js files, camoufox_integration.py, tls_fingerprint.py, cookie_warmer.py, behavioral_biometrics.py, adaptive.py, browser_pool.py, network.py, captcha_solver.py, shadow_dom.py, lazy_load.py, signup_automator.py, and auth_connector.py. The BSL 1.1 includes an Additional Use Grant that permits personal use, non-production testing, academic research, non-commercial open source software development, and 90-day evaluation. The BSL Change License is MIT, not GPLv2. The Change Date is set to 2029-01-01, after which all BSL-licensed files automatically convert to MIT.
|
||||
|
||||
## Consequences
|
||||
|
||||
Positive: The MIT-licensed core engine will enable community growth, attract GitHub stars, build user trust through code transparency, and drive adoption of the MCP server and template marketplace. External contributors can submit improvements to the scraping engine, parser, and integrations.
|
||||
|
||||
Positive: Users and auditors can inspect the core scraping engine for security vulnerabilities, data handling practices, and code quality. This transparency is essential for enterprise adoption and regulatory compliance.
|
||||
|
||||
Positive: The BSL 1.1 protection on anti-detection files preserves the project's competitive moat. Competitors can see and copy the core engine code under MIT, but cannot legally copy the anti-detection techniques that differentiate Pry from other scraping tools.
|
||||
|
||||
Negative: Competitors can freely use, modify, and redistribute the core scraping engine code. This may lead to derivative projects that compete with Pry's commercial offerings, though they will lack the protected anti-detection capabilities.
|
||||
|
||||
Negative: The dual licensing model introduces complexity in license compliance. Contributors and users must understand which files are under MIT versus BSL 1.1, and the BSL 1.1 Additional Use Grant limitations must be clearly communicated.
|
||||
|
||||
## Alternatives
|
||||
|
||||
Full MIT license: Rejected because it would expose all anti-detection techniques to competitors, eliminating the project's primary competitive advantage. Without protection of these techniques, the project would have no sustainable differentiation in the market.
|
||||
|
||||
Full proprietary license: Rejected because it resulted in zero community engagement, zero contributors, and zero adoption. This model proved incompatible with building the community moat necessary to compete with Firecrawl and Crawl4AI.
|
||||
|
||||
AGPL version 3 plus commercial license: Rejected because the AGPL's viral copyleft nature would deter many potential users and contributors who prefer permissive licensing. The AGPL also creates compatibility issues with the MIT-licensed ecosystem that the project depends upon.
|
||||
|
||||
BSL 1.1 on all files: Rejected because applying BSL to the entire codebase would prevent the community adoption and trust-building that only a permissive license can provide. Users would be unable to freely use the core engine without evaluating the BSL's Additional Use Grant limitations.
|
||||
|
||||
## References
|
||||
|
||||
- LICENSE (MIT) - Core scraping engine license file
|
||||
- LICENSE-BSL-STEALTH (BSL 1.1) - Anti-detection components license file
|
||||
- https://mariadb.com/bsl11/ - Business Source License 1.1 official documentation
|
||||
- ADR-0000-template.md - Architecture Decision Record template used for this document
|
||||
- LICENSING_PRICING_STRATEGY.md - Previous licensing strategy document (superseded by this ADR)
|
||||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Email Inbox Scraping.
|
||||
Connect Gmail/Outlook, extract structured data from emails."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import base64
|
||||
import logging
|
||||
import re
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Data Enrichment Pipeline.
|
||||
Enrich scraped data with company info, social profiles, tech stack detection."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Any
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — typed error hierarchy.
|
||||
Every API error flows through here for consistent JSON responses."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
from typing import Any
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — structured extraction strategies.
|
||||
CSS/XPath-based extraction (no LLM needed) + chunking strategies for LLM extraction."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import logging
|
||||
import math
|
||||
import re
|
||||
|
|
|
|||
|
|
@ -4,6 +4,12 @@ LLM failures fall back gracefully to pattern mode.
|
|||
No hallucination: JSON output is always parsed and validated.
|
||||
"""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import re
|
||||
from typing import Any
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Adaptive Freshness Scheduling.
|
||||
Conditional scraping, content fingerprinting, staleness dashboard, adaptive frequency."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
|
|
|
|||
6
gdpr.py
6
gdpr.py
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — GDPR Compliance Portal.
|
||||
Data deletion API, consent management, retention policies, audit log."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
|
|
|
|||
|
|
@ -1,3 +1,8 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
"""Pry — Real GDPR compliance: data subject access requests, data portability, real audit log."""
|
||||
|
||||
import contextlib
|
||||
|
|
|
|||
48
glama.json
48
glama.json
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"$schema": "https://glama.ai/mcp/schema.json",
|
||||
"name": "pry",
|
||||
"title": "Pry — Web Scraping & Browser Automation MCP",
|
||||
"title": "Pry \u2014 Web Scraping & Browser Automation MCP",
|
||||
"description": "Self-hosted web intelligence platform: scrape, crawl, extract, automate, and parse any website. Exposed as a Model Context Protocol server.",
|
||||
"version": "3.0.0",
|
||||
"license": "MIT",
|
||||
|
|
@ -27,14 +27,38 @@
|
|||
}
|
||||
],
|
||||
"tools": [
|
||||
{ "name": "pry_scrape", "description": "Scrape a URL to clean markdown" },
|
||||
{ "name": "pry_crawl", "description": "Crawl a website from a starting URL" },
|
||||
{ "name": "pry_extract", "description": "Extract structured data with CSS selectors" },
|
||||
{ "name": "pry_template", "description": "Execute a pre-built scraper template" },
|
||||
{ "name": "pry_search_templates", "description": "Search scraper templates" },
|
||||
{ "name": "pry_enrich", "description": "Enrich a URL with company and tech intelligence" },
|
||||
{ "name": "pry_x402_pricing", "description": "Get x402 pay-per-call pricing" },
|
||||
{ "name": "pry_referrals", "description": "Get the referral catalog" }
|
||||
{
|
||||
"name": "pry_scrape",
|
||||
"description": "Scrape a URL to clean markdown"
|
||||
},
|
||||
{
|
||||
"name": "pry_crawl",
|
||||
"description": "Crawl a website from a starting URL"
|
||||
},
|
||||
{
|
||||
"name": "pry_extract",
|
||||
"description": "Extract structured data with CSS selectors"
|
||||
},
|
||||
{
|
||||
"name": "pry_template",
|
||||
"description": "Execute a pre-built scraper template"
|
||||
},
|
||||
{
|
||||
"name": "pry_search_templates",
|
||||
"description": "Search scraper templates"
|
||||
},
|
||||
{
|
||||
"name": "pry_enrich",
|
||||
"description": "Enrich a URL with company and tech intelligence"
|
||||
},
|
||||
{
|
||||
"name": "pry_x402_pricing",
|
||||
"description": "Get x402 pay-per-call pricing"
|
||||
},
|
||||
{
|
||||
"name": "pry_referrals",
|
||||
"description": "Get the referral catalog"
|
||||
}
|
||||
],
|
||||
"tags": [
|
||||
"web-scraping",
|
||||
|
|
@ -43,5 +67,7 @@
|
|||
"x402",
|
||||
"self-hosted",
|
||||
"mcp"
|
||||
]
|
||||
}
|
||||
],
|
||||
"_license": "MIT",
|
||||
"_copyright": "Copyright (c) 2026 Rug Munch Media LLC"
|
||||
}
|
||||
|
|
@ -3,6 +3,12 @@ Detects GraphQL endpoints, runs introspection queries, generates optimized queri
|
|||
Many modern sites (Shopify, GitHub, Twitter/X, etc.) have GraphQL APIs that are
|
||||
10-100x more efficient than scraping HTML."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Competitive Intelligence Engine.
|
||||
Historical snapshots, anomaly detection, natural-language alerts, weekly reports."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — batch job queue with Redis and webhook callbacks.
|
||||
Async processing engine inspired by Firecrawl's webhook system."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -1,3 +1,9 @@
|
|||
# SPDX-License-Identifier: BSL-1.1
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — Stealth / Anti-Detection Module
|
||||
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
|
||||
# Change Date: 2029-01-01 (converts to MIT).
|
||||
"""Pry — lazy load and infinite scroll handling."""
|
||||
|
||||
import logging
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Real LLM-powered features. Replaces regex stubs with actual LLM calls.
|
||||
Used by compliance, SEO, entity reconciliation, PII redaction, and other AI features."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Any
|
||||
|
|
|
|||
|
|
@ -0,0 +1,5 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
|
@ -2,6 +2,12 @@
|
|||
Supports pluggable providers: OpenAI, Anthropic, Google, Cohere, Mistral, Ollama, OpenRouter.
|
||||
Includes referral/affiliate link tracking for revenue sharing."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
|
|
|
|||
|
|
@ -1,3 +1,8 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
"""Concrete LLM provider implementations."""
|
||||
|
||||
import logging
|
||||
|
|
|
|||
|
|
@ -1,3 +1,8 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
"""LLM provider registry with fallback chain and cost tracking."""
|
||||
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — markdown generation strategies with content filtering.
|
||||
Raw, Fit (pruning), and Fit+BM25 (query-relevant) markdown generators."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import logging
|
||||
import math
|
||||
import re
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry Config — runtime configuration for proxy, Tor, VPN, and anti-detection.
|
||||
Users can configure at startup (env vars) or at runtime (API calls)."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,12 @@ Standard MCP features implemented:
|
|||
Run: python -m mcp_production
|
||||
"""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry MCP Server — expose scrape, crawl, automate as MCP tools.
|
||||
Enables any MCP-compatible AI agent (Claude, Hermes, Cursor) to use Pry."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -10,6 +10,12 @@ Session state is kept in memory. For production multi-worker deployments, use a
|
|||
Redis-backed queue or sticky sessions.
|
||||
"""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — scheduled monitors with AI change detection.
|
||||
Cron-based monitors that diff content and judge meaningful changes."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import difflib
|
||||
import json
|
||||
import logging
|
||||
|
|
|
|||
|
|
@ -1,6 +1,13 @@
|
|||
"""Pry — network capture and lazy load handling.
|
||||
Captures XHR/fetch requests from JS-heavy SPAs and handles lazy content."""
|
||||
|
||||
# SPDX-License-Identifier: BSL-1.1
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — Stealth / Anti-Detection Module
|
||||
# Licensed under Business Source License 1.1 — see LICENSE-BSL-STEALTH.
|
||||
# Change Date: 2029-01-01 (converts to MIT).
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
|
|
|
|||
|
|
@ -1,3 +1,8 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
"""Pry — Observability: Prometheus metrics, OpenTelemetry tracing, structured logging."""
|
||||
|
||||
import logging
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Image OCR using Tesseract.
|
||||
Extract text from images on web pages. Uses pytesseract + Pillow."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
|
|
|
|||
|
|
@ -3,6 +3,12 @@ Uses asyncio subprocess for CPU-bound OCR to avoid blocking the event loop.
|
|||
Temp files are always cleaned up in finally blocks.
|
||||
"""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import asyncio
|
||||
import io
|
||||
import os
|
||||
|
|
|
|||
|
|
@ -2,6 +2,12 @@
|
|||
Extracts structured tables from PDF documents (financial reports, invoices, etc.)
|
||||
Uses pdfplumber, camelot, and pdfminer as fallback methods."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — pipeline hook system for the scraping workflow.
|
||||
Allows plugins and custom transformations at every stage."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import logging
|
||||
from collections.abc import Awaitable, Callable
|
||||
from typing import Any, cast
|
||||
|
|
|
|||
|
|
@ -4,6 +4,12 @@ the engine executes them sequentially with branching and error handling.
|
|||
|
||||
A UI can render these steps as drag-and-drop blocks."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
|
|
|
|||
|
|
@ -2,6 +2,12 @@
|
|||
Tries free proxies first (Tor, public rotating). When premium proxy is needed,
|
||||
prompts user to sign up via affiliate links. Pre-wired to 5+ providers."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
|
|
|
|||
|
|
@ -5,6 +5,12 @@ Usage:
|
|||
result = await mc.scrape("https://example.com")
|
||||
"""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — WebSocket streaming, scheduler, batch-file, recorder, transforms.
|
||||
New capabilities that make Pry unbeatable."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import csv
|
||||
import io
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -17,6 +17,12 @@ jobs:
|
|||
|
||||
"""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import json
|
||||
import os
|
||||
|
||||
|
|
|
|||
|
|
@ -1,14 +1,18 @@
|
|||
{
|
||||
"$schema": "https://pulsemcp.com/schema.json",
|
||||
"name": "pry",
|
||||
"display_name": "Pry — Web Scraping & Browser Automation",
|
||||
"display_name": "Pry \u2014 Web Scraping & Browser Automation",
|
||||
"description": "Scrape, crawl, extract, and automate any website. Self-hosted, x402 pay-per-call, MCP-compatible.",
|
||||
"version": "3.0.0",
|
||||
"license": "MIT",
|
||||
"author": "Rug Munch Media LLC",
|
||||
"homepage": "https://pry.dev",
|
||||
"repository": "https://github.com/cryptorugmuncher/pry",
|
||||
"categories": ["Web Scraping", "Browser Automation", "Data Extraction"],
|
||||
"categories": [
|
||||
"Web Scraping",
|
||||
"Browser Automation",
|
||||
"Data Extraction"
|
||||
],
|
||||
"install": {
|
||||
"type": "pip",
|
||||
"package": "pry",
|
||||
|
|
@ -32,5 +36,13 @@
|
|||
"pry_x402_pricing",
|
||||
"pry_referrals"
|
||||
],
|
||||
"tags": ["scraping", "automation", "extraction", "x402", "self-hosted"]
|
||||
}
|
||||
"tags": [
|
||||
"scraping",
|
||||
"automation",
|
||||
"extraction",
|
||||
"x402",
|
||||
"self-hosted"
|
||||
],
|
||||
"_license": "MIT",
|
||||
"_copyright": "Copyright (c) 2026 Rug Munch Media LLC"
|
||||
}
|
||||
|
|
@ -1,3 +1,7 @@
|
|||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
[build-system]
|
||||
requires = ["setuptools>=69.0"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
|
@ -8,8 +12,23 @@ version = "3.0.0"
|
|||
description = "Free web scraping + browser automation API — self-hosted, no API keys needed"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
license = { text = "Proprietary" }
|
||||
license = { text = "MIT AND BSL-1.1" }
|
||||
authors = [{ name = "Rug Munch Media LLC" }]
|
||||
keywords = ["scraping", "crawler", "browser-automation", "mcp", "x402", "ai-agents"]
|
||||
classifiers = [
|
||||
"Development Status :: 4 - Beta",
|
||||
"License :: OSI Approved :: MIT License",
|
||||
"License :: Other/Proprietary License",
|
||||
"Operating System :: OS Independent",
|
||||
"Programming Language :: Python :: 3",
|
||||
"Programming Language :: Python :: 3.11",
|
||||
"Programming Language :: Python :: 3.12",
|
||||
"Topic :: Internet :: WWW/HTTP :: Browsers",
|
||||
"Topic :: Software Development :: Libraries :: Python Modules",
|
||||
"Typing :: Typed",
|
||||
]
|
||||
[project.license-files]
|
||||
paths = ["LICENSE", "LICENSE-BSL-STEALTH"]
|
||||
dependencies = [
|
||||
"fastapi>=0.115.0",
|
||||
"uvicorn[standard]>=0.32.0",
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Data Quality SLA Dashboard.
|
||||
Per-extraction quality metrics, anomaly detection, freshness tracking."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import difflib
|
||||
import hashlib
|
||||
import json
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — token-bucket rate limiter per IP.
|
||||
Prevents abuse while keeping the service responsive."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import time
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,12 @@
|
|||
"""Pry — Multi-Source Entity Reconciliation.
|
||||
Cross-source entity matching, unified schema mapping, diff dashboard."""
|
||||
|
||||
# SPDX-License-Identifier: MIT
|
||||
# Copyright (c) 2026 Rug Munch Media LLC
|
||||
#
|
||||
# Part of Pry — https://git.rugmunch.io/RugMunchMedia/pryscraper
|
||||
# Licensed under MIT. See LICENSE.
|
||||
|
||||
import difflib
|
||||
import hashlib
|
||||
import logging
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue