fix: restore Cinnabox fleet config files after merge
Restore home-directory .editorconfig, .mise.toml, README.md that were shadowed by degenfeed-web project versions during unrelated-history merge.
This commit is contained in:
parent
d652bf4fab
commit
8d51b57488
3 changed files with 162 additions and 48 deletions
|
|
@ -4,26 +4,27 @@ root = true
|
|||
charset = utf-8
|
||||
end_of_line = lf
|
||||
indent_style = space
|
||||
indent_size = 2
|
||||
indent_size = 4
|
||||
insert_final_newline = true
|
||||
trim_trailing_whitespace = true
|
||||
max_line_length = 100
|
||||
|
||||
[*.md]
|
||||
trim_trailing_whitespace = false
|
||||
|
||||
[Makefile]
|
||||
indent_style = tab
|
||||
|
||||
[*.{py,pyi}]
|
||||
indent_size = 4
|
||||
max_line_length = 100
|
||||
|
||||
[*.{yml,yaml}]
|
||||
indent_size = 2
|
||||
|
||||
[*.{json,jsonc}]
|
||||
[*.{js,ts,jsx,tsx,json,jsonc}]
|
||||
indent_size = 2
|
||||
|
||||
[*.{md,mdown,mkd,markdown}]
|
||||
trim_trailing_whitespace = false
|
||||
max_line_length = 120
|
||||
|
||||
[Makefile]
|
||||
indent_style = tab
|
||||
|
||||
[*.py]
|
||||
indent_size = 4
|
||||
max_line_length = 100
|
||||
|
||||
[{*.bat,*.cmd,*.ps1}]
|
||||
end_of_line = crlf
|
||||
end_of_line = crlf
|
||||
|
|
|
|||
11
.mise.toml
11
.mise.toml
|
|
@ -1,3 +1,12 @@
|
|||
[tools]
|
||||
python = "3.12"
|
||||
node = "22"
|
||||
pnpm = "11.7.0"
|
||||
bun = "1"
|
||||
|
||||
[env]
|
||||
# Managed by mise — reproducible dev environments across Cinnabox/Talos/Hydra
|
||||
|
||||
[settings]
|
||||
always_keep_download = false
|
||||
legacy_version_file = true
|
||||
experimental = true
|
||||
|
|
|
|||
172
README.md
172
README.md
|
|
@ -1,44 +1,148 @@
|
|||
# DegenFeed Web
|
||||
# Pry — Pry open any website.
|
||||
|
||||
**Repo:** git.rugmunch.io/RugMunchMedia/degenfeed-web
|
||||
**Status:** Active development
|
||||
**Note:** This repo is the Next.js social-aggregator client for Nostr, Farcaster, Lens, and Bluesky. It is NOT the deployed Mastodon instance at degenfeed.xyz.
|
||||
> **Web scraping + Cloudflare bypass + browser automation + change monitoring.**
|
||||
> Self-hosted. One command. Yours forever. No monthly bills.
|
||||
|
||||
## What this repo is
|
||||
A web client that aggregates open social protocols into a single feed and composer.
|
||||
Planned stack: Next.js 14 App Router, React 19, Tailwind, Radix, Zustand, TanStack Query.
|
||||
```bash
|
||||
docker run -d -p 8005:8002 pry/pry
|
||||
curl -X POST http://localhost:8005/v1/scrape \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"url":"https://example.com"}'
|
||||
```
|
||||
|
||||
## What this repo is NOT
|
||||
- The deployed DegenFeed Mastodon server runs at /srv/degenfeed/ container degenfeed-web on port 4002
|
||||
- A backend API. Protocol adapters live in packages/.
|
||||
---
|
||||
|
||||
## Status
|
||||
- apps/landing is a static landing page live at degenfeed.xyz
|
||||
- apps/web is scaffolding; not deployed
|
||||
- This repo currently has no health endpoint because apps/web is not running
|
||||
## Why Pry?
|
||||
|
||||
## Quick start when apps/web exists
|
||||
npm install
|
||||
npm run dev
|
||||
| vs | Firecrawl | ScraperAPI | ScrapingBee | **Pry** |
|
||||
|---|---|---|---|---|
|
||||
| Price | $99/mo | $49/mo | $49/mo | **$29 once** |
|
||||
| Cloudflare bypass | Paid addon | ❌ | ❌ | **✅ Built-in** |
|
||||
| Browser automation | ❌ | ❌ | ❌ | **✅ Sessions** |
|
||||
| Change monitoring | $99/mo | ❌ | ❌ | **✅ Free** |
|
||||
| Local AI extraction | ❌ | ❌ | ❌ | **✅ Via Ollama** |
|
||||
| Tor proxy | ❌ | ❌ | ❌ | **✅ 1-click** |
|
||||
| Self-hosted | ❌ | ❌ | ❌ | **✅ Docker** |
|
||||
| Real-time streaming | ❌ | ❌ | ❌ | **✅ WebSocket** |
|
||||
|
||||
## Deployed Mastodon separate
|
||||
ssh netcup
|
||||
cd /srv/degenfeed/
|
||||
docker compose ps
|
||||
---
|
||||
|
||||
## Current issues
|
||||
- No PRODUCT/ARCHITECTURE/ROADMAP/TODO docs now added
|
||||
- No pre-commit config
|
||||
- No Makefile
|
||||
- No CI
|
||||
- Repo README conflated this client with the Mastodon deployment
|
||||
## Features
|
||||
|
||||
## Standards
|
||||
This repo follows Rug Munch Media standards:
|
||||
https://git.rugmunch.io/RugMunchMedia/standards
|
||||
### 🛡️ Cloudflare? Never heard of it
|
||||
Pry's 4-tier engine burns through every protection:
|
||||
1. **Direct** — rotating UAs, browser-like headers
|
||||
2. **FlareSolverr** — Cloudflare bypass on-demand
|
||||
3. **Playwright stealth** — full browser with fingerprint spoofing
|
||||
4. **Googlebot fallback** — last resort with search engine trust
|
||||
|
||||
### 🤖 Browser automation that persists
|
||||
Login once. Scrape for hours. Sessions persist via cookies.
|
||||
|
||||
```python
|
||||
# Login, then scrape authenticated pages
|
||||
steps = [
|
||||
{"action": "navigate", "url": "https://site.com/login"},
|
||||
{"action": "type", "selector": "#email", "value": "user@x.com"},
|
||||
{"action": "click", "selector": "#login-btn"},
|
||||
{"action": "screenshot"},
|
||||
]
|
||||
```
|
||||
|
||||
### 🔍 AI-powered extraction
|
||||
Point at any page. Local Ollama finds the structure automatically.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8005/v1/scrape \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"url":"https://store.com/product","jsonSchema":{"price":"the price","name":"product name"}}'
|
||||
```
|
||||
|
||||
### 👀 Watch pages for changes
|
||||
Free change monitoring. Get Slack/Discord alerts when pages update.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8005/v1/watch \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"url":"https://site.com/page","webhook":"https://hooks.slack.com/..."}'
|
||||
```
|
||||
|
||||
### 📋 One-file config
|
||||
Define everything in `pry.yml`:
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
- name: prices
|
||||
url: https://store.com/products
|
||||
schedule: every 1h
|
||||
extract:
|
||||
title: "h1"
|
||||
price: ".price"
|
||||
output: csv
|
||||
webhook: slack://C012345
|
||||
```
|
||||
|
||||
Run it: `pry run`
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Docker (recommended)
|
||||
```bash
|
||||
docker run -d -p 8005:8002 --name pry pry/pry
|
||||
```
|
||||
|
||||
### 2. Compose (with Cloudflare bypass)
|
||||
```bash
|
||||
curl -O https://pry.dev/docker-compose.yml
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### 3. pip
|
||||
```bash
|
||||
pip install pry
|
||||
pry serve
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pricing
|
||||
|
||||
| Tier | Price | What you get |
|
||||
|------|-------|-------------|
|
||||
| **Free** (GitHub) | $0 | Source code, Docker, self-host, AGPL |
|
||||
| **Pro** | **$29 once** | Pre-built image, 30 templates, CLI autocomplete, commercial license, 1yr updates |
|
||||
|
||||
**$29. Lifetime. No subscription.**
|
||||
|
||||
---
|
||||
|
||||
## Docs
|
||||
- PRODUCT.md — charter
|
||||
- ARCHITECTURE.md — packages and protocol adapters
|
||||
- ROADMAP.md — next 90 days
|
||||
- TODO.md — current tasks
|
||||
|
||||
Full API reference at [pry.dev/docs](https://pry.dev/docs)
|
||||
|
||||
### Core endpoints
|
||||
| Endpoint | Method | What it does |
|
||||
|----------|--------|-------------|
|
||||
| `/v1/scrape` | POST | Scrape any URL |
|
||||
| `/v1/crawl` | POST | Crawl multiple pages |
|
||||
| `/v1/automate` | POST | Browser automation |
|
||||
| `/v1/batch` | POST | Batch scrape 50 URLs |
|
||||
| `/v1/watch` | POST | Monitor page changes |
|
||||
| `/v1/stream` | WS | Real-time data stream |
|
||||
| `/v1/parse` | POST | Parse PDF/DOCX/images |
|
||||
| `/v1/transform` | POST | Convert data format |
|
||||
| `/v1/run` | POST | Execute pry.yml jobs |
|
||||
| `/dashboard` | GET | Health dashboard |
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
- **Community Edition:** AGPL v3 — free for self-hosted use
|
||||
- **Pro Edition:** Commercial license — $29 one-time
|
||||
|
||||
---
|
||||
|
||||
*Built for developers who refuse to pay $99/month for a scraper.*
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue