RugMunchMedia/rmi-backend
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

merge: chore/cleanup-remove-bloat-and-secrets into main
Some checks are pending
RMI CI / heartbeat (push) Waiting to run
Details
RMI CI / lint (push) Waiting to run
Details
RMI CI / typecheck (push) Waiting to run
Details
RMI CI / test (push) Waiting to run
Details
RMI CI / security (push) Waiting to run
Details
RMI CI / openapi (push) Waiting to run
Details
RMI CI / qdrant-cleanup (push) Waiting to run
Details
Deploy RMI Backend / deploy (push) Waiting to run
Details

Browse source
This commit is contained in:
crmuncher 2026-07-02 01:24:22 +07:00
commit e5133f91d4
No known key found for this signature in database
GPG key ID: 58D4300721937626
449 changed files with 91 additions and 120537 deletions
Download patch file Download diff file Expand all files Collapse all files

27
.gitignore vendored
View file

@ -60,3 +60,30 @@ data/*.sqlite
*.pkl
data/papers/
.envrc
# === SECURITY: never commit secrets ===
.secrets/
*.pem
*.key
*.crt
*.p12
.env
.env.*
!.env.example
# === DATA: don't commit binary data blobs ===
*.zip
*.tar.gz
*.tar
*.parquet
*.duckdb
*.sqlite
*.bin
*.safetensors
*.pt
*.pth
*.onnx
*.gguf
*.h5
*.hdf5

5
.secrets/ghost_session_cookies
View file

@ -1,5 +0,0 @@
# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
#HttpOnly_blog.rugmunch.io FALSE /ghost TRUE 1795687044 ghost-admin-api-session s%3ABGoSwf8Oax8WvUeqjtfseBlJFSToM-0-.qtb6flmTMlOFDfoxgQDPR%2FI7YmWYmVIcsxD4y6Kq3kw

0
5s}
View file

0
7s}
View file

0
=2.0.0
View file

13
AGENTS.md
View file

@ -1,3 +1,16 @@
## 🚨 CRITICAL RULES (enforced by gitleaks pre-commit hook)
1. **No nested repos.** Don't commit other complete project trees inside this one.
2. **No secrets.** Never commit `.secrets/`, `.env`, `*.pem`, `*.key`, `*.crt`, cookie jars.
3. **No data blobs.** Don't commit `*.zip`, `*.parquet`, model weights, sqlite files.
4. **No broken files.** Shell heredoc accidents must be caught before commit.
5. **No duplicate scaffolds.** If a `backend/` or `frontend/` subdir exists at root,
it's either THE app or bloat — investigate before adding more.
See [CLEANUP.md](CLEANUP.md) for cleanup history.
---
# AGENTS.md — rmi-backend
# Every AI agent working on backend code reads this first.
# Last updated: 2026-06-29

51
CLEANUP.md Normal file
View file

@ -0,0 +1,51 @@
# CLEANUP.md — rmi-backend
> Audit + cleanup log. Most recent first.
## 2026-07-02 — bloat + secrets removal
**Branch**: `chore/cleanup-remove-bloat-and-secrets`
### 🚨 SECURITY: removed `.secrets/`
- `.secrets/ghost_session_cookies` was committed to main. Content was a Netscape cookie jar.
- **CRITICAL ACTION REQUIRED: rotate any cookies/secrets that were in this file.**
- The file may have been browsable by anyone with repo read access.
- Added `.secrets/` to `.gitignore` (was missing).
### 🗑️ Bloat removed
- `rmi-frontend/` subtree (~2.7MB) — React app lives in its own repo `RugMunchMedia/rmi-frontend`
- `backend/` subtree (273 files, ~1.7MB) — duplicate scaffold, NOT the source of truth
### 🚮 Broken files removed
- `, r.stdout)...` — shell heredoc accident with literal newlines in filename
- `5s}`, `7s}`, `=2.0.0` — fragment files from another shell accident
### 🛡️ .gitignore hardened
Added patterns for:
- Secrets (`.secrets/`, `*.pem`, `*.key`, `*.crt`, `.env`)
- Data blobs (`*.zip`, `*.parquet`, `*.sqlite`, `*.duckdb`)
- Model weights (`*.bin`, `*.safetensors`, `*.pt`, `*.onnx`)
## Files deliberately NOT removed
These look "extra" but are legit:
- `worker.py`, `main.py`, `Dockerfile.worker` — runtime entry points
- `alembic.ini`, `alembic/` — DB migrations
- `ruff.toml`, `mypy.ini`, `pytest.ini` — lint/type/test config
- `safe_deploy.sh`, `databus_warm_cron.py` — ops scripts
- `justfile`, `uv.lock`, `requirements.txt` — modern Python tooling
- `rmi_sdk.py`, `rmi_langchain.py`, `x402_tool_builder.py` — domain modules
- `smithery.yaml`, `huggingface.yaml` — service configs
- `docker-compose.email.yml` — email service config
- `supabase/*.sql` — Supabase migrations
- `PROPRIETARY_REGISTRATION.txt` — legal doc
- `main.py.bak` — backup, consider deleting in future PR
- `backend.log` — log file, should be ignored (will be cleaned by better .gitignore)
## Pre-cleanup size
134MB → ~70MB
## Lessons learned
- Pre-commit hooks must include gitleaks (it WAS configured but this file slipped through)
- `.secretsallow` is an allowlist — anything not on it must be blocked
- Empty/short-lived subtrees should be deleted immediately, not left as legacy
- A `CLEANUP.md` in each repo prevents this pattern from recurring

86
backend/.github/workflows/ci.yml vendored
View file

@ -1,86 +0,0 @@
name: CI
on:
push:
branches: ["**"]
pull_request:
branches: [main]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: pip
- run: pip install ruff
- run: ruff check app/ --select E,F,W --ignore E501,E402,W503
- run: ruff format app/ --check --diff
test:
runs-on: ubuntu-latest
services:
redis:
image: redis:7-alpine
ports: ["6379:6379"]
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
env:
REDIS_URL: redis://localhost:6379/0
PYTHONPATH: /app
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: pip
- run: pip install -r requirements.txt pytest pytest-asyncio pytest-cov 2>/dev/null || pip install fastapi uvicorn redis httpx pydantic pytest pytest-asyncio pytest-cov
- run: |
python -m pytest tests/ -x --tb=short --cov=app --cov-report=term-missing 2>&1 || \
echo "::warning::Some tests failed — non-blocking for initial CI setup"
- uses: actions/upload-artifact@v4
if: always()
with:
name: test-results-${{ github.sha }}
path: |
.coverage
coverage.xml
retention-days: 7
deploy:
needs: [lint, test]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: appleboy/ssh-action@v1.0.3
with:
host: ${{ secrets.VPS_HOST }}
username: root
key: ${{ secrets.VPS_SSH_KEY }}
script: |
cd /root/backend
git pull origin main
docker compose -f /srv/rugmuncher-backend/docker-compose.yml up -d --build backend
sleep 10
for i in 1 2 3 4 5; do
if curl -fsS http://localhost:8000/health | python3 -c "import json,sys; exit(0 if json.load(sys.stdin).get('status')=='healthy' else 1)"; then
echo "DEPLOY OK — health check passed"
exit 0
fi
echo "Health check attempt $i failed, retrying..."
sleep 5
done
echo "HEALTH CHECK FAILED — rolling back"
cd /srv/rugmuncher-backend
docker compose up -d --build backend
exit 1

61
backend/.github/workflows/security.yml vendored
View file

@ -1,61 +0,0 @@
name: Security
on:
push:
branches: ["**"]
pull_request:
branches: [main]
schedule:
- cron: "0 6 * * *" # nightly scan of main
jobs:
trivy:
name: Container scan (Trivy)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- uses: docker/build-push-action@v6
with:
context: .
tags: rmi:scan
push: false
load: true
- name: Run Trivy
uses: aquasecurity/trivy-action@master
with:
image-ref: rmi:scan
format: sarif
output: trivy-results.sarif
severity: CRITICAL,HIGH
exit-code: 1
- uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: trivy-results.sarif
semgrep:
name: SAST (Semgrep)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: returntocorp/semgrep-action@v1
with:
config: >-
p/owasp-top-ten
p/python
p/sql-injection
p/xss
p/command-injection
p/secrets
gitleaks:
name: Secret scan (gitleaks)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

3
backend/.gitignore vendored
View file

@ -1,3 +0,0 @@
__pycache__/
*.pyc
.pytest_cache/

43
backend/.pre-commit-config.yaml
View file

@ -1,43 +0,0 @@
# .pre-commit-config.yaml — RMI v4.0
# Install: pip install pre-commit && pre-commit install
# Run manually: pre-commit run --all-files
repos:
# ── Generic checks ──
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.6.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
args: [--allow-multiple-documents]
- id: check-json
- id: check-toml
- id: check-added-large-files
args: [--maxkb=5000]
- id: check-merge-conflict
- id: detect-private-key
- id: mixed-line-ending
args: [--fix=lf]
# ── Python: ruff lint + format ──
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.6.9
hooks:
- id: ruff
args: [--fix, --exit-non-zero-on-fix]
- id: ruff-format
# ── Secrets scanning ──
- repo: https://github.com/gitleaks/gitleaks
rev: v8.19.0
hooks:
- id: gitleaks
# ── Conventional commits ──
- repo: https://github.com/compilerla/conventional-pre-commit
rev: v3.2.0
hooks:
- id: conventional-pre-commit
stages: [commit-msg]
args: [feat, fix, security, docs, style, refactor, perf, test, chore, ci, build, revert]

76
backend/AGENTS.md
View file

@ -1,76 +0,0 @@
# /root/backend/ — RMI CANONICAL BACKEND
## ⚠️ THIS IS THE ONE AND ONLY BACKEND
All other copies are dead. If you find code at `/srv/rugmuncher-backend/main.py`,
`/root/rmi/backend/`, or anywhere else — it's STALE. Work here ONLY.
## Architecture
```
/root/backend/
├── main.py # FastAPI app (5040 lines) — entry point
├── Dockerfile # Backend container build
├── Dockerfile.worker # Worker container build
├── requirements.txt # Python dependencies
├── .env.example # All required env vars documented
├── generate_env.py # Auto-generate .env from Hermes config
├── app/ # All application modules
│ ├── news_service.py # 15+ source news aggregator
│ ├── rag_service.py # Redis-based RAG vector store
│ ├── auth.py # Authentication
│ ├── payments.py # Payment processing
│ ├── content_syndicate.py # Multi-platform content publishing
│ └── ... (80+ modules)
```
## How to Use
### Backend changes (Python):
```bash
# Edit files here. Volume mount means changes are LIVE:
docker restart rmi-backend
# Or for dependency changes, rebuild:
cd /srv/rugmuncher-backend
docker compose build backend --no-cache
docker compose up -d backend
```
### Environment setup:
```bash
python3 generate_env.py --force
# Then edit .env to fill in missing values
```
### API documentation:
- Swagger: http://localhost:8000/docs
- Health: http://localhost:8000/health
## Docker Compose
Compose file: `/srv/rugmuncher-backend/docker-compose.yml`
Context: `context: /root/backend` (builds from here)
Mount: `/root/backend:/app` (live code, no rebuild needed)
All container names use hyphens: `rmi-backend`, `rmi-worker`, `rmi-n8n`, etc.
## Development Rules
1. **Never edit files outside this directory** for backend work
2. **Always `python3 generate_env.py`** after adding new env vars
3. **`.env` never committed** — use `.env.example` as template
4. **Test with `curl localhost:8000/health`** after changes
5. **Volume mount = live reload** — just restart the container
## Related Systems
| System | Location | Container |
|--------|----------|-----------|
| n8n workflows | /root/n8n-data/ | rmi-n8n |
| Orchestrator | /srv/rugmuncher-backend/orchestrator/ | rmi-orchestrator |
| Telegram bot | /srv/rugmuncher-backend/bots/telegram/ | rmi-telegram-bot |
| Frontend | /srv/rugmuncher-backend/rmi-frontend/ | Vercel/Cloudflare |
| Hermes AI | /root/.hermes/ | CLI process |
| Langfuse | /srv/langfuse/ | Separate compose |
| Redis | Composed | rmi-redis |

519
backend/DARKROOM_ADMIN_V2.md
View file

@ -1,519 +0,0 @@
# RMI Darkroom — Complete Backend Documentation
## RugMunch Intelligence Platform — Admin Backend v2
---
## Overview
The RMI Darkroom is a comprehensive, enterprise-grade admin backend for the RugMunch Intelligence crypto security platform. It provides full control over users, content, wallets, payments, security, analytics, and token deployment across multiple blockchains.
**Status:** Production-ready | **Version:** 2.0 | **Date:** May 31, 2026
---
## Architecture
```
┌─────────────────────────────────────────────────────────────┐
│ RMI Darkroom Backend │
├─────────────────────────────────────────────────────────────┤
│ Admin SPA (/admin) │ Darkroom UI (/darkroom) │
│ ├─ Dashboard │ ├─ Token Deployer │
│ ├─ User Management │ ├─ Airdrop Manager │
│ ├─ Security Center │ ├─ Multi-chain Launch │
│ ├─ Wallet Manager │ └─ Custom Snapshots │
│ ├─ Analytics │ │
│ ├─ Bulletin Board │ │
│ ├─ Financial │ │
│ └─ Configuration │ │
├─────────────────────────────────────────────────────────────┤
│ API Layer (757+ endpoints) │
│ ├─ /api/v1/admin/backend/* — Admin operations │
│ ├─ /api/v1/wallets/v2/* — Wallet management │
│ ├─ /api/v1/analytics/* — Real-time analytics │
│ ├─ /api/v1/admin/bulletin/* — Content management │
│ ├─ /api/v1/admin/tokens/* — Token deployment │
│ └─ /api/v1/bulletin/* — Public content │
├─────────────────────────────────────────────────────────────┤
│ Core Engines │
│ ├─ Admin Backend (RBAC, Audit, Sessions) │
│ ├─ Wallet Manager v2 (25+ chains, HD, Rotation) │
│ ├─ Security Defense (Bot Detection, WAF, DDoS) │
│ ├─ Analytics Engine (Real-time, Prometheus, Grafana) │
│ ├─ Bulletin Board (CMS, Moderation, SEO) │
│ ├─ Token Deployer (5 chains, Blacklist, Anti-bot) │
│ └─ Plugin System (Extensible architecture) │
├─────────────────────────────────────────────────────────────┤
│ Data Layer │
│ ├─ Redis — Sessions, rate limits, caching │
│ ├─ Supabase — Users, audit logs, wallet data │
│ ├─ File System — Vault, configs, backups │
│ └─ ClickHouse — Analytics time-series (optional) │
└─────────────────────────────────────────────────────────────┘
```
---
## Key Features
### 1. Role-Based Access Control (RBAC)
- **5 roles:** superadmin, admin, moderator, viewer, support
- **Permission matrix** with 30+ granular permissions
- **IP allowlists** for admin access
- **Session management** with 8-hour timeout, max 3 concurrent
- **2FA support** (TOTP-ready)
### 2. Wallet Manager v2
- **25+ chains:** Bitcoin, Ethereum, Solana, TRON, Base, BSC, Polygon, Arbitrum, Optimism, Avalanche, Fantom, Gnosis, Dogecoin, Litecoin, and more
- **HD Wallets:** BIP39/BIP44/BIP49/BIP84 mnemonic support
- **Key Rotation:** Scheduled automatic rotation with notifications
- **Payment Integration:** x402 micropayments, subscription tiers
- **Balance Monitoring:** Real-time tracking across all chains
- **AES-256-GCM encryption** with Argon2id key derivation
- **Multi-signature ready** architecture
### 3. Security Defense System
- **Bot Detection:** Behavioral analysis, fingerprinting, heuristics
- **Anomaly Detection:** Statistical analysis on request patterns
- **Honeypot Endpoints:** 10 trap endpoints that auto-ban attackers
- **DDoS Protection:** Circuit breaker pattern, rate limiting
- **IP Reputation:** Integration-ready for AbuseIPDB
- **Geo-blocking:** Country-based access control
- **Request Fingerprinting:** Canvas, WebGL, font analysis
### 4. Analytics Engine
- **Real-time Metrics:** CPU, memory, requests, errors, latency
- **4 Default Dashboards:** System Health, Financial, Security, Users
- **Trend Detection:** Automatic anomaly detection with 3-sigma analysis
- **Prometheus Export:** Compatible with Prometheus/Grafana stack
- **WebSocket-ready:** Real-time streaming data
- **Custom Dashboards:** Configurable widget layouts
### 5. Bulletin Board / CMS
- **Post Management:** CRUD with versioning, scheduling, expiry
- **Categories:** news, alert, update, promo, system, community, announcement, tutorial
- **Targeting:** Audience segmentation (free, premium, pro, admins)
- **Moderation:** Draft/review/published/archived workflow
- **SEO:** Meta tags, OpenGraph, slug generation
- **Comments:** Threaded discussions with moderation
### 6. Token Deployer (Darkroom)
- **5 Chains:** Ethereum, Base, BSC, Solana, TRON
- **Features:** Blacklist, anti-bot, anti-sniper, team allocation, vesting
- **Airdrop System:** 1:1 exact-match airdrop, multi-chain snapshots
- **Custom Snapshots:** JSON, CSV, manual upload
- **Anti-gaming:** Sybil detection, multi-account filtering
---
## API Endpoints Summary
| Category | Endpoints | Auth |
|----------|-----------|------|
| Admin Auth | 5 | Public/Session |
| Dashboard | 2 | dashboard.read |
| Users | 5 | users.read/write |
| Security | 8 | security.read/write |
| System | 5 | system.read/write |
| Content | 3 | content.read/write |
| Financial | 3 | financial.read |
| API Keys | 3 | api_keys.read/write |
| Admin Mgmt | 4 | superadmin only |
| Backups | 2 | superadmin only |
| Webhooks | 2 | webhooks.read/write |
| **Wallet Manager v2** | **19** | token_deploy.read/write |
| **Analytics** | **11** | analytics.read |
| **Bulletin Board** | **18** | content.read/write |
| **Token Deployer** | **24** | X-Admin-Key |
| **Public Bulletin** | **5** | None |
| **Total** | **757+** | Mixed |
---
## File Structure
```
/root/backend/
├── app/
│ ├── admin_backend.py # Core admin engine (RBAC, audit, sessions)
│ ├── wallet_manager_v2.py # Wallet management engine
│ ├── security_defense.py # Bot detection, WAF, DDoS protection
│ ├── analytics_engine.py # Real-time metrics and dashboards
│ ├── bulletin_board.py # CMS engine
│ ├── plugin_system.py # Plugin architecture
│ ├── token_deployer.py # Multi-chain token deployer
│ ├── multichain_airdrop.py # Airdrop engine
│ └── routers/
│ ├── admin_backend.py # Admin API (36 endpoints)
│ ├── wallet_manager_v2.py # Wallet API (19 endpoints)
│ ├── analytics.py # Analytics API (11 endpoints)
│ ├── bulletin_board.py # Bulletin API (18 endpoints)
│ ├── darkroom_tokens.py # Token deployer API
│ ├── darkroom_airdrop.py # Airdrop API
│ └── darkroom_multichain.py # Multi-chain API
├── static/
│ ├── admin.html # Admin SPA (52KB)
│ └── darkroom.html # Token deployer UI (43KB)
└── main.py # FastAPI app (757+ routes)
```
---
## Security Features
### Authentication
- bcrypt password hashing
- JWT session tokens with expiry
- Rate limiting on all endpoints
- IP blocking with auto-ban
- Failed login tracking (auto-ban after 5 attempts)
- Session invalidation on logout
- Concurrent session limits
### Authorization
- Role-based access control
- Permission matrix per endpoint
- Admin-only endpoints for sensitive operations
- Audit logging of all actions
- Before/after state tracking
### Data Protection
- AES-256-GCM encryption for wallet keys
- Argon2id key derivation
- File permissions (chmod 600) on vault files
- No private keys in memory longer than necessary
- Secure session storage in Redis
### Network Security
- Bot detection with behavioral analysis
- Honeypot endpoints (auto-ban on trigger)
- DDoS circuit breaker
- Request fingerprinting
- Anomaly detection on traffic patterns
- Geo-blocking capability
---
## Wallet Manager v2
### Supported Chains
| Chain | Family | Address Pattern | HD Path |
|-------|--------|----------------|---------|
| Bitcoin | Bitcoin | 1/3/bc1... | m/44'/0'/0'/0/0 |
| Bitcoin SegWit | Bitcoin | 3/bc1... | m/49'/0'/0'/0/0 |
| Bitcoin Native SegWit | Bitcoin | bc1... | m/84'/0'/0'/0/0 |
| Ethereum | EVM | 0x... | m/44'/60'/0'/0/0 |
| Base | EVM | 0x... | m/44'/60'/0'/0/0 |
| Polygon | EVM | 0x... | m/44'/60'/0'/0/0 |
| Arbitrum | EVM | 0x... | m/44'/60'/0'/0/0 |
| Optimism | EVM | 0x... | m/44'/60'/0'/0/0 |
| Avalanche | EVM | 0x... | m/44'/60'/0'/0/0 |
| BSC | EVM | 0x... | m/44'/60'/0'/0/0 |
| Fantom | EVM | 0x... | m/44'/60'/0'/0/0 |
| Gnosis | EVM | 0x... | m/44'/60'/0'/0/0 |
| Solana | Solana | Base58 | m/44'/501'/0'/0' |
| TRON | TRON | T... | m/44'/195'/0'/0/0 |
| Dogecoin | Secp256k1 | D... | m/44'/3'/0'/0/0 |
| Litecoin | Secp256k1 | L/M/ltc1... | m/44'/2'/0'/0/0 |
### Wallet Tiers
| Tier | Use Case | Security |
|------|----------|----------|
| hot | Active trading | Standard |
| warm | Regular operations | Enhanced |
| cold | Long-term storage | High |
| vault | Maximum security | Multi-sig ready |
### Payment Integration
- **x402:** Enable per-wallet with price in USD
- **Subscriptions:** Tier-based (free, basic, pro, enterprise)
- **Payment Types:** x402, subscription, one-time, marketplace, refund, withdrawal, deposit, fee, reward
---
## Analytics Dashboards
### System Health Dashboard
- CPU Usage (gauge + line chart)
- Memory Usage (gauge + line chart)
- Disk Usage (gauge)
- Requests/minute (counter)
- Response Latency (line chart)
- Error Rate (line chart)
### Financial Dashboard
- Total Revenue (counter)
- MRR (counter)
- ARPU (counter)
- Churn Rate (gauge)
- Revenue Trend (line chart)
- Payment Count (line chart)
### Security Dashboard
- Threats Blocked (counter)
- Bot Requests (counter)
- Attacks Detected (counter)
- Blocked IPs (counter)
- Threat Types (pie chart)
- Attack Timeline (line chart)
### User Analytics Dashboard
- DAU (counter)
- MAU (counter)
- New Users (counter)
- Retention Rate (gauge)
- User Growth (line chart)
- User Tiers (pie chart)
---
## Plugin System
### Plugin Types
- **connector** — Data sources (exchanges, APIs, oracles)
- **scanner** — Security scanners (contract, wallet, token)
- **analyzer** — Analysis engines (risk, sentiment, on-chain)
- **notifier** — Alert channels (email, telegram, webhook)
- **exporter** — Data export (CSV, PDF, API, webhook)
- **wallet** — Wallet integrations (hardware, custodial)
- **payment** — Payment processors (x402, stripe, crypto)
- **ml** — ML models (fraud detection, prediction)
- **security** — Security tools (WAF, firewall)
- **analytics** — Analytics integrations (Grafana, Prometheus)
### Built-in Plugins
- PrometheusExporter — Export metrics to Prometheus format
- WebhookNotifier — Send notifications to webhooks
- RedisCache — Redis caching and pub/sub connector
### Plugin Directory
```
/root/backend/plugins/
├── connector/
├── scanner/
├── analyzer/
├── notifier/
├── exporter/
├── wallet/
├── payment/
├── ml/
├── security/
└── analytics/
```
---
## Deployment
### Requirements
- Python 3.10+
- Redis 6.0+
- FastAPI + Uvicorn
- Optional: Supabase, ClickHouse, Prometheus, Grafana
### Environment Variables
```bash
# Core
JWT_SECRET=your-jwt-secret
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=your-redis-password
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_KEY=your-service-key
# Wallet Vault
WALLET_VAULT_PASSWORD=your-vault-password
# Admin
ADMIN_API_KEY=your-admin-key
# x402
X402_EVM_PAY_TO=your-wallet-address
# Security
ABUSEIPDB_API_KEY=your-abuseipdb-key # optional
```
### Startup
```bash
cd /root/backend
uvicorn main:app --host 0.0.0.0 --port 8000 --workers 4
```
### Health Check
```bash
curl http://localhost:8000/health
```
---
## Admin Access
### Default Admin
- **Email:** admin@rugmunch.io
- **Password:** Darkroom2025!
- **Role:** superadmin
### Admin UI
- **URL:** https://your-domain.com/admin
- **Login:** Email + Password + optional 2FA
- **Session:** 8-hour expiry, max 3 concurrent
### Darkroom (Token Deployer)
- **URL:** https://your-domain.com/darkroom
- **Auth:** X-Admin-Key header
---
## API Usage Examples
### Generate Wallet
```bash
curl -X POST https://api.rugmunch.io/api/v1/wallets/v2/generate \
-H "X-Admin-Session: sess_xxx" \
-H "Content-Type: application/json" \
-d '{"chain": "eth", "purpose": "payments", "tier": "hot"}'
```
### Record Payment
```bash
curl -X POST https://api.rugmunch.io/api/v1/wallets/v2/payments \
-H "X-Admin-Session: sess_xxx" \
-H "Content-Type: application/json" \
-d '{
"wallet_id": "wal_eth_123",
"wallet_address": "0x...",
"chain": "eth",
"payment_type": "x402",
"amount": 0.01,
"amount_usd": 25.00,
"user_id": "user_123"
}'
```
### Get Analytics
```bash
curl https://api.rugmunch.io/api/v1/analytics/dashboards/system \
-H "X-Admin-Session: sess_xxx"
```
### Prometheus Metrics
```bash
curl https://api.rugmunch.io/api/v1/analytics/prometheus
```
---
## Monitoring & Alerting
### Prometheus Metrics
All system metrics are exportable in Prometheus format at `/api/v1/analytics/prometheus`.
### Key Metrics
- `rmi_cpu_percent` — CPU usage
- `rmi_memory_percent` — Memory usage
- `rmi_requests_per_minute` — Request rate
- `rmi_response_time_ms` — Response latency
- `rmi_error_rate` — Error percentage
- `rmi_revenue_usd` — Total revenue
- `rmi_threats_blocked` — Threats blocked
- `rmi_active_users` — Active users
### Grafana Integration
Import the Prometheus endpoint into Grafana for visualization.
---
## Backup & Recovery
### Wallet Vault
- Encrypted JSON file at `/root/.rmi/wallets/vault_v2.json`
- Keystore at `/root/.rmi/wallets/keystore.enc`
- Payment log at `/root/.rmi/wallets/payments.jsonl`
### Backup Strategy
1. Daily encrypted backups to secure storage
2. Seed phrase recovery for HD wallets
3. Multi-signature backup for vault wallets
4. Audit log retention: 90 days
---
## Development
### Adding a New Plugin
```python
from app.plugin_system import Plugin, PluginType
class MyPlugin(Plugin):
@property
def name(self): return "my_plugin"
@property
def version(self): return "1.0.0"
@property
def plugin_type(self): return PluginType.ANALYZER
@property
def description(self): return "My custom analyzer"
def _setup(self):
# Initialize your plugin
pass
```
### Adding a Dashboard Widget
```python
from app.analytics_engine import DashboardWidget
widget = DashboardWidget(
widget_id="my_widget",
widget_type="line",
title="My Metric",
metric_name="my_metric",
width=6,
height=4,
)
engine.add_widget("system", widget)
```
---
## Security Checklist
- [ ] Change default admin password
- [ ] Set strong WALLET_VAULT_PASSWORD
- [ ] Enable Redis AUTH
- [ ] Configure IP allowlists for admin access
- [ ] Set up AbuseIPDB API key
- [ ] Enable 2FA for superadmin accounts
- [ ] Configure backup schedule
- [ ] Set up Prometheus/Grafana monitoring
- [ ] Enable HTTPS only
- [ ] Review audit logs weekly
- [ ] Rotate wallet keys quarterly
- [ ] Test disaster recovery plan
---
## Support
- **Email:** admin@rugmunch.io
- **Docs:** https://docs.rugmunch.io
- **API:** https://api.rugmunch.io/docs
- **Status:** https://status.rugmunch.io
---
## License
Proprietary and confidential. Unauthorized use, distribution, or reproduction is strictly prohibited.
Copyright (c) 2026 RugMunch Intelligence. All rights reserved.
---
**Built with:** FastAPI, Redis, Supabase, Python 3.12, love for crypto security.
**The Bloomberg Terminal of Shitcoins.**

625
backend/DESIGN.md
View file

@ -1,625 +0,0 @@
# RMI Backend — 2026 Architecture Design
## Why this exists
The current backend (`/root/backend/`) has grown organically:
- `main.py` is 10,305 lines
- 124 router files flat in `app/routers/`
- 14 RAG modules scattered at top of `app/`
- `token_scanner.py` is 4,109 lines
- `x402_tools.py` is 5,817 lines
- Cross-cutting concerns (redis, auth, errors) duplicated across modules
- Domain logic entangled with FastAPI
- No tests, no type safety, no clear boundaries
15 mechanical refactor tasks would patch symptoms. This design fixes the architecture.
## Target Layout
```
/root/backend/
├── pyproject.toml uv + ruff + mypy + pytest config (single source)
├── .pre-commit-config.yaml ruff + mypy + size cap + gitleaks
├── Dockerfile
├── alembic/ async migrations
│
├── app/
│ ├── main.py <100 lines: app factory + lifespan + middleware ONLY
│ ├── config.py pydantic-settings, env loading
│ │
│ ├── core/ cross-cutting, NO business logic
│ │ ├── logging.py structlog JSON + correlation ID
│ │ ├── errors.py AppError hierarchy + FastAPI handlers
│ │ ├── redis.py async client + get_redis() Depends()
│ │ ├── db.py async SQLAlchemy session
│ │ ├── auth.py JWT decode + role guards
│ │ ├── lifespan.py startup/shutdown
│ │ ├── middleware.py CORS, rate limit, correlation ID
│ │ ├── websocket.py WS connection manager
│ │ ├── tracing.py OpenTelemetry + Langfuse v4 init
│ │ ├── http.py async httpx client
│ │ ├── pagination.py cursor-based
│ │ ├── metrics.py Prometheus /metrics endpoint (P0 v3)
│ │ └── legacy.py @legacy decorator for deprecation tracking
│ │
│ ├── api/ HTTP transport, thin routes
│ │ ├── deps.py shared Depends (current_user, redis, etc)
│ │ ├── v1/
│ │ │ ├── public/ no auth — scanner, wallet, token, pricing, health
│ │ │ ├── auth/ JWT — portfolio, alerts, intel, profile
│ │ │ ├── admin/ admin — users, system, ops
│ │ │ ├── x402/ paid — tools, tokens, wallets, defi, security
│ │ │ └── mcp/ MCP — tools.py
│ │ └── ws/ WebSocket
│ │ └── alerts.py
│ │
│ ├── domain/ pure business logic, NO FastAPI imports
│ │ ├── scanner/ core + honeypot + rugcheck + holders + contract + deployer + models + service
│ │ ├── wallet/ analyzer + labels + behavior + models + service
│ │ ├── token/ discovery + supply + models + service
│ │ ├── rag/ embeddings + chunking + search + ingest + firehose + feedback + agentic + evaluation + tracing + router + permanence + models + service
│ │ ├── x402/ facilitator + tokens + enforcement + settlement + models + service
│ │ ├── intel/ feeds + narratives + graph + models + service
│ │ ├── scam/ classifier + patterns + models + service
│ │ ├── databus/ client + chains(96) + models + service
│ │ └── bulletin/ board + models + service
│ │
│ ├── infra/ external integrations
│ │ ├── ollama.py
│ │ ├── langfuse.py
│ │ ├── vector_store.py
│ │ ├── chains/ evm + solana + bitcoin + base + ...
│ │ ├── apis/ coingecko + etherscan + birdeye + goplus + arkham + dune + ...
│ │ └── providers/ ollama + openrouter + huggingface + ...
│ │
│ ├── agents/ v3 NEW — bounded-task AI agent loops (M1)
│ │ ├── loop.py frozen: TaskInput/LoopBudget/TaskOutput Pydantic models
│ │ ├── fact_store.py Redis-backed long-term memory
│ │ ├── kill_switches.py budget enforcers (iterations, tokens, wall, spend)
│ │ └── executor.py bounded loop runner
│ │
│ ├── mcp/ v3 NEW — MCP mesh v2 (M2)
│ │ ├── manifest.py frozen: MCPToolManifest Pydantic model
│ │ ├── registry.py FastAPI on :8643 — list/resolve tools
│ │ ├── auth.py gopass-backed auth scopes
│ │ └── servers/ opencti + dify + n8n FastMCP servers
│ │
│ ├── middleware/ v3 NEW — Starlette middleware classes (M7)
│ │ ├── cost_tracking.py frozen: per-tenant/per-route cost middleware
│ │ ├── rate_limit_v2.py per-tier sliding window
│ │ ├── auth_deps.py JWT/API key on legacy routes
│ │ ├── security_headers.py CSP, HSTS, X-Frame-Options
│ │ └── shadow_traffic.py v1/v2 response diffing
│ │
│ └── workers/ background jobs (separate from API)
│ ├── firehose.py
│ ├── scanner_queue.py
│ ├── ingest_cron.py
│ └── cleanup.py
│
├── docs/
│ ├── adr/ v3 NEW — Architecture Decision Records (M6)
│ ├── runbooks/ v3 NEW — incident response (M6)
│ ├── postmortems/ v3 NEW — incident learnings (M6)
│ ├── oncall/ v3 NEW — rotation doc (M6)
│ ├── load/ v3 NEW — k6 baseline reports (P2 #26)
│ ├── contract/ v3 NEW — Pact contract specs (P2 #27)
│ └── plans/
│ ├── UNFUCK-V1.md Frozen — superseded
│ ├── UNFUCK-V2.md Frozen — superseded
│ ├── UNFUCK-V3.md ACTIVE — AI-Forward Ship Edition (this document's parent)
│ └── INTEGRATION-LOG.md append-only worklog of v3 execution
│
└── tests/
├── conftest.py
├── unit/
│ ├── core/ 33+ core tests
│ └── domain/ 58+ domain tests
├── e2e/ v3 NEW — Playwright (P2 #25)
├── contract/ v3 NEW — Pact (P2 #25)
├── load/ v3 NEW — k6 scripts (P2 #25)
└── rag/
├── eval_suite.py v3 NEW — retrieval quality regression (M3)
└── prompts/
└── test_snapshots.py v3 NEW — prompt regression
```
## Key Design Principles
1. **STRICT LAYERING.** `api → domain → infra`. Never reverse. Domain knows nothing about HTTP.
2. **ONE SOURCE OF TRUTH for cross-cutting.** redis/auth/errors/logging live in `core/` exactly once. Routes import, never redefine.
3. **HARD SIZE CAP.** 500 lines per file. Enforced in pre-commit. No 4,109-line `token_scanner.py` ever again.
4. **THIN ROUTES.** Routes parse → call service → return. No business logic in HTTP layer.
5. **DOMAIN = PURE PYTHON.** `domain/scanner/` can be unit tested without spinning up FastAPI. This is the test that proves the architecture.
6. **WORKERS SEPARATED.** Background jobs don't pollute the API. firehose, scanner_queue, ingest_cron live in `workers/`.
7. **PYDANTIC V2 EVERYWHERE.** Every domain has `models.py`. No `dict` types crossing boundaries.
8. **ASYNC-ONLY.** No sync I/O in handlers. Same shape for the whole codebase.
9. **OBSERVABILITY BY DEFAULT.** structlog JSON + correlation ID + OTel + Langfuse in `core/tracing.py`. Every endpoint instrumented without opt-in.
10. **STRANGLER FIG MIGRATION.** New skeleton co-exists with old code. Old `main.py` keeps importing the old routers. New routes added alongside. Per-domain cutover, not big-bang.
## Migration Order
| Order | Domain | Why |
|-------|--------|-----|
| 0 | `rag_engine` shim | unblock prod crash, temp until `app/rag/` lands |
| 1 | `core/` | foundation everyone depends on |
| 2 | `infra/` | external integrations domain depends on |
| 3 | `alerts` | smallest, well-bounded, has WS + JWT + redis — proves full pattern |
| 4 | `wallet` | high-value, used by frontend |
| 5 | `token` | high-value |
| 6 | `scanner` | biggest (4,109 lines), do last when pattern is mature |
| 7 | `x402` | payment system, critical, mature pattern by then |
| 8 | `intel`, `scam`, `databus`, `bulletin` | long tail |
| 9 | `rag` consolidation (was 14 files) | last because it's the most coupled |
---
# THE 14-POINT MODERN BUILDER BAR (v3 — non-negotiable quality gate)
Every file that touches the unfuck MUST clear this. Pre-commit enforces what can be enforced.
```
# Standard Status Enforcement
1 Pydantic v2 strict (ConfigDict) PARTIAL pre-commit mypy
2 async-only handlers DONE pre-commit AST scan
3 <500 lines/file (pre-commit) DONE pre-commit file-size-cap
4 pre-commit hooks (ruff+mypy+gitleaks+
detect-secrets+bandit+file-size+pytest) PARTIAL install on dev + CI
5 typed errors (18/18) — currently 4/18 4/18 pre-commit AST scan
6 1 type per layer (api→domain→core→infra) PARTIAL architecture test
7 structlog everywhere DONE pre-commit no-print()
8 event bus (Redis pub/sub) PARTIAL manual review
9 per-domain /health/{domain} PARTIAL manual review
10 per-route cost tracking NONE M7 middleware (P2)
11 no from-main-import DONE pre-commit AST scan
12 single get_redis() DONE pre-commit no-new-get-redis
13 OTel trace coverage >85% 0% M4 wiring (P1)
14 Langfuse span coverage >95% LLM calls PARTIAL M4 wiring (P1)
```
**Items 5, 10, 13, 14 are the v3 additions.**
---
# THE 30 MUST-DO ITEMS (v3 — ACTIVE PLAN)
## P0 — Week 1 (32h total — 16h human + 16h M3)
| # | Item | Hours | Owner | Status | File |
|---|------|-------|-------|--------|------|
| 1 | main.py lifespan merge crash fixed | 1 | human | ✅ DONE | `main.py` (94 lines) |
| 2 | Audit 53 uncommitted modifications | 3 | human | ⏳ TODO | `git diff HEAD` |
| 3 | 65 broken crons → Tailscale IP | 4 | human | ⏳ TODO | crons 167.86.116.51 → 100.100.18.18 |
| 4 | /metrics endpoint | 2 | M3 | ⏳ TODO | `app/core/metrics.py` (NEW) |
| 5 | AlertManager → Telegram | 2 | M3 | ⏳ TODO | `docker-compose.yml` + `alertmanager.yml` |
| 6 | Embedder dual-dim wrapper (M3) | 8 | M3 | ⏳ TODO | `app/rag/embedder.py` (NEW) |
| 7 | WalletService proper async impl | 6 | M3 | ⏳ STUBS DONE, REAL PENDING | `app/domain/wallet/service.py` |
| 8 | Frontend SPA migrate Contabo→netcup | 2 | human | ⏳ TODO | `/var/www/rmi/` |
| 9 | GHOST_ADMIN_KEY from gopass | 1 | human | ⏳ TODO | `.env` |
| 10 | Load avg 14.19 → identify cause | 3 | human | ⏳ TODO | Erigon or Ollama? |
## P1 — Week 2 (37h total — 3h human + 34h M3)
| # | Item | Hours | Owner | Status | File |
|---|------|-------|-------|--------|------|
| 11 | GlitchTip self-host + wire SDK | 4 | M3 | ⏳ TODO | `docker-compose.glitchtip.yml` (NEW) |
| 12 | OTel auto-instrumentation (M4) | 6 | M3 | ⏳ TODO | `app/core/tracing.py` wire into lifespan |
| 13 | GitHub Actions CI | 8 | M3 | ⏳ TODO | `.github/workflows/ci.yml` (NEW) |
| 14 | Renovate + Dependabot | 2 | M3 | ⏳ TODO | `.github/dependabot.yml` (NEW) |
| 15 | Trivy in CI | 2 | M3 | ⏳ TODO | `.github/workflows/ci.yml` |
| 16 | Semgrep SAST in CI | 2 | M3 | ⏳ TODO | `.github/workflows/ci.yml` |
| 17 | Gitleaks + Trufflehog | 2 | M3 | ⏳ TODO | `.pre-commit-config.yaml` + CI |
| 18 | Vault rotation script (gopass) | 3 | human | ⏳ TODO | `scripts/rotate_secrets.sh` (NEW) |
| 19 | Rate limit middleware on all /api/v1/* | 4 | M3 | ⏳ TODO | `app/core/middleware.py` |
| 20 | Auth Depends() on legacy routes | 4 | M3 | ⏳ TODO | `app/core/middleware.py` |
## P2 — Week 3 (46h total — 12h human + 34h M3)
| # | Item | Hours | Owner | Status | File |
|---|------|-------|-------|--------|------|
| 21 | Security headers (CSP, HSTS, X-Frame) | 3 | M3 | ⏳ TODO | `app/core/middleware.py` |
| 22 | Backup restore drill | 3 | human | ⏳ TODO | R2 bucket test |
| 23 | 4 Grafana dashboards | 5 | M3 | ⏳ TODO | `grafana/dashboards/*.json` |
| 24 | Container memory limits (all 36) | 3 | M3 | ⏳ TODO | `docker-compose.yml` |
| 25 | tests/{e2e,contract,load}/ scaffolding | 4 | M3 | ⏳ TODO | `tests/e2e/` `tests/contract/` `tests/load/` |
| 26 | k6 baselines (top 50 routes) | 6 | M3 | ⏳ TODO | `tests/load/baseline.js` |
| 27 | Pact contracts + openapi-generator SDK | 5 | M3 | ⏳ TODO | `tests/contract/` |
| 28 | Top 10 ADRs | 5 | human | ⏳ TODO | `docs/adr/0001-0010.md` |
| 29 | Runbooks + postmortem + oncall | 4 | human | ⏳ TODO | `docs/runbooks/` `docs/postmortems/` `docs/oncall/` |
| 30 | Agent Loop Spec productionization (M1) | 8 | M3 | ⏳ TODO | `app/agents/loop.py` (NEW, frozen on creation) |
## P3 — Defer until revenue
Multi-region read replicas. SOC2 audit. Paid third-party pen-test. SBOM distribution. AsyncAPI WebSocket spec. Frontend RUM (Sentry browser SDK, web vitals). Pyroscope continuous profiling. OWASP ZAP.
---
# THE 8 AI-FORWARD MODULES (v3)
## M1: Agent Loop Productionization (P2 #30)
**Deliverable: `app/agents/loop.py` (frozen on creation)**
```python
from pydantic import BaseModel, Field, ConfigDict
class TaskInput(BaseModel):
model_config = ConfigDict(strict=True, frozen=True)
task_id: str = Field(..., pattern=r"^\d+-[a-z0-9-]+$")
description: str = Field(..., max_length=500)
context_budget_tokens: int = Field(default=8000, le=32000)
allowed_tools: list[str] = Field(..., min_length=1)
success_criteria: str = Field(..., max_length=300)
verify_command: str = Field(..., max_length=200)
class LoopBudget(BaseModel):
model_config = ConfigDict(strict=True, frozen=True)
max_iterations: int = Field(default=20, le=100)
max_tokens: int = Field(default=50_000, le=500_000)
max_wallclock_seconds: int = Field(default=900, le=3600)
max_spend_usd: float = Field(default=5.0, le=50.0)
class TaskOutput(BaseModel):
task_id: str
files_touched: list[str]
lines_added: int
lines_removed: int
verify_passed: bool
worklog_entry: str
spend_usd: float
iterations_used: int
```
**Kill switches** (every iteration):
- max_iterations ≤ 20 (max 100)
- max_tokens ≤ 50K (max 500K)
- max_wallclock ≤ 900s (max 3600s)
- max_spend ≤ $5 (max $50)
- verify_command_failed_3x → human review
**fact_store companion** (`app/agents/fact_store.py`):
- Redis key: `fact:{namespace}:{key}`
- TTL: 86400s
- Loaded at loop start, written at loop end
- Prevents re-discovery of known facts
**Acceptance gate: 7 consecutive days zero runaway.**
## M2: MCP Mesh v2 (P2 #24, D9)
**Deliverable: `app/mcp/manifest.py` (frozen on creation)**
```python
from pydantic import BaseModel, Field
from typing import Any
class MCPToolManifest(BaseModel):
name: str = Field(..., pattern=r"^[a-z_]+:[a-z_]+$") # e.g. "rmi-netcup:docker_ps"
version: str = Field(..., pattern=r"^\d+\.\d+\.\d+$")
server: str = Field(..., pattern=r"^[a-z-]+$")
description: str = Field(..., max_length=200)
input_schema: dict[str, Any]
output_schema: dict[str, Any]
auth_scope: str = Field(..., pattern=r"^[a-z_]+:[a-z_]+$") # gopass scope
deprecated: bool = False
successor: str | None = None
```
**Mesh registry on netcup `:8643`** with:
- GET /tools (list all with filters)
- GET /tools/{name} (full manifest)
- POST /resolve (name+MAJOR version → executor endpoint)
**3 unwired servers to wire in D9:**
- OpenCTI (40 tools) — wire with FastMCP
- Dify (18 tools) — wire with FastMCP
- n8n (20 tools) — wire with FastMCP
**Acceptance gate: 332+ tools versioned, all gopass-backed auth, OpenCTI+Dify+n8n wired.**
## M3: RAG Dual-Dim Wrapper (P0 #6)
**Deliverable: `app/rag/embedder.py` (wraps frozen `app/crypto_embeddings.py`)**
```python
from enum import Enum
class EmbedderBackend(Enum):
BGE_M3 = "bge-m3" # 1024d, legacy
QWEN3_4B = "qwen3-embedding:4b" # 2048d, target
class DualDimEmbedder:
def __init__(self, backend: EmbedderBackend):
self.backend = backend
self.dim = 1024 if backend == EmbedderBackend.BGE_M3 else 2048
async def embed(self, texts: list[str]) -> list[list[float]]:
return await self._ollama_embed(texts, self.backend.value)
async def reindex_collection(self, name: str, target: EmbedderBackend) -> bool:
old = DualDimEmbedder(EmbedderBackend.BGE_M3)
new = DualDimEmbedder(target)
# 1. Read all docs from collection
# 2. Re-embed with new backend
# 3. Write to new collection {name}_v2
# 4. Atomic swap: {name} → {name}_legacy, {name}_v2 → {name}
# 5. Verify: query 10 known docs, confirm retrieval
return True
```
**The math:**
| Metric | bge-m3 | qwen3-4b | Delta |
|--------|--------|----------|-------|
| Dimensions | 1024 | 2048 | +2x |
| RAM (loaded) | 1.2GB | 2.0GB | +800MB |
| Embed latency | ~80ms | ~140ms | +60ms |
| Retrieval (nDCG@10) | 0.71 | 0.83 (est) | +12% |
| Reindex time | — | 6h overnight | one-time |
| Storage/coll | ~50MB | ~100MB | +650MB total |
**Strategy: collection-by-collection overnight, smallest first, verify each morning.** If <8%, defer.
**Acceptance gate: All 13 collections reindexed, eval harness +8% nDCG@10, bge-m3 unloaded.**
## M4: Observability Triangle (P1 #11, #12, P2 #21)
**Three pillars wired into the 94-line lifespan:**
```python
# main.py additions (~15 lines to existing 94)
from contextlib import asynccontextmanager
from app.core.tracing import setup_otel
from app.core.langfuse import langfuse
from app.core.errors import init_glitchtip
@asynccontextmanager
async def lifespan(app: FastAPI):
_otel = setup_otel() # OTel: httpx, redis, asyncpg, neo4j
init_glitchtip() # Sentry-compatible SDK
await app.state.redis.initialize()
yield
_otel.shutdown()
langfuse.flush()
await app.state.redis.close()
app = FastAPI(lifespan=lifespan)
```
**Pillars:**
- **OTel**: auto-instrument httpx/redis/asyncpg/neo4j, export to local OTel collector
- **Langfuse**: `@observe()` decorator on every LLM call, >95% coverage
- **GlitchTip**: Sentry-compatible SDK, DSN from gopass, captures every unhandled exception
**Acceptance gate: 7 consecutive days triangle lit.**
## M5: CI/CD Standards (P1 #13-17, P1 #19, P1 #20)
**Pipeline stages:**
```
PRE-COMMIT (local): ruff + mypy + gitleaks + detect-secrets + bandit + file-size + pytest-smoke
CI (GH Actions): lint + test-full + build + trivy + semgrep + gitleaks-history + grype
DEPLOY (blue-green): build-green → start → health-gate → swap-router → verify-5xx → keep-blue-10min
ROLLBACK (auto): swap-back-to-blue on 5xx spike → page → freeze-deploys-1h
```
**Justfile deploy target:**
```makefile
deploy:
docker compose -f docker-compose.green.yml build
docker compose -f docker-compose.green.yml up -d
./scripts/health-check.sh http://127.0.0.1:8001/health 60
./scripts/swap-upstream.sh green
./scripts/watch-errors.sh 600 0.01
docker compose -f docker-compose.blue.yml stop
```
## M6: ADRs + Runbooks + Postmortems + Oncall (P2 #28, #29)
**Top 10 ADRs (write these first):**
1. Why FastAPI over Litestar/Flask/Django
2. Why 5 DBs (Postgres+Redis+CH+Neo4j+Qdrant)
3. Why strangler-fig over rewrite
4. Why bge-m3 → qwen3-embedding:4b (M3)
5. Why self-host GlitchTip over Sentry SaaS
6. Why single-VPS blue-green over k3s
7. Why gopass over Vault/Doppler/AWS SM
8. Why Hermes cron over systemd/Airflow
9. Why MCP mesh over direct API calls
10. Why Tailscale over WireGuard/ZeroTier
**ADR Template:**
```
# ADR-XXXX: [Title]
## Status: Accepted | Rejected | Superseded
## Date: YYYY-MM-DD
## Decider: @cryptorugmunch
## Context: What we're solving
## Decision: What we chose
## Alternatives Considered: What we rejected
## Consequences: Easier / Harder / Mitigations
```
## M7: Cost Tracking + SLOs + Error Budgets (P2 #21)
**Per-tenant cost middleware (`app/middleware/cost_tracking.py`):**
```python
class CostTrackingMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request, call_next):
tenant = request.headers.get("X-Tenant-ID", "anonymous")
t0 = time.monotonic()
response = await call_next(request)
elapsed_ms = (time.monotonic() - t0) * 1000
cost = self._estimate_cost(route, response, elapsed_ms)
await self._buffer_cost(tenant, route, cost, elapsed_ms)
return response
```
**SLO Targets:**
| Service | SLO | Error budget |
|---------|-----|--------------|
| /api/v1/* p99 latency | < 500ms | 43min/mo |
| /api/v2/* p99 latency | < 200ms | 43min/mo |
| /health uptime | ≥ 99.5% | 3.6h/mo |
| /api/* error rate (5xx) | < 0.1% | 43min of 5xx/mo |
| RAG retrieval success | > 95% | 43min of failures/mo |
| LLM calls timeout rate | < 1% | 7.2h of timeouts/mo |
**Auto-freeze deploys on 4x burn:**
```yaml
- alert: SLOBurnRate4x
expr: (sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h]))) > 0.02
for: 2m
annotations:
webhook: "http://127.0.0.1:8642/ci/freeze"
```
## M8: AI Tool Wiring Matrix
| Task | Best tool | Worst tool |
|------|-----------|------------|
| Bounded code edit (1 file, <50 lines) | **aider** | claude-code |
| Multi-file refactor (3–10 files) | **claude-code** | aider |
| Codebase search | **Bloop** (when installed) | grep |
| Test generation | **claude-code** | aider |
| Doc generation (ADR, runbook) | **GLM-5.2** | aider |
| Security review | **Semgrep** (not LLM) | any LLM |
| Dependency upgrade | **Renovate** (not LLM) | any LLM |
| Prompt engineering | **GLM-5.2 + DSPy** | claude-code |
| RAG eval | Custom harness + LLM-as-judge | any LLM alone |
| Long-context (≥100K) | **Kimi K2.7** (256K) | DeepSeek (slow) |
| Vision | **Gemini 2.5-pro** | text-only LLM |
| Fast small (≤500 tokens) | **Groq** | DeepSeek |
| Autonomous (bounded loop) | **Hermes** | claude-code |
---
# 10 INCIDENT SCENARIOS (v3)
## v2's 7 (preserved)
1. rmi-redis crashes at 3am → auto-restart
2. Disk full on netcup → AlertManager page at 80%
3. Postgres corruption → restore from nightly backup
4. SSL cert expiry → Cloudflare manages, Tailscale renew
5. OOM kill on rmi-backend → Docker limit + py-spy/memray
6. DDoS on rugmunch.io → Cloudflare "Under Attack" mode
7. DeepSeek balance hits $0 → failover to Ollama Cloud/Gemini
## v3's 3 NEW (AI-specific)
### Scenario 8 — Agent Loop Runaway
**Symptom:** LLM spend spikes ($50+/h). Hermes logs show task iterating >50×.
**Detect:** Daily spend alert (M7) at $20/h. Hermes /cron/health shows iterations >20.
**Mitigate:** `pkill -f "hermes.*task_id=X"`. Disable cron job.
**Recover:** Audit input contract — likely missing success_criteria or verify_command.
**Root cause:** vague description or missing verify_command.
### Scenario 9 — RAG Poisoning
**Symptom:** RAG retrieval returns adversarial content. User reports "the AI told me to send funds to X."
**Detect:** Langfuse span shows suspicious source. Eval harness flags quality regression.
**Mitigate:** Disable affected collection. Roll back FAISS index.
**Recover:** Add source allowlist to ingest step.
**Root cause:** open ingestion endpoint or untrusted RSS feed.
### Scenario 10 — MCP Tool Compromise
**Symptom:** Unusual gopass access pattern. MCP tool called from unexpected IP. Lateral movement.
**Detect:** gopass audit log. CrowdSec flags anomalous SSH. Tailscale ACL violation.
**Mitigate:** `gopass rm rmi/infra/compromised/scope` + rotate. Revoke Tailscale node. Block IP.
**Recover:** Audit all gopass accesses in last 30 days. Rotate ALL keys.
**Root cause:** over-broad auth_scope or leaked Tailscale key. SEV1.
---
# FROZEN FILE MANIFEST v3 (14 entries)
| # | File | Why frozen |
|---|------|------------|
| 1 | `main.py` | Front door (was 8475, now 94) |
| 2 | `_legacy_main.py` | Strangler-fig target |
| 3 | `app/core/redis.py` | Single Redis source |
| 4 | `app/core/config.py` | Env var loader |
| 5 | `app/databus/core.py` | 96-chain fetch interface |
| 6 | `app/rag/pipeline.py` | RAG pipeline (frozen for M3) |
| 7 | `app/crypto_embeddings.py` | bge-m3 embedder (replaced via wrapper) |
| 8 | `app/api/v1/__init__.py` | Router registry |
| 9 | `docker-compose.yml` | All 36 containers |
| 10 | `app/core/tracing.py` | OTel (not wired — M4) |
| 11 | `app/agents/loop.py` | M1 deliverable (frozen on creation) |
| 12 | `app/mcp/server.py` | M2 deliverable (frozen on creation) |
| 13 | `app/core/metrics.py` | P0 deliverable (frozen on creation) |
| 14 | `app/middleware/cost_tracking.py` | M7 deliverable (frozen on creation) |
**Rule:** Modified only via bounded-task delegation with Task ID in `/home/z/my-project/worklog.md`. Out-of-process edits reverted.
---
# THE MATH (v3 BRUTAL TRUTH)
```
Solo With M3
P0 + P1 + P2 hours 115h 115h
Available budget (21d×8h) 168h 168h human + 152h M3 = 320h
Buffer (30% incident tax) −50h −50h human, −46h M3
Effective capacity 118h 224h
NET (capacity − work) +3h +109h
↑ Phase 4 fits here
Plus 20h passive waits (eval soaks, mesh verifications)
→ Solo: 138h needed vs 118h available — slips to Day 25
→ M3: 138h needed vs 224h available — fits comfortably
```
**Verdict: Solo, v3 is a 25-day plan. With M3 parallel, v3 is a 21-day plan.**
---
# 90-DAY ROADMAP (after the unfuck)
## Month 2 — Agent Mesh + RAG Eval + SDK
- Agent mesh: Hermes orchestrating multiple claude-code/aider in parallel
- RAG eval harness: automated retrieval-quality regression on every push
- SDK v1: TypeScript + Python clients from OpenAPI spec
## Month 3 — Multi-Region + SOC2 + Paid Pilots
- Multi-region read replicas (Contabo → Cloudflare LB)
- SOC2 prep (ADRs + runbooks = 60% of evidence)
- 3 paid pilots, $500/mo each, 60-day commitment
## Month 4+ — Hire or Stay Solo
- If 3 pilots + 1 conversion → hire senior backend engineer
- If pilots but no conversion → focus on conversion
- If no pilots → pivot
---
# What Ships This Pass (Foundation)
1. ✅ Fix crash — `rag_engine` re-export shim, backend healthy (Phase 0)
2. ✅ `pyproject.toml` — uv + ruff + mypy strict + pytest
3. ✅ `.pre-commit-config.yaml` — ruff + mypy + size cap (500) + gitleaks
4. ✅ `app/core/` — 11 modules, each <200 lines
5. ✅ `app/api/v1/__init__.py` — router aggregator
6. ✅ `main.py` — 94 lines (Phase 0 just restored)
7. ✅ Verify: backend boots, all 1249 routes respond, health 200
8. ✅ Commit + deploy (`4a8a16d`)
## What Does NOT Ship This Pass (v3 P0-P2)
- Migrating alerts/wallet/token/scanner to new `domain/`. That's the current unfuck.
- The 15 mechanical refactors. Replaced with the layered architecture.
- Deleting old code. Strangler fig — old stays until domain is migrated.
- **v3 new**: M1 Agent Loop, M2 MCP Mesh, M3 RAG dual-dim, M4 Observability Triangle, M5 CI/CD, M6 ADRs, M7 Cost+SLOs, M8 AI Tool Matrix.
## Phase 2: Alerts Vertical Slice (proves the pattern) — COMPLETE
After foundation lands, migrate `alerts` end-to-end as the reference:
```
app/domain/alerts/
├── models.py # Alert, AlertRule, Notification — Pydantic v2
├── repository.py # async SQLAlchemy queries
├── service.py # business logic, pure Python
└── broadcaster.py # WebSocket broadcast helper
app/api/v1/auth/alerts.py # thin route: parse → call service → return
```
This proves the pattern works: domain is pure Python, route is <100 lines, can be unit tested without HTTP.

24
backend/Dockerfile
View file

@ -1,24 +0,0 @@
FROM python:3.11-slim
WORKDIR /app
# System deps
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc libpq-dev curl && \
rm -rf /var/lib/apt/lists/*
# Python deps
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy app
COPY . .
# Health check
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1
EXPOSE 8000
CMD ["python", "-u", "main.py"]

160
backend/EMAIL_SETUP.md
View file

@ -1,160 +0,0 @@
# Email Forwarding Setup Guide
# =============================
#
# This system forwards emails from your domains to a Gmail inbox,
# then polls that inbox and forwards emails to your Telegram admin bot.
#
# NO AUTO-REPLIES - emails appear in Telegram for admin review.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
SETUP STEPS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
1. CREATE GMAIL RECEIVER ACCOUNT
------------------------------
Create a new Gmail account (or use existing):
Example: rugmunch.admin@gmail.com
IMPORTANT: Enable 2FA and create an App Password:
- Go to: https://myaccount.google.com/security
- Enable 2-Step Verification (if not already)
- Go to: https://myaccount.google.com/apppasswords
- Create a new app password (select "Mail" and your device)
- COPY THE 16-CHARACTER PASSWORD (no spaces)
You'll need:
- Email: rugmunch.admin@gmail.com (or your choice)
- App Password: XXXX XXXX XXXX XXXX (16 chars)
2. CONFIGURE CLOUDFLARE EMAIL ROUTING
-----------------------------------
For rugmunch.io:
- Log in to Cloudflare dashboard
- Go to your rugmunch.io zone
- Click "Email" → "Email Routing"
- Click "Add Route"
- Create these addresses (all forward to same Gmail):
admin@rugmunch.io → rugmunch.admin@gmail.com
support@rugmunch.io → rugmunch.admin@gmail.com
contact@rugmunch.io → rugmunch.admin@gmail.com
For cryptorugmunch.com:
- Go to your cryptorugmunch.com zone
- Click "Email" → "Email Routing"
- Click "Add Route"
- Create these addresses:
admin@cryptorugmunch.com → rugmunch.admin@gmail.com
team@cryptorugmunch.com → rugmunch.admin@gmail.com
info@cryptorugmunch.com → rugmunch.admin@gmail.com
IMPORTANT: Make sure Email Routing is ENABLED (toggle ON)
3. CONFIGURE BACKEND ENVIRONMENT VARIABLES
----------------------------------------
Add these to your /root/.secrets/project_envs/rmi-backend.env:
# Email Forwarding
EMAIL_RECEIVER=rugmunch.admin@gmail.com
EMAIL_PASSWORD=xxxx xxxx xxxx xxxx (app password, no spaces)
TELEGRAM_ADMIN_CHAT_ID=123456789 (your admin Telegram chat ID)
EMAIL_POLL_INTERVAL=60 # check every 60 seconds
# Get your Telegram Chat ID:
- Message your bot: @userinfobot
- Send any message
- It will reply with your ID (e.g., 123456789)
4. START RESTART BACKEND
----------------------
Restart your backend service:
sudo systemctl restart rmi-backend
Or if running manually:
cd /srv/rmi/backend
source venv/bin/activate
python main.py
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
VERIFICATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Check logs for email polling startup:
sudo journalctl -u rmi-backend -f
You should see:
[RMI] 📧 Email polling enabled - starting IMAP service
Send a test email to:
admin@rugmunch.io
Wait 1-2 minutes. You should receive a Telegram message in your admin chat:
📧 New Email
From: sender@example.com
Domain: rugmunch.io
Subject: Test Email
Time: 2026-05-01 12:34:56
━━━━━━━━━━━━━━━━━━━━
This is the email body...
Inbox: rugmunch.admin@gmail.com
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
FAQ
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Q: Can users reply to these emails?
A: Emails can be replied to (standard email), but your system
won't auto-reply. Admins review in Telegram and respond manually
via their email client.
Q: What happens if Gmail IMAP fails?
A: The poller logs errors and retries on next interval. Emails
are marked as read, so they won't be re-sent to Telegram.
Q: Can I forward to a different email?
A: Yes, just change EMAIL_RECEIVER to any Gmail/IMAP-enabled
address. You'll need an app password for Gmail.
Q: How many emails can I receive?
A: Gmail free accounts: 15GB storage (roughly 10,000+ emails)
Cloudflare Email Routing: Unlimited forwards
Q: Do I need to configure MX records?
A: NO! Cloudflare Email Routing handles this automatically when
you enable Email Routing for your zone.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TROUBLESHOOTING
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
"IMAP login failed":
- Check EMAIL_PASSWORD is correct (use app password, not regular password)
- Ensure IMAP is enabled in Gmail settings
- Try logging into Gmail IMAP manually: telnet imap.gmail.com 993
"No emails appearing in Telegram":
- Verify Cloudflare Email Routing is ENABLED
- Check that emails are actually arriving in Gmail inbox
- Look at backend logs for polling errors
- Wait up to 2 minutes (poll interval)
"Duplicate emails in Telegram":
- Emails are marked as read after first poll
- Check Gmail isn't moving emails back to inbox
- Reset seen_message_ids by restarting backend
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

85
backend/PORT_MAP.md
View file

@ -1,85 +0,0 @@
# Backend Port Map — v3 (Jun 21 2026)
> **v3 reality**: The legacy multi-port dev instances (8002/8003/8005/8006/8010)
> are gone. There is ONE backend on **port 8000** running the new clean
> `main.py` (243 lines, no `_legacy_main` import). The legacy 8,475-line
> monolith is preserved on disk at `/root/backend/_legacy_main.py` but is
> no longer served.
## Active Services (host-level listen sockets)
| Port | Bind | Service | Status | Purpose |
|------|------|---------|--------|---------|
| 8000 | 127.0.0.1 | docker-proxy → rmi-backend | UP (healthy) | FastAPI backend (v3) |
| 9090 | 127.0.0.1 | docker-proxy → rmi-prometheus | UP | Prometheus scrape + alert eval |
| 9093 | 127.0.0.1 | docker-proxy (orphan) | STALE | Old alertmanager docker-proxy, no container behind it |
| 9094 | 0.0.0.0 | docker-proxy (langfuse?) | UP | Langfuse web (alt port) |
| 9095 | * | prometheus-alertmanager (host binary) | UP | AlertManager → Telegram routing |
| 2368 | 127.0.0.1 | docker-proxy → rmi-ghost | UP | Ghost CMS |
| 3000 | 127.0.0.1 | docker-proxy → rmi-grafana | UP | Grafana dashboards |
| 3002 | 0.0.0.0 | docker-proxy → langfuse-langfuse-web | UP | Langfuse LLM observability |
| 6379 | 127.0.0.1 | docker-proxy → rmi-redis | UP (auth) | Redis (RMI_PROD_REDIS_2026) |
| 7474 | 127.0.0.1 | docker-proxy → rmi-neo4j | UP | Neo4j HTTP |
| 7687 | 127.0.0.1 | docker-proxy → rmi-neo4j | UP | Neo4j Bolt |
| 8095 | 0.0.0.0 | docker-proxy → opencti | UP | OpenCTI |
| 8545 | 127.0.0.1 | docker-proxy → rmi-reth | UP | Reth Ethereum RPC (HTTP) |
| 8546 | 127.0.0.1 | docker-proxy → rmi-reth | UP | Reth Ethereum RPC (WS) |
| 8880 | 0.0.0.0 | nginx (system) | UP | Frontend SPA / public |
| 25/143/465/587/993/995 | * | docker-proxy → rmi-mail | UP | Postfix + Dovecot |
| 4190 | * | docker-proxy → rmi-mail | UP | Dovecot managesieve |
## Ports no longer in use
- **8002 / 8003 / 8005 / 8006 / 8010** — Legacy dev instances, gone in v3.
- **9100** — node_exporter (target down, not currently running)
## Quick health checks
```bash
# Backend (v3)
curl http://localhost:8000/health
# → {"status":"ok","service":"rmi-backend","version":"2026.06.21",
# "deploy_mode":"new-system (no _legacy_main)"}
# Prometheus
curl http://localhost:9090/-/healthy
# AlertManager
curl http://localhost:9095/-/healthy
# v1 API surface (auto-discover)
curl http://localhost:8000/openapi.json | jq '.paths | keys'
```
## Backend ↔ Prometheus Network
Both run on Docker bridge `rmi_network`:
- rmi-backend: 172.19.0.3:8000
- rmi-prometheus: 172.19.0.8:9090
- gateway (host): 172.19.0.1
iptables INPUT chain policy is DROP. Required ACCEPT rules:
- 8880 (nginx)
- 11434 (ollama)
- 9095 (added Jun 21, alertmanager <-> prometheus communication)
## Redis access
Redis runs in `rmi-redis` container with auth. The v1 backend reads:
- `REDIS_HOST=rmi-redis`
- `REDIS_PORT=6379`
- `REDIS_DB=0`
- `REDIS_PASSWORD=RMI_PROD_REDIS_2026`
The old `redis-server --port 6379` (no auth) recipe is gone — production
Redis is authenticated. Local development must use the same env vars.
## Modern port-discovery (replaces netstat)
```bash
# What replaces `netstat -tlnp | grep :8000`
ss -tlnp 'sport = :8000'
# Free-port picker
python3 -c "import socket; s=socket.socket(); s.bind(('', 0)); print(s.getsockname()[1]); s.close()"
```

186
backend/PRICING_ARCHITECTURE.md
View file

@ -1,186 +0,0 @@
"""
RMI Pricing & Subscription Architecture v2
=============================================
INTELLIGENT SCAN ECONOMY — One Scan, Many Uses
-------------------------------------------------
The key insight: When a user scans a token address, that's ONE data event.
But it can power DOZENS of downstream analyses without re-fetching.
Example: User scans token 0xDEAD...BEEF
→ 1 API call fetches: price, liquidity, holders, contract bytecode
→ Powers: risk scan, holder analysis, bubble map, contract audit,
funding trace, social sentiment, whale tracking, cross-chain check
→ All from one scan input, shared across the DataBus cache
This means we can offer SCAN PACKS where one scan credit
actually delivers comprehensive intelligence across ALL our tools,
because the DataBus deduplicates and caches the underlying data.
COMPETITIVE ANALYSIS (June 2026)
---------------------------------
| Platform | Free Tier | Pro Tier | Enterprise |
|-----------------|-----------------------|--------------------|--------------------|
| GoPlus Security | 150K CU/mo | $199/mo (6M CU) | $799/mo (37.5M CU)|
| Arkham | Limited views | $99/mo | $999/mo |
| Nansen | Basic dashboard | $150/mo (Vital) | $1,000/mo (Onchain)|
| DexScreener | Free basic | — | Custom |
| Bubblemaps | Free V2 | $29/mo pro | B2B custom |
| TokenSniffer | Free basic | $99/mo (SnifferPro)| Custom |
| Honeypot.is | Free basic | — | — |
| Chainalysis KYT | None | — | $50K+/yr |
| TRM Labs | None | — | $30K+/yr |
| De.Fi | Free basic | $19.99/mo | Custom |
| RugCheck | Free token checks | — | — |
KEY INSIGHT: We're the ONLY platform that gives ONE scan = ALL intelligence.
GoPlus charges per CU. Nansen charges per month for limited chains.
Arkham gives entity data but no risk scoring. TokenSniffer gives scores only.
RMI covers 38 chains, 67 data providers, real-time caching, AND risk scoring
in a single scan. That's worth a serious premium.
PRICING TIERS (v2 — REVISED June 2026)
---------------------------------------
FREE TIER (Anonymous / Fingerprint)
- 3 basic scans per day (urlcheck, pulse, token_age)
- 1 market overview per day
- Limited data per scan (summary only, no deep analysis)
- No wallet tracking, no real-time alerts
- Powered-by branding on all outputs
SCOUT PACK — $4.99 (25 scan credits)
- 25 scan credits, each = ONE address scanned
- Each scan unlocks EVERY tool for that address for 24 hours
- Includes: risk scan, holder analysis, bubble map, funding trace,
contract audit, whale tracking, social sentiment, cross-chain
- Smart money queries: 10 per pack
- Market overview: unlimited
- Credits never expire
- PER-SCAN VALUE: $0.20 per scan (competitive with TokenSniffer's $0.01-0.05
per basic scan, but we deliver 10-20x more data per scan)
HUNTER PACK — $14.99 (150 scan credits)
- 150 scan credits, same "one scan = full intelligence" model
- 70% discount vs Scout per scan ($0.10/scan)
- Includes everything in Scout plus:
- Arkham entity intelligence (5 queries)
- Deep SENTINEL forensic scans
- Nansen smart money labels (10 queries)
- Prediction market signals (unlimited)
- Real-time alerts (24h per activation)
- Portfolio dashboard (3 wallets)
WHALE PACK — $49.99 (750 scan credits)
- 750 scan credits ($0.067/scan — bulk rate)
- 85% discount vs Scout per scan
- Includes everything in Hunter plus:
- Unlimited Arkham entity lookups
- Unlimited SENTINEL forensic scans
- Unlimited smart money queries
- 30-day real-time alerts
- Portfolio tracking (25 wallets)
- Priority queue (cache bypass)
- x402 API access for automation
MONTHLY SUBSCRIPTIONS
─────────────────────
SCOUT MONTHLY — $19.99/mo
- 75 scan credits/month (rolls over 1 month)
- All Scout Pack features
- Weekly intelligence digest email
- Community Discord access
HUNTER MONTHLY — $49.99/mo
- 350 scan credits/month (rolls over 1 month)
- All Hunter Pack features
- Daily watchlist alerts
- Priority support
WHALE MONTHLY — $149.99/mo
- 1,500 scan credits/month (rolls over 1 month)
- All Whale Pack features
- Dedicated Telegram alert channel
- Custom webhooks
- API access with higher rate limits
- Account manager
ENTERPRISE — $499/mo (or custom)
- Unlimited scans, all tools, all data
- Full API access (databus.fetch with admin key)
- WebSocket real-time streams
- Custom data pipelines
- White-label options
- Dedicated support & SLA
COMMUNITY DISCOUNT — 50% OFF for CRM / $cryptorugmunch holders
- Verify: Check wallet balance > 0 of CRM (Solana) or
$cryptorugmunch (Base/Zora) at purchase time
- Applied automatically when wallet connected
- Works on ALL tiers (packs and subscriptions)
- CRM Solana: 6pnitzwjumnzsvfyfejf9mijzpc4iuqh1xugfwvdf8wb
- $cryptorugmunch Base: 0x93c4f6f6f8a14a255e78de0273d6490719d8538e17dfcc9b72907df6a0d72bf204
PRICE JUSTIFICATION
────────────────────
Why $4.99 for 25 scans when GoPlus gives 150K calls/mo free?
- GoPlus gives RAW API calls. Most are useless without interpretation.
- Our 1 scan = 15-20 underlying API calls, all aggregated and scored.
- Real value: risk assessment, not raw data. A rug pull warning saves $1K+.
- Users don't buy API calls; they buy protection.
Why $14.99 for 150 scans?
- Cheaper than Nansen ($150/mo) for a serious trader
- More comprehensive than Arkham ($99/mo) for security
- Deep analysis that TokenSniffer can't match
Why $49.99 for 750 scans?
- Active investigators use 20-30 scans/day
- Cheaper per-scan than any competitor at this volume
- Priority access means better data freshness
Why subscriptions?
- Recurring revenue for sustainability
- Lower monthly cost vs. buying packs repeatedly
- Roll-over credits reduce purchase anxiety
WHY NOT CHEAPER?
- $0.99 for 50 scans devalues the intelligence. Our free tier already
gives 3 scans/day. The paid product must feel like a significant step up.
- Crypto security is a serious business. Users spending $500-5K on a rug
pull want serious tools, not dollar-store pricing.
- The 50% community discount already gives holders $2.50/25 or $7.50/150
scans — aggressive discount without cheapening the brand.
IMPLEMENTATION
──────────────
Scan credit tracking: Redis key x402:scan_credits:{wallet}
Community discount: Check wallet balance of CRM/$cryptorugmunch tokens
Pack purchase: x402 payment (USDC on Base or SOL)
Credit deduction: On first API call per unique address per 24h window
Address reuse: Same address within 24h = no additional credit deduction
x402 Tool Pricing (per-call, no pack):
- urlcheck: Free (loss leader)
- pulse: Free (loss leader)
- risk_scan: $0.05
- holder_analysis: $0.08
- bubble_map: $0.10
- contract_audit: $0.15
- funding_trace: $0.08
- whale_watch: $0.12
- sentiment: $0.05
- cross_chain: $0.08
- arkham_entity: $0.20
- sentinel_deep: $0.25
Pack scanning: 1 credit = all above tools for 1 address for 24h
→ Per-credit value: $1.00+ of individual tool calls
→ Effective per-scan price: $0.067-$0.20 depending on pack size
"""

161
backend/RAG_MODERNIZATION.md
View file

@ -1,161 +0,0 @@
# RMI RAG Modernization — 2026 Standards
# ======================================
# Design document for upgrading RMI's RAG system to production-grade
# modern standards. Based on audit of all 40+ endpoints, 4 pipelines,
# 9 collections, and 3 embedders.
## Current State Audit
### Collections (crypto_embeddings.py)
wallet_profiles, token_analysis, scam_patterns, forensic_reports,
market_intel, contract_audits, known_scams, news_articles,
transaction_patterns
### Embedders (INCONSISTENT — 3 different models)
- nomic-embed-text (768d) — rag_engine.py, smart_ai_engine.py
- bge-m3 (1024d) — rag_ingestion.py, rag_supreme.py
- bge-small-en-v1.5 (384d) — crypto_embeddings.py (primary)
### Pipelines (4 separate, overlapping)
- rag_engine.py — Qdrant REST API, nomic-embed-text, 5 collections
- rag_service.py — FAISS ANN, bge-small, 9 collections, 3-pillar search
- rag_supreme.py — 15-win pipeline, bge-m3, 5 Qdrant collections
- rag_firehose.py — continuous ingestion engine (designed, not fully wired)
### Gaps Identified
1. NO historical scam ingestion (Rekt DB, Chainabuse, DeFi hacks)
2. NO structured chunking — raw text embedding, no overlap
3. NO evaluation running (RAGAS mentioned, not active)
4. Embedding model inconsistency across pipelines
5. Firehose sources not wired (cadences defined, fetchers missing)
6. NO query transformation in production path
7. NO feedback loop active
8. Redis SCARD bug (FIXED 2026-06-17)
9. FAISS disk indexes exist but Redis backing data evicted for 7/9 collections
## Modern Standards (2025-2026 Industry Consensus)
### 1. Chunking Strategy
- DEFAULT: Recursive character splitting, 512 tokens, 15% overlap
- For code: add class/function boundary separators
- For news: sentence-based chunking preserves coherence
- For scam reports: semantic chunking on topic boundaries
- Overlap: 10-20% (test for your domain — some studies show no benefit)
### 2. Embedding Models
- STANDARDIZE on bge-m3 (1024d) — best open-source, multilingual
- Fallback: bge-small-en-v1.5 (384d) for fast/local
- Multi-head: different dims for different content types
- Contract code: 128d structural features (already in crypto_embeddings.py)
- Scam patterns: 384d behavioral embedding
- News/articles: 1024d semantic (bge-m3)
- Wallet profiles: 64d behavioral fingerprint
### 3. Retrieval Architecture
- HYBRID: Dense (70%) + BM25/Sparse (30%) — 5-15% recall improvement
- RRF fusion (Reciprocal Rank Fusion) — proven best for hybrid
- Cross-encoder rerank: top-20 → rerank → top-5
- MMR dedup: remove near-duplicate results
- Query expansion: generate 3 variants, fuse results
### 4. Ingestion Pipeline (UNIFIED)
- SINGLE entry point: POST /api/v1/rag/ingest
- Pipeline: Parse → Chunk → Dedup → Classify → Embed → Store → Index
- Dedup: content hash in Redis (MD5 of normalized text)
- Quality filter: skip docs below quality threshold
- Rate limiting: per-collection docs/minute
- Batch embedding: groups of 25-50, async
### 5. Historical Data Sources (NEW)
- Rekt DB (de.fi/rekt-database) — 3,000+ DeFi hacks since 2020
- Chainabuse — scam reports with addresses
- TRM Labs Crypto Crime Report — annual typologies
- Elliptic State of Crypto Scams — annual report
- Chainalysis Crypto Crime Report — annual trends
- SlowMist Hacked Archive — detailed exploit analysis
- Immunefi Bug Bounty Reports — vulnerability patterns
- CertiK Audit Findings — smart contract vulnerabilities
- Solana Compromised Accounts — known drained wallets
- Etherscan Labels — 115K+ labeled addresses (already have)
### 6. Evaluation Framework
- RAGAS metrics: faithfulness, answer_relevancy, context_precision, context_recall
- Golden test set: 50 known scam queries with expected answers
- Run weekly, alert on regression
- Track: Hit@5, MRR, NDCG@10
### 7. Feedback Loop
- Scanner hits → boost source weight
- False positives → penalize
- User corrections → update embeddings
- Track helpful docs, boost in future searches
## Implementation Plan
### Phase 1: Standardize & Consolidate (NOW)
1. Standardize embedder: bge-m3 (1024d) primary, bge-small (384d) fallback
2. Add recursive chunking to ingest pipeline
3. Wire firehose sources (Rekt DB, Chainabuse, Etherscan labels)
4. Add content hash dedup to all ingestion paths
### Phase 2: Historical Data Ingestion (THIS WEEK)
5. Build Rekt DB scraper → forensic_reports collection
6. Build Chainabuse scraper → known_scams collection
7. Ingest TRM/Elliptic/Chainalysis annual reports → market_intel
8. Ingest SlowMist/Immunefi/CertiK findings → contract_audits
### Phase 3: Evaluation & Feedback (NEXT WEEK)
9. Activate RAGAS evaluation pipeline
10. Build golden test set (50 queries)
11. Wire feedback loop (scanner hits → boost)
12. Add query transformation (HyDE, expansion)
### Phase 4: Advanced Retrieval (ONGOING)
13. Cross-encoder reranking (bge-reranker-v2-m3)
14. Parent-child retrieval for long documents
15. Multi-modal: code + text + transaction patterns
16. Streaming response for agentic investigation
## New Unified Ingestion Pipeline
```
POST /api/v1/rag/ingest
{
"documents": [...],
"collection": "known_scams",
"source": "rekt_db",
"chunking": "recursive" // or "semantic", "sentence", "none"
}
Pipeline:
1. PARSE — extract text, metadata, entities
2. CHUNK — recursive split (512 tokens, 15% overlap)
3. DEDUP — MD5 hash check against Redis
4. QUALITY — score content, skip if < threshold
5. CLASSIFY — route to correct collection
6. EMBED — batch embed via bge-m3 (Ollama)
7. STORE — Redis (hot) + FAISS (index) + R2 (cold)
8. INDEX — update ANN index version
```
## New Collections to Add
| Collection | Source | Dims | Purpose |
|-----------|--------|------|---------|
| defi_hacks | Rekt DB, SlowMist | 1024d | Historical DeFi exploits |
| rug_timeline | Chainabuse, SENTINEL | 1024d | Rug pull chronology |
| vuln_patterns | Immunefi, CertiK | 1024d | Smart contract vulnerabilities |
| crime_reports | TRM, Elliptic, Chainalysis | 1024d | Annual crime typologies |
| compromised_wallets | Solana, Etherscan | 384d | Known drained addresses |
| exploit_techniques | All sources | 1024d | How hacks were executed |
## Success Metrics
- RAG total_docs: 2,473 → 50,000+ (20x)
- Collections with data: 2/9 → 9/9 + 6 new
- Embedding consistency: 3 models → 1 primary + 1 fallback
- Ingestion cadence: ad-hoc → continuous (firehose)
- Evaluation: none → weekly RAGAS
- Chunking: none → recursive 512-token
- Dedup: none → content hash
- Cold storage: partial → full R2 permanence

26
backend/RAG_R2_SETUP.md
View file

@ -1,26 +0,0 @@
# RAG R2 Storage — Setup Required
## One-time Cloudflare setup:
1. Create R2 bucket "rmi-rag-storage" in Cloudflare dashboard
2. Generate R2 API token with Object Read & Write permissions
3. Set environment variables:
- R2_ACCESS_KEY (the Access Key ID from R2 token)
- R2_SECRET_KEY (the Secret Access Key from R2 token)
## Architecture (already deployed):
Hot → Redis (in-memory, fast queries, always available)
Warm → Local /data/rag-storage (7-day cache, auto-cleaned)
Cold → Cloudflare R2 (permanent, 10GB free, zero egress)
## Endpoints (all working, all bypass write middleware):
POST /api/v1/rag/permanence/snapshot → Save all collections to R2
POST /api/v1/rag/permanence/restore → Pull latest from R2 into Redis
POST /api/v1/rag/permanence/nightly → Full cycle: snapshot→R2, clean local, rebuild ANN
GET /api/v1/rag/permanence/stats → R2 usage + local cache stats
## Cron (active):
cd0f23b963f2 — runs nightly at 3 AM UTC — full RAG persistence cycle

308
backend/README.md
View file

@ -1,308 +0,0 @@
# Rug Munch Intelligence — Platform Documentation
## The Bloomberg Terminal of Shitcoins
Rug Munch Intelligence (RMI) is a unified crypto intelligence platform providing **270+ tools** for token security, wallet forensics, whale tracking, market data, and blockchain queries. Every data call routes through our **DataBus pipeline** — 38 data chains, 67 providers, automatic failover, multi-layer caching.
**Access:** `https://mcp.rugmunch.io` | **Docs:** `https://rugmunch.io/docs` | **Operations:** `@rmialerts` (Telegram)
---
## v3 Architecture (Jun 21 2026)
The backend was rebuilt via **strangler-fig pattern** during Jun 21 crisis ops.
The legacy 8,475-line `_legacy_main.py` is preserved but **no longer imported**.
The new clean `main.py` (243 lines) mounts v1 routers exclusively.
**Frozen files** (do not edit without explicit un-freeze):
- `main.py`, `_legacy_main.py`, `app/core/redis.py`, `app/core/config.py`,
`app/databus/core.py`, `app/rag/pipeline.py`, `app/crypto_embeddings.py`,
`app/api/v1/__init__.py`, `docker-compose.yml`, `app/core/tracing.py`,
`app/agents/loop.py`, `app/mcp/server.py`, `app/core/metrics.py`,
`app/middleware/cost_tracking.py`
**AI-Forward Modules (M1-M8)** — 8 mandated modules shipping through Jun:
- M1 Risk Explainer · M2 News Classifier · M3 Sentiment Stream
- M4 On-chain Detective · M5 Threat Modeler · M6 Cost Optimizer
- M7 SLO + Error Budgets · M8 Self-Healing Loops
**Telegram channel surface** (rugmunchbot has admin on all):
- @cryptorugmuncher (main) · @rmicryptonews (news)
- @rmiupdates (changelog) · @rmialerts (operations — wired to AlertManager)
- @rmialpha (intel) · @rmiscans (scan feed)
**Observability stack:**
- Prometheus on 172.19.0.8:9090, scrapes rmi-backend at 172.19.0.3:8000
- 11 alert rules across `rmi-backend-critical`, `rmi-infra-warning`, `rmi-slo-burn`
- AlertManager on host:9095, routes to @rmialerts via rugmunchbot
- `/api/v1/admin/alerts/{webhook,critical,recent}` — in-app alert history
---
## Quick Start
```bash
# Discover the platform
curl https://mcp.rugmunch.io/.well-known/mcp
# List all tools
curl https://mcp.rugmunch.io/mcp/tools
# Check platform health
curl https://mcp.rugmunch.io/mcp/health
# Get SDK examples
curl https://mcp.rugmunch.io/mcp/sdk
```
**MCP Native (Claude Desktop, Cursor, Windsurf):**
```json
{
"mcpServers": {
"rug-munch": {
"url": "https://mcp.rugmunch.io/mcp",
"transport": "streamable-http"
}
}
}
```
---
## Platform Capabilities
### Tool Categories (234+ total, growing — check `/mcp/status` for live count)
| Category | Count | Description |
|----------|-------|-------------|
| Security Scanning | 45 | Rug pulls, honeypots, audits, clone detection, MEV protection |
| Wallet Intelligence | 38 | PnL, clustering, insider networks, whale tracking, forensics |
| Market Data | 32 | Prices, liquidity, volume, arbitrage, trends, OHLCV |
| Token Analytics | 28 | Holder distribution, sniper detection, deployer history |
| DeFi Analytics | 24 | TVL, yields, protocol risk, liquidity flow, bridges |
| Social Signals | 18 | Sentiment, KOL tracking, profile flips, meme scoring |
| Caching Shield | 13 | Internal multi-provider data access layer |
| Local MCP | 85 | Self-hosted Solana RPC (60) + EVM (25 tools, 86 networks) |
| Free Public MCP | 50 | Boar blockchain (ETH, ENS, contracts, keyless) |
### Supported Chains (96+ via DataBus, 13 native + 86 EVM via local MCP)
DataBus is the single source of truth: 96 chains, 119 providers, automatic failover.
See `app/databus/core.py` and `app/api/v1/public/databus/` for the live surface.
### Payment Facilitators (8 — plus x402 MCP marketplace gateway)
Coinbase CDP, EIP-7702 Universal EVM, Cloudflare x402, PayAI, Asterpay (SEPA),
TRON Self-Verify, Bitcoin Self-Verify, PrimeV. The x402 MCP marketplace gateway
(`x402-mcp-gateway` skill) handles tool-level micropayments in USDC on Base.
---
## Pricing & Access
### Free Trials
Every paid tool includes 1-5 free trial calls. No payment required until trials exhausted. Fingerprint-gated anti-abuse. Monthly reset.
### Individual Tools
$0.01 - $0.40 per call. Pay only for what you use. Full automatic refund within 48 hours if tool returns no data.
### Scan Packs (50-53% off)
| Pack | Tools | Price |
|------|-------|-------|
| Token Hunter Pack | Fresh pairs, snipers, security, deployer, clone detection | $0.09 |
| Whale Watcher Suite | Whale scan, accumulation, smart money, syndicates, PnL | $0.14 |
| Wallet Forensics | Funding trace, insider network, wash trading, wallet graph | $0.17 |
| Market Pulse | Prices, liquidity, arbitrage, sentiment, listings | $0.12 |
### Membership Tiers (60-90% discount)
| Tier | Price/mo | Daily Calls | Best For |
|------|----------|-------------|----------|
| Scout | $4.99 | 50 | Casual traders, hobby agents |
| Hunter | $14.99 | 200 | Active traders, alpha groups |
| Whale | $49.99 | 1,000 | Professional funds, market makers |
| Institution | $199.99 | 5,000 | Enterprises, high-frequency agents |
### Streaming Feeds
Real-time data via WebSocket + webhook delivery:
- New Token Firehose ($0.50/hr) — Every new token across all chains
- Whale Alert Stream ($0.75/hr) — Large transfers, positions, accumulation
- Multi-Chain Price Feed ($0.30/hr) — OHLCV, volume, arbitrage
- Security Alert Feed ($0.60/hr) — Rug pulls, exploits, suspicious activity
### Deep Research Reports
- Token Deep Dive ($0.75) — Full contract audit + deployer + holders + sentiment
- Wallet Intelligence Profile ($0.50) — PnL, style, associations, risk
- Chain Health Report ($0.25) — Gas, congestion, MEV, TVL, governance
- Cross-Chain Fund Trace ($1.50) — Follow money through bridges and mixers
### Batch Processing (75-90% off)
- Batch Token Scanner — 100 tokens, $0.05 per 10
- Batch Wallet Analysis — 50 wallets, $0.03 per 10
- Batch Pre-Buy Screen — 200 tokens, $0.02 per 50
### AI Data Feeds
- Market Context Feed ($9.99/mo) — LLM-optimized market summaries
- Alpha Signal Feed ($19.99/mo) — Scored trading signals
- Entity Relationship Graph ($14.99/mo) — Pre-computed wallet clusters
---
## Agent Skills (18 Workflows)
Every agent gets 18 guided workflows teaching best practices:
**Security & Vetting:** Pre-Buy Token Vetting, Scam Investigation, Post-Rug Forensics, Compliance Screening
**Trading & Execution:** Launch Day Playbook, MEV/Sandwich Avoidance, Market Making Intelligence, Portfolio Defense
**Alpha Discovery:** Alpha Discovery Pipeline, Whale Movement Tracking, CEX Listing Prediction, Insider Trading Detection
**DeFi & Yield:** Yield Farming Optimizer, Cross-Chain Bridge Monitor, DAO Governance Intelligence
**NFTs & Influencers:** NFT Mint Sniper, KOL Performance Tracker, Airdrop Hunting
Access at `GET /mcp/skills` — includes anti-abuse rules and 4 ready-to-use agent prompts.
---
## Technical Architecture
### Caching Shield
Every data call passes through three layers:
1. **L1 Memory Cache** — Sub-millisecond TTL lookup (8s-1hr depending on data type)
2. **Rate Limiter** — Token bucket per provider (prevents burning free tier quotas)
3. **Provider Chain** — Ordered fallback (3-4 providers per data type)
### Provider Fallback Chains
| Data Type | Primary | Fallback 1 | Fallback 2 | Fallback 3 |
|-----------|---------|------------|------------|------------|
| Token Price | Jupiter | Solana Tracker | DexScreener | Binance |
| Token Metadata | Helius DAS | Solana Tracker | Jupiter | DexScreener |
| Wallet Balance | Helius | QuickNode | Alchemy | PublicNode |
| Risk Scan | GoPlus | RugCheck | Honeypot | Local Labels |
| EVM Funding | Blockscout | Etherscan | Public RPC | Boar MCP |
### Infrastructure
- **Server:** Bare metal VPS, Docker Compose (30 containers)
- **Secrets:** GPG-encrypted vault (76 secrets), age-encrypted runtime injection
- **CI/CD:** GitHub Actions auto-deploy on push to main
- **Edge:** Cloudflare Workers for x402 payment gateway
- **Observability:** Langfuse cloud with smart sampling (20% normal, 100% errors)
### Local MCP Servers
Two self-hosted MCP servers run on our infrastructure:
- **Solana SVM MCP** (Rust, 32MB binary) — 60 RPC tools, WebSocket subscriptions, built-in x402
- **EVM MCP** (TypeScript, bun runtime) — 25 tools across 86 networks, ENS, contracts, gas
---
## API Reference
### MCP Protocol (Streamable HTTP)
```
POST /mcp JSON-RPC endpoint
GET /mcp/tools Tool catalog with input schemas
GET /mcp/call/{tool_id} Direct tool execution
GET /.well-known/mcp Server discovery
GET /.well-known/x402 Payment protocol discovery
```
### Platform Endpoints
```
GET /mcp/health Uptime + response time
GET /mcp/status Cache stats, provider health
GET /mcp/manifest Auto-updating platform manifest
GET /mcp/skills 18 agent workflow guides
GET /mcp/membership Plans, pricing, scan packs
GET /mcp/sdk Python/TS/curl quick-start
GET /mcp/changelog Version history
GET /mcp/trials Free trial status
GET /mcp/earnings Revenue dashboard
```
### REST API
```
POST /api/v1/investigate/trace Wallet funding source tracing
POST /api/v1/investigate/scan Full investigation
GET /api/v1/investigate/chains Supported chains
GET /api/v1/cache/health Caching shield status
```
### Dashboard Pages
```
/earnings Revenue dashboard (auto-refreshing)
/investigate Wallet investigation tool
```
---
## SDKs & Integration
**Python:** `pip install rmi-agent-sdk`
```python
from rmi_agent import RMIAgent
agent = RMIAgent() # auto-discovers via /.well-known/mcp
result = agent.call("rug_pull_predictor", {"token": "So111..."})
```
**TypeScript:** `npm install @rugmunch/agent-sdk`
```typescript
import { RMIAgent } from "@rugmunch/agent-sdk";
const agent = new RMIAgent();
const result = await agent.call("rug_pull_predictor", { token: "So111..." });
```
---
## Directory Listings
- **Smithery:** `https://smithery.ai/server/@cryptorugmuncher/rug-munch-intelligence`
- **Glama:** `https://glama.ai/mcp/servers/@cryptorugmuncher/rug-munch-intelligence`
- **mcp.so:** `https://mcp.so/server/rug-munch-intelligence`
- **GitHub:** `https://github.com/Rug-Munch-Media-LLC/rug-munch-intelligence-mcp`
- **HuggingFace:** `https://huggingface.co/cryptorugmunch/rug-munch-intelligence`
---
## Version History
### v3.4.0 (2026-06-21) — Crisis Ops Rebuild
- Backend rewritten via strangler-fig: clean main.py (243 lines) replaces 8,475-line legacy
- DataBus promoted to single source of truth (96 chains, 119 providers)
- 14 frozen files — explicit un-freeze required before edit
- AI-Forward Modules M1-M8 mandated across the system
- Prometheus + AlertManager wired to @rmialerts Telegram channel
- x402 MCP marketplace gateway live for tool-level micropayments
- 5 new v1 routers: admin/alerts_webhook, databus, x402/payments, rag/search, scanner
### v3.3.0 (2026-06-01)
- 18 agent skills with workflow guides and anti-abuse rules
- 4 membership tiers with daily call limits (60-90% discount)
- 4 scan packs at 50-53% off individual tools
- 4 real-time streaming feeds (WebSocket + webhook)
- 4 deep research report products
- 3 batch scanning products (75-90% off)
- 3 AI-optimized data feeds for LLM consumption
- 85 local MCP tools (Solana RPC + EVM 86 networks)
- 50 free Boar blockchain tools (ETH, ENS, contracts)
- Multi-provider caching shield on every data call
- Platform manifest — single source of truth, auto-syncing
- Earnings dashboard with wallet tracking
- Quality endpoints: health, status, SDK, changelog, trials
### v3.2.0 (2026-05-15)
- POST /mcp JSON-RPC handler
- inputSchema on every tool
- Dynamic facilitator count
- CORS headers for browser clients
### v3.1.0 (2026-04-01)
- /.well-known/mcp discovery
- llms.txt for AI agent discovery
- x402 payment protocol support
- 8 payment facilitators across 13 chains
---
**Contact:** mcp@rugmunch.io | **GitHub:** github.com/Rug-Munch-Media-LLC
**Operations:** @rmialerts (Telegram, Prometheus-wired)

281
backend/RMI_SYSTEM_MAP.md
View file

@ -1,281 +0,0 @@
# Rug Munch Intelligence (RMI) — System Map & Build Status
## LIVE SYSTEM OVERVIEW
```
┌─────────────────────────────────────────────────────────┐
│ FRONTEND (React) │
│ /root/frontend/ — 20 pages, dist/index.html │
│ RugMaps, RugCharts, Alerts, Markets, News, │
│ Intelligence, Investigation, ScamSchool, MCP Docs │
└────────────────────┬────────────────────────────────────┘
│ Supabase + REST API
┌────────────────────▼────────────────────────────────────┐
│ BACKEND (FastAPI) │
│ /root/backend/ — 379 endpoints, 6 routers │
│ Docker: rmi_backend (volume-mounted /app/app) │
│ 66 backend modules, 8 data connectors │
│ │
│ WALLET-CLUSTERING ROUTER (14 endpoints) │
│ POST /contract-scan → holders → clusters → bundles │
│ POST /cluster/detect → 7-method detection │
│ POST /cluster/analyze → behavioral fingerprinting │
│ GET /health → cache + GNN + spam stats │
│ │
│ FORENSICS ROUTER (12 endpoints) │
│ POST /threat-check → CryptoScamDB + GoPlus + Januus │
│ POST /deep-scan → full wallet forensics │
│ POST /cross-chain → multi-chain correlation │
│ │
│ RUGMAPS ROUTER (8 endpoints) │
│ GET /analyze/{address} → bubble map generation │
│ GET /health │
│ │
│ CROSS-TOKEN ROUTER (8 endpoints) │
│ GET /connections/{wallet} → cross-project links │
│ │
│ DISCOVERY ROUTER (8 endpoints) │
│ GET /tokens → new token discovery │
│ │
│ X402 TOOLS ROUTER (142 endpoints) │
└────────────────────┬────────────────────────────────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────────┐
│ Helius x3 │ │QuickNode │ │ DexScreener │
│ (primary) │ │(fallback)│ │ (free tier) │
└──────────┘ └──────────┘ └──────────────┘
▼ ▼ ▼
┌─────────────────────────────────────────────────────────┐
│ UNIFIED PROVIDER (7-source cascade) │
│ Helius → Birdeye → Solscan → GMGN → DexScreener → │
│ QuickNode → Blockchair Rate-limited 5 req/sec │
└─────────────────────────────────────────────────────────┘
```
## DATA SOURCES (18 API Keys)
| Source | Key File | Purpose | Status |
|---|---|---|---|
| Helius x3 | helius_api_key, _2, _3 | Solana RPC primary | LIVE |
| QuickNode | quicknode_api_key | Solana RPC fallback | LIVE |
| Birdeye | birdeye_api_key | Token data, whale tracking | LIVE |
| GMGN | gmgn_api_key | Token discovery | LIVE |
| Moralis | moralis_api_key | Multi-chain EVM data | CONFIGURED |
| Arkham | arkham_api_key | Entity labeling | CONFIGURED |
| CoinGecko | coingecko_api_key | Price data | CONFIGURED |
| Dune | dune_api_key | SQL queries on-chain | CONFIGURED |
| Nansen | nansen_api_key | Smart money tracking | CONFIGURED |
| Solscan | solscan_api_key | Solana transaction data | CONFIGURED |
| NVIDIA | nvidia_api_key, dev_api_key | AI inference | CONFIGURED |
| OpenRouter | openrouter_api_key | LLM routing | CONFIGURED |
| Groq | groq_api_key | Fast LLM inference | CONFIGURED |
| SiliconFlow | siliconflow_api_key, _2 | LLM inference | CONFIGURED |
| Kimi | kimi_api_key | LLM (Moonshot) | CONFIGURED |
| Mistral | mistral_api_key | LLM inference | CONFIGURED |
| Gemini | gemini_api_key | Google AI | CONFIGURED |
| HuggingFace | huggingface_token | Model downloads | CONFIGURED |
| Cloudflare | cloudflare_api_token | Workers, DNS | LIVE |
| Telegram | telegram_bot_token | Bot integration | LIVE |
## DETECTION PIPELINE
```
Token/Address Input
│
▼
┌──────────────────┐
│ ENTITY REGISTRY │ ← 50+ CEX/DeFi/Mixer addresses
│ filter_infra() │ ← 100K+ Solana labels from CSV
└────────┬─────────┘
▼
┌──────────────────┐
│ SPAM REGISTRY │ ← 2,530 Scam Sniffer addresses
│ check_token() │ ← GoldRush 8M spam tokens (6 chains)
└────────┬─────────┘ ← OpenSanctions OFAC, Guardian phishing
▼
┌──────────────────┐
│ HOLDER ANALYSIS │ ← Helius getProgramAccounts (228K JTO)
│ (unified_provider)│ ← Multi-source fallback cascade
└────────┬─────────┘
▼
┌──────────────────┐
│ BUNDLE DETECTION │ 5 signals:
│ (bundle_detector)│ ← atomic_block, common_funder, temporal,
└────────┬─────────┘ ← distribution_anomaly, concentration
▼
┌──────────────────┐
│ CLUSTER DETECTION│ 7 methods:
│ (wallet_clustering)│ ← temporal, counterparty, behavioral,
└────────┬─────────┘ ← funding, pattern, ML similarity, sleeper
▼
┌──────────────────┐
│ GNN FRAUD SCORE │ ← Random Forest fallback (CPU-only)
│ (fraud_gnn) │ ← HuggingFace sklearn (gated, not loaded)
└────────┬─────────┘
▼
┌──────────────────┐
│ THREAT INTEL │ ← CryptoScamDB (MIT, free)
│ (threat_feeds) │ ← GoPlus Security (free tier)
└────────┬─────────┘ ← Januus risk scores (open-source)
▼
┌──────────────────┐
│ CROSS-CHAIN │ ← Behavioral fingerprinting
│ (correlator) │ ← CEX deposit pattern matching
└────────┬─────────┘ ← Union-find entity grouping
▼
RISK SCORE OUTPUT
```
## LOCAL DATA FILES
| File | Lines | Purpose |
|---|---|---|
| wallet-labels/solana_cex_labels.csv | 100,001 | All known CEX hot wallets on Solana |
| wallet-labels/solana_defi_labels.csv | 1,481 | DeFi protocol addresses (Jupiter, Raydium, etc.) |
| wallet-labels/solana_dapp_labels.csv | 1,791 | Dapp addresses |
| wallet-labels/etherscan_malicious_labels.csv | 7,781 | Etherscan-flagged malicious contracts |
| wallet-labels/malicious_smart_contracts.csv | 754 | Additional malicious contracts |
| wallet-labels/ofac_sanctions.json | 0 | (Empty - needs seeding) |
| spam/scamsniffer_blacklist.json | 2,531 | Scam Sniffer address blacklist |
| SOSANA-CRM-2024.json | 101,916 | Full CRM data dump |
| wallet_database.json | 364 | Wallet profiles DB |
| rmi.db | 9 | SQLite state |
## SUPABASE TABLES (8 tables)
- profiles — user profiles
- wallet_labels — labeled wallet data
- token_analysis — token analysis results
- scam_reports — scam report submissions
- alerts — alert configurations
- market_intel — market intelligence cache
- news — news article cache
- forensic_reports — forensic analysis results
## INFRASTRUCTURE
| Service | Container | Status | Purpose |
|---|---|---|---|
| Backend API | rmi_backend | UP (healthy) | FastAPI, 379 endpoints |
| n8n Automation | rmi_n8n | UP | Workflow automation |
| Worker | rmi_worker | UP (healthy) | Background job processing |
| Telegram Bot | telegram-mcp | UP | Telegram bot integration |
| Listmonk | rmi-listmonk | UP | Email newsletters |
| Dragonfly (Redis) | rmi_dragonfly | UP (healthy) | Caching |
| Cloudflare Worker | rmi_cloudflare | UP | CF tunnel/edge |
| Ghost CMS | rmi-ghost | UP | Blog/content |
| MySQL | rmi-mysql | UP | Database |
| Langfuse | langfuse stack | UP | LLM observability |
| CF Edge Worker | rag.rugmunch.io | LIVE | RAG caching |
## WHAT'S WORKING (VERIFIED)
- [x] Helius RPC — 228K JTO holders detected via getProgramAccounts
- [x] Multi-source cascade — Helius → QuickNode → DexScreener fallback
- [x] Entity Registry — Binance/Uniswap/Tornado correctly identified
- [x] Bundle Detection — JTO = 0.09 confidence (correctly low)
- [x] Spam Registry — 2,530 Scam Sniffer addresses loaded
- [x] GNN Scoring — Random Forest fallback active (HuggingFace model gated)
- [x] Threat Feeds — GoPlus + CryptoScamDB + Januus integrated
- [x] All 5 health endpoints returning "ok"
- [x] All 10 core modules importing clean
- [x] RAG Edge Worker at rag.rugmunch.io returning health ok
- [x] n8n running with database
- [x] Telegram bot connected to Telegram servers
## WHAT'S BROKEN / NEEDS WORK
### Critical
- [ ] **RAG collections empty** — wallet_profiles, scam_patterns, forensic_reports all 0 docs. n8n needs to feed these. Only news_articles has 4 docs.
- [ ] **OFAC sanctions empty** — wallet-labels/ofac_sanctions.json is 0 lines. Needs seeding from opensanctions.org
- [ ] **563 uncommitted files** — backend has substantial uncommitted changes (Dockerfile, x402, entity_labeler, portfolio_tracker, etc.)
- [ ] **Frontend only has index.html** — 20 page source files exist but dist/ only has index.html. Other pages (docs, pricing, tools, x402) were deleted.
### High Priority
- [ ] **HuggingFace model gated** — fraud_gnn.py falls back to heuristic Random Forest because the sklearn model requires auth. Need to either get access or train a proper model.
- [ ] **n8n workflows not queryable** — 6 workflows claimed but API returned empty. Need to verify they're running.
- [ ] **CryptoGuard/GoPlus integration testing** — threat_feeds.py has the code but needs live testing with known scam addresses.
- [ ] **Entity labeler refactoring** — entity_labeler.py has 1084 lines of changes uncommitted, needs cleanup.
- [ ] **Exchange flow analyzer** — exchange_flow_analyzer.py reworked but uncommitted.
### Medium Priority
- [ ] **Frontend build pipeline** — Need to build and deploy the React frontend properly with all 20 pages.
- [ ] **Telegram bot features** — Bot is connected but needs command handlers for RMI features (scan, alert, etc.)
- [ ] **Email alerts** — Listmonk is running but not wired to RMI alert system.
- [ ] **CF Worker source** — rag.rugmunch.io is live but worker source code not in repo.
- [ ] **DexScreener connector** — Listed in unified_provider but not tested in cascade.
- [ ] **Blockchair connector** — Exists but not wired into unified_provider cascade.
- [ ] **EVM connector** — File exists but not tested against real EVM chains.
### Low Priority / Nice-to-Have
- [ ] **x402 payment system** — 142+ endpoints in x402_tools but uncommitted changes.
- [ ] **GNN model training** — Train a local sklearn model on known fraud data instead of HF gated model.
- [ ] **Cross-chain EVM testing** — cross_chain_correlator has Ethereum CEX addresses but Solana-only testing.
- [ ] **Mempool sentinel** — mempool_sentinel.py exists but unclear if active.
- [ ] **Wallet monitor** — wallet_monitor.py exists but not connected to alerts.
## BUILD PLAN — NEXT STEPS
### Phase 1: Stabilize & Commit (Day 1)
1. Commit all 563 uncommitted backend files
2. Seed RAG collections (wallet profiles from labels, known scam patterns)
3. Seed OFAC sanctions data from OpenSanctions
4. Test all threat feeds end-to-end with known scam addresses
### Phase 2: Frontend & Bot (Day 2-3)
5. Build frontend properly — `npm run build` in /root/frontend
6. Wire Telegram bot commands: /scan, /alert, /watch, /status
7. Deploy frontend to CF Pages or VPS
### Phase 3: Data Pipeline (Day 3-4)
8. Wire n8n workflows to feed RAG collections continuously
9. Set up scheduled GoldRush spam token sync (6 chains)
10. Set up OpenSanctions daily sync
11. Set up Scam Sniffer blacklist auto-update
### Phase 4: Testing & Hardening (Day 4-5)
12. End-to-end test with known scam tokens (not just JTO)
13. EVM chain testing with Ethereum addresses
14. Load testing on /contract-scan endpoint
15. Documentation: API docs, setup guide, architecture diagram
### Phase 5: Production (Day 5+)
16. Set up GitHub Actions CI/CD
17. Auto-deploy on merge to main
18. Monitoring (Langfuse, uptime checks)
19. Rate limiting on public endpoints
20. Authentication on sensitive endpoints
## KEY FILES QUICK REFERENCE
```
/root/backend/app/
├── chain_client.py # Rate-limited Solana RPC (Helius + QuickNode)
├── chain_cache.py # LRU cache with TTL (500 entries)
├── chain_feeder.py # Wallet TX feeding into clustering engine
├── unified_provider.py # 7-source data cascade
├── bundle_detector.py # 5-signal bundle detection
├── entity_registry.py # 50+ CEX/DeFi/Mixer address exclusion
├── threat_feeds.py # CryptoScamDB + GoPlus + Januus
├── fraud_gnn.py # Random Forest fraud scoring (CPU-only)
├── spam_registry.py # 2,530 scam addresses + GoldRush integration
├── cross_chain_correlator.py # Multi-chain entity resolution
├── wallet_clustering.py # 7-method clustering engine
├── cluster_detection.py # Cluster detection orchestrator
├── bubble_maps.py # RugMaps visualization engine
├── token_discovery.py # New token scanning
├── rag_service.py # RAG query service
├── routers/
│ ├── wallet_clustering_router.py # 14 endpoints
│ ├── forensics_router.py # 12 endpoints
│ ├── bubble_maps_router.py # 8 endpoints
│ ├── cross_token_router.py # 8 endpoints
│ ├── discovery_router.py # 8 endpoints
│ └── ... (admin, chat, x402, etc.)
├── data/
│ ├── spam/scamsniffer_blacklist.json # 2,531 lines
│ ├── wallet-labels/ # 100K+ Solana labels
│ └── SOSANA-CRM-2024.json # Full CRM dump
```

48
backend/SECURITY.md
View file

@ -1,48 +0,0 @@
# Security Policy
## Supported Versions
| Version | Supported |
| ------- | ------------------ |
| 2.x | ✅ Active support |
| 1.x | ❌ End of life |
## Reporting a Vulnerability
**DO NOT OPEN A PUBLIC ISSUE.** This is a commercial security product. Vulnerabilities in our code directly affect our customers' safety.
**Email:** security@rugmunch.io
**PGP Key:** [Available on request]
**Response time:** Within 24 hours
**Disclosure:** Coordinated disclosure after fix deployment (max 90 days)
### What to include:
- Type of vulnerability (RCE, auth bypass, data exposure, etc.)
- Affected endpoint/component
- Steps to reproduce
- Proof of concept (if available)
- Impact assessment
### What you'll receive:
- Confirmation within 24 hours
- Regular status updates
- Credit in release notes (unless you request anonymity)
- Bug bounty at our discretion (contact us for current program details)
## Security Best Practices for Contributors
1. **Never commit secrets** — API keys, tokens, passwords, private keys go in environment variables only
2. **Use `.env` (gitignored)** for local development credentials
3. **Sign your commits** with GPG (`git config commit.gpgsign true`)
4. **Review your own diffs** before pushing — check for accidental credential exposure
5. **Use branch protection** — all changes to main must go through PR review
6. **Run `git-sync.py --dry-run`** before pushing to verify no secrets are staged
## Our Security Stack
- Pre-commit hooks scan every staged file for secrets
- Pre-push hooks block force pushes and re-scan for secrets
- GitHub Actions CI runs secret scanning on every PR
- Dependabot monitors dependencies for known CVEs
- Production secrets stored in GitHub Secrets vault + environment variables
- Backend .env never committed (in .gitignore)

98
backend/SECURITY_STACK.md
View file

@ -1,98 +0,0 @@
# RugMunch Intelligence — Security Stack Summary
# Generated: 2026-05-08
# WARNING: This file describes the security tooling deployed on this host.
# Do NOT share externally — contains system architecture details.
# Access: chmod 600, root-only.
══════════════════════════════════════════════════════════════════
INSTALLED OPEN-SOURCE SECURITY TOOLS
══════════════════════════════════════════════════════════════════
SAST (Static Analysis):
bandit 1.9.4 Python security linter → pipx install bandit
semgrep 1.162.0 Cross-language static analysis → pipx install semgrep
Secret Detection:
gitleaks 8.25.1 Git + filesystem secret scanning → binary download
Dependency Scan:
pip-audit 2.10.0 PyPI vulnerability audit → pipx install pip-audit
Container Scanning:
trivy 0.70.0 Container + filesystem + secrets → binary download
IPS / WAF:
crowdsec 1.7.7 Collaborative intrusion detection → apt install
fail2ban active IP-based brute-force blocker → apt install
Pre-commit:
pre-commit 4.6.0 Git hook automation → pipx install pre-commit
══════════════════════════════════════════════════════════════════
NEW FILES CREATED
══════════════════════════════════════════════════════════════════
/srv/rmi/backend/.bandit.yaml Bandit config (excludes B105/B311 false-posit, dirs)
/srv/rmi/backend/.gitleaks.toml Gitleaks allowlist (public SOL addresses + static/)
/srv/rmi/backend/.trivyignore Trivy ignore (investigation evidence files)
/srv/rmi/backend/.pre-commit-config.yaml Pre-commit: bandit + gitleaks + isort + black + pip-audit
/srv/rmi/backend/run-security.sh Full security suite runner
/srv/rmi/backend/tmp/ fail2ban templates + GitHub Actions template
══════════════════════════════════════════════════════════════════
CODE FIXES APPLIED
══════════════════════════════════════════════════════════════════
DOCKERFILE:
- FROM python:3.12-slim (was 3.11)
- Added non-root `rmi` user + USER rmi
- Upgraded known-vulnerable packages: jaraco.context + wheel
CODE (md5 → sha256 - CWE-327):
app/fallback_engine.py:83 Cache key hash
app/routers/news_feed.py:237 Article deduplication hash
app/routers/rugmaps.py:462 Token cluster seed
app/routers/social.py:435 Like hash
app/rugmaps_analyzer.py:99 Analyzer seed
BUG FIXES:
app/routers/daily_briefing.py:280 Fixed unterminated string literal syntax error
══════════════════════════════════════════════════════════════════
SCAN RESULTS (latest run)
══════════════════════════════════════════════════════════════════
Bandit: 0 HIGH, 42 MEDIUM (excludes B105 false-positives)
Semgrep: 4 findings (2 INFO + 2 WARNING — all in x402-gateway/*.ts, not backend)
pip-audit: 0 known dependency vulnerabilities
Gitleaks: 0 leaks (after allowlist for public SOL addresses + dist/)
Trivy fs: 0 HIGH/CRITICAL (after .trivyignore for investigation evidence)
══════════════════════════════════════════════════════════════════
MANUAL DEPLOY STEPS
══════════════════════════════════════════════════════════════════
Deploy fail2ban API abuse protection:
sudo cp /srv/rmi/backend/tmp/rmi-api.conf /etc/fail2ban/filter.d/rmi-api.conf
sudo cp /srv/rmi/backend/tmp/rmi-api-jail.conf /etc/fail2ban/jail.d/rmi-api.conf
sudo systemctl restart fail2ban
Deploy GitHub Actions when repo hooks up:
mkdir -p .github/workflows
cp /srv/rmi/backend/tmp/github-workflow.yml .github/workflows/security.yml
══════════════════════════════════════════════════════════════════
COMMAND REFERENCE
══════════════════════════════════════════════════════════════════
Quick scan: cd /srv/rmi/backend && ./run-security.sh
Full scan: cd /srv/rmi/backend && ./run-security.sh --full
Bandit only: bandit -r app/ -c .bandit.yaml
Semgrep only: semgrep --config=auto
pip-audit: pip-audit -r requirements.txt
Gitleaks: gitleaks detect --source . --no-git --config .gitleaks.toml
Trivy fs: trivy fs --scanners vuln,secret,misconfig .
Trivy image: trivy image --severity HIGH,CRITICAL rmi-backend:latest
Pre-commit: pre-commit run --all-files
CrowdSec stats: cscli metrics + cscli decisions list
Fail2ban ban: sudo fail2ban-client status rmi-api

196
backend/STANDARDS.md
View file

@ -1,196 +0,0 @@
# RMI Development Standards — AI-Forward + Web3 Best Practices
> **v3 (Jun 21 2026) — Crisis Ops Rebuild**: These standards reflect the
> rebuilt architecture. The legacy 8,475-line `_legacy_main.py` is frozen;
> do not edit. New code goes into `app/api/v1/` via the aggregator in
> `app/api/v1/__init__.py`. See `docs/adr/0001-why-fastapi.md` and
> `docs/adr/0003-strangler-fig-not-rewrite.md` for the architectural decisions.
## v3 Hard Rules
1. **No mass regex.** Each transformation explicit, one-line-per-replace.
2. **New files only** — never modify the 14 frozen files without explicit un-freeze.
3. **Strangler-fig migration**: legacy routes stay mounted in `main.py` until
explicitly removed. New v1 routes mount at the same path — first match wins.
4. **One-line main.py edits only.** Even when adding routes, prefer the
`app/api/v1/__init__.py` aggregator + `v1_modules` list pattern.
5. **No `_legacy_main` import.** The new main.py does not import legacy.
6. **Real data flows only.** A v1 route that returns 404 / placeholder is not
done. Acceptance = a real route serves real data end-to-end.
7. **Prometheus metrics required.** Every new router emits `rmi_requests_total`
counter (auto via CostTrackingMiddleware).
8. **Telegram alerts via @rmialerts only.** No personal chat routing.
See `app/api/v1/admin/alerts_webhook.py`.
## v3 Modern Builder Bar (14 points)
See `~/.hermes/skills/rmi/modern-builder-compliance/SKILL.md` for the full bar.
Highlights:
- Async/await for all I/O · Pydantic v2 everywhere · Structured logging
- Type hints on all public functions · No bare except: · Frozen file respect
- Real data acceptance test · Documented in DESIGN.md or relevant ADR
- One concern per file · Tests before merge · No silent failures
- Idempotent endpoints · Pagination on list endpoints · Rate-limited public routes
---
## ⚠️ FIRST: Read /root/DEVELOPERS.md for canonical paths.
---
## BACKEND DEVELOPMENT
### Pre-commit checklist (run before every commit):
```bash
bash /root/backend/scripts/pre-commit.sh
```
Checks: Python syntax, hardcoded secrets, env var consistency, stale path references.
### Environment variables:
```bash
# Auto-generate from Hermes config:
python3 /root/backend/generate_env.py --force
# Then fill in missing values:
nano /root/backend/.env
```
### Live development (no rebuild needed):
```bash
# Volume mount means code changes are instant:
docker restart rmi-backend
# Verify:
curl http://localhost:8000/health
```
### Adding new env vars:
1. Add to code: `os.getenv("MY_VAR")`
2. Add to `/root/backend/.env.example` with comment
3. Run `python3 /root/backend/generate_env.py --force`
4. Add to `/srv/rugmuncher-backend/docker-compose.yml` if container needs it
---
## AI AGENT DEVELOPMENT WORKFLOW
This system is designed for AI-assisted development. Here's the stack:
```
hermes-agent (CLI)
│
├── Terminal tool → docker exec, git, curl, python
├── Web tool → API testing, research
├── File tool → Edit /root/backend/ directly
├── Delegate → Spawn sub-agents for parallel work
└── Cron jobs → Automated tasks
```
### How Hermes develops the backend:
1. **Discover**: Reads AGENTS.md in /root/backend/
2. **Edit**: Patches files directly (volume mount = live)
3. **Test**: `curl localhost:8000/health` after changes
4. **Rebuild**: `docker compose build && docker compose up -d`
5. **Verify**: Checks logs, API responses
### n8n workflow development:
- UI: http://localhost:5678 (admin / RugMuncher2024)
- Direct DB: `sqlite3 /root/n8n-data/database.sqlite`
- Import: Copy workflow JSONs into `/root/n8n-workflows/`
- Test: Check execution history in UI
### Orchestrator swarm:
- API: http://localhost:8081
- Health: `curl http://localhost:8081/health`
- Bots: `curl http://localhost:8081/orchestrator/bots`
- Create task: `POST /orchestrator/task`
---
## WEB3 SECURITY BEST PRACTICES
### Secrets management:
- **NO hardcoded secrets** in any `.py` file
- All secrets in `/root/.secrets/` or `/root/.hermes/.env`
- App passwords preferred over account passwords
- Rotate API keys quarterly
### Key scanning:
```bash
# Run before any commit:
grep -rn '0x[0-9a-fA-F]\{64\}\|sk-[a-zA-Z0-9]\{20,\}' /root/backend/app/ --include='*.py'
```
### RPC security:
- Use dedicated RPC URLs, never public endpoints in production
- Rate limit all on-chain queries
- Cache blockchain data aggressively (Redis)
---
## CODE QUALITY
### Python:
- Type hints on all public functions
- Docstrings for modules and classes
- Async/await for all I/O operations
- Use Pydantic for data models
### TypeScript (Frontend):
- Components in `/srv/rugmuncher-backend/rmi-frontend/src/components/`
- Services in `/srv/rugmuncher-backend/rmi-frontend/src/services/`
- Types shared via `/srv/rugmuncher-backend/rmi-frontend/src/types.ts`
---
## MONITORING
### Health checks:
```bash
# All services:
curl http://localhost:8000/health # Backend
curl http://localhost:8081/health # Orchestrator
curl http://localhost:5678/healthz # n8n
curl http://localhost:9001/api/health # Listmonk
```
### Logs:
```bash
docker logs rmi-backend --tail 50
docker logs rmi-n8n --tail 50
journalctl -u hermes -n 50
```
### Cron jobs:
```bash
# List all:
cronjob action='list'
# Check status of specific job:
cronjob action='list' # look for last_status
```
---
## DEPLOYMENT
### Full stack restart:
```bash
cd /srv/rugmuncher-backend
docker compose down
docker compose up -d
```
### Rebuild with cache clear:
```bash
docker compose build --no-cache backend worker orchestrator
docker compose up -d
```
### Rollback (if something breaks):
```bash
# Restore backup:
cp /root/backups/n8n/$(date +%Y-%m)/database.sqlite /root/n8n-data/
docker restart rmi-n8n
# Rebuild from known-good commit:
cd /root/backend && git checkout <commit-hash>
docker restart rmi-backend
```

90
backend/SUPABASE_ARCHITECTURE.md
View file

@ -1,90 +0,0 @@
# RMI Supabase Integration — AI-First, Web3-Forward
## Architecture
```
┌─────────────────────────────────────────────────────────┐
│ RMI Backend (FastAPI) │
│ │
│ supabase_router.py supabase_oauth_router.py │
│ supabase_auth_router.py supabase_service.py │
│ supabase_rag.py db_client.py │
└──────────────┬──────────────────────────────────────────┘
│ httpx + service_role key
▼
┌─────────────────────────────────────────────────────────┐
│ Supabase │
│ │
│ ┌───────────┐ ┌───────────┐ ┌───────────────────┐ │
│ │ Auth │ │ Postgres │ │ Row Level │ │
│ │ (JWT+OAuth)│ │ (Database)│ │ Security (RLS) │ │
│ └───────────┘ └───────────┘ └───────────────────┘ │
│ │
│ ┌───────────┐ ┌───────────┐ ┌───────────────────┐ │
│ │ Storage │ │ Edge │ │ Real-time │ │
│ │ (Files) │ │ Functions │ │ Subscriptions │ │
│ └───────────┘ └───────────┘ └───────────────────┘ │
└─────────────────────────────────────────────────────────┘
```
## Key Integration Points
### 1. Authentication (Web3 + Traditional)
- **JWT auth** via `supabase-auth` skill + `auth.py`
- **OAuth providers**: GitHub, Google (configured in `supabase_oauth_router.py`)
- **Wallet auth**: EVM + Solana wallet connection (non-custodial)
- **x402 trial tracking**: Device fingerprint + wallet-based quotas
### 2. Database (Postgres via Supabase)
- **User profiles**: `users` table with premium tiers, notification prefs
- **Intelligence data**: whale_movements, market_trending_tokens, scam_alerts
- **Content**: content posts, comments, upvotes, gamification events
- **x402 payments**: transaction logs, tool usage, trial tracking
- **Retention**: 90-day auto-cleanup for non-security data
### 3. RAG / Vector Store
- Redis-based vector store for crypto intelligence (`rag_service.py`)
- Lightweight SQLite+TF-IDF fallback (`rag_lightweight.py`)
- Collections: wallet_profiles, token_analysis, scam_patterns, forensic_reports, market_intel
- n8n workflow ingests news articles into RAG
### 4. Env Vars Required
```
SUPABASE_URL=https://<project>.supabase.co
SUPABASE_ANON_KEY=eyJh...
SUPABASE_SERVICE_KEY=eyJh...
SUPABASE_JWT_SECRET=...
```
## MCP (Model Context Protocol) Integration
- **`/api/v1/x402/tools-catalog`** — Full MCP catalog of 51 (44 MCP + 7 bundles) tools
- **`app/mcp/x402_mcp_server.py`** — MCP server implementation
- **`app/mcp_router.py`** — Routes MCP tool calls to backend functions
- **GitHub repo**: `Rug-Munch-Media-LLC/rug-munch-intelligence-mcp` (public)
## x402 Payment Protocol
- **`/.well-known/x402`** — Protocol discovery document
- **7 chains**: Solana (Facilitator), Base (Facilitator), ETH/BSC/ARB/OPT/POL (Self-verify)
- **Trial**: 1 free call (no wallet), 3 free calls (with wallet)
- **Payment**: USDC micropayments via HTTP 402
- **Repos**: `x402-gateway-solana`, `x402-gateway-base`, `x402-twitter-view`
## AI-Forward Architecture
```
User Request → Backend API → Orchestrator (9 agents)
│ │
├── Supabase ├── Wallet clustering
├── Redis RAG ├── Scam detection
├── News Agg ├── Threat intel
└── x402 Gate └── Cross-chain analysis
```
## Web3 Best Practices Applied
- **Non-custodial**: No private keys stored server-side
- **RLS**: Row Level Security on all Supabase tables
- **Device fingerprinting**: Anti-abuse for trial system
- **On-chain verification**: Self-verify mode for 5 chains
- **Facilitator mode**: Cloudflare Workers for Base/Solana

233
backend/X402_ARCHITECTURE.md
View file

@ -1,233 +0,0 @@
# x402 Protocol — Complete System Architecture
## MUST READ for all future RMI developers
### Auto-audited: May 23, 2026 — 59 tools, 7 chains, all endpoints verified
---
## SYSTEM OVERVIEW
```
INTERNET
│
▼
Cloudflare Tunnel (rmi-cloudflare)
┌─────────────────────────────┐
│ rugmunch.io │
│ mcp.rugmunch.io │
│ n8n.rugmunch.io │
└─────────────┬───────────────┘
│
┌─────────────▼───────────────┐
│ nginx (:80, :443) │
│ Routes: │
│ /api/* → :8000 │
│ /.well-known/* → :8000 │
│ /mcp/* → :8000 │
│ /health → :8000 │
│ / → static │
└─────────────┬───────────────┘
│
┌───────────────────┼───────────────────┐
│ │ │
▼ ▼ ▼
┌─────────┐ ┌──────────┐ ┌──────────┐
│ Backend │ │ Orchestrator│ │ n8n │
│ :8000 │ │ :8081 │ │ :5678 │
│ 59 tools│ │ 9 agents │ │ 2 flows │
└────┬────┘ └──────────┘ └──────────┘
│
┌────┼────────────────────┐
│ │ │
▼ ▼ ▼
┌──────┐ ┌──────┐ ┌──────────┐
│Redis │ │Supabase│ │ Langfuse │
│:6379 │ │ (API) │ │ :3100 │
└──────┘ └──────┘ └──────────┘
```
---
## FILE MAP — Every x402 file and what it does
### Core Backend (Python/FastAPI)
| File | Lines | Purpose |
|------|-------|---------|
| `app/routers/x402_enforcement.py` | 1290 | **Payment gatekeeper** — intercepts all `/api/v1/x402-tools/*`, verifies x402 payment headers, enforces trials, builds 402 Payment Required responses. 7-chain support. |
| `app/routers/x402_tools.py` | 3581 | **Tool handlers** — 48 route implementations. Each `@router.post("/audit")` is a tool. Also serves AI framework adapters (OpenAI, Anthropic, Gemini, LangChain formats). |
| `app/routers/x402_catalog.py` | 255 | **Auto-discovery** — parses gateway index.ts files + scans route decorators. Builds unified catalog. No hardcoded tool lists. |
| `app/routers/x402_forensic_tools.py` | 237 | **Forensic bundles** — 3 premium tools: forensic_valuation, osint_identity_hunt, investigation_report |
| `app/routers/x402_dashboard.py` | 493 | **Analytics** — usage tracking, revenue per tool, top users, trial exhaustion stats |
| `app/routers/x402_middleware.py` | 685 | **Anti-abuse** — device fingerprinting, trial tracking per device/wallet, rate limiting |
| `app/mcp/x402_mcp_server.py` | 682 | **MCP protocol server** — translates x402 tools into MCP format for Claude/Cursor/Windsurf |
### Cloudflare Workers (TypeScript)
| File | Lines | Purpose |
|------|-------|---------|
| `x402-gateway/base/index.ts` | 2650 | **Base + EVM gateway** — Payment verification via PayAI facilitator. 44 tool definitions. Routes to backend. |
| `x402-gateway/solana/index.ts` | 2650 | **Solana gateway** — Payment verification via PayAI facilitator. 35 tool definitions. Routes to backend. |
| `x402-twitter-view/src/index.ts` | ~200 | **Twitter data worker** — profiles, timelines, search. Self-healing with failover. |
### GitHub Repos (public)
| Repo | Purpose |
|------|---------|
| `rug-munch-intelligence-mcp` | Public pip package. Thin MCP wrapper around x402 API. |
| `x402-gateway-solana` | Solana gateway source — deploys to Cloudflare Workers |
| `x402-gateway-base` | Base + EVM gateway source — deploys to Cloudflare Workers |
| `x402-twitter-view` | Twitter data worker source |
---
## PAYMENT FLOW — Step by step
```
1. User/bot calls POST /api/v1/x402-tools/{tool}
│
2. x402_enforcement middleware intercepts
├── Check: Has user paid? (x-pay header with tx hash)
├── Check: Is trial available? (device fingerprint + wallet)
├── If unpaid AND no trials → build 402 Payment Required
│ └── Returns: payment addresses per chain, amounts, timeout
│
3. If paid or trial available → forward to tool handler
│
4. Tool handler (x402_tools.py) executes
├── Call backend connectors (Helius, Etherscan, DeFiLlama, etc.)
├── Aggregate multi-source data
└── Return JSON response
│
5. Payment verification (if paid):
├── Base/Solana → PayAI facilitator verifies USDC transfer
└── ETH/BSC/ARB/OPT/POL → Self-verify via Etherscan on-chain check
```
### Payment Addresses
- **All EVM chains**: `0x1E3AC01d0fdb976179790BDD02823196A92705C9`
- **Solana**: `Gix4P9AmwcZRGzr2hCEME5m2QAvY86dBfm8c7e7MpFzv`
- **Token**: USDC on all chains
- **Amounts**: $0.01 - $0.50 per tool (defined in gateway index.ts)
---
## TOOL DISCOVERY — How the catalog works
```
MCP Catalog (/api/v1/x402/tools-catalog)
│
├── Step 1: parse_gateway_tools()
│ └── Reads x402-gateway/{base,solana}/index.ts
│ └── Regex extracts each tool from RMI_TOOLS object
│ └── Finds: name, description, price, category, trialFree, method
│ └── Result: 44 tools (22 unique to base, 0 unique to solana)
│
├── Step 2: discover_route_tools()
│ └── Scans x402_tools.py + x402_forensic_tools.py
│ └── Finds @router.get/post decorators
│ └── Extracts docstrings as descriptions
│ └── Result: 5 unique tools not in gateways
│
└── Step 3: Merge + Deduplicate
└── Same tool ID = merge chains
└── Result: 59 total tools
```
### Output formats
| Format | Endpoint | For |
|--------|----------|-----|
| Full catalog | `/api/v1/x402/tools-catalog` | Humans, dashboards |
| x402 protocol | `/.well-known/x402` | AI agents, protocol discovery |
| OpenAI | `/api/v1/x402-tools/openai-tools` | ChatGPT, OpenAI-compatible |
| Anthropic | `/api/v1/x402-tools/anthropic-tools` | Claude, Cursor |
| Gemini | `/api/v1/x402-tools/gemini-tools` | Google Gemini |
| LangChain | `/api/v1/x402-tools/langchain-tools` | LangChain agents |
---
## PRICING & TRIALS
| Tier | Calls | Requirement |
|------|-------|-------------|
| Anonymous | 1 free per tool | Device fingerprint |
| Wallet connected | 3 free per tool | MetaMask/Phantom |
| Paid | Unlimited | USDC payment per call |
**Refund**: Full refund if tool returns no data. POST `/api/v1/x402/refund` with tx hash within 48h.
**Anti-abuse**: Device fingerprinting survives VPN/incognito. Identity hierarchy: wallet > device_id > turnstile > fingerprint.
---
## CHAIN SUPPORT MATRIX
| Chain | Network ID | USDC Address | Verification |
|-------|-----------|-------------|--------------|
| Base | eip155:8453 | 0x833589...a02913 | PayAI facilitator |
| Solana | solana:5eykt4... | EPjFWdd5...TDt1v | PayAI facilitator |
| Ethereum | eip155:1 | 0xA0b869...eB48 | Self-verify |
| BSC | eip155:56 | 0x8AC76a...d580d | Self-verify |
| Arbitrum | eip155:42161 | 0xaf88d0...5831 | Self-verify |
| Optimism | eip155:10 | 0x0b2C63...Ff85 | Self-verify |
| Polygon | eip155:137 | 0x3c499c...3359 | Self-verify |
---
## CONNECTOR APIS — What data we have
| Connector | API Key | Status | Used By |
|-----------|---------|--------|---------|
| Helius | ✅ Working | Solana RPC, webhooks, transactions | wallet, cluster, whale, forensics |
| Etherscan | ✅ Working | Contract source, ABI, TX history | contract_inspect, tx_decoder |
| DeFiLlama | 🆓 Free | TVL, protocols, yields | protocol_research, yield_scanner |
| Birdeye | ✅ Working | Trending, token data | trending_tokens |
| CoinGecko | ✅ Working | Prices, categories, trending | market_price, market_sectors |
| DexScreener | 🆓 Free | Pairs, liquidity, volume | dex_activity, market_price |
| Moralis | ❌ Key invalid | Multi-chain wallet/token data | NOT USED |
| GMGN | ❌ Access denied | KOL tracking, trending | NOT USED |
| Arkham | ⚠️ Untested | Entity labeling | NOT USED |
| Nansen | ⚠️ Untested | Smart money, token god mode | NOT USED |
| Dune | ⚠️ Untested | Custom queries | NOT USED |
---
## TESTING
```bash
# Test all 59 tools (expect 403 = x402 enforcement working):
bash /root/backend/scripts/test_all_tools.sh
# Status dashboard:
python3 /root/scripts/rmi-status
# Pre-commit check:
bash /root/backend/scripts/pre-commit.sh
```
---
## COMMON ISSUES & FIXES
| Issue | Symptom | Fix |
|-------|---------|-----|
| Gateway files missing | Catalog shows 0 tools | Clone gateways to `/root/backend/x402-gateway/` |
| Tool returns 403 | x402 enforcement active | Expected — tools require payment or trial |
| Catalog stale after adding tools | Old count | Restart backend: `docker restart rmi-backend` |
| Payment verification fails | 402 responses | Check USDC addresses, network config |
| Self-signed cert on external | curl fails without -k | Cloudflare provides edge cert — use -k or browser |
---
## WHEN ADDING NEW TOOLS
1. Add definition to `x402-gateway/base/index.ts` (and solana if applicable)
2. Add route handler to `x402_tools.py`:
```python
@router.post("/my_new_tool")
async def my_new_tool(req: SomeRequest):
"""Description of what this tool does."""
# Implementation using existing connectors
```
3. Restart backend: `docker restart rmi-backend`
4. Verify: `curl http://localhost:8000/api/v1/x402-tools/my_new_tool`
5. Check catalog auto-updated: `curl http://localhost:8000/api/v1/x402/tools-catalog`

219
backend/X_AUDIT_AND_STRATEGY.md
View file

@ -1,219 +0,0 @@
# CryptoRugMunch X (Twitter) Complete Audit & Strategy
## Generated: June 2, 2026
---
## ACCOUNT SNAPSHOT
| Metric | Value |
|--------|-------|
| Handle | @CryptoRugMunch |
| Display Name | Crypto Rug Muncher ✓ (verified) |
| Joined | March 23, 2024 |
| Followers | 66,699 |
| Following | 505 |
| Total Posts | ~14,395 |
| Bio | "Rug Munch Intelligence: Terminal for dev tracking, KOL rep cards, & deep token analysis" |
| Website | t.me/cryptorugmuncher |
---
## COMPLETE TWEKE INVENTORY (Discovered)
### TIER 1: High-Performing Investigative Threads (50-158 likes)
| Date | ID | Topic | Est. Likes |
|------|----|-------|-----------|
| 2026-01-13 | 2011121865268273169 | 🚨 RUG PULL WARNING: $USOR bundled scam | 158 |
| 2026-03-13 | 2032249064440431012 | DavinciJeremie expose — "turned followers into exit liquidity" | 50 |
| 2026-02-23 | 2025778728123666684 | The Paid Shill Pipeline: How KOLs Get Rich (Pump.fun lawsuit) | 41 |
| 2026-01-22 | 2014168260812685576 | More bundled garbage: $USR playing off $USOR success | ~30 |
| 2025-03-15 | 1900961816672694749 | WallStreetBets + Hayden Davis scam connection | 74 |
| 2025-02-14 | 1890538027115503700 | $LIBRA deployer multiple rug pulls (GMGN data) | ~60 |
| 2025-02-04 | 1886805232878858528 | Finixio/Clickout Media presale scams list Feb 2025 | ~35 |
| 2024-12-24 | 1871615662814056757 | Results of typical Finixio/Clickout Media presale scam | 35 |
| 2024-12-22 | 1870863989946609894 | Removed $ALICE post after feedback, community accountability | ~20 |
### TIER 2: Product/Brand Announcements (10-31 likes)
| Date | ID | Topic |
|------|----|-------|
| 2026-02-15 | 2023164661852418141 | Chrome/Firefox web extensions launching this week |
| 2026-02-18 | 2024167740219462025 | Coinbase AgentKit plugin for Rug Intel risk checks (x402) |
| 2025-09-26 | 1971604084756218356 | $CRM token supply, distribution, and controls for CoinGecko |
| 2026-01-29 | 2016675199387914592 | Bringing on additional developer, reworking website |
| 2026-03-01 | 2038909240178536655 | V2 $CRM token relaunch plan (3-step: forensics, product, then token) |
| 2026-02-08 | 2020370336940806508 | $BEAM shilled by serial scammer warning |
| 2025-04-05 | 1908461438894821690 | Wallet warning — remove immediately |
| 2025-02-14 | 1890529174802096440 | $LIBRA / JMilei — 3 wallets control 80% of supply |
### TIER 3: Scam Warnings & Call-Outs (19-35 likes)
| Date | ID | Topic |
|------|----|-------|
| 2024-06-12 | 1800953406137209006 | $DOGEVERSE scam — reports of staking theft |
| 2024-06-16 | 1802326083338945012 | Presale scam analysis: countdown clock, promotional tactics |
| 2024-06-17 | 1802697920606552549 | YouTube shill promotion, undoxxed team |
| 2024-06-18 | 1803060093740617960 | Paid advertorials in major crypto news outlets |
| 2024-06-19 | 1803439240203710756 | Same team behind $DOGEVERSE, $SMOG, $SLOTH, $SEALANA |
| 2024-06-25 | 1805662250101076378 | $SEAL team = $SLOTH + $DOGEVERSE + $DOGE20 + $SMOG |
| 2024-06-28 | 1806730519129805187 | $TIME presale scam — founder Erdem Nazli promoting to 167k |
| 2024-07-18 | 1813973110216925618 | Another presale scam format |
| 2024-09-03 | 1830974957700153375 | Cabal's 2024 End-of-Year Party, $NEIRO |
| 2024-11-08 | 1854947038968045621 | BlockDAG — inflated numbers, likely to become biggest presale scam |
| 2024-12-12 | 1867248464414867620 | Pepe Unchained $PEPU at $500M mcap — skeptics proven right? |
| 2024-12-18 | 1869175591804846368 | Clickout Media / Finixio — can't stop scamming |
| 2024-12-18 | 1869391339923570852 | Related Clickout expose |
| 2025-01-16 | 1811452834162110886 | Scam analysis continuation |
| 2025-11-01 | 1984657092507005033 | D Poppin collaboration/profile |
### TIER 4: Community / Personal
| Date | ID | Topic |
|------|----|-------|
| 2024-06-19 | 1803520713695105516 | OnlyFans behind-the-scenes access announcement |
| 2024-06-18 | 1803119961226756139 | "Who has been here?" — engagement post (2337 views) |
| 2026-03 | 2042987696889696307 | "I hear all of you... the silence has been frustrating... I had to learn to code and build" |
| 2025-02-04 | 1886835371331174834 | Top "traders" of bundled $ALPHA |
| 2026-01-16 | 2011984602559365328 | "Verify everything before you buy" |
### PRODUCT & TECH MILESTONES
| Date | ID | Topic |
|------|----|-------|
| 2026-02-15 | 2023164661852418141 | Web extensions (Chrome + Firefox) launching |
| 2026-02-18 | 2024167740219462025 | Coinbase AgentKit plugin with x402 risk checks |
| 2025-09-26 | 1971604084756218356 | $CRM token supply/distribution documentation |
| 2026-03-01 | 2038909240178536655 | V2 $CRM relaunch (3-step: forensics public → product live → token relaunch) |
| ~2025 | GitHub | Rug Munch MCP server (19 tools for crypto risk intelligence) |
| ~2025 | HuggingFace | x402-gateway-solana (Solana payment gateway) |
| ~2025 | Phantom | Listed on Phantom App Store |
| ~2025 | RNWY / Smithery | MCP Server directory listing |
| ~2025 | DexScreener | KOL scanner integration |
---
## CONTENT ANALYSIS BY CATEGORY
### Category Breakdown (% of discovered tweets)
| Category | % | Avg Likes | Quality |
|----------|---|-----------|---------|
| 🔍 Scam/Investigation Expose | 40% | 50-158 | HIGH — core value prop |
| 🚨 Rug Pull Warnings | 25% | 20-40 | MEDIUM — high volume, lower per-tweet impact |
| 🛠️ Product Announcements | 10% | 10-31 | LOW — poor product-to-engagement conversion |
| 🗣️ Community/Personal | 10% | 5-15 | LOW — but necessary for trust |
| 🔁 Thread Continuations | 15% | 5-20 | MEDIUM — follow-through is good |
### STRENGTHS
1. **Deep investigative work** — the KOL expose thread (41 likes), $LIBRA research, Finixio/Clickout series are genuinely valuable
2. **Consistent anti-scam voice** — never wavering from the core mission
3. **Real on-chain evidence** — citing GMGN, wallet data, transaction analysis
4. **Thread discipline** — most investigations are properly threaded with evidence
5. **Brand recognition** — 66K followers in crypto security niche is solid
### WEAKNESSES (Critical)
1. **POSTING FREQUENCY IS ERRATIC** — massive gaps (weeks/months of silence), then bursts. The "I hear you all" tweet from March 2026 acknowledges this directly.
2. **PRODUCT ANNOUNCEMENTS HAVE ZERO HYPE STRATEGY** — Chrome/Firefox extension launch got 31 likes. AgentKit got buried. These should be 500+ likes announcements. The gap between product capability and audience awareness is enormous.
3. **NO VISUAL BRANDING** — no consistent color scheme, no branded graphics, no template for warnings vs. investigations vs. announcements. Every top crypto security account uses branded templates.
4. **NO ENGAGEMENT FUNNEL** — 66K followers but average engagement is 20-80 likes. That's a 0.03-0.12% engagement rate. Crypto Twitter avg for this size is 0.5-2%. Something is deeply wrong.
5. **INCONSISTENT THREAD LENGTH** — some bangers are 1-tweet wonders, others are 15-part threads. No standard format.
6. **NO RECURRING CONTENT SERIES** — no "Scam of the Week", no daily digest, no regular format that builds habit.
7. **ONLYFANS STUNT** — the June 2024 OnlyFans post was engagement bait that confused the serious security brand. Never again.
8. **$CRM TOKEN MISHANDLING** — the token launch, then silence, then "V2 relaunch" 6 months later creates massive trust erosion. The 3-step plan is good but should have been communicated DURING the gap, not after.
9. **NO COLLABORATION STRATEGY** — zero threads tagging or quoting other security researchers (ZachXBT, Coffeezilla, etc.). Self-contained bubble.
10. **THREADBOLDS/FORMAT INCONSISTENCY** — mix of 🚨 emojis and plain text, no visual hierarchy standard.
---
## COMPETITIVE ANALYSIS vs TOP CRYPTO SECURITY ACCOUNTS
| Account | Followers | Avg Likes | Engagement Rate | Content Type |
|---------|-----------|-----------|----------------|-------------|
| @zabxXBT (ZachXBT) | 650K | 2,000-10,000 | 1.5-3% | Investigative threads |
| @Coffeezilla | 1.2M | 5,000-50,000 | 0.8-4% | Video + thread exposes |
| @lookonchain | 450K | 500-5,000 | 0.5-1.5% | On-chain data threads |
| @CryptoRugMunch | 66.7K | 20-158 | 0.03-0.24% | Scam warnings + investigations |
| @ape_scanner | 15K | 50-200 | 0.5-1.3% | Token security alerts |
**KEY INSIGHT**: RMI's engagement rate is 5-10x BELOW comparable accounts. The content quality is there but the distribution and format strategy is fundamentally broken.
---
## ACTIONABLE IMPROVEMENTS
### 1. POSTING CADENCE (Critical)
- **Minimum 2 posts/day**: 1 morning alert, 1 evening analysis
- **1 major thread/week**: Deep investigation (Tuesday 2pm ET)
- **Daily scam digest**: Top 3-5 scams to avoid that day (morning, 8am ET)
- **Fill the silence gaps**: If building, post "building in public" updates weekly
### 2. VISUAL BRANDING STACK
- Create 3 branded templates:
- 🚨 RUG ALERT (red/black, high urgency)
- 🔍 INVESTIGATION (blue/white, analytical)
- 🛡️ PRODUCT NEWS (green/dark, positive)
- Use consistent header bars with RMI logo
- All threads start with a branded image/graphic
### 3. ENGAGEMENT FUNNEL
- End every thread with a CTA: "Scan any token free at cryptorugmunch.com"
- Quote-tweet other researchers (ZachXBT, Lookonchain) with added context
- Reply to every major scam news within 60 minutes
- Use polls 1x/week for engagement bait ("How many of you lost money to [scam type]?")
### 4. RECURRING SERIES (Builds Habit)
- **"Scam School" weekly thread**: Educational deep-dive into one scam technique
- **"Monday Munchies"**: Top 5 projects to avoid this week
- **"Whale Watch Wednesday"**: Following smart money / whale wallet movements
- **"Verification Friday"**: Legit projects that passed RMI's full scan
- **Monthly State of Scams**: Comprehensive monthly report (great for bookmarks)
### 5. PRODUCT LAUNCH PLAYBOOK
- **7-day teaser campaign** before any launch
- **Launch day**: Thread storm (5+ tweets), video walkthrough, CTA
- **48-hour follow-up**: Share user results/stats
- **Week after**: "What we built vs. what you asked for" thread
### 6. CROSS-PLATFORM AMPLIFICATION
- Mirror every thread to Telegram channel (existing)
- Create YouTube Shorts from top threads (60-sec summaries)
- Reddit posts in r/CryptoCurrency for major investigations
- Cross-post to Mirror/Medium for long-form
### 7. HASHTAG STRATEGY
- Primary: #RugMunch #RugAlert #CryptoSecurity
- Secondary: #ScamAlert #DeFiSafety #OnChain
- Campaign: #ScanFirst (our equivalent of #DYOR but specific to RMI)
### 8. COMMUNITY BUILDING
- Weekly AMAs on X Spaces
- Create a "RMI Verified" badge for projects that pass full scan
- Community reports: let users submit scams, credit them in posts
- Reward top community members with premium access
---
## ENGAGEMENT TARGETS (30/60/90 Day)
| Metric | Current | 30 Day | 60 Day | 90 Day |
|--------|---------|--------|--------|--------|
| Posts/week | ~2-3 | 14 | 14 | 14 |
| Avg likes/tweet | 30 | 80 | 150 | 250 |
| Engagement rate | 0.08% | 0.5% | 1.0% | 1.5% |
| Major threads/month | 1-2 | 4 | 6 | 8 |
| Thread avg likes | 50 | 200 | 400 | 600 |
| Followers | 66.7K | 70K | 78K | 90K |
This requires: consistent posting, visual branding, engagement funnel, and collaboration strategy as outlined above.

4
backend/_check_mcp.py
View file

@ -1,4 +0,0 @@
from app.api.v1.mcp import router
print("MCP router loaded:", len(router.routes), "routes")
for r in router.routes:
print(" ", r.methods, r.path)

8478
backend/_legacy_main.py
View file

File diff suppressed because it is too large Load diff

1081
backend/app/admin_backend.py
View file

File diff suppressed because it is too large Load diff

155
backend/app/agents/fact_store.py
View file

@ -1,155 +0,0 @@
"""fact_store — Agent loop long-term memory (M1 companion).
Frozen on creation. Modified only via bounded-task delegation with Task ID.
Redis-backed KV of verified system facts. The agent loop reads at loop start
and writes at loop end, preventing re-discovery of known facts on every
iteration.
Schema:
Key: fact:{namespace}:{key}
Value: JSON-encoded arbitrary data
TTL: 24h default (override per call)
Examples:
await write_fact("agents", "postgres.host", "rmi-postgres", ttl=86400)
await write_fact("agents", "embedder.current", "bge-m3", ttl=86400)
facts = await load_facts("agents")
"""
from __future__ import annotations
import json
from typing import Any
# Lazy import — get_redis() may not be importable in unit tests.
try:
from app.core.redis import get_redis
except ImportError:
get_redis = None # type: ignore[assignment]
KEY_PREFIX = "fact:"
DEFAULT_TTL_SECONDS = 86400
async def load_facts(namespace: str) -> dict[str, Any]:
"""Load all facts in a namespace.
Returns an empty dict if Redis is unavailable (e.g. unit tests).
"""
if get_redis is None:
return {}
try:
r = await get_redis()
except Exception:
return {}
keys = await r.keys(f"{KEY_PREFIX}{namespace}:*")
if not keys:
return {}
vals = await r.mget(*keys)
out: dict[str, Any] = {}
for k, v in zip(keys, vals, strict=True):
if v is None:
continue
try:
# Strip the prefix to get just the key portion.
short_key = k.decode() if isinstance(k, bytes) else k
short_key = short_key.removeprefix(f"{KEY_PREFIX}{namespace}:")
out[short_key] = json.loads(v)
except (json.JSONDecodeError, ValueError):
continue
return out
async def write_fact(
namespace: str,
key: str,
value: Any,
ttl: int = DEFAULT_TTL_SECONDS,
) -> bool:
"""Write one fact to the store.
Returns False if Redis is unavailable (graceful degradation).
"""
if get_redis is None:
return False
try:
r = await get_redis()
await r.setex(
f"{KEY_PREFIX}{namespace}:{key}",
ttl,
json.dumps(value, default=str),
)
return True
except Exception:
return False
async def delete_fact(namespace: str, key: str) -> bool:
"""Delete one fact. Returns False if not found or Redis unavailable."""
if get_redis is None:
return False
try:
r = await get_redis()
deleted = await r.delete(f"{KEY_PREFIX}{namespace}:{key}")
return bool(deleted)
except Exception:
return False
async def list_namespaces() -> list[str]:
"""Return all distinct namespaces currently in the store."""
if get_redis is None:
return []
try:
r = await get_redis()
keys = await r.keys(f"{KEY_PREFIX}*")
namespaces: set[str] = set()
for k in keys:
s = k.decode() if isinstance(k, bytes) else k
parts = s.removeprefix(KEY_PREFIX).split(":", 1)
if parts:
namespaces.add(parts[0])
return sorted(namespaces)
except Exception:
return []
# ── Seed function for first-time setup ─────────────────────────────────
SEED_FACTS: list[tuple[str, str, Any, int]] = [
# (namespace, key, value, ttl)
("agents", "postgres.host", "rmi-postgres", 86400),
("agents", "postgres.port", 5432, 86400),
("agents", "redis.host", "rmi-redis", 86400),
("agents", "redis.port", 6379, 86400),
("agents", "clickhouse.host", "rmi-clickhouse", 86400),
("agents", "neo4j.host", "rmi-neo4j", 86400),
("agents", "neo4j.port", 7474, 86400),
("agents", "qdrant.host", "rmi-qdrant", 86400),
("agents", "ollama.url", "http://ollama:11434", 86400),
("agents", "embedder.current", "bge-m3", 86400),
("agents", "embedder.target", "qwen3-embedding:4b", 86400),
("agents", "embedder.current_dims", 1024, 86400),
("agents", "embedder.target_dims", 2048, 86400),
("agents", "main.path", "/root/backend/main.py", 86400),
("agents", "main.lines_target", 94, 86400),
("agents", "legacy.lines", 8475, 86400),
("agents", "frozen_manifest.count", 14, 86400),
("agents", "mustdo.total", 30, 86400),
("agents", "phase.current", "0", 86400),
("agents", "deploy.method", "blue-green", 86400),
]
async def seed_facts() -> int:
"""Seed the store with the top 20 verified system facts.
Returns the number of facts written. Idempotent — overwrites existing.
"""
written = 0
for namespace, key, value, ttl in SEED_FACTS:
if await write_fact(namespace, key, value, ttl):
written += 1
return written

260
backend/app/agents/loop.py
View file

@ -1,260 +0,0 @@
"""Agent Loop Specification (M1 — P2 #30 in v3 unfuck plan).
Frozen on creation. Modified only via bounded-task delegation with Task ID
in /home/z/my-project/worklog.md. See DESIGN.md §M1.
Defines the formal contract for any AI agent loop that runs on RMI infrastructure:
Hermes, claude-code, aider, GLM-5.2. Without this spec, loops are unbounded —
they iterate forever, burn tokens, produce no verifiable artifacts.
Usage from a Hermes cron task:
from app.agents.loop import BoundedAgentLoop, TaskInput, LoopBudget
loop = BoundedAgentLoop(
task=TaskInput(
task_id="30-a",
description="Add the new typed error class",
success_criteria="error class exists and passes mypy",
verify_command="python -c 'from app.core.errors import NewError'",
),
budget=LoopBudget(max_iterations=5, max_tokens=10_000),
)
result = await loop.run()
if result.verify_passed:
...
"""
from __future__ import annotations
import time
from typing import Any
from pydantic import BaseModel, ConfigDict, Field
# ── Input Contract ──────────────────────────────────────────────────────
class TaskInput(BaseModel):
"""What every agent loop receives."""
model_config = ConfigDict(strict=True, frozen=True)
task_id: str = Field(
...,
pattern=r"^\d+-[a-z0-9-]+$",
description="Globally-unique ID matching ^\\d+-[a-z0-9-]+$. Logged before delegation.",
)
description: str = Field(
...,
max_length=500,
description="One-paragraph task description. No multi-page briefs — split the task.",
)
context_budget_tokens: int = Field(
default=8000,
le=32000,
description="How much context the agent loads. Larger = more expensive.",
)
allowed_tools: list[str] = Field(
...,
min_length=1,
description="Whitelist of MCP tools the agent may call. Anything else trips a kill switch.",
)
success_criteria: str = Field(
...,
max_length=300,
description="One-sentence definition of 'done.' Verifiable by verify_command.",
)
verify_command: str = Field(
...,
max_length=200,
description="Shell command that returns 0 if success_criteria is met.",
)
# ── Budget / Kill Switches ─────────────────────────────────────────────
class LoopBudget(BaseModel):
"""The four kill switches. Checked every iteration."""
model_config = ConfigDict(strict=True, frozen=True)
max_iterations: int = Field(default=20, le=100)
max_tokens: int = Field(default=50_000, le=500_000)
max_wallclock_seconds: int = Field(default=900, le=3600)
max_spend_usd: float = Field(default=5.0, le=50.0)
# ── Output Contract ─────────────────────────────────────────────────────
class TaskOutput(BaseModel):
"""What every agent loop returns."""
model_config = ConfigDict(strict=True)
task_id: str
files_touched: list[str] = Field(default_factory=list)
lines_added: int = 0
lines_removed: int = 0
verify_passed: bool = False
worklog_entry: str = ""
spend_usd: float = 0.0
iterations_used: int = 0
aborted: bool = False
abort_reason: str | None = None
# ── Kill switch reasons ─────────────────────────────────────────────────
class BudgetExceededError(RuntimeError):
"""Raised when any kill switch trips."""
# ── Bounded Agent Loop ──────────────────────────────────────────────────
class BoundedAgentLoop:
"""Wraps any agent loop with kill switches and a verifiable output contract.
This is the production runtime. For the actual loop body, subclass and
override _step(). The base class enforces the budget and produces the
output contract.
"""
def __init__(self, task: TaskInput, budget: LoopBudget | None = None) -> None:
self.task = task
self.budget = budget or LoopBudget()
self._iterations_used = 0
self._tokens_used = 0
self._spend_usd = 0.0
self._files_touched: list[str] = []
self._lines_added = 0
self._lines_removed = 0
self._start_time = 0.0
self._aborted = False
self._abort_reason: str | None = None
async def run(self) -> TaskOutput:
"""Execute the loop until done, budget exceeded, or verify passes."""
self._start_time = time.monotonic()
# Load long-term memory from fact_store at loop start.
facts = await self._load_facts()
context = self._build_initial_context(facts)
while not self._aborted:
self._check_budget()
self._iterations_used += 1
try:
step_result = await self._step(context)
except BudgetExceededError as exc:
self._aborted = True
self._abort_reason = str(exc)
break
self._record_step(step_result)
context = self._update_context(context, step_result)
# Check verify_command after each step (cheap path).
if await self._verify():
break
# Final verification.
verify_passed = await self._verify()
return TaskOutput(
task_id=self.task.task_id,
files_touched=self._files_touched,
lines_added=self._lines_added,
lines_removed=self._lines_removed,
verify_passed=verify_passed,
worklog_entry=self._build_worklog_entry(),
spend_usd=self._spend_usd,
iterations_used=self._iterations_used,
aborted=self._aborted,
abort_reason=self._abort_reason,
)
# ── To be overridden by subclasses ──────────────────────────────────
async def _step(self, context: Any) -> dict[str, Any]:
"""One iteration of the agent loop. Subclass and implement."""
raise NotImplementedError
# ── Built-in budget enforcement ─────────────────────────────────────
def _check_budget(self) -> None:
"""Throws BudgetExceededError if any kill switch is tripped."""
if self._iterations_used + 1 > self.budget.max_iterations:
raise BudgetExceededError(
f"max_iterations={self.budget.max_iterations} exceeded"
)
elapsed = time.monotonic() - self._start_time
if elapsed > self.budget.max_wallclock_seconds:
raise BudgetExceededError(
f"max_wallclock_seconds={self.budget.max_wallclock_seconds} exceeded"
)
if self._tokens_used > self.budget.max_tokens:
raise BudgetExceededError(
f"max_tokens={self.budget.max_tokens} exceeded"
)
if self._spend_usd > self.budget.max_spend_usd:
raise BudgetExceededError(
f"max_spend_usd={self.budget.max_spend_usd} exceeded"
)
# ── Helpers (override-friendly) ─────────────────────────────────────
async def _load_facts(self) -> dict[str, Any]:
"""Load facts from fact_store at loop start."""
from app.agents.fact_store import load_facts
return await load_facts(namespace="agents")
async def _verify(self) -> bool:
"""Run verify_command and return True if exit code is 0."""
import asyncio
try:
proc = await asyncio.create_subprocess_shell(
self.task.verify_command,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
stdout, stderr = await asyncio.wait_for(
proc.communicate(), timeout=120
)
return proc.returncode == 0
except (asyncio.TimeoutError, OSError):
return False
def _build_initial_context(self, facts: dict[str, Any]) -> Any:
"""Build the initial context for step 0. Override for custom merging."""
return {
"task": self.task.model_dump(),
"facts": facts,
"budget": self.budget.model_dump(),
"iteration": 0,
}
def _update_context(self, context: Any, step_result: dict[str, Any]) -> Any:
"""Update context for the next iteration."""
context = dict(context)
context["iteration"] = context.get("iteration", 0) + 1
context["last_step"] = step_result
return context
def _record_step(self, step_result: dict[str, Any]) -> None:
"""Update internal counters from step result."""
self._tokens_used += int(step_result.get("tokens_used", 0))
self._spend_usd += float(step_result.get("spend_usd", 0))
self._files_touched.extend(step_result.get("files_touched", []))
self._lines_added += int(step_result.get("lines_added", 0))
self._lines_removed += int(step_result.get("lines_removed", 0))
def _build_worklog_entry(self) -> str:
"""Build the worklog entry for this task."""
return (
f"task_id: {self.task.task_id}\n"
f"description: {self.task.description}\n"
f"iterations_used: {self._iterations_used}\n"
f"tokens_used: {self._tokens_used}\n"
f"spend_usd: ${self._spend_usd:.3f}\n"
f"files_touched: {len(self._files_touched)}\n"
f"lines_added: {self._lines_added}\n"
f"lines_removed: {self._lines_removed}\n"
f"aborted: {self._aborted}"
+ (f" (reason: {self._abort_reason})" if self._abort_reason else "")
)

1
backend/app/api/__init__.py
View file

@ -1 +0,0 @@
"""HTTP transport layer. Routes are thin: parse → call domain service → return."""

33
backend/app/api/deps.py
View file

@ -1,33 +0,0 @@
"""Shared FastAPI dependencies.
Use these in route signatures to inject cross-cutting concerns:
from app.api.deps import get_redis, get_current_user, get_settings
Actual implementations live in `app/core/`. This module is a re-export
facade so route authors don't need to know which core module owns what.
"""
from __future__ import annotations
# Re-exports — actual implementations come from app/core/.
# Core modules are populated by the parallel DeepSeek tasks (DS-1..DS-10).
# Until then, these imports will fail; routes should not depend on them yet.
try:
from app.core.redis import get_redis # noqa: F401
except ImportError:
get_redis = None # type: ignore[assignment]
try:
from app.core.db import get_db # noqa: F401
except ImportError:
get_db = None # type: ignore[assignment]
try:
from app.core.auth import get_current_user, get_optional_user # noqa: F401
except ImportError:
get_current_user = None # type: ignore[assignment]
get_optional_user = None # type: ignore[assignment]
try:
from app.core.config import settings # noqa: F401
except ImportError:
from app.config import settings # fallback until core.config lands

73
backend/app/api/v1/__init__.py
View file

@ -1,73 +0,0 @@
"""V1 API router aggregator.
The strangle: new v1 routes are added here as domains migrate. The legacy
main.py still mounts all old routes; we ADD new v1 routes on top so they
co-exist until cutover.
To add a new domain:
1. Create app/api/v1/<group>/<domain>.py with APIRouter
2. Import and append it to `api_v1_router` below
3. Mount the route prefix in the domain's __init__.py
"""
from __future__ import annotations
from fastapi import APIRouter
# Aggregator list — populated as domains migrate.
# Each entry is an APIRouter from app/api/v1/<group>/<domain>.py.
api_v1_router: list[APIRouter] = []
# Aggregator router — single mount point for v1.
# When domains migrate, replace this with a real aggregator:
# from app.api.v1.public import router as public_router
# api_v1_router.append(public_router)
router = APIRouter(prefix="/api/v1", tags=["v1"])
# ── Migrated domains ───────────────────────────────────────────────────
# Each migrated domain is imported here. The router exposes endpoints
# at /api/v1/<domain>/* (path defined per-router).
#
# During strangelfig, the LEGACY /api/v1/<domain>/* endpoints remain
# mounted in main.py. The new v1 router is mounted at the same path
# (FastAPI handles prefix-based routing) — first match wins, so the
# legacy stays until we explicitly remove it.
from app.api.v1.auth.alerts import router as alerts_router # noqa: E402
api_v1_router.append(alerts_router)
from app.api.v1.public.wallet import router as wallet_router # noqa: E402
api_v1_router.append(wallet_router)
from app.api.v1.public.token import router as token_router # noqa: E402
api_v1_router.append(token_router)
from app.api.v1.public.scanner import router as scanner_router # noqa: E402
api_v1_router.append(scanner_router)
# x402 moved to app.domain.x402 (T34 v2)
# Old app/api/v1/x402/payments.py removed to avoid model conflicts
from app.api.v1.rag.search import router as rag_v2_router # noqa: E402
api_v1_router.append(rag_v2_router)
from app.api.v1.admin.alerts_webhook import router as admin_alerts_webhook_router # noqa: E402
api_v1_router.append(admin_alerts_webhook_router)
from app.api.v1.catalog import router as catalog_router # noqa: E402
api_v1_router.append(catalog_router)
def build_v1_router() -> APIRouter:
"""Construct the v1 aggregator with all migrated routes mounted."""
aggregated = APIRouter(prefix="/api/v1")
for r in api_v1_router:
aggregated.include_router(r)
return aggregated

4
backend/app/api/v1/admin/__init__.py
View file

@ -1,4 +0,0 @@
"""Admin routes — admin role required.
Target: user management, system config, ops, bulletin moderation.
"""

215
backend/app/api/v1/admin/alerts_webhook.py
View file

@ -1,215 +0,0 @@
"""V1 Admin alerts webhook — receives AlertManager notifications.
This is the HTTP bridge between Prometheus AlertManager and the RMI backend.
AlertManager POSTs JSON payloads here for two receivers:
- /api/v1/admin/alerts/webhook — default (all severities, from rmi-alerts receiver)
- /api/v1/admin/alerts/critical — critical only (from rmi-critical receiver)
Each handler:
1. Parses the AlertManager v2 webhook payload
2. Persists to Redis (sorted set, capped) for in-app display
3. Logs structured record (JSON, single line)
4. Returns 200 OK immediately so AlertManager doesn't retry
AlertManager webhook payload shape (Prometheus):
{
"version": "4",
"groupKey": "<strings>",
"status": "firing|resolved",
"receiver": "rmi-alerts",
"groupLabels": {"alertname": "...", ...},
"commonLabels": {...},
"commonAnnotations": {...},
"externalURL": "...",
"alerts": [
{
"status": "firing|resolved",
"labels": {...},
"annotations": {...},
"startsAt": "RFC3339",
"endsAt": "RFC3339",
"generatorURL": "..."
}
]
}
"""
from __future__ import annotations
import json
import logging
import os
import time
from typing import Any
from fastapi import APIRouter, Request
from pydantic import BaseModel, Field
logger = logging.getLogger("rmi.admin.alerts_webhook")
# Cap the in-Redis alert history so it doesn't grow unbounded.
_REDIS_CAP = int(os.getenv("RMI_ALERTS_HISTORY_CAP", "500"))
router = APIRouter(prefix="/api/v1/admin/alerts", tags=["admin-alerts"])
class AlertmanagerAlert(BaseModel):
status: str
labels: dict[str, str] = Field(default_factory=dict)
annotations: dict[str, str] = Field(default_factory=dict)
startsAt: str | None = None
endsAt: str | None = None
generatorURL: str | None = None
class AlertmanagerPayload(BaseModel):
version: str | None = None
groupKey: str | None = None
status: str
receiver: str | None = None
groupLabels: dict[str, str] = Field(default_factory=dict)
commonLabels: dict[str, str] = Field(default_factory=dict)
commonAnnotations: dict[str, str] = Field(default_factory=dict)
externalURL: str | None = None
alerts: list[AlertmanagerAlert] = Field(default_factory=list)
def _redis_url_from_env() -> str:
"""Build a Redis URL from discrete REDIS_HOST/PORT/DB/PASSWORD env vars.
Falls back to REDIS_URL if set, otherwise localhost. Returns a URL with
auth credentials embedded if REDIS_PASSWORD is set.
"""
explicit = os.getenv("REDIS_URL")
if explicit:
return explicit
host = os.getenv("REDIS_HOST", "localhost")
port = os.getenv("REDIS_PORT", "6379")
db = os.getenv("REDIS_DB", "0")
password = os.getenv("REDIS_PASSWORD", "")
if password:
return f"redis://:{password}@{host}:{port}/{db}"
return f"redis://{host}:{port}/{db}"
def _redis_client():
"""Lazy Redis import — avoids forcing a Redis dep at module load."""
try:
import redis.asyncio as redis_async # type: ignore
url = _redis_url_from_env()
return redis_async.from_url(url, decode_responses=True)
except Exception as exc: # pragma: no cover - degraded mode
logger.warning("redis_unavailable", extra={"err": str(exc)})
return None
async def _persist_to_redis(payload: AlertmanagerPayload, severity: str) -> int:
"""Push payload to Redis sorted set capped at _REDIS_CAP entries.
Returns number of alerts persisted (0 if Redis is down).
"""
client = _redis_client()
if client is None:
return 0
try:
score = time.time()
record = json.dumps(
{
"received_at": score,
"severity_bucket": severity,
"receiver": payload.receiver,
"status": payload.status,
"group_labels": payload.groupLabels,
"common_labels": payload.commonLabels,
"common_annotations": payload.commonAnnotations,
"alerts": [a.model_dump() for a in payload.alerts],
},
default=str,
)
key = "rmi:alerts:webhook"
async with client.pipeline(transaction=False) as pipe:
pipe.zadd(key, {record: score})
pipe.zremrangebyrank(key, 0, -(_REDIS_CAP + 1))
pipe.expire(key, 7 * 24 * 3600) # 7 days
await pipe.execute()
return len(payload.alerts)
except Exception as exc:
logger.warning("redis_persist_failed", extra={"err": str(exc)})
return 0
finally:
try:
await client.aclose()
except Exception:
pass
def _log_payload(payload: AlertmanagerPayload, severity: str, persisted: int) -> None:
"""Single-line JSON log so log aggregators can index cleanly."""
record = {
"ts": time.time(),
"event": "alertmanager_webhook",
"severity_bucket": severity,
"receiver": payload.receiver,
"status": payload.status,
"alert_count": len(payload.alerts),
"persisted": persisted,
"alertname": payload.groupLabels.get("alertname"),
"common_labels": payload.commonLabels,
}
logger.info(json.dumps(record, default=str))
@router.post("/webhook", status_code=200)
async def alerts_webhook(payload: AlertmanagerPayload, request: Request) -> dict[str, Any]:
"""Default receiver webhook — all severities."""
severity = payload.commonLabels.get("severity", "unknown")
persisted = await _persist_to_redis(payload, severity)
_log_payload(payload, severity, persisted)
return {
"ok": True,
"received": len(payload.alerts),
"persisted": persisted,
"severity": severity,
"alertname": payload.groupLabels.get("alertname"),
}
@router.post("/critical", status_code=200)
async def alerts_critical_webhook(
payload: AlertmanagerPayload, request: Request
) -> dict[str, Any]:
"""Critical-only webhook — escalations only (severity=critical)."""
# Defensive: if someone misroutes a non-critical here, log and accept
# (we don't want to lose data). Severity bucket stays "critical" because
# the receiver name implies it.
severity = "critical"
persisted = await _persist_to_redis(payload, severity)
_log_payload(payload, severity, persisted)
return {
"ok": True,
"received": len(payload.alerts),
"persisted": persisted,
"severity": severity,
"alertname": payload.groupLabels.get("alertname"),
"bucket": "critical",
}
@router.get("/recent", status_code=200)
async def alerts_recent(limit: int = 50) -> dict[str, Any]:
"""Read recent alerts (debug endpoint — admin only in production)."""
client = _redis_client()
if client is None:
return {"ok": False, "error": "redis_unavailable", "items": []}
try:
# Newest first
raw = await client.zrevrange("rmi:alerts:webhook", 0, max(0, limit - 1))
items = [json.loads(r) for r in raw]
return {"ok": True, "count": len(items), "items": items}
except Exception as exc:
return {"ok": False, "error": str(exc), "items": []}
finally:
try:
await client.aclose()
except Exception:
pass

60
backend/app/api/v1/admin/glitchtip_test.py
View file

@ -1,60 +0,0 @@
"""T07 GlitchTip test endpoint.
POST /api/v1/_test/glitchtip
{"type": "error", "message": "test error"}
POST /api/v1/_test/exception
→ triggers a real exception, captured by GlitchTip
POST /api/v1/_test/message
→ captures an info-level message
Used for:
- Verifying the GlitchTip pipeline works
- Smoke testing after deploys
- Demonstrating the secret-scrubbing before_send hook
"""
from __future__ import annotations
import logging
from fastapi import APIRouter
from pydantic import BaseModel
log = logging.getLogger(__name__)
router = APIRouter(prefix="/api/v1/_test", tags=["test"])
class GlitchtipTestRequest(BaseModel):
type: str = "error" # error | exception | message
message: str = "test event from RMI"
secret: str | None = None # should be REDACTED in Sentry
@router.post("/glitchtip")
async def test_glitchtip(req: GlitchtipTestRequest) -> dict:
"""Trigger a test event. Tests the secret-scrubbing before_send hook."""
if req.type == "exception":
try:
raise ValueError(req.message)
except Exception as e:
try:
from app.core.observability import capture_exception
capture_exception(e, secret=req.secret, route="/api/v1/_test/glitchtip")
except ImportError:
log.exception("test_exception_no_sentry")
return {"captured": "exception", "message": req.message}
if req.type == "message":
try:
from app.core.observability import capture_message
capture_message(req.message, level="warning", secret=req.secret)
except ImportError:
log.warning(f"test_message_no_sentry: {req.message}")
return {"captured": "message", "message": req.message}
# default: error log + capture
log.error(f"test_error: {req.message} (secret={req.secret})")
try:
from app.core.observability import capture_message
capture_message(req.message, level="error", secret=req.secret)
except ImportError:
pass
return {"captured": "error", "message": req.message, "secret_redacted_in_sentry": True}

4
backend/app/api/v1/auth/__init__.py
View file

@ -1,4 +0,0 @@
"""Authenticated routes — JWT required.
Target: portfolio, alerts, intel feeds, profile, settings.
"""

70
backend/app/api/v1/auth/alerts.py
View file

@ -1,70 +0,0 @@
"""V1 alerts route — thin HTTP layer over app.domain.alerts.
Rules (per 2026 standards):
- Parse request → call service → return response. No business logic here.
- Pydantic models for request/response, defined in app.domain.alerts.models.
- No direct Redis or DB access. Goes through AlertService.
- All errors raised as AppError subclasses, handled by core.errors handlers.
"""
from __future__ import annotations
from typing import Annotated, Any
from fastapi import APIRouter, Depends
from app.core.auth import get_optional_user
from app.core.errors import AuthError, NotFoundError
from app.domain.alerts import (
AlertService,
AlertSubscription,
CreateAlertRequest,
)
from app.models import PaginatedResponse
router = APIRouter(prefix="/api/v1/alerts", tags=["alerts"])
def _service() -> AlertService:
"""Dependency: a fresh AlertService per request (stateless)."""
return AlertService()
@router.post("/subscribe", response_model=AlertSubscription, status_code=201)
async def subscribe(
req: CreateAlertRequest,
user: Annotated[dict[str, Any] | None, Depends(get_optional_user)],
svc: Annotated[AlertService, Depends(_service)],
) -> AlertSubscription:
"""Create a new alert subscription for a token."""
owner_id = (user or {}).get("id")
return await svc.create_subscription(req, owner_id=owner_id)
@router.get("", response_model=PaginatedResponse)
async def list_alerts(
user: Annotated[dict[str, Any] | None, Depends(get_optional_user)],
svc: Annotated[AlertService, Depends(_service)],
) -> PaginatedResponse:
"""List alert subscriptions. Authenticated users see their own; anonymous sees all (legacy behavior)."""
owner_id = (user or {}).get("id")
items = await svc.list_subscriptions(owner_id=owner_id)
return PaginatedResponse(
items=[i.model_dump(mode="json") for i in items],
total=len(items),
)
@router.delete("/{alert_id}", status_code=204)
async def cancel(
alert_id: str,
user: Annotated[dict[str, Any] | None, Depends(get_optional_user)],
svc: Annotated[AlertService, Depends(_service)],
) -> None:
"""Cancel (delete) a subscription. Owner only."""
owner_id = (user or {}).get("id")
if owner_id is None:
raise AuthError("authentication required to cancel alerts")
ok = await svc.cancel_subscription(alert_id, owner_id=owner_id)
if not ok:
raise NotFoundError(f"alert {alert_id} not found or not owned by you")
return None

4
backend/app/api/v1/catalog/__init__.py
View file

@ -1,4 +0,0 @@
"""Catalog v1 routes — thin HTTP layer."""
from .router import router
__all__ = ["router"]

BIN
backend/app/api/v1/catalog/__pycache__/__init__.cpython-311.pyc
View file

Binary file not shown.

BIN
backend/app/api/v1/catalog/__pycache__/router.cpython-311.pyc
View file

Binary file not shown.

155
backend/app/api/v1/catalog/router.py
View file

@ -1,155 +0,0 @@
"""T27B HTTP routes — CatalogService endpoints.
Per v4.0 §T27. The thin HTTP layer over app.catalog.service.CatalogService.
"""
from __future__ import annotations
from typing import Any
from fastapi import APIRouter, HTTPException
from pydantic import BaseModel, Field
from app.catalog.models import Chain
from app.catalog.service import get_catalog
router = APIRouter(prefix="/api/v1/catalog", tags=["catalog"])
# ── Request models ───────────────────────────────────────────────
class RagIngestRequest(BaseModel):
content: str = Field(..., min_length=1)
collection: str = "scam_intel"
doc_id: str | None = None
metadata: dict[str, Any] = Field(default_factory=dict)
class RagSearchRequest(BaseModel):
query: str = Field(..., min_length=1)
collection: str = "scam_intel"
top_k: int = Field(default=5, ge=1, le=50)
class ResolveEntityRequest(BaseModel):
wallet_id: str
max_chains: int = Field(default=5, ge=1, le=20)
class FindRiskyTokensRequest(BaseModel):
min_rug_count: int = Field(default=1, ge=1)
chain: str | None = None
limit: int = Field(default=50, ge=1, le=200)
class AttachRagRequest(BaseModel):
chain: str
address: str
qdrant_point_id: str
# ── Health / introspection ───────────────────────────────────────
@router.get("/stats")
async def stats() -> dict:
"""Catalog stats: which stores are reachable + entity counts."""
return await get_catalog().stats()
@router.get("/probe")
async def probe() -> dict:
"""Probe which stores are reachable from this container."""
return await get_catalog().probe_stores()
# ── Token endpoints ─────────────────────────────────────────────
@router.get("/tokens/{chain}/{address}")
async def get_token(chain: str, address: str) -> dict:
"""Get a token by chain+address. Returns full Token model + provenance."""
try:
c = Chain(chain)
except ValueError:
raise HTTPException(400, f"unknown chain: {chain}")
tok = await get_catalog().get_token(c, address)
if not tok:
raise HTTPException(404, "token not found")
return tok.model_dump(mode="json")
@router.get("/tokens/{chain}/{address}/risk")
async def get_token_risk(chain: str, address: str) -> dict:
"""Recipe 3 — Real-time risk score. Composes Redis + Postgres + Neo4j."""
try:
c = Chain(chain)
except ValueError:
raise HTTPException(400, f"unknown chain: {chain}")
return await get_catalog().get_token_risk(c, address)
@router.post("/tokens/risky-by-deployer")
async def risky_tokens(req: FindRiskyTokensRequest) -> dict:
"""Recipe 1 — Find tokens deployed by wallets with rug history."""
chain_enum = None
if req.chain:
try:
chain_enum = Chain(req.chain)
except ValueError:
raise HTTPException(400, f"unknown chain: {req.chain}")
tokens = await get_catalog().find_tokens_by_deployer_history(
min_rug_count=req.min_rug_count, chain=chain_enum, limit=req.limit
)
return {
"count": len(tokens),
"tokens": [t.model_dump(mode="json") for t in tokens],
}
# ── Wallet endpoints ─────────────────────────────────────────────
@router.get("/wallets/{chain}/{address}")
async def get_wallet(chain: str, address: str) -> dict:
try:
c = Chain(chain)
except ValueError:
raise HTTPException(400, f"unknown chain: {chain}")
w = await get_catalog().get_wallet(c, address)
if not w:
raise HTTPException(404, "wallet not found")
return w.model_dump(mode="json")
# ── Entity resolution (Recipe 5) ────────────────────────────────
@router.post("/entities/resolve")
async def resolve_entity(req: ResolveEntityRequest) -> dict:
"""Cross-chain entity resolution via Neo4j Cypher."""
return await get_catalog().resolve_entity(req.wallet_id, req.max_chains)
# ── RAG bridge endpoints ────────────────────────────────────────
@router.post("/rag/search")
async def rag_search(req: RagSearchRequest) -> dict:
"""Search the RAG system. Returns ranked hits with RRF scores."""
hits = await get_catalog().rag_search(
query=req.query, collection=req.collection, top_k=req.top_k
)
return {"count": len(hits), "hits": hits}
@router.post("/rag/ingest")
async def rag_ingest(req: RagIngestRequest) -> dict:
"""Ingest content into RAG. Returns qdrant_point_id for cross-store linking."""
return await get_catalog().rag_ingest(
content=req.content,
collection=req.collection,
doc_id=req.doc_id,
metadata=req.metadata,
)
@router.post("/tokens/{chain}/{address}/attach-rag")
async def attach_rag(chain: str, address: str, req: AttachRagRequest) -> dict:
"""Link an existing RAG embedding (Qdrant point) to a Token row."""
try:
c = Chain(chain)
except ValueError:
raise HTTPException(400, f"unknown chain: {chain}")
ok = await get_catalog().attach_rag_to_token(c, address, req.qdrant_point_id)
if not ok:
raise HTTPException(404, "token not found or update failed")
return {"ok": True, "chain": chain, "address": address, "rag_embedding_id": req.qdrant_point_id}

4
backend/app/api/v1/mcp/__init__.py
View file

@ -1,4 +0,0 @@
"""MCP v1 routes."""
from .router import router
__all__ = ["router"]

BIN
backend/app/api/v1/mcp/__pycache__/__init__.cpython-311.pyc
View file

Binary file not shown.

BIN
backend/app/api/v1/mcp/__pycache__/router.cpython-311.pyc
View file

Binary file not shown.

120
backend/app/api/v1/mcp/router.py
View file

@ -1,120 +0,0 @@
"""T33 MCP Server — HTTP wrapper for SSE transport.
Per v4.0 §T33. Endpoints:
POST /mcp JSON-RPC 2.0 endpoint
GET /mcp/tools Tool catalog
POST /mcp/call/{tool_id} Direct tool execution (no JSON-RPC)
The server speaks the Model Context Protocol natively. Claude Desktop
and Cursor connect via:
{"mcpServers": {"rugmunch": {"url": "https://mcp.rugmunch.io/mcp", "transport": "sse"}}}
"""
from __future__ import annotations
import json
import logging
from typing import Any
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel
from app.mcp.server import TOOL_CATALOG, call_tool
router = APIRouter(prefix="/mcp", tags=["mcp"])
log = logging.getLogger(__name__)
class JsonRpcRequest(BaseModel):
jsonrpc: str = "2.0"
method: str
params: dict[str, Any] = {}
id: int | str | None = None
class JsonRpcResponse(BaseModel):
jsonrpc: str = "2.0"
result: Any | None = None
error: dict | None = None
id: int | str | None = None
@router.post("")
async def jsonrpc_handler(req: JsonRpcRequest) -> dict:
"""JSON-RPC 2.0 endpoint for MCP clients.
Methods:
- initialize → returns server info
- tools/list → returns tool catalog
- tools/call → dispatches to backend
- resources/list → empty
- prompts/list → empty
"""
if req.jsonrpc != "2.0":
return {"jsonrpc": "2.0", "error": {"code": -32600, "message": "invalid jsonrpc version"}, "id": req.id}
if req.method == "initialize":
return {
"jsonrpc": "2.0",
"result": {
"protocolVersion": "2024-11-05",
"serverInfo": {
"name": "rugmunch-intelligence",
"version": "4.0.0",
"description": "Crypto intelligence platform — 13+ chains, 8 MCP tools, x402 paid tier",
},
"capabilities": {"tools": {}, "resources": {}, "prompts": {}},
},
"id": req.id,
}
if req.method == "tools/list":
return {
"jsonrpc": "2.0",
"result": {"tools": TOOL_CATALOG},
"id": req.id,
}
if req.method == "tools/call":
name = req.params.get("name", "")
arguments = req.params.get("arguments", {})
if not name:
return {"jsonrpc": "2.0", "error": {"code": -32602, "message": "tool name required"}, "id": req.id}
result = await call_tool(name, arguments)
return {
"jsonrpc": "2.0",
"result": {
"content": [{"type": "text", "text": json.dumps(result, default=str)[:50000]}],
"isError": "error" in result,
},
"id": req.id,
}
if req.method == "resources/list":
return {"jsonrpc": "2.0", "result": {"resources": []}, "id": req.id}
if req.method == "prompts/list":
return {"jsonrpc": "2.0", "result": {"prompts": []}, "id": req.id}
if req.method == "notifications/initialized":
return {"jsonrpc": "2.0", "result": {}, "id": req.id}
return {
"jsonrpc": "2.0",
"error": {"code": -32601, "message": f"method not found: {req.method}"},
"id": req.id,
}
@router.get("/tools")
async def list_tools() -> dict:
"""Plain JSON endpoint (for direct integration, no JSON-RPC)."""
return {"server": "rugmunch-intelligence", "version": "4.0.0", "tools": TOOL_CATALOG}
@router.post("/call/{tool_id}")
async def direct_call(tool_id: str, request: Request) -> dict:
"""Direct tool execution (no JSON-RPC). For curl/scripts."""
body = await request.json() if request.headers.get("content-type", "").startswith("application/json") else {}
arguments = body.get("arguments", body) if isinstance(body, dict) else {}
result = await call_tool(tool_id, arguments)
return result

4
backend/app/api/v1/public/__init__.py
View file

@ -1,4 +0,0 @@
"""Public routes — no authentication required.
Target: scanner, wallet lookup, token info, pricing, health.
"""

39
backend/app/api/v1/public/scanner.py
View file

@ -1,39 +0,0 @@
"""V1 scanner route — thin HTTP layer over app.domain.scanner.
The actual scan logic lives in app.token_scanner (legacy 4,109-line
monolith). The new domain/ layer is a Pydantic facade that:
- Validates the request
- Calls the legacy scan_token
- Wraps the result in a Pydantic response
Per-domain migration: as check_* functions are rewritten, the service
stops calling the legacy monolith and uses the new modules instead.
"""
from __future__ import annotations
from typing import Annotated, Any
from fastapi import APIRouter, Depends
from app.core.auth import get_optional_user
from app.domain.scanner import ScanRequest, ScanResponse, ScannerService
router = APIRouter(prefix="/api/v1/scanner", tags=["scanner"])
def _service() -> ScannerService:
return ScannerService()
@router.post("/scan", response_model=ScanResponse)
async def scan(
req: ScanRequest,
svc: Annotated[ScannerService, Depends(_service)],
_user: Annotated[dict[str, Any] | None, Depends(get_optional_user)],
) -> ScanResponse:
"""Full token scan. Multi-module. Returns Pydantic response.
The legacy /api/v1/scanner/scan (and /api/v1/token/scan) are still
served — they take priority on the same path during strangelfig.
"""
return await svc.scan(req)

78
backend/app/api/v1/public/token.py
View file

@ -1,78 +0,0 @@
"""V1 token route — thin HTTP layer over app.domain.token."""
from __future__ import annotations
from typing import Annotated, Any
from fastapi import APIRouter, Depends, Query
from app.core.auth import get_optional_user
from app.domain.token import (
TokenDetail,
TokenRisk,
TokenScanRequest,
TokenScanResult,
TokenService,
)
from app.models import PaginatedResponse
router = APIRouter(prefix="/api/v1/token", tags=["token"])
def _service() -> TokenService:
return TokenService()
@router.get("/{address}", response_model=TokenDetail)
async def get_detail(
address: str,
svc: Annotated[TokenService, Depends(_service)],
chain: str = Query(default="solana"),
) -> TokenDetail:
"""Token metadata + supply + verification."""
return await svc.get_detail(address, chain=chain)
@router.get("/{address}/holders", response_model=PaginatedResponse)
async def get_holders(
address: str,
svc: Annotated[TokenService, Depends(_service)],
chain: str = Query(default="solana"),
limit: int = Query(default=100, ge=1, le=500),
) -> PaginatedResponse:
"""Top token holders."""
holders = await svc.get_holders(address, chain=chain, limit=limit)
return PaginatedResponse(
items=[h.model_dump(mode="json") for h in holders],
total=len(holders),
)
@router.get("/{address}/liquidity", response_model=dict)
async def get_liquidity(
address: str,
svc: Annotated[TokenService, Depends(_service)],
chain: str = Query(default="solana"),
) -> dict:
"""Token liquidity + pool info."""
liq = await svc.get_liquidity(address, chain=chain)
return liq.model_dump(mode="json")
@router.get("/{address}/risk", response_model=TokenRisk)
async def get_risk(
address: str,
svc: Annotated[TokenService, Depends(_service)],
chain: str = Query(default="solana"),
) -> TokenRisk:
"""Combined risk: holders + liquidity + heuristics."""
return await svc.get_risk(address, chain=chain)
@router.post("/scan", response_model=TokenScanResult)
async def scan(
req: TokenScanRequest,
svc: Annotated[TokenService, Depends(_service)],
_user: Annotated[dict[str, Any] | None, Depends(get_optional_user)],
) -> TokenScanResult:
"""Full token scan. Multi-module. Freemium rate-limit applied in legacy router."""
return await svc.scan(req)

74
backend/app/api/v1/public/wallet.py
View file

@ -1,74 +0,0 @@
"""V1 wallet route — thin HTTP layer over app.domain.wallet.
Rules (per 2026 standards):
- Parse request → call service → return response. No business logic.
- Pydantic models defined in app.domain.wallet.models.
- No direct Redis/DB access. Goes through WalletService.
- Errors raised as AppError subclasses, handled by core.errors.
"""
from __future__ import annotations
from typing import Annotated, Any
from fastapi import APIRouter, Depends, Query
from app.core.auth import get_optional_user
from app.domain.wallet import (
Balance,
ScanRequest,
ScanResult,
WalletAnalysis,
WalletService,
)
from app.models import PaginatedResponse
router = APIRouter(prefix="/api/v1/wallet", tags=["wallet"])
def _service() -> WalletService:
return WalletService()
@router.get("/{address}/analysis", response_model=WalletAnalysis)
async def get_analysis(
address: str,
svc: Annotated[WalletService, Depends(_service)],
chain: str = Query(default="solana"),
) -> WalletAnalysis:
"""Full wallet analysis: risk score + tokens + recent transactions."""
return await svc.analyze(address, chain=chain)
@router.get("/{address}/balance", response_model=Balance)
async def get_balance(
address: str,
svc: Annotated[WalletService, Depends(_service)],
chain: str = Query(default="solana"),
) -> Balance:
"""Wallet balance + token holdings."""
return await svc.get_balance(address, chain=chain)
@router.get("/{address}/transactions", response_model=PaginatedResponse)
async def get_transactions(
address: str,
svc: Annotated[WalletService, Depends(_service)],
chain: str = Query(default="solana"),
limit: int = Query(default=50, ge=1, le=200),
) -> PaginatedResponse:
"""Recent transactions for a wallet."""
txs = await svc.get_transactions(address, chain=chain, limit=limit)
return PaginatedResponse(
items=[t.model_dump(mode="json") for t in txs],
total=len(txs),
)
@router.post("/scan", response_model=ScanResult)
async def scan(
req: ScanRequest,
svc: Annotated[WalletService, Depends(_service)],
_user: Annotated[dict[str, Any] | None, Depends(get_optional_user)],
) -> ScanResult:
"""Multi-chain threat scan. Auth optional — freemium rate-limit applied in legacy router."""
return await svc.scan(req)

BIN
backend/app/api/v1/rag/__pycache__/search.cpython-311.pyc
View file

Binary file not shown.

81
backend/app/api/v1/rag/search.py
View file

@ -1,81 +0,0 @@
"""V1 RAG route — thin HTTP layer over app.rag.
The RAG system is the most coupled module (14 legacy files). This
facade exposes the most-used operations: search, ingest, feedback.
"""
from __future__ import annotations
from typing import Annotated, Any
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, Field
from app.rag import (
FeedbackRecord,
IngestRequest,
IngestResult,
RAGService,
SearchRequest,
SearchResponse,
)
from app.rag.engine import bulk_ingest as engine_bulk_ingest
from app.rag.engine import get_stats as engine_get_stats
router = APIRouter(prefix="/api/v1/rag/v2", tags=["rag"])
def _service() -> RAGService:
return RAGService()
class BulkIngestRequest(BaseModel):
collection: str = "scam_intel"
items: list[dict[str, Any]] = Field(default_factory=list)
@router.post("/search", response_model=SearchResponse)
async def search(
req: SearchRequest,
svc: Annotated[RAGService, Depends(_service)],
) -> SearchResponse:
"""RAG search. Returns Pydantic response with hits + scores."""
return await svc.search(req)
@router.post("/ingest", response_model=IngestResult)
async def ingest(
req: IngestRequest,
svc: Annotated[RAGService, Depends(_service)],
) -> IngestResult:
"""Ingest a document into the RAG system."""
return await svc.ingest(req)
@router.post("/feedback", response_model=IngestResult)
async def feedback(
record: FeedbackRecord,
svc: Annotated[RAGService, Depends(_service)],
) -> IngestResult:
"""Record scanner → RAG feedback. Ingests known scam into known_scams collection."""
ok = await svc.record_feedback(record)
return IngestResult(
doc_id=record.token_address,
collection="known_scams",
status="ok" if ok else "failed",
)
@router.get("/stats")
async def stats() -> dict:
"""Per-collection vector counts + active embedder backend."""
return engine_get_stats()
@router.post("/bulk-ingest")
async def bulk(req: BulkIngestRequest) -> dict:
"""Ingest many items into a collection sequentially (max 500 per call)."""
if not req.items:
raise HTTPException(status_code=400, detail="items must be non-empty")
if len(req.items) > 500:
raise HTTPException(status_code=400, detail="bulk limit 500 per call")
return await engine_bulk_ingest(items=req.items, collection=req.collection)

40
backend/app/api/v1/test_errors.py
View file

@ -1,40 +0,0 @@
"""Test endpoint for verifying error handlers work end-to-end.
This is a development-only endpoint that raises domain errors to verify
the FastAPI exception handlers in app/core/errors.py work correctly.
"""
from __future__ import annotations
from app.core.errors import (
HoneypotDetectedError,
InsufficientFundsError,
PaymentRequiredError,
WalletNotFoundError,
)
from fastapi import APIRouter
router = APIRouter(prefix="/api/v1/_test_errors", tags=["test-errors"])
@router.get("/wallet_not_found/{address}")
async def test_wallet_not_found(address: str):
"""Raises WalletNotFoundError → 404 with code=wallet_not_found."""
raise WalletNotFoundError(address, "ethereum")
@router.get("/insufficient_funds")
async def test_insufficient_funds():
"""Raises InsufficientFundsError → 402 with code=insufficient_funds."""
raise InsufficientFundsError(required=1.5, available=0.3, asset="SOL")
@router.get("/honeypot/{address}")
async def test_honeypot(address: str):
"""Raises HoneypotDetectedError → 422 with code=honeypot_detected."""
raise HoneypotDetectedError(address, "solana")
@router.get("/payment_required")
async def test_payment_required():
"""Raises PaymentRequiredError → 402 with code=payment_required."""
raise PaymentRequiredError("scan_token", 0.05, "solana")

48
backend/app/api/v1/test_ratelimit.py
View file

@ -1,48 +0,0 @@
"""Test endpoint for verifying the rate limiter works end-to-end.
This is a dev-only endpoint that uses rate_limit_dep to enforce
per-tier rate limits. Returns RateLimitInfo in the response.
"""
from __future__ import annotations
from typing import Annotated
from fastapi import APIRouter, Depends, Response
from app.core.errors import RateLimitError
from app.core.rate_limit import RateLimitInfo, rate_limit_dep
router = APIRouter(prefix="/api/v1/_test_ratelimit", tags=["test-ratelimit"])
@router.get("/ping")
async def ping(
response: Response,
rate: Annotated[RateLimitInfo, Depends(rate_limit_dep)],
) -> dict:
"""Hit this endpoint repeatedly to see rate limit kick in.
Returns 200 with rate info, or 429 when limit is exceeded.
Headers: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset.
"""
# Surface rate info in response headers
response.headers["X-RateLimit-Limit"] = str(rate.limit)
response.headers["X-RateLimit-Remaining"] = str(rate.remaining)
response.headers["X-RateLimit-Reset"] = rate.reset_at
response.headers["X-RateLimit-Tier"] = rate.tier
if not rate.allowed:
raise RateLimitError(
f"Rate limit exceeded: {rate.used}/{rate.limit}",
details={
"tier": rate.tier,
"used": rate.used,
"limit": rate.limit,
"reset_at": rate.reset_at,
},
)
return {
"ok": True,
"rate_limit": rate.model_dump(mode="json"),
}

4
backend/app/api/v1/x402/__init__.py
View file

@ -1,4 +0,0 @@
"""x402 paid routes — crypto micropayment gated.
Target: tools (split from legacy x402_tools.py), tokens, wallets, defi, security.
"""

45
backend/app/api/v1/x402/payments.py
View file

@ -1,45 +0,0 @@
"""V1 x402 route — thin HTTP layer over app.domain.x402.
The actual x402 logic lives in 25+ legacy routers (x402_tools,
x402_enforcement, x402_databus_tools, etc.). The new domain layer
is a Pydantic facade that:
- Validates requests
- Calls the legacy routers
- Wraps results in Pydantic models
Per-router cutover: as each x402 router is rewritten, the service
stops calling the legacy code.
"""
from __future__ import annotations
from typing import Annotated
from fastapi import APIRouter, Depends
from app.domain.x402 import (
PaymentFacilitator,
ToolCatalog,
X402Service,
)
router = APIRouter(prefix="/api/v1/x402", tags=["x402"])
def _service() -> X402Service:
return X402Service()
@router.get("/catalog", response_model=ToolCatalog)
async def get_catalog(
svc: Annotated[X402Service, Depends(_service)],
) -> ToolCatalog:
"""Full x402 tool catalog — list of paid tools + categories + pricing."""
return await svc.get_catalog()
@router.get("/facilitators", response_model=list[PaymentFacilitator])
async def get_facilitators(
svc: Annotated[X402Service, Depends(_service)],
) -> list[PaymentFacilitator]:
"""List of enabled payment facilitators and their health status."""
return await svc.get_facilitators()

4
backend/app/api/ws/__init__.py
View file

@ -1,4 +0,0 @@
"""WebSocket endpoints.
Target: real-time alerts, scanner results, intel feeds.
"""

600
backend/app/arkham_counterparties.py
View file

@ -1,600 +0,0 @@
"""
Arkham Counterparties — Entity Relationship Graph & Money Flow Analysis
=====================================================================
Maps address relationships and traces fund flows between entities.
Signals detected:
- Direct trading counterparties (who trades with whom)
- Fund flow paths (where money comes from and goes)
- Entity clustering (wallets controlled by same entity)
- Money flow volume analysis (total in/out, net flow)
- Risk-weighted counterparties (scam/sanctioned exposure)
- Cross-chain relationship mapping
- Historical relationship persistence
Tier : Elite ($0.20)
Price : 200000 atoms
Endpoint: POST /api/v1/x402-tools/arkham_counterparties
"""
import logging
import os
from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from typing import Any
import httpx
logger = logging.getLogger(__name__)
# ── Constants ──────────────────────────────────────────────────────
ARKHAM_API_BASE = "https://api.arkhamintelligence.com"
CACHE_TTL = 300 # 5 minutes
MAX_TRANSACTIONS = 500
MAX_COUNTERPARTIES = 50
# Risk thresholds
HIGH_RISK_THRESHOLD = 75
MEDIUM_RISK_THRESHOLD = 40
LOW_RISK_THRESHOLD = 15
# ── Enums ───────────────────────────────────────────────────────────
class RelationshipType(str, Enum):
DIRECT_TRADE = "direct_trade"
FUNDING_SOURCE = "funding_source"
WITHDRAWAL_DEST = "withdrawal_dest"
SAME_ENTITY = "same_entity"
CROSS_CHAIN = "cross_chain"
SMART_CONTRACT = "smart_contract"
class RiskLevel(str, Enum):
CRITICAL = "critical"
HIGH = "high"
MEDIUM = "medium"
LOW = "low"
NONE = "none"
# ── Data Models ───────────────────────────────────────────────────
@dataclass
class Counterparty:
"""A single counterparty entity related to the target address."""
address: str
chain: str
entity_name: str = ""
entity_category: str = "unknown"
relationship_type: RelationshipType = RelationshipType.DIRECT_TRADE
relationship_strength: float = 0.0 # 0.0 to 1.0
total_volume_in_usd: float = 0.0
total_volume_out_usd: float = 0.0
net_volume_usd: float = 0.0
tx_count: int = 0
first_interaction: int = 0 # unix timestamp
last_interaction: int = 0
risk_score: float = 0.0
risk_level: str = "none"
def to_dict(self) -> dict[str, Any]:
return {
"address": self.address,
"chain": self.chain,
"entity_name": self.entity_name,
"entity_category": self.entity_category,
"relationship_type": (
self.relationship_type.value
if isinstance(self.relationship_type, RelationshipType)
else self.relationship_type
),
"relationship_strength": round(self.relationship_strength, 3),
"total_volume_in_usd": round(self.total_volume_in_usd, 2),
"total_volume_out_usd": round(self.total_volume_out_usd, 2),
"net_volume_usd": round(self.net_volume_usd, 2),
"tx_count": self.tx_count,
"first_interaction": self.first_interaction,
"last_interaction": self.last_interaction,
"risk_score": round(self.risk_score, 1),
"risk_level": self.risk_level,
}
@dataclass
class MoneyFlow:
"""Money flow path between addresses."""
source: str
destination: str
amount_usd: float
token: str = ""
chain: str = ""
tx_hash: str = ""
timestamp: int = 0
hop: int = 1 # distance from original source
def to_dict(self) -> dict[str, Any]:
return {
"source": self.source,
"destination": self.destination,
"amount_usd": round(self.amount_usd, 2),
"token": self.token,
"chain": self.chain,
"tx_hash": self.tx_hash[:18] + "..." if len(self.tx_hash) > 18 else self.tx_hash,
"timestamp": self.timestamp,
"hop": self.hop,
}
@dataclass
class EntityCluster:
"""A cluster of related addresses/wallets."""
cluster_id: str
addresses: list[str]
total_volume_usd: float
entity_hint: str = "" # Likely entity type or name
confidence: float = 0.0 # 0-100
def to_dict(self) -> dict[str, Any]:
return {
"cluster_id": self.cluster_id,
"address_count": len(self.addresses),
"addresses": [a[:12] + "..." for a in self.addresses[:20]],
"total_volume_usd": round(self.total_volume_usd, 2),
"entity_hint": self.entity_hint,
"confidence": round(self.confidence, 1),
}
@dataclass
class CounterpartyReport:
"""Complete entity relationship analysis report."""
target_address: str
chain: str
target_entity_name: str = ""
target_entity_category: str = "unknown"
# Summary stats
total_interactions: int = 0
unique_counterparties: int = 0
total_volume_in_usd: float = 0.0
total_volume_out_usd: float = 0.0
net_volume_usd: float = 0.0
# Key findings
counterparties: list[Counterparty] = field(default_factory=list)
top_counterparties_by_volume: list[Counterparty] = field(default_factory=list)
risk_exposed_counterparties: list[Counterparty] = field(default_factory=list)
# Flow analysis
money_flows: list[MoneyFlow] = field(default_factory=list)
fund_sources: list[str] = field(default_factory=list) # top funding sources
# Clustering
entity_clusters: list[EntityCluster] = field(default_factory=list)
# Risk assessment
max_risk_score: float = 0.0
aggregate_risk_level: str = "none"
scam_exposure_count: int = 0
sanctioned_exposure_count: int = 0
errors: list[str] = field(default_factory=list)
generated_at: str = field(
default_factory=lambda: datetime.now(timezone.utc).isoformat()
)
def to_dict(self) -> dict[str, Any]:
return {
"target_address": self.target_address,
"chain": self.chain,
"target_entity_name": self.target_entity_name,
"target_entity_category": self.target_entity_category,
"summary": {
"total_interactions": self.total_interactions,
"unique_counterparties": self.unique_counterparties,
"total_volume_in_usd": round(self.total_volume_in_usd, 2),
"total_volume_out_usd": round(self.total_volume_out_usd, 2),
"net_volume_usd": round(self.net_volume_usd, 2),
"max_risk_score": round(self.max_risk_score, 1),
"aggregate_risk_level": self.aggregate_risk_level,
},
"counterparties": [c.to_dict() for c in self.counterparties[:MAX_COUNTERPARTIES]],
"top_counterparties_by_volume": [
c.to_dict() for c in self.top_counterparties_by_volume[:10]
],
"risk_exposed_counterparties": [
c.to_dict() for c in self.risk_exposed_counterparties[:10]
],
"fund_sources": self.fund_sources[:5],
"entity_clusters": [c.to_dict() for c in self.entity_clusters[:10]],
"scam_exposure_count": self.scam_exposure_count,
"sanctioned_exposure_count": self.sanctioned_exposure_count,
"generated_at": self.generated_at,
"errors": self.errors,
}
def summary(self) -> str:
risk_emoji = {
"critical": "🔴 CRITICAL",
"high": "🟠 HIGH",
"medium": "🟡 MEDIUM",
"low": "🔵 LOW",
"none": "✅ CLEAN",
}.get(self.aggregate_risk_level, "⚪ UNKNOWN")
return (
f"{risk_emoji} Counterparties — {self.target_address[:12]}... | "
f"Interactions: {self.total_interactions} | "
f"Unique counterparties: {self.unique_counterparties} | "
f"Net flow: ${self.net_volume_usd:,.0f} | "
f"Scam exposure: {self.scam_exposure_count} | "
f"Sanctioned: {self.sanctioned_exposure_count}"
)
# ── Core Detector ─────────────────────────────────────────────────
class ArkhamCounterparties:
"""Fetches and analyzes entity relationships and money flows."""
def __init__(self, api_key: str = "", cache_ttl: int = CACHE_TTL):
self._api_key = api_key or os.getenv("ARKHAM_API_KEY", "")
self._cache_ttl = cache_ttl
self._local_cache: dict[str, CounterpartyReport] = {}
def _get_cached(self, address: str) -> CounterpartyReport | None:
"""Check cache for existing report."""
cached = self._local_cache.get(address)
if cached:
# Check TTL
if cached.generated_at:
cached_time = datetime.fromisoformat(cached.generated_at).timestamp()
if (datetime.now().timestamp() - cached_time) < self._cache_ttl:
return cached
del self._local_cache[address]
return None
def _set_cache(self, address: str, report: CounterpartyReport):
"""Store report in cache."""
self._local_cache[address] = report
async def analyze(
self,
address: str,
chain: str = "ethereum",
depth: int = 2,
max_transactions: int = MAX_TRANSACTIONS,
) -> CounterpartyReport:
"""
Analyze entity relationships and money flows for an address.
Args:
address: Target wallet address to analyze
chain: Blockchain name (ethereum, solana, bsc, etc.)
depth: How many hops to trace fund flows (1-3)
max_transactions: Max transactions to analyze
Returns:
CounterpartyReport with full relationship analysis
"""
# Check cache first
cached = self._get_cached(address)
if cached:
return cached
report = CounterpartyReport(
target_address=address,
chain=chain,
)
try:
# Fetch transaction history
txs = await self._fetch_transactions(address, chain, max_transactions)
report.total_interactions = len(txs)
if not txs:
report.errors.append("No transactions found for address")
return report
# Extract counterparties
counterparties = self._extract_counterparties(txs, address, chain)
# Remove self from counterparties
counterparties = [c for c in counterparties if c.address.lower() != address.lower()]
report.counterparties = counterparties
report.unique_counterparties = len(counterparties)
# Calculate summary stats
report.total_volume_in_usd = sum(c.total_volume_in_usd for c in counterparties)
report.total_volume_out_usd = sum(c.total_volume_out_usd for c in counterparties)
report.net_volume_usd = (
report.total_volume_in_usd - report.total_volume_out_usd
)
# Sort by volume
report.top_counterparties_by_volume = sorted(
counterparties, key=lambda x: x.total_volume_in_usd + x.total_volume_out_usd, reverse=True
)[:10]
# Identify risk-exposed counterparties
report.risk_exposed_counterparties = [
c for c in counterparties if c.risk_score >= MEDIUM_RISK_THRESHOLD
]
# Track fund sources
report.fund_sources = self._identify_fund_sources(txs, address)[:5]
# Cluster related addresses
report.entity_clusters = self._cluster_addresses(counterparties)[:10]
# Calculate aggregate risk
report.max_risk_score = max((c.risk_score for c in counterparties), default=0.0)
report.scam_exposure_count = sum(
1 for c in counterparties if c.entity_category in ("scam", "sanctioned")
)
report.sanctioned_exposure_count = sum(
1 for c in counterparties if c.entity_category == "sanctioned"
)
if report.max_risk_score >= HIGH_RISK_THRESHOLD:
report.aggregate_risk_level = "critical"
elif report.max_risk_score >= MEDIUM_RISK_THRESHOLD:
report.aggregate_risk_level = "high"
elif report.max_risk_score >= LOW_RISK_THRESHOLD:
report.aggregate_risk_level = "medium"
elif report.risk_exposed_counterparties:
report.aggregate_risk_level = "low"
else:
report.aggregate_risk_level = "none"
except Exception as e:
logger.error(f"Counterparty analysis failed for {address}: {e}")
report.errors.append(str(e))
report.aggregate_risk_level = "error"
self._set_cache(address, report)
return report
async def _fetch_transactions(
self, address: str, chain: str, limit: int
) -> list[dict[str, Any]]:
"""Fetch transaction history from Arkham API or fallback source."""
txs = []
if self._api_key and httpx:
try:
txs = await self._fetch_from_arkham(address, chain, limit)
except Exception as e:
logger.warning(f"Arkham API fetch failed: {e}")
# Fallback: return empty if no API key (would normally use other sources)
return txs
async def _fetch_from_arkham(
self, address: str, chain: str, limit: int
) -> list[dict[str, Any]]:
"""Fetch transactions from Arkham Intelligence API."""
headers = {
"API-Key": self._api_key,
"Content-Type": "application/json",
}
params = {
"address": address,
"chain": chain,
"limit": min(limit, 500),
}
async with httpx.AsyncClient(timeout=30.0) as client:
resp = await client.get(
f"{ARKHAM_API_BASE}/v0/transactions",
headers=headers,
params=params,
)
if resp.status_code == 200:
data = resp.json()
return data.get("transactions", [])
resp.raise_for_status()
return []
def _extract_counterparties(
self, txs: list[dict[str, Any]], target: str, chain: str
) -> list[Counterparty]:
"""Extract and aggregate counterparty data from transactions."""
counterparties: dict[str, Counterparty] = {}
for tx in txs:
# Determine counterparty (the other side of the transaction)
counterparty_addr = self._get_counterparty_address(tx, target)
if not counterparty_addr:
continue
if counterparty_addr not in counterparties:
counterparties[counterparty_addr] = Counterparty(
address=counterparty_addr,
chain=tx.get("chain", chain),
entity_name=tx.get("counterparty", {}).get("name", ""),
entity_category=tx.get("counterparty", {}).get("type", "unknown"),
)
cp = counterparties[counterparty_addr]
# Determine relationship type
cp.relationship_type = self._determine_relationship_type(tx)
# Aggregate volumes
amount = float(tx.get("amount", 0))
usd_value = float(tx.get("usd_price", 0)) * amount
if self._is_incoming(tx, target):
cp.total_volume_in_usd += usd_value
else:
cp.total_volume_out_usd += usd_value
cp.tx_count += 1
cp.net_volume_usd = cp.total_volume_in_usd - cp.total_volume_out_usd
# Update timestamps
ts = int(tx.get("timestamp", 0))
if ts:
if cp.first_interaction == 0 or ts < cp.first_interaction:
cp.first_interaction = ts
if ts > cp.last_interaction:
cp.last_interaction = ts
# Calculate risk
cp.risk_score, cp.risk_level = self._calculate_risk(cp.entity_category)
cp.relationship_strength = min(1.0, cp.tx_count / 10.0)
return list(counterparties.values())
def _get_counterparty_address(
self, tx: dict[str, Any], target: str
) -> str:
"""Get the counterparty address from a transaction."""
target_lower = target.lower()
# Check from/to fields
tx_from = (tx.get("from", "") or "").lower()
tx_to = (tx.get("to", "") or "").lower()
if tx_from and tx_from != target_lower:
return tx_from
if tx_to and tx_to != target_lower:
return tx_to
# Check for counterparty in nested structure
counterparty = tx.get("counterparty", {}).get("address", "")
if counterparty:
return counterparty
return ""
def _is_incoming(self, tx: dict[str, Any], target: str) -> bool:
"""Determine if transaction is incoming to target address."""
target_lower = target.lower()
tx_to = (tx.get("to", "") or "").lower()
return tx_to == target_lower
def _determine_relationship_type(self, tx: dict[str, Any]) -> RelationshipType:
"""Determine the type of relationship from transaction data."""
tx_type = (tx.get("type", "") or "").lower()
category = (tx.get("counterparty", {}).get("type", "") or "").lower()
if "swap" in tx_type or "trade" in tx_type:
return RelationshipType.DIRECT_TRADE
if "fund" in tx_type or "transfer" in tx_type:
return RelationshipType.FUNDING_SOURCE
if category in ("exchange", "cex"):
return RelationshipType.FUNDING_SOURCE
if category in ("contract", "smart_contract"):
return RelationshipType.SMART_CONTRACT
return RelationshipType.DIRECT_TRADE
def _calculate_risk(
self, category: str
) -> tuple[float, str]:
"""Calculate risk score based on entity category."""
cat_low = category.lower()
if cat_low in ("scam", "sanctioned"):
return 90.0, "critical"
if cat_low in ("malicious", "phishing"):
return 75.0, "high"
if cat_low in ("suspicious", "high_risk"):
return 50.0, "medium"
if cat_low in ("rug", "hacker"):
return 80.0, "high"
return 0.0, "none"
def _identify_fund_sources(
self, txs: list[dict[str, Any]], target: str
) -> list[str]:
"""Identify primary sources of funds (both incoming and outgoing)."""
sources = []
for tx in txs[:100]: # Limit analysis
if self._is_incoming(tx, target):
# Incoming: who sent funds
source = self._get_counterparty_address(tx, target)
if source and source not in sources:
sources.append(source)
else:
# Outgoing: where funds went
dest = self._get_counterparty_address(tx, target)
if dest and dest not in sources:
sources.append(dest)
return sources
def _cluster_addresses(
self, counterparties: list[Counterparty]
) -> list[EntityCluster]:
"""Cluster addresses that may be controlled by the same entity."""
clusters: list[EntityCluster] = []
# Group by entity name/hint
by_entity: dict[str, list[Counterparty]] = {}
for cp in counterparties:
key = cp.entity_name or cp.entity_category or "unknown"
if key not in by_entity:
by_entity[key] = []
by_entity[key].append(cp)
for entity_name, addrs in by_entity.items():
if len(addrs) < 2:
continue # Need at least 2 for a cluster
total_vol = sum(a.total_volume_in_usd + a.total_volume_out_usd for a in addrs)
confidence = min(100.0, len(addrs) * 15.0) # More addresses = higher confidence
clusters.append(
EntityCluster(
cluster_id=entity_name[:20].replace(" ", "_").lower(),
addresses=[a.address for a in addrs],
total_volume_usd=total_vol,
entity_hint=entity_name,
confidence=confidence,
)
)
return sorted(clusters, key=lambda x: x.total_volume_usd, reverse=True)
# ── CLI Entry Point ──────────────────────────────────────────────
if __name__ == "__main__":
import asyncio
import sys
if len(sys.argv) < 2:
print("Usage: python -m app.arkham_counterparties <address> [chain]")
sys.exit(1)
addr = sys.argv[1]
chain = sys.argv[2] if len(sys.argv) > 2 else "ethereum"
async def main():
analyzer = ArkhamCounterparties()
report = await analyzer.analyze(addr, chain)
print(report.summary())
print(f"\nTop counterparties by volume:")
for cp in report.top_counterparties_by_volume[:5]:
print(f" - {cp.entity_name or cp.address[:12]}... | ${cp.net_volume_usd:,.0f} net | {cp.tx_count} tx")
asyncio.run(main())

728
backend/app/arkham_entity.py
View file

@ -1,728 +0,0 @@
"""
Arkham Entity Resolver
======================
Map any blockchain address to its real-world owner with multi-source confidence scoring.
Entity types detected:
1. CEX/DEX Wallets — Binance, Coinbase, Kraken, OKX, Uniswap, SushiSwap, Curve
2. DeFi Protocols — Aave, Compound, MakerDAO, Lido, EigenLayer, Ethena
3. Institutional Funds — Grayscale, Pantera, a16z, Multicoin, Paradigm
4. Smart Money / Whales — High-value wallets with repeat profitable trades
5. Bridge / Cross-chain — LayerZero, Wormhole, Stargate, Across
6. Scam / Sanctioned — Lazarus, Ronin exploiter, known bad actors
7. NFT / Gaming — BAYC, Opensea, Blur, Magic Eden wallets
8. Unknown / Uncategorized — Addresses with no known entity label
Competitive advantage:
- Arkham Intelligence paid ($299/mo+) vs our bundled x402 access ($0.10/tool)
- Chainalysis/TRM Labs require enterprise contracts (50K+/yr)
- Our multi-source hybrid approach (Arkham API + local entity DB + heuristic scoring)
means we still return useful results when the API is down
- Confidence scoring (0-100) lets users calibrate trust per use case
- Integrates with existing RMI tooling (wallet_graph, entity_clustering, reputation_score)
Usage:
from app.arkham_entity import ArkhamEntityResolver
resolver = ArkhamEntityResolver()
result = await resolver.resolve("0xBE0eB53FC46b790099138e3d32C721856d41e865")
print(f"Entity: {result.entity_name}")
print(f"Confidence: {result.confidence}/100")
print(f"Category: {result.category}")
for label in result.labels:
print(f" [{label.source}] {label.label}")
CLI:
python3 -m app.arkham_entity 0xBE0eB53FC46b790099138e3d32C721856d41e865
"""
import asyncio
import logging
import os
import re
import sys
from dataclasses import dataclass, field, asdict
from datetime import datetime, timezone
from enum import Enum
from typing import Any
try:
from app.arkham_connector import ArkhamClient, ARKHAM_API_KEY
except ImportError:
ArkhamClient = None # type: ignore
ARKHAM_API_KEY = ""
try:
from app.entity_registry import KNOWN_CEX_WALLETS
except ImportError:
KNOWN_CEX_WALLETS = {}
try:
from app.entity_labeler import EXCHANGES, PROTOCOLS
except ImportError:
EXCHANGES = {}
PROTOCOLS = {}
logger = logging.getLogger(__name__)
# ── Constants ─────────────────────────────────────────────────────────────────
ETH_ADDRESS_RE = re.compile(r"^0x[a-fA-F0-9]{40}$")
SOL_ADDRESS_RE = re.compile(r"^[1-9A-HJ-NP-Za-km-z]{32,44}$")
# Known entity database — manual curation supplementing Arkham data
KNOWN_ENTITIES: dict[str, dict[str, Any]] = {
# Exchanges
"0xBE0eB53FC46b790099138e3d32C721856d41e865": {
"name": "Binance", "category": "exchange", "label": "Binance Hot Wallet 7",
"tags": "cex,hot_wallet,high_volume",
},
"0xF977814e90dA44bFA03b6295A0616a897441aceC": {
"name": "Binance", "category": "exchange", "label": "Binance Hot Wallet 8",
"tags": "cex,hot_wallet,high_volume",
},
"0x28C6c06298d514Db089934071355E5743bf21d60": {
"name": "Binance", "category": "exchange", "label": "Binance Hot Wallet 14",
"tags": "cex,hot_wallet,high_volume",
},
"0x503828976D22510aad0201ac7EC88293211D23Da": {
"name": "Coinbase", "category": "exchange", "label": "Coinbase Hot Wallet 1",
"tags": "cex,hot_wallet",
},
"0xddfAbCdc4D8fFC17086Ea2cBcebe71504184443C": {
"name": "Coinbase", "category": "exchange", "label": "Coinbase Hot Wallet 2",
"tags": "cex,hot_wallet",
},
"0x267be1C1D684F78cb4F6a176C4911b741E4Ffdc0": {
"name": "Kraken", "category": "exchange", "label": "Kraken Hot Wallet",
"tags": "cex,hot_wallet",
},
"0x6cC5F688a315f3dC28A7781717a9A798a59fDA7b": {
"name": "OKX", "category": "exchange", "label": "OKX Hot Wallet",
"tags": "cex,hot_wallet",
},
# DeFi Protocols
"0x7a250d5630b4cf139281983dce37532e7d5c9196": {
"name": "Uniswap V2 Router", "category": "defi", "label": "Uniswap V2 Router",
"tags": "dex,router,swap",
},
"0x7d2768de32b0b8013e039f31fbacf67128c9c3d8": {
"name": "Aave V2 Lending Pool", "category": "defi", "label": "Aave V2 Lending Pool",
"tags": "lending,liquidity,pools",
},
"0xdAC17F958D2ee523a2206206994597C13D831ec7": {
"name": "Tether (USDT)", "category": "token", "label": "Tether USDT Contract",
"tags": "stablecoin,erc20,high_volume",
},
"0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48": {
"name": "Circle (USDC)", "category": "token", "label": "USD Coin Contract",
"tags": "stablecoin,erc20,high_volume",
},
# Known bad actors / scams
"0x1CBd3b2770909D4e10f157cABC84C7264073C9Ec": {
"name": "Lazarus Group", "category": "scam", "label": "Lazarus Group (Sanctioned)",
"tags": "sanctioned,north_korea,exploiter",
},
"0x098B716B8Aaf21512996dC57EB0615e2383E2f96": {
"name": "Ronin Bridge Exploiter", "category": "scam", "label": "Ronin Bridge Exploiter",
"tags": "exploiter,hack,6.2m_eth",
},
}
# Normalize known entity addresses to lowercase
KNOWN_ENTITIES = {k.lower(): v for k, v in KNOWN_ENTITIES.items()}
# Entity category weights for confidence scoring
CATEGORY_WEIGHTS: dict[str, float] = {
"exchange": 0.95,
"defi": 0.90,
"token": 0.90,
"scam": 0.85,
"bridge": 0.85,
"fund": 0.80,
"nft": 0.70,
"whale": 0.60,
"unknown": 0.20,
}
# ── Enums ─────────────────────────────────────────────────────────────────────
class EntityCategory(str, Enum):
"""Top-level entity classification."""
EXCHANGE = "exchange"
DEFI = "defi"
TOKEN = "token"
BRIDGE = "bridge"
FUND = "fund"
WHALE = "whale"
NFT = "nft"
GAMING = "gaming"
SCAM = "scam"
SANCTIONED = "sanctioned"
UNKNOWN = "unknown"
class ResolverSource(str, Enum):
"""Source that provided the entity resolution."""
ARKHAM_API = "arkham_api"
LOCAL_DB = "local_db"
ENTITY_REGISTRY = "entity_registry"
ENTITY_LABELER = "entity_labeler"
HEURISTIC = "heuristic"
# ── Dataclasses ───────────────────────────────────────────────────────────────
@dataclass
class EntityLabel:
"""A single label for an address from a specific source."""
label: str
source: ResolverSource
confidence: float # 0.0 to 1.0
category: str = "unknown"
@dataclass
class ResolvedEntity:
"""Complete entity resolution result for a single address."""
address: str
chain: str
entity_name: str
category: EntityCategory
confidence: float # 0-100
labels: list[EntityLabel] = field(default_factory=list)
tags: list[str] = field(default_factory=list)
related_addresses: list[str] = field(default_factory=list)
source: ResolverSource = ResolverSource.HEURISTIC
arkham_data: dict[str, Any] | None = None
resolved_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())
raw_signals: list[dict[str, Any]] = field(default_factory=list)
def to_dict(self) -> dict[str, Any]:
return {
"address": self.address,
"chain": self.chain,
"entity_name": self.entity_name,
"category": self.category.value if isinstance(self.category, EntityCategory) else self.category,
"confidence": self.confidence,
"labels": [asdict(l) for l in self.labels],
"tags": self.tags,
"source": self.source.value if isinstance(self.source, ResolverSource) else self.source,
"resolved_at": self.resolved_at,
"related_count": len(self.related_addresses),
}
@dataclass
class EntityReport:
"""Full entity resolution report for one or more addresses."""
query_addresses: list[str]
chain: str
entities: list[ResolvedEntity]
summary: dict[str, Any]
generated_at: str = field(default_factory=lambda: datetime.now(timezone.utc).isoformat())
error: str | None = None
def to_dict(self) -> dict[str, Any]:
return {
"query_addresses": self.query_addresses,
"chain": self.chain,
"entities": [e.to_dict() for e in self.entities],
"summary": self.summary,
"generated_at": self.generated_at,
"error": self.error,
}
# ── Entity Resolver ───────────────────────────────────────────────────────────
class ArkhamEntityResolver:
"""
Multi-source entity resolution engine.
Pipeline:
1. Local DB lookup (fastest, always available)
2. Entity Registry lookup (CEX wallets, known protocols)
3. Entity Labeler lookup (advanced protocol detection)
4. Arkham API lookup (requires API key, richest data)
5. Heuristic inference (pattern matching, fallback)
Each source contributes confidence-weighted labels. Final confidence
is aggregated from all sources.
"""
def __init__(self, cache_ttl: int = 300):
self._arkham: ArkhamClient | None = None
self._cache_ttl = cache_ttl
async def _get_arkham(self) -> ArkhamClient | None:
"""Lazy-init Arkham client if API key is available."""
if self._arkham is None and ArkhamClient is not None and bool(ARKHAM_API_KEY):
try:
self._arkham = ArkhamClient(cache_ttl=self._cache_ttl)
except Exception as e:
logger.warning(f"Failed to init Arkham client: {e}")
return self._arkham
async def close(self):
"""Close the underlying Arkham HTTP client."""
if self._arkham is not None:
await self._arkham.close()
self._arkham = None
def _detect_chain(self, address: str) -> str:
"""Heuristically detect chain from address format."""
if ETH_ADDRESS_RE.match(address):
return "ethereum"
if SOL_ADDRESS_RE.match(address):
return "solana"
# Could expand with more chain patterns
return "unknown"
def _lookup_local_db(self, address: str) -> ResolvedEntity | None:
"""Step 1: Check hardcoded known entity database."""
addr = address.lower()
entry = KNOWN_ENTITIES.get(addr)
if not entry:
return None
labels = [
EntityLabel(
label=str(entry.get("label", entry["name"])),
source=ResolverSource.LOCAL_DB,
confidence=0.95,
category=str(entry.get("category", "unknown")),
)
]
tags_raw = entry.get("tags", "")
tags = [t.strip() for t in tags_raw.split(",") if t.strip()]
category = EntityCategory(entry["category"]) if entry["category"] in EntityCategory._value2member_map_ else EntityCategory.UNKNOWN # type: ignore
return ResolvedEntity(
address=address,
chain=self._detect_chain(address),
entity_name=str(entry["name"]),
category=category,
confidence=95.0,
labels=labels,
tags=tags,
source=ResolverSource.LOCAL_DB,
)
def _lookup_entity_registry(self, address: str) -> ResolvedEntity | None:
"""Step 2: Check KNOWN_CEX_WALLETS from entity_registry."""
addr = address.lower()
labels: list[EntityLabel] = []
for exchange_name, wallets in KNOWN_CEX_WALLETS.items():
for w in wallets:
if w.lower() == addr:
labels.append(EntityLabel(
label=f"{exchange_name.capitalize()} Wallet",
source=ResolverSource.ENTITY_REGISTRY,
confidence=0.90,
category="exchange",
))
if not labels:
return None
return ResolvedEntity(
address=address,
chain=self._detect_chain(address),
entity_name=labels[0].label.split(" Wallet")[0],
category=EntityCategory.EXCHANGE,
confidence=90.0,
labels=labels,
tags=["cex", "exchange"],
source=ResolverSource.ENTITY_REGISTRY,
)
def _lookup_entity_labeler(self, address: str) -> ResolvedEntity | None:
"""Step 3: Check EXCHANGES and PROTOCOLS from entity_labeler."""
addr = address.lower()
labels: list[EntityLabel] = []
# Check EXCHANGES dict
for name, addresses in EXCHANGES.items():
for a in addresses:
if isinstance(a, str) and a.lower() == addr:
labels.append(EntityLabel(
label=f"{name.capitalize()} (Labeler)",
source=ResolverSource.ENTITY_LABELER,
confidence=0.85,
category="exchange",
))
# Check PROTOCOLS dict
for name, info in PROTOCOLS.items():
if isinstance(info, dict):
proto_addr = info.get("address", "")
if isinstance(proto_addr, str) and proto_addr.lower() == addr:
labels.append(EntityLabel(
label=f"{name.capitalize()} (Labeler)",
source=ResolverSource.ENTITY_LABELER,
confidence=0.90,
category="defi",
))
if not labels:
return None
top = labels[0]
cat = EntityCategory.EXCHANGE if top.category == "exchange" else EntityCategory.DEFI
return ResolvedEntity(
address=address,
chain=self._detect_chain(address),
entity_name=top.label.split(" (Labeler")[0],
category=cat,
confidence=85.0,
labels=labels,
tags=[top.category],
source=ResolverSource.ENTITY_LABELER,
)
async def _lookup_arkham_api(self, address: str) -> ResolvedEntity | None:
"""Step 4: Query Arkham Intelligence API for entity data."""
arkham = await self._get_arkham()
if not arkham:
return None
try:
data = await arkham.get_entity(address)
if not data or "error" in data:
return None
entity_name = data.get("name", data.get("entity", ""))
if not entity_name:
return None
category_raw = data.get("category", "unknown")
category = EntityCategory.UNKNOWN
if category_raw in EntityCategory._value2member_map_:
category = EntityCategory(category_raw)
labels = [
EntityLabel(
label=data.get("label", entity_name),
source=ResolverSource.ARKHAM_API,
confidence=min(1.0, float(data.get("confidence", 0.85))),
category=category_raw,
)
]
return ResolvedEntity(
address=address,
chain=self._detect_chain(address),
entity_name=entity_name,
category=category,
confidence=min(100.0, float(data.get("confidence", 85)) * 100),
labels=labels,
tags=data.get("tags", []),
source=ResolverSource.ARKHAM_API,
arkham_data=data,
)
except Exception as e:
logger.warning(f"Arkham API lookup failed for {address}: {e}")
return None
def _heuristic_inference(self, address: str) -> ResolvedEntity | None:
"""Step 5: Fallback heuristic — pattern-based entity inference."""
addr = address.lower()
signals: list[dict[str, Any]] = []
labels: list[EntityLabel] = []
tags: list[str] = []
# Check address patterns
if addr.endswith("dead") or addr.endswith("0000"):
signals.append({"type": "burn_pattern", "detail": "Address ends with dead/0000", "weight": 0.3})
labels.append(EntityLabel(
label="Likely Burn Address",
source=ResolverSource.HEURISTIC,
confidence=0.6,
category="burn",
))
tags.append("burn")
if addr[:6] == "0x0000" and len(addr) == 42:
signals.append({"type": "zero_prefix", "detail": "Heavy zero-prefixed address", "weight": 0.2})
# Check if it looks like a contract (common patterns)
# Most contracts start with non-random patterns
if not labels:
return None
confidence = max(0.0, min(100.0, 20.0 + sum(s.get("weight", 0) for s in signals) * 100))
return ResolvedEntity(
address=address,
chain=self._detect_chain(address),
entity_name="Unknown (Heuristic Match)",
category=EntityCategory.UNKNOWN,
confidence=round(confidence, 1),
labels=labels,
tags=tags,
source=ResolverSource.HEURISTIC,
raw_signals=signals,
)
async def resolve(
self,
address: str,
chain: str | None = None,
use_arkham: bool = True,
) -> ResolvedEntity:
"""
Resolve a single blockchain address to an entity.
Pipeline: Local DB → Entity Registry → Entity Labeler → Arkham API → Heuristic
Args:
address: Blockchain address to resolve.
chain: Chain override (auto-detected if None).
use_arkham: Whether to attempt Arkham API lookup.
Returns:
ResolvedEntity with best available data.
"""
if not chain:
chain = self._detect_chain(address)
# Validate address format
input_entity = address.strip()
if not (ETH_ADDRESS_RE.match(input_entity) or SOL_ADDRESS_RE.match(input_entity)):
return ResolvedEntity(
address=address,
chain=chain,
entity_name="Invalid Address",
category=EntityCategory.UNKNOWN,
confidence=0.0,
labels=[],
tags=["invalid"],
source=ResolverSource.HEURISTIC,
)
# Pipeline — stop at first positive match
result = self._lookup_local_db(input_entity)
if result:
result.chain = chain or result.chain
return result
result = self._lookup_entity_registry(input_entity)
if result:
result.chain = chain or result.chain
return result
result = self._lookup_entity_labeler(input_entity)
if result:
result.chain = chain or result.chain
return result
if use_arkham:
result = await self._lookup_arkham_api(input_entity)
if result:
result.chain = chain or result.chain
return result
result = self._heuristic_inference(input_entity)
if result:
result.chain = chain or result.chain
return result
# No match at all
return ResolvedEntity(
address=address,
chain=chain,
entity_name="Unknown",
category=EntityCategory.UNKNOWN,
confidence=0.0,
labels=[],
tags=[],
source=ResolverSource.HEURISTIC,
)
async def resolve_batch(
self,
addresses: list[str],
chain: str | None = None,
use_arkham: bool = True,
) -> list[ResolvedEntity]:
"""
Resolve multiple addresses in sequence.
Args:
addresses: List of blockchain addresses.
chain: Chain override for all addresses.
use_arkham: Whether to attempt Arkham API lookup.
Returns:
List of ResolvedEntity objects.
"""
results: list[ResolvedEntity] = []
for addr in addresses:
result = await self.resolve(addr, chain=chain, use_arkham=use_arkham)
results.append(result)
return results
# ── Aggregation & Reporting ────────────────────────────────────────────────────
async def analyze_addresses(
addresses: list[str],
chain: str | None = None,
use_arkham: bool = True,
) -> EntityReport:
"""
Main entry point: resolve addresses and produce a structured report.
Args:
addresses: One or more blockchain addresses to resolve.
chain: Chain override (auto-detected if None).
use_arkham: Whether to query Arkham API.
Returns:
EntityReport with resolved entities and summary statistics.
"""
if not addresses:
return EntityReport(
query_addresses=[],
chain=chain or "unknown",
entities=[],
summary={"total": 0, "known": 0, "unknown": 0, "avg_confidence": 0.0},
error="No addresses provided",
)
resolver = ArkhamEntityResolver()
try:
if chain is None:
chain = resolver._detect_chain(addresses[0])
entities = await resolver.resolve_batch(addresses, chain=chain, use_arkham=use_arkham)
known = [e for e in entities if e.confidence >= 50.0]
unknown = [e for e in entities if e.confidence < 50.0]
categories: dict[str, int] = {}
for e in entities:
cat = e.category.value if isinstance(e.category, EntityCategory) else str(e.category)
categories[cat] = categories.get(cat, 0) + 1
avg_conf = sum(e.confidence for e in entities) / len(entities) if entities else 0.0
summary = {
"total": len(entities),
"known": len(known),
"unknown": len(unknown),
"avg_confidence": round(avg_conf, 1),
"categories": categories,
"sources_used": list({e.source.value if isinstance(e.source, ResolverSource) else e.source for e in entities}),
}
return EntityReport(
query_addresses=addresses,
chain=chain,
entities=entities,
summary=summary,
)
finally:
await resolver.close()
def format_report(report: EntityReport) -> str:
"""
Format an EntityReport as a human-readable string.
Args:
report: The report to format.
Returns:
Formatted string suitable for CLI or API response.
"""
lines = [
"┌────────────────────────────────────────────────────────────┐",
"│ Arkham Entity Report │",
"├────────────────────────────────────────────────────────────┤",
]
if report.error:
lines.append(f"│ ERROR: {report.error}")
lines.append("└────────────────────────────────────────────────────────────┘")
return "\n".join(lines)
lines.append(f"│ Chain: {report.chain}")
lines.append(f"│ Addresses: {len(report.query_addresses)}")
lines.append(f"│ Generated: {report.generated_at}")
lines.append("├────────────────────────────────────────────────────────────┤")
for i, entity in enumerate(report.entities):
lines.append(f"│ [{i + 1}] {entity.address[:42]}")
lines.append(f"│ Entity: {entity.entity_name}")
cat_str = entity.category.value if isinstance(entity.category, EntityCategory) else str(entity.category)
lines.append(f"│ Category: {cat_str}")
lines.append(f"│ Confidence: {entity.confidence:.0f}/100")
src_str = entity.source.value if isinstance(entity.source, ResolverSource) else str(entity.source)
lines.append(f"│ Source: {src_str}")
if entity.tags:
lines.append(f"│ Tags: {', '.join(entity.tags[:5])}")
if entity.labels:
for label in entity.labels:
lines.append(f"│ Label: [{label.source.value}] {label.label} (conf: {label.confidence:.0%})")
if entity.related_addresses:
lines.append(f"│ Related: {len(entity.related_addresses)} addresses")
if i < len(report.entities) - 1:
lines.append("│" + "─" * 60 + "│")
# Summary footer
lines.append("├────────────────────────────────────────────────────────────┤")
s = report.summary
lines.append(f"│ Summary: {s.get('total', 0)} total, {s.get('known', 0)} known, {s.get('unknown', 0)} unknown")
lines.append(f"│ Avg Confidence: {s.get('avg_confidence', 0):.1f}/100")
cats = s.get("categories", {})
if cats:
cat_str = ", ".join(f"{k}={v}" for k, v in sorted(cats.items()))
lines.append(f"│ Categories: {cat_str}")
lines.append("└────────────────────────────────────────────────────────────┘")
return "\n".join(lines)
# ── CLI Entry Point ────────────────────────────────────────────────────────────
async def main():
"""CLI entry point for entity resolution."""
import argparse
parser = argparse.ArgumentParser(
description="Arkham Entity Resolver — map addresses to real-world entities",
)
parser.add_argument("addresses", nargs="+", help="Blockchain address(es) to resolve")
parser.add_argument("--chain", "-c", default=None, help="Chain override (auto-detect if omitted)")
parser.add_argument("--no-arkham", action="store_true", help="Skip Arkham API lookup")
parser.add_argument("--json", action="store_true", help="Output as JSON")
args = parser.parse_args()
report = await analyze_addresses(
addresses=args.addresses,
chain=args.chain,
use_arkham=not args.no_arkham,
)
if args.json:
import json
print(json.dumps(report.to_dict(), indent=2))
else:
print(format_report(report))
return 0 if not report.error else 1
if __name__ == "__main__":
exit(asyncio.run(main()))

744
backend/app/audit_report_validator.py
View file

@ -1,744 +0,0 @@
"""
Fake Audit Report / Security Review Validator
=============================================
Validates security audit report claims made by tokens/projects.
Detects forged audit reports from Certik, Hacken, SlowMist, and
other major security firms used to promote scam tokens.
Signals detected:
- Forged report IDs/hashes that don't match auditor databases
- Metadata anomalies (dates, auditor name misspellings, logo mismatches)
- Template reuse — similar report text across unrelated projects
- URL/domain analysis for fake audit hosting pages
- Claim-vs-reality discrepancy (audit claims "safe" but contract is malicious)
- Report timeline anomalies (audit after deploy, future dates)
- Standard language detection (generic copy-paste report text)
- Invite-only audit scams (non-existent "Certik Priority" programs)
- Verified badge farming via fake audit blogs
- Known fake auditor wallet addresses deploying tokens
- Cross-referencing with public auditor verified lists
- Dashboard embed scams (iframe fake audit dashboards)
Tier : Premium ($0.08)
Price : 80000 atoms
Endpoint: POST /api/v1/x402-tools/audit_validate
"""
import json
import logging
import re
import time
from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from typing import Any
from urllib.parse import urlparse
import httpx
logger = logging.getLogger(__name__)
# ── Constants ──────────────────────────────────────────────────────
KNOWN_AUDITORS: dict[str, dict[str, Any]] = {
"certik": {
"name": "Certik",
"url": "https://www.certik.com",
"verify_url": "https://www.certik.com/projects/{}",
"known_domains": ["certik.com", "certik.org", "certik.io", "skynet.certik.com"],
},
"hacken": {
"name": "Hacken",
"url": "https://hacken.io",
"verify_url": "https://hacken.io/audits/#{}",
"known_domains": ["hacken.io", "hacken.com", "proofofhacken.io"],
},
"slowmist": {
"name": "SlowMist",
"url": "https://www.slowmist.com",
"verify_url": "https://www.slowmist.com/en/audit-{}.html",
"known_domains": ["slowmist.com", "slowmist.io"],
},
"trailofbits": {
"name": "Trail of Bits",
"url": "https://www.trailofbits.com",
"verify_url": "https://blog.trailofbits.com/?s={}",
"known_domains": ["trailofbits.com"],
},
"consensys": {
"name": "ConsenSys Diligence",
"url": "https://consensys.io/diligence",
"verify_url": "https://consensys.io/diligence/audits/{}",
"known_domains": ["consensys.io", "diligence.consensys.io"],
},
"openzeppelin": {
"name": "OpenZeppelin",
"url": "https://www.openzeppelin.com/security-audits",
"verify_url": "https://blog.openzeppelin.com/{}",
"known_domains": ["openzeppelin.com", "docs.openzeppelin.com"],
},
"quantstamp": {
"name": "Quantstamp",
"url": "https://quantstamp.com",
"verify_url": "https://quantstamp.com/audit/{}",
"known_domains": ["quantstamp.com"],
},
"peckshield": {
"name": "PeckShield",
"url": "https://peckshield.com",
"verify_url": "https://peckshield.com/audit/{}",
"known_domains": ["peckshield.com", "peckshield.io"],
},
"salus": {
"name": "Salus Security",
"url": "https://salusec.io",
"verify_url": "https://salusec.io/audit-reports/{}",
"known_domains": ["salusec.io", "salus.xyz"],
},
"verichains": {
"name": "Verichains",
"url": "https://www.verichains.io",
"verify_url": "https://www.verichains.io/audits/{}",
"known_domains": ["verichains.io"],
},
"solidproof": {
"name": "SolidProof",
"url": "https://solidproof.io",
"verify_url": "https://github.com/solidproof/projects/{}",
"known_domains": ["solidproof.io"],
},
"rugdoc": {
"name": "RugDoc",
"url": "https://rugdoc.io",
"verify_url": "https://rugdoc.io/audit/{}",
"known_domains": ["rugdoc.io"],
},
"goplus": {
"name": "GoPlus Security",
"url": "https://gopluslabs.io",
"verify_url": "https://gopluslabs.io/audits/{}",
"known_domains": ["gopluslabs.io"],
},
}
# Fake auditor names commonly seen in scam tokens
FAKE_AUDITOR_PATTERNS: list[str] = [
"certik", "certick", "certi.guide", "certik.pro", "certik-verify",
"hacken", "hackenpro", "hacken-verify", "hackenaudit",
"slowmist", "slow-mist", "slowmistpro", "slowmist.io",
"solidproof", "solid-proof", "solid.proof",
"audited", "secured by", "verified by",
"pangolin audit", "mythx certified", "mythril scanned",
"goplus", "gopluslabs",
]
# Generic / template report sentences that suggest copy-paste audit
TEMPLATE_PHRASES: list[str] = [
"we have thoroughly reviewed the smart contract",
"no critical vulnerabilities were found",
"the contract appears to be secure",
"our team of experienced auditors",
"this report is provided as is",
"the audit does not guarantee",
"all findings have been resolved",
"the code follows best practices",
"we found no security issues",
"the project has passed our security review",
"this confirms the safety of the contract",
"we have completed the security audit",
"the token contract has been audited",
"no centralization risks found",
"liquidity is locked permanently",
"ownership has been renounced",
]
SUSPICIOUS_TLD_PATTERNS: list[str] = [".xyz", ".top", ".loan", ".click", ".work", ".gq", ".tk", ".ml", ".cf"]
# Maximum allowed difference between deploy and audit claim dates (days)
MAX_AUDIT_ANTEDATE_DAYS = 7 # audit before deploy is suspicious
MAX_AUDIT_POSTDATE_DAYS = 180 # audit more than 6 months after deploy unlikely
SENTENCE_END_RE = re.compile(r"[.!?]\s+")
# ── Risk Levels ──────────────────────────────────────────────────
class AuditRisk(Enum):
CRITICAL = "critical"
HIGH = "high"
MEDIUM = "medium"
LOW = "low"
SAFE = "safe"
class SignalType(Enum):
FORGED_REPORT = "forged_report"
METADATA_ANOMALY = "metadata_anomaly"
TEMPLATE_REUSE = "template_reuse"
SUSPICIOUS_DOMAIN = "suspicious_domain"
TIMELINE_ANOMALY = "timeline_anomaly"
GENERIC_LANGUAGE = "generic_language"
FAKE_AUDITOR_NAME = "fake_auditor_name"
CLAIM_DISCREPANCY = "claim_discrepancy"
# ── Data Models ──────────────────────────────────────────────────
@dataclass
class AuditClaim:
"""A parsed audit claim from token metadata or website."""
auditor_name: str
report_url: str | None = None
report_id: str | None = None
report_date: str | None = None
verified_badge_url: str | None = None
report_text: str | None = None
@dataclass
class AuditSignal:
"""A single detection signal."""
signal_type: SignalType
severity: AuditRisk
description: str
detail: str = ""
@dataclass
class AuditValidationResult:
"""Complete validation result."""
token_address: str
chain: str
risk_level: AuditRisk
risk_score: float # 0.0 (safe) to 1.0 (critical)
signals: list[AuditSignal] = field(default_factory=list)
matched_auditor: str | None = None
verified_on_chain: bool = False
deploy_timestamp: int | None = None
report_timestamp: int | None = None
analysis_time: float = 0.0
error: str | None = None
# ── Validation Engine ────────────────────────────────────────────
class AuditReportValidator:
"""
Validates audit report claims for crypto tokens.
Analyzes:
- Report metadata (dates, auditor names, IDs)
- URL/domain hosting of audit reports
- Report text for template/proof-of-fraud signals
- Timeline consistency (deploy vs audit dates)
- Cross-referencing with known auditor databases
"""
def __init__(self) -> None:
self._known_auditors_lower = {k.lower(): v for k, v in KNOWN_AUDITORS.items()}
# ── Public API ──────────────────────────────────────────────
async def validate(
self,
token_address: str,
chain: str = "ethereum",
claims: list[dict[str, Any]] | None = None,
deploy_timestamp: int | None = None,
) -> AuditValidationResult:
"""
Validate audit claims for a token.
Args:
token_address: The token contract address.
chain: Blockchain name (ethereum, bsc, solana, etc.).
claims: List of audit claim dicts. Each dict may contain:
auditor_name, report_url, report_id, report_date,
verified_badge_url, report_text.
deploy_timestamp: Unix timestamp of token deployment (optional).
Returns:
AuditValidationResult with findings.
"""
start = time.time()
result = AuditValidationResult(
token_address=token_address,
chain=chain,
risk_level=AuditRisk.SAFE,
risk_score=0.0,
deploy_timestamp=deploy_timestamp,
)
if not claims:
# No claims means no audit to validate
result.analysis_time = time.time() - start
return result
parsed_claims = [AuditClaim(**c) if isinstance(c, dict) else c for c in claims]
for claim in parsed_claims:
signals = await self._validate_claim(claim, deploy_timestamp)
result.signals.extend(signals)
# Score aggregation
await self._score_result(result)
result.analysis_time = time.time() - start
return result
# ── Validation Methods ──────────────────────────────────────
async def _validate_claim(
self,
claim: AuditClaim,
deploy_ts: int | None,
) -> list[AuditSignal]:
"""Run all validation checks on a single claim."""
signals: list[AuditSignal] = []
# Signal 1: Fake auditor name detection
signals.extend(self._check_auditor_name(claim.auditor_name))
# Signal 2: Suspicious domain / URL analysis
if claim.report_url:
signals.extend(self._check_report_url(claim.report_url))
# Signal 3: Badge URL analysis
if claim.verified_badge_url:
signals.extend(self._check_badge_url(claim.verified_badge_url))
# Signal 4: Report ID validation
if claim.report_id:
signals.extend(self._check_report_id(claim.report_id, claim.auditor_name))
# Signal 5: Timeline analysis (deploy vs audit date)
if claim.report_date:
signals.extend(self._check_timeline(claim.report_date, deploy_ts))
# Signal 6: Template / generic language detection
if claim.report_text:
signals.extend(self._check_report_text(claim.report_text))
return signals
def _check_auditor_name(self, auditor_name: str) -> list[AuditSignal]:
"""Signal 1: Detect fake/misspelled auditor names."""
signals: list[AuditSignal] = []
name_lower = auditor_name.lower().strip()
# Check for misspellings of known auditors
for known_key, known_info in self._known_auditors_lower.items():
known_name_lower = known_info["name"].lower()
# Levenshtein-like: check character overlap and common misspellings
if self._is_fake_auditor_misspelling(name_lower, known_key, known_name_lower):
signals.append(AuditSignal(
signal_type=SignalType.FAKE_AUDITOR_NAME,
severity=AuditRisk.HIGH,
description=f"Suspicious auditor name resembling '{known_info['name']}'",
detail=f"Claimed: '{auditor_name}' — possible impersonation of {known_info['name']}",
))
# Check for known scam patterns in auditor name
for pattern in FAKE_AUDITOR_PATTERNS:
if pattern in name_lower and name_lower != pattern:
signals.append(AuditSignal(
signal_type=SignalType.FAKE_AUDITOR_NAME,
severity=AuditRisk.MEDIUM,
description=f"Auditor name contains known scam pattern keyword",
detail=f"Pattern matched: '{pattern}' in '{auditor_name}'",
))
break
# Check for generic sounding auditor names
generic_indicators = ["audit", "security", "verified", "certified", "safe", "guard", "labs", "consulting"]
generic_count = sum(1 for g in generic_indicators if g in name_lower)
if generic_count >= 2 and not any(k in name_lower for k in self._known_auditors_lower):
signals.append(AuditSignal(
signal_type=SignalType.FAKE_AUDITOR_NAME,
severity=AuditRisk.MEDIUM,
description="Generic-sounding auditor name not matching known firms",
detail=f"Name '{auditor_name}' sounds generic with {generic_count} generic indicators",
))
return signals
def _normalize_url(self, url: str) -> str:
"""Normalize URL: lowercase scheme+host, strip fragments/trailing junk."""
try:
parsed = urlparse(url.lower().strip())
# Reconstruct without fragment
clean = f"{parsed.scheme}://{parsed.netloc}{parsed.path}"
if parsed.query:
clean += f"?{parsed.query}"
return clean
except Exception:
return url.lower().strip()
def _extract_domain(self, url: str) -> str:
"""Extract the full hostname from a URL."""
try:
parsed = urlparse(url.lower().strip())
return parsed.netloc or url.split("://")[-1].split("/")[0].lower()
except Exception:
return url.lower().strip()
def _check_report_url(self, url: str) -> list[AuditSignal]:
"""Signal 2: Analyze report hosting URL for suspicion."""
signals: list[AuditSignal] = []
normalized = self._normalize_url(url)
url_lower = normalized
# Check if URL uses a suspicious TLD
domain = self._extract_domain(url)
for tld in SUSPICIOUS_TLD_PATTERNS:
if domain.endswith(tld):
signals.append(AuditSignal(
signal_type=SignalType.SUSPICIOUS_DOMAIN,
severity=AuditRisk.MEDIUM,
description=f"Audit report hosted on suspicious TLD '{tld}'",
detail=f"Domain: {domain}",
))
# Check if URL impersonates a known auditor domain
# Only flag if NONE of the auditor's official domains match
for known_key, known_info in self._known_auditors_lower.items():
official_domains = known_info.get("known_domains", [])
# Check if any official domain matches the URL domain
matches_official = any(
domain == od or domain.endswith("." + od)
for od in official_domains
)
if not matches_official:
# Check if the URL contains the base auditor name
base_names = {d.split(".")[-2] if len(d.split(".")) > 1 else d for d in official_domains}
for base_name in base_names:
if base_name in domain and len(base_name) >= 4:
signals.append(AuditSignal(
signal_type=SignalType.SUSPICIOUS_DOMAIN,
severity=AuditRisk.HIGH,
description=f"URL possibly impersonates {known_info['name']} domain",
detail=f"Domain '{domain}' contains '{base_name}' but doesn't match official domains",
))
# Check for PDF/image hosting on free file services (should be on official auditor site)
free_hosting_patterns = [
"drive.google.com", "docs.google.com", "dropbox.com",
"github.io", "githubusercontent.com", "ipfs.io",
"cdn.discord", "telegra.ph", "imgur.com",
"docsend.com", "scribd.com", "docdroid.net",
]
for pattern in free_hosting_patterns:
if pattern in url_lower:
signals.append(AuditSignal(
signal_type=SignalType.SUSPICIOUS_DOMAIN,
severity=AuditRisk.MEDIUM,
description=f"Audit report hosted on free file service ({pattern})",
detail=f"Legitimate audits are hosted on the auditor's official domain",
))
# Check for suspicious subdomain patterns
suspicious_subdomains = ["verify", "audit", "secure", "check", "certify", "validate"]
domain_prefix = domain.split(".")[0].lower()
if domain_prefix in suspicious_subdomains:
signals.append(AuditSignal(
signal_type=SignalType.SUSPICIOUS_DOMAIN,
severity=AuditRisk.LOW,
description=f"URL uses suspicious subdomain '{domain_prefix}'",
detail=f"Domain: {domain}",
))
return signals
def _check_badge_url(self, url: str) -> list[AuditSignal]:
"""Signal 3: Validate 'verified' badge embed URLs."""
signals: list[AuditSignal] = []
url_lower = url.lower()
# Legitimate badge URLs should come from the auditor's domain
found_legitimate = False
for known_key, known_info in self._known_auditors_lower.items():
for known_domain in known_info.get("known_domains", []):
domain_part = known_domain.split("://")[-1].split("/")[0].lower()
if domain_part in url_lower:
found_legitimate = True
break
if not found_legitimate:
# Check if it's an iframe embed or data URL (common in fake badges)
if url_lower.startswith("data:") or "iframe" in url_lower or "embed" in url_lower:
signals.append(AuditSignal(
signal_type=SignalType.FORGED_REPORT,
severity=AuditRisk.HIGH,
description="Verified badge uses data URI or iframe embed instead of trusted domain",
detail=f"Badge URL: {url[:100]}",
))
elif not url_lower.startswith("http"):
signals.append(AuditSignal(
signal_type=SignalType.METADATA_ANOMALY,
severity=AuditRisk.LOW,
description="Unusual badge URL format",
detail=f"Badge URL: {url}",
))
return signals
def _check_report_id(self, report_id: str, auditor_name: str) -> list[AuditSignal]:
"""Signal 4: Validate report ID format against known auditor patterns."""
signals: list[AuditSignal] = []
name_lower = auditor_name.lower().strip()
# Certik report IDs are typically: "Certik-${project-name}-${number}" or UUID
# Hacken: alphanumeric IDs or UUIDs
# SlowMist: date-prefixed IDs
# Check for obviously fake UUIDs/invalid formats
for known_key, known_info in self._known_auditors_lower.items():
if known_key in name_lower:
# Certik IDs should be findable on certik.com
if known_key == "certik" and len(report_id) < 5:
signals.append(AuditSignal(
signal_type=SignalType.METADATA_ANOMALY,
severity=AuditRisk.HIGH,
description=f"Unusually short Certik report ID: '{report_id}'",
detail="Legitimate Certik reports have longer project-specific IDs",
))
# Generic check: report IDs with only digits are suspicious
if report_id.isdigit() and len(report_id) > 8:
signals.append(AuditSignal(
signal_type=SignalType.METADATA_ANOMALY,
severity=AuditRisk.LOW,
description="Report ID is an all-numeric string",
detail="May be auto-generated instead of auditor-issued",
))
return signals
def _check_timeline(self, report_date_str: str, deploy_ts: int | None) -> list[AuditSignal]:
"""Signal 5: Check if audit timeline makes sense."""
signals: list[AuditSignal] = []
try:
# Try multiple date formats
report_ts = self._parse_date(report_date_str)
if report_ts is None:
signals.append(AuditSignal(
signal_type=SignalType.METADATA_ANOMALY,
severity=AuditRisk.LOW,
description="Could not parse audit report date",
detail=f"Date string: '{report_date_str}'",
))
return signals
report_dt = datetime.fromtimestamp(report_ts, tz=timezone.utc)
# Future dates (report dated in future)
now_ts = int(time.time())
if report_ts > now_ts + 86400: # More than 1 day in the future
signals.append(AuditSignal(
signal_type=SignalType.TIMELINE_ANOMALY,
severity=AuditRisk.HIGH,
description="Audit report dated in the future",
detail=f"Report date: {report_dt.date()}. Current: {datetime.now(timezone.utc).date()}",
))
# Still check deploy-timeline if available
if deploy_ts is not None:
self._add_deploy_timeline_signal(report_ts, deploy_ts, report_dt, signals)
return signals
# Deploy-timestamp-dependent checks
if deploy_ts is not None:
self._add_deploy_timeline_signal(report_ts, deploy_ts, report_dt, signals)
except (ValueError, OverflowError) as e:
signals.append(AuditSignal(
signal_type=SignalType.METADATA_ANOMALY,
severity=AuditRisk.LOW,
description=f"Date parsing error: {e}",
detail=f"Date string: '{report_date_str}'",
))
return signals
def _add_deploy_timeline_signal(
self, report_ts: int, deploy_ts: int,
report_dt: datetime, signals: list[AuditSignal],
) -> None:
"""Check deploy vs audit date consistency."""
deploy_dt = datetime.fromtimestamp(deploy_ts, tz=timezone.utc)
diff_days = (report_dt - deploy_dt).days
# Audit before deploy (or impossible future date)
if diff_days < -MAX_AUDIT_ANTEDATE_DAYS:
signals.append(AuditSignal(
signal_type=SignalType.TIMELINE_ANOMALY,
severity=AuditRisk.HIGH,
description=f"Audit report dated {-diff_days} days BEFORE token deployment",
detail=f"Deploy: {deploy_dt.date()}, Audit: {report_dt.date()}. Impossible for a pre-launch audit.",
))
elif diff_days < 0:
signals.append(AuditSignal(
signal_type=SignalType.TIMELINE_ANOMALY,
severity=AuditRisk.LOW,
description=f"Audit report slightly before deployment ({-diff_days} days)",
detail="Possible if audit was performed before deploy, verify dates carefully.",
))
# Audit too long after deploy
if diff_days > MAX_AUDIT_POSTDATE_DAYS:
signals.append(AuditSignal(
signal_type=SignalType.TIMELINE_ANOMALY,
severity=AuditRisk.MEDIUM,
description=f"Audit report dated {diff_days} days AFTER deployment",
detail=f"Late audits may indicate the audit was procured after scam launch.",
))
def _check_report_text(self, text: str) -> list[AuditSignal]:
"""Signal 6: Analyze report text for template/generic language."""
signals: list[AuditSignal] = []
# Count template phrases
text_lower = text.lower()
phrase_matches = [p for p in TEMPLATE_PHRASES if p in text_lower]
if len(phrase_matches) >= 4:
signals.append(AuditSignal(
signal_type=SignalType.TEMPLATE_REUSE,
severity=AuditRisk.HIGH,
description=f"Report text contains {len(phrase_matches)} template/generic phrases",
detail=f"Matched phrases: {', '.join(phrase_matches[:5])}",
))
elif len(phrase_matches) >= 2:
signals.append(AuditSignal(
signal_type=SignalType.GENERIC_LANGUAGE,
severity=AuditRisk.MEDIUM,
description=f"Report text contains {len(phrase_matches)} generic phrases",
detail=f"May indicate copy-paste/fake report. Matched: {', '.join(phrase_matches[:3])}",
))
# Check for suspiciously short report text
word_count = len(text.split())
if word_count < 50 and word_count > 0:
signals.append(AuditSignal(
signal_type=SignalType.GENERIC_LANGUAGE,
severity=AuditRisk.MEDIUM,
description=f"Report text is surprisingly short ({word_count} words)",
detail="Legitimate audit reports are typically 5-50+ pages long",
))
# Check for missing technical terms that should be in a real audit
missing_terms: list[str] = []
technical_terms = [
"reentrancy", "overflow", "access control", "front-running",
"timestamp dependence", "tx.origin", "gas limit",
"integer overflow", "logic flaw", "centralization risk",
]
for term in technical_terms:
if term not in text_lower:
missing_terms.append(term)
if len(missing_terms) >= 8 and word_count > 100:
signals.append(AuditSignal(
signal_type=SignalType.GENERIC_LANGUAGE,
severity=AuditRisk.LOW,
description=f"Report lacks standard security terminology ({len(missing_terms)}/{len(technical_terms)} terms missing)",
detail=f"Missing: {', '.join(missing_terms[:5])}",
))
return signals
# ── Scoring ──────────────────────────────────────────────────
async def _score_result(self, result: AuditValidationResult) -> None:
"""Compute aggregate risk score from signals."""
if not result.signals:
result.risk_level = AuditRisk.SAFE
result.risk_score = 0.0
return
severity_scores = {
AuditRisk.CRITICAL: 1.0,
AuditRisk.HIGH: 0.7,
AuditRisk.MEDIUM: 0.4,
AuditRisk.LOW: 0.15,
}
total = 0.0
count = len(result.signals)
for signal in result.signals:
total += severity_scores.get(signal.severity, 0.1)
# Base score is average severity
base_score = total / max(count, 1)
# Boost for multiple signals
multiplier = min(1.0 + (count - 1) * 0.15, 1.5)
# Boost for critical signals
critical_count = sum(1 for s in result.signals if s.severity == AuditRisk.CRITICAL)
multiplier += critical_count * 0.2
result.risk_score = round(min(base_score * multiplier, 1.0), 4)
# Map score to risk level
if result.risk_score >= 0.7:
result.risk_level = AuditRisk.CRITICAL
elif result.risk_score >= 0.45:
result.risk_level = AuditRisk.HIGH
elif result.risk_score >= 0.2:
result.risk_level = AuditRisk.MEDIUM
elif result.risk_score > 0:
result.risk_level = AuditRisk.LOW
else:
result.risk_level = AuditRisk.SAFE
# ── Helpers ──────────────────────────────────────────────────
def _is_fake_auditor_misspelling(self, name: str, known_key: str, known_name: str) -> bool:
"""Check if name is a likely misspelling/impersonation of a known auditor."""
if name == known_name or name == known_key:
return False
# Check character-level similarity (simple approach)
known_key_len = len(known_key)
if known_key_len >= 4 and known_key in name:
# If the known key is a substring but not the whole thing: impersonation
remaining = name.replace(known_key, "").strip()
if remaining in {"", "-", "_", "."}:
return False # exact match after stripping
# It's a misspelling variant like "certikpro", "hackenverify"
return True
# Check for common character swaps/typos
if known_key_len >= 4 and len(name) >= 4:
# Simple edit distance check - if they share first 3 chars and last 2
if name[:3] == known_key[:3] and name[-2:] == known_key[-2:]:
if name != known_key:
return True
return False
def _parse_date(self, date_str: str) -> int | None:
"""Parse various date formats into a Unix timestamp."""
formats = [
"%Y-%m-%d",
"%Y/%m/%d",
"%d/%m/%Y",
"%m/%d/%Y",
"%d %B %Y",
"%B %d, %Y",
"%d %b %Y",
"%b %d, %Y",
"%Y-%m-%dT%H:%M:%S",
"%Y-%m-%dT%H:%M:%SZ",
"%Y-%m-%d %H:%M:%S",
]
date_str_clean = date_str.strip().strip('"').strip("'")
for fmt in formats:
try:
return int(datetime.strptime(date_str_clean, fmt).timestamp())
except ValueError:
continue
return None

876
backend/app/bundler_detect.py
View file

@ -1,876 +0,0 @@
"""
Supply Manipulation / Bundler Detector
=======================================
Detects bundled token launches where insiders control disproportionate
supply through sniper-controlled wallet distributions.
Signals detected:
- Bundled initial buys (multiple wallets funded from same source,
buying within same block/seconds)
- Supply concentration across linked wallets (top holders controlled
by same entity)
- Fund flow analysis (same funding source → multiple snipers)
- TIMEO (This Is My Eyes Only) token distribution patterns
- Sniper cluster detection (wallets that only buy this token)
- Launch timing anomalies (coordinated buys in first blocks)
- Holder overlap with known bundler addresses
- Supply distribution entropy analysis
Tier : Premium ($0.08)
Price : 80000 atoms
Endpoint: POST /api/v1/x402-tools/bundler_detect
"""
import logging
import math
import os
import re
import time
from dataclasses import dataclass, field
from enum import Enum
from typing import Any
import httpx
logger = logging.getLogger(__name__)
# ── Constants ──────────────────────────────────────────────────────
SOLANA_ADDR_RE = re.compile(r"^[1-9A-HJ-NP-Za-km-z]{32,44}$")
EVM_ADDR_RE = re.compile(r"^0x[a-fA-F0-9]{40}$")
EVM_CHAINS = frozenset(
{
"ethereum",
"bsc",
"polygon",
"arbitrum",
"optimism",
"avalanche",
"base",
"fantom",
"linea",
"zksync",
"scroll",
"mantle",
}
)
SUPPORTED_CHAINS = [*EVM_CHAINS, "solana"]
# DEX API endpoints
DEXSCREENER_API = "https://api.dexscreener.com/latest/dex"
# Free Solana RPC for account info
SOLANA_RPC = "https://api.mainnet-beta.solana.com"
# Birdeye public API (no key needed for basic queries)
BIRDEYE_PUBLIC = "https://public-api.birdeye.so"
# Known bundler wallet addresses (publicly flagged on-chain)
KNOWN_BUNDLER_SEEDS: set[str] = set()
# ── Risk Levels ──────────────────────────────────────────────────
class BundlerRisk(Enum):
CRITICAL = "critical"
HIGH = "high"
MEDIUM = "medium"
LOW = "low"
NONE = "none"
# ── Data Models ──────────────────────────────────────────────────
@dataclass
class BundledBuy:
"""A single suspicious buy event identified as potentially bundled."""
wallet: str
amount_usd: float
buy_block: int
buy_timestamp: float
tx_hash: str = ""
funding_source: str = ""
is_sniper: bool = False
def to_dict(self) -> dict[str, Any]:
return {
"wallet": self.wallet,
"amount_usd": round(self.amount_usd, 2),
"buy_block": self.buy_block,
"buy_timestamp": self.buy_timestamp,
"tx_hash": self.tx_hash,
"funding_source": self.funding_source,
"is_sniper": self.is_sniper,
}
@dataclass
class HolderCluster:
"""A cluster of wallets suspected to be controlled by one entity."""
wallets: list[str]
total_supply_pct: float
funding_overlap_score: float # 0-1, how much funding sources overlap
buy_time_similarity: float # 0-1, how clustered buys were in time
common_funding_source: str = ""
def to_dict(self) -> dict[str, Any]:
return {
"wallet_count": len(self.wallets),
"wallets": self.wallets[:20], # cap at 20 in output
"total_supply_pct": round(self.total_supply_pct, 2),
"funding_overlap_score": round(self.funding_overlap_score, 3),
"buy_time_similarity": round(self.buy_time_similarity, 3),
"common_funding_source": self.common_funding_source,
}
@dataclass
class BundlerReport:
"""Full supply manipulation analysis result."""
token_address: str
chain: str
name: str = ""
symbol: str = ""
# Core scores (0-100)
bundler_score: float = 0.0
supply_concentration_score: float = 0.0
sniper_cluster_score: float = 0.0
launch_timing_anomaly_score: float = 0.0
fund_flow_risk_score: float = 0.0
# Findings
suspected_bundled_buys: list[BundledBuy] = field(default_factory=list)
holder_clusters: list[HolderCluster] = field(default_factory=list)
top_10_holder_concentration: float = 0.0
dev_hold_pct: float = 0.0
unique_buyers_first_block: int = 0
total_buys_first_blocks: int = 0
buys_from_same_funding: int = 0
estimated_unique_entities: int = 0
risk_label: str = "none"
errors: list[str] = field(default_factory=list)
raw: dict[str, Any] = field(default_factory=dict)
def to_dict(self) -> dict[str, Any]:
return {
"token_address": self.token_address,
"chain": self.chain,
"name": self.name,
"symbol": self.symbol,
"bundler_score": round(self.bundler_score, 1),
"risk_label": self.risk_label,
"signals": {
"supply_concentration": round(self.supply_concentration_score, 1),
"sniper_cluster": round(self.sniper_cluster_score, 1),
"launch_timing_anomaly": round(self.launch_timing_anomaly_score, 1),
"fund_flow_risk": round(self.fund_flow_risk_score, 1),
},
"suspected_bundled_buys": [b.to_dict() for b in self.suspected_bundled_buys[:50]],
"holder_clusters": [c.to_dict() for c in self.holder_clusters[:10]],
"top_10_holder_concentration": round(self.top_10_holder_concentration, 2),
"dev_hold_pct": round(self.dev_hold_pct, 2),
"unique_buyers_first_block": self.unique_buyers_first_block,
"total_buys_first_blocks": self.total_buys_first_blocks,
"buys_from_same_funding": self.buys_from_same_funding,
"estimated_unique_entities": self.estimated_unique_entities,
}
def summary(self) -> str:
flags = []
if self.top_10_holder_concentration > 80:
flags.append(f"top10hld:{self.top_10_holder_concentration:.0f}%")
if self.buys_from_same_funding > 3:
flags.append(f"shared_fund:{self.buys_from_same_funding}x")
if self.suspected_bundled_buys:
flags.append(f"bundled:{len(self.suspected_bundled_buys)}buys")
if self.holder_clusters:
total_cluster_pct = sum(c.total_supply_pct for c in self.holder_clusters)
flags.append(f"clustered:{total_cluster_pct:.0f}%")
flag_str = f" [{', '.join(flags)}]" if flags else ""
return (
f"[{self.risk_label.upper()}] {self.token_address[:14]}... "
f"({self.name}/{self.symbol}) — "
f"Bundler score: {self.bundler_score:.0f}/100 | "
f"{len(self.holder_clusters)} clusters | "
f"{self.estimated_unique_entities} entities estimated"
f"{flag_str}"
)
# ── Scoring Helpers ──────────────────────────────────────────────
def _gini_coefficient(values: list[float]) -> float:
"""Compute Gini coefficient for supply distribution (0=equal, 1=max concentration)."""
if not values:
return 0.0
sorted_vals = sorted(values)
n = len(sorted_vals)
cumulative = 0.0
for i, v in enumerate(sorted_vals):
cumulative += (i + 1) * v
gini = (2 * cumulative) / (n * sum(sorted_vals)) - (n + 1) / n
return max(0.0, min(gini, 1.0))
def _entropy(values: list[float]) -> float:
"""Shannon entropy of a distribution (lower = more concentrated).
Returns normalized [0, 1] where 1 = perfectly uniform, 0 = fully concentrated.
"""
total = sum(values)
if total <= 0:
return 0.0
n = len(values)
if n <= 1:
return 1.0 # Single bin = trivially uniform
raw = 0.0
for v in values:
p = v / total
if p > 0:
raw -= p * math.log2(p)
max_entropy = math.log2(n)
return raw / max_entropy if max_entropy > 0 else 0.0
def _time_cluster_similarity(timestamps: list[float]) -> float:
"""Score how tightly clustered timestamps are (0=spread, 1=all at once)."""
if len(timestamps) < 2:
return 0.0
min_ts = min(timestamps)
max_ts = max(timestamps)
span = max_ts - min_ts
if span == 0:
return 1.0
# If all buys happened within 60 seconds, high similarity
if span <= 60:
return 1.0 - (span / 60) * 0.5 # 0.5-1.0
# If within 5 minutes, medium
if span <= 300:
return 0.5 - (span - 60) / (300 - 60) * 0.3 # 0.2-0.5
return max(0.0, 0.2 - (span - 300) / 3600)
def _funding_overlap(funding_sources: list[str]) -> float:
"""Score how many wallets share the same funding source (0-1)."""
if not funding_sources:
return 0.0
total = len(funding_sources)
if total < 2:
return 0.0
# Count how many share a source with at least one other
from collections import Counter
source_counts = Counter(funding_sources)
shared = sum(c for c in source_counts.values() if c > 1)
return shared / total
def _label_risk(score: float) -> str:
if score >= 75:
return "critical"
if score >= 50:
return "high"
if score >= 25:
return "medium"
if score > 0:
return "low"
return "none"
# ── Core Detector ────────────────────────────────────────────────
class BundlerDetector:
"""Main detector for bundled/supply-manipulated token launches."""
def __init__(self, http_timeout: float = 15.0):
self.http = httpx.AsyncClient(timeout=http_timeout)
self._birdeye_api_key = os.environ.get("BIRDEYE_API_KEY", "")
async def close(self):
await self.http.aclose()
# ── Public API ──────────────────────────────────────────────
async def scan(self, address: str, chain: str) -> BundlerReport:
"""Full supply manipulation analysis for a token."""
if not self._validate_address(address, chain):
return BundlerReport(
token_address=address,
chain=chain,
errors=[f"Invalid address format for chain: {chain}"],
risk_label="error",
)
chain = chain.lower()
if chain not in SUPPORTED_CHAINS:
return BundlerReport(
token_address=address,
chain=chain,
errors=[f"Unsupported chain: {chain}"],
risk_label="error",
)
report = BundlerReport(token_address=address, chain=chain)
try:
# 1. Fetch token metadata and pair info
metadata = await self._fetch_metadata(address, chain)
report.name = metadata.get("name", "Unknown")
report.symbol = metadata.get("symbol", "UNKNOWN")
report.raw["metadata"] = metadata
# 2. Fetch holder data
holders = await self._fetch_holders(address, chain)
report.raw["holders_raw"] = holders
if not holders:
report.errors.append("No holder data available")
report.risk_label = "error"
return report
# 3. Compute supply concentration
top10_pct = self._compute_top_holder_pct(holders, 10)
report.top_10_holder_concentration = top10_pct
report.dev_hold_pct = self._extract_dev_hold_pct(holders, metadata)
# 4. Fetch and analyze buys for bundling patterns
buys = await self._fetch_buys(address, chain)
report.raw["buys_raw"] = buys
# 5. Detect bundled buys (same funding source, same block)
bundled_buys, buys_from_same_funding = self._detect_bundled_buys(buys)
report.suspected_bundled_buys = bundled_buys
report.buys_from_same_funding = buys_from_same_funding
# 6. Analyze launch timing
timing_info = self._analyze_launch_timing(buys)
report.unique_buyers_first_block = timing_info["unique_buyers_first_block"]
report.total_buys_first_blocks = timing_info["total_buys_first_blocks"]
# 7. Cluster wallets by funding source and timing
clusters = self._cluster_wallets(buys, holders)
report.holder_clusters = clusters
# 8. Estimate unique entities
report.estimated_unique_entities = self._estimate_entities(holders, clusters, len(bundled_buys))
# 9. Compute all scores
report.supply_concentration_score = self._score_supply_concentration(holders, top10_pct)
report.sniper_cluster_score = self._score_sniper_clusters(clusters, bundled_buys)
report.launch_timing_anomaly_score = self._score_launch_timing(timing_info, buys, holders)
report.fund_flow_risk_score = self._score_fund_flow(bundled_buys, buys_from_same_funding, clusters)
# 10. Composite bundler score
report.bundler_score = self._compute_bundler_score(report)
report.risk_label = _label_risk(report.bundler_score)
except Exception as e:
logger.error(f"Bundler scan error for {address}: {e}")
report.errors.append(str(e))
report.risk_label = "error"
return report
async def quick_check(self, address: str, chain: str) -> dict[str, Any]:
"""Quick supply concentration check — holder data only."""
if not self._validate_address(address, chain):
return {"error": f"Invalid address for chain {chain}"}
chain = chain.lower()
metadata = await self._fetch_metadata(address, chain)
holders = await self._fetch_holders(address, chain)
if not holders:
return {
"address": address,
"chain": chain,
"name": metadata.get("name", ""),
"symbol": metadata.get("symbol", ""),
"error": "No holder data available",
}
top10 = self._compute_top_holder_pct(holders, 10)
gini = _gini_coefficient([h.get("percentage", 0) for h in holders[:100]])
score = 0.0
if top10 > 80:
score += 40
elif top10 > 60:
score += 25
if gini > 0.8:
score += 30
elif gini > 0.6:
score += 15
return {
"address": address,
"chain": chain,
"name": metadata.get("name", ""),
"symbol": metadata.get("symbol", ""),
"supply_concentration_score": min(score, 100),
"risk_label": _label_risk(min(score, 100)),
"top_10_holder_pct": round(top10, 2),
"gini_coefficient": round(gini, 3),
}
# ── Validation ──────────────────────────────────────────────
def _validate_address(self, address: str, chain: str) -> bool:
chain = chain.lower()
if chain == "solana":
return bool(SOLANA_ADDR_RE.match(address))
if chain in EVM_CHAINS:
return bool(EVM_ADDR_RE.match(address))
return bool(EVM_ADDR_RE.match(address) or SOLANA_ADDR_RE.match(address))
# ── Data Fetching ───────────────────────────────────────────
async def _fetch_metadata(self, address: str, chain: str) -> dict[str, Any]:
"""Fetch token metadata from DexScreener."""
try:
url = f"{DEXSCREENER_API}/tokens/{address}"
resp = await self.http.get(url, timeout=10)
if resp.status_code != 200:
return {}
data = resp.json()
pairs = data.get("pairs", [])
if not pairs:
return {}
pair = pairs[0]
return {
"name": pair.get("baseToken", {}).get("name", ""),
"symbol": pair.get("baseToken", {}).get("symbol", ""),
"decimals": pair.get("baseToken", {}).get("decimals"),
"price_usd": pair.get("priceUsd", ""),
"liquidity_usd": pair.get("liquidity", {}).get("usd", 0),
"fdv": pair.get("fdv", 0),
"pair_address": pair.get("pairAddress", ""),
"dex": pair.get("dexId", ""),
"url": pair.get("url", ""),
"social": {
"twitter": pair.get("info", {}).get("twitter", ""),
"website": pair.get("info", {}).get("website", ""),
"telegram": pair.get("info", {}).get("telegram", ""),
},
"creation_block": None, # May not be available
}
except Exception as e:
logger.debug(f"Metadata fetch error: {e}")
return {}
async def _fetch_holders(self, address: str, chain: str) -> list[dict[str, Any]]:
"""Fetch top holders from Birdeye public API or Solscan."""
try:
if chain == "solana":
return await self._fetch_solana_holders(address)
# EVM chains — try Birdeye first
return await self._fetch_evm_holders(address, chain)
except Exception as e:
logger.debug(f"Holder fetch error: {e}")
return []
async def _fetch_solana_holders(self, address: str) -> list[dict[str, Any]]:
"""Fetch Solana token holders via Birdeye public API."""
try:
url = f"{BIRDEYE_PUBLIC}/defi/holder/tokenlist?tokenAddress={address}&limit=100"
headers = {"Accept": "application/json"}
if self._birdeye_api_key:
headers["X-API-KEY"] = self._birdeye_api_key
resp = await self.http.get(url, headers=headers, timeout=10)
if resp.status_code == 200:
data = resp.json()
items = data.get("data", {}).get("items", [])
return [
{
"address": h.get("holder", ""),
"amount": h.get("amount", 0),
"percentage": h.get("percent", 0),
}
for h in items
]
except Exception as e:
logger.debug(f"Solana holder fetch error: {e}")
# Fallback: Solscan free API
try:
url = f"https://public-api.solscan.io/token/holders?tokenAddress={address}&limit=100&offset=0"
resp = await self.http.get(url, timeout=10)
if resp.status_code == 200:
data = resp.json()
items = data if isinstance(data, list) else data.get("data", [])
return [
{
"address": h.get("owner", h.get("address", "")),
"amount": h.get("amount", h.get("balance", 0)),
"percentage": h.get("percentage", h.get("percent", 0)),
}
for h in items
]
except Exception as e:
logger.debug(f"Solscan holder fallback error: {e}")
return []
async def _fetch_evm_holders(self, address: str, chain: str) -> list[dict[str, Any]]:
"""Fetch EVM token holders via Birdeye public API."""
try:
url = f"{BIRDEYE_PUBLIC}/defi/holder/tokenlist?tokenAddress={address}&limit=100"
headers = {"Accept": "application/json"}
if self._birdeye_api_key:
headers["X-API-KEY"] = self._birdeye_api_key
resp = await self.http.get(url, headers=headers, timeout=10)
if resp.status_code == 200:
data = resp.json()
items = data.get("data", {}).get("items", [])
return [
{
"address": h.get("holder", ""),
"amount": h.get("amount", 0),
"percentage": h.get("percent", 0),
}
for h in items
]
except Exception as e:
logger.debug(f"EVM holder fetch error: {e}")
return []
async def _fetch_buys(self, address: str, chain: str) -> list[dict[str, Any]]:
"""Fetch recent buy transactions for the token."""
buys: list[dict[str, Any]] = []
try:
url = f"{DEXSCREENER_API}/tokens/{address}"
resp = await self.http.get(url, timeout=10)
if resp.status_code == 200:
data = resp.json()
pairs = data.get("pairs", [])
for pair in pairs[:5]: # Check top 5 pairs
txns = pair.get("txns", {})
# Extract buys from recent transactions
m5 = txns.get("m5", {}) or {}
h1 = txns.get("h1", {}) or {}
h6 = txns.get("h6", {}) or {}
buys.append(
{
"type": "buy",
"m5_buys": m5.get("buys", 0),
"m5_sells": m5.get("sells", 0),
"h1_buys": h1.get("buys", 0),
"h1_sells": h1.get("sells", 0),
"h6_buys": h6.get("buys", 0),
"h6_sells": h6.get("sells", 0),
"pair_address": pair.get("pairAddress", ""),
"creation_block": None, # May not be available
}
)
# Try to get volume per tx for bundling analysis
volume_m5 = pair.get("volume", {}).get("m5", 0) or 0
if m5.get("buys", 0) > 0:
avg_buy = float(volume_m5) / max(1, m5.get("buys", 1))
buys[-1]["avg_buy_value"] = avg_buy
except Exception as e:
logger.debug(f"Buy fetch error: {e}")
return buys
# ── Analysis ────────────────────────────────────────────────
@staticmethod
def _compute_top_holder_pct(holders: list[dict[str, Any]], top_n: int) -> float:
"""Calculate the percentage of supply held by top N holders."""
sorted_h = sorted(holders, key=lambda h: h.get("percentage", 0), reverse=True)
top = sorted_h[:top_n]
return sum(h.get("percentage", 0) for h in top if h.get("percentage") is not None)
@staticmethod
def _extract_dev_hold_pct(holders: list[dict[str, Any]], metadata: dict[str, Any]) -> float:
"""Extract developer/allocation wallet holding percentage."""
if not holders:
return 0.0
return holders[0].get("percentage", 0) if holders else 0.0
def _detect_bundled_buys(self, buys: list[dict[str, Any]]) -> tuple[list[BundledBuy], int]:
"""Detect buys that appear bundled (same source, time clustering)."""
bundled: list[BundledBuy] = []
same_funding_count = 0
# From aggregated transaction data, detect anomalous patterns
for buy in buys:
m5_buys = buy.get("m5_buys", 0)
h1_buys = buy.get("h1_buys", 0)
# If buys/minute in first 5min is very high relative to later
if m5_buys > 0 and h1_buys > 0:
m5_rate = m5_buys / 5
h1_rate = h1_buys / 60
if m5_rate > h1_rate * 3 and m5_buys >= 10:
# High initial buy concentration — suspicious
bundled.append(
BundledBuy(
wallet=f"cluster:{buy.get('pair_address', '')[:12]}",
amount_usd=0, # aggregated
buy_block=0,
buy_timestamp=time.time(),
tx_hash="",
funding_source="aggregated",
is_sniper=True,
)
)
same_funding_count += m5_buys
return bundled, same_funding_count
def _analyze_launch_timing(self, buys: list[dict[str, Any]]) -> dict[str, Any]:
"""Analyze launch timing for anomalous patterns."""
result = {
"unique_buyers_first_block": 0,
"total_buys_first_blocks": 0,
"buy_concentration_ratio": 0.0,
}
for buy in buys:
m5_buys = buy.get("m5_buys", 0)
h1_buys = buy.get("h1_buys", 0)
h6_buys = buy.get("h6_buys", 0)
total = m5_buys + h1_buys + h6_buys
if total > 0:
# What % of all buys happened in first 5 minutes?
first_5m_pct = m5_buys / total if total > 0 else 0
result["buy_concentration_ratio"] = max(result["buy_concentration_ratio"], first_5m_pct)
result["total_buys_first_blocks"] += m5_buys
# Estimate unique from m5 vs h1 ratio
if h1_buys > 0 and m5_buys > 0:
result["unique_buyers_first_block"] = max(
result["unique_buyers_first_block"],
min(m5_buys, h1_buys), # rough proxy
)
return result
def _cluster_wallets(self, buys: list[dict[str, Any]], holders: list[dict[str, Any]]) -> list[HolderCluster]:
"""Cluster wallets by funding overlap and timing patterns."""
clusters: list[HolderCluster] = []
if not holders:
return clusters
# Identify clusters based on supply concentration
sorted_h = sorted(holders, key=lambda h: h.get("percentage", 0), reverse=True)
# If top 3 holders control >60%, they form a natural cluster
top3 = sorted_h[:3]
top3_pct = sum(h.get("percentage", 0) for h in top3 if h.get("percentage") is not None)
if top3_pct > 60 and len(top3) >= 2:
clusters.append(
HolderCluster(
wallets=[h.get("address", "") for h in top3 if h.get("address")],
total_supply_pct=top3_pct,
funding_overlap_score=0.7 if top3_pct > 80 else 0.5,
buy_time_similarity=0.8 if top3_pct > 80 else 0.6,
common_funding_source="top_holders_cluster",
)
)
# Check for wallet groupings with 5-15% each (typical bundler pattern)
cluster_wallets: list[dict[str, Any]] = []
cluster_pct = 0.0
for h in sorted_h[3:]: # Skip top 3
pct = h.get("percentage", 0)
if pct and 2 <= pct <= 15:
cluster_wallets.append(h)
cluster_pct += pct
if len(cluster_wallets) >= 5 and cluster_pct >= 15:
break
if len(cluster_wallets) >= 5 and cluster_pct >= 15:
clusters.append(
HolderCluster(
wallets=[h.get("address", "") for h in cluster_wallets],
total_supply_pct=cluster_pct,
funding_overlap_score=0.6,
buy_time_similarity=0.7,
common_funding_source="mid_holder_belt",
)
)
return clusters
@staticmethod
def _estimate_entities(
holders: list[dict[str, Any]],
clusters: list[HolderCluster],
bundled_buys_count: int,
) -> int:
"""Estimate number of truly independent entities behind the token."""
total_holders = len(holders)
# Each cluster represents 1 entity instead of N wallets
cluster_wallet_count = sum(len(c.wallets) for c in clusters)
# Reduce estimated entities by clustered wallets
entities = max(1, total_holders - cluster_wallet_count)
# Further reduce if many bundled buys detected
if bundled_buys_count > 20:
entities = max(1, entities - bundled_buys_count // 5)
return entities
# ── Scoring ─────────────────────────────────────────────────
def _score_supply_concentration(self, holders: list[dict[str, Any]], top10_pct: float) -> float:
"""Score supply distribution risk (0-100)."""
score = 0.0
# Top 10 concentration
if top10_pct >= 90:
score += 50
elif top10_pct >= 75:
score += 35
elif top10_pct >= 50:
score += 20
elif top10_pct >= 30:
score += 10
# Gini coefficient
amounts = [h.get("percentage", 0) for h in holders[:100] if h.get("percentage") is not None]
gini = _gini_coefficient(amounts)
if gini >= 0.9:
score += 40
elif gini >= 0.8:
score += 30
elif gini >= 0.6:
score += 15
# Entropy (low entropy = concentrated)
ent = _entropy(amounts)
if ent < 0.3:
score += 15
elif ent < 0.5:
score += 8
return min(score, 100)
def _score_sniper_clusters(self, clusters: list[HolderCluster], bundled_buys: list[BundledBuy]) -> float:
"""Score sniper cluster risk (0-100)."""
score = 0.0
# High-funding-overlap clusters
high_overlap = [c for c in clusters if c.funding_overlap_score > 0.6]
if high_overlap:
total_pct = sum(c.total_supply_pct for c in high_overlap)
if total_pct >= 50:
score += 50
elif total_pct >= 30:
score += 35
elif total_pct >= 15:
score += 20
# Bundled buys
if bundled_buys:
score += min(len(bundled_buys) * 5, 30)
# Time clustering in clusters
high_time = [c for c in clusters if c.buy_time_similarity > 0.7]
if high_time:
score += min(len(high_time) * 10, 25)
return min(score, 100)
def _score_launch_timing(
self,
timing_info: dict[str, Any],
buys: list[dict[str, Any]],
holders: list[dict[str, Any]],
) -> float:
"""Score launch timing anomalies (0-100)."""
score = 0.0
# High buy concentration in first 5 minutes
ratio = timing_info.get("buy_concentration_ratio", 0)
if ratio >= 0.8:
score += 50
elif ratio >= 0.6:
score += 35
elif ratio >= 0.4:
score += 20
# Very few unique buyers relative to total buys
unique = timing_info.get("unique_buyers_first_block", 0)
total = timing_info.get("total_buys_first_blocks", 0)
if total > 0 and unique > 0:
repeat_rate = total / max(1, unique)
if repeat_rate >= 5:
score += 30
elif repeat_rate >= 3:
score += 20
# Holder count vs buy count mismatch
holder_count = len(holders)
if holder_count > 0 and total > 0:
buys_per_holder = total / holder_count
if buys_per_holder >= 3:
score += 15
return min(score, 100)
def _score_fund_flow(
self,
bundled_buys: list[BundledBuy],
same_funding_count: int,
clusters: list[HolderCluster],
) -> float:
"""Score fund flow risk (0-100)."""
score = 0.0
# Same funding source buys
if same_funding_count >= 20:
score += 45
elif same_funding_count >= 10:
score += 30
elif same_funding_count >= 5:
score += 15
# Clusters with high funding overlap
high_overlap = [c for c in clusters if c.funding_overlap_score > 0.7]
if high_overlap:
score += min(len(high_overlap) * 15, 30)
# Overall cluster funding overlap average
if clusters:
avg_overlap = sum(c.funding_overlap_score for c in clusters) / len(clusters)
score += avg_overlap * 20
return min(score, 100)
def _compute_bundler_score(self, report: BundlerReport) -> float:
"""Weighted composite bundler score."""
weights = {
"supply_concentration": 0.30,
"sniper_cluster": 0.25,
"launch_timing_anomaly": 0.20,
"fund_flow_risk": 0.25,
}
score = (
report.supply_concentration_score * weights["supply_concentration"]
+ report.sniper_cluster_score * weights["sniper_cluster"]
+ report.launch_timing_anomaly_score * weights["launch_timing_anomaly"]
+ report.fund_flow_risk_score * weights["fund_flow_risk"]
)
return min(score, 100)

932
backend/app/canonical_tools.py
View file

@ -1,932 +0,0 @@
"""
Canonical Tool Prices — Single Source of Truth
127 tools. Enforcement + databus merged.
This file is THE authoritative list. All endpoints (MCP discovery, x402 catalog, human marketplace) read from this.
"""
CANONICAL_TOOL_PRICES = {
"airdrop_check": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "market",
"trial_free": 2,
"description": "airdrop_check",
},
"airdrop_finder": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "intelligence",
"trial_free": 2,
"description": "airdrop_finder",
},
"alpha_digest": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "intelligence",
"trial_free": 1,
"description": "alpha_digest",
},
"anomaly": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "security",
"trial_free": 0,
"description": "anomaly",
},
"arbitrage_scan": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "market",
"trial_free": 2,
"description": "arbitrage_scan",
},
"arkham_counterparties": {
"price_usd": 0.2,
"price_atoms": "200000",
"category": "elite",
"trial_free": 0,
"description": "Counterparty intelligence — entity relationship graph and money flow analysis",
},
"arkham_entity": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "premium",
"trial_free": 1,
"description": "Entity resolution — map any address to its real-world owner with confidence scoring",
},
"arkham_labels": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "premium",
"trial_free": 1,
"description": "Institutional entity labels — fund names, exchange wallets, known addresses",
},
"arkham_portfolio": {
"price_usd": 0.25,
"price_atoms": "250000",
"category": "elite",
"trial_free": 0,
"description": "Institutional portfolio intelligence — complete holdings, historical performance, attribution",
},
"arkham_transfers": {
"price_usd": 0.2,
"price_atoms": "200000",
"category": "elite",
"trial_free": 0,
"description": "Cross-chain transfer tracer — full movement history with entity labeling",
},
"audit": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "security",
"trial_free": 1,
"description": "audit",
},
"bridge_security": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "security",
"trial_free": 0,
"description": "bridge_security",
},
"bubble_map": {
"price_usd": 0.02,
"price_atoms": "20000",
"category": "basic",
"trial_free": 3,
"description": "Holder concentration map — visualize whale clusters and distribution",
},
"bundle_detect": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "premium",
"trial_free": 1,
"description": "Bot detector — same-block bundling, MEV patterns, sniper wallet identification",
},
"bundler_detect": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "security",
"trial_free": 1,
"description": "Supply Manipulation Detector — bundled launches, sniper-clustered distributions, and multi-wallet insider patterns",
},
"catalog": {
"price_usd": 0.0,
"price_atoms": "0",
"category": "api",
"trial_free": 999,
"description": "Browse available tools, pricing, and chain support",
},
"chain_health": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "market",
"trial_free": 0,
"description": "chain_health",
},
"clone_detect": {
"price_usd": 0.02,
"price_atoms": "20000",
"category": "security",
"trial_free": 3,
"description": "clone_detect",
},
"cluster": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "intelligence",
"trial_free": 0,
"description": "cluster",
},
"composite_score": {
"price_usd": 0.25,
"price_atoms": "250000",
"category": "premium",
"trial_free": 1,
"description": "RMI Composite Score — one number combining ALL signals for instant buy/sell/avoid decisions",
},
"comprehensive_audit": {
"price_usd": 0.5,
"price_atoms": "500000",
"category": "security",
"trial_free": 1,
"description": "comprehensive_audit",
},
"contract_scan": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "premium",
"trial_free": 1,
"description": "Deep contract audit — static analysis, honeypot detection, vulnerability scan",
},
"copy_trade_finder": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "intelligence",
"trial_free": 0,
"description": "copy_trade_finder",
},
"cross_chain": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "premium",
"trial_free": 1,
"description": "Cross-chain activity — find the same entity across multiple blockchains",
},
"defi_position": {
"price_usd": 0.15,
"price_atoms": "150000",
"category": "defi",
"trial_free": 1,
"description": "DeFi position analyzer — LP holdings, impermanent loss estimation, yield sustainability, protocol risk",
},
"defi_protocols": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "basic",
"trial_free": 5,
"description": "DeFi protocol tracker — TVL, chains, categories, revenue metrics",
},
"defi_yield_scanner": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "market",
"trial_free": 1,
"description": "defi_yield_scanner",
},
"deployer_history": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "security",
"trial_free": 2,
"description": "deployer_history",
},
"dex_data": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "basic",
"trial_free": 5,
"description": "DEX pool data — liquidity depth, volume, price impact for any token",
},
"entity_intel": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "premium",
"trial_free": 1,
"description": "Entity intelligence — who is this wallet, linked addresses, risk assessment",
},
"fee_manipulation_detect": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "security",
"trial_free": 1,
"description": "Fee & Slippage Manipulation Detector — hidden token taxes, dynamic fees, cooldowns, blacklists, and sell traps",
},
"forensic_pack": {
"price_usd": 0.35,
"price_atoms": "350000",
"category": "bundle",
"trial_free": 1,
"description": "Forensic Investigation Pack — valuation + OSINT + report at 33% discount",
},
"forensic_valuation": {
"price_usd": 0.25,
"price_atoms": "250000",
"category": "premium",
"trial_free": 1,
"description": "Institutional-grade token valuation — DCF intrinsic value, comparable analysis with outlier detection, scam probability scoring",
},
"forensics": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "analysis",
"trial_free": 1,
"description": "forensics",
},
"fresh_pair": {
"price_usd": 0.03,
"price_atoms": "30000",
"category": "security",
"trial_free": 3,
"description": "fresh_pair",
},
"funding_source": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "premium",
"trial_free": 1,
"description": "Trace where a wallet's funds came from — multi-hop origin analysis",
},
"gas_forecast": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "market",
"trial_free": 0,
"description": "gas_forecast",
},
"gmgn_smart_money": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "premium",
"trial_free": 1,
"description": "Smart money narratives — trending wallets and their trade patterns",
},
"history": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "analysis",
"trial_free": 2,
"description": "Historical scanner time-series — risk/liquidity/volume/price trends over hours",
},
"honeypot_check": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "security",
"trial_free": 2,
"description": "honeypot_check",
},
"human-execute": {
"price_usd": 0.02,
"price_atoms": "20000",
"category": "api",
"trial_free": 2,
"description": "Human-in-the-loop execution — wallet-based payment for manual crypto investigation tasks",
},
"insider": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "intelligence",
"trial_free": 0,
"description": "insider",
},
"insider_network": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "intelligence",
"trial_free": 0,
"description": "insider_network",
},
"investigation_report": {
"price_usd": 0.2,
"price_atoms": "200000",
"category": "premium",
"trial_free": 1,
"description": "Full investigation report — on-chain forensics, financial valuation, OSINT findings, scam scoring in one deliverable",
},
"kol_performance": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "intelligence",
"trial_free": 0,
"description": "kol_performance",
},
"launch": {
"price_usd": 0.03,
"price_atoms": "30000",
"category": "launchpad",
"trial_free": 2,
"description": "launch",
},
"launch_intel": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "launchpad",
"trial_free": 2,
"description": "launch_intel",
},
"liquidity_depth": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "market",
"trial_free": 2,
"description": "liquidity_depth",
},
"liquidity_flow": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "intelligence",
"trial_free": 0,
"description": "liquidity_flow",
},
"liquidity_migration": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "security",
"trial_free": 2,
"description": "liquidity_migration",
},
"listing_predictor": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "intelligence",
"trial_free": 1,
"description": "listing_predictor",
},
"launch_fairness": {
"price_usd": 0.10,
"price_atoms": "100000",
"category": "security",
"trial_free": 2,
"description": "Launch fairness & bot activity analyzer — sniped distributions, bundled launches, LP manipulation, bot activity, presale concentration with 0-100 fairness score",
},
"market_movers": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "basic",
"trial_free": 5,
"description": "Top gainers, losers, and volume movers across all chains",
},
"market_overview": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "market",
"trial_free": 0,
"description": "market_overview",
},
"mcp-proxy": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "api",
"trial_free": 5,
"description": "MCP protocol proxy — route tool calls through the x402 payment layer",
},
"meme_vibe_score": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "social",
"trial_free": 3,
"description": "Meme token vibe scoring — sentiment, community strength, and virality analysis",
},
"mev_alert": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "security",
"trial_free": 1,
"description": "mev_alert",
},
"mev_detect": {
"price_usd": 0.15,
"price_atoms": "150000",
"category": "security",
"trial_free": 2,
"description": "MEV/Sandwich attack detection — sandwich attacks, frontrunning, arbitrage extraction, MEV bot identification",
},
"mev_protection": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "security",
"trial_free": 0,
"description": "mev_protection",
},
"nansen_labels": {
"price_usd": 0.15,
"price_atoms": "150000",
"category": "elite",
"trial_free": 0,
"description": "Smart money labels — fund tags, whale classifications, and institutional wallet mapping",
},
"nansen_smart_money": {
"price_usd": 0.15,
"price_atoms": "150000",
"category": "elite",
"trial_free": 0,
"description": "Smart money tracker — top trader activity, position tracking, alpha signals",
},
"narrative": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "social",
"trial_free": 3,
"description": "Market narrative engine — what is the market saying about this token RIGHT NOW",
},
"news": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "basic",
"trial_free": 5,
"description": "Crypto news feed — aggregated headlines, filtered by topic",
},
"nft_wash_detector": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "analysis",
"trial_free": 1,
"description": "nft_wash_detector",
},
"osint_identity_hunt": {
"price_usd": 0.15,
"price_atoms": "150000",
"category": "premium",
"trial_free": 2,
"description": "Cross-platform OSINT investigation — hunt usernames across 400+ networks, domain intelligence, stealth page capture",
},
"portfolio": {
"price_usd": 0.15,
"price_atoms": "150000",
"category": "elite",
"trial_free": 0,
"description": "Multi-wallet portfolio — consolidated holdings, PnL, and risk across all wallets",
},
"portfolio_aggregate": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "analysis",
"trial_free": 1,
"description": "portfolio_aggregate",
},
"portfolio_risk": {
"price_usd": 0.2,
"price_atoms": "200000",
"category": "premium",
"trial_free": 1,
"description": "Cross-chain portfolio risk dashboard — unified risk across multiple wallets and chains",
},
"portfolio_tracker": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "analysis",
"trial_free": 0,
"description": "portfolio_tracker",
},
"prediction_markets": {
"price_usd": 0.02,
"price_atoms": "20000",
"category": "basic",
"trial_free": 3,
"description": "Prediction market odds — event probabilities and trading volumes",
},
"prediction_signals": {
"price_usd": 0.02,
"price_atoms": "20000",
"category": "basic",
"trial_free": 3,
"description": "Trading signals — sentiment, momentum, and contrarian indicators",
},
"profile_flip": {
"price_usd": 0.03,
"price_atoms": "30000",
"category": "security",
"trial_free": 3,
"description": "profile_flip",
},
"protocol_risk": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "security",
"trial_free": 1,
"description": "protocol_risk",
},
"pulse": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "market",
"trial_free": 3,
"description": "pulse",
},
"rag_search": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "premium",
"trial_free": 2,
"description": "Knowledge search — query 17K+ crypto documents for research, analysis, and deep answers",
},
"reputation_score": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "premium",
"trial_free": 1,
"description": "Comprehensive 0-100 trust score combining wallet labels, scam databases, deployer history, and RAG similarity matching",
},
"risk_monitor": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "security",
"trial_free": 1,
"description": "risk_monitor",
},
"risk_scan": {
"price_usd": 0.02,
"price_atoms": "20000",
"category": "basic",
"trial_free": 3,
"description": "Quick rug risk scan — honeypot, liquidity lock, ownership, and contract flags",
},
"rug_probability": {
"price_usd": 0.15,
"price_atoms": "150000",
"category": "premium",
"trial_free": 1,
"description": "Predictive rug pull probability 0-100 — honeypot + liquidity + deployer + social signals",
},
"rug_pull_predictor": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "security",
"trial_free": 0,
"description": "rug_pull_predictor",
},
"rugmaps_analysis": {
"price_usd": 0.02,
"price_atoms": "20000",
"category": "basic",
"trial_free": 3,
"description": "Holder distribution analysis — risk scoring, dump patterns, concentration",
},
"rugshield": {
"price_usd": 0.02,
"price_atoms": "20000",
"category": "security",
"trial_free": 3,
"description": "rugshield",
},
"scam_database": {
"price_usd": 0.03,
"price_atoms": "30000",
"category": "security",
"trial_free": 3,
"description": "scam_database",
},
"sentiment": {
"price_usd": 0.03,
"price_atoms": "30000",
"category": "social",
"trial_free": 0,
"description": "sentiment",
},
"sentiment_spike": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "social",
"trial_free": 2,
"description": "sentiment_spike",
},
"sentinel_deep": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "premium",
"trial_free": 1,
"description": "Full threat scan — deep contract analysis, risk scoring, threat intelligence",
},
"smart_money": {
"price_usd": 0.2,
"price_atoms": "200000",
"category": "intelligence",
"trial_free": 1,
"description": "Smart Money P&L Tracker — real profitability-based wallet tracking, find the actual profitable traders",
},
"smart_money_alpha": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "intelligence",
"trial_free": 3,
"description": "Smart money alpha signals — track wallets that consistently outperform the market",
},
"smartmoney": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "intelligence",
"trial_free": 1,
"description": "smartmoney",
},
"sniper_alert": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "launchpad",
"trial_free": 2,
"description": "sniper_alert",
},
"sniper_detect": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "intelligence",
"trial_free": 1,
"description": "sniper_detect",
},
"social_feed": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "basic",
"trial_free": 5,
"description": "Social sentiment feed — what crypto Twitter and Telegram are saying",
},
"social_signal": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "social",
"trial_free": 0,
"description": "social_signal",
},
"socialfi_resolve": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "basic",
"trial_free": 3,
"description": "Resolve social identity — ENS names, Farcaster profiles, linked addresses",
},
"syndicate_scan": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "intelligence",
"trial_free": 1,
"description": "syndicate_scan",
},
"syndicate_track": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "intelligence",
"trial_free": 1,
"description": "syndicate_track",
},
"threat_check": {
"price_usd": 0.02,
"price_atoms": "20000",
"category": "basic",
"trial_free": 3,
"description": "Threat intelligence check — known scams, malicious patterns, risk scoring",
},
"token_age": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "security",
"trial_free": 5,
"description": "token_age",
},
"token_comparison": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "analysis",
"trial_free": 0,
"description": "token_comparison",
},
"token_deep_dive": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "analysis",
"trial_free": 0,
"description": "token_deep_dive",
},
"token_detail": {
"price_usd": 0.02,
"price_atoms": "20000",
"category": "basic",
"trial_free": 3,
"description": "Full token intelligence — market cap, volume, liquidity, holders, risk flags",
},
"token_price": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "basic",
"trial_free": 5,
"description": "Get real-time token price with consensus from multiple sources",
},
"token_watch_alerts": {
"price_usd": 0.0,
"price_atoms": "0",
"category": "monitoring",
"trial_free": 999,
"description": "token_watch_alerts",
},
"token_watch_check": {
"price_usd": 0.03,
"price_atoms": "30000",
"category": "monitoring",
"trial_free": 5,
"description": "One-shot token status check — current LP, price, volume, and rug risk warnings",
},
"token_watch_create": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "monitoring",
"trial_free": 3,
"description": "Set token monitoring watch — alerts when LP drops, price changes, or rug indicators detected",
},
"token_watch_list": {
"price_usd": 0.0,
"price_atoms": "0",
"category": "monitoring",
"trial_free": 999,
"description": "token_watch_list",
},
"trending": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "basic",
"trial_free": 5,
"description": "Trending tokens across chains — hottest movers right now",
},
"tvl": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "basic",
"trial_free": 5,
"description": "DeFi TVL data — protocol-level totals, chain breakdowns, yields",
},
"tw_profile": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "social",
"trial_free": 0,
"description": "tw_profile",
},
"tw_search": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "social",
"trial_free": 0,
"description": "tw_search",
},
"tw_timeline": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "social",
"trial_free": 0,
"description": "tw_timeline",
},
"unlock_calendar": {
"price_usd": 0.03,
"price_atoms": "30000",
"category": "market",
"trial_free": 3,
"description": "unlock_calendar",
},
"urlcheck": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "security",
"trial_free": 3,
"description": "urlcheck",
},
"wallet": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "analysis",
"trial_free": 1,
"description": "wallet",
},
"wallet_balance": {
"price_usd": 0.01,
"price_atoms": "10000",
"category": "basic",
"trial_free": 3,
"description": "Check any wallet's balance across chains — multi-chain support",
},
"wallet_cluster": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "premium",
"trial_free": 1,
"description": "Syndicate mapper — find related wallets via funding patterns and heuristics",
},
"wallet_graph": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "intelligence",
"trial_free": 0,
"description": "wallet_graph",
},
"wallet_labels": {
"price_usd": 0.02,
"price_atoms": "20000",
"category": "basic",
"trial_free": 3,
"description": "Identify who owns a wallet — entity labels, tags, and known affiliations",
},
"wallet_pnl": {
"price_usd": 0.1,
"price_atoms": "100000",
"category": "analysis",
"trial_free": 0,
"description": "wallet_pnl",
},
"wallet_profile": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "premium",
"trial_free": 1,
"description": "Complete wallet profile — labels, PnL summary, risk score, related wallets",
},
"wallet_tokens": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "premium",
"trial_free": 1,
"description": "All tokens held by a wallet — balances, USD values, allocation breakdown",
},
"wash_trade_detect": {
"price_usd": 0.15,
"price_atoms": "150000",
"category": "security",
"trial_free": 2,
"description": "Wash Trading & Insider Detection — artificial volume, coordinated buying, insider accumulation patterns",
},
"wash_trading": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "security",
"trial_free": 1,
"description": "wash_trading",
},
"webhook_list": {
"price_usd": 0.0,
"price_atoms": "0",
"category": "monitoring",
"trial_free": 999,
"description": "List registered webhooks for an address",
},
"webhook_register": {
"price_usd": 0.02,
"price_atoms": "20000",
"category": "monitoring",
"trial_free": 2,
"description": "Register webhook URL for real-time monitoring alerts — rug pulls, whale moves, price crashes",
},
"whale": {
"price_usd": 0.15,
"price_atoms": "150000",
"category": "intelligence",
"trial_free": 1,
"description": "whale",
},
"whale_accumulation": {
"price_usd": 0.08,
"price_atoms": "80000",
"category": "intelligence",
"trial_free": 1,
"description": "whale_accumulation",
},
"whale_profile": {
"price_usd": 0.05,
"price_atoms": "50000",
"category": "intelligence",
"trial_free": 2,
"description": "whale_profile",
},
"whale_scan": {
"price_usd": 0.03,
"price_atoms": "30000",
"category": "intelligence",
"trial_free": 3,
"description": "whale_scan",
},
"rug_imminence": {
"price_usd": 0.20,
"price_atoms": "200000",
"category": "security",
"trial_free": 1,
"description": "Rug Pull Imminence Predictor — AI-powered early warning system fusing 7 signals to predict imminent rug pulls before they happen",
},
"liquidation_cascade": {
"price_usd": 0.15,
"price_atoms": "150000",
"category": "defi",
"trial_free": 1,
"description": "Liquidation Cascade Risk Analyzer — cross-chain DeFi position monitoring, health factor computation, and cascade scenario simulation for Aave/Compound positions",
},
"bridge_health": {
"price_usd": 0.10,
"price_atoms": "100000",
"category": "security",
"trial_free": 2,
"description": "Cross-Chain Bridge Health & Exploit Monitor — real-time TVL tracking, anomaly detection, trust model scoring, and exploit signal detection across 12 major bridges (LayerZero, Stargate, Wormhole, Across, Hop, Synapse, Axelar, Celer, DeBridge, CCIP, Connext, Orbiter)",
},
}

86
backend/app/catalog/__init__.py
View file

@ -1,86 +0,0 @@
"""Catalog domain — the unified read/write API for RMI.
T27 of the v4.0 guide. Every store read/write goes through CatalogService.
Domain facades call the catalog; they never touch stores directly.
Architecture (per v4.0 §T27):
app/catalog/models.py Pydantic v2 entity schemas
app/catalog/service.py CatalogService — fan-out reads, fan-in writes
app/catalog/reputation.py Deployer reputation scoring (T31)
app/catalog/rag_bridge.py Bridge to existing app/rag/ engine
app/catalog/llm_router.py LiteLLM proxy for AI analysis
Cross-store ref shape (per v4.0):
"chain:address" Wallet, Token, Contract IDs
UUID Entity, Alert, NewsItem, RAGFinding, Report IDs
Qdrant point_id RAGFinding.vector_id, rag_embedding_id on Token
Public API (re-exported):
CatalogService, get_catalog
Entity, Wallet, Deployer, Token, Alert, NewsItem, RAGFinding, ScanReport
DeployerReputation, RECIPES
"""
from __future__ import annotations
from app.core import health as health_mod
from app.core.health import DomainHealth
async def _health_check() -> DomainHealth:
"""Catalog health: which stores are reachable + how many entities."""
try:
from app.catalog.service import get_catalog
cat = get_catalog()
reach = await cat.probe_stores()
healthy = any(reach.values())
return DomainHealth(
name="catalog",
healthy=healthy,
details={
"stores_reachable": {k: v for k, v in reach.items()},
"primary": "redis+rag" if healthy else "none",
},
)
except Exception as e:
return DomainHealth(name="catalog", healthy=False, error=str(e))
health_mod.register_health_check("catalog", _health_check)
# Public API
from app.catalog.models import ( # noqa: E402
COLLECTIONS,
CHAIN_REGISTRY,
Chain,
Entity,
EntityLabel,
RiskTier,
Token,
Wallet,
Deployer,
Alert,
NewsItem,
RAGFinding,
ScanReport,
)
from app.catalog.service import CatalogService, get_catalog # noqa: E402
__all__ = [
"CatalogService",
"get_catalog",
"Entity",
"Wallet",
"Deployer",
"Token",
"Alert",
"NewsItem",
"RAGFinding",
"ScanReport",
"EntityLabel",
"Chain",
"RiskTier",
"COLLECTIONS",
"CHAIN_REGISTRY",
]

BIN
backend/app/catalog/__pycache__/__init__.cpython-311.pyc
View file

Binary file not shown.

BIN
backend/app/catalog/__pycache__/llm_router.cpython-311.pyc
View file

Binary file not shown.

BIN
backend/app/catalog/__pycache__/models.cpython-311.pyc
View file

Binary file not shown.

BIN
backend/app/catalog/__pycache__/service.cpython-311.pyc
View file

Binary file not shown.

172
backend/app/catalog/init_schema.py
View file

@ -1,172 +0,0 @@
"""Catalog database schema — initial migration.
Creates the tables referenced by the v4.0 catalog models.
Idempotent: safe to run multiple times.
"""
from __future__ import annotations
import asyncio
import os
import asyncpg
SCHEMA = """
-- Tokens (T27)
CREATE TABLE IF NOT EXISTS tokens (
token_id TEXT PRIMARY KEY,
chain TEXT NOT NULL,
address TEXT NOT NULL,
symbol TEXT,
name TEXT,
decimals INT,
deployer_wallet_id TEXT,
deployed_at TIMESTAMPTZ NOT NULL,
initial_supply BIGINT,
current_supply BIGINT,
is_honeypot BOOLEAN,
is_mintable BOOLEAN,
is_proxy BOOLEAN,
tax_buy_bps INT,
tax_sell_bps INT,
risk_tier TEXT,
risk_score INT,
risk_factors TEXT[],
rag_embedding_id TEXT,
updated_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_tokens_chain ON tokens(chain);
CREATE INDEX IF NOT EXISTS idx_tokens_deployer ON tokens(deployer_wallet_id);
CREATE INDEX IF NOT EXISTS idx_tokens_risk ON tokens(risk_score DESC) WHERE risk_score IS NOT NULL;
-- Alerts
CREATE TABLE IF NOT EXISTS alerts (
alert_id TEXT PRIMARY KEY,
token_id TEXT,
wallet_id TEXT,
chain TEXT,
alert_type TEXT NOT NULL,
severity TEXT NOT NULL,
title TEXT NOT NULL,
description TEXT,
evidence JSONB DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
resolved_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_alerts_token ON alerts(token_id);
CREATE INDEX IF NOT EXISTS idx_alerts_wallet ON alerts(wallet_id);
CREATE INDEX IF NOT EXISTS idx_alerts_created ON alerts(created_at DESC);
CREATE INDEX IF NOT EXISTS idx_alerts_severity ON alerts(severity);
-- News items (T28)
CREATE TABLE IF NOT EXISTS news_items (
news_id TEXT PRIMARY KEY,
url TEXT NOT NULL,
title TEXT NOT NULL,
summary TEXT,
body_markdown TEXT,
source TEXT NOT NULL,
published_at TIMESTAMPTZ NOT NULL,
ingested_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
chains_mentioned TEXT[] DEFAULT ARRAY[]::TEXT[],
tokens_mentioned TEXT[] DEFAULT ARRAY[]::TEXT[],
wallets_mentioned TEXT[] DEFAULT ARRAY[]::TEXT[],
sentiment_score REAL,
ai_analysis TEXT,
rag_embedding_id TEXT,
body_tsv tsvector
);
CREATE INDEX IF NOT EXISTS idx_news_published ON news_items(published_at DESC);
CREATE INDEX IF NOT EXISTS idx_news_source ON news_items(source);
CREATE INDEX IF NOT EXISTS idx_news_sentiment ON news_items(sentiment_score);
CREATE INDEX IF NOT EXISTS idx_news_body_tsv ON news_items USING gin(body_tsv);
-- Auto-update the body_tsv column on insert/update
CREATE OR REPLACE FUNCTION news_items_tsv_update() RETURNS trigger AS $$
BEGIN
NEW.body_tsv := to_tsvector('english', COALESCE(NEW.title, '') || ' ' ||
COALESCE(NEW.summary, '') || ' ' ||
COALESCE(NEW.body_markdown, ''));
RETURN NEW;
END
$$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS news_items_tsv_trigger ON news_items;
CREATE TRIGGER news_items_tsv_trigger
BEFORE INSERT OR UPDATE ON news_items
FOR EACH ROW EXECUTE FUNCTION news_items_tsv_update();
-- Scan reports (T29)
CREATE TABLE IF NOT EXISTS scan_reports (
report_id TEXT PRIMARY KEY,
subject_type TEXT NOT NULL,
subject_id TEXT NOT NULL,
generated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
generated_by_model TEXT NOT NULL,
risk_score INT NOT NULL,
risk_tier TEXT NOT NULL,
sections JSONB DEFAULT '{}'::jsonb,
markdown_url TEXT,
paid_via_x402 TEXT
);
CREATE INDEX IF NOT EXISTS idx_reports_subject ON scan_reports(subject_type, subject_id);
CREATE INDEX IF NOT EXISTS idx_reports_generated ON scan_reports(generated_at DESC);
-- RAG findings metadata (Qdrant has the vectors; this is metadata for query)
CREATE TABLE IF NOT EXISTS rag_findings (
finding_id TEXT PRIMARY KEY,
source_type TEXT NOT NULL,
source_url TEXT,
source_token_id TEXT,
source_wallet_id TEXT,
claim TEXT NOT NULL,
confidence REAL NOT NULL,
extracted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
qdrant_point_id TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_findings_token ON rag_findings(source_token_id);
CREATE INDEX IF NOT EXISTS idx_findings_wallet ON rag_findings(source_wallet_id);
-- x402 receipts (T34)
CREATE TABLE IF NOT EXISTS x402_receipts (
tx_hash TEXT PRIMARY KEY,
agent_id TEXT,
tool TEXT NOT NULL,
amount_usd REAL NOT NULL,
chain TEXT,
paid_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
tier TEXT
);
CREATE INDEX IF NOT EXISTS idx_x402_paid_at ON x402_receipts(paid_at DESC);
CREATE INDEX IF NOT EXISTS idx_x402_tool ON x402_receipts(tool);
CREATE INDEX IF NOT EXISTS idx_x402_agent ON x402_receipts(agent_id);
"""
async def main():
"""Run the schema migration."""
cfg = {
"host": os.getenv("POSTGRES_HOST", "rmi-postgres"),
"port": int(os.getenv("POSTGRES_PORT", "5432")),
"user": os.getenv("POSTGRES_USER", "rmi"),
"password": os.getenv("POSTGRES_PASSWORD", "RMI_PROD_POSTGRES_2026"),
"database": os.getenv("POSTGRES_DB", "rmi"),
}
print(f"Connecting to postgres at {cfg['host']}:{cfg['port']} db={cfg['database']}")
conn = await asyncpg.connect(**cfg)
try:
await conn.execute(SCHEMA)
print("✓ Schema applied")
# Verify
tables = await conn.fetch(
"SELECT tablename FROM pg_tables WHERE schemaname='public' ORDER BY tablename"
)
print(f"Tables ({len(tables)}):")
for t in tables:
print(f" {t['tablename']}")
finally:
await conn.close()
if __name__ == "__main__":
asyncio.run(main())

128
backend/app/catalog/llm_router.py
View file

@ -1,128 +0,0 @@
"""T27 LLM Router — sovereign-first LiteLLM proxy for catalog AI.
Per v4.0 §T28 (News analysis), §T29 (Report generation).
Self-hosted LiteLLM proxy at litellm.rugmunch.io routes to:
- DeepSeek-V3 (cost-effective analysis)
- Qwen (summaries)
- Local Llama-3 (free-tier fallback)
We never call OpenAI directly. If LiteLLM is unreachable, the catalog
operations that need LLM analysis return None/empty with a logged warning
rather than failing the whole request.
"""
from __future__ import annotations
import logging
import os
from typing import Any
import httpx
log = logging.getLogger(__name__)
# ── Config ─────────────────────────────────────────────────────────
LITELLM_URL = os.getenv("LITELLM_URL", "http://litellm.rugmunch.io")
LITELLM_API_KEY = os.getenv("LITELLM_API_KEY", "")
DEFAULT_MODEL = os.getenv("LITELLM_DEFAULT_MODEL", "deepseek-v3")
NEWS_ANALYSIS_PROMPT = """You are an analyst at RugMunch Intelligence, a crypto
scam-detection platform. Analyze the following news item and produce a
structured Markdown summary.
NEWS ITEM:
- Title: {title}
- Source: {source}
- Published: {published_at}
- Body: {body_truncated_to_2000_chars}
Produce a summary with these sections (use Markdown headers):
## Summary
2-3 sentence plain-English summary.
## Affected Tokens
List any tokens mentioned, with their chain and address if known.
## Affected Wallets
List any wallets mentioned.
## Sentiment
One of: bullish | bearish | neutral | risk-elevating | risk-reducing
1-sentence justification.
## RugMunch Action
What should our platform do in response? Options:
- (none)
- (flag mentioned tokens for re-scan)
- (alert subscribers)
- (update deployer reputation)
- (cross-chain entity resolution trigger)
Be concise. Do not speculate beyond what the article says.
"""
class LLMRouter:
"""Async client for the self-hosted LiteLLM proxy.
Falls back to None if the proxy is unreachable — catalog operations
that need LLM output will skip the AI analysis but still complete
the rest of the workflow.
"""
def __init__(self, url: str | None = None, api_key: str | None = None) -> None:
self.url = (url or LITELLM_URL).rstrip("/")
self.api_key = api_key or LITELLM_API_KEY
self._client: httpx.AsyncClient | None = None
async def _get_client(self) -> httpx.AsyncClient:
if self._client is None:
headers = {"Content-Type": "application/json"}
if self.api_key:
headers["Authorization"] = f"Bearer {self.api_key}"
self._client = httpx.AsyncClient(
base_url=self.url, headers=headers, timeout=30.0
)
return self._client
async def chat(
self,
prompt: str,
model: str | None = None,
max_tokens: int = 800,
temperature: float = 0.3,
) -> str | None:
"""Send a chat completion. Returns text or None on failure."""
try:
client = await self._get_client()
r = await client.post(
"/chat/completions",
json={
"model": model or DEFAULT_MODEL,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": max_tokens,
"temperature": temperature,
},
)
if r.status_code != 200:
log.warning("llm_http_%d: %s", r.status_code, r.text[:200])
return None
data = r.json()
return data["choices"][0]["message"]["content"]
except Exception as e:
log.warning("llm_chat_fail: %s", e)
return None
async def analyze_news(
self, news_item: "NewsItem" # type: ignore # noqa: F821
) -> str | None:
"""Generate AI analysis for a NewsItem. Per v4.0 §T28."""
prompt = NEWS_ANALYSIS_PROMPT.format(
title=news_item.title,
source=news_item.source,
published_at=news_item.published_at.isoformat(),
body_truncated_to_2000_chars=(news_item.body_markdown or news_item.summary)[:2000],
)
return await self.chat(prompt, max_tokens=800, temperature=0.3)

323
backend/app/catalog/models.py
View file

@ -1,323 +0,0 @@
"""T27A — Canonical entity models for the RMI data catalog.
Pydantic v2 (per ADR-0002). Every entity is persisted in exactly one primary
store, with cross-store references (string IDs) to related entities in other
stores. CatalogService resolves these references transparently.
Cross-store ID conventions:
Token, Wallet, Contract: f"{chain.value}:{address}"
Entity, Alert, NewsItem, RAGFinding, ScanReport: UUID4 hex
Qdrant point_id: 16-byte UUID hex (matches RAG engine)
Persistence (per v4.0 §T27 "Why each store exists"):
Redis: hot token data, rate limits, alert state, cron locks
Postgres: users, api_keys, subscriptions, x402 receipts, audit
alerts, news_items, scan_reports
Neo4j: entities, wallets, deployers, contracts, entity labels
(graph traversal, Cypher)
Qdrant: RAG embeddings, token similarity, news embeddings
MinIO: news raw HTML, report markdown, RAG source docs
Filesystem: ingest tmp, log rotation (transient)
"""
from __future__ import annotations
from datetime import datetime, UTC
from enum import Enum
from typing import Any, Literal, Optional
from pydantic import BaseModel, ConfigDict, Field, HttpUrl, field_validator
# ── Enums ───────────────────────────────────────────────────────────
class Chain(str, Enum):
"""All chains the platform indexes. Per v4.0 §T27."""
SOLANA = "solana"
ETHEREUM = "ethereum"
BASE = "base"
ARBITRUM = "arbitrum"
OPTIMISM = "optimism"
POLYGON = "polygon"
BSC = "bsc"
TRON = "tron"
BITCOIN = "bitcoin"
AVALANCHE = "avalanche"
FANTOM = "fantom"
GNOSIS = "gnosis"
# EVM subnets (sample — full 96 in CHAIN_REGISTRY)
SEPOLIA = "sepolia"
LINEA = "linea"
SCROLL = "scroll"
ZKSYNC = "zksync"
BLAST = "blast"
MANTLE = "mantle"
# Full chain registry — v4.0 says 96 chains. We track the canonical 20 here
# plus extend via CHAIN_REGISTRY for the remaining 76. Adding a chain is a
# one-line edit.
CHAIN_REGISTRY: dict[str, dict[str, str]] = {
"solana": {"name": "Solana", "type": "svm", "native": "SOL", "explorer": "https://solscan.io"},
"ethereum": {"name": "Ethereum", "type": "evm", "native": "ETH", "explorer": "https://etherscan.io"},
"base": {"name": "Base", "type": "evm", "native": "ETH", "explorer": "https://basescan.org"},
"arbitrum": {"name": "Arbitrum One", "type": "evm", "native": "ETH", "explorer": "https://arbiscan.io"},
"optimism": {"name": "Optimism", "type": "evm", "native": "ETH", "explorer": "https://optimistic.etherscan.io"},
"polygon": {"name": "Polygon", "type": "evm", "native": "MATIC", "explorer": "https://polygonscan.com"},
"bsc": {"name": "BNB Smart Chain", "type": "evm", "native": "BNB", "explorer": "https://bscscan.com"},
"tron": {"name": "TRON", "type": "tvm", "native": "TRX", "explorer": "https://tronscan.org"},
"bitcoin": {"name": "Bitcoin", "type": "utxo", "native": "BTC", "explorer": "https://mempool.space"},
"avalanche": {"name": "Avalanche C-Chain", "type": "evm", "native": "AVAX", "explorer": "https://snowtrace.io"},
"fantom": {"name": "Fantom Opera", "type": "evm", "native": "FTM", "explorer": "https://ftmscan.com"},
"gnosis": {"name": "Gnosis Chain", "type": "evm", "native": "xDAI", "explorer": "https://gnosisscan.io"},
}
class RiskTier(str, Enum):
LOW = "low"
MEDIUM = "medium"
HIGH = "high"
CRITICAL = "critical"
# ── Entities (Neo4j primary) ───────────────────────────────────────
class Entity(BaseModel):
"""A logical entity resolved across chains. Neo4j primary."""
model_config = ConfigDict(extra="ignore")
entity_id: str = Field(..., description="UUID, Neo4j primary key")
label: Optional[str] = None
aliases: list[str] = Field(default_factory=list)
first_seen: datetime
last_seen: datetime
risk_score: Optional[int] = Field(None, ge=0, le=100)
tags: list[str] = Field(default_factory=list)
notes: Optional[str] = None
store: Literal["neo4j"] = "neo4j"
class EntityLabel(BaseModel):
"""A label attached to an entity. Neo4j primary."""
model_config = ConfigDict(extra="ignore")
entity_id: str
label: str
source: Literal["manual", "heuristic", "third_party"]
confidence: float = Field(ge=0.0, le=1.0)
added_at: datetime
added_by: str
store: Literal["neo4j"] = "neo4j"
# ── Wallets + Deployers (Neo4j primary) ───────────────────────────
class Wallet(BaseModel):
"""An on-chain wallet. Neo4j primary; linked to Entity."""
model_config = ConfigDict(extra="ignore")
wallet_id: str = Field(..., description='"chain:address", e.g. "solana:7Np41..."')
chain: Chain
address: str
entity_id: Optional[str] = None
first_seen: datetime
last_seen: datetime
tx_count: int = 0
total_volume_usd: float = 0.0
is_deployer: bool = False
is_known_exchange: bool = False
is_suspicious: bool = False
reputation_score: Optional[int] = Field(None, ge=0, le=100)
store: Literal["neo4j"] = "neo4j"
@field_validator("wallet_id")
@classmethod
def _check_id_format(cls, v: str) -> str:
if ":" not in v:
raise ValueError("wallet_id must be 'chain:address'")
chain, _ = v.split(":", 1)
if chain not in {c.value for c in Chain}:
# Allow unknown chains (forward compat) but flag them
pass
return v
class Deployer(Wallet):
"""A wallet that has deployed at least one token. Extends Wallet."""
model_config = ConfigDict(extra="ignore")
deployments: list[str] = Field(default_factory=list, description="token_ids")
rug_count: int = 0
legit_count: int = 0
avg_token_lifetime_days: float = 0.0
reputation_score: Optional[int] = Field(
None,
ge=0,
le=100,
description="Weighted: legit_count * 1.0 - rug_count * 3.0 + age_bonus - news_penalty. Cached in Redis TTL 1h.",
)
# ── Tokens (Postgres primary) ──────────────────────────────────────
class Token(BaseModel):
"""A token contract. Postgres primary; references Deployer in Neo4j."""
model_config = ConfigDict(extra="ignore")
token_id: str = Field(..., description='"chain:address"')
chain: Chain
address: str
symbol: str
name: str
decimals: int
deployer_wallet_id: Optional[str] = Field(
None, description="Cross-store ref to Wallet (Neo4j)"
)
deployed_at: datetime
initial_supply: int
current_supply: Optional[int] = None
is_honeypot: Optional[bool] = None
is_mintable: Optional[bool] = None
is_proxy: Optional[bool] = None
tax_buy_bps: Optional[int] = None
tax_sell_bps: Optional[int] = None
risk_tier: Optional[RiskTier] = None
risk_score: Optional[int] = Field(None, ge=0, le=100)
risk_factors: list[str] = Field(default_factory=list)
rag_embedding_id: Optional[str] = Field(
None, description="Cross-store ref to Qdrant point (16-byte hex)"
)
store: Literal["postgres"] = "postgres"
# ── Alerts (Postgres primary) ──────────────────────────────────────
class Alert(BaseModel):
"""A risk alert. Postgres primary."""
model_config = ConfigDict(extra="ignore")
alert_id: str = Field(..., description="UUID")
token_id: Optional[str] = None
wallet_id: Optional[str] = None
chain: Optional[Chain] = None
alert_type: Literal[
"rug_detected",
"deployer_history",
"liquidity_drain",
"honeypot_detected",
"high_tax_change",
"whale_dump",
]
severity: Literal["info", "warning", "critical"]
title: str
description: str
evidence: dict[str, Any] = Field(default_factory=dict)
created_at: datetime
resolved_at: Optional[datetime] = None
store: Literal["postgres"] = "postgres"
# ── News (Postgres primary + Qdrant embeddings) ────────────────────
class NewsItem(BaseModel):
"""A news article from RSS. Postgres primary; embeddings in Qdrant."""
model_config = ConfigDict(extra="ignore")
news_id: str = Field(..., description="UUID")
url: HttpUrl
title: str
summary: str
body_markdown: Optional[str] = None
source: str
published_at: datetime
ingested_at: datetime
chains_mentioned: list[Chain] = Field(default_factory=list)
tokens_mentioned: list[str] = Field(default_factory=list)
wallets_mentioned: list[str] = Field(default_factory=list)
sentiment_score: Optional[float] = Field(None, ge=-1.0, le=1.0)
ai_analysis: Optional[str] = None
rag_embedding_id: Optional[str] = None
store: Literal["postgres"] = "postgres"
# ── RAG Findings (Qdrant primary + Postgres metadata) ───────────────
class RAGFinding(BaseModel):
"""A fact extracted by RAG. Qdrant primary (vector); metadata in Postgres."""
model_config = ConfigDict(extra="ignore")
finding_id: str = Field(..., description="UUID")
source_type: Literal["news", "onchain", "audit", "social", "manual"]
source_url: Optional[HttpUrl] = None
source_token_id: Optional[str] = None
source_wallet_id: Optional[str] = None
claim: str
confidence: float = Field(ge=0.0, le=1.0)
extracted_at: datetime
qdrant_point_id: str
store: Literal["qdrant"] = "qdrant"
# ── Reports (Postgres + MinIO) ──────────────────────────────────────
class ScanReport(BaseModel):
"""A research report. Postgres primary; markdown in MinIO."""
model_config = ConfigDict(extra="ignore")
report_id: str = Field(..., description="UUID")
subject_type: Literal["token", "wallet", "deployer"]
subject_id: str
generated_at: datetime
generated_by_model: str
risk_score: int = Field(ge=0, le=100)
risk_tier: RiskTier
sections: dict[str, str] = Field(default_factory=dict)
markdown_url: Optional[HttpUrl] = None
paid_via_x402: Optional[str] = None
store: Literal["postgres", "minio"] = "postgres"
def to_markdown(self) -> str:
"""Render report sections to a single Markdown document."""
parts = [
f"# Research Report: {self.subject_type.title()} `{self.subject_id}`",
"",
f"**Generated:** {self.generated_at.isoformat()}",
f"**Generated by:** {self.generated_by_model}",
f"**Risk score:** {self.risk_score}/100 ({self.risk_tier.value.upper()})",
"",
]
for section, body in self.sections.items():
parts.append(f"## {section.replace('_', ' ').title()}")
parts.append("")
parts.append(body)
parts.append("")
parts.append("---")
parts.append(f"*Report ID: {self.report_id}*")
return "\n".join(parts)
# ── Wire-format helpers ─────────────────────────────────────────────
def utcnow() -> datetime:
"""Timezone-aware UTC now. Pydantic serializes to ISO 8601."""
return datetime.now(UTC)
# ── RAG engine collections (kept here so catalog + RAG share the list) ─
COLLECTIONS: list[str] = [
# Per v4.0 catalog/RAG bridge — these are the canonical 13 RAG
# collections that also have Token/Wallet/etc cross-refs.
"scam_intel",
"deployer_history",
"wallet_labels",
"contract_audit",
"phishing_db",
"defi_hacks",
"rug_timeline",
"vuln_patterns",
"crime_reports",
"transaction_patterns",
"known_scams",
"token_analysis",
"market_intel",
]

179
backend/app/catalog/reputation.py
View file

@ -1,179 +0,0 @@
"""T01 — Bayesian Deployer Reputation System.
Per MINIMAX_M3_TASKS.md T01. Beta-Binomial posterior replaces the
weighted-sum that conflated probabilities with volumes.
The legacy 0-100 score is kept for backward compatibility (every
existing consumer reads it). The new authoritative output is:
probability — P(rug) = alpha / (alpha + beta)
credible_interval_95 — 95% Bayesian CI from Beta distribution
observations — {successes, failures, total}
We start with a uniform prior Beta(1,1). Each rug increments beta.
Each legitimate deployment increments alpha. News sentiment < -0.3
adds 2 to beta (pessimistic prior). News sentiment > 0.3 adds 2 to
alpha (optimistic prior). Age and volume are logged but not folded
into the prior (they are orthogonal signals, not evidence).
The legacy 0-100 score is derived deterministically from probability:
score = round((1 - probability) * 100)
"""
from __future__ import annotations
import logging
import math
from datetime import datetime, UTC
from app.catalog.models import Deployer, utcnow
log = logging.getLogger(__name__)
# ── Prior adjustments (Bayesian update weights) ────────────────
PRIOR_WEIGHTS: dict[str, int] = {
"prior_alpha": 1, # Beta(1,1) = uniform prior
"prior_beta": 1,
"news_pessimistic_shift": 2, # +2 to beta if avg sentiment < -0.3
"news_optimistic_shift": 2, # +2 to alpha if avg sentiment > 0.3
"news_window_hours": 720, # 30 days
"news_negative_threshold": -0.3,
"news_positive_threshold": 0.3,
}
def _beta_credible_interval_95(alpha: float, beta: float) -> tuple[float, float]:
"""Approximate 95% credible interval for Beta(alpha, beta).
Uses the normal approximation to the Beta distribution, which is
accurate for alpha+beta > 30 (our regime: typically dozens of
observations per deployer). For low-observation regimes, falls back
to a wider quantile-based interval.
"""
n = alpha + beta
if n <= 0:
return (0.0, 1.0)
if n < 30:
# Wider interval for low-data regime
mean = alpha / n
var = (alpha * beta) / (n * n * (n + 1))
sd = math.sqrt(var)
# Use 1.96 but clamp to [0,1]
lo = max(0.0, mean - 1.96 * sd)
hi = min(1.0, mean + 1.96 * sd)
return (lo, hi)
# High-data regime: tighter interval
mean = alpha / n
var = (alpha * beta) / (n * n * (n + 1))
sd = math.sqrt(var)
lo = max(0.0, mean - 1.96 * sd)
hi = min(1.0, mean + 1.96 * sd)
return (lo, hi)
async def compute_deployer_posterior(
deployer: Deployer,
catalog: "CatalogService",
) -> dict:
"""Compute Bayesian reputation for a deployer.
Returns:
{
"probability": float, # P(rug), 0..1
"credible_interval_95": [lo, hi], # 95% Bayesian CI
"observations": {
"rugs": int, "legit": int, "total": int,
"alpha": float, "beta": float,
},
"news_sentiment": float | None, # -1..+1 if available
"score": int, # legacy 0-100 (backward compat)
"computed_at": str, # ISO8601
}
"""
cache_key = f"catalog:deployer_rep:v2:{deployer.wallet_id}"
if catalog._health.redis:
try:
cached = await catalog._redis.get(cache_key)
if cached:
import json as _json
return _json.loads(cached)
except Exception:
pass
# ── Update prior from observations ──
alpha = float(PRIOR_WEIGHTS["prior_alpha"])
beta = float(PRIOR_WEIGHTS["prior_beta"])
rugs = max(0, deployer.rug_count)
legit = max(0, len(deployer.deployments) - rugs)
alpha += legit
beta += rugs
# ── News sentiment prior adjustment ──
news_sentiment = None
if catalog._health.postgres:
try:
async with catalog._pg_pool.acquire() as conn:
rows = await conn.fetch(
"""SELECT sentiment_score FROM news_items
WHERE $1 = ANY(wallets_mentioned)
AND published_at > NOW() - make_interval(hours => $2)
LIMIT 20""",
deployer.wallet_id,
PRIOR_WEIGHTS["news_window_hours"],
)
scores = [r["sentiment_score"] for r in rows if r["sentiment_score"] is not None]
if scores:
news_sentiment = sum(scores) / len(scores)
if news_sentiment < PRIOR_WEIGHTS["news_negative_threshold"]:
beta += PRIOR_WEIGHTS["news_pessimistic_shift"]
elif news_sentiment > PRIOR_WEIGHTS["news_positive_threshold"]:
alpha += PRIOR_WEIGHTS["news_optimistic_shift"]
except Exception as e:
log.debug("reputation_news_fail: %s", e)
# ── Posterior ──
total = alpha + beta
probability = alpha / total if total > 0 else 0.5
lo, hi = _beta_credible_interval_95(alpha, beta)
result = {
"probability": round(probability, 4),
"credible_interval_95": [round(lo, 4), round(hi, 4)],
"observations": {
"rugs": int(rugs),
"legit": int(legit),
"total": int(deployer.total_volume_usd and len(deployer.deployments) or 0),
"alpha": alpha,
"beta": beta,
},
"news_sentiment": round(news_sentiment, 4) if news_sentiment is not None else None,
# Legacy 0-100 score: probability of legitness scaled to 0..100
# probability = P(rug), so legitness = 1 - probability
"score": int(round((1.0 - probability) * 100)),
"computed_at": utcnow().isoformat(),
}
if catalog._health.redis:
try:
import json as _json
await catalog._redis.setex(cache_key, 3600, _json.dumps(result))
except Exception:
pass
return result
# ── Backward-compatible wrapper (returns just the int score) ──
async def compute_deployer_reputation(
deployer: Deployer,
catalog: "CatalogService",
) -> int:
"""Legacy 0-100 reputation score.
Returns the integer score derived from the Bayesian posterior.
New code should call compute_deployer_posterior() directly for the
full probability + CI.
"""
posterior = await compute_deployer_posterior(deployer, catalog)
return posterior["score"]

609
backend/app/catalog/service.py
View file

@ -1,609 +0,0 @@
"""T27 CatalogService — unified read/write API for RMI.
Per v4.0 §T27. The CatalogService is the ONLY sanctioned way to read or
write catalog data. Domain facades call this; they never touch stores directly.
Architecture:
- Lazy store clients (init on first use, retry on failure)
- Graceful degradation: unreachable stores return None/empty, not crash
- Redis cache layer (always available since rmi-redis is up)
- Cross-store ID conventions from app.catalog.models
Recipe coverage:
Recipe 1: find_tokens_by_deployer_history (Postgres + Neo4j)
Recipe 2: find_similar_tokens (Qdrant + Postgres)
Recipe 3: get_token_risk (Redis + Postgres + Neo4j)
Recipe 4: news_price_correlation (Postgres, basic v1)
Recipe 5: resolve_entity (Neo4j Cypher)
"""
from __future__ import annotations
import asyncio
import json
import logging
import os
import time
from typing import Any, Optional
import httpx
from app.catalog.models import (
Alert,
COLLECTIONS,
Chain,
Deployer,
Entity,
NewsItem,
RAGFinding,
ScanReport,
Token,
Wallet,
utcnow,
)
from app.rag.engine import three_pillar_search as rag_three_pillar_search
from app.rag.engine import ingest_document as rag_ingest_document
log = logging.getLogger(__name__)
# ── Connection config (from env, with sensible defaults to netcup IPs) ──
# These defaults point to the actual IPs on netcup rmi_network.
# Override via env vars in docker-compose.yml for portability.
DEFAULT_CONFIG: dict[str, Any] = {
"redis": {
"host": os.getenv("REDIS_HOST", "rmi-redis"),
"port": int(os.getenv("REDIS_PORT", "6379")),
"password": os.getenv("REDIS_PASSWORD", "RMI_PROD_REDIS_2026"),
"db": int(os.getenv("REDIS_DB", "0")),
},
"postgres": {
"host": os.getenv("POSTGRES_HOST", "rmi-postgres"),
"port": int(os.getenv("POSTGRES_PORT", "5432")),
"user": os.getenv("POSTGRES_USER", "rmi"),
"password": os.getenv("POSTGRES_PASSWORD", ""),
"database": os.getenv("POSTGRES_DB", "rmi"),
},
"neo4j": {
"uri": os.getenv("NEO4J_URI", "bolt://rmi-neo4j:7687"),
"user": os.getenv("NEO4J_USER", "neo4j"),
"password": os.getenv("NEO4J_PASSWORD", ""),
},
"qdrant": {
"url": os.getenv("QDRANT_URL", "http://rmi-qdrant:6333"),
"api_key": os.getenv("QDRANT_API_KEY", ""),
},
"minio": {
"endpoint": os.getenv("MINIO_ENDPOINT", "rmi-minio:9000"),
"access_key": os.getenv("MINIO_ACCESS_KEY", ""),
"secret_key": os.getenv("MINIO_SECRET_KEY", ""),
},
}
class StoreHealth:
"""Tracks which stores are reachable. Updated on each probe."""
def __init__(self) -> None:
self.redis: bool = False
self.postgres: bool = False
self.neo4j: bool = False
self.qdrant: bool = False
self.minio: bool = False
self.last_checked: float = 0.0
class CatalogService:
"""Unified read/write API for RMI data.
Use `get_catalog()` to get the singleton instance. All methods are
async and graceful-degrade — if a store is unreachable, the method
returns None or an empty result with a logged warning.
"""
def __init__(self, config: dict[str, Any] | None = None) -> None:
self.config = config or DEFAULT_CONFIG
self._health = StoreHealth()
self._redis = None
self._pg_pool: Any = None
self._neo_driver: Any = None
self._qdrant: Any = None # httpx.AsyncClient
self._init_lock = asyncio.Lock()
# ── Store init (lazy) ────────────────────────────────────────────
async def _init_stores(self) -> None:
async with self._init_lock:
if self._redis is not None:
return # already init
# Redis (always try first — used for everything)
try:
import redis.asyncio as aioredis
cfg = self.config["redis"]
self._redis = aioredis.Redis(
host=cfg["host"],
port=cfg["port"],
password=cfg["password"],
db=cfg["db"],
decode_responses=True,
)
await self._redis.ping()
self._health.redis = True
log.info("catalog_redis_ok")
except Exception as e:
log.warning("catalog_redis_fail: %s", e)
self._health.redis = False
# Postgres
try:
import asyncpg
cfg = self.config["postgres"]
if cfg["password"]:
self._pg_pool = await asyncpg.create_pool(
host=cfg["host"],
port=cfg["port"],
user=cfg["user"],
password=cfg["password"],
database=cfg["database"],
min_size=1,
max_size=8,
command_timeout=10,
)
self._health.postgres = True
log.info("catalog_postgres_ok")
except Exception as e:
log.warning("catalog_postgres_fail: %s", e)
self._health.postgres = False
# Neo4j (NEO4J_AUTH=none means no password)
try:
from neo4j import GraphDatabase
cfg = self.config["neo4j"]
auth = (
(cfg["user"], cfg["password"]) if cfg["password"] else None
)
self._neo_driver = GraphDatabase.driver(cfg["uri"], auth=auth)
with self._neo_driver.session() as s:
s.run("RETURN 1").consume()
self._health.neo4j = True
log.info("catalog_neo4j_ok")
except Exception as e:
log.warning("catalog_neo4j_fail: %s", e)
self._health.neo4j = False
# MinIO (HTTP health probe)
try:
import socket
host, port = self.config["minio"]["endpoint"].rsplit(":", 1)
s = socket.socket()
s.settimeout(3)
s.connect((host, int(port)))
s.close()
self._health.minio = True
log.info("catalog_minio_ok")
except Exception as e:
log.warning("catalog_minio_fail: %s", e)
self._health.minio = False
# Qdrant (HTTP)
try:
cfg = self.config["qdrant"]
headers = {"api-key": cfg["api_key"]} if cfg["api_key"] else {}
self._qdrant = httpx.AsyncClient(
base_url=cfg["url"], headers=headers, timeout=5.0
)
r = await self._qdrant.get("/collections")
if r.status_code == 200:
self._health.qdrant = True
log.info("catalog_qdrant_ok")
else:
log.warning("catalog_qdrant_http_%d", r.status_code)
except Exception as e:
log.warning("catalog_qdrant_fail: %s", e)
self._health.qdrant = False
self._health.last_checked = time.time()
# ── Health probe ────────────────────────────────────────────────
async def probe_stores(self) -> dict[str, bool]:
await self._init_stores()
return {
"redis": self._health.redis,
"postgres": self._health.postgres,
"neo4j": self._health.neo4j,
"qdrant": self._health.qdrant,
"minio": self._health.minio,
}
# ── Tokens (Postgres primary + Redis cache) ─────────────────────
async def get_token(self, chain: Chain, address: str) -> Token | None:
await self._init_stores()
cache_key = f"catalog:token:{chain.value}:{address}"
if self._health.redis:
try:
cached = await self._redis.get(cache_key)
if cached:
return Token.model_validate_json(cached)
except Exception as e:
log.debug("token_cache_get_fail: %s", e)
if not self._health.postgres:
return None
try:
async with self._pg_pool.acquire() as conn:
row = await conn.fetchrow(
"SELECT * FROM tokens WHERE chain=$1 AND address=$2",
chain.value, address,
)
if not row:
return None
token = Token(**dict(row))
if self._health.redis:
try:
await self._redis.setex(cache_key, 3600, token.model_dump_json())
except Exception as e:
log.debug("token_cache_set_fail: %s", e)
return token
except Exception as e:
log.warning("token_get_fail: %s", e)
return None
async def save_token(self, token: Token) -> bool:
await self._init_stores()
if not self._health.postgres:
log.warning("token_save_no_postgres")
return False
try:
async with self._pg_pool.acquire() as conn:
await conn.execute(
"""
INSERT INTO tokens (
token_id, chain, address, symbol, name, decimals,
deployer_wallet_id, deployed_at, initial_supply,
current_supply, is_honeypot, is_mintable, is_proxy,
tax_buy_bps, tax_sell_bps, risk_tier, risk_score,
risk_factors, rag_embedding_id
) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,
$14,$15,$16,$17,$18,$19)
ON CONFLICT (token_id) DO UPDATE SET
symbol=EXCLUDED.symbol,
name=EXCLUDED.name,
current_supply=EXCLUDED.current_supply,
is_honeypot=EXCLUDED.is_honeypot,
is_mintable=EXCLUDED.is_mintable,
risk_tier=EXCLUDED.risk_tier,
risk_score=EXCLUDED.risk_score,
risk_factors=EXCLUDED.risk_factors
""",
token.token_id, token.chain.value, token.address,
token.symbol, token.name, token.decimals,
token.deployer_wallet_id, token.deployed_at,
token.initial_supply, token.current_supply,
token.is_honeypot, token.is_mintable, token.is_proxy,
token.tax_buy_bps, token.tax_sell_bps,
token.risk_tier.value if token.risk_tier else None,
token.risk_score, token.risk_factors, token.rag_embedding_id,
)
# Invalidate cache
if self._health.redis:
try:
await self._redis.delete(
f"catalog:token:{token.chain.value}:{token.address}"
)
except Exception:
pass
return True
except Exception as e:
log.warning("token_save_fail: %s", e)
return False
# ── Wallets (Neo4j primary) ──────────────────────────────────────
async def get_wallet(self, chain: Chain, address: str) -> Wallet | None:
await self._init_stores()
if not self._health.neo4j:
return None
wallet_id = f"{chain.value}:{address}"
try:
with self._neo_driver.session() as s:
result = s.run(
"MATCH (w:Wallet {wallet_id: $id}) RETURN w",
id=wallet_id,
).single()
if not result:
return None
node = dict(result["w"])
# Normalize datetime strings
for k in ("first_seen", "last_seen"):
if isinstance(node.get(k), str):
from datetime import datetime
try:
node[k] = datetime.fromisoformat(node[k].replace("Z", "+00:00"))
except Exception:
pass
return Wallet(**node)
except Exception as e:
log.warning("wallet_get_fail: %s", e)
return None
async def save_wallet(self, wallet: Wallet) -> bool:
await self._init_stores()
if not self._health.neo4j:
return False
try:
with self._neo_driver.session() as s:
s.run(
"""
MERGE (w:Wallet {wallet_id: $wallet_id})
SET w.chain=$chain, w.address=$address,
w.first_seen=$first_seen, w.last_seen=$last_seen,
w.tx_count=$tx_count, w.total_volume_usd=$total_volume_usd,
w.is_deployer=$is_deployer,
w.is_known_exchange=$is_known_exchange,
w.is_suspicious=$is_suspicious,
w.reputation_score=$reputation_score
""",
wallet_id=wallet.wallet_id,
chain=wallet.chain.value,
address=wallet.address,
first_seen=wallet.first_seen.isoformat(),
last_seen=wallet.last_seen.isoformat(),
tx_count=wallet.tx_count,
total_volume_usd=wallet.total_volume_usd,
is_deployer=wallet.is_deployer,
is_known_exchange=wallet.is_known_exchange,
is_suspicious=wallet.is_suspicious,
reputation_score=wallet.reputation_score,
)
return True
except Exception as e:
log.warning("wallet_save_fail: %s", e)
return False
# ── Recipe 1: find tokens by deployer history ───────────────────
async def find_tokens_by_deployer_history(
self, min_rug_count: int = 1, chain: Chain | None = None, limit: int = 50
) -> list[Token]:
"""Find tokens deployed by wallets that have rug-pulled before.
Cross-store: Neo4j for the wallet filter, Postgres for the tokens.
"""
await self._init_stores()
if not (self._health.neo4j and self._health.postgres):
log.warning("recipe1_stores_unavailable")
return []
try:
# Step 1: Neo4j — find wallets with rug_count >= min_rug_count
with self._neo_driver.session() as s:
wallets = [
r["w.wallet_id"]
for r in s.run(
"MATCH (w:Deployer) WHERE w.rug_count >= $min "
"RETURN w.wallet_id LIMIT 200",
min=min_rug_count,
)
]
if not wallets:
return []
# Step 2: Postgres — find tokens deployed by those wallets
async with self._pg_pool.acquire() as conn:
query = (
"SELECT * FROM tokens WHERE deployer_wallet_id = ANY($1::text[]) "
)
params: list[Any] = [wallets]
if chain:
query += "AND chain=$2 "
params.append(chain.value)
query += "ORDER BY deployed_at DESC LIMIT $3"
params.append(limit)
rows = await conn.fetch(query, *params)
return [Token(**dict(r)) for r in rows]
except Exception as e:
log.warning("recipe1_fail: %s", e)
return []
# ── Recipe 3: get_token_risk (3-store composition) ──────────────
async def get_token_risk(
self, chain: Chain, address: str
) -> dict[str, Any]:
"""Real-time risk score — composes Redis cache + Postgres + Neo4j.
Returns dict (not Token) because it's a cross-store projection.
Cache TTL 60s.
"""
await self._init_stores()
cache_key = f"catalog:risk:{chain.value}:{address}"
if self._health.redis:
try:
cached = await self._redis.get(cache_key)
if cached:
return json.loads(cached)
except Exception:
pass
token = await self.get_token(chain, address)
deployer_reputation: int | None = None
if token and token.deployer_wallet_id and self._health.neo4j:
try:
with self._neo_driver.session() as s:
r = s.run(
"MATCH (d:Deployer {wallet_id: $id}) "
"RETURN d.reputation_score AS rep, d.rug_count AS rugs",
id=token.deployer_wallet_id,
).single()
if r:
deployer_reputation = r["rep"]
except Exception as e:
log.debug("risk_neo4j_fail: %s", e)
token_score = token.risk_score if token and token.risk_score is not None else 50
if deployer_reputation is not None:
score = int(0.6 * token_score + 0.4 * (100 - deployer_reputation))
else:
score = token_score
score = max(0, min(100, score))
if score < 25:
tier = "low"
elif score < 50:
tier = "medium"
elif score < 75:
tier = "high"
else:
tier = "critical"
result: dict[str, Any] = {
"score": score,
"tier": tier,
"factors": token.risk_factors if token else [],
"token": token.model_dump() if token else None,
"deployer_reputation": deployer_reputation,
"fetched_at": utcnow().isoformat(),
}
if self._health.redis:
try:
await self._redis.setex(cache_key, 60, json.dumps(result, default=str))
except Exception:
pass
return result
# ── Recipe 5: resolve_entity (Neo4j Cypher) ─────────────────────
async def resolve_entity(
self, wallet_id: str, max_chains: int = 5
) -> dict[str, Any]:
"""Cross-chain entity resolution via Neo4j.
Uses the SAME_AS / FUNDED_BY_SAME / CLONE_OF / BEHAVIORAL_MATCH
edges defined in v4.0 §T32 with path-weighted confidence.
"""
await self._init_stores()
if not self._health.neo4j:
return {"entity_id": None, "wallets": [], "note": "neo4j unavailable"}
try:
with self._neo_driver.session() as s:
result = s.run(
"""
MATCH (start:Wallet {wallet_id: $wallet_id})
OPTIONAL MATCH path = (start)-[:SAME_AS|FUNDED_BY_SAME|CLONE_OF|BEHAVIORAL_MATCH*1..3]-(other:Wallet)
WHERE start <> other
WITH start, other, path,
reduce(conf = 1.0, r IN relationships(path) | conf * r.confidence) AS path_conf
ORDER BY path_conf DESC
LIMIT $limit
RETURN start.entity_id AS entity_id,
collect({
wallet_id: other.wallet_id,
chain: other.chain,
address: other.address,
confidence: path_conf
}) AS cross_chain_wallets
""",
wallet_id=wallet_id, limit=max_chains,
).single()
if not result:
return {"entity_id": None, "wallets": [], "note": "no edges"}
wallets = [w for w in (result["cross_chain_wallets"] or []) if w.get("wallet_id")]
return {
"entity_id": result["entity_id"],
"wallets": wallets,
"note": f"{len(wallets)} cross-chain matches",
}
except Exception as e:
log.warning("resolve_entity_fail: %s", e)
return {"entity_id": None, "wallets": [], "error": str(e)}
# ── RAG bridge: wire app/rag/engine.py into the catalog ─────────
async def rag_search(
self, query: str, collection: str = "scam_intel", top_k: int = 5
) -> list[dict]:
"""Search the RAG system. Returns raw hits (catalog-agnostic).
Use this when you want RAG results as raw search hits.
Use get_token_risk / etc. when you want catalog-typed results.
"""
try:
return await rag_three_pillar_search(
query=query, collection=collection, top_k=top_k
)
except Exception as e:
log.warning("rag_search_fail: %s", e)
return []
async def rag_ingest(
self,
content: str,
collection: str = "scam_intel",
doc_id: str | None = None,
metadata: dict | None = None,
) -> dict:
"""Ingest a fact into RAG and (if it's about a token) link to Token.rag_embedding_id.
Returns the RAG ingest result plus the Qdrant point_id for cross-store ref.
"""
from uuid import uuid4
doc_id = doc_id or f"finding:{uuid4().hex[:12]}"
try:
r = await rag_ingest_document(
collection=collection,
doc_id=doc_id,
content=content,
metadata=metadata or {},
)
r["qdrant_point_id"] = doc_id # Engine returns the same
return r
except Exception as e:
log.warning("rag_ingest_fail: %s", e)
return {"status": "failed", "error": str(e)}
async def attach_rag_to_token(
self, chain: Chain, address: str, qdrant_point_id: str
) -> bool:
"""Link an existing Qdrant point to a Token row as rag_embedding_id."""
token = await self.get_token(chain, address)
if not token:
return False
token.rag_embedding_id = qdrant_point_id
return await self.save_token(token)
# ── Stats / introspection ────────────────────────────────────────
async def stats(self) -> dict[str, Any]:
await self._init_stores()
out: dict[str, Any] = {
"stores": await self.probe_stores(),
"collections": COLLECTIONS,
}
if self._health.postgres:
try:
async with self._pg_pool.acquire() as conn:
out["tokens"] = await conn.fetchval("SELECT COUNT(*) FROM tokens") or 0
out["alerts"] = await conn.fetchval("SELECT COUNT(*) FROM alerts") or 0
except Exception as e:
out["postgres_error"] = str(e)
if self._health.neo4j:
try:
with self._neo_driver.session() as s:
out["wallets"] = s.run("MATCH (w:Wallet) RETURN COUNT(w) AS c").single()["c"] or 0
out["deployers"] = s.run("MATCH (d:Deployer) RETURN COUNT(d) AS c").single()["c"] or 0
except Exception as e:
out["neo4j_error"] = str(e)
if self._health.qdrant:
try:
cols = await self._qdrant.get("/collections")
if cols.status_code == 200:
out["qdrant_collections"] = [
c["name"] for c in cols.json().get("result", {}).get("collections", [])
]
except Exception as e:
out["qdrant_error"] = str(e)
return out
# ── Singleton accessor ──────────────────────────────────────────────
_catalog: CatalogService | None = None
def get_catalog() -> CatalogService:
"""Get the global CatalogService. Lazy-init on first call."""
global _catalog
if _catalog is None:
_catalog = CatalogService()
return _catalog

78
backend/app/config.py
View file

@ -1,78 +0,0 @@
"""Application configuration.
Pydantic-settings reads from environment variables (and .env in dev).
Import: `from app.config import settings`
"""
from __future__ import annotations
from functools import lru_cache
from typing import Literal
from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
"""All runtime configuration. Add new env vars here as needed."""
model_config = SettingsConfigDict(
env_file=".env",
env_file_encoding="utf-8",
case_sensitive=False,
extra="ignore",
)
# ── Runtime ──────────────────────────────────────────────────────
environment: Literal["dev", "staging", "prod"] = "prod"
log_level: str = "INFO"
port: int = 8000
# ── Database / Cache ────────────────────────────────────────────
database_url: str = "postgresql+asyncpg://rmi:rmi@localhost/rmi"
redis_url: str = "redis://localhost:6379/0"
# ── Auth ────────────────────────────────────────────────────────
jwt_secret: str = Field(default="dev-secret-CHANGE-ME")
jwt_algorithm: str = "HS256"
jwt_expire_minutes: int = 60 * 24
# ── CORS ────────────────────────────────────────────────────────
cors_origins: list[str] = ["*"]
# ── Rate limiting ───────────────────────────────────────────────
rate_limit_per_minute: int = 100
# ── AI providers ────────────────────────────────────────────────
ollama_url: str = "http://localhost:11434"
openrouter_api_key: str = ""
huggingface_token: str = ""
# ── Langfuse v4 (observability) ─────────────────────────────────
langfuse_public_key: str = ""
langfuse_secret_key: str = ""
langfuse_host: str = "http://localhost:3002"
# ── External APIs ───────────────────────────────────────────────
coingecko_api_key: str = ""
etherscan_api_key: str = ""
birdeye_api_key: str = ""
goplus_api_key: str = ""
# ── RMI-specific ────────────────────────────────────────────────
rag_collections: list[str] = [
"scam_intel",
"deployer_history",
"wallet_labels",
"contract_audit",
"phishing_db",
]
@lru_cache(maxsize=1)
def get_settings() -> Settings:
"""Cached settings instance."""
return Settings()
# Module-level singleton for convenience.
settings = get_settings()

355
backend/app/contract_scan.py
View file

@ -1,355 +0,0 @@
"""
Deep Contract Audit / Contract Scanner
======================================
Performs static analysis, honeypot detection, and vulnerability scanning on
smart contracts across EVM and Solana chains. The free, comprehensive alternative
to paid audit services.
What it does:
1. Static Analysis — Bytecode pattern matching for known malicious opcodes
2. Honeypot Detection — Sell restrictions, blacklist/whitelist patterns
3. Ownership & Admin Checks — Owner privileges, proxy patterns, upgrade capabilities
4. Fee Analysis — Hidden taxes, dynamic fees, cooldown mechanisms
5. Liquidity Verification — LP lock status, minting controls, supply manipulation
6. Vulnerability Scanning — Reentrancy, overflow, unchecked external calls
7. Risk Scoring — Composite score with detailed findings
Signals detected:
- honeypot patterns (sell disabled, whitelist-only, blacklist on sell)
- ownership renouncement status
- proxy/upgradeable contract patterns
- mint/burn privileges
- transfer restrictions (cooldown, tax on sell)
- anti-whale mechanisms
- known scam function signatures
- flashloan attack vectors
Tier : Premium ($0.08)
Price : 80000 atoms
Endpoint: POST /api/v1/x402-tools/contract_scan
Usage:
from app.contract_scan import ContractScanner
scanner = ContractScanner()
result = await scanner.scan("0x...")
print(result.score, result.findings)
CLI:
python3 contract_scan.py 0x1234... --chain ethereum
"""
import asyncio
import json
import logging
import re
from dataclasses import dataclass, field
from enum import Enum
from typing import Any
import httpx
logger = logging.getLogger(__name__)
# ── Constants ──────────────────────────────────────────────────────────────
SOLANA_ADDR_RE = re.compile(r"^[1-9A-HJ-NP-Za-km-z]{32,44}$")
EVM_ADDR_RE = re.compile(r"^0x[a-fA-F0-9]{40}$")
EVM_CHAINS = frozenset({
"ethereum", "bsc", "polygon", "arbitrum", "optimism",
"avalanche", "base", "fantom", "linea", "zksync", "scroll",
})
SUPPORTED_CHAINS = [*EVM_CHAINS, "solana"]
# Known vulnerable opcode patterns (Solidity)
VULNERABLE_PATTERNS = {
"reentrancy": [r"call\.value", r"\.call\(", r"call\.gas\("],
"unchecked_send": [r"\.send\(", r"\.transfer\("],
"integer_overflow": [r"unchecked\s*\{", r"\+\+.*\+\+"],
"delegatecall": [r"delegatecall", r"DELEGATECALL"],
"selfdestruct": [r"selfdestruct", r"suicide\("],
"tx_origin": [r"tx\.origin"],
}
# Known honeypot function signatures
HONEYPOT_SIGS = [
"0x181f3e6", # transfer
"0xa9059cbb", # transfer
"0x23b872dd", # transferFrom
"0x095ea7b3", # approve
"0x70a08231", # balanceOf
]
# DEX APIs
DEXSCREENER_API = "https://api.dexscreener.com/latest/dex/tokens"
BIRDEYE_API = "https://public-api.birdeye.so"
# Risk thresholds
SCORE_HIGH_RISK = 70
SCORE_MEDIUM_RISK = 30
SCORE_LOW_RISK = 10
# ── Enums & Types ────────────────────────────────────────────────────────
class RiskLevel(Enum):
SAFE = "safe"
LOW = "low"
MEDIUM = "medium"
HIGH = "high"
CRITICAL = "critical"
class FindingType(Enum):
HONEYPOT = "honeypot"
OWNED = "owned"
UPGRADEABLE = "upgradeable"
MINT_PRIVILEGE = "mint_privilege"
BLACKLIST = "blacklist"
WHITELIST = "whitelist"
HIGH_TAX = "high_tax"
COOLDOWN = "cooldown"
ANTI_WHALE = "anti_whale"
REENTRANCY = "reentrancy"
DELEGATECALL = "delegatecall"
SELFDESTRUCT = "selfdestruct"
# ── Data Models ───────────────────────────────────────────────────────────
@dataclass
class Finding:
"""A single security finding."""
finding_type: FindingType
severity: RiskLevel
description: str
evidence: str = ""
def to_dict(self) -> dict[str, Any]:
return {
"type": self.finding_type.value,
"severity": self.severity.value,
"description": self.description,
"evidence": self.evidence,
}
@dataclass
class ContractScanResult:
"""Result of contract scan."""
contract_address: str
chain: str
score: float # 0-100, higher = riskier
risk_level: RiskLevel
findings: list[Finding] = field(default_factory=list)
bytecode_length: int = 0
is_proxy: bool = False
is_verified: bool = False
constructor_args: str = ""
proxy_implementation: str = ""
def to_dict(self) -> dict[str, Any]:
return {
"contract_address": self.contract_address,
"chain": self.chain,
"score": self.score,
"risk_level": self.risk_level.value,
"findings": [f.to_dict() for f in self.findings],
"bytecode_length": self.bytecode_length,
"is_proxy": self.is_proxy,
"is_verified": self.is_verified,
"proxy_implementation": self.proxy_implementation,
}
# ── Main Scanner Class ───────────────────────────────────────────────────
class ContractScanner:
"""Deep contract scanner for security analysis."""
def __init__(self, alchemy_key: str | None = None):
self.alchemy_key = alchemy_key or None
self.client = httpx.AsyncClient(timeout=30.0)
async def scan(
self,
contract_address: str,
chain: str = "ethereum",
include_source: bool = False,
) -> ContractScanResult:
"""Scan a contract for security issues."""
chain = chain.lower()
if chain not in SUPPORTED_CHAINS:
raise ValueError(f"Unsupported chain: {chain}")
result = ContractScanResult(
contract_address=contract_address,
chain=chain,
score=0.0,
risk_level=RiskLevel.SAFE,
)
# Fetch contract bytecode
bytecode = await self._get_bytecode(contract_address, chain)
if bytecode:
result.bytecode_length = len(bytecode)
await self._analyze_bytecode(bytecode, result)
# Check verification and proxy status
result.is_verified, result.is_proxy, result.proxy_implementation = await self._get_contract_metadata(contract_address, chain)
if result.is_proxy:
result.findings.append(Finding(
finding_type=FindingType.UPGRADEABLE,
severity=RiskLevel.HIGH,
description="Contract is upgradeable via proxy pattern",
evidence=f"Implementation: {result.proxy_implementation[:20]}..."
))
# Check liquidity
await self._check_liquidity(contract_address, chain, result)
# Calculate final score
result.score = self._calculate_score(result.findings)
result.risk_level = self._get_risk_level(result.score)
return result
async def _get_bytecode(self, address: str, chain: str) -> str | None:
"""Fetch contract bytecode from chain RPC."""
if chain in EVM_CHAINS:
try:
# Use public RPC
rpc_url = f"https://{chain}.llamarpc.com"
payload = {
"jsonrpc": "2.0",
"method": "eth_getCode",
"params": [address, "latest"],
"id": 1,
}
resp = await self.client.post(rpc_url, json=payload)
data = resp.json()
return data.get("result", "")
except Exception as e:
logger.warning(f"Failed to fetch bytecode: {e}")
return None
async def _get_contract_metadata(self, address: str, chain: str) -> tuple[bool, bool, str]:
"""Get contract verification and proxy status."""
try:
# Etherscan/DexScreener
url = f"https://api.dexscreener.com/latest/dex/tokens/{address}"
resp = await self.client.get(url)
if resp.status_code == 200:
data = resp.json()
# Check for proxy implementation
proxy_impl = data.get("proxy", {}).get("implementation", "")
return True, bool(proxy_impl), proxy_impl
except Exception as e:
logger.debug(f"Metadata fetch failed: {e}")
return False, False, ""
async def _analyze_bytecode(self, bytecode: str, result: ContractScanResult) -> None:
"""Analyze bytecode for security patterns."""
# Convert to lowercase for matching
bc_lower = bytecode.lower()
# Check honeypot signatures
for sig in HONEYPOT_SIGS:
if sig not in bc_lower:
continue
# If transfer sig exists but sell patterns missing, could be honeypot
if sig == "0xa9059cbb": # transfer
# Check for sell restrictions
if "0x70a08231" not in bc_lower: # balanceOf missing
result.findings.append(Finding(
finding_type=FindingType.HONEYPOT,
severity=RiskLevel.CRITICAL,
description="Potential honeypot: transfer function without balance query"
))
# Check vulnerable patterns
for vuln_type, patterns in VULNERABLE_PATTERNS.items():
for pattern in patterns:
if re.search(pattern, bytecode, re.IGNORECASE):
ft = FindingType.HONEYPOT if vuln_type == "honeypot" else (
FindingType.DELEGATECALL if vuln_type == "delegatecall" else
FindingType.REENTRANCY if vuln_type == "reentrancy" else
FindingType.SELFDESTRUCT if vuln_type == "selfdestruct" else None
)
if ft:
result.findings.append(Finding(
finding_type=ft,
severity=RiskLevel.HIGH,
description=f"Vulnerable pattern detected: {vuln_type}",
evidence=f"Pattern: {pattern}"
))
async def _check_liquidity(self, address: str, chain: str, result: ContractScanResult) -> None:
"""Check liquidity status on DEXs."""
try:
url = f"{DEXSCREENER_API}/{address}"
resp = await self.client.get(url)
if resp.status_code != 200:
return
data = resp.json()
pairs = data.get("pairs", [])
if not pairs:
result.findings.append(Finding(
finding_type=FindingType.HIGH_TAX,
severity=RiskLevel.MEDIUM,
description="No liquidity pairs found on DEXs"
))
except Exception as e:
logger.debug(f"Liquidity check failed: {e}")
def _calculate_score(self, findings: list[Finding]) -> float:
"""Calculate risk score from findings."""
score = 0.0
weights = {
RiskLevel.CRITICAL: 25,
RiskLevel.HIGH: 15,
RiskLevel.MEDIUM: 8,
RiskLevel.LOW: 3,
}
for finding in findings:
score += weights.get(finding.severity, 0)
return min(100, score)
def _get_risk_level(self, score: float) -> RiskLevel:
"""Convert score to risk level."""
if score >= SCORE_HIGH_RISK:
return RiskLevel.CRITICAL
if score >= SCORE_MEDIUM_RISK:
return RiskLevel.HIGH
if score >= SCORE_LOW_RISK:
return RiskLevel.MEDIUM
return RiskLevel.SAFE
async def close(self) -> None:
"""Close HTTP client."""
await self.client.aclose()
# ── CLI Interface ────────────────────────────────────────────────────────
async def main():
"""CLI entry point."""
import argparse
p = argparse.ArgumentParser(description="Deep Contract Scanner")
p.add_argument("address", help="Contract address")
p.add_argument("--chain", default="ethereum", help="Chain name")
args = p.parse_args()
scanner = ContractScanner()
try:
result = await scanner.scan(args.address, args.chain)
print(json.dumps(result.to_dict(), indent=2))
finally:
await scanner.close()
if __name__ == "__main__":
asyncio.run(main())

31
backend/app/core/__init__.py
View file

@ -1,31 +0,0 @@
"""Cross-cutting concerns. NO business logic."""
from app.core.agent_memory import get_conversation, get_user_context, store_conversation
from app.core.ai_stream import StreamRequest, ai_route, list_providers, stream_ai
from app.core.auth import AuthMiddleware, is_public_path
from app.core.cerebras_provider import cerebras_chat
from app.core.config import Settings, get_settings
from app.core.cost_tracker import cheapest_for_task, get_cheapest_model, get_rates, get_usage, log_usage
from app.core.databus_extras import provider_dashboard, task_queue_stats
from app.core.db import get_supabase, get_supabase_sync
from app.core.db_pool import get_postgres, pool_stats, return_postgres
from app.core.errors import AppError, AuthError, NotFoundError, RateLimitError, register_error_handlers
from app.core.http import close_http_client
from app.core.lifespan import lifespan
from app.core.llm_cache import get_cache_stats, get_cached, set_cached
from app.core.logging import get_logger, setup_logging
from app.core.metrics import setup_metrics
from app.core.middleware import cache_middleware, emergency_lockdown_middleware, hsts_middleware, payload_size_limit_middleware, request_id_middleware, secure_cookie_middleware
from app.core.mistral_provider import mistral_chat, mistral_embed, mistral_moderate
from app.core.model_eval import compare_models, evaluate_model
from app.core.model_router import RoutingDecision, TaskType, route_task, smart_route
from app.core.prompt_registry import get_prompt, get_prompt_info, list_prompts, load_all_prompts, reload_prompts, render_prompt
from app.core.rate_limiter import Tier, UpgradeRequest, check_rate_limit, get_tiers, get_user_tier, my_tier, payment_links, upgrade_tier
from app.core.redis import get_redis, get_redis_async, invalidate_redis
from app.core.signal_generator import fetch_trending, main, publish_signal, scan_and_signal
from app.core.task_queue import enqueue, get_queue_stats, process_tasks, register_task
from app.core.tracing import end_span, setup_tracing, start_span
from app.core.tron_provider import tron_balance, tron_transactions
from app.core.websocket import active_connections, broadcast_alert, broadcast_scan, register_connection, unregister_connection
__all__ = ['AppError', 'AuthError', 'AuthMiddleware', 'NotFoundError', 'RateLimitError', 'RoutingDecision', 'Settings', 'StreamRequest', 'TaskType', 'Tier', 'UpgradeRequest', 'active_connections', 'ai_route', 'broadcast_alert', 'broadcast_scan', 'cache_middleware', 'cerebras_chat', 'cheapest_for_task', 'check_rate_limit', 'close_http_client', 'compare_models', 'emergency_lockdown_middleware', 'end_span', 'enqueue', 'evaluate_model', 'fetch_trending', 'get_cache_stats', 'get_cached', 'get_cheapest_model', 'get_conversation', 'get_logger', 'get_postgres', 'get_prompt', 'get_prompt_info', 'get_queue_stats', 'get_rates', 'get_redis', 'get_redis', 'get_redis_async', 'get_settings', 'get_supabase', 'get_supabase_sync', 'get_tiers', 'get_usage', 'get_user_context', 'get_user_tier', 'hsts_middleware', 'invalidate_redis', 'is_public_path', 'lifespan', 'list_prompts', 'list_providers', 'load_all_prompts', 'log_usage', 'main', 'mistral_chat', 'mistral_embed', 'mistral_moderate', 'my_tier', 'payload_size_limit_middleware', 'payment_links', 'pool_stats', 'process_tasks', 'provider_dashboard', 'publish_signal', 'register_connection', 'register_error_handlers', 'register_task', 'reload_prompts', 'render_prompt', 'request_id_middleware', 'return_postgres', 'route_task', 'scan_and_signal', 'secure_cookie_middleware', 'set_cached', 'setup_logging', 'setup_metrics', 'setup_tracing', 'smart_route', 'start_span', 'store_conversation', 'stream_ai', 'task_queue_stats', 'tron_balance', 'tron_transactions', 'unregister_connection', 'upgrade_tier']

98
backend/app/core/agent_memory.py
View file

@ -1,98 +0,0 @@
"""#9 — Agent Memory Layer. Stores conversation history in Memgraph for long-term agent memory.
Enables agents to remember past interactions across sessions."""
import os
from datetime import UTC, datetime
from fastapi import APIRouter
MEMGRAPH_URI = os.getenv("MEMGRAPH_URI", "bolt://localhost:7687")
MEMGRAPH_USER = os.getenv("MEMGRAPH_USER", "")
MEMGRAPH_PASS = os.getenv("MEMGRAPH_PASSWORD", "")
router = APIRouter(prefix="/api/v1/memory", tags=["agent-memory"])
async def _run_query(query: str, params: dict = None) -> list:
"""Run a Cypher query against Memgraph."""
import requests
try:
r = requests.post(
"http://localhost:7444/db/memgraph/query",
json={"query": query, "parameters": params or {}},
headers={"Content-Type": "application/json"},
timeout=10,
)
if r.status_code == 200:
return r.json().get("data", [])
except Exception:
pass
return []
@router.post("/conversation")
async def store_conversation(user_id: str, agent_id: str, message: str, role: str = "user"):
"""Store a conversation message in agent memory graph."""
query = """
MERGE (u:User {id: $user_id})
MERGE (a:Agent {id: $agent_id})
MERGE (c:Conversation {id: $conv_id})
ON CREATE SET c.created_at = $timestamp
CREATE (m:Message {
role: $role,
content: $message,
timestamp: $timestamp
})
MERGE (u)-[:PARTICIPATES_IN]->(c)
MERGE (a)-[:PARTICIPATES_IN]->(c)
MERGE (c)-[:HAS_MESSAGE]->(m)
MERGE (m)-[:SENT_BY]->(CASE WHEN $role = 'user' THEN u ELSE a END)
"""
conv_id = f"{user_id}:{agent_id}"
await _run_query(
query,
{
"user_id": user_id,
"agent_id": agent_id,
"conv_id": conv_id,
"role": role,
"message": message,
"timestamp": datetime.now(UTC).isoformat(),
},
)
return {"stored": True, "conversation_id": conv_id}
@router.get("/conversation/{user_id}/{agent_id}")
async def get_conversation(user_id: str, agent_id: str, limit: int = 20):
"""Retrieve conversation history for an agent."""
query = """
MATCH (u:User {id: $user_id})-[:PARTICIPATES_IN]->(c:Conversation)<-[:PARTICIPATES_IN]-(a:Agent {id: $agent_id})
MATCH (c)-[:HAS_MESSAGE]->(m:Message)
RETURN m.role as role, m.content as content, m.timestamp as timestamp
ORDER BY m.timestamp DESC LIMIT $limit
"""
rows = await _run_query(query, {"user_id": user_id, "agent_id": agent_id, "limit": limit})
return {
"user_id": user_id,
"agent_id": agent_id,
"messages": [{"role": r[0], "content": r[1], "timestamp": r[2]} for r in reversed(rows)] if rows else [],
"count": len(rows) if rows else 0,
}
@router.get("/context/{user_id}")
async def get_user_context(user_id: str):
"""Get all agent conversations + preferences for a user."""
query = """
MATCH (u:User {id: $user_id})-[:PARTICIPATES_IN]->(c:Conversation)
MATCH (a:Agent)-[:PARTICIPATES_IN]->(c)
RETURN a.id as agent, count(*) as messages
ORDER BY messages DESC
"""
rows = await _run_query(query, {"user_id": user_id})
return {
"user_id": user_id,
"agents": [{"agent": r[0], "messages": r[1]} for r in rows] if rows else [],
}

80
backend/app/core/ai_stream.py
View file

@ -1,80 +0,0 @@
"""SSE Response Streaming — real-time token streaming for AI endpoints."""
import asyncio
import json
from fastapi import APIRouter
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from app.core.model_router import route_task, TaskType
router = APIRouter(prefix="/api/v1/ai", tags=["ai-streaming"])
class StreamRequest(BaseModel):
prompt: str
task: str = "fast" # fast|cheap|complex|bulk|code
prefer: str = "fast" # fast|cheap
async def _stream_ollama(prompt: str, model: str):
"""Stream from Ollama."""
import httpx
async with httpx.AsyncClient(timeout=120) as c:
async with c.stream("POST", "http://localhost:11434/api/generate", json={
"model": model, "prompt": prompt
}) as r:
async for line in r.aiter_lines():
if line:
try:
chunk = json.loads(line)
if chunk.get("done"):
yield f"data: {json.dumps({'done': True, 'model': model})}\n\n"
break
yield f"data: {json.dumps({'token': chunk.get('response', '')})}\n\n"
except json.JSONDecodeError:
continue
@router.post("/stream")
async def stream_ai(req: StreamRequest):
"""Stream AI response in real-time. Tokens appear as generated."""
decision = route_task(TaskType(req.task), req.prefer)
if decision.provider == "ollama":
return StreamingResponse(
_stream_ollama(req.prompt, decision.model),
media_type="text/event-stream",
headers={"X-Model": decision.model, "X-Provider": decision.provider, "X-Latency-Ms": str(decision.estimated_latency_ms)}
)
# For non-Ollama providers, fall back to blocking + stream result
async def _blocking_stream():
from app.core.model_router import smart_route
result = await smart_route(req.prompt, req.task, req.prefer)
if result and "response" in result:
words = result["response"].split()
for word in words:
yield f"data: {json.dumps({'token': word + ' '})}\n\n"
await asyncio.sleep(0.05)
yield f"data: {json.dumps({'done': True, 'model': decision.model})}\n\n"
return StreamingResponse(
_blocking_stream(),
media_type="text/event-stream",
headers={"X-Model": decision.model, "X-Provider": decision.provider}
)
@router.post("/route")
async def ai_route(req: StreamRequest):
"""Non-streaming: auto-route to best model, return complete response."""
from app.core.model_router import smart_route
result = await smart_route(req.prompt, req.task, req.prefer)
return result if result else {"error": "All providers failed"}
@router.get("/providers")
async def list_providers():
"""List all available AI providers with capabilities."""
from app.core.model_router import ROUTING_TABLE
return {
"providers": {
task.value: [{"model": m[0], "provider": m[1], "latency_ms": m[2]*1000, "cost_per_1M_input": m[3]}
for m in models]
for task, models in ROUTING_TABLE.items()
}
}

98
backend/app/core/auth.py
View file

@ -1,98 +0,0 @@
"""RMI Backend — Auth middleware, API key verification, and JWT user identity.
This module is the single source of truth for auth. Routes import
`get_current_user` / `get_optional_user` from here (re-exported via
app.api.deps for convenience).
"""
from __future__ import annotations
import os
from typing import Any
from fastapi import Request
from fastapi.responses import JSONResponse
from starlette.middleware.base import BaseHTTPMiddleware
RMI_AUTH_TOKEN = os.getenv("RMI_AUTH_TOKEN", "")
PUBLIC_WRITE_PREFIXES = [
"/api/v1/auth/",
"/api/v1/x402/",
"/api/v1/x402-tools/",
"/api/v1/x402-databus/",
"/api/v1/databus/",
"/api/v1/alerts/",
"/api/v1/admin/",
"/api/v1/content/",
"/api/v1/bulletin/",
"/api/v1/rag/permanence/",
"/api/v1/token/",
"/api/v1/ai/",
"/api/v1/premium/",
"/api/v1/rag/",
"/api/v1/protect/",
"/api/v1/wallet-manager/",
]
PUBLIC_GET_PREFIXES = [
"/api/v1/token/",
"/api/v1/databus/",
"/api/v1/alerts/",
"/api/v1/x402-databus/",
"/api/v1/x402-tools/",
]
def is_public_path(path: str, method: str) -> bool:
"""Check if a path is publicly accessible without auth."""
if path in ("/health", "/ready", "/docs", "/openapi.json", "/redoc", "/", "/favicon.ico"):
return True
if path.startswith("/ws/") or not path.startswith("/api/"):
return True
if method in ("POST", "PUT", "DELETE", "PATCH"):
return any(path.startswith(p) for p in PUBLIC_WRITE_PREFIXES)
return True # GET/HEAD always public
class AuthMiddleware(BaseHTTPMiddleware):
async def dispatch(self, request: Request, call_next):
path = request.url.path
method = request.method
if is_public_path(path, method):
return await call_next(request)
api_key = request.headers.get("X-API-Key", "")
if RMI_AUTH_TOKEN and api_key != RMI_AUTH_TOKEN:
return JSONResponse(
status_code=401,
content={"detail": "Unauthorized - valid X-API-Key header required for write operations"},
)
return await call_next(request)
# ── JWT user identity (FastAPI dependencies) ─────────────────────────────
# Delegates to the legacy app.auth JWT logic during strangelfig migration.
# Once legacy auth.py is migrated to the new pattern, these become the
# canonical implementation. Until then they reuse the working logic so
# new routes (like app/api/v1/auth/alerts.py) can use modern Depends().
async def get_optional_user(request: Request) -> dict[str, Any] | None:
"""Return the authenticated user dict, or None if not authenticated.
Reads the Authorization: Bearer <jwt> header. Returns the user dict
(id, email, tier, role) on success, None if no/invalid token.
Use this for endpoints that work with OR without auth.
"""
from app.auth import get_current_user as _legacy_get_user # local import to avoid cycles
return await _legacy_get_user(request)
async def get_current_user(request: Request) -> dict[str, Any]:
"""Require an authenticated user. Raises 401 if missing.
Use this for endpoints that REQUIRE auth.
"""
from app.auth import require_auth as _legacy_require # local import to avoid cycles
return await _legacy_require(request)

49
backend/app/core/cerebras_provider.py
View file

@ -1,49 +0,0 @@
"""Cerebras provider — GPT-OSS-120B, fastest inference on Earth (9ms).
Free tier: 14,400 req/day, 1M tokens/day."""
import logging
import os
import httpx
logger = logging.getLogger(__name__)
CEREBRAS_KEY = os.getenv("CEREBRAS_API_KEY", "")
BASE = "https://api.cerebras.ai/v1"
async def cerebras_chat(
prompt: str, system: str = None, temperature: float = 0.7, max_tokens: int = 1024
) -> dict | None:
"""GPT-OSS-120B via Cerebras — 9ms latency. Use for real-time, latency-sensitive tasks."""
if not CEREBRAS_KEY:
return None
messages = []
if system:
messages.append({"role": "system", "content": system})
messages.append({"role": "user", "content": prompt})
try:
async with httpx.AsyncClient(timeout=15) as c:
r = await c.post(
f"{BASE}/chat/completions",
json={
"model": "gpt-oss-120b",
"messages": messages,
"max_tokens": max_tokens,
"temperature": temperature,
},
headers={"Authorization": f"Bearer {CEREBRAS_KEY}"},
)
if r.status_code == 200:
d = r.json()
choice = d["choices"][0]["message"]
content = choice.get("content") or choice.get("reasoning", "")
return {
"response": content.strip(),
"model": "gpt-oss-120b",
"tokens": d.get("usage", {}).get("total_tokens", 0),
"latency_ms": round(d.get("time_info", {}).get("total_time", 0) * 1000),
"provider": "cerebras",
}
except Exception as e:
logger.warning(f"Cerebras failed: {e}")
return None

81
backend/app/core/config.py
View file

@ -1,81 +0,0 @@
"""
Central configuration — single source of truth for all settings.
Loaded from .env via pydantic-settings.
Usage:
from app.core.config import settings
print(settings.REDIS_HOST)
"""
from __future__ import annotations
import os
from functools import lru_cache
from typing import Optional
class Settings:
"""Application settings loaded from environment variables."""
def __init__(self) -> None:
# Redis
self.REDIS_HOST: str = os.getenv("REDIS_HOST", "rmi-redis")
self.REDIS_PORT: int = int(os.getenv("REDIS_PORT", "6379"))
self.REDIS_PASSWORD: str = os.getenv("REDIS_PASSWORD", "")
self.REDIS_DB: int = int(os.getenv("REDIS_DB", "0"))
# Supabase
self.SUPABASE_URL: str = os.getenv("SUPABASE_URL", "")
self.SUPABASE_SERVICE_KEY: str = os.getenv("SUPABASE_SERVICE_KEY", "") or os.getenv("SUPABASE_KEY", "")
# Admin
self.ADMIN_API_KEY: str = os.getenv("ADMIN_API_KEY", "dev-key-change-me")
# API Keys
self.OPENROUTER_API_KEY: str = os.getenv("OPENROUTER_API_KEY", "")
self.ETHERSCAN_API_KEY: str = os.getenv("ETHERSCAN_API_KEY", "")
self.HELIUS_API_KEY: str = os.getenv("HELIUS_API_KEY", "")
self.DEEPSEEK_API_KEY: str = os.getenv("DEEPSEEK_API_KEY", "")
self.GEMINI_API_KEY: str = os.getenv("GEMINI_API_KEY", "")
self.GEMINI_API_KEY_2: str = os.getenv("GEMINI_API_KEY_2", "")
# Langfuse
self.LANGFUSE_PUBLIC_KEY: str = os.getenv("LANGFUSE_PUBLIC_KEY", "")
self.LANGFUSE_SECRET_KEY: str = os.getenv("LANGFUSE_SECRET_KEY", "")
self.LANGFUSE_HOST: str = os.getenv("LANGFUSE_HOST", "http://langfuse-langfuse-web-1:3000")
# Ollama
self.OLLAMA_HOST: str = os.getenv("OLLAMA_HOST", "http://ollama:11434")
# Wallet
self.WALLET_VAULT_PASSWORD: str = os.getenv("WALLET_VAULT_PASSWORD", "")
# R2
self.R2_API_TOKEN: str = os.getenv("R2_API_TOKEN", "")
self.R2_ACCOUNT_ID: str = os.getenv("R2_ACCOUNT_ID", "")
self.R2_BUCKET: str = os.getenv("R2_BUCKET", "rag-backup")
def validate(self) -> list[str]:
"""Return list of missing critical env vars. Empty list = all good."""
critical = [
"REDIS_HOST",
"SUPABASE_URL",
"SUPABASE_SERVICE_KEY",
"ADMIN_API_KEY",
"WALLET_VAULT_PASSWORD",
]
missing = []
for var in critical:
if not getattr(self, var, None):
missing.append(var)
return missing
@lru_cache(maxsize=1)
def get_settings() -> Settings:
"""Cached singleton — call this, don't instantiate Settings directly."""
return Settings()
# Module-level singleton — import this everywhere
settings = get_settings()

111
backend/app/core/cost_tracker.py
View file

@ -1,111 +0,0 @@
"""#10 — Cost-Per-Model Tracking. Tracks $/1K tokens per model/provider.
Auto-routes to cheapest model that meets quality threshold."""
from datetime import UTC, datetime
from fastapi import APIRouter, Query
router = APIRouter(prefix="/api/v1/costs", tags=["cost-tracking"])
# Cost per 1M tokens (USD) — updated June 2026
MODEL_COSTS = {
"deepseek-v4-flash": {"input": 0.14, "output": 0.28, "provider": "deepseek"},
"deepseek-v4-pro": {"input": 0.55, "output": 2.19, "provider": "deepseek"},
# Gemini pricing (paid tier, per 1M tokens)
"gemini-2.5-flash": {"input": 0.15, "output": 0.60, "provider": "gemini"},
"gemini-2.5-pro": {"input": 1.25, "output": 10.00, "provider": "gemini"},
"gemini-3.5-flash": {"input": 1.50, "output": 9.00, "provider": "gemini"},
"mistral-small-latest": {"input": 0.0, "output": 0.0, "provider": "mistral", "note": "Free tier — 1B tokens/mo"},
"mistral-medium-latest": {"input": 0.0, "output": 0.0, "provider": "mistral", "note": "Free tier — use sparingly"},
"mistral-embed": {
"input": 0.0,
"output": 0.0,
"provider": "mistral",
"note": "Free tier — state of art embeddings",
},
"mistral-large": {"input": 2.00, "output": 6.00, "provider": "mistral"},
"mistral-small": {"input": 0.20, "output": 0.60, "provider": "mistral"},
"qwen2.5-coder:7b": {"input": 0.0, "output": 0.0, "provider": "ollama"},
"gpt-oss-120b": {
"input": 0.0,
"output": 0.0,
"provider": "cerebras",
"note": "Free tier — 14.4K req/day, 9ms latency",
},
"mistral:7b": {"input": 0.0, "output": 0.0, "provider": "ollama"},
"bge-m3": {"input": 0.0, "output": 0.0, "provider": "ollama"},
}
_usage_log: list[dict] = []
def log_usage(model: str, input_tokens: int, output_tokens: int, latency_ms: float):
"""Log model usage for cost tracking."""
costs = MODEL_COSTS.get(model, {"input": 0, "output": 0, "provider": "unknown"})
input_cost = (input_tokens / 1_000_000) * costs["input"]
output_cost = (output_tokens / 1_000_000) * costs["output"]
total_cost = input_cost + output_cost
_usage_log.append(
{
"timestamp": datetime.now(UTC).isoformat(),
"model": model,
"provider": costs["provider"],
"input_tokens": input_tokens,
"output_tokens": output_tokens,
"cost_usd": round(total_cost, 6),
"latency_ms": latency_ms,
}
)
def get_cheapest_model(models: list[str]) -> str:
"""Pick the cheapest model from a list that meets quality."""
best = None
best_cost = float("inf")
for m in models:
c = MODEL_COSTS.get(m, {})
cost = c.get("output", 1.0)
if cost < best_cost:
best_cost = cost
best = m
return best or models[0]
@router.get("/rates")
async def get_rates():
"""Get current model pricing."""
return {"models": MODEL_COSTS, "updated": "2026-06-15"}
@router.get("/usage")
async def get_usage(limit: int = Query(50, le=200)):
"""Get recent usage log."""
recent = _usage_log[-limit:]
total_cost = sum(e["cost_usd"] for e in recent)
total_tokens = sum(e["input_tokens"] + e["output_tokens"] for e in recent)
return {
"total_cost_usd": round(total_cost, 4),
"total_tokens": total_tokens,
"entries": len(recent),
"log": recent,
}
@router.get("/cheapest")
async def cheapest_for_task(quality: str = Query("medium", description="minimum quality tier")):
"""Get cheapest model for a given quality tier."""
if quality == "high":
candidates = ["deepseek-v4-pro", "gemini-2.5-pro", "mistral-large"]
elif quality == "medium":
candidates = ["deepseek-v4-flash", "gemini-2.5-flash", "mistral-small", "qwen2.5-coder:7b"]
else:
candidates = ["qwen2.5-coder:7b", "mistral:7b", "deepseek-v4-flash"]
cheapest = get_cheapest_model(candidates)
return {
"quality_tier": quality,
"candidates": candidates,
"cheapest": cheapest,
"cost_per_1M_output": MODEL_COSTS.get(cheapest, {}).get("output", "?"),
}

48
backend/app/core/databus_extras.py
View file

@ -1,48 +0,0 @@
"""#2 SSE Streaming + #3 Provider Dashboard endpoints."""
from fastapi import APIRouter
import httpx
import os
router = APIRouter(prefix="/api/v1/databus", tags=["databus-extras"])
BACKEND = os.getenv("BACKEND_URL", "http://localhost:8000")
@router.get("/providers/dashboard")
async def provider_dashboard():
"""Real-time provider health dashboard data — feed into Grafana."""
try:
async with httpx.AsyncClient(timeout=10) as c:
r = await c.get(f"{BACKEND}/api/v1/databus/providers/health", headers={"X-RMI-Key": os.getenv("RMI_INTERNAL_KEY", "rmi-internal-2026")})
if r.status_code == 200:
data = r.json()
providers = data.get("providers", data)
# Format for Grafana
panels = []
for name, health in (providers.items() if isinstance(providers, dict) else []):
panels.append({
"provider": name,
"status": "healthy" if health.get("healthy", True) else "degraded",
"latency_ms": health.get("avg_latency_ms", 0),
"error_rate": health.get("error_rate", 0),
"circuit": health.get("circuit_state", "closed"),
})
return {
"providers": panels,
"summary": {
"total": len(panels),
"healthy": sum(1 for p in panels if p["status"] == "healthy"),
"degraded": sum(1 for p in panels if p["status"] != "healthy"),
}
}
except Exception:
pass
return {"providers": [], "note": "Provider health API unavailable — check backend"}
@router.get("/queue/stats")
async def task_queue_stats():
"""Background task queue statistics."""
from app.core.task_queue import get_queue_stats
return await get_queue_stats()

60
backend/app/core/db.py
View file

@ -1,60 +0,0 @@
"""
Supabase client singleton — single source of truth.
Replaces scattered supabase client creation across routers.
Usage:
from app.core.db import get_supabase
client = await get_supabase()
result = await client.table("alerts").select("*").execute()
"""
from __future__ import annotations
import logging
import os
from typing import Any
logger = logging.getLogger(__name__)
_client: Any = None
async def get_supabase() -> Any:
"""Get or create the async Supabase client."""
global _client
if _client is not None:
return _client
url = os.getenv("SUPABASE_URL", "")
key = os.getenv("SUPABASE_SERVICE_KEY", "") or os.getenv("SUPABASE_KEY", "")
if not url or not key:
logger.warning("supabase_not_configured")
return None
try:
from supabase import create_client as _create_sync
_client = _create_sync(url, key)
logger.info("supabase_connected", url=url[:30])
return _client
except ImportError:
logger.error("supabase_package_missing — pip install supabase")
return None
except Exception as e:
logger.error("supabase_init_failed", error=str(e))
return None
def get_supabase_sync() -> Any:
"""Synchronous Supabase client for scripts and non-async contexts."""
url = os.getenv("SUPABASE_URL", "")
key = os.getenv("SUPABASE_SERVICE_KEY", "") or os.getenv("SUPABASE_KEY", "")
if not url or not key:
return None
try:
from supabase import create_client
return create_client(url, key)
except (ImportError, Exception):
return None

60
backend/app/core/db_pool.py
View file

@ -1,60 +0,0 @@
"""Database connection pooling — Redis + Postgres with auto-reconnect."""
import logging
import os
logger = logging.getLogger(__name__)
REDIS_URL = os.getenv("REDIS_URL", "redis://localhost:6379/0")
PG_URL = os.getenv("DATABASE_URL", "postgresql://postgres:postgres@localhost:5432/postgres")
_redis_pool = None
_pg_pool = None
def get_redis():
"""Get or create Redis connection pool. Auto-reconnects."""
global _redis_pool
if _redis_pool is None:
import redis
try:
_redis_pool = redis.ConnectionPool.from_url(
REDIS_URL, max_connections=20, retry_on_timeout=True, health_check_interval=30
)
logger.info("Redis pool created (max 20)")
except Exception as e:
logger.error(f"Redis pool failed: {e}")
return None
import redis
return redis.Redis(connection_pool=_redis_pool)
def get_postgres():
"""Get or create Postgres connection pool."""
global _pg_pool
if _pg_pool is None:
try:
from psycopg2 import pool
_pg_pool = pool.ThreadedConnectionPool(5, 20, PG_URL)
logger.info("Postgres pool created (5-20)")
except Exception as e:
logger.error(f"Postgres pool failed: {e}")
return None
return _pg_pool.getconn()
def return_postgres(conn):
"""Return connection to pool."""
if _pg_pool and conn:
_pg_pool.putconn(conn)
def pool_stats() -> dict:
"""Get connection pool statistics."""
return {
"redis": {"pool_size": 20, "active": "unknown"} if _redis_pool else {"error": "no pool"},
"postgres": {"min": 5, "max": 20} if _pg_pool else {"error": "no pool"},
}

301
backend/app/core/errors.py
View file

@ -1,301 +0,0 @@
"""Domain-specific error hierarchy.
Each domain defines its own error subclasses with HTTP status + code.
The FastAPI exception handler (in core.errors) returns a consistent
ErrorResponse envelope.
Why domain errors over a generic AppError:
- OpenAPI schema documents each error code (clients can switch on code).
- Self-documenting: a wallet domain error says "WalletNotFound" not
"AppError with code 404".
- Each domain owns its error vocabulary. New errors don't require
touching the global error catalog.
- The legacy code has scattered HTTPException raises. Domain errors
replace these with typed exceptions that the handler converts.
Migration: existing code that raises AppError/NotFoundError/AuthError
keeps working (those are base classes). New code should use
domain-specific subclasses.
"""
from __future__ import annotations
from typing import Any, Optional
class AppError(Exception):
"""Base for all RMI errors. Each subclass declares its HTTP status."""
status_code: int = 500
code: str = "internal_error"
def __init__(
self,
message: str = "",
*,
details: Optional[dict[str, Any]] = None,
) -> None:
super().__init__(message)
self.message = message or self.code
self.details = details or {}
def to_dict(self) -> dict[str, Any]:
return {
"code": self.code,
"message": self.message,
"details": self.details,
}
# ── Generic categories (subdomain-agnostic) ───────────────────────────
class NotFoundError(AppError):
status_code = 404
code = "not_found"
class AuthError(AppError):
status_code = 401
code = "unauthorized"
class ForbiddenError(AppError):
status_code = 403
code = "forbidden"
class RateLimitError(AppError):
status_code = 429
code = "rate_limited"
class ValidationError(AppError):
status_code = 400
code = "validation_error"
class ConflictError(AppError):
status_code = 409
code = "conflict"
class UpstreamError(AppError):
"""External API failed (chain RPC, databus provider, etc)."""
status_code = 502
code = "upstream_error"
# ── Domain-specific errors ────────────────────────────────────────────
class WalletError(AppError):
"""Base for all wallet-domain errors."""
code = "wallet_error"
class WalletNotFoundError(WalletError, NotFoundError):
code = "wallet_not_found"
def __init__(self, address: str, chain: str = "unknown") -> None:
super().__init__(
f"Wallet {address[:12]}... not found",
details={"address": address, "chain": chain},
)
class InsufficientFundsError(WalletError):
status_code = 402
code = "insufficient_funds"
def __init__(self, required: float, available: float, asset: str = "native") -> None:
super().__init__(
f"Insufficient {asset}: need {required}, have {available}",
details={"required": required, "available": available, "asset": asset},
)
class TokenError(AppError):
"""Base for all token-domain errors."""
code = "token_error"
class TokenNotScannedError(TokenError, NotFoundError):
code = "token_not_scanned"
def __init__(self, address: str, chain: str = "unknown") -> None:
super().__init__(
f"Token {address[:12]}... has not been scanned",
details={"address": address, "chain": chain},
)
class HoneypotDetectedError(TokenError):
status_code = 422
code = "honeypot_detected"
def __init__(self, address: str, chain: str = "unknown") -> None:
super().__init__(
f"Token {address[:12]}... is a honeypot — cannot trade",
details={"address": address, "chain": chain},
)
class ScanError(TokenError, UpstreamError):
code = "scan_failed"
def __init__(self, address: str, reason: str) -> None:
super().__init__(
f"Scan failed for {address[:12]}...: {reason}",
details={"address": address, "reason": reason},
)
class AlertError(AppError):
"""Base for alert-domain errors."""
code = "alert_error"
class AlertNotFoundError(AlertError, NotFoundError):
code = "alert_not_found"
def __init__(self, alert_id: str) -> None:
super().__init__(
f"Alert {alert_id} not found",
details={"alert_id": alert_id},
)
class AlertQuotaExceededError(AlertError, RateLimitError):
code = "alert_quota_exceeded"
def __init__(self, limit: int, used: int) -> None:
super().__init__(
f"Alert quota exceeded: {used}/{limit}",
details={"limit": limit, "used": used},
)
class PaymentError(AppError):
"""Base for x402 payment errors."""
code = "payment_error"
class PaymentRequiredError(PaymentError):
status_code = 402
code = "payment_required"
def __init__(self, tool: str, price_usd: float, chain: str = "solana") -> None:
super().__init__(
f"Payment required for {tool}: ${price_usd}",
details={"tool": tool, "price_usd": price_usd, "chain": chain},
)
class PaymentFailedError(PaymentError, UpstreamError):
code = "payment_failed"
def __init__(self, tx_hash: str | None, reason: str) -> None:
super().__init__(
f"Payment failed: {reason}",
details={"tx_hash": tx_hash, "reason": reason},
)
class RAGError(AppError):
"""Base for RAG errors."""
code = "rag_error"
class RAGSearchError(RAGError, UpstreamError):
code = "rag_search_failed"
def __init__(self, query: str, reason: str) -> None:
super().__init__(
f"RAG search failed: {reason}",
details={"query": query[:100], "reason": reason},
)
# ── Helpers ──────────────────────────────────────────────────────────
def domain_error_response(error: AppError) -> dict[str, Any]:
"""Convert an AppError to the standard error envelope."""
return {
"code": error.code,
"message": error.message,
"details": error.details,
"status": error.status_code,
}
# ── FastAPI exception handlers ──────────────────────────────────────
def register_error_handlers(app: Any, debug: bool = False) -> None:
"""Register exception handlers on the FastAPI app.
Handlers:
- AppError subclasses → domain_error_response, status from class
- StarletteHTTPException → standard {error, code, request_id} envelope
- ValueError → 400 validation error
- Exception → 500 with optional traceback (dev only)
"""
import traceback
import uuid
from fastapi import Request
from fastapi.responses import JSONResponse
from starlette.exceptions import HTTPException as StarletteHTTPException
@app.exception_handler(AppError)
async def app_error_handler(request: Request, exc: AppError):
request_id = getattr(request.state, "request_id", str(uuid.uuid4()))
body = domain_error_response(exc)
body["request_id"] = request_id
return JSONResponse(status_code=exc.status_code, content=body)
@app.exception_handler(StarletteHTTPException)
async def http_exception_handler(request: Request, exc: StarletteHTTPException):
request_id = getattr(request.state, "request_id", str(uuid.uuid4()))
return JSONResponse(
status_code=exc.status_code,
content={
"code": exc.status_code,
"message": str(exc.detail),
"details": {},
"request_id": request_id,
},
)
@app.exception_handler(ValueError)
async def value_error_handler(request: Request, exc: ValueError):
request_id = getattr(request.state, "request_id", str(uuid.uuid4()))
return JSONResponse(
status_code=400,
content={
"code": "validation_error",
"message": str(exc),
"details": {},
"request_id": request_id,
},
)
@app.exception_handler(Exception)
async def unhandled_exception_handler(request: Request, exc: Exception):
request_id = getattr(request.state, "request_id", str(uuid.uuid4()))
body = {
"code": "internal_error",
"message": "Internal server error" if not debug else str(exc),
"details": {},
"request_id": request_id,
}
if debug:
body["details"]["traceback"] = traceback.format_exc().split("\n")
return JSONResponse(status_code=500, content=body)

169
backend/app/core/health.py
View file

@ -1,169 +0,0 @@
"""Health check hierarchy: /live, /ready, /health.
Three tiers, kubernetes-grade:
/live — process up. Always 200 if the Python process is alive.
Use this for liveness probes (restart-on-fail).
/ready — critical deps reachable: redis, databus, vector store.
200 if all healthy, 503 if any critical dep down.
Use this for readiness probes (route traffic only when ready).
/health — deep per-domain checks. Each domain registers a health_check()
function. The endpoint runs them all and returns per-domain
status. 200 if all healthy, 503 if any critical domain down.
Use this for monitoring dashboards and incident response.
Design: each domain provides a health_check() function via
register_health_check(). The health module doesn't import the domains
directly — they register themselves. This keeps core/ free of domain
coupling.
"""
from __future__ import annotations
import asyncio
import time
from dataclasses import dataclass, field
from typing import Awaitable, Callable, Optional
from app.core.logging import get_logger
log = get_logger(__name__)
HealthCheck = Callable[[], Awaitable["DomainHealth"]]
@dataclass
class DomainHealth:
"""Result of a single domain's health check."""
name: str
healthy: bool
latency_ms: int = 0
details: dict = field(default_factory=dict)
error: Optional[str] = None
def to_dict(self) -> dict:
return {
"name": self.name,
"healthy": self.healthy,
"latency_ms": self.latency_ms,
"details": self.details,
"error": self.error,
}
# Global registry of domain health checks
_HEALTH_CHECKS: dict[str, HealthCheck] = {}
def register_health_check(name: str, check: HealthCheck) -> None:
"""Register a domain health check. Idempotent (overwrites)."""
_HEALTH_CHECKS[name] = check
def unregister_health_check(name: str) -> None:
"""Remove a registered health check (mostly for tests)."""
_HEALTH_CHECKS.pop(name, None)
def list_registered_checks() -> list[str]:
"""Names of all registered domain health checks."""
return sorted(_HEALTH_CHECKS.keys())
async def liveness() -> dict:
"""Liveness probe — process is alive. Always 200 unless the process is dead."""
return {"status": "alive"}
async def readiness() -> tuple[bool, dict]:
"""Readiness probe — critical deps reachable. Returns (healthy, details)."""
details: dict = {}
healthy = True
# Redis check
try:
from app.core.redis import get_redis
r = get_redis()
if r.ping():
details["redis"] = {"healthy": True}
else:
details["redis"] = {"healthy": False, "error": "ping returned falsy"}
healthy = False
except Exception as e:
details["redis"] = {"healthy": False, "error": str(e)}
healthy = False
# Databus check (best-effort, only if client available)
try:
from app.databus.client import get_databus_client
client = get_databus_client()
if hasattr(client, "ping"):
await asyncio.wait_for(client.ping(), timeout=2.0)
details["databus"] = {"healthy": True}
else:
details["databus"] = {"healthy": True, "note": "no ping() method — assumed up"}
except Exception as e:
details["databus"] = {"healthy": False, "error": str(e)}
# Databus is best-effort — don't fail readiness just because it's slow
details["databus"]["best_effort"] = True
return healthy, details
async def deep_health() -> tuple[bool, dict]:
"""Deep health — runs every registered domain health check in parallel."""
if not _HEALTH_CHECKS:
return True, {"domains": {}, "registered": 0}
start = time.monotonic()
tasks = []
for name, check in _HEALTH_CHECKS.items():
tasks.append((name, _safe_check(name, check)))
results = await asyncio.gather(*(t[1] for t in tasks), return_exceptions=True)
domains: dict[str, dict] = {}
healthy = True
for (name, _), result in zip(tasks, results):
if isinstance(result, Exception):
domains[name] = DomainHealth(
name=name, healthy=False, error=str(result),
).to_dict()
healthy = False
elif isinstance(result, DomainHealth):
domains[name] = result.to_dict()
if not result.healthy:
healthy = False
else:
domains[name] = {"name": name, "healthy": False, "error": f"unexpected: {result!r}"}
healthy = False
total_ms = int((time.monotonic() - start) * 1000)
return healthy, {
"domains": domains,
"registered": len(_HEALTH_CHECKS),
"total_latency_ms": total_ms,
}
async def _safe_check(name: str, check: HealthCheck) -> DomainHealth:
"""Run a single check with a 5s timeout. Returns DomainHealth, never raises."""
start = time.monotonic()
try:
result = await asyncio.wait_for(check(), timeout=5.0)
if isinstance(result, DomainHealth):
return result
# Some checks return a bool — wrap it
return DomainHealth(
name=name,
healthy=bool(result),
latency_ms=int((time.monotonic() - start) * 1000),
)
except Exception as e:
return DomainHealth(
name=name,
healthy=False,
latency_ms=int((time.monotonic() - start) * 1000),
error=str(e),
)

48
backend/app/core/health_route.py
View file

@ -1,48 +0,0 @@
"""Health route — exposes the health check hierarchy at HTTP.
This is the 2026 framework push #6: kubernetes-grade liveness, readiness,
and deep health, with per-domain registration.
Mounted by main.py after the legacy /live, /ready, /health routes.
The legacy ones still serve during strangelfig.
"""
from __future__ import annotations
from fastapi import APIRouter
from fastapi.responses import JSONResponse
from app.core import health as health_mod
router = APIRouter(tags=["health"])
@router.get("/live")
async def live() -> dict:
"""Liveness probe. Process up. Always 200 unless the process is dead."""
return await health_mod.liveness()
@router.get("/ready")
async def ready() -> JSONResponse:
"""Readiness probe. 200 if critical deps reachable, 503 otherwise."""
healthy, details = await health_mod.readiness()
status_code = 200 if healthy else 503
return JSONResponse(
status_code=status_code,
content={"status": "ready" if healthy else "degraded", "dependencies": details},
)
@router.get("/health")
async def health() -> JSONResponse:
"""Deep health — runs every registered domain health check.
Each domain registers its own check via core.health.register_health_check().
Returns 200 if all domains healthy, 503 if any critical domain down.
"""
healthy, details = await health_mod.deep_health()
status_code = 200 if healthy else 503
return JSONResponse(
status_code=status_code,
content={"status": "healthy" if healthy else "degraded", **details},
)

27
backend/app/core/http.py
View file

@ -1,27 +0,0 @@
"""
Shared HTTP client with connection pooling.
Use this instead of creating ad-hoc httpx clients.
Usage:
from app.core.http import http_client
resp = await http_client.get("https://api.example.com/data")
"""
from __future__ import annotations
import httpx
# Single pooled client — reuse across all requests
http_client = httpx.AsyncClient(
limits=httpx.Limits(
max_connections=100,
max_keepalive_connections=20,
),
timeout=httpx.Timeout(30.0),
follow_redirects=True,
)
async def close_http_client() -> None:
"""Call on app shutdown."""
await http_client.aclose()

105
backend/app/core/lifespan.py
View file

@ -1,105 +0,0 @@
"""RMI Backend - Application lifespan (startup/shutdown events)."""
from contextlib import asynccontextmanager
import asyncio
import os
import httpx
from fastapi import FastAPI
from app.core.logging import get_logger
logger = get_logger(__name__)
# Background task imports (lazy, at startup time)
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Application lifespan: startup checks, background tasks, graceful shutdown."""
vault_pw = os.getenv("WALLET_VAULT_PASSWORD", "").strip()
if not vault_pw:
raise RuntimeError(
"CRITICAL: WALLET_VAULT_PASSWORD environment variable is missing or empty. "
"The backend will not start without it to prevent silent wallet key loss."
)
app.state.http_client = httpx.AsyncClient(
limits=httpx.Limits(max_connections=100, max_keepalive_connections=20), timeout=10.0
)
await _verify_indexes(app)
await _start_background_tasks(app)
yield # App runs here
await app.state.http_client.aclose()
logger.info("shutdown_complete")
async def _start_background_tasks(app: FastAPI) -> None:
"""Start all background monitoring / cleanup tasks."""
tasks = [
("facilitator_health", "app.routers.facilitator_health", "health_check_loop", "60s"),
("status_page", "app.routers.status_page", "status_monitor_loop", "30s"),
("webhook_dispatcher", "app.routers.webhook_dispatcher", "webhook_dispatcher_loop", "5s"),
("x402_trial_cleanup", "app.routers.x402_enforcement", "trial_cleanup_loop", "hourly"),
("auto_sweep", "app.wallet_manager_v2", "auto_sweep_loop", ""),
]
for name, module_path, func_name, interval in tasks:
try:
mod = __import__(module_path, fromlist=[func_name])
fn = getattr(mod, func_name)
if name == "auto_sweep":
asyncio.create_task(fn())
else:
asyncio.create_task(fn())
logger.info("background_task_started", task=name, interval=interval)
except Exception as e:
logger.warning("background_task_failed", task=name, error=str(e))
# Cache warmer (passes app instance)
try:
from app.databus.core import cache_warm_loop
asyncio.create_task(cache_warm_loop(app))
logger.info("background_task_started", task="databus_cache_warmer", interval="")
except Exception as e:
logger.warning("background_task_failed", task="databus_cache_warmer", error=str(e))
async def _verify_indexes(app: FastAPI) -> None:
"""Verify and create database indexes on startup."""
try:
supabase_url = os.getenv("SUPABASE_URL", "")
supabase_key = os.getenv("SUPABASE_SERVICE_KEY", "") or os.getenv("SUPABASE_KEY", "")
if not supabase_url or not supabase_key:
return
indexes = [
"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_scan_results_created_at ON scan_results(created_at DESC);",
"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_scan_results_token_address ON scan_results(token_address);",
"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_whale_alerts_created_at ON whale_alerts(created_at DESC);",
"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_security_alerts_created_at ON security_alerts(created_at DESC);",
"CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_x402_payments_tx_hash ON x402_payments(tx_hash);",
]
for sql in indexes:
try:
res = await app.state.http_client.post(
f"{supabase_url}/rest/v1/rpc/exec_sql",
json={"query": sql},
headers={
"apikey": supabase_key,
"Authorization": f"Bearer {supabase_key}",
"Content-Type": "application/json",
},
)
if res.status_code in [200, 204]:
logger.info("index_verified", table=sql.split("ON ")[1].split("(")[0].strip())
else:
logger.warning("index_skipped", status=res.status_code, sql=sql[:50])
except Exception as e:
logger.warning("index_verify_failed", error=str(e))
except Exception as e:
logger.warning("index_verification_skipped", error=str(e))

49
backend/app/core/llm_cache.py
View file

@ -1,49 +0,0 @@
"""Semantic LLM Cache for DataBus — caches identical + similar prompts. Redis-backed."""
import hashlib
import json
import os
REDIS_URL = os.getenv("REDIS_CACHE_URL", "redis://localhost:6379/1")
CACHE_TTL = int(os.getenv("LLM_CACHE_TTL", "3600"))
def _cache_key(prompt: str, model: str) -> str:
return f"llm_cache:{hashlib.sha256(f'{model}:{prompt}'.encode()).hexdigest()[:16]}"
def get_cached(prompt: str, model: str) -> dict | None:
"""Check if prompt+model result is cached."""
import redis
try:
r = redis.from_url(REDIS_URL, decode_responses=True)
data = r.get(_cache_key(prompt, model))
if data:
return json.loads(data)
except Exception:
pass
return None
def set_cached(prompt: str, model: str, result: dict, ttl: int = CACHE_TTL):
"""Cache a prompt result."""
import redis
try:
r = redis.from_url(REDIS_URL, decode_responses=True)
r.setex(_cache_key(prompt, model), ttl, json.dumps(result))
except Exception:
pass
def get_cache_stats() -> dict:
"""Get cache statistics."""
import redis
try:
r = redis.from_url(REDIS_URL, decode_responses=True)
keys = r.keys("llm_cache:*")
return {"cached_prompts": len(keys)}
except Exception:
return {"cached_prompts": 0, "error": "redis unavailable"}

40
backend/app/core/logging.py
View file

@ -1,40 +0,0 @@
"""RMI Backend - Structured logging with structlog + request ID tracking."""
import logging
import sys
import structlog
def setup_logging(level: str = "INFO") -> None:
"""Configure structlog with JSON output for production, console for dev."""
is_prod = level == "INFO" # JSON in prod, colored console in debug
structlog.configure(
processors=[
structlog.contextvars.merge_contextvars,
structlog.stdlib.filter_by_level,
structlog.stdlib.add_logger_name,
structlog.stdlib.add_log_level,
structlog.stdlib.PositionalArgumentsFormatter(),
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.StackInfoRenderer(),
structlog.processors.format_exc_info,
structlog.processors.UnicodeDecoder(),
structlog.dev.ConsoleRenderer() if not is_prod else structlog.processors.JSONRenderer(),
],
context_class=dict,
logger_factory=structlog.stdlib.LoggerFactory(),
wrapper_class=structlog.stdlib.BoundLogger,
cache_logger_on_first_use=True,
)
# Set root logger level
logging.basicConfig(format="%(message)s", stream=sys.stdout, level=getattr(logging, level))
logging.getLogger("uvicorn.access").setLevel(logging.WARNING)
logging.getLogger("httpx").setLevel(logging.WARNING)
def get_logger(name: str) -> structlog.stdlib.BoundLogger:
"""Get a structlog logger for a module."""
return structlog.get_logger(name)

178
backend/app/core/metrics.py
View file

@ -1,178 +0,0 @@
"""Prometheus /metrics endpoint (P0 #4 in v3 unfuck plan).
Auto-instruments every request via FastAPI middleware. Exposes:
- rmi_requests_total{method,path,status} — Counter
- rmi_request_duration_seconds{method,path} — Histogram
- rmi_active_requests — Gauge (in-flight)
- rmi_errors_total{type,route} — Counter for typed errors
Mount in main.py:
from app.core.metrics import router as metrics_router
_legacy_main.app.include_router(metrics_router)
from app.core.metrics import PrometheusMiddleware
_legacy_main.app.add_middleware(PrometheusMiddleware)
"""
from __future__ import annotations
import time
from typing import Awaitable, Callable
from fastapi import APIRouter, Request, Response
from prometheus_client import (
CONTENT_TYPE_LATEST,
CollectorRegistry,
Counter,
Gauge,
Histogram,
generate_latest,
)
from starlette.middleware.base import BaseHTTPMiddleware
# Use a dedicated registry so we don't conflict with the default global one.
REGISTRY = CollectorRegistry(auto_describe=True)
# ── Metrics ─────────────────────────────────────────────────────────────
REQUEST_COUNT = Counter(
"rmi_requests_total",
"Total HTTP requests handled by the backend.",
("method", "path", "status"),
registry=REGISTRY,
)
REQUEST_LATENCY = Histogram(
"rmi_request_duration_seconds",
"Request latency in seconds.",
("method", "path"),
buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0),
registry=REGISTRY,
)
ACTIVE_REQUESTS = Gauge(
"rmi_active_requests",
"Number of in-flight requests.",
registry=REGISTRY,
)
ERROR_COUNT = Counter(
"rmi_errors_total",
"Typed errors raised by the backend.",
("type", "route"),
registry=REGISTRY,
)
LLM_COST_USD = Counter(
"rmi_llm_cost_usd_total",
"Cumulative LLM cost in USD (from cost_tracking middleware).",
("tenant", "model"),
registry=REGISTRY,
)
# ── /metrics endpoint ───────────────────────────────────────────────────
router = APIRouter(tags=["metrics"])
@router.get("/metrics", include_in_schema=False)
async def metrics() -> Response:
"""Prometheus scrape endpoint."""
return Response(content=generate_latest(REGISTRY), media_type=CONTENT_TYPE_LATEST)
# ── Middleware ──────────────────────────────────────────────────────────
class PrometheusMiddleware(BaseHTTPMiddleware):
"""Records request count + latency for every request.
Path labels are normalized to the route template (e.g. /api/v2/wallet/{address})
rather than the actual path — this prevents cardinality explosion from
addresses / IDs being used as labels.
"""
async def dispatch(
self,
request: Request,
call_next: Callable[[Request], Awaitable[Response]],
) -> Response:
# Skip metrics endpoint itself to avoid recursive metrics.
if request.url.path == "/metrics":
return await call_next(request)
method = request.method
path_template = self._resolve_route_template(request)
ACTIVE_REQUESTS.inc()
start = time.monotonic()
try:
response = await call_next(request)
except Exception as exc: # noqa: BLE001 — typed errors raised below
elapsed = time.monotonic() - start
REQUEST_COUNT.labels(method, path_template, "500").inc()
REQUEST_LATENCY.labels(method, path_template).observe(elapsed)
ERROR_COUNT.labels(type=type(exc).__name__, route=path_template).inc()
ACTIVE_REQUESTS.dec()
raise
else:
elapsed = time.monotonic() - start
REQUEST_COUNT.labels(method, path_template, str(response.status_code)).inc()
REQUEST_LATENCY.labels(method, path_template).observe(elapsed)
return response
finally:
ACTIVE_REQUESTS.dec()
@staticmethod
def _resolve_route_template(request: Request) -> str:
"""Return the FastAPI route path template (e.g. /api/v2/wallet/{address})
instead of the literal URL — keeps label cardinality bounded.
"""
route = request.scope.get("route")
if route is not None and getattr(route, "path", None):
return str(route.path)
# Fallback: bucket the literal path to avoid unbounded cardinality.
return _bucket_path(request.url.path)
_BUCKET_PREFIXES = (
"/api/v1/scanner/",
"/api/v1/wallet/",
"/api/v1/token/",
"/api/v1/alerts/",
"/api/v1/rag/",
"/api/v1/x402/",
"/api/v2/scanner/",
"/api/v2/wallet/",
"/api/v2/token/",
"/api/v2/alerts/",
"/api/v2/rag/",
"/api/v2/x402/",
)
def _bucket_path(path: str) -> str:
"""Bucket unknown paths to avoid label cardinality explosion."""
for prefix in _BUCKET_PREFIXES:
if path.startswith(prefix):
return prefix + "{id}"
return path if len(path) <= 64 else "/{short_path}"
# ── v1 router compatibility shim ───────────────────────────────────────
def setup_metrics() -> None:
"""No-op setup for v1 router compatibility.
The v1 routers call this at import time to ensure metrics are wired.
The actual middleware is registered in main.py's lifespan via
PrometheusMiddleware. This function exists for import compatibility
only — it does nothing.
"""
return None
def record_llm_cost(tenant: str, model: str, cost_usd: float) -> None:
"""Increment the LLM cost counter for a tenant + model.
Called by the cost tracking middleware or by LLM wrappers.
"""
LLM_COST_USD.labels(tenant=tenant or "anonymous", model=model or "unknown").inc(
cost_usd
)

133
backend/app/core/middleware.py
View file

@ -1,133 +0,0 @@
"""RMI Backend — Core Middleware."""
import json
import os
import uuid
from fastapi import Request
from fastapi.responses import JSONResponse
# ═══════════════════════════════════════════════════════════════
# Rate-limit config
# ═══════════════════════════════════════════════════════════════
CACHEABLE_TOOLS = {"token_price", "token_metadata", "wallet_tokens", "entity_intel"}
# ═══════════════════════════════════════════════════════════════
# Payload size limit
# ═══════════════════════════════════════════════════════════════
MAX_PAYLOAD_SIZE = 1_048_576 # 1MB for standard JSON APIs
async def cache_middleware(request: Request, call_next):
"""Check Redis cache before executing tool. Store result after."""
path = request.url.path
if not path.startswith("/api/v1/x402-tools/"):
return await call_next(request)
if request.method != "POST":
return await call_next(request)
tool = path.rstrip("/").split("/")[-1]
if tool not in CACHEABLE_TOOLS:
return await call_next(request)
try:
body = await request.body()
params = json.loads(body) if body else {}
except Exception:
return await call_next(request)
try:
from app.routers.x402_advanced_tools import get_cached, set_cached
cached = get_cached(tool, params)
if cached:
return JSONResponse(content=cached, headers={"X-Cache": "HIT", "X-Cache-TTL": "60"})
except Exception:
pass
response = await call_next(request)
if response.status_code == 200:
try:
resp_body = b""
async for chunk in response.body_iterator:
resp_body += chunk
result = json.loads(resp_body)
set_cached(tool, params, result)
return JSONResponse(
content=result,
status_code=response.status_code,
headers={**dict(response.headers), "X-Cache": "MISS"},
)
except Exception:
pass
return response
async def emergency_lockdown_middleware(request: Request, call_next):
"""Check for emergency lockdown status. Block non-admin routes if active."""
if request.url.path in ["/health", "/api/v1/admin/emergency-lockdown", "/api/v1/admin/emergency-status"]:
return await call_next(request)
try:
import redis.asyncio as redis_lib
r = redis_lib.Redis(
host=os.getenv("REDIS_HOST", "localhost"),
port=int(os.getenv("REDIS_PORT", "6379")),
password=os.getenv("REDIS_PASSWORD", ""),
decode_responses=True,
)
is_locked = await r.exists("rmi:emergency_lockdown")
if is_locked:
auth_header = request.headers.get("Authorization", "")
session_token = request.headers.get("X-Admin-Session", "")
if not auth_header and not session_token:
return JSONResponse(
status_code=503,
content={"error": "Service Unavailable", "detail": "System is in emergency lockdown."},
)
except Exception:
pass
return await call_next(request)
async def hsts_middleware(request: Request, call_next):
"""Force HTTPS and prevent protocol downgrade attacks."""
response = await call_next(request)
response.headers["Strict-Transport-Security"] = "max-age=31536000; includeSubDomains; preload"
return response
async def request_id_middleware(request: Request, call_next):
"""Generate unique request ID for log correlation."""
request_id = request.headers.get("X-Request-ID") or str(uuid.uuid4())
response = await call_next(request)
response.headers["X-Request-ID"] = request_id
return response
async def payload_size_limit_middleware(request: Request, call_next):
"""Enforce 1MB payload limit on non-upload routes."""
content_length = request.headers.get("content-length")
if content_length and int(content_length) > MAX_PAYLOAD_SIZE:
if not request.url.path.startswith("/api/v1/admin/backend/upload/"):
return JSONResponse(
status_code=413,
content={"error": "Payload Too Large", "detail": "Maximum payload size is 1MB"},
)
return await call_next(request)
async def secure_cookie_middleware(request: Request, call_next):
"""Enforce secure cookie flags for admin sessions."""
response = await call_next(request)
if "set-cookie" in response.headers:
cookie_val = response.headers["set-cookie"]
if "HttpOnly" not in cookie_val:
cookie_val += "; HttpOnly"
if "Secure" not in cookie_val:
cookie_val += "; Secure"
if "SameSite=Strict" not in cookie_val and "SameSite=Lax" not in cookie_val:
cookie_val += "; SameSite=Strict"
response.headers["set-cookie"] = cookie_val
return response

106
backend/app/core/mistral_provider.py
View file

@ -1,106 +0,0 @@
"""Mistral AI provider for DataBus — Free tier: 1B tokens/month, 1 req/sec.
Credit-conserving: uses Small 4 for bulk, Medium 3.5 only when needed."""
import logging
import os
import httpx
logger = logging.getLogger(__name__)
MISTRAL_KEY = os.getenv("MISTRAL_API_KEY", "")
MISTRAL_BASE = "https://api.mistral.ai/v1"
# Model selection by task — free tier optimized
MODELS = {
"fast": "mistral-small-latest", # Small 4 — 90% of calls, ~$0.1/1M tokens
"smart": "mistral-medium-latest", # Medium 3.5 — complex analysis only
"embed": "mistral-embed", # Embeddings — state of art
"code": "mistral-small-latest", # Small 4 handles code well
"moderate": "mistral-moderation-latest", # Content moderation
}
# ⚠️ Deprecated — do NOT use
# mistral-small-2506 → deprecated, retiring July 2026
# mistral-medium-2508 → deprecated, retiring Aug 2026
async def mistral_chat(
prompt: str, model: str = None, system: str = None, max_tokens: int = 512, temperature: float = 0.7
) -> dict | None:
"""Chat completion via Mistral. Conserves free credits by using fast models."""
if not MISTRAL_KEY:
return None
model = model or MODELS["fast"]
messages = []
if system:
messages.append({"role": "system", "content": system})
messages.append({"role": "user", "content": prompt})
try:
async with httpx.AsyncClient(timeout=30) as c:
r = await c.post(
f"{MISTRAL_BASE}/chat/completions",
json={
"model": model,
"messages": messages,
"max_tokens": max_tokens,
"temperature": temperature,
},
headers={"Authorization": f"Bearer {MISTRAL_KEY}"},
)
if r.status_code == 200:
d = r.json()
return {
"response": d["choices"][0]["message"]["content"],
"model": model,
"tokens": d.get("usage", {}).get("total_tokens", 0),
"provider": "mistral",
}
elif r.status_code == 429:
logger.warning("Mistral rate limit hit — waiting...")
except Exception as e:
logger.warning(f"Mistral chat failed: {e}")
return None
async def mistral_embed(text: str) -> list | None:
"""Generate embeddings via Mistral Embed — state of art."""
if not MISTRAL_KEY:
return None
try:
async with httpx.AsyncClient(timeout=15) as c:
r = await c.post(
f"{MISTRAL_BASE}/embeddings",
json={"model": MODELS["embed"], "input": [text]},
headers={"Authorization": f"Bearer {MISTRAL_KEY}"},
)
if r.status_code == 200:
return r.json()["data"][0]["embedding"]
except Exception as e:
logger.warning(f"Mistral embed failed: {e}")
return None
async def mistral_moderate(text: str) -> dict | None:
"""Content moderation — jailbreak, toxicity, PII detection."""
if not MISTRAL_KEY:
return None
try:
async with httpx.AsyncClient(timeout=10) as c:
r = await c.post(
f"{MISTRAL_BASE}/chat/completions",
json={
"model": MODELS["moderate"],
"messages": [{"role": "user", "content": text}],
"max_tokens": 10,
"temperature": 0,
},
headers={"Authorization": f"Bearer {MISTRAL_KEY}"},
)
if r.status_code == 200:
return {"flagged": False, "provider": "mistral"}
except Exception:
pass
return None

128
backend/app/core/model_eval.py
View file

@ -1,128 +0,0 @@
#!/usr/bin/env python3
"""#8 — Model Evaluation Harness. Benchmarks models on Real-CATS scam data.
Runs lm-eval locally or via Ollama. Picks the best model per task."""
import asyncio
import json
import os
import time
from pathlib import Path
from typing import Any
import httpx
OLLAMA = os.getenv("OLLAMA_HOST", "http://localhost:11434")
REAL_CATS_PATH = Path(os.getenv("REAL_CATS_PATH", str(Path.home() / "rmi/backend/data/real_cats.json")))
# Test prompts for scam classification
BENCHMARK_TASKS = {
"scam_detection": {
"prompts": [
{
"input": "Token has mint authority enabled, liquidity is 0.5 SOL unlocked, deployer created 50 tokens before. Is this a scam?",
"expected": "yes",
},
{
"input": "Token has renounced mint, liquidity locked for 1 year, verified contract, audited by CertiK. Is this a scam?",
"expected": "no",
},
{
"input": "Token has honeypot detection enabled, 99% sell tax, unverified contract, anonymous team. Is this a scam?",
"expected": "yes",
},
{
"input": "Token listed on Binance, $50M market cap, 100K holders, 2 years old. Is this a scam?",
"expected": "no",
},
],
"metric": "accuracy",
},
}
async def evaluate_model(model: str, task_name: str) -> dict[str, Any]:
"""Evaluate a model on a benchmark task."""
task = BENCHMARK_TASKS.get(task_name)
if not task:
return {"error": f"Unknown task: {task_name}"}
correct = 0
total = 0
total_time = 0.0
results = []
async with httpx.AsyncClient(timeout=60) as c:
for item in task["prompts"]:
start = time.perf_counter()
try:
r = await c.post(
f"{OLLAMA}/api/generate",
json={
"model": model,
"prompt": f"Answer only YES or NO. {item['input']}",
"stream": False,
"options": {"num_predict": 5, "temperature": 0.1},
},
)
elapsed = time.perf_counter() - start
total_time += elapsed
response = r.json().get("response", "").strip().upper()
is_correct = item["expected"].upper() in response
if is_correct:
correct += 1
total += 1
results.append(
{
"input": item["input"][:80],
"expected": item["expected"],
"got": response[:20],
"correct": is_correct,
"time_ms": round(elapsed * 1000),
}
)
except Exception as e:
results.append({"input": item["input"][:80], "error": str(e)})
total += 1
accuracy = (correct / total * 100) if total > 0 else 0
return {
"model": model,
"task": task_name,
"accuracy": round(accuracy, 1),
"correct": correct,
"total": total,
"avg_time_ms": round((total_time / total) * 1000) if total > 0 else 0,
"results": results,
}
async def compare_models(models: list[str], task: str = "scam_detection"):
"""Compare multiple models on a benchmark task."""
scores = []
for model in models:
result = await evaluate_model(model, task)
scores.append(result)
scores.sort(key=lambda s: s["accuracy"], reverse=True)
return {
"task": task,
"models_compared": len(scores),
"leaderboard": [
{"model": s["model"], "accuracy": s["accuracy"], "avg_time_ms": s["avg_time_ms"]} for s in scores
],
"best_model": scores[0]["model"] if scores else None,
}
if __name__ == "__main__":
async def main():
print("Model Evaluation Harness")
print("=" * 40)
models = ["qwen2.5-coder:7b", "mistral:7b"]
results = await compare_models(models)
print(json.dumps(results["leaderboard"], indent=2))
print(f"\nBest model for scam detection: {results['best_model']}")
asyncio.run(main())

123
backend/app/core/model_router.py
View file

@ -1,123 +0,0 @@
"""Intelligent Model Router — auto-routes to best provider by task type, cost, latency.
Priority: real-time → Cerebras (9ms), cheap → Ollama ($0), complex → DeepSeek, bulk → Mistral."""
from dataclasses import dataclass
from enum import Enum
class TaskType(Enum):
FAST = "fast" # < 100ms needed — Cerebras, Groq
CHEAP = "cheap" # cost-sensitive — Ollama, Mistral free tier
COMPLEX = "complex" # reasoning needed — DeepSeek V4 Pro
BULK = "bulk" # high volume — Mistral Small 4
VISION = "vision" # image understanding — Gemini
EMBED = "embed" # embeddings — Mistral Embed
CODE = "code" # code generation — DeepSeek, qwen2.5-coder
ROUTING_TABLE = {
TaskType.FAST: [
("gpt-oss-120b", "cerebras", 0.009, 0.0), # 9ms, free
("llama-3.3-70b", "groq", 0.1, 0.0), # 100ms, free
],
TaskType.CHEAP: [
("qwen2.5-coder:7b", "ollama", 2.0, 0.0), # 2s, free
("mistral-small-latest", "mistral", 0.2, 0.0), # 200ms, free
],
TaskType.COMPLEX: [
("deepseek-v4-pro", "deepseek", 0.5, 0.55), # 500ms, $0.55/1M input
("mistral-medium-latest", "mistral", 0.3, 0.0), # 300ms, free
],
TaskType.BULK: [
("mistral-small-latest", "mistral", 0.2, 0.0), # 200ms, free, 2M TPM
("deepseek-v4-flash", "deepseek", 0.3, 0.14), # 300ms, cheap
],
TaskType.VISION: [
("gemini-2.5-flash", "gemini", 0.3, 0.0), # 300ms, free tier
],
TaskType.EMBED: [
("mistral-embed", "mistral", 0.1, 0.0), # 100ms, 20M TPM free
("bge-m3", "ollama", 2.0, 0.0), # 2s, local
],
TaskType.CODE: [
("deepseek-v4-flash", "deepseek", 0.3, 0.14), # 300ms, cheap
("qwen2.5-coder:7b", "ollama", 2.0, 0.0), # 2s, free
("mistral-small-latest", "mistral", 0.2, 0.0), # 200ms, free
],
}
@dataclass
class RoutingDecision:
model: str
provider: str
estimated_latency_ms: float
cost_per_1m_input: float
fallback_model: str | None = None
fallback_provider: str | None = None
def route_task(task_type: TaskType, prefer: str = "fast") -> RoutingDecision:
"""Route a task to the best model. Falls back to next on failure."""
candidates = ROUTING_TABLE.get(task_type, ROUTING_TABLE[TaskType.FAST])
if prefer == "cheap":
candidates = sorted(candidates, key=lambda c: c[3]) # sort by cost
elif prefer == "fast":
candidates = sorted(candidates, key=lambda c: c[2]) # sort by latency
primary = candidates[0]
fallback = candidates[1] if len(candidates) > 1 else None
return RoutingDecision(
model=primary[0],
provider=primary[1],
estimated_latency_ms=primary[2] * 1000,
cost_per_1m_input=primary[3],
fallback_model=fallback[0] if fallback else None,
fallback_provider=fallback[1] if fallback else None,
)
async def smart_route(prompt: str, task_type: str = "fast", prefer: str = "fast", **kwargs):
"""Auto-route a prompt to the best model. Returns response or falls back."""
decision = route_task(TaskType(task_type), prefer)
# Try primary
result = await _call_provider(decision.provider, decision.model, prompt, **kwargs)
if result:
return {**result, "routing": vars(decision), "fallback_used": False}
# Try fallback
if decision.fallback_model:
result = await _call_provider(decision.fallback_provider, decision.fallback_model, prompt, **kwargs)
if result:
return {**result, "routing": vars(decision), "fallback_used": True}
return {"error": "All providers failed", "routing": vars(decision)}
async def _call_provider(provider: str, model: str, prompt: str, **kwargs):
"""Call a specific provider. Returns dict or None."""
try:
if provider == "cerebras":
from app.core.cerebras_provider import cerebras_chat
return await cerebras_chat(prompt, **kwargs)
elif provider == "mistral":
from app.core.mistral_provider import mistral_chat
return await mistral_chat(prompt, model=model, **kwargs)
elif provider == "ollama":
import httpx
async with httpx.AsyncClient(timeout=60) as c:
r = await c.post(
"http://localhost:11434/api/generate", json={"model": model, "prompt": prompt, "stream": False}
)
if r.status_code == 200:
return {"response": r.json()["response"], "model": model, "provider": "ollama"}
# DeepSeek, Groq, Gemini handled via existing DataBus providers
except Exception:
pass
return None

151
backend/app/core/observability.py
View file

@ -1,151 +0,0 @@
"""T07 GlitchTip — Sentry SDK integration.
Per v4.0 §T07. Self-hosted Sentry-compatible error tracking at
glitchtip.rugmunch.io (Sentry SDK pointed at our own instance).
Key principle: NEVER leak secrets. The before_send hook strips
authorization headers, X-API-Key, passwords, tokens, etc.
Per v3 unfuck rule #7: SDK init must be at module level, not in lifespan.
But the SDK itself uses lazy init — setup_sentry() is called once at startup.
"""
from __future__ import annotations
import logging
import os
from typing import Any
log = logging.getLogger(__name__)
# Config — defaults to local GlitchTip; override via env
DEFAULT_DSN = "http://rmi-glitchtip-web:8000/1"
DEFAULT_ENV = "production"
DEFAULT_SAMPLE_RATE = 0.1 # 10% of transactions traced
# Sensitive field names that must never be sent to error tracking
SENSITIVE_KEYS = {
"authorization", "x-api-key", "x-payment", "x-agent-id",
"password", "secret", "token", "cookie", "set-cookie",
"api_key", "apikey", "access_token", "refresh_token",
"private_key", "credit_card", "ssn",
}
def _scrub_secrets(data: Any) -> Any:
"""Recursively replace sensitive values with '[REDACTED]'.
Walks dicts and lists, checking each key against SENSITIVE_KEYS.
"""
if isinstance(data, dict):
return {
k: "[REDACTED]" if k.lower() in SENSITIVE_KEYS else _scrub_secrets(v)
for k, v in data.items()
}
if isinstance(data, list):
return [_scrub_secrets(x) for x in data]
return data
def setup_sentry() -> bool:
"""Initialize the Sentry SDK pointed at our self-hosted GlitchTip.
Returns True if initialized, False if DSN not configured or
sentry_sdk is not installed (graceful — backend still works).
"""
dsn = os.getenv("GLITCHTIP_DSN") or os.getenv("SENTRY_DSN")
if not dsn:
log.info("sentry_disabled no_dsn")
return False
try:
import sentry_sdk
from sentry_sdk.integrations.fastapi import FastApiIntegration
from sentry_sdk.integrations.httpx import HttpxIntegration
from sentry_sdk.integrations.redis import RedisIntegration
except ImportError as exc:
log.info(f"sentry_sdk_not_installed err={exc}")
return False
env = os.getenv("ENVIRONMENT", DEFAULT_ENV)
traces_sample_rate = float(os.getenv("SENTRY_TRACES_SAMPLE_RATE", DEFAULT_SAMPLE_RATE))
sentry_sdk.init(
dsn=dsn,
environment=env,
traces_sample_rate=traces_sample_rate,
send_default_pii=False, # PII stays in our infra
integrations=[
FastApiIntegration(transaction_style="endpoint"),
HttpxIntegration(),
RedisIntegration(),
],
before_send=_before_send,
before_send_transaction=_before_send_transaction,
)
log.info(f"sentry_initialized dsn={dsn[:30]}... env={env} sample_rate={traces_sample_rate}")
return True
def _before_send(event: dict, hint: dict) -> dict | None:
"""Strip secrets before sending to GlitchTip.
Per v4.0 §T07: secrets scrubbed before send. This is a hard
requirement — never log full request bodies with credentials.
"""
try:
if "request" in event:
if "headers" in event["request"]:
event["request"]["headers"] = _scrub_secrets(event["request"]["headers"])
if "data" in event["request"]:
event["request"]["data"] = _scrub_secrets(event["request"]["data"])
if "extra" in event:
event["extra"] = _scrub_secrets(event["extra"])
if "user" in event:
# Strip email/IP — keep only id
event["user"] = {"id": event["user"].get("id", "anon")}
return event
except Exception as e:
log.warning(f"sentry_scrub_fail err={e}")
return event # Better to log it than to drop it
def _before_send_transaction(event: dict, hint: dict) -> dict | None:
"""Drop transactions that are health checks / metrics scrapes (high volume, low value)."""
url = event.get("transaction") or event.get("request", {}).get("url", "")
if any(s in url for s in ("/health", "/ready", "/live", "/metrics", "/openapi.json")):
return None
return event
def capture_exception(error: BaseException, **extra: Any) -> None:
"""Manually capture an exception. For use in catch blocks where
we want to log to Sentry but continue execution."""
try:
import sentry_sdk
with sentry_sdk.push_scope() as scope:
for k, v in extra.items():
scope.set_extra(k, v)
sentry_sdk.capture_exception(error)
except Exception as e:
log.warning(f"sentry_capture_fail err={e}")
def capture_message(message: str, level: str = "info", **extra: Any) -> None:
"""Manually capture a message. For non-exception events."""
try:
import sentry_sdk
with sentry_sdk.push_scope() as scope:
for k, v in extra.items():
scope.set_extra(k, v)
sentry_sdk.capture_message(message, level=level)
except Exception as e:
log.warning(f"sentry_message_fail err={e}")
def flush_sentry(timeout: float = 2.0) -> None:
"""Flush pending events. Call before shutdown."""
try:
import sentry_sdk
sentry_sdk.flush(timeout=timeout)
except Exception:
pass

Some files were not shown because too many files have changed in this diff Show more