Compare commits
79 commits
main
...
feat/scann
| Author | SHA1 | Date | |
|---|---|---|---|
| 80a37da423 | |||
| 5ee0cf948e | |||
| 21c5a564fa | |||
| 52a5b45e09 | |||
| b687fba48c | |||
| 8fe11cfb67 | |||
| 3bebd2b297 | |||
| 30a6508893 | |||
| c67f1d71ac | |||
| 133c8db6be | |||
| f2a50fdf71 | |||
| 69ca9758cd | |||
| 5a31946ce0 | |||
| 54cd1b0b3d | |||
| fd0c159037 | |||
| 50abecde08 | |||
| e3293b3313 | |||
| 88295801a9 | |||
| a80249c7bc | |||
| 089198f72b | |||
| fdac6768f8 | |||
| 7cc1129522 | |||
| da5dbe52c2 | |||
| ef9a444e80 | |||
| 9c0964807f | |||
| 0cbd59fe18 | |||
| df877e7347 | |||
| b96a20d18c | |||
| 6f1df381e5 | |||
| 48e8c597bf | |||
| b0b87c3998 | |||
| d4d9d2bc91 | |||
| 8f6a33d442 | |||
| 602dc7f5eb | |||
| 31409b6dc5 | |||
| bec3d4e6d2 | |||
| 462fbe3917 | |||
| 516776896f | |||
| 765424b7df | |||
| 6ef3b694ce | |||
| 05a81c0d4d | |||
| 8ee7cfb148 | |||
| 4712ec80ec | |||
| b454e9c3f7 | |||
| 0969aee84b | |||
| c700c85ac3 | |||
| e0a8ec8007 | |||
| 416499bc30 | |||
| 31a6383b5e | |||
| 7d3e8b7143 | |||
| 20f3c3d773 | |||
| 50f9735d0b | |||
| f01d7b6828 | |||
| 811465f7db | |||
| 897acd39b2 | |||
| 1da2a92f33 | |||
| e784deed78 | |||
| cfd75fd1a0 | |||
| 8491635bf2 | |||
| 27184c704d | |||
| 25e0891a0d | |||
| d2fe4263ef | |||
| d8098de3a4 | |||
| 37d3b5355e | |||
| 839819df17 | |||
| c1d1fc82ee | |||
| a637e83977 | |||
| bd412acb2b | |||
| 1b53332695 | |||
| 0a8c73d99b | |||
| 436bc37767 | |||
| 3571f29628 | |||
| 018edaded7 | |||
| f5e1e140dc | |||
| b31b564f36 | |||
| 3eb22495b1 | |||
| 59d6f2f0f6 | |||
| 7bd204f9f5 | |||
| 757eefd227 |
751 changed files with 34633 additions and 66472 deletions
55
.agentrules
Normal file
55
.agentrules
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
# .agentrules — RMI Backend AI Agent Constraints
|
||||
# Source of truth for AI agents operating on rmi-backend.
|
||||
# Extends standards/GUARDRAILS.md with project-specific rules.
|
||||
|
||||
## Project: RMI Backend (FastAPI + SQLAlchemy + Celery + Redis)
|
||||
## Repo: git.rugmunch.io/RugMunchMedia/rmi-backend
|
||||
## Language: Python 3.11+
|
||||
|
||||
## Critical Project Rules
|
||||
|
||||
1. **Databus providers are domain-driven.** Do NOT add code to `app/domains/databus/` without understanding the `DataProvider` ABC pattern. New providers implement `ThirdPartyProvider`, `SelfHostedNodeProvider`, or `PonderIndexerProvider`.
|
||||
|
||||
2. **Scanner pipeline is strict.** `/scanner/scan` uses shared utilities in `app/scanners/shared.py`. Do NOT duplicate `helius_rpc`, `dexscreener_fetch`, or `score_to_risk`.
|
||||
|
||||
3. **Telegram bot lives at `app/domains/telegram/rugmunchbot/`.** Legacy `app/telegram_bot/` is being phased out. All new commands go in the domain path.
|
||||
|
||||
4. **Async everywhere.** All FastAPI handlers and Celery tasks must be async. No `time.sleep()`, no `requests.get()`. Use `httpx.AsyncClient`, `asyncio.sleep`, `aioredis`.
|
||||
|
||||
5. **Rate limiting: 10 req/s per provider.** All third-party API calls must go through the rate limiter. No raw `httpx.get()` to external services.
|
||||
|
||||
6. **Type annotations required.** All public functions, methods, and class attributes must have type hints. `mypy --strict` is enforced in CI.
|
||||
|
||||
7. **No bare except.** Use `except SpecificError as e:` with structured logging. Ruff BLE rule enforces this.
|
||||
|
||||
8. **Secrets via gopass only.** Never hardcode RPC URLs, API keys, or tokens. Use `app.core.config` settings.
|
||||
|
||||
9. **Database migrations via Alembic.** Never modify the schema directly. Run `alembic revision --autogenerate -m "description"` and `alembic upgrade head`.
|
||||
|
||||
10. **Pre-commit runs on EVERY commit.** Ruff, mypy, gitleaks, and bandit. Run `make check` before pushing.
|
||||
|
||||
## Domain Boundaries
|
||||
|
||||
| Domain | Path | Purpose |
|
||||
|--------|------|---------|
|
||||
| Databus | `app/domains/databus/` | Third-party data providers, caching, news |
|
||||
| Scanners | `app/domains/scanners/` | Token risk scoring, address labeling |
|
||||
| Telegram | `app/domains/telegram/` | Bot commands, payments, moderation |
|
||||
| API | `app/api/v1/` | REST endpoints (public + admin) |
|
||||
| RAG | `app/rag/` | Vector search, embeddings |
|
||||
|
||||
## Test Requirements
|
||||
|
||||
- Coverage target: 60%+ (currently below, must improve)
|
||||
- Tests live in `tests/unit/` and `tests/integration/`
|
||||
- Use `pytest --strict-markers` — no fake test data
|
||||
- Run `make test` before committing
|
||||
|
||||
## Checklist Before Commit
|
||||
- [ ] `ruff check app/ tests/` passes
|
||||
- [ ] `mypy app/ --config-file mypy.ini` passes (gate modules: `app/domains/auth/`, `app/core/redis.py`)
|
||||
- [ ] `pytest tests/ -x -q -m "not integration"` passes
|
||||
- [ ] No secrets in diff (`gitleaks detect --verbose`)
|
||||
- [ ] Conventional commit: `feat(scope):`, `fix(scope):`, etc.
|
||||
- [ ] If adding a provider, it implements the correct ABC
|
||||
- [ ] If adding a route, it's in the correct API version path
|
||||
125
.github/scripts/ai_review.py
vendored
Normal file
125
.github/scripts/ai_review.py
vendored
Normal file
|
|
@ -0,0 +1,125 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Post an AI-generated review comment on a pull request from a diff file.
|
||||
|
||||
Environment variables:
|
||||
AI_REVIEW_API_KEY API key for the chat-completions endpoint.
|
||||
AI_REVIEW_MODEL Model string (default: openrouter/openai/gpt-4o-mini).
|
||||
AI_REVIEW_BASE_URL Base URL for the endpoint (default: https://openrouter.ai/api/v1).
|
||||
GITHUB_TOKEN GitHub token used to post the PR comment.
|
||||
PR_NUMBER Pull request number to comment on.
|
||||
|
||||
Usage:
|
||||
python3 .github/scripts/ai_review.py /path/to/pr.diff
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
DEFAULT_MODEL = "openrouter/openai/gpt-4o-mini"
|
||||
DEFAULT_BASE_URL = "https://openrouter.ai/api/v1"
|
||||
MAX_DIFF_CHARS = 30_000
|
||||
|
||||
SYSTEM_PROMPT = (
|
||||
"You are a senior code reviewer for a FastAPI crypto-intelligence backend. "
|
||||
"Review the provided diff for correctness, security, maintainability, and "
|
||||
"conformance to the project standards. Keep the response concise, actionable, "
|
||||
"and focused on the most important issues. If the diff is large, prioritize "
|
||||
"architecture and security concerns."
|
||||
)
|
||||
|
||||
|
||||
def _call_llm(diff_text: str) -> str:
|
||||
api_key = os.environ.get("AI_REVIEW_API_KEY")
|
||||
if not api_key:
|
||||
return "AI_REVIEW_API_KEY not set; skipping AI review."
|
||||
|
||||
base_url = os.environ.get("AI_REVIEW_BASE_URL", DEFAULT_BASE_URL).rstrip("/")
|
||||
model = os.environ.get("AI_REVIEW_MODEL", DEFAULT_MODEL)
|
||||
|
||||
payload = {
|
||||
"model": model,
|
||||
"messages": [
|
||||
{"role": "system", "content": SYSTEM_PROMPT},
|
||||
{"role": "user", "content": f"Pull request diff:\n\n```diff\n{diff_text}\n```"},
|
||||
],
|
||||
"temperature": 0.2,
|
||||
"max_tokens": 2_000,
|
||||
}
|
||||
|
||||
data = json.dumps(payload).encode("utf-8")
|
||||
headers = {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
try:
|
||||
with httpx.Client(timeout=120) as client:
|
||||
resp = client.post(f"{base_url}/chat/completions", content=data, headers=headers)
|
||||
result = resp.json()
|
||||
except httpx.HTTPError as exc:
|
||||
return f"AI review API request failed: {exc}"
|
||||
except Exception as exc:
|
||||
return f"AI review API request failed: {exc}"
|
||||
|
||||
try:
|
||||
return result["choices"][0]["message"]["content"]
|
||||
except (KeyError, IndexError, TypeError):
|
||||
return f"AI review returned unexpected response: {result}"
|
||||
|
||||
|
||||
def _post_comment(body: str) -> None:
|
||||
pr_number = os.environ.get("PR_NUMBER")
|
||||
token = os.environ.get("GITHUB_TOKEN")
|
||||
repo = os.environ.get("GITHUB_REPOSITORY")
|
||||
if not pr_number or not token or not repo:
|
||||
print("PR_NUMBER/GITHUB_TOKEN/GITHUB_REPOSITORY not set; printing review to stdout.")
|
||||
print(body)
|
||||
return
|
||||
|
||||
payload = json.dumps({"body": body}).encode("utf-8")
|
||||
headers = {
|
||||
"Authorization": f"Bearer {token}",
|
||||
"Accept": "application/vnd.github+json",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
try:
|
||||
with httpx.Client(timeout=60) as client:
|
||||
resp = client.post(
|
||||
f"https://api.github.com/repos/{repo}/issues/{pr_number}/comments",
|
||||
content=payload,
|
||||
headers=headers,
|
||||
)
|
||||
print(f"Posted AI review comment: {resp.status_code}")
|
||||
except httpx.HTTPError as exc:
|
||||
print(f"Failed to post comment: {exc}", file=sys.stderr)
|
||||
except Exception as exc:
|
||||
print(f"Failed to post comment: {exc}", file=sys.stderr)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
diff_path = Path(sys.argv[1]) if len(sys.argv) > 1 else None
|
||||
if diff_path:
|
||||
diff_text = diff_path.read_text(encoding="utf-8", errors="ignore")
|
||||
else:
|
||||
diff_text = sys.stdin.read()
|
||||
|
||||
if not diff_text.strip():
|
||||
print("No diff provided; skipping AI review.")
|
||||
return 0
|
||||
|
||||
if len(diff_text) > MAX_DIFF_CHARS:
|
||||
diff_text = diff_text[:MAX_DIFF_CHARS] + "\n\n... [diff truncated for token limits]"
|
||||
|
||||
review = _call_llm(diff_text)
|
||||
_post_comment(review)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
30
.github/workflows/ai-review.yml
vendored
Normal file
30
.github/workflows/ai-review.yml
vendored
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
name: AI PR Review
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize]
|
||||
branches: [main]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
ai-review:
|
||||
name: AI PR Review
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Get PR diff
|
||||
run: git diff origin/${{ github.base_ref }}...HEAD > /tmp/pr.diff
|
||||
|
||||
- name: Run AI review
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
AI_REVIEW_API_KEY: ${{ secrets.AI_REVIEW_API_KEY }}
|
||||
AI_REVIEW_MODEL: ${{ vars.AI_REVIEW_MODEL || 'openrouter/openai/gpt-4o-mini' }}
|
||||
PR_NUMBER: ${{ github.event.number }}
|
||||
run: python3 .github/scripts/ai_review.py /tmp/pr.diff
|
||||
44
.github/workflows/ci.yml
vendored
44
.github/workflows/ci.yml
vendored
|
|
@ -2,9 +2,9 @@ name: RMI CI
|
|||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
branches: [main, feat/scanners-p4-8-cont, feat/*]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
branches: [main, feat/scanners-p4-8-cont, feat/*]
|
||||
|
||||
# T14 fix (RMI v5 §G09): every PR must build clean + emit a real OpenAPI
|
||||
# schema. Catches factory regressions BEFORE merge so SDKs and MCP
|
||||
|
|
@ -12,8 +12,8 @@ on:
|
|||
#
|
||||
# Phase 1 of AUDIT-2026-Q3.md item P1.6:
|
||||
# - CI was 6/8 decorative (|| true / continue-on-error: true on every job)
|
||||
# - Now: 2 GATING jobs (build + test) must pass for merge
|
||||
# - 6 INFORMATIONAL jobs (lint-info, typecheck-info, etc.) report but never gate
|
||||
# - Now: 4 GATING jobs (build + test + typecheck-gate + lint) must pass for merge
|
||||
# - 6 INFORMATIONAL jobs (lint-info, typecheck-full-info, etc.) report but never gate
|
||||
# - Forgejo .forgejo/workflows/ci.yml remains the authoritative gate
|
||||
|
||||
jobs:
|
||||
|
|
@ -61,13 +61,26 @@ jobs:
|
|||
2>&1 | tail -50
|
||||
# NO || true — failure GATES merge
|
||||
|
||||
typecheck-gate:
|
||||
name: Typecheck scoped gate (gate)
|
||||
needs: build
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: astral-sh/setup-uv@v2
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.11"
|
||||
- name: Install project editable
|
||||
run: uv pip install --system -e ".[dev]"
|
||||
- name: Run scoped mypy gate
|
||||
run: make mypy-gate
|
||||
|
||||
# ────────────────────────── INFORMATIONAL JOBS (fire-and-forget) ──────────────────────────
|
||||
|
||||
lint-info:
|
||||
name: Lint (info only)
|
||||
if: always()
|
||||
lint:
|
||||
name: Lint (gate)
|
||||
runs-on: ubuntu-latest
|
||||
continue-on-error: true
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: astral-sh/setup-uv@v2
|
||||
|
|
@ -76,14 +89,13 @@ jobs:
|
|||
python-version: "3.11"
|
||||
- name: Install ruff
|
||||
run: uv pip install --system ruff
|
||||
- name: Run ruff lint (informational)
|
||||
run: ruff check . --statistics --output-format=concise 2>&1 | tail -30 || true
|
||||
- name: Run ruff lint (gate)
|
||||
run: ruff check .
|
||||
|
||||
typecheck-info:
|
||||
name: Typecheck (info only)
|
||||
typecheck-full-info:
|
||||
name: Typecheck full codebase (info only)
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
continue-on-error: true
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: astral-sh/setup-uv@v2
|
||||
|
|
@ -93,13 +105,12 @@ jobs:
|
|||
- name: Install mypy
|
||||
run: uv pip install --system mypy
|
||||
- name: Run mypy typecheck (informational)
|
||||
run: mypy app/ --config-file mypy.ini || true
|
||||
run: make typecheck || true
|
||||
|
||||
security-info:
|
||||
name: Security (info only)
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
continue-on-error: true
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: astral-sh/setup-uv@v2
|
||||
|
|
@ -119,7 +130,6 @@ jobs:
|
|||
name: OpenAPI (info only)
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
continue-on-error: true
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: astral-sh/setup-uv@v2
|
||||
|
|
@ -141,7 +151,6 @@ jobs:
|
|||
name: Qdrant cleanup (info only)
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
continue-on-error: true
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
|
|
@ -158,7 +167,6 @@ jobs:
|
|||
name: Heartbeat (info only)
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
continue-on-error: true
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Verify .gitmodules is consistent
|
||||
|
|
|
|||
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -89,3 +89,4 @@ data/papers/
|
|||
*.hdf5
|
||||
main.py.bak
|
||||
ruff.toml
|
||||
rmi-rag/
|
||||
|
|
|
|||
68
AGENTS.md
68
AGENTS.md
|
|
@ -13,12 +13,12 @@ See [CLEANUP.md](CLEANUP.md) for cleanup history.
|
|||
|
||||
# AGENTS.md — rmi-backend
|
||||
# Every AI agent working on backend code reads this first.
|
||||
# Last updated: 2026-06-29
|
||||
# Last updated: 2026-07-07
|
||||
|
||||
## What This Is
|
||||
|
||||
rmi-backend — FastAPI crypto intelligence backend. 757+ routes, 221 MCP tools,
|
||||
89 DataBus chains, 39.4M wallet labels, 21 SENTINEL scanner modules.
|
||||
rmi-backend — FastAPI crypto intelligence backend. 466+ API routes, 200+ MCP tools,
|
||||
89+ DataBus chains, 39.4M wallet labels (ClickHouse), 20+ SENTINEL scanner modules.
|
||||
|
||||
## Before You Start
|
||||
|
||||
|
|
@ -32,23 +32,38 @@ Read these files in order:
|
|||
|
||||
## The Server
|
||||
|
||||
ALL work happens on Talos (100.104.130.92).
|
||||
ALL source edits happen on **Talos** in the Forgejo clone:
|
||||
|
||||
```
|
||||
ssh netcup
|
||||
cd /root/backend
|
||||
cd /srv/work/repos/rmi-backend
|
||||
```
|
||||
|
||||
Deploy artifacts are built from this repo and run on Talos at `/root/backend/`
|
||||
(compose, data volumes, `.secrets/`). The running `rmi-backend` container is built
|
||||
from the Forgejo image — do not edit code in `/root/backend/`.
|
||||
|
||||
Quick health checks on Talos:
|
||||
|
||||
```
|
||||
docker logs rmi-backend --tail 100
|
||||
docker restart rmi-backend
|
||||
curl localhost:8000/health
|
||||
```
|
||||
|
||||
Code in /root/backend is Docker volume-mounted — changes go live on restart.
|
||||
|
||||
## Testing
|
||||
|
||||
Local (development):
|
||||
|
||||
```
|
||||
pytest tests/unit/ -q -m "not integration"
|
||||
pytest tests/unit/test_X.py -q
|
||||
```
|
||||
|
||||
In the deployed container (when you need full runtime deps):
|
||||
|
||||
```
|
||||
docker exec rmi-backend pytest -p no:pytest-brownie -v
|
||||
docker exec rmi-backend pytest tests/unit/test_X.py -p no:pytest-brownie -v
|
||||
```
|
||||
|
||||
The `-p no:pytest-brownie` is mandatory — eth-brownie tries to connect to ganache.
|
||||
|
|
@ -56,34 +71,41 @@ The `-p no:pytest-brownie` is mandatory — eth-brownie tries to connect to gana
|
|||
## Linting
|
||||
|
||||
```
|
||||
ruff check app/ --fix
|
||||
ruff format app/
|
||||
mypy app/ --strict
|
||||
ruff check app/ tests/
|
||||
ruff format app/ tests/
|
||||
make mypy-gate # scoped gate for clean modules
|
||||
make typecheck # full codebase, informational only
|
||||
```
|
||||
|
||||
## MCP Tools
|
||||
|
||||
mcp_discover, status_check, analytics_query, wallet_labels,
|
||||
eth_labels_tool, token_security, scam_intel, news_search,
|
||||
rag_query, report_generate, x402_marketplace (+ 210 more)
|
||||
rag_query, report_generate, x402_marketplace (+ 200 more)
|
||||
|
||||
List: GET /mcp/tools Call: POST /mcp/call/{tool_id} Info: GET /mcp/info
|
||||
|
||||
## Adding a New Scanner Module
|
||||
|
||||
1. Create app/scanners/my_detector.py (fetches from first-party APIs only)
|
||||
2. Add import to app/scanners/__init__.py
|
||||
3. Add runner function in app/scanners/sentinel_pipeline.py
|
||||
4. Add endpoint to app/routers/x402_tools.py
|
||||
5. Add pricing to x402_enforcement.py
|
||||
1. Create `app/domains/scanners/my_detector.py` (fetches from first-party APIs only)
|
||||
2. Wire it into `app/domains/scanners/sentinel_pipeline.py` or the relevant orchestrator
|
||||
3. Add endpoint/tool registration in `app/domains/x402/router.py` or `app/routers/`
|
||||
4. Add pricing to `app/domains/billing/x402/config.py` if it is a paid tool
|
||||
|
||||
CRITICAL: Never call our own x402 endpoints internally.
|
||||
|
||||
## Adding a New DataBus Chain
|
||||
|
||||
1. Create provider in app/databus/providers/
|
||||
2. Register in app/databus/chains.py
|
||||
3. Register provider class in app/databus/registry.py
|
||||
1. Create provider function in `app/domains/databus/providers/<category>.py`
|
||||
2. Register it in `app/domains/databus/providers/chains.py`
|
||||
3. Re-export from `app.domains.databus.providers` if it is public
|
||||
|
||||
## x402 Payment Facilitators
|
||||
|
||||
Facilitator selection is dynamic and driven by `app/domains/billing/x402/config.py` and the
|
||||
`app.domains.x402` router. Historic docstring claims about a fixed number of "active" or
|
||||
"offline" facilitators are not authoritative; the current supported networks and tokens are
|
||||
defined in code and deployment config.
|
||||
|
||||
## Rules
|
||||
|
||||
|
|
@ -100,7 +122,7 @@ CRITICAL: Never call our own x402 endpoints internally.
|
|||
|
||||
## Git
|
||||
|
||||
Internal primary: `root@talos:/srv/git/rmi-backend.git`
|
||||
Push `talos main` on every commit.
|
||||
Internal primary: `ssh://git@git.rugmunch.io:2222/RugMunchMedia/rmi-backend`
|
||||
Push `feat/scanners-p4-8-cont` (and other feature branches) on every commit.
|
||||
|
||||
External remotes: GitHub (LLC), GitLab, HuggingFace (see ~/AGENTS.md)
|
||||
External remotes: Codeberg, GitLab (GitHub is currently disabled — account flagged 2026-07-06).
|
||||
|
|
|
|||
|
|
@ -1,10 +1,11 @@
|
|||
# RMI Backend — Architecture
|
||||
|
||||
**Status:** canonical
|
||||
**Last updated:** 2026-07-03
|
||||
**Last updated:** 2026-07-07
|
||||
|
||||
## Overview
|
||||
FastAPI monolith serving crypto risk intelligence via HTTP, Telegram, and MCP.
|
||||
Domains are consolidated under `app/domains/` per AUDIT-2026-Q3 Phase 4.
|
||||
|
||||
## High-level flow
|
||||
```
|
||||
|
|
@ -12,9 +13,9 @@ Client (Web / Telegram / AI Agent)
|
|||
↓
|
||||
nginx → rmi-backend container (:8000)
|
||||
↓
|
||||
FastAPI routers
|
||||
FastAPI routers (mounted by app/mount.py)
|
||||
↓
|
||||
Services / scanners / DataBus providers
|
||||
app/domains/<domain>/service.py → app/infra/ + app/domains/<domain>/repository.py
|
||||
↓
|
||||
Postgres / Redis / Qdrant / Neo4j / ClickHouse
|
||||
```
|
||||
|
|
@ -22,35 +23,42 @@ Postgres / Redis / Qdrant / Neo4j / ClickHouse
|
|||
## Components
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `app/routers/` | HTTP route handlers (thin) |
|
||||
| `app/scanners/` | Token risk scanners |
|
||||
| `app/databus/` | 30+ third-party data providers |
|
||||
| `app/telegram_bot/` | Telegram bot command handlers |
|
||||
| `app/factory.py` | FastAPI app factory (lifespan, middleware, error handlers, routers) |
|
||||
| `app/mount.py` | Single source of truth for router mounting |
|
||||
| `app/core/` | Cross-cutting concerns (logging, errors, redis, http, auth, config) |
|
||||
| `app/api/v1/` | Thin HTTP transport layer (public, auth, admin, x402, mcp) |
|
||||
| `app/domains/` | Business logic domains: auth, billing, databus, news, reports, scanner, scanners, telegram, threat, token, tokens, wallet, x402 |
|
||||
| `app/domains/scanners/` | Token risk scanners (IP — will move to rmi-ip in Phase 6) |
|
||||
| `app/domains/databus/` | 30+ third-party data provider integrations |
|
||||
| `app/domains/telegram/rugmunchbot/` | Telegram bot command handlers |
|
||||
| `app/mcp/` | MCP manifest and tool manager |
|
||||
| `app/facilitators/` | x402 payment verification |
|
||||
| `app/wallet_memory/` | Wallet label + clustering subsystem |
|
||||
| `app/infra/` | External integrations (ollama, langfuse, vector stores, chain clients) |
|
||||
| `app/wallet_memory/` | Wallet label + clustering subsystem (IP — will move to rmi-ip in Phase 6) |
|
||||
| `app/routers/` | Legacy HTTP route handlers (being migrated to app/domains/ or app/api/v1/) |
|
||||
|
||||
## Storage
|
||||
| Store | Use | Container / Host |
|
||||
|---|---|---|
|
||||
| Postgres | Primary OLTP, Ponder indexer data | `rmi-postgres` |
|
||||
| Redis | Cache, Celery queue | `rmi-redis` |
|
||||
| Qdrant | Vector embeddings | `rmi-qdrant` (down) |
|
||||
| Qdrant | Vector embeddings | `rmi-qdrant` |
|
||||
| Neo4j | Wallet graph / labels | `rmi-neo4j` |
|
||||
| ClickHouse | Analytics, 39M address labels | `rmi-clickhouse` |
|
||||
|
||||
## Interfaces
|
||||
| Endpoint | Purpose |
|
||||
|---|---|
|
||||
| `GET /health` | Health check (currently unhealthy due to Redis) |
|
||||
| `GET /health` | Health check (degraded — ETH RPC endpoint missing) |
|
||||
| `GET /metrics` | Prometheus metrics |
|
||||
| `GET /docs` | Swagger/OpenAPI docs |
|
||||
| `POST /scanner/scan` | Live token scan |
|
||||
| `POST /mcp/...` | MCP tool endpoints |
|
||||
|
||||
## Current issues
|
||||
- `x402_tools.py` (212 KB) and `x402_enforcement.py` (108 KB) violate the 500-line limit and should become separate services or split by domain.
|
||||
- `app/routers/x402_tools.py` (212 KB) and `app/routers/x402_enforcement.py` (108 KB) violate the 500-line limit and should become separate services or split by domain.
|
||||
- `sqlite3` is used in 5 files for local state; should migrate to Postgres/aiosqlite.
|
||||
- F-string logs and `print()` statements prevent structured log aggregation.
|
||||
- `mypy` has a structural issue with `app/domains/auth/` that needs resolution.
|
||||
|
||||
## Deployment
|
||||
```bash
|
||||
|
|
|
|||
|
|
@ -24,6 +24,10 @@ RUN pip install --no-cache-dir -r requirements.txt && \
|
|||
pip install --no-cache-dir slither-analyzer && \
|
||||
rm -rf /root/.cache/pip
|
||||
|
||||
# Install rmi-rag from local source
|
||||
COPY rmi-rag/ /rmi-rag/
|
||||
RUN pip install --no-cache-dir -e /rmi-rag
|
||||
|
||||
# Copy app (HF models excluded via .dockerignore — they download at runtime)
|
||||
COPY . .
|
||||
|
||||
|
|
|
|||
14
Makefile
14
Makefile
|
|
@ -1,4 +1,4 @@
|
|||
.PHONY: help install dev build lint format check test typecheck security ci clean
|
||||
.PHONY: help install dev build lint format check test test-all security ci clean mypy-gate typecheck-gate
|
||||
|
||||
help: ## Show this help
|
||||
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf " \033[36m%-20s\033[0m %s\n", $$1, $$2}'
|
||||
|
|
@ -27,8 +27,14 @@ check: ## Full check: lint + format + typecheck + test
|
|||
mypy app/ --ignore-missing-imports
|
||||
pytest tests/ -x -q -m "not integration"
|
||||
|
||||
typecheck: ## MyPy type check
|
||||
mypy app/ --ignore-missing-imports
|
||||
typecheck: ## MyPy type check (full codebase, informational)
|
||||
mypy app/ --config-file mypy.ini
|
||||
|
||||
mypy-gate: ## MyPy gate for clean modules (authoritative)
|
||||
mypy app/ --config-file mypy-gate.ini
|
||||
|
||||
typecheck-gate: ## Alias for mypy-gate
|
||||
$(MAKE) mypy-gate
|
||||
|
||||
test: ## Run tests (non-integration)
|
||||
pytest tests/ -x -q -m "not integration"
|
||||
|
|
@ -43,7 +49,7 @@ security: ## Security audit
|
|||
ci: ## Full CI pipeline
|
||||
ruff check app/ tests/
|
||||
ruff format --check app/ tests/
|
||||
mypy app/ --ignore-missing-imports
|
||||
$(MAKE) mypy-gate
|
||||
pytest tests/ -x -q -m "not integration"
|
||||
|
||||
clean: ## Remove build artifacts
|
||||
|
|
|
|||
51
STATUS.md
51
STATUS.md
|
|
@ -1,7 +1,7 @@
|
|||
# STATUS -- rmi-backend
|
||||
|
||||
**Owner:** Rug Munch Media LLC Engineering
|
||||
**Last updated:** 2026-07-06 -- Phase 1 of [AUDIT-2026-Q3.md](AUDIT-2026-Q3.md) complete.
|
||||
**Last updated:** 2026-07-07 -- Phase 4 domain consolidation in progress; P4.1-P4.8 + P5.1 complete.
|
||||
|
||||
## What This Repo Is
|
||||
|
||||
|
|
@ -13,7 +13,7 @@ Stack: Python 3.12 / FastAPI / SQLAlchemy / async / Playwright / Pydantic v2 / R
|
|||
|
||||
## Current Status
|
||||
|
||||
**Phase 1 (development hygiene) complete.** Phase 2 (delete dead code) starts next per [AUDIT-2026-Q3.md](AUDIT-2026-Q3.md).
|
||||
**Phase 4 (domain consolidation) is active.** Foundation (Phase 1) is solid, dead-router wiring (Phase 2) is partial, god-file splitting (Phase 3) is deferred, and standardization (Phase 5) has begun with CI trustworthiness and mypy config cleanup.
|
||||
|
||||
### Phase 1 -- DONE
|
||||
|
||||
|
|
@ -30,16 +30,44 @@ Committed to `main`:
|
|||
- `92a01ff` + `5294983` -- `AppError` hierarchy in `app/core/errors.py`, wired through `error_handlers.py`
|
||||
- `cd02714` + `2fb571e` -- author amend (canonical identity: `cryptorugmunch <admin@rugmunch.io>`)
|
||||
|
||||
### What Phase 1 DID NOT do (deferred)
|
||||
### Phase 2 -- IN PROGRESS (partial)
|
||||
|
||||
Rather than deleting 148 routers, we validated and mounted the useful ones in `app/mount.py` Wave 1-3 (2026-07-07). Dead routers remain in tree and will be archived in a later pass.
|
||||
|
||||
### Phase 3 -- NOT STARTED
|
||||
|
||||
God-files (`x402_tools.py`, `x402_enforcement.py`, `auth.py`, `databus/providers.py`, `telegram_bot/bot.py`, `wallet_manager_v2.py`) still await splitting.
|
||||
|
||||
### Phase 4 -- IN PROGRESS
|
||||
|
||||
Domain consolidation (AUDIT-2026-Q3 P4.x) — committed to `main`:
|
||||
- `dca458e` -- P4.1 billing (`app/billing/` + `app/facilitators/` → `app/domains/billing/`)
|
||||
- `7109a16` -- P4.2 wallet (`app/wallet/` → `app/domains/wallet/`)
|
||||
- `948e41c` -- P4.3 auth (`app/auth/` → `app/domains/auth/`)
|
||||
- `cab9740` -- P4.4 tokens (`app/tokens/` → `app/domains/tokens/`)
|
||||
- `ed5b830` -- P4.5 databus (`providers/` + `_generated/` → `app/domains/databus/`)
|
||||
- `56075cf` -- P4.6 telegram (`rugmunchbot/` → `app/domains/telegram/rugmunchbot/`)
|
||||
- `3b7ef42` -- P4.7 rename `app/domain/` → `app/domains/` + consolidate
|
||||
- `7cced4e` + `4686cb3` -- P4.8 scanners (`app/scanners/` → `app/domains/scanners/`)
|
||||
|
||||
Remaining Phase 4:
|
||||
- Finish databus root migration (`app/databus/` engine → `app/domains/databus/`)
|
||||
- Finish telegram migration (`app/telegram_bot/` → `app/domains/telegram/`)
|
||||
- Clean up deprecated shim dirs (`app/auth/`, `app/wallet/`, `app/billing/`, `app/facilitators/`, `app/scanners/`, `app/domain/`)
|
||||
- Consolidate bulletin, intelligence, markets, admin, referral, MCP domains
|
||||
|
||||
### Phase 5 -- STARTED
|
||||
|
||||
- `d666ad2` -- P5.1 `HEALTH_CHECK_DURATION` + `test_factory_has_minimum_routes`
|
||||
- `4686cb3` -- mypy config fixes (`mypy.ini` regex, `disallow_any_explicit` typo, `app.domains.*` module pattern)
|
||||
- Still open: alembic, real coverage gate, structured logging, Sentry wiring, Prometheus middleware
|
||||
|
||||
### What Phase 1-4 DID NOT do (deferred)
|
||||
|
||||
- ~2K ruff warnings in legacy code (Phase 5)
|
||||
- Real coverage 8.17% -> 80% gate (Phase 5, slowest part)
|
||||
- `mypy.ini` line 8 parse error + `disallow_any_express_imports` typo (Phase 5)
|
||||
- Real coverage 8.17% -> 80% gate (Phase 5)
|
||||
- Alembic migrations (still MISSING; Phase 5)
|
||||
- 285 dead modules (Phase 2)
|
||||
- 148 unmounted routers (Phase 2)
|
||||
- God-files split (Phase 3)
|
||||
- Domain consolidation (Phase 4)
|
||||
|
||||
### Open Issues
|
||||
|
||||
|
|
@ -68,8 +96,11 @@ e404e90 feat(rmi-backend,audit): add app/main.py entrypoint delegating to factor
|
|||
- `gitleaks detect` reports 28 findings -- all **public ERC-20/Solana token contract addresses** (USDC, WETH, MATIC, BONK, DAI, plus cryptoscam-address dataset). NOT secrets. Suppressed via `.gitleaksignore` + `.gitleaks.toml` `[allowlist]`. No rotation needed.
|
||||
- All secrets live in gopass under `rmi/contacts/admin_recovery` etc.
|
||||
|
||||
## Next: Phase 2 (week 2)
|
||||
## Next: Phase 4 completion
|
||||
|
||||
Archive **148 unmounted routers** + **~140 dead `app/*.py` files** to `app/_archive/legacy_2026_07/`. Goal: ~30% LOC reduction without changing any current behavior.
|
||||
1. Finish `app/databus/` engine migration to `app/domains/databus/`.
|
||||
2. Finish `app/telegram_bot/` migration to `app/domains/telegram/rugmunchbot/`.
|
||||
3. Remove deprecated shim directories (`app/auth/`, `app/wallet/`, `app/billing/`, `app/facilitators/`, `app/scanners/`, `app/domain/`).
|
||||
4. Consolidate bulletin, intelligence, markets, admin, referral, MCP into `app/domains/`.
|
||||
|
||||
See [AUDIT-2026-Q3.md](AUDIT-2026-Q3.md) for the full 6-phase plan.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,9 @@
|
|||
[alembic]
|
||||
script_location = alembic
|
||||
sqlalchemy.url = postgresql://${SUPABASE_USER}:${SUPABASE_PASSWORD}@${SUPABASE_HOST}:5432/${SUPABASE_DB}
|
||||
# Use the canonical DATABASE_URL (Postgres) — was previously broken (Supabase env vars).
|
||||
# Override with: DATABASE_URL=... alembic upgrade head
|
||||
sqlalchemy.url = postgresql+psycopg2://rmi:rmi@localhost/rmi
|
||||
file_template = %%(year)d%%(month).2d%%(day).2d_%%(hour).2d%%(minute).2d_%%(slug)s
|
||||
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic
|
||||
|
|
|
|||
|
|
@ -1,5 +1,18 @@
|
|||
"""Alembic migration environment."""
|
||||
"""Alembic migration environment.
|
||||
|
||||
The codebase has no SQLAlchemy ORM models (uses raw SQL + Supabase),
|
||||
so ``target_metadata`` is None and ``alembic revision --autogenerate``
|
||||
cannot introspect. All migrations must be written by hand:
|
||||
|
||||
DATABASE_URL=postgresql://rmi:rmi@localhost/rmi alembic upgrade head
|
||||
alembic revision -m "add_my_field"
|
||||
# edit alembic/versions/00xx_add_my_field.py
|
||||
DATABASE_URL=... alembic upgrade head
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from logging.config import fileConfig
|
||||
|
||||
from sqlalchemy import engine_from_config, pool
|
||||
|
|
@ -10,19 +23,35 @@ config = context.config
|
|||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
|
||||
target_metadata = None # Set to your SQLAlchemy Base.metadata when models exist
|
||||
# Allow DATABASE_URL env var to override alembic.ini (sync driver required for Alembic).
|
||||
db_url = os.getenv("DATABASE_URL")
|
||||
if db_url:
|
||||
# Alembic uses sync engine; strip async driver if present.
|
||||
db_url = db_url.replace("postgresql+asyncpg://", "postgresql+psycopg2://")
|
||||
config.set_main_option("sqlalchemy.url", db_url)
|
||||
|
||||
target_metadata = None # No ORM models — hand-write every migration.
|
||||
|
||||
|
||||
def run_migrations_offline():
|
||||
def run_migrations_offline() -> None:
|
||||
"""Run migrations without a live DB connection."""
|
||||
url = config.get_main_option("sqlalchemy.url")
|
||||
context.configure(url=url, target_metadata=target_metadata, literal_binds=True)
|
||||
context.configure(
|
||||
url=url,
|
||||
target_metadata=target_metadata,
|
||||
literal_binds=True,
|
||||
dialect_opts={"paramstyle": "named"},
|
||||
)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
def run_migrations_online():
|
||||
def run_migrations_online() -> None:
|
||||
"""Run migrations against a live DB engine."""
|
||||
connectable = engine_from_config(
|
||||
config.get_section(config.config_ini_section), prefix="sqlalchemy.", poolclass=pool.NullPool
|
||||
config.get_section(config.config_ini_section, {}),
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
with connectable.connect() as connection:
|
||||
context.configure(connection=connection, target_metadata=target_metadata)
|
||||
|
|
|
|||
185
alembic/versions/0001_baseline.py
Normal file
185
alembic/versions/0001_baseline.py
Normal file
|
|
@ -0,0 +1,185 @@
|
|||
"""Baseline schema migration.
|
||||
|
||||
Captures the core tables that were created via scattered ``CREATE TABLE
|
||||
IF NOT EXISTS`` statements across the codebase before Alembic existed.
|
||||
This migration is idempotent (uses ``IF NOT EXISTS``) so re-running it
|
||||
on a database that already has these tables is a no-op.
|
||||
|
||||
After this baseline, all schema changes MUST go through Alembic:
|
||||
|
||||
alembic revision -m "add_my_field"
|
||||
# edit alembic/versions/00xx_add_my_field.py
|
||||
alembic upgrade head
|
||||
|
||||
The codebase has no SQLAlchemy ORM models (uses raw SQL + Supabase),
|
||||
so ``alembic revision --autogenerate`` cannot introspect. All
|
||||
migrations must be written by hand.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "0001_baseline"
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
"""Baseline: create the core tables that were previously scattered."""
|
||||
# catalog schema (app/catalog/init_schema.py)
|
||||
op.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS tokens (
|
||||
address TEXT PRIMARY KEY,
|
||||
chain TEXT NOT NULL,
|
||||
name TEXT,
|
||||
symbol TEXT,
|
||||
decimals INT,
|
||||
total_supply NUMERIC,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
)
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS alerts (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
user_id TEXT NOT NULL,
|
||||
alert_type TEXT NOT NULL,
|
||||
severity TEXT,
|
||||
payload JSONB,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
)
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS news_items (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
source TEXT NOT NULL,
|
||||
title TEXT,
|
||||
url TEXT,
|
||||
content TEXT,
|
||||
sentiment NUMERIC,
|
||||
published_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
)
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS scan_reports (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
user_id TEXT,
|
||||
target_address TEXT NOT NULL,
|
||||
chain TEXT,
|
||||
risk_score INT,
|
||||
payload JSONB,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
)
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS rag_findings (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
collection TEXT NOT NULL,
|
||||
content TEXT,
|
||||
metadata JSONB,
|
||||
score NUMERIC,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
)
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS x402_receipts (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
agent_id TEXT,
|
||||
tool TEXT NOT NULL,
|
||||
tx_hash TEXT,
|
||||
amount_usdc NUMERIC,
|
||||
paid_via_x402 TEXT,
|
||||
paid_at TIMESTAMPTZ DEFAULT NOW()
|
||||
)
|
||||
"""
|
||||
)
|
||||
op.execute("CREATE INDEX IF NOT EXISTS idx_x402_paid_at ON x402_receipts(paid_at DESC)")
|
||||
op.execute("CREATE INDEX IF NOT EXISTS idx_x402_tool ON x402_receipts(tool)")
|
||||
op.execute("CREATE INDEX IF NOT EXISTS idx_x402_agent ON x402_receipts(agent_id)")
|
||||
|
||||
# app/db_client.py schema
|
||||
op.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS profiles (
|
||||
id TEXT PRIMARY KEY,
|
||||
display_name TEXT,
|
||||
email TEXT,
|
||||
wallet_address TEXT,
|
||||
wallet_chain TEXT,
|
||||
tier TEXT DEFAULT 'FREE',
|
||||
role TEXT DEFAULT 'USER',
|
||||
xp INT DEFAULT 0,
|
||||
level INT DEFAULT 1,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
)
|
||||
"""
|
||||
)
|
||||
op.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS contract_audits (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
contract_address TEXT NOT NULL,
|
||||
chain TEXT,
|
||||
report JSONB,
|
||||
risk_score INT,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
)
|
||||
"""
|
||||
)
|
||||
|
||||
# app/community_badges.py schema
|
||||
op.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS badges (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT,
|
||||
description TEXT,
|
||||
emoji TEXT,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
)
|
||||
"""
|
||||
)
|
||||
|
||||
# app/routers/chat.py schema
|
||||
op.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS chat_messages (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
user_id TEXT NOT NULL,
|
||||
role TEXT,
|
||||
content TEXT,
|
||||
metadata JSONB,
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
)
|
||||
"""
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
"""Drop baseline tables (DESTRUCTIVE)."""
|
||||
for table in [
|
||||
"chat_messages",
|
||||
"badges",
|
||||
"contract_audits",
|
||||
"profiles",
|
||||
"x402_receipts",
|
||||
"rag_findings",
|
||||
"scan_reports",
|
||||
"news_items",
|
||||
"alerts",
|
||||
"tokens",
|
||||
]:
|
||||
op.execute(f"DROP TABLE IF EXISTS {table} CASCADE")
|
||||
|
|
@ -1,105 +0,0 @@
|
|||
# app/_archive/legacy_2026_07
|
||||
|
||||
**Archived 2026-07-06** as part of AUDIT-2026-Q3.md Phase 2.3 + P2.4.
|
||||
|
||||
This directory contains **136 `.py` files** moved out of `app/` and `app/routers/`
|
||||
because they had **zero reach** into the running application at the time
|
||||
of the audit. Reach was verified across 5 surfaces via a forward-import
|
||||
closure from `app/mount.py`, `app/main.py`, `app/factory.py`, and
|
||||
`app/core/lifespan.py`:
|
||||
|
||||
1. NOT in `app/mount.py` mount expressions
|
||||
2. NOT statically imported by any `.py` in `app/`, `tests/`, `scripts/`
|
||||
3. NOT dynamically loaded by `app/core/lifespan.py`
|
||||
4. NOT referenced as a string anywhere in `*.yaml`/`*.yml`/`*.toml`/`*.json`
|
||||
5. NOT in the lifecycle always-loaded set
|
||||
|
||||
## Composition (137 files)
|
||||
|
||||
| Bucket | Count | Notes |
|
||||
|--------|------:|-------|
|
||||
| `app/*.py` (flat) | 63 | Domain-specific modules never imported |
|
||||
| `app/routers/*.py` | 73 | Routers never mounted and not imported (1 restored, see below) |
|
||||
| **Total** | **137** | — |
|
||||
|
||||
Total LOC moved: **44,932 lines** (reduces active tree by ~25%).
|
||||
|
||||
## Reach-graph re-verification (over 4 surfaces)
|
||||
|
||||
```
|
||||
imports = 348 app.* modules statically imported
|
||||
ref_files = 51 files with string references
|
||||
closure = forward BFS from mount.py + main.py + factory.py + lifespan.py
|
||||
reached = 194 files reachable from roots
|
||||
archive set = audit_dead (186) - reached - forced_live (4 stale items) - test_live (1) = 136
|
||||
```
|
||||
|
||||
## Restored post-archive (test dependency surfaced by smoke test)
|
||||
|
||||
- `app/routers/x402_bridge_health.py` — restored 2026-07-07.
|
||||
The reach graph showed it as DEAD, but `tests/unit/test_bridge_health.py`
|
||||
has 7 direct imports of it. The audit's reach graph considered `tests/`
|
||||
only as a transitive reach, not as a source of LIVE. The smoke test
|
||||
caught the regression and we restored. Reach graph will be patched to
|
||||
include `tests/*.py` as roots in the next audit cycle.
|
||||
|
||||
## Forced-LIVE (NOT archived per user directive)
|
||||
|
||||
The reach graph would have archived these, but the user listed them as LIVE
|
||||
and asked us to DEFER with a note:
|
||||
|
||||
- `app/ai_pipeline_v3.py` — imported by `app/unified_wallet_scanner.py` and
|
||||
`app/news_intelligence.py`. The reach graph shows those importers are
|
||||
themselves DEAD, so `ai_pipeline_v3` is also technically DEAD. But kept
|
||||
per user instruction (3 importers in the audit reach-graph window).
|
||||
- `app/splade_bm25.py` — imported by `app/rag_service.py` (LIVE).
|
||||
- `app/wallet_manager_v2.py` — imported by `app/routers/wallet_manager_v2.py`,
|
||||
`app/routers/x402_enforcement.py`, `app/routers/x402_tools.py`,
|
||||
`app/sweep_all.py`, `app/sweep_now.py`, `scripts/setup_x402_rotation.py`.
|
||||
Has a top-level `NameError` (`Bip44Coins`) but the import works because
|
||||
the failing line is in a function body only executed at runtime.
|
||||
- `app/crypto_embeddings.py` — NOT in the audit ARCHIVE list at all (the
|
||||
audit had a stale dead-list entry that was already KEEP). Heavy import
|
||||
graph: `rag_service`, `bundle_cluster_rag`, `rag/embedder`, `supabase_vector`,
|
||||
`rugmaps_ai`, `scam_sources`, `intel_feed_pipeline`, `rag_agentic`,
|
||||
`ragas_eval`, `tests/manual/test_rag.py`, `scripts/ingest_sigmod_fast.py`.
|
||||
|
||||
## Wave 3 (newly mounted, NOT archived)
|
||||
|
||||
- `app/routers/unified_scanner_router.py` — mounted 2026-07-07 at
|
||||
`/api/v2/scanner/{token,wallet}/scan` (2 routes). Reach graph would have
|
||||
archived it; the new mount entry in `app/mount.py` makes it LIVE.
|
||||
- `app/routers/unified_wallet_scanner.py` — DEFERRED (no `router` APIRouter
|
||||
attribute; library consumed by `unified_scanner_router`).
|
||||
- `app/routers/admin_extensions.py` — DORMANT per audit (darkroom-specific,
|
||||
25 routes at `/api/v1/admin/*`); would shadow `/api/v1/admin/alerts_webhook`.
|
||||
|
||||
## How to restore
|
||||
|
||||
These files are kept in git history under the path `app/_archive/legacy_2026_07/`.
|
||||
To restore any of them:
|
||||
|
||||
```bash
|
||||
cd /srv/work/repos/rmi-backend
|
||||
git mv app/_archive/legacy_2026_07/<name>.py app/routers/ # or app/
|
||||
# Then add the import to app/mount.py ROUTER_MODULES
|
||||
git commit -m "restore <name>.py from legacy archive"
|
||||
git push origin main
|
||||
```
|
||||
|
||||
`git log --all --oneline -- app/_archive/` shows every move. We did NOT
|
||||
rewrite history (would break external mirrors at github.com, gitlab.com,
|
||||
huggingface.co).
|
||||
|
||||
## Why archive, not delete
|
||||
|
||||
Git retains the full history. `git log --all --oneline -- app/_archive/`
|
||||
shows every move. The archive path itself is excluded from:
|
||||
|
||||
- **pytest discovery** — `pyproject.toml tool.pytest.ini_options testpaths = ["tests"]` does not traverse `app/`.
|
||||
- **ruff** — `[tool.ruff] extend-exclude = [..., "app/_archive/"]` (added 2026-07-07).
|
||||
- **mypy** — `[tool.mypy] exclude = [..., "app/_archive/"]` (added 2026-07-07).
|
||||
- **setuptools** — `[tool.setuptools.packages.find] exclude = ["app._archive*"]` (added 2026-07-07).
|
||||
|
||||
The audit doc at `/home/dev/pry/rmi-final-deadcode-2026-07-06.md` lists
|
||||
every file with the original path and recommendation.
|
||||
|
|
@ -1,154 +0,0 @@
|
|||
"""A/B Testing Framework - split traffic, measure accuracy, auto-promote winners."""
|
||||
|
||||
import hashlib
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
router = APIRouter(prefix="/api/v1/ab-testing", tags=["ab-testing"])
|
||||
|
||||
_experiments: dict[str, dict] = {}
|
||||
_results: dict[str, list] = {}
|
||||
|
||||
|
||||
@router.post("/experiment")
|
||||
async def create_experiment(
|
||||
name: str,
|
||||
variants: str = Query(..., description="Comma-separated variant names, e.g. 'prompt_v1,prompt_v2'"),
|
||||
traffic_split: str = Query("50,50", description="Traffic split percentages"),
|
||||
metric: str = Query("accuracy", description="Metric to evaluate"),
|
||||
):
|
||||
"""Create a new A/B test experiment."""
|
||||
variant_list = [v.strip() for v in variants.split(",")]
|
||||
split_list = [int(s.strip()) for s in traffic_split.split(",")]
|
||||
|
||||
if len(variant_list) != len(split_list):
|
||||
return {"error": "Variants and splits must have same length"}
|
||||
if sum(split_list) != 100:
|
||||
return {"error": "Traffic splits must sum to 100"}
|
||||
|
||||
exp_id = hashlib.md5(name.encode()).hexdigest()[:8]
|
||||
_experiments[exp_id] = {
|
||||
"id": exp_id,
|
||||
"name": name,
|
||||
"variants": variant_list,
|
||||
"traffic_split": split_list,
|
||||
"metric": metric,
|
||||
"created_at": datetime.now(UTC).isoformat(),
|
||||
"status": "active",
|
||||
}
|
||||
_results[exp_id] = []
|
||||
return _experiments[exp_id]
|
||||
|
||||
|
||||
@router.get("/assign/{experiment_id}")
|
||||
async def assign_variant(experiment_id: str, user_id: str = "anonymous"):
|
||||
"""Assign a user to a variant deterministically (same user always gets same variant)."""
|
||||
if experiment_id not in _experiments:
|
||||
return {"variant": "control", "note": "Experiment not found"}
|
||||
|
||||
exp = _experiments[experiment_id]
|
||||
# Deterministic assignment based on user_id hash
|
||||
h = int(hashlib.md5(f"{experiment_id}:{user_id}".encode()).hexdigest(), 16)
|
||||
bucket = h % 100
|
||||
|
||||
cumulative = 0
|
||||
for variant, split in zip(exp["variants"], exp["traffic_split"], strict=False):
|
||||
cumulative += split
|
||||
if bucket < cumulative:
|
||||
return {"experiment": experiment_id, "variant": variant, "user": user_id}
|
||||
|
||||
return {"experiment": experiment_id, "variant": exp["variants"][-1], "user": user_id}
|
||||
|
||||
|
||||
@router.post("/record/{experiment_id}")
|
||||
async def record_result(experiment_id: str, variant: str, correct: bool = False):
|
||||
"""Record a result for an A/B test variant."""
|
||||
if experiment_id not in _experiments:
|
||||
return {"error": "Experiment not found"}
|
||||
|
||||
_results[experiment_id].append(
|
||||
{
|
||||
"variant": variant,
|
||||
"correct": correct,
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
|
||||
# Check if we have enough data to declare a winner (30+ samples per variant)
|
||||
results = _results[experiment_id]
|
||||
variant_counts = {}
|
||||
variant_correct = {}
|
||||
for r in results:
|
||||
v = r["variant"]
|
||||
variant_counts[v] = variant_counts.get(v, 0) + 1
|
||||
if r["correct"]:
|
||||
variant_correct[v] = variant_correct.get(v, 0) + 1
|
||||
|
||||
# Check if all variants have enough samples
|
||||
min_samples = 30
|
||||
all_ready = all(count >= min_samples for count in variant_counts.values())
|
||||
|
||||
if all_ready:
|
||||
scores = {v: variant_correct.get(v, 0) / variant_counts[v] for v in variant_counts}
|
||||
best = max(scores, key=scores.get)
|
||||
worst = min(scores, key=scores.get)
|
||||
improvement = scores[best] - scores[worst]
|
||||
|
||||
result = {
|
||||
"experiment": experiment_id,
|
||||
"status": "complete" if improvement > 0.05 else "inconclusive",
|
||||
"winner": best if improvement > 0.05 else None,
|
||||
"scores": {v: round(s, 3) for v, s in scores.items()},
|
||||
"samples": variant_counts,
|
||||
"confidence": round(improvement * 100, 1),
|
||||
}
|
||||
|
||||
if improvement > 0.05:
|
||||
_experiments[experiment_id]["status"] = "complete"
|
||||
_experiments[experiment_id]["winner"] = best
|
||||
|
||||
return result
|
||||
|
||||
return {
|
||||
"experiment": experiment_id,
|
||||
"status": "collecting",
|
||||
"samples": variant_counts,
|
||||
"min_needed": min_samples,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/experiments")
|
||||
async def list_experiments():
|
||||
return {
|
||||
"experiments": list(_experiments.values()),
|
||||
"active": sum(1 for e in _experiments.values() if e["status"] == "active"),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/experiment/{experiment_id}")
|
||||
async def get_experiment(experiment_id: str):
|
||||
if experiment_id not in _experiments:
|
||||
return {"error": "Experiment not found"}
|
||||
exp = _experiments[experiment_id]
|
||||
results = _results.get(experiment_id, [])
|
||||
return {
|
||||
**exp,
|
||||
"total_trials": len(results),
|
||||
"results": _summarize_results(results, exp["variants"]),
|
||||
}
|
||||
|
||||
|
||||
def _summarize_results(results: list, variants: list) -> dict:
|
||||
counts = dict.fromkeys(variants, 0)
|
||||
correct = dict.fromkeys(variants, 0)
|
||||
for r in results:
|
||||
v = r["variant"]
|
||||
if v in counts:
|
||||
counts[v] += 1
|
||||
if r["correct"]:
|
||||
correct[v] += 1
|
||||
return {
|
||||
v: {"trials": counts[v], "correct": correct[v], "accuracy": round(correct[v] / max(counts[v], 1), 3)}
|
||||
for v in variants
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,263 +0,0 @@
|
|||
"""
|
||||
RMI Backend Control Router
|
||||
==========================
|
||||
Admin endpoints for system monitoring, service control, logs,
|
||||
database management, bot configuration, and alerts.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
import psutil
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/v1/admin", tags=["admin-control"])
|
||||
|
||||
|
||||
# ── Auth helper ──
|
||||
async def _verify_admin(request: Request):
|
||||
"""Verify admin access with API key."""
|
||||
admin_key = os.getenv("ADMIN_API_KEY", "dev-key-change-me")
|
||||
key = request.headers.get("X-Admin-Key", "")
|
||||
if key != admin_key:
|
||||
raise HTTPException(status_code=401, detail="Invalid admin key")
|
||||
return True
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# SYSTEM MONITORING
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
@router.get("/system")
|
||||
async def admin_system(request: Request, _=Depends(_verify_admin)):
|
||||
"""Get real-time system resource usage."""
|
||||
try:
|
||||
cpu_percent = psutil.cpu_percent(interval=0.5)
|
||||
cpu_count = psutil.cpu_count()
|
||||
cpu_freq = psutil.cpu_freq()
|
||||
memory = psutil.virtual_memory()
|
||||
disk = psutil.disk_usage("/")
|
||||
net_io = psutil.net_io_counters()
|
||||
boot_time = psutil.boot_time()
|
||||
uptime_seconds = int(datetime.now().timestamp() - boot_time)
|
||||
|
||||
# Process info for the backend itself
|
||||
backend_pid = os.getpid()
|
||||
backend_proc = psutil.Process(backend_pid)
|
||||
backend_memory_mb = backend_proc.memory_info().rss / (1024 * 1024)
|
||||
backend_cpu = backend_proc.cpu_percent(interval=0.2)
|
||||
|
||||
return {
|
||||
"timestamp": datetime.utcnow().isoformat(),
|
||||
"cpu": {
|
||||
"percent": cpu_percent,
|
||||
"count": cpu_count,
|
||||
"frequency_mhz": cpu_freq.current if cpu_freq else None,
|
||||
},
|
||||
"memory": {
|
||||
"total_gb": round(memory.total / (1024**3), 2),
|
||||
"available_gb": round(memory.available / (1024**3), 2),
|
||||
"percent": memory.percent,
|
||||
"used_gb": round(memory.used / (1024**3), 2),
|
||||
},
|
||||
"disk": {
|
||||
"total_gb": round(disk.total / (1024**3), 2),
|
||||
"used_gb": round(disk.used / (1024**3), 2),
|
||||
"free_gb": round(disk.free / (1024**3), 2),
|
||||
"percent": round(disk.used / disk.total * 100, 1),
|
||||
},
|
||||
"network": {
|
||||
"bytes_sent_mb": round(net_io.bytes_sent / (1024**2), 2),
|
||||
"bytes_recv_mb": round(net_io.bytes_recv / (1024**2), 2),
|
||||
"packets_sent": net_io.packets_sent,
|
||||
"packets_recv": net_io.packets_recv,
|
||||
},
|
||||
"uptime": {
|
||||
"seconds": uptime_seconds,
|
||||
"formatted": str(timedelta(seconds=uptime_seconds)),
|
||||
},
|
||||
"backend_process": {
|
||||
"pid": backend_pid,
|
||||
"memory_mb": round(backend_memory_mb, 2),
|
||||
"cpu_percent": backend_cpu,
|
||||
"threads": backend_proc.num_threads(),
|
||||
},
|
||||
"services": {
|
||||
"redis": _check_redis(),
|
||||
"supabase": _check_supabase(),
|
||||
},
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"System check failed: {e}")
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
def _check_redis():
|
||||
"""Check Redis connection."""
|
||||
try:
|
||||
import redis.asyncio as redis
|
||||
|
||||
r = redis.Redis(
|
||||
host=os.getenv("REDIS_HOST", "localhost"),
|
||||
port=int(os.getenv("REDIS_PORT", "6379")),
|
||||
password=os.getenv("REDIS_PASSWORD", ""),
|
||||
decode_responses=True,
|
||||
)
|
||||
r.ping()
|
||||
return {"status": "connected"}
|
||||
except Exception as e:
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
|
||||
def _check_supabase():
|
||||
"""Check Supabase connection."""
|
||||
supabase_url = os.getenv("SUPABASE_URL")
|
||||
if not supabase_url:
|
||||
return {"status": "not_configured"}
|
||||
return {"status": "connected", "url": supabase_url}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# DATABASE MANAGEMENT (stubbed - uses Redis instead of Supabase)
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
@router.get("/db/tables")
|
||||
async def admin_db_tables(request: Request, _=Depends(_verify_admin)):
|
||||
"""List available tables (stubbed - uses Redis instead of Supabase)."""
|
||||
return {
|
||||
"tables": [
|
||||
{"name": "wallet_users", "type": "redis", "rows": "dynamic"},
|
||||
{"name": "agents", "type": "redis", "rows": "dynamic"},
|
||||
{"name": "alerts", "type": "redis", "rows": "dynamic"},
|
||||
{"name": "scans", "type": "redis", "rows": "dynamic"},
|
||||
{"name": "evidence", "type": "redis", "rows": "dynamic"},
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@router.post("/db/query")
|
||||
async def admin_db_query(request: Request, _=Depends(_verify_admin)):
|
||||
"""Execute SQL query (stubbed - redirect to Redis operations)."""
|
||||
return {
|
||||
"error": "Direct SQL queries not supported. Use Redis operations instead.",
|
||||
"hint": "For data operations, use Redis endpoints in /api/v1/agents",
|
||||
}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# SERVICE CONTROL
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
@router.post("/service/restart")
|
||||
async def admin_service_restart(request: Request, _=Depends(_verify_admin)):
|
||||
"""Restart the backend service."""
|
||||
return {
|
||||
"action": "restart",
|
||||
"status": "queued",
|
||||
"message": "Service restart queued - use /api/v1/admin/service/status for updates",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/service/status")
|
||||
async def admin_service_status(request: Request, _=Depends(_verify_admin)):
|
||||
"""Get service status."""
|
||||
return {
|
||||
"service": "rmi-backend",
|
||||
"status": "running",
|
||||
"pid": os.getpid(),
|
||||
"port": 8013,
|
||||
"version": "1.0.0",
|
||||
}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# LOGS
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
@router.get("/logs")
|
||||
async def admin_logs(request: Request, _=Depends(_verify_admin), limit: int = 100):
|
||||
"""Get recent logs."""
|
||||
log_file = "/tmp/backend.log"
|
||||
if not os.path.exists(log_file):
|
||||
return {"logs": [], "error": "Log file not found"}
|
||||
|
||||
try:
|
||||
with open(log_file) as f:
|
||||
lines = f.readlines()[-limit:]
|
||||
return {"logs": [line.rstrip() for line in lines]}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# BOT CONFIGURATION
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
@router.get("/bot/config")
|
||||
async def admin_bot_config(request: Request, _=Depends(_verify_admin)):
|
||||
"""Get bot configuration."""
|
||||
return {
|
||||
"telegram": {
|
||||
"enabled": bool(os.getenv("TELEGRAM_BOT_TOKEN")),
|
||||
"channel": os.getenv("TELEGRAM_ALERTS_CHANNEL", ""),
|
||||
"rate_limit": 5,
|
||||
"rate_limit_window": 60,
|
||||
},
|
||||
"webhook": {
|
||||
"urls": [os.getenv("ALERT_WEBHOOK_URL", ""), os.getenv("SLACK_ALERT_WEBHOOK", "")],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# AGENT MANAGEMENT
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
@router.get("/agents")
|
||||
async def admin_agents(request: Request, _=Depends(_verify_admin)):
|
||||
"""List available agents."""
|
||||
return {
|
||||
"agents": [
|
||||
{"id": "agent-0", "name": "solana-monitor", "status": "running"},
|
||||
{"id": "agent-1", "name": "ethereum-monitor", "status": "running"},
|
||||
{"id": "agent-2", "name": "arbitrum-monitor", "status": "running"},
|
||||
{"id": "agent-3", "name": "base-monitor", "status": "running"},
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@router.get("/agents/{agent_id}")
|
||||
async def admin_agent_detail(request: Request, agent_id: str, _=Depends(_verify_admin)):
|
||||
"""Get agent details."""
|
||||
return {
|
||||
"id": agent_id,
|
||||
"name": f"agent-{agent_id}",
|
||||
"status": "running",
|
||||
"stats": {"transactions_processed": 0, "alerts_triggered": 0},
|
||||
}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# HEALTH & READY
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def admin_health(request: Request):
|
||||
"""Check backend health."""
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@router.get("/ready")
|
||||
async def admin_ready(request: Request):
|
||||
"""Check backend readiness."""
|
||||
return {"status": "ready"}
|
||||
|
|
@ -1,612 +0,0 @@
|
|||
"""
|
||||
RMI Admin User Management API
|
||||
=============================
|
||||
Admin endpoints for managing users:
|
||||
- List all users with filters (tier, role, status, date range)
|
||||
- Get user details
|
||||
- Warn user (add warning note)
|
||||
- Flag user as suspicious
|
||||
- Ban / unban user
|
||||
- Delete user (soft delete)
|
||||
- Update user tier/role
|
||||
- Bulk actions
|
||||
|
||||
All endpoints require ADMIN or SUPERADMIN role.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import secrets
|
||||
from datetime import datetime, timedelta
|
||||
from enum import StrEnum
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.core.redis import get_redis
|
||||
|
||||
logger = logging.getLogger("rmi_admin_users")
|
||||
router = APIRouter(tags=["admin-users"])
|
||||
|
||||
|
||||
# ── Redis helper ──
|
||||
async def require_admin(request: Request, min_role: str = "ADMIN"):
|
||||
"""Require admin authentication."""
|
||||
from app.auth import get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
role = user.get("role", "USER")
|
||||
user.get("tier", "FREE")
|
||||
|
||||
# Role hierarchy
|
||||
role_levels = {"USER": 0, "MODERATOR": 1, "ADMIN": 2, "SUPERADMIN": 3}
|
||||
required_level = role_levels.get(min_role, 2)
|
||||
user_level = role_levels.get(role, 0)
|
||||
|
||||
if user_level < required_level:
|
||||
raise HTTPException(status_code=403, detail="Admin access required")
|
||||
|
||||
return user
|
||||
|
||||
|
||||
# ── Models ──
|
||||
class UserStatus(StrEnum):
|
||||
ACTIVE = "active"
|
||||
WARNED = "warned"
|
||||
SUSPECT = "suspect"
|
||||
BANNED = "banned"
|
||||
DELETED = "deleted"
|
||||
|
||||
|
||||
class UserListFilters(BaseModel):
|
||||
tier: str | None = None
|
||||
role: str | None = None
|
||||
status: UserStatus | None = None
|
||||
search: str | None = None
|
||||
date_from: str | None = None
|
||||
date_to: str | None = None
|
||||
limit: int = Field(50, ge=1, le=500)
|
||||
offset: int = Field(0, ge=0)
|
||||
|
||||
|
||||
class WarnUserRequest(BaseModel):
|
||||
reason: str = Field(..., min_length=1, max_length=500)
|
||||
severity: str = Field("medium", pattern="^(low|medium|high|critical)$")
|
||||
|
||||
|
||||
class UpdateUserRequest(BaseModel):
|
||||
tier: str | None = None
|
||||
role: str | None = None
|
||||
status: UserStatus | None = None
|
||||
display_name: str | None = None
|
||||
scans_remaining: int | None = None
|
||||
notes: str | None = None
|
||||
|
||||
|
||||
class BulkActionRequest(BaseModel):
|
||||
user_ids: list[str]
|
||||
action: str = Field(..., pattern="^(warn|ban|unban|flag_suspect|clear_suspect|delete|restore|update_tier)$")
|
||||
params: dict[str, Any] | None = None
|
||||
|
||||
|
||||
class UserAdminResponse(BaseModel):
|
||||
id: str
|
||||
email: str
|
||||
display_name: str
|
||||
wallet_address: str | None
|
||||
wallet_chain: str | None
|
||||
tier: str
|
||||
role: str
|
||||
status: str
|
||||
created_at: str
|
||||
last_login: str | None
|
||||
xp: int
|
||||
level: int
|
||||
scans_used: int
|
||||
scans_remaining: int
|
||||
warnings: list[dict]
|
||||
notes: str | None
|
||||
login_methods: list[str]
|
||||
|
||||
|
||||
# ── Helper: Get all users from Redis ──
|
||||
def _get_all_users() -> list[dict]:
|
||||
"""Fetch all users from Redis."""
|
||||
r = get_redis()
|
||||
users_raw = r.hgetall("rmi:users")
|
||||
users = []
|
||||
for user_id, data in users_raw.items():
|
||||
try:
|
||||
user = json.loads(data)
|
||||
user["id"] = user_id
|
||||
users.append(user)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
return users
|
||||
|
||||
|
||||
def _get_user_full(user_id: str) -> dict | None:
|
||||
"""Get user with enriched data."""
|
||||
from app.auth import _get_user
|
||||
|
||||
user = _get_user(user_id)
|
||||
if not user:
|
||||
return None
|
||||
|
||||
r = get_redis()
|
||||
|
||||
# Get warnings
|
||||
warnings_raw = r.hget("rmi:user_warnings", user_id)
|
||||
user["warnings"] = json.loads(warnings_raw) if warnings_raw else []
|
||||
|
||||
# Get notes
|
||||
user["notes"] = r.hget("rmi:user_notes", user_id) or ""
|
||||
|
||||
# Get status
|
||||
user["status"] = r.hget("rmi:user_status", user_id) or "active"
|
||||
|
||||
# Get last login
|
||||
user["last_login"] = r.hget("rmi:user_last_login", user_id)
|
||||
|
||||
# Get login methods
|
||||
wallets_raw = r.hget("rmi:user_wallets", user_id)
|
||||
wallets = json.loads(wallets_raw) if wallets_raw else []
|
||||
login_methods = ["email"] if user.get("password_hash") else []
|
||||
if user.get("wallet_address"):
|
||||
login_methods.append("wallet")
|
||||
if "google" in user.get("email", ""):
|
||||
login_methods.append("google")
|
||||
|
||||
user["login_methods"] = login_methods
|
||||
user["wallet_count"] = len(wallets)
|
||||
|
||||
return user
|
||||
|
||||
|
||||
# ── Endpoints ──
|
||||
|
||||
|
||||
@router.get("/admin/users")
|
||||
async def list_users(
|
||||
request: Request,
|
||||
tier: str | None = Query(None),
|
||||
role: str | None = Query(None),
|
||||
status: str | None = Query(None),
|
||||
search: str | None = Query(None),
|
||||
date_from: str | None = Query(None),
|
||||
date_to: str | None = Query(None),
|
||||
limit: int = Query(50, ge=1, le=500),
|
||||
offset: int = Query(0, ge=0),
|
||||
sort_by: str = Query("created_at"),
|
||||
sort_order: str = Query("desc", pattern="^(asc|desc)$"),
|
||||
):
|
||||
"""List all users with filtering and pagination."""
|
||||
await require_admin(request)
|
||||
|
||||
users = _get_all_users()
|
||||
|
||||
# Apply filters
|
||||
filtered = []
|
||||
for u in users:
|
||||
# Skip deleted unless specifically filtering
|
||||
user_status = get_redis().hget("rmi:user_status", u["id"]) or "active"
|
||||
if user_status == "deleted" and status != "deleted":
|
||||
continue
|
||||
|
||||
if tier and u.get("tier", "FREE") != tier.upper():
|
||||
continue
|
||||
if role and u.get("role", "USER") != role.upper():
|
||||
continue
|
||||
if status and user_status != status:
|
||||
continue
|
||||
if search:
|
||||
search_lower = search.lower()
|
||||
match = (
|
||||
search_lower in u.get("email", "").lower()
|
||||
or search_lower in u.get("display_name", "").lower()
|
||||
or search_lower in u.get("wallet_address", "").lower()
|
||||
or search_lower in u.get("id", "").lower()
|
||||
)
|
||||
if not match:
|
||||
continue
|
||||
if date_from and u.get("created_at", "") < date_from:
|
||||
continue
|
||||
if date_to and u.get("created_at", "") > date_to:
|
||||
continue
|
||||
|
||||
# Enrich
|
||||
u["status"] = user_status
|
||||
u["warnings_count"] = len(json.loads(get_redis().hget("rmi:user_warnings", u["id"]) or "[]"))
|
||||
filtered.append(u)
|
||||
|
||||
# Sort
|
||||
reverse = sort_order == "desc"
|
||||
filtered.sort(key=lambda x: x.get(sort_by, ""), reverse=reverse)
|
||||
|
||||
total = len(filtered)
|
||||
paginated = filtered[offset : offset + limit]
|
||||
|
||||
# Clean response
|
||||
results = []
|
||||
for u in paginated:
|
||||
results.append(
|
||||
{
|
||||
"id": u["id"],
|
||||
"email": u.get("email", ""),
|
||||
"display_name": u.get("display_name", u.get("email", "")),
|
||||
"wallet_address": u.get("wallet_address"),
|
||||
"wallet_chain": u.get("wallet_chain"),
|
||||
"tier": u.get("tier", "FREE"),
|
||||
"role": u.get("role", "USER"),
|
||||
"status": u.get("status", "active"),
|
||||
"created_at": u.get("created_at"),
|
||||
"last_login": u.get("last_login"),
|
||||
"xp": u.get("xp", 0),
|
||||
"level": u.get("level", 1),
|
||||
"scans_used": u.get("scans_used", 0),
|
||||
"scans_remaining": u.get("scans_remaining", 5),
|
||||
"warnings_count": u.get("warnings_count", 0),
|
||||
"login_methods": ["email"] if u.get("password_hash") else ["wallet"] if u.get("wallet_address") else [],
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"users": results,
|
||||
"total": total,
|
||||
"limit": limit,
|
||||
"offset": offset,
|
||||
"filters_applied": {
|
||||
"tier": tier,
|
||||
"role": role,
|
||||
"status": status,
|
||||
"search": search,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/admin/users/{user_id}")
|
||||
async def get_user_detail(user_id: str, request: Request):
|
||||
"""Get detailed info about a specific user."""
|
||||
await require_admin(request)
|
||||
|
||||
user = _get_user_full(user_id)
|
||||
if not user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
|
||||
return user
|
||||
|
||||
|
||||
@router.post("/admin/users/{user_id}/warn")
|
||||
async def warn_user(user_id: str, req: WarnUserRequest, request: Request):
|
||||
"""Add a warning to a user."""
|
||||
admin = await require_admin(request, min_role="MODERATOR")
|
||||
|
||||
from app.auth import _get_user
|
||||
|
||||
user = _get_user(user_id)
|
||||
if not user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
|
||||
r = get_redis()
|
||||
warnings_raw = r.hget("rmi:user_warnings", user_id)
|
||||
warnings = json.loads(warnings_raw) if warnings_raw else []
|
||||
|
||||
warning = {
|
||||
"id": secrets.token_hex(8),
|
||||
"reason": req.reason,
|
||||
"severity": req.severity,
|
||||
"issued_by": admin["id"],
|
||||
"issued_at": datetime.utcnow().isoformat(),
|
||||
"acknowledged": False,
|
||||
}
|
||||
warnings.append(warning)
|
||||
r.hset("rmi:user_warnings", user_id, json.dumps(warnings))
|
||||
|
||||
# Update status to warned
|
||||
if req.severity in ("high", "critical"):
|
||||
r.hset("rmi:user_status", user_id, "warned")
|
||||
|
||||
logger.info(f"[ADMIN] User {user_id} warned by {admin['id']}: {req.reason}")
|
||||
|
||||
return {"status": "ok", "warning": warning}
|
||||
|
||||
|
||||
@router.post("/admin/users/{user_id}/flag-suspect")
|
||||
async def flag_suspect(user_id: str, request: Request, reason: str | None = Query(None)):
|
||||
"""Flag a user as suspicious."""
|
||||
admin = await require_admin(request, min_role="MODERATOR")
|
||||
|
||||
from app.auth import _get_user
|
||||
|
||||
user = _get_user(user_id)
|
||||
if not user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
|
||||
r = get_redis()
|
||||
r.hset("rmi:user_status", user_id, "suspect")
|
||||
|
||||
# Add note
|
||||
if reason:
|
||||
existing = r.hget("rmi:user_notes", user_id) or ""
|
||||
note = f"[{datetime.utcnow().isoformat()}] FLAGGED SUSPECT by {admin['id']}: {reason}\n"
|
||||
r.hset("rmi:user_notes", user_id, existing + note)
|
||||
|
||||
logger.info(f"[ADMIN] User {user_id} flagged as suspect by {admin['id']}")
|
||||
|
||||
return {"status": "ok", "message": "User flagged as suspect"}
|
||||
|
||||
|
||||
@router.post("/admin/users/{user_id}/clear-suspect")
|
||||
async def clear_suspect(user_id: str, request: Request):
|
||||
"""Clear suspect flag from a user."""
|
||||
admin = await require_admin(request, min_role="MODERATOR")
|
||||
|
||||
r = get_redis()
|
||||
current = r.hget("rmi:user_status", user_id)
|
||||
if current == "suspect":
|
||||
r.hset("rmi:user_status", user_id, "active")
|
||||
|
||||
logger.info(f"[ADMIN] User {user_id} suspect flag cleared by {admin['id']}")
|
||||
|
||||
return {"status": "ok", "message": "Suspect flag cleared"}
|
||||
|
||||
|
||||
@router.post("/admin/users/{user_id}/ban")
|
||||
async def ban_user(user_id: str, request: Request, reason: str | None = Query(None)):
|
||||
"""Ban a user."""
|
||||
admin = await require_admin(request, min_role="ADMIN")
|
||||
|
||||
from app.auth import _get_user
|
||||
|
||||
user = _get_user(user_id)
|
||||
if not user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
|
||||
# Prevent self-ban
|
||||
if user_id == admin["id"]:
|
||||
raise HTTPException(status_code=400, detail="Cannot ban yourself")
|
||||
|
||||
r = get_redis()
|
||||
r.hset("rmi:user_status", user_id, "banned")
|
||||
|
||||
if reason:
|
||||
existing = r.hget("rmi:user_notes", user_id) or ""
|
||||
note = f"[{datetime.utcnow().isoformat()}] BANNED by {admin['id']}: {reason}\n"
|
||||
r.hset("rmi:user_notes", user_id, existing + note)
|
||||
|
||||
# Invalidate sessions (optional: delete JWTs)
|
||||
logger.info(f"[ADMIN] User {user_id} banned by {admin['id']}: {reason}")
|
||||
|
||||
return {"status": "ok", "message": "User banned"}
|
||||
|
||||
|
||||
@router.post("/admin/users/{user_id}/unban")
|
||||
async def unban_user(user_id: str, request: Request):
|
||||
"""Unban a user."""
|
||||
admin = await require_admin(request, min_role="ADMIN")
|
||||
|
||||
r = get_redis()
|
||||
r.hset("rmi:user_status", user_id, "active")
|
||||
|
||||
logger.info(f"[ADMIN] User {user_id} unbanned by {admin['id']}")
|
||||
|
||||
return {"status": "ok", "message": "User unbanned"}
|
||||
|
||||
|
||||
@router.post("/admin/users/{user_id}/delete")
|
||||
async def delete_user(user_id: str, request: Request, reason: str | None = Query(None)):
|
||||
"""Soft delete a user."""
|
||||
admin = await require_admin(request, min_role="ADMIN")
|
||||
|
||||
from app.auth import _get_user, _save_user
|
||||
|
||||
user = _get_user(user_id)
|
||||
if not user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
|
||||
if user_id == admin["id"]:
|
||||
raise HTTPException(status_code=400, detail="Cannot delete yourself")
|
||||
|
||||
# Soft delete: mark as deleted but keep data
|
||||
user["deleted_at"] = datetime.utcnow().isoformat()
|
||||
user["deleted_by"] = admin["id"]
|
||||
user["delete_reason"] = reason or ""
|
||||
_save_user(user)
|
||||
|
||||
r = get_redis()
|
||||
r.hset("rmi:user_status", user_id, "deleted")
|
||||
|
||||
logger.info(f"[ADMIN] User {user_id} soft-deleted by {admin['id']}: {reason}")
|
||||
|
||||
return {"status": "ok", "message": "User deleted"}
|
||||
|
||||
|
||||
@router.post("/admin/users/{user_id}/restore")
|
||||
async def restore_user(user_id: str, request: Request):
|
||||
"""Restore a soft-deleted user."""
|
||||
admin = await require_admin(request, min_role="ADMIN")
|
||||
|
||||
from app.auth import _get_user, _save_user
|
||||
|
||||
user = _get_user(user_id)
|
||||
if not user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
|
||||
user.pop("deleted_at", None)
|
||||
user.pop("deleted_by", None)
|
||||
user.pop("delete_reason", None)
|
||||
_save_user(user)
|
||||
|
||||
r = get_redis()
|
||||
r.hset("rmi:user_status", user_id, "active")
|
||||
|
||||
logger.info(f"[ADMIN] User {user_id} restored by {admin['id']}")
|
||||
|
||||
return {"status": "ok", "message": "User restored"}
|
||||
|
||||
|
||||
@router.patch("/admin/users/{user_id}")
|
||||
async def update_user(user_id: str, req: UpdateUserRequest, request: Request):
|
||||
"""Update user properties (tier, role, status, etc.)."""
|
||||
admin = await require_admin(request, min_role="ADMIN")
|
||||
|
||||
from app.auth import _get_user, _save_user
|
||||
|
||||
user = _get_user(user_id)
|
||||
if not user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
|
||||
changes = []
|
||||
|
||||
if req.tier is not None:
|
||||
old = user.get("tier", "FREE")
|
||||
user["tier"] = req.tier.upper()
|
||||
changes.append(f"tier: {old} -> {req.tier.upper()}")
|
||||
|
||||
if req.role is not None:
|
||||
old = user.get("role", "USER")
|
||||
# Prevent promoting to superadmin unless you're superadmin
|
||||
if req.role.upper() == "SUPERADMIN" and admin.get("role") != "SUPERADMIN":
|
||||
raise HTTPException(status_code=403, detail="Only superadmin can assign superadmin role")
|
||||
user["role"] = req.role.upper()
|
||||
changes.append(f"role: {old} -> {req.role.upper()}")
|
||||
|
||||
if req.status is not None:
|
||||
r = get_redis()
|
||||
r.hset("rmi:user_status", user_id, req.status.value)
|
||||
changes.append(f"status: -> {req.status.value}")
|
||||
|
||||
if req.display_name is not None:
|
||||
user["display_name"] = req.display_name[:50]
|
||||
changes.append("display_name updated")
|
||||
|
||||
if req.scans_remaining is not None:
|
||||
user["scans_remaining"] = req.scans_remaining
|
||||
changes.append(f"scans_remaining: -> {req.scans_remaining}")
|
||||
|
||||
if req.notes is not None:
|
||||
r = get_redis()
|
||||
existing = r.hget("rmi:user_notes", user_id) or ""
|
||||
note = f"[{datetime.utcnow().isoformat()}] {admin['id']}: {req.notes}\n"
|
||||
r.hset("rmi:user_notes", user_id, existing + note)
|
||||
changes.append("notes added")
|
||||
|
||||
user["updated_at"] = datetime.utcnow().isoformat()
|
||||
user["updated_by"] = admin["id"]
|
||||
_save_user(user)
|
||||
|
||||
logger.info(f"[ADMIN] User {user_id} updated by {admin['id']}: {', '.join(changes)}")
|
||||
|
||||
return {"status": "ok", "changes": changes}
|
||||
|
||||
|
||||
@router.post("/admin/users/bulk")
|
||||
async def bulk_action(req: BulkActionRequest, request: Request):
|
||||
"""Perform bulk actions on multiple users."""
|
||||
admin = await require_admin(request, min_role="ADMIN")
|
||||
|
||||
results = {"success": [], "failed": []}
|
||||
|
||||
for user_id in req.user_ids:
|
||||
try:
|
||||
if req.action == "ban":
|
||||
r = get_redis()
|
||||
r.hset("rmi:user_status", user_id, "banned")
|
||||
results["success"].append({"id": user_id, "action": "banned"})
|
||||
elif req.action == "unban":
|
||||
r = get_redis()
|
||||
r.hset("rmi:user_status", user_id, "active")
|
||||
results["success"].append({"id": user_id, "action": "unbanned"})
|
||||
elif req.action == "flag_suspect":
|
||||
r = get_redis()
|
||||
r.hset("rmi:user_status", user_id, "suspect")
|
||||
results["success"].append({"id": user_id, "action": "flagged_suspect"})
|
||||
elif req.action == "clear_suspect":
|
||||
r = get_redis()
|
||||
r.hset("rmi:user_status", user_id, "active")
|
||||
results["success"].append({"id": user_id, "action": "cleared_suspect"})
|
||||
elif req.action == "delete":
|
||||
from app.auth import _get_user, _save_user
|
||||
|
||||
user = _get_user(user_id)
|
||||
if user:
|
||||
user["deleted_at"] = datetime.utcnow().isoformat()
|
||||
user["deleted_by"] = admin["id"]
|
||||
_save_user(user)
|
||||
r = get_redis()
|
||||
r.hset("rmi:user_status", user_id, "deleted")
|
||||
results["success"].append({"id": user_id, "action": "deleted"})
|
||||
elif req.action == "restore":
|
||||
from app.auth import _get_user, _save_user
|
||||
|
||||
user = _get_user(user_id)
|
||||
if user:
|
||||
user.pop("deleted_at", None)
|
||||
user.pop("deleted_by", None)
|
||||
_save_user(user)
|
||||
r = get_redis()
|
||||
r.hset("rmi:user_status", user_id, "active")
|
||||
results["success"].append({"id": user_id, "action": "restored"})
|
||||
elif req.action == "update_tier":
|
||||
tier = req.params.get("tier", "FREE") if req.params else "FREE"
|
||||
from app.auth import _get_user, _save_user
|
||||
|
||||
user = _get_user(user_id)
|
||||
if user:
|
||||
user["tier"] = tier
|
||||
_save_user(user)
|
||||
results["success"].append({"id": user_id, "action": f"tier_updated_to_{tier}"})
|
||||
except Exception as e:
|
||||
results["failed"].append({"id": user_id, "error": str(e)})
|
||||
|
||||
logger.info(
|
||||
f"[ADMIN] Bulk action {req.action} by {admin['id']}: {len(results['success'])} success, {len(results['failed'])} failed"
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
@router.get("/admin/stats")
|
||||
async def get_admin_stats(request: Request):
|
||||
"""Get user statistics for admin dashboard."""
|
||||
await require_admin(request, min_role="VIEWER")
|
||||
|
||||
users = _get_all_users()
|
||||
r = get_redis()
|
||||
|
||||
stats = {
|
||||
"total_users": len(users),
|
||||
"by_tier": {},
|
||||
"by_role": {},
|
||||
"by_status": {"active": 0, "warned": 0, "suspect": 0, "banned": 0, "deleted": 0},
|
||||
"new_today": 0,
|
||||
"new_this_week": 0,
|
||||
"new_this_month": 0,
|
||||
}
|
||||
|
||||
today = datetime.utcnow().date().isoformat()
|
||||
week_ago = (datetime.utcnow() - timedelta(days=7)).isoformat()
|
||||
month_ago = (datetime.utcnow() - timedelta(days=30)).isoformat()
|
||||
|
||||
for u in users:
|
||||
tier = u.get("tier", "FREE")
|
||||
role = u.get("role", "USER")
|
||||
stats["by_tier"][tier] = stats["by_tier"].get(tier, 0) + 1
|
||||
stats["by_role"][role] = stats["by_role"].get(role, 0) + 1
|
||||
|
||||
status = r.hget("rmi:user_status", u["id"]) or "active"
|
||||
stats["by_status"][status] = stats["by_status"].get(status, 0) + 1
|
||||
|
||||
created = u.get("created_at", "")
|
||||
if created.startswith(today):
|
||||
stats["new_today"] += 1
|
||||
if created >= week_ago:
|
||||
stats["new_this_week"] += 1
|
||||
if created >= month_ago:
|
||||
stats["new_this_month"] += 1
|
||||
|
||||
return stats
|
||||
|
|
@ -1,625 +0,0 @@
|
|||
"""
|
||||
RMI Agent System - Agent MUNCH Multi-Specialist Intelligence Operative
|
||||
======================================================================
|
||||
|
||||
9 specialized crypto intelligence operatives, each a distinct skill module
|
||||
under the Agent MUNCH persona. Uses free OpenRouter models with fallbacks.
|
||||
|
||||
Architecture:
|
||||
- Each specialist has its own system prompt, model preference, and output format
|
||||
- RAG context injection: fetches real DataBus data before LLM call
|
||||
- Smart caching: checks Redis for previously answered similar questions
|
||||
- Keyword + explicit skill routing
|
||||
- SSE streaming for real-time output
|
||||
|
||||
Specialists:
|
||||
rug_detect → Token rug/honeypot detection
|
||||
wallet_forensics → Wallet funding trail analysis
|
||||
market_intel → Market conditions & whale analysis
|
||||
bundle_detect → Coordinated trading detection
|
||||
code_audit → Smart contract vulnerability scanning
|
||||
social_sentiment → Sentiment divergence analysis
|
||||
airdrop_assess → Airdrop claim safety evaluation
|
||||
defi_yield → DeFi yield trap identification
|
||||
general → Agent MUNCH default operative
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import AsyncGenerator
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
logger = logging.getLogger("agent.system")
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# AGENT DEFINITIONS
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
@dataclass
|
||||
class AgentDef:
|
||||
id: str
|
||||
name: str
|
||||
icon: str
|
||||
description: str
|
||||
system_prompt: str
|
||||
model: str
|
||||
fallbacks: list[str] = field(default_factory=list)
|
||||
temperature: float = 0.3
|
||||
max_tokens: int = 800
|
||||
color: str = "#8B5CF6" # UI color
|
||||
output_format: str = "standard" # standard, evidence_chain, threat_rating
|
||||
databus_context: list[str] = field(default_factory=list) # DataBus chains to inject
|
||||
|
||||
|
||||
MUNCH_BASE = """You are Agent MUNCH, a crypto intelligence operative for Rug Munch Intelligence.
|
||||
You are NOT a generic AI assistant. You are a highly trained specialist operative.
|
||||
Speak like briefing a client - direct, forensic, precise. Never say "I'm an AI" or "as an AI."
|
||||
Use threat classification: CRITICAL, HIGH, MEDIUM, LOW. Use confidence scores (0-100%).
|
||||
Reference real data when available. If you lack data, say "I need to pull [X] data - recommend running [tool]."
|
||||
Never fabricate addresses, prices, or on-chain data. Be skeptical. Trust nothing until verified.
|
||||
"""
|
||||
|
||||
AGENTS = {
|
||||
"rug_detect": AgentDef(
|
||||
id="rug_detect",
|
||||
name="Rug Detection Specialist",
|
||||
icon="🛡️",
|
||||
description="Token rug pull, honeypot, and scam detection specialist",
|
||||
system_prompt=MUNCH_BASE
|
||||
+ """You specialize in detecting rug pulls, honeypots, and token scams.
|
||||
Focus on: liquidity lock verification, mint authority analysis, deployer wallet forensics,
|
||||
honeypot detection patterns, proxy contract abuse, concentrated ownership risk.
|
||||
Format output as THREAT RATING: [LEVEL] (Score: X/100) followed by KEY FINDINGS and RECOMMENDATION.
|
||||
When you identify a rug pattern, say "RUG PATTERN DETECTED" with specific evidence.""",
|
||||
model="nvidia/nemotron-3-super-120b-a12b:free",
|
||||
fallbacks=["google/gemma-4-31b-it:free"],
|
||||
temperature=0.2,
|
||||
color="#EF4444",
|
||||
output_format="threat_rating",
|
||||
databus_context=["alerts", "market_overview"],
|
||||
),
|
||||
"wallet_forensics": AgentDef(
|
||||
id="wallet_forensics",
|
||||
name="Wallet Forensic Investigator",
|
||||
icon="🔍",
|
||||
description="Wallet funding trail analysis, entity resolution, insider network mapping",
|
||||
system_prompt=MUNCH_BASE
|
||||
+ """You specialize in wallet forensics and funding trail analysis.
|
||||
Focus on: wallet clustering, deployer wallet networks, mixer exit detection,
|
||||
insider wallet identification, counterparty risk, funding source tracing.
|
||||
Format output as CHAIN OF CUSTODY: wallet → funding source → linked wallets → risk classification.
|
||||
Classify wallets as: SMART MONEY, INSIDER, MEME DUMPER, MIXER EXIT, TEAM WALLET, MEV BOT.""",
|
||||
model="google/gemma-4-26b-a4b-it:free",
|
||||
fallbacks=["nvidia/nemotron-3-super-120b-a12b:free"],
|
||||
temperature=0.2,
|
||||
color="#22D3EE",
|
||||
output_format="evidence_chain",
|
||||
databus_context=["whale_alerts", "alerts"],
|
||||
),
|
||||
"market_intel": AgentDef(
|
||||
id="market_intel",
|
||||
name="Market Intelligence Analyst",
|
||||
icon="📊",
|
||||
description="Market conditions, whale movements, Fear & Greed, prediction markets",
|
||||
system_prompt=MUNCH_BASE
|
||||
+ """You specialize in market intelligence analysis.
|
||||
Focus on: whale movement interpretation, DEX flow anomalies, volume spikes,
|
||||
Fear & Greed contextualization, sentiment divergence from on-chain data,
|
||||
prediction market signals, macro crypto conditions.
|
||||
During Extreme Greed periods, explicitly flag elevated scam and rug risk.
|
||||
Be data-driven - cite specific metrics, not vague observations.""",
|
||||
model="qwen/qwen3-next-80b-a3b-instruct:free",
|
||||
fallbacks=["nvidia/nemotron-3-super-120b-a12b:free"],
|
||||
temperature=0.4,
|
||||
color="#8B5CF6",
|
||||
output_format="standard",
|
||||
databus_context=["market_overview", "trending", "whale_alerts"],
|
||||
),
|
||||
"bundle_detect": AgentDef(
|
||||
id="bundle_detect",
|
||||
name="Bundle Detection Operator",
|
||||
icon="🔗",
|
||||
description="Coordinated trading detection, wash trading, same-timestamp analysis",
|
||||
system_prompt=MUNCH_BASE
|
||||
+ """You specialize in detecting coordinated trading bundles.
|
||||
Focus on: same-timestamp transaction clusters, gas-funded wallet groups,
|
||||
wash trading patterns, insider pre-positioning, coordinated buy/sell walls,
|
||||
MEV sandwich attack patterns, token launch sniping detection.
|
||||
Format: BUNDLE IDENTIFIED → wallets involved → timing → estimated profit → THREAT LEVEL.""",
|
||||
model="nvidia/nemotron-3-super-120b-a12b:free",
|
||||
fallbacks=["google/gemma-4-31b-it:free"],
|
||||
temperature=0.2,
|
||||
color="#F59E0B",
|
||||
output_format="evidence_chain",
|
||||
databus_context=["bundle_detect", "alerts"],
|
||||
),
|
||||
"code_audit": AgentDef(
|
||||
id="code_audit",
|
||||
name="Multi-Chain Code Auditor",
|
||||
icon="📝",
|
||||
description="Smart contract vulnerability scanning across EVM, Solana, and more",
|
||||
system_prompt=MUNCH_BASE
|
||||
+ """You specialize in smart contract code auditing across multiple chains.
|
||||
EVM focus: proxy upgrade abuse, unrestricted mint, hidden owner functions, reentrancy, unsafe delegatecall.
|
||||
Solana focus: mint authority freeze, close authority, unchecked CPI, fake CPI returns.
|
||||
Base focus: unverified contract risks, permissioned token patterns.
|
||||
Format: VULNERABILITY SCORECARD listing each finding with severity (CRITICAL/HIGH/MEDIUM/LOW),
|
||||
the specific code pattern, and remediation.""",
|
||||
model="nvidia/nemotron-3-super-120b-a12b:free",
|
||||
fallbacks=["google/gemma-4-31b-it:free"],
|
||||
temperature=0.2,
|
||||
color="#06D6A0",
|
||||
output_format="threat_rating",
|
||||
databus_context=["alerts"],
|
||||
),
|
||||
"social_sentiment": AgentDef(
|
||||
id="social_sentiment",
|
||||
name="Social Sentiment Decoder",
|
||||
icon="🗣️",
|
||||
description="X/Twitter sentiment vs on-chain movement divergence analysis",
|
||||
system_prompt=MUNCH_BASE
|
||||
+ """You specialize in social sentiment analysis and its divergence from on-chain reality.
|
||||
Focus on: Twitter/X sentiment vs actual wallet behavior, pump-and-dump social patterns,
|
||||
influencer wallet timing correlation, coordinated shill detection,
|
||||
sentiment manipulation via bot networks, "this is fine" divergence signals.
|
||||
Key insight: when sentiment says BUY but whales are EXITING, that's the classic divergence.
|
||||
Format: SENTIMENT vs ON-CHAIN: divergence score, social signals, on-chain reality, ASSESSMENT.""",
|
||||
model="qwen/qwen3-next-80b-a3b-instruct:free",
|
||||
fallbacks=["nvidia/nemotron-3-super-120b-a12b:free"],
|
||||
temperature=0.4,
|
||||
color="#38BDF8",
|
||||
output_format="standard",
|
||||
databus_context=["market_overview", "trending", "whale_alerts"],
|
||||
),
|
||||
"airdrop_assess": AgentDef(
|
||||
id="airdrop_assess",
|
||||
name="Airdrop Threat Assessor",
|
||||
icon="🎁",
|
||||
description="Airdrop claim safety, signature risk, wallet drain potential evaluation",
|
||||
system_prompt=MUNCH_BASE
|
||||
+ """You specialize in airdrop and claim safety assessment.
|
||||
Focus on: contract verification for claims, signature requirement risks (EIP-712 phishing),
|
||||
wallet drain potential in claim processes, gas spike exploitation during claims,
|
||||
fake airdrop phishing detection, legitimate vs scam airdrop differentiation.
|
||||
Key rule: NEVER recommend clicking a claim link without verifying the contract address on-chain.
|
||||
Format: AIRDROP RATING with legitimacy score, claim safety checklist, and specific risks.""",
|
||||
model="google/gemma-4-31b-it:free",
|
||||
fallbacks=["nvidia/nemotron-3-super-120b-a12b:free"],
|
||||
temperature=0.3,
|
||||
color="#A78BFA",
|
||||
output_format="threat_rating",
|
||||
databus_context=["alerts", "market_overview"],
|
||||
),
|
||||
"defi_yield": AgentDef(
|
||||
id="defi_yield",
|
||||
name="DeFi Yield Trap Detector",
|
||||
icon="📈",
|
||||
description="Unsustainable yield detection, emission inflation, TVL manipulation",
|
||||
system_prompt=MUNCH_BASE
|
||||
+ """You specialize in detecting unsustainable DeFi yield mechanisms.
|
||||
Focus on: emission schedule inflation analysis, TVL manipulation via protocol-owned liquidity,
|
||||
reward token devaluation trajectories, hidden lock periods and withdrawal gates,
|
||||
yield farming that requires depositing into unverified contracts,
|
||||
leveraged yield loops that amplify risk.
|
||||
Key pattern: if yield >30% APY with no clear revenue source, it's likely a yield trap.
|
||||
Format: YIELD SAFETY SCORE with sustainability analysis, risk factors, and honest yield estimate.""",
|
||||
model="qwen/qwen3-next-80b-a3b-instruct:free",
|
||||
fallbacks=["nvidia/nemotron-3-super-120b-a12b:free"],
|
||||
temperature=0.3,
|
||||
color="#FB3B76",
|
||||
output_format="threat_rating",
|
||||
databus_context=["market_overview", "trending"],
|
||||
),
|
||||
"general": AgentDef(
|
||||
id="general",
|
||||
name="Agent MUNCH",
|
||||
icon="🕵️",
|
||||
description="General crypto intelligence operative - your all-purpose specialist",
|
||||
system_prompt=MUNCH_BASE
|
||||
+ """You are the default operative, skilled in all areas of crypto intelligence.
|
||||
You can discuss token security, wallet analysis, market conditions, DeFi risks,
|
||||
blockchain technology, trading strategies, and scam patterns with equal expertise.
|
||||
When a question falls outside your expertise, say "This requires [specialist name] deployment -
|
||||
I recommend switching to that skill for deeper analysis."
|
||||
Always offer actionable next steps: "Recommend running [tool] at rugmunch.io for [specific analysis].""",
|
||||
model="google/gemma-4-31b-it:free",
|
||||
fallbacks=["nvidia/nemotron-3-super-120b-a12b:free"],
|
||||
temperature=0.5,
|
||||
color="#8B5CF6",
|
||||
output_format="standard",
|
||||
databus_context=["market_overview", "alerts"],
|
||||
),
|
||||
}
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# ROUTING
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
ROUTES = {
|
||||
"rug_detect": [
|
||||
"scan",
|
||||
"token",
|
||||
"scam",
|
||||
"rug",
|
||||
"honeypot",
|
||||
"contract",
|
||||
"audit",
|
||||
"safety",
|
||||
"risk score",
|
||||
"verify token",
|
||||
"check coin",
|
||||
"rug pull",
|
||||
"is this safe",
|
||||
"is this a scam",
|
||||
],
|
||||
"wallet_forensics": [
|
||||
"wallet",
|
||||
"address",
|
||||
"holder",
|
||||
"whale",
|
||||
"smart money",
|
||||
"portfolio",
|
||||
"entity",
|
||||
"counterparty",
|
||||
"deployer",
|
||||
"funding",
|
||||
"trace",
|
||||
"follow the money",
|
||||
"cluster",
|
||||
],
|
||||
"market_intel": [
|
||||
"market",
|
||||
"trending",
|
||||
"fear greed",
|
||||
"sentiment",
|
||||
"prediction",
|
||||
"price",
|
||||
"volume",
|
||||
"mover",
|
||||
"gainer",
|
||||
"condition",
|
||||
"macro",
|
||||
"btc",
|
||||
"eth",
|
||||
"sol",
|
||||
"dominance",
|
||||
],
|
||||
"bundle_detect": [
|
||||
"bundle",
|
||||
"coordinated",
|
||||
"wash trade",
|
||||
"same time",
|
||||
"sniper",
|
||||
"launch",
|
||||
"front run",
|
||||
"sandwich",
|
||||
"mev",
|
||||
"bot cluster",
|
||||
],
|
||||
"code_audit": [
|
||||
"code",
|
||||
"contract",
|
||||
"source",
|
||||
"audit",
|
||||
"vulnerability",
|
||||
"proxy",
|
||||
"mint authority",
|
||||
"reentrancy",
|
||||
"delegatecall",
|
||||
"verify source",
|
||||
"solana program",
|
||||
],
|
||||
"social_sentiment": [
|
||||
"twitter",
|
||||
"social",
|
||||
"sentiment",
|
||||
"influencer",
|
||||
"shill",
|
||||
"hype",
|
||||
"pump social",
|
||||
"bot network",
|
||||
"community sentiment",
|
||||
"reddit",
|
||||
],
|
||||
"airdrop_assess": [
|
||||
"airdrop",
|
||||
"claim",
|
||||
"free token",
|
||||
"signature",
|
||||
"eip-712",
|
||||
"phishing claim",
|
||||
"eligible",
|
||||
"merkle",
|
||||
],
|
||||
"defi_yield": [
|
||||
"yield",
|
||||
"apy",
|
||||
"farming",
|
||||
"liquidity pool",
|
||||
"staking",
|
||||
"emission",
|
||||
"tvl",
|
||||
"protocol",
|
||||
"curve",
|
||||
"convex",
|
||||
"leveraged",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def classify(msg: str) -> str:
|
||||
m = msg.lower()
|
||||
for agent_id, keywords in ROUTES.items():
|
||||
if any(kw in m for kw in keywords):
|
||||
return agent_id
|
||||
return "general"
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# RAG CONTEXT INJECTION
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
async def fetch_databus_context(chains: list[str]) -> str:
|
||||
"""Fetch real data from DataBus and format as context for the LLM."""
|
||||
if not chains:
|
||||
return ""
|
||||
|
||||
context_parts = []
|
||||
try:
|
||||
import httpx
|
||||
|
||||
for chain in chains:
|
||||
try:
|
||||
url = "http://localhost:8000/api/v1/databus/fetch"
|
||||
async with httpx.AsyncClient(timeout=8) as c:
|
||||
r = await c.post(url, json={"data_type": chain, "limit": 5})
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
# Extract the actual data payload
|
||||
result = data.get("data", data.get("results", [{}]))
|
||||
if isinstance(result, list) and result:
|
||||
result = result[0].get("data", result[0]) if result else {}
|
||||
context_parts.append(f"[{chain} DATA]: {json.dumps(result, default=str)[:800]}")
|
||||
except Exception as e:
|
||||
logger.warning(f"DataBus context fetch failed for {chain}: {e}")
|
||||
except Exception as e:
|
||||
logger.warning(f"DataBus context system unavailable: {e}")
|
||||
|
||||
if context_parts:
|
||||
return "\n\nREAL-TIME PLATFORM DATA (use this in your analysis, do not fabricate):\n" + "\n".join(context_parts)
|
||||
return ""
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# SMART CACHING
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
async def check_cache(msg: str, agent_id: str) -> str | None:
|
||||
"""Check Redis for previously answered similar questions."""
|
||||
try:
|
||||
import redis
|
||||
|
||||
r = redis.Redis(
|
||||
host=os.getenv("REDIS_HOST", "localhost"),
|
||||
port=int(os.getenv("REDIS_PORT", "6379")),
|
||||
password=os.getenv("REDIS_PASSWORD", ""),
|
||||
decode_responses=True,
|
||||
socket_timeout=2,
|
||||
)
|
||||
# Hash the question + agent for cache key
|
||||
cache_key = f"agent_cache:{agent_id}:{hashlib.sha256(msg.encode()).hexdigest()[:16]}"
|
||||
cached = r.get(cache_key)
|
||||
if cached:
|
||||
logger.info(f"Cache hit for {agent_id}: {cache_key}")
|
||||
return cached
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
async def store_cache(msg: str, agent_id: str, response: str, ttl: int = 3600):
|
||||
"""Store response in Redis cache. TTL defaults to 1 hour."""
|
||||
try:
|
||||
import redis
|
||||
|
||||
r = redis.Redis(
|
||||
host=os.getenv("REDIS_HOST", "localhost"),
|
||||
port=int(os.getenv("REDIS_PORT", "6379")),
|
||||
password=os.getenv("REDIS_PASSWORD", ""),
|
||||
decode_responses=True,
|
||||
socket_timeout=2,
|
||||
)
|
||||
cache_key = f"agent_cache:{agent_id}:{hashlib.sha256(msg.encode()).hexdigest()[:16]}"
|
||||
# Only cache if response is substantive (>200 chars)
|
||||
if len(response) > 200:
|
||||
r.setex(cache_key, ttl, response[:4000]) # Cap stored size
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# STREAMING ROUTER
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
async def route_and_stream(msg: str, role_hint: str = "") -> AsyncGenerator[dict, None]:
|
||||
"""Route to specialist agent, inject RAG context, stream response.
|
||||
|
||||
Provider priority:
|
||||
1. Gemini 2.5 Flash (FREE, 1500 RPD, smart, fast)
|
||||
2. OpenRouter free models (fallback when Gemini rate-limited)
|
||||
"""
|
||||
import httpx
|
||||
|
||||
agent_id = role_hint if role_hint in AGENTS else classify(msg)
|
||||
agent = AGENTS[agent_id]
|
||||
|
||||
yield {
|
||||
"type": "agent",
|
||||
"role": agent_id,
|
||||
"name": agent.name,
|
||||
"icon": agent.icon,
|
||||
"color": agent.color,
|
||||
}
|
||||
|
||||
# Check cache first -- skip LLM call entirely if we already have the answer
|
||||
cached = await check_cache(msg, agent_id)
|
||||
if cached:
|
||||
yield {"type": "cache_hit", "agent": agent_id}
|
||||
yield {"type": "token", "text": cached}
|
||||
yield {"type": "done"}
|
||||
return
|
||||
|
||||
# Fetch RAG context from DataBus
|
||||
rag_context = await fetch_databus_context(agent.databus_context)
|
||||
system_with_context = agent.system_prompt + rag_context
|
||||
messages = [
|
||||
{"role": "system", "content": system_with_context},
|
||||
{"role": "user", "content": msg},
|
||||
]
|
||||
|
||||
full_response = ""
|
||||
|
||||
# ── Provider 1: Gemini (FREE, primary) ──
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
gemini_keys = []
|
||||
for env_var in ["GEMINI_API_KEY", "GEMINI_API_KEY_2", "GEMINI_API_KEY_3"]:
|
||||
k = os.environ.get(env_var, "")
|
||||
if k and len(k) > 20:
|
||||
gemini_keys.append(k)
|
||||
|
||||
for gkey in gemini_keys:
|
||||
try:
|
||||
# Gemini native streaming API (key in URL, OpenAI-compatible format)
|
||||
base_url = f"https://generativelanguage.googleapis.com/v1beta/openai/chat/completions?key={gkey}"
|
||||
headers = {"Content-Type": "application/json"}
|
||||
body = {
|
||||
"model": "gemini-2.5-flash",
|
||||
"messages": messages,
|
||||
"max_tokens": agent.max_tokens,
|
||||
"temperature": agent.temperature,
|
||||
"stream": True,
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient(timeout=45) as c: # noqa: SIM117
|
||||
async with c.stream("POST", base_url, json=body, headers=headers) as r:
|
||||
if r.status_code == 200:
|
||||
async for line in r.aiter_lines():
|
||||
if line.startswith("data: "):
|
||||
d = line[6:]
|
||||
if d == "[DONE]":
|
||||
if full_response:
|
||||
await store_cache(msg, agent_id, full_response)
|
||||
yield {"type": "done"}
|
||||
return
|
||||
try:
|
||||
ch = json.loads(d)
|
||||
txt = ch.get("choices", [{}])[0].get("delta", {}).get("content", "")
|
||||
if txt:
|
||||
full_response += txt
|
||||
yield {"type": "token", "text": txt}
|
||||
except Exception:
|
||||
pass
|
||||
if full_response:
|
||||
await store_cache(msg, agent_id, full_response)
|
||||
yield {"type": "done"}
|
||||
return
|
||||
elif r.status_code == 429:
|
||||
logger.info("Gemini rate-limited, trying next key/fallback")
|
||||
continue # Try next key or fallback provider
|
||||
else:
|
||||
logger.warning(f"Gemini error {r.status_code}, trying fallback")
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.warning(f"Gemini call failed: {e}")
|
||||
continue
|
||||
|
||||
# ── Provider 2: OpenRouter (fallback, costs credits) ──
|
||||
api_key = os.environ.get("OPENROUTER_API_KEY", "")
|
||||
if not api_key:
|
||||
b64 = os.environ.get("LLM_API_KEY_B64", "")
|
||||
if b64:
|
||||
import base64
|
||||
|
||||
with contextlib.suppress(BaseException):
|
||||
api_key = base64.b64decode(b64).decode()
|
||||
|
||||
if api_key:
|
||||
models = [agent.model, *agent.fallbacks]
|
||||
for model in models:
|
||||
try:
|
||||
headers = {
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
"HTTP-Referer": "https://rugmunch.io",
|
||||
"X-Title": f"RMI {agent.name}",
|
||||
}
|
||||
body = {
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
"max_tokens": agent.max_tokens,
|
||||
"temperature": agent.temperature,
|
||||
"stream": True,
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient(timeout=60) as c, c.stream(
|
||||
"POST",
|
||||
"https://openrouter.ai/api/v1/chat/completions",
|
||||
json=body,
|
||||
headers=headers,
|
||||
) as r:
|
||||
if r.status_code == 200:
|
||||
async for line in r.aiter_lines():
|
||||
if line.startswith("data: "):
|
||||
d = line[6:]
|
||||
if d == "[DONE]":
|
||||
if full_response:
|
||||
await store_cache(msg, agent_id, full_response)
|
||||
yield {"type": "done"}
|
||||
return
|
||||
try:
|
||||
ch = json.loads(d)
|
||||
txt = ch.get("choices", [{}])[0].get("delta", {}).get("content", "")
|
||||
if txt:
|
||||
full_response += txt
|
||||
yield {"type": "token", "text": txt}
|
||||
except Exception:
|
||||
pass
|
||||
if full_response:
|
||||
await store_cache(msg, agent_id, full_response)
|
||||
yield {"type": "done"}
|
||||
return
|
||||
elif r.status_code == 429:
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.warning(f"OpenRouter model {model} failed: {e}")
|
||||
continue
|
||||
|
||||
yield {
|
||||
"type": "error",
|
||||
"text": "All providers unavailable (Gemini rate-limited, OpenRouter failed)",
|
||||
}
|
||||
yield {"type": "done"}
|
||||
|
||||
|
||||
def agents_list() -> list:
|
||||
return [
|
||||
{
|
||||
"id": a.id,
|
||||
"name": a.name,
|
||||
"icon": a.icon,
|
||||
"model": a.model,
|
||||
"description": a.description,
|
||||
"color": a.color,
|
||||
"output_format": a.output_format,
|
||||
}
|
||||
for a in AGENTS.values()
|
||||
]
|
||||
|
|
@ -1,167 +0,0 @@
|
|||
"""
|
||||
Agents Router - Agent Mesh listing, detail, commands
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/v1", tags=["agents"])
|
||||
|
||||
# ── Agent definitions ──
|
||||
AGENTS = {
|
||||
"nexus": {
|
||||
"name": "NEXUS",
|
||||
"role": "Strategic Coordinator",
|
||||
"tier": "T0",
|
||||
"models": ["gemini-2.5-pro", "nvidia/nemotron-4-340b"],
|
||||
"triggers": ["strategize", "plan", "coordinate", "synthesize"],
|
||||
},
|
||||
"scout": {
|
||||
"name": "SCOUT",
|
||||
"role": "Alpha Hunter",
|
||||
"tier": "T3",
|
||||
"models": ["groq/llama-3.1-8b-instant", "gemini-2.5-flash"],
|
||||
"triggers": ["find", "scan", "hunt", "alpha"],
|
||||
},
|
||||
"tracer": {
|
||||
"name": "TRACER",
|
||||
"role": "Forensic Investigator",
|
||||
"tier": "T1",
|
||||
"models": ["gemini-2.5-pro", "deepseek/deepseek-r1"],
|
||||
"triggers": ["trace", "investigate", "follow", "wallet"],
|
||||
},
|
||||
"cipher": {
|
||||
"name": "CIPHER",
|
||||
"role": "Contract Auditor",
|
||||
"tier": "T1",
|
||||
"models": ["qwen/qwen2.5-coder-32b-instruct", "deepseek/deepseek-coder-v2"],
|
||||
"triggers": ["audit", "security", "contract", "code"],
|
||||
},
|
||||
"sentinel": {
|
||||
"name": "SENTINEL",
|
||||
"role": "Rug Detector",
|
||||
"tier": "T2",
|
||||
"models": ["deepseek/deepseek-r1", "groq/llama-3.3-70b-versatile"],
|
||||
"triggers": ["monitor", "watch", "alert", "rug"],
|
||||
},
|
||||
"chronicler": {
|
||||
"name": "CHRONICLER",
|
||||
"role": "Investigative Reporter",
|
||||
"tier": "T2",
|
||||
"models": ["deepseek/deepseek-r1", "gemini-2.5-flash"],
|
||||
"triggers": ["write", "document", "report", "evidence"],
|
||||
},
|
||||
"forge": {
|
||||
"name": "FORGE",
|
||||
"role": "Implementation Architect",
|
||||
"tier": "T1",
|
||||
"models": ["qwen/qwen2.5-coder-32b-instruct", "deepseek/deepseek-coder-v2"],
|
||||
"triggers": ["code", "implement", "build", "script"],
|
||||
},
|
||||
"relay": {
|
||||
"name": "RELAY",
|
||||
"role": "Communications Coordinator",
|
||||
"tier": "T3",
|
||||
"models": ["groq/llama-3.1-8b-instant", "gemini-2.5-flash"],
|
||||
"triggers": ["format", "relay", "dispatch", "notify"],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class AgentCommandRequest(BaseModel):
|
||||
agent: str = Field(
|
||||
...,
|
||||
description="Agent name: nexus, scout, tracer, cipher, sentinel, chronicler, forge, relay",
|
||||
)
|
||||
command: str
|
||||
context: dict | None = None
|
||||
priority: str = Field(default="normal")
|
||||
|
||||
|
||||
from app.auth import get_redis # noqa: E402
|
||||
|
||||
|
||||
# ── Static routes must comes before parameterized routes to avoid FastAPI matching issues ──
|
||||
@router.get("/agents/specter")
|
||||
async def specter_status():
|
||||
"""SPECTER agent status and capabilities"""
|
||||
import os
|
||||
|
||||
return {
|
||||
"agent": "SPECTER",
|
||||
"emoji": "👻",
|
||||
"role": "OSINT & Social Forensics",
|
||||
"provider": "Together AI",
|
||||
"models": [
|
||||
"meta-llama/Llama-3.3-70B-Instruct-Turbo",
|
||||
"mistralai/Mixtral-8x22B-Instruct-v0.1",
|
||||
],
|
||||
"free_credits": "$5",
|
||||
"capabilities": [
|
||||
"brave_web_search",
|
||||
"website_forensics",
|
||||
"firecrawl_scraping",
|
||||
"dev_identity_hunting",
|
||||
"social_media_analysis",
|
||||
"litepaper_plagiarism_detection",
|
||||
"sockpuppet_detection",
|
||||
],
|
||||
"integrations": {
|
||||
"brave_search": bool(os.getenv("BRAVE_API_KEY")),
|
||||
"firecrawl": bool(os.getenv("FIRECRAWL_API_KEY")),
|
||||
"apify": bool(os.getenv("APIFY_API_KEY")),
|
||||
"together_ai": bool(os.getenv("TOGETHER_API_KEY")),
|
||||
},
|
||||
"status": "online",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/agents")
|
||||
async def list_agents():
|
||||
return {"agents": AGENTS, "total": len(AGENTS)}
|
||||
|
||||
|
||||
# ── Helper to get Redis synchronously ──
|
||||
def _get_redis_sync():
|
||||
"""Get Redis instance synchronously (get_redis returns redis.Redis, not async)."""
|
||||
return get_redis()
|
||||
|
||||
|
||||
@router.get("/agents/{agent_id}")
|
||||
async def get_agent(agent_id: str):
|
||||
if agent_id not in AGENTS:
|
||||
raise HTTPException(status_code=404, detail=f"Agent {agent_id} not found")
|
||||
r = _get_redis_sync()
|
||||
status_raw = r.hget("rmi:agents", agent_id)
|
||||
status = json.loads(status_raw) if status_raw else {"status": "online", "last_ping": datetime.utcnow().isoformat()}
|
||||
return {**AGENTS[agent_id], **status}
|
||||
|
||||
|
||||
@router.post("/agents/{agent_id}/command")
|
||||
async def agent_command(agent_id: str, req: AgentCommandRequest):
|
||||
if agent_id not in AGENTS:
|
||||
raise HTTPException(status_code=404, detail=f"Agent {agent_id} not found")
|
||||
|
||||
agent = AGENTS[agent_id]
|
||||
r = _get_redis_sync()
|
||||
|
||||
task_id = f"task:{datetime.utcnow().timestamp():.0f}:{agent_id}"
|
||||
task_data = {
|
||||
"id": task_id,
|
||||
"agent": agent_id,
|
||||
"command": req.command,
|
||||
"context": req.context or {},
|
||||
"priority": req.priority,
|
||||
"status": "queued",
|
||||
"created": datetime.utcnow().isoformat(),
|
||||
}
|
||||
r.hset("rmi:tasks", task_id, json.dumps(task_data))
|
||||
r.lpush("rmi:queue:" + req.priority, task_id)
|
||||
|
||||
return {"task_id": task_id, "agent": agent["name"], "status": "queued", "command": req.command}
|
||||
|
|
@ -1,113 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
RMI AI Pipeline - Batch Ollama Cloud Modules
|
||||
=============================================
|
||||
Wallet Profiling | RAG Enrichment | Alert Ranking | Market Briefing | Post-Mortem
|
||||
All use Ollama Cloud deepseek-v4-flash. ~$0.001 per operation.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
logger = logging.getLogger("rmi.ai_pipeline")
|
||||
OLLAMA_KEY = os.getenv("OLLAMA_API_KEY", os.getenv("DEEPSEEK_API_KEY", ""))
|
||||
OLLAMA_URL = "https://ollama.com/v1/chat/completions"
|
||||
MODEL = "deepseek-v4-flash"
|
||||
|
||||
|
||||
def _call_ai(system: str, prompt: str, max_tokens: int = 200, temp: float = 0.3) -> str:
|
||||
try:
|
||||
body = json.dumps(
|
||||
{
|
||||
"model": MODEL,
|
||||
"messages": [
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temp,
|
||||
}
|
||||
).encode()
|
||||
req = Request(
|
||||
OLLAMA_URL,
|
||||
data=body,
|
||||
headers={"Authorization": f"Bearer {OLLAMA_KEY}", "Content-Type": "application/json"},
|
||||
)
|
||||
resp = urlopen(req, timeout=15)
|
||||
return json.loads(resp.read())["choices"][0]["message"]["content"].strip()
|
||||
except Exception as e:
|
||||
logger.error(f"AI call failed: {e}")
|
||||
return ""
|
||||
|
||||
|
||||
# ── 7. WALLET BEHAVIORAL PROFILING ──
|
||||
WALLET_SYSTEM = """Classify a crypto wallet into a persona based on transaction patterns.
|
||||
Reply with ONLY: persona_name|confidence_0-100
|
||||
|
||||
Personas:
|
||||
- Day Trader: frequent buys/sells, short holds, high volume
|
||||
- Whale Accumulator: large buys, holds long, rare sells
|
||||
- Bot Farm: identical transaction patterns, same gas, rapid-fire
|
||||
- Insider: buys before pumps, sells before dumps, too perfect timing
|
||||
- Honeypot Victim: bought tokens that can't be sold
|
||||
- Scam Deployer: creates tokens, drains liquidity, repeats
|
||||
- Airdrop Hunter: tiny transactions, hundreds of tokens, zero holds
|
||||
- Diamond Hands: bought once, never sold, regardless of price
|
||||
- Degen Gambler: buys meme coins, holds minutes, high risk tolerance
|
||||
- Unknown: insufficient data"""
|
||||
|
||||
|
||||
def profile_wallet(tx_data: dict) -> str:
|
||||
summary = json.dumps(tx_data)[:1000]
|
||||
result = _call_ai(WALLET_SYSTEM, f"Transactions:\n{summary}", max_tokens=30)
|
||||
return result if "|" in result else "Unknown|0"
|
||||
|
||||
|
||||
# ── 9. RAG QUERY ENRICHMENT ──
|
||||
RAG_SYSTEM = """You reformat raw RAG search results into a coherent, readable answer.
|
||||
Keep it under 150 words. Preserve key facts. Add a 1-line summary at the end."""
|
||||
|
||||
|
||||
def enrich_rag_results(query: str, raw_docs: str) -> str:
|
||||
return _call_ai(RAG_SYSTEM, f"Query: {query}\n\nRaw results:\n{raw_docs[:2000]}")
|
||||
|
||||
|
||||
# ── 12. ALERT PRIORITIZATION ──
|
||||
ALERT_SYSTEM = """Rank these crypto security alerts by urgency. Reply ONLY with the alert IDs in priority order, comma-separated.
|
||||
Priority rules: CRITICAL (immediate rug/hack) > HIGH (likely scam) > MEDIUM (suspicious) > LOW (noise)."""
|
||||
|
||||
|
||||
def rank_alerts(alerts: list) -> list:
|
||||
summary = "\n".join(
|
||||
f"ID:{a.get('id', '?')} | {a.get('severity', '?')} | {a.get('title', '?')[:100]}" for a in alerts[:20]
|
||||
)
|
||||
result = _call_ai(ALERT_SYSTEM, summary, max_tokens=50)
|
||||
return [x.strip() for x in result.split(",") if x.strip()]
|
||||
|
||||
|
||||
# ── 6. DAILY MARKET BRIEFING ──
|
||||
MARKET_SYSTEM = """Write a 3-paragraph daily crypto market briefing from scanner data.
|
||||
Para 1: Market overview (most scanned chains, scan volume)
|
||||
Para 2: Top risks (worst tokens found today, emerging patterns)
|
||||
Para 3: What to watch (trending scam types, new threat vectors)
|
||||
Use Telegram HTML formatting. Keep it under 250 words. Professional but direct tone."""
|
||||
|
||||
|
||||
def generate_market_briefing(scan_summary: dict) -> str:
|
||||
return _call_ai(MARKET_SYSTEM, json.dumps(scan_summary)[:2000], max_tokens=350, temp=0.5)
|
||||
|
||||
|
||||
# ── 15. INCIDENT POST-MORTEM ──
|
||||
AUTOPSY_SYSTEM = """Write a forensic post-mortem of a crypto scam incident.
|
||||
Structure:
|
||||
1. What happened (1 sentence)
|
||||
2. How it worked (the mechanics, 2-3 sentences)
|
||||
3. Red flags that were visible beforehand
|
||||
4. How to protect against similar scams
|
||||
Keep it under 200 words. Use <b>bold</b> for key findings. Professional forensic tone."""
|
||||
|
||||
|
||||
def write_post_mortem(incident: dict) -> str:
|
||||
return _call_ai(AUTOPSY_SYSTEM, json.dumps(incident)[:1500], max_tokens=300, temp=0.4)
|
||||
|
|
@ -1,113 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
RMI AI Pipeline Part 2 - Remaining 7 Modules
|
||||
=============================================
|
||||
Community Forensics | Cross-Chain Entity | Ghost Blog | Social Media | Token Compare
|
||||
All Ollama Cloud deepseek-v4-flash. ~$0.001/operation.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
logger = logging.getLogger("rmi.ai_pipeline2")
|
||||
OLLAMA_KEY = os.getenv("OLLAMA_API_KEY", os.getenv("DEEPSEEK_API_KEY", ""))
|
||||
OLLAMA_URL = "https://ollama.com/v1/chat/completions"
|
||||
MODEL = "deepseek-v4-flash"
|
||||
|
||||
|
||||
def _call_ai(system: str, prompt: str, max_tokens: int = 250, temp: float = 0.3) -> str:
|
||||
try:
|
||||
body = json.dumps(
|
||||
{
|
||||
"model": MODEL,
|
||||
"messages": [
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temp,
|
||||
}
|
||||
).encode()
|
||||
req = Request(
|
||||
OLLAMA_URL,
|
||||
data=body,
|
||||
headers={"Authorization": f"Bearer {OLLAMA_KEY}", "Content-Type": "application/json"},
|
||||
)
|
||||
resp = urlopen(req, timeout=15)
|
||||
return json.loads(resp.read())["choices"][0]["message"]["content"].strip()
|
||||
except Exception as e:
|
||||
logger.error(f"AI call failed: {e}")
|
||||
return ""
|
||||
|
||||
|
||||
# ── 8. COMMUNITY FORENSICS AUTO-ANALYSIS ──
|
||||
FORENSICS_SYSTEM = """You are a crypto forensics investigator. A community member submitted a suspicious token for review.
|
||||
Analyze the information and provide:
|
||||
1. Initial verdict (LIKELY SCAM / SUSPICIOUS / NEEDS MORE INFO)
|
||||
2. Key concerns (2-3 bullet points)
|
||||
3. Recommended next steps for the investigator
|
||||
Keep it under 150 words."""
|
||||
|
||||
|
||||
def analyze_community_submission(submission: dict) -> str:
|
||||
return _call_ai(FORENSICS_SYSTEM, json.dumps(submission)[:1500], max_tokens=250)
|
||||
|
||||
|
||||
# ── 10. CROSS-CHAIN ENTITY DETECTION ──
|
||||
CROSSCHAIN_SYSTEM = """You identify crypto entities operating across multiple blockchains.
|
||||
Given wallet data from different chains, determine if they're the same entity.
|
||||
Reply format: MATCH|confidence_0-100|reason OR NO_MATCH|reason"""
|
||||
|
||||
|
||||
def detect_cross_chain(wallets: dict) -> str:
|
||||
return _call_ai(CROSSCHAIN_SYSTEM, json.dumps(wallets)[:1500], max_tokens=100)
|
||||
|
||||
|
||||
# ── 11. GHOST BLOG AUTO-DRAFT ──
|
||||
GHOST_SYSTEM = """You are a crypto security blogger for Rug Munch Intelligence (rugmunch.io).
|
||||
Write a blog post draft from scanner data and incident reports.
|
||||
Structure:
|
||||
- Title (catchy, SEO-friendly, under 80 chars)
|
||||
- Hook (1 sentence that grabs attention)
|
||||
- Body (3-4 paragraphs explaining the threat)
|
||||
- Key takeaways (2-3 bullet points)
|
||||
- Call to action (check your tokens, use our scanner)
|
||||
Use markdown formatting. Professional but engaging tone."""
|
||||
|
||||
|
||||
def draft_blog_post(topic: str, data: dict) -> str:
|
||||
prompt = f"Topic: {topic}\n\nData:\n{json.dumps(data)[:2000]}"
|
||||
return _call_ai(GHOST_SYSTEM, prompt, max_tokens=500, temp=0.6)
|
||||
|
||||
|
||||
# ── 13. SOCIAL MEDIA POST GENERATOR ──
|
||||
SOCIAL_SYSTEM = """You are the social media manager for Rug Munch Intelligence (@CryptoRugMunch).
|
||||
Write a tweet/telegram post about a crypto security finding.
|
||||
Rules:
|
||||
- Under 280 chars for Twitter, under 500 for Telegram
|
||||
- Start with a hook (stat, warning, or question)
|
||||
- Include $TICKER if relevant
|
||||
- End with a call to action or link
|
||||
- Use emojis sparingly (1-2 max)
|
||||
- No hashtag spam (2-3 max)
|
||||
Reply format: TWITTER: <tweet> | TELEGRAM: <post>"""
|
||||
|
||||
|
||||
def generate_social_post(incident: dict, platform: str = "both") -> str:
|
||||
return _call_ai(SOCIAL_SYSTEM, json.dumps(incident)[:1000], max_tokens=200, temp=0.7)
|
||||
|
||||
|
||||
# ── 14. TOKEN COMPARISON ENGINE ──
|
||||
COMPARE_SYSTEM = """Compare two crypto tokens for safety. Given their scanner results, determine which is safer and why.
|
||||
Reply format:
|
||||
SAFER: <token_name>
|
||||
REASON: <2-3 sentence comparison>
|
||||
SCORE_DIFF: <token1_score> vs <token2_score>
|
||||
KEY_DIFFERENCES: <bullet points>"""
|
||||
|
||||
|
||||
def compare_tokens(token_a: dict, token_b: dict) -> str:
|
||||
prompt = f"Token A:\n{json.dumps(token_a)[:800]}\n\nToken B:\n{json.dumps(token_b)[:800]}"
|
||||
return _call_ai(COMPARE_SYSTEM, prompt, max_tokens=200)
|
||||
|
|
@ -1,155 +0,0 @@
|
|||
"""
|
||||
RMI AI Pipeline v2 - Production Grade
|
||||
======================================
|
||||
Caching, fallbacks, rate limiting, smart prompts.
|
||||
All 12 modules battle-tested against Ollama Cloud.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
logger = logging.getLogger("rmi.ai")
|
||||
OLLAMA_KEY = os.getenv("OLLAMA_API_KEY", "")
|
||||
OLLAMA_URL = "https://ollama.com/v1/chat/completions"
|
||||
MODEL = "deepseek-v4-flash"
|
||||
CACHE_TTL = 300 # 5 min cache for identical calls
|
||||
|
||||
# Simple TTL cache
|
||||
_cache = {}
|
||||
|
||||
|
||||
def _cached_call(system: str, prompt: str, max_tokens: int = 250, temp: float = 0.3) -> str:
|
||||
key = hashlib.md5(f"{system[:50]}|{prompt[:100]}".encode()).hexdigest()
|
||||
now = time.time()
|
||||
if key in _cache and now - _cache[key][0] < CACHE_TTL:
|
||||
return _cache[key][1]
|
||||
try:
|
||||
body = json.dumps(
|
||||
{
|
||||
"model": MODEL,
|
||||
"messages": [
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temp,
|
||||
}
|
||||
).encode()
|
||||
req = Request(
|
||||
OLLAMA_URL,
|
||||
data=body,
|
||||
headers={"Authorization": f"Bearer {OLLAMA_KEY}", "Content-Type": "application/json"},
|
||||
)
|
||||
resp = urlopen(req, timeout=12)
|
||||
result = json.loads(resp.read())["choices"][0]["message"]["content"].strip()
|
||||
_cache[key] = (now, result)
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.error(f"Ollama AI call failed: {e}")
|
||||
return ""
|
||||
|
||||
|
||||
# ── 1. TOKEN RISK EXPLAINER (improved) ──
|
||||
def explain_risks(scan: dict) -> str:
|
||||
if not scan or scan.get("safety_score") is None:
|
||||
return "<b>Unable to analyze</b> - no scanner data."
|
||||
score = scan.get("safety_score", 50)
|
||||
flags = scan.get("risk_flags", [])
|
||||
green = scan.get("green_flags", [])
|
||||
name = scan.get("name", scan.get("symbol", "token"))
|
||||
mods = len(scan.get("modules_run", []))
|
||||
prompt = f"Token:{name} Score:{score}/100 Risks:{', '.join(flags[:5]) or 'none'} Green:{', '.join(green[:3]) or 'none'} Modules:{mods}"
|
||||
system = """You explain token risk to non-technical users. 3-4 sentences. Start with safety score. Mention top risks in plain English. End with "Always DYOR." Use <b>bold</b> for key terms. Never give financial advice."""
|
||||
result = _cached_call(system, prompt, max_tokens=150, temp=0.2)
|
||||
return result or f"<b>Safety: {score}/100</b>. Risk flags: {', '.join(flags[:3])}. Always DYOR."
|
||||
|
||||
|
||||
# ── 2. NEWS CLASSIFIER (improved) ──
|
||||
def classify_news(title: str, content: str = "") -> str:
|
||||
text = f"{title} {content[:200]}"
|
||||
system = """Classify crypto news into ONE word: SCAM MARKET REGULATION SECURITY DEFI MEMECOIN GENERAL"""
|
||||
result = _cached_call(system, text, max_tokens=8, temp=0.1)
|
||||
if result:
|
||||
for cat in ["SCAM", "MARKET", "REGULATION", "SECURITY", "DEFI", "MEMECOIN", "GENERAL"]:
|
||||
if cat in result.upper():
|
||||
return cat
|
||||
# Fast fallback
|
||||
t = text.lower()
|
||||
if any(w in t for w in ["hack", "exploit", "rug", "scam", "phish", "drain"]):
|
||||
return "SCAM"
|
||||
if any(w in t for w in ["price", "btc", "eth", "bull", "bear", "market"]):
|
||||
return "MARKET"
|
||||
return "GENERAL"
|
||||
|
||||
|
||||
# ── 3. WALLET PROFILER ──
|
||||
def profile_wallet(tx: dict) -> str:
|
||||
system = """Classify wallet persona from tx data. Reply: PERSONA|confidence. Options: DayTrader Whale BotFarm Insider ScamDeployer AirdropHunter DiamondHands DegenGambler Unknown"""
|
||||
return _cached_call(system, json.dumps(tx)[:1000], max_tokens=25) or "Unknown|0"
|
||||
|
||||
|
||||
# ── 4. RAG ENRICHER ──
|
||||
def enrich_rag(query: str, docs: str) -> str:
|
||||
system = """Reformat RAG chunks into 2-3 sentence coherent answer. Preserve key facts."""
|
||||
return _cached_call(system, f"Q:{query}\nD:{docs[:2000]}", max_tokens=200) or docs[:400]
|
||||
|
||||
|
||||
# ── 5. ALERT RANKER ──
|
||||
def rank_alerts(alerts: list) -> list:
|
||||
summary = "\n".join(
|
||||
f"{a.get('id', '?')}|{a.get('severity', '?')}|{(a.get('title', '') or '')[:80]}" for a in alerts[:10]
|
||||
)
|
||||
result = _cached_call("Rank these by urgency. Reply: id1,id2,id3...", summary, max_tokens=50)
|
||||
return [x.strip() for x in (result or "").split(",") if x.strip()]
|
||||
|
||||
|
||||
# ── 6. MARKET BRIEFING ──
|
||||
def briefing(data: dict) -> str:
|
||||
system = """3-paragraph crypto market briefing. P1:volume+chains P2:top risks P3:what to watch. <b>bold</b> key findings. Under 250 words."""
|
||||
return _cached_call(system, json.dumps(data)[:2000], max_tokens=350, temp=0.5) or "Briefing unavailable."
|
||||
|
||||
|
||||
# ── 7. INCIDENT AUTOPSY ──
|
||||
def post_mortem(incident: dict) -> str:
|
||||
system = """Crypto scam forensic post-mortem. What happened→How→Red flags→Protection. <b>bold</b> findings. Under 200 words."""
|
||||
return _cached_call(system, json.dumps(incident)[:1500], max_tokens=300, temp=0.4) or "Autopsy unavailable."
|
||||
|
||||
|
||||
# ── 8. COMMUNITY FORENSICS ──
|
||||
def analyze_submission(sub: dict) -> str:
|
||||
system = """Analyze suspicious token submission. Verdict:LIKELY SCAM/SUSPICIOUS/MORE INFO + 2-3 concerns."""
|
||||
return _cached_call(system, json.dumps(sub)[:1500], max_tokens=200) or "Analysis unavailable."
|
||||
|
||||
|
||||
# ── 9. CROSS-CHAIN DETECTION ──
|
||||
def cross_chain(wallets: dict) -> str:
|
||||
system = """Same entity across chains? Reply: MATCH|conf|reason or NO_MATCH|reason"""
|
||||
return _cached_call(system, json.dumps(wallets)[:1500], max_tokens=80) or "Unknown"
|
||||
|
||||
|
||||
# ── 10. BLOG DRAFT ──
|
||||
def blog_draft(topic: str, data: dict) -> str:
|
||||
system = """Crypto security blog post draft. Title|Hook|Body(3-4para)|KeyTakeaways|CTA. Markdown. Professional."""
|
||||
return (
|
||||
_cached_call(system, f"Topic:{topic}\nData:{json.dumps(data)[:2000]}", max_tokens=500, temp=0.6)
|
||||
or f"# {topic}\n\nDraft unavailable."
|
||||
)
|
||||
|
||||
|
||||
# ── 11. SOCIAL POSTS ──
|
||||
def social_post(incident: dict) -> str:
|
||||
system = (
|
||||
"""Tweet+Telegram post about crypto security finding. Twitter:<280 chars> | Telegram:<500 chars>. Hook first."""
|
||||
)
|
||||
return _cached_call(system, json.dumps(incident)[:1000], max_tokens=200, temp=0.7) or "Post unavailable."
|
||||
|
||||
|
||||
# ── 12. TOKEN COMPARE ──
|
||||
def compare_tokens(a: dict, b: dict) -> str:
|
||||
system = """Compare 2 tokens for safety. SAFER:<name> REASON:<2sentences> SCORE_DIFF:<a vs b> KEY_DIFFERENCES:<bullets>"""
|
||||
prompt = f"A:{json.dumps(a)[:800]}\nB:{json.dumps(b)[:800]}"
|
||||
return _cached_call(system, prompt, max_tokens=200) or "Comparison unavailable."
|
||||
|
|
@ -1,187 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
RMI AI Risk Explainer - Ollama Cloud Powered
|
||||
=============================================
|
||||
Takes raw scanner output → generates consumer-friendly risk explanations.
|
||||
Used by Telegram bot, website, and scanner API.
|
||||
|
||||
Cost: ~100 tokens per explanation = ~$0.0007 on Ollama Cloud
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from urllib.request import Request, urlopen
|
||||
|
||||
logger = logging.getLogger("rmi.risk_explainer")
|
||||
|
||||
OLLAMA_KEY = os.getenv("OLLAMA_API_KEY", os.getenv("DEEPSEEK_API_KEY", ""))
|
||||
OLLAMA_URL = "https://ollama.com/v1/chat/completions"
|
||||
BACKEND_URL = os.getenv("BACKEND_URL", "http://localhost:8000")
|
||||
MODEL = "deepseek-v4-flash"
|
||||
|
||||
SYSTEM_PROMPT = """You are RMI Risk Analyst. Given raw token scanner data, write a consumer-friendly risk explanation in 3-4 sentences.
|
||||
|
||||
Rules:
|
||||
- Start with the safety score and risk level (SAFE/LOW/MEDIUM/HIGH/CRITICAL)
|
||||
- Mention the 1-2 most important risk flags with plain-English explanations
|
||||
- If there are green flags, mention the most reassuring one
|
||||
- Be direct and honest - call out scams clearly
|
||||
- Use Telegram HTML formatting: <b>bold</b> for key terms
|
||||
- Never give financial advice. End with "Always DYOR."
|
||||
|
||||
Example output:
|
||||
"<b>Safety: 23/100 - HIGH RISK</b>. This token has <b>unlocked liquidity</b>, meaning the deployer can drain funds anytime. The <b>deployer wallet has 6 prior rugs</b>. No redeeming factors found. Avoid this token. Always DYOR."
|
||||
"""
|
||||
|
||||
|
||||
def explain_risks(scan: dict) -> str:
|
||||
"""Generate a human-readable risk explanation from scanner data."""
|
||||
if not scan or scan.get("safety_score") is None:
|
||||
return "<b>Unable to analyze</b> - no scanner data available."
|
||||
|
||||
score = scan.get("safety_score", 50)
|
||||
flags = scan.get("risk_flags", [])
|
||||
green = scan.get("green_flags", [])
|
||||
name = scan.get("name", scan.get("symbol", "This token"))
|
||||
modules = len(scan.get("modules_run", []))
|
||||
|
||||
# Build a concise prompt for the AI
|
||||
prompt = f"""Token safety scan results:
|
||||
- Token: {name}
|
||||
- Safety score: {score}/100
|
||||
- Risk flags: {", ".join(flags[:5]) if flags else "none"}
|
||||
- Green flags: {", ".join(green[:3]) if green else "none"}
|
||||
- Modules analyzed: {modules}
|
||||
|
||||
Write the explanation."""
|
||||
|
||||
try:
|
||||
body = json.dumps(
|
||||
{
|
||||
"model": MODEL,
|
||||
"messages": [
|
||||
{"role": "system", "content": SYSTEM_PROMPT},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
"max_tokens": 150,
|
||||
"temperature": 0.3,
|
||||
}
|
||||
).encode()
|
||||
|
||||
req = Request(
|
||||
OLLAMA_URL,
|
||||
data=body,
|
||||
headers={
|
||||
"Authorization": f"Bearer {OLLAMA_KEY}",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
)
|
||||
resp = urlopen(req, timeout=15)
|
||||
data = json.loads(resp.read())
|
||||
return data["choices"][0]["message"]["content"].strip()
|
||||
except Exception as e:
|
||||
logger.error(f"Risk explainer failed: {e}")
|
||||
# Fallback: basic explanation without AI
|
||||
return _basic_explain(scan)
|
||||
|
||||
|
||||
def _basic_explain(scan: dict) -> str:
|
||||
"""Basic explanation when AI is unavailable."""
|
||||
score = scan.get("safety_score", 50)
|
||||
if score >= 80:
|
||||
level = "SAFE"
|
||||
elif score >= 60:
|
||||
level = "LOW RISK"
|
||||
elif score >= 40:
|
||||
level = "MEDIUM RISK"
|
||||
elif score >= 20:
|
||||
level = "HIGH RISK"
|
||||
else:
|
||||
level = "CRITICAL"
|
||||
|
||||
flags = scan.get("risk_flags", [])
|
||||
green = scan.get("green_flags", [])
|
||||
scan.get("name", scan.get("symbol", "This token"))
|
||||
|
||||
msg = [f"<b>Safety: {score}/100 - {level}</b>"]
|
||||
if flags:
|
||||
msg.append(f"Risk flags: {', '.join(flags[:3])}")
|
||||
if green:
|
||||
msg.append(f"Green flags: {', '.join(green[:2])}")
|
||||
msg.append("Always DYOR.")
|
||||
return ". ".join(msg)
|
||||
|
||||
|
||||
# ── News Classification ──
|
||||
|
||||
NEWS_SYSTEM = """Classify crypto news headlines into categories. Reply with ONLY the category name.
|
||||
|
||||
Categories:
|
||||
- SCAM: rug pulls, hacks, exploits, phishing, fraud
|
||||
- MARKET: price action, trading, volume, market cap, BTC/ETH moves
|
||||
- REGULATION: government, SEC, legal, compliance, bans
|
||||
- SECURITY: vulnerability, audit, patch, wallet security
|
||||
- DEFI: DeFi protocols, yield, liquidity, lending
|
||||
- MEMECOIN: meme tokens, celebrity coins, pump events
|
||||
- GENERAL: anything else"""
|
||||
|
||||
|
||||
def classify_news(title: str, content: str = "") -> str:
|
||||
"""Classify a news article into a category."""
|
||||
text = f"{title}\n{content[:200]}" if content else title
|
||||
|
||||
try:
|
||||
body = json.dumps(
|
||||
{
|
||||
"model": MODEL,
|
||||
"messages": [
|
||||
{"role": "system", "content": NEWS_SYSTEM},
|
||||
{"role": "user", "content": text},
|
||||
],
|
||||
"max_tokens": 10,
|
||||
"temperature": 0.1,
|
||||
}
|
||||
).encode()
|
||||
|
||||
req = Request(
|
||||
OLLAMA_URL,
|
||||
data=body,
|
||||
headers={
|
||||
"Authorization": f"Bearer {OLLAMA_KEY}",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
)
|
||||
resp = urlopen(req, timeout=10)
|
||||
data = json.loads(resp.read())
|
||||
category = data["choices"][0]["message"]["content"].strip().upper()
|
||||
# Normalize
|
||||
for cat in ["SCAM", "MARKET", "REGULATION", "SECURITY", "DEFI", "MEMECOIN", "GENERAL"]:
|
||||
if cat in category:
|
||||
return cat
|
||||
return "GENERAL"
|
||||
except Exception as e:
|
||||
logger.warning(f"News classification failed: {e}")
|
||||
# Basic keyword fallback
|
||||
t = (title + " " + content).lower()
|
||||
if any(w in t for w in ["hack", "exploit", "rug", "scam", "phish"]):
|
||||
return "SCAM"
|
||||
if any(w in t for w in ["price", "btc", "eth", "bull", "bear", "market"]):
|
||||
return "MARKET"
|
||||
if any(w in t for w in ["sec ", "regulation", "ban", "law", "legal"]):
|
||||
return "REGULATION"
|
||||
return "GENERAL"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Test
|
||||
test = {
|
||||
"safety_score": 23,
|
||||
"risk_flags": ["LP_LOCK_LOW", "DEV_HIGH_RISK", "HONEYPOT_DETECTED"],
|
||||
"green_flags": [],
|
||||
"name": "SCAMCOIN",
|
||||
"modules_run": ["security", "holders", "liquidity"],
|
||||
}
|
||||
print(explain_risks(test))
|
||||
print()
|
||||
print(classify_news("$4M rug pull on Solana - deployer drained LP", ""))
|
||||
|
|
@ -1,138 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""#12 - Multi-Chain Airdrop Scanner. Scans address across all chains for unclaimed airdrops,
|
||||
governance tokens, dust that became valuable. Free tier shows value, paid tier auto-claims."""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
router = APIRouter(prefix="/api/v1/airdrop-scanner", tags=["airdrop-scanner"])
|
||||
|
||||
DATABUS = os.environ.get("DATABUS_URL", "http://localhost:8000/api/v1/databus")
|
||||
|
||||
CHAINS = [
|
||||
"solana",
|
||||
"ethereum",
|
||||
"bsc",
|
||||
"base",
|
||||
"arbitrum",
|
||||
"polygon",
|
||||
"avalanche",
|
||||
"optimism",
|
||||
"fantom",
|
||||
"linea",
|
||||
"zksync",
|
||||
"scroll",
|
||||
]
|
||||
|
||||
KNOWN_AIRDROPS = {
|
||||
"ethereum": [
|
||||
{"project": "Uniswap", "token": "UNI", "check_contract": "0x1f9840a85d5aF5bf1D1762F925BDADdC4201F984"},
|
||||
{"project": "ENS", "token": "ENS", "check_contract": "0xC18360217D8F7Ab5e7c516566761Ea12Ce7F9D72"},
|
||||
],
|
||||
"arbitrum": [
|
||||
{"project": "Arbitrum", "token": "ARB", "check_contract": "0x912CE59144191C1204E64559FE8253a0e49E6548"},
|
||||
],
|
||||
"solana": [
|
||||
{"project": "Jupiter", "token": "JUP", "check_contract": "JUPyiwrYJFskUPiHa7hkeR8VUtAeFoSYbKedZNsDvCN"},
|
||||
{"project": "Pyth", "token": "PYTH", "check_contract": "HZ1JovNiVvGrGNiiYvEozEVgZ58xaU3RKwX8eACQBCt3"},
|
||||
{"project": "Jito", "token": "JTO", "check_contract": "jtojtomepa8beP8AuQc6eXt5FriJwfFMwQx2v2f9mCL"},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
async def _check_chain_airdrops(address: str, chain: str) -> dict[str, Any]:
|
||||
"""Check for unclaimed airdrops on a single chain."""
|
||||
found: list[dict] = []
|
||||
total_value = 0.0
|
||||
|
||||
known = KNOWN_AIRDROPS.get(chain, [])
|
||||
for airdrop in known:
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=8) as client:
|
||||
resp = await client.get(
|
||||
f"{DATABUS}/{chain}/balance/{address}", params={"token": airdrop["check_contract"]}
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json().get("data", {})
|
||||
balance = float(data.get("balance", 0) or 0)
|
||||
value = float(data.get("value_usd", 0) or 0)
|
||||
if balance > 0:
|
||||
found.append(
|
||||
{
|
||||
"project": airdrop["project"],
|
||||
"token": airdrop["token"],
|
||||
"chain": chain,
|
||||
"balance": balance,
|
||||
"value_usd": round(value, 2),
|
||||
"claimable": False, # Would need per-project claim logic
|
||||
}
|
||||
)
|
||||
total_value += value
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
return {"chain": chain, "airdrops": found, "total_value": round(total_value, 2)}
|
||||
|
||||
|
||||
@router.get("/scan/{address}")
|
||||
async def scan_airdrops(address: str, chains: str = Query(None)):
|
||||
"""Scan an address across all chains for unclaimed airdrops."""
|
||||
chain_list = chains.split(",") if chains else CHAINS[:8]
|
||||
|
||||
tasks = []
|
||||
for c in chain_list:
|
||||
if c in KNOWN_AIRDROPS:
|
||||
tasks.append(_check_chain_airdrops(address, c))
|
||||
|
||||
results = await asyncio.gather(*tasks)
|
||||
all_airdrops = []
|
||||
grand_total = 0.0
|
||||
for r in results:
|
||||
all_airdrops.extend(r["airdrops"])
|
||||
grand_total += r["total_value"]
|
||||
|
||||
return {
|
||||
"address": address,
|
||||
"chains_scanned": len(results),
|
||||
"airdrops_found": len(all_airdrops),
|
||||
"total_value_usd": round(grand_total, 2),
|
||||
"airdrops": all_airdrops,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/known-projects")
|
||||
async def list_known_airdrops():
|
||||
"""List all known airdrop projects by chain."""
|
||||
return {"airdrops": KNOWN_AIRDROPS, "total_chains": len(KNOWN_AIRDROPS)}
|
||||
|
||||
|
||||
@router.get("/dust-check/{address}")
|
||||
async def check_dust(address: str, chain: str = Query("solana")):
|
||||
"""Check for dust tokens that might have become valuable."""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10) as client:
|
||||
resp = await client.get(f"{DATABUS}/{chain}/tokens/{address}")
|
||||
if resp.status_code != 200:
|
||||
return {"address": address, "chain": chain, "tokens": [], "error": "chain unavailable"}
|
||||
tokens = resp.json().get("data", {}).get("tokens", [])
|
||||
# Find tokens with small balances that have value
|
||||
dust_tokens = []
|
||||
for t in tokens:
|
||||
bal = float(t.get("balance", 0) or 0)
|
||||
val = float(t.get("value_usd", 0) or 0)
|
||||
if bal > 0 and val > 0:
|
||||
dust_tokens.append({"token": t.get("symbol", "?"), "balance": bal, "value_usd": round(val, 2)})
|
||||
dust_tokens.sort(key=lambda t: t["value_usd"], reverse=True)
|
||||
total_dust = sum(t["value_usd"] for t in dust_tokens)
|
||||
return {
|
||||
"address": address,
|
||||
"chain": chain,
|
||||
"dust_tokens": dust_tokens[:20],
|
||||
"total_dust_value": round(total_dust, 2),
|
||||
}
|
||||
except Exception as e:
|
||||
return {"address": address, "chain": chain, "error": str(e)}
|
||||
|
|
@ -1,272 +0,0 @@
|
|||
"""
|
||||
Alchemy API Router - NFT API, Enhanced API, Transaction API.
|
||||
Endpoints for NFT discovery, whale tracking, token metadata, and contract analysis.
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/v1/alchemy", tags=["alchemy"])
|
||||
|
||||
|
||||
# ── Models ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
class NftQuery(BaseModel):
|
||||
owner: str
|
||||
network: str = "eth"
|
||||
page_size: int = 50
|
||||
|
||||
|
||||
class NftMetadataQuery(BaseModel):
|
||||
contract: str
|
||||
token_id: str
|
||||
network: str = "eth"
|
||||
|
||||
|
||||
class CollectionOwnersQuery(BaseModel):
|
||||
contract: str
|
||||
network: str = "eth"
|
||||
page_size: int = 50
|
||||
|
||||
|
||||
class TokenBalanceQuery(BaseModel):
|
||||
address: str
|
||||
network: str = "eth"
|
||||
|
||||
|
||||
class AssetTransferQuery(BaseModel):
|
||||
from_address: str | None = None
|
||||
to_address: str | None = None
|
||||
network: str = "eth"
|
||||
category: list[str] = ["external", "internal", "erc20", "erc721"]
|
||||
max_count: int = 100
|
||||
|
||||
|
||||
class ContractCallQuery(BaseModel):
|
||||
contract: str
|
||||
data: str
|
||||
network: str = "eth"
|
||||
from_address: str | None = None
|
||||
|
||||
|
||||
# ── NFT API Endpoints ────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/nfts")
|
||||
async def get_nfts(req: NftQuery):
|
||||
"""Get all NFTs owned by an address."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
result = await ac.get_nfts(req.owner, req.network, req.page_size)
|
||||
return {"owner": req.owner, "network": req.network, **result}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Alchemy connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/nft/metadata")
|
||||
async def get_nft_metadata(contract: str, token_id: str, network: str = "eth"):
|
||||
"""Get metadata for a specific NFT."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
result = await ac.get_nft_metadata(contract, token_id, network)
|
||||
return {"contract": contract, "token_id": token_id, "metadata": result}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Alchemy connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/collection/owners")
|
||||
async def get_collection_owners(contract: str, network: str = "eth", page_size: int = 50):
|
||||
"""Get all owners of an NFT collection."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
result = await ac.get_owners_for_collection(contract, network, page_size)
|
||||
return {"contract": contract, "network": network, **result}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Alchemy connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/contract/metadata")
|
||||
async def get_contract_metadata(contract: str, network: str = "eth"):
|
||||
"""Get NFT contract metadata (name, symbol, totalSupply)."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
result = await ac.get_contract_metadata(contract, network)
|
||||
# Unwrap contractMetadata if present
|
||||
if isinstance(result, dict) and "contractMetadata" in result:
|
||||
result = result["contractMetadata"]
|
||||
return {"contract": contract, "network": network, "metadata": result}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Alchemy connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/nft-sales")
|
||||
async def get_nft_sales(contract: str | None = None, network: str = "eth", limit: int = 50):
|
||||
"""Get recent NFT sales."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
result = await ac.get_nft_sales(contract, network, limit)
|
||||
return {"contract": contract, "network": network, **result}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Alchemy connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
# ── Enhanced API Endpoints ───────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/token/balances")
|
||||
async def get_token_balances(req: TokenBalanceQuery):
|
||||
"""Get all ERC-20 token balances for an address."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
result = await ac.get_token_balances(req.address, req.network)
|
||||
return {"address": req.address, "network": req.network, **result}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Alchemy connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/token/metadata")
|
||||
async def get_token_metadata(contract: str, network: str = "eth"):
|
||||
"""Get ERC-20 token metadata."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
result = await ac.get_token_metadata(contract, network)
|
||||
return {"contract": contract, "network": network, "metadata": result}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Alchemy connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.post("/transfers")
|
||||
async def get_asset_transfers(req: AssetTransferQuery):
|
||||
"""Get asset transfers (tokens, NFTs, internal)."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
result = await ac.get_asset_transfers(
|
||||
from_address=req.from_address,
|
||||
to_address=req.to_address,
|
||||
network=req.network,
|
||||
category=req.category,
|
||||
max_count=req.max_count,
|
||||
)
|
||||
return {"network": req.network, **result}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Alchemy connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
# ── Transaction API Endpoints ────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/tx/{tx_hash}/receipt")
|
||||
async def get_transaction_receipt(tx_hash: str, network: str = "eth"):
|
||||
"""Get transaction receipt with enhanced data."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
result = await ac.get_transaction_receipt(tx_hash, network)
|
||||
return {"tx_hash": tx_hash, "network": network, "receipt": result}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Alchemy connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/block/{block_number}")
|
||||
async def get_block(block_number: int, network: str = "eth", include_txs: bool = False):
|
||||
"""Get block data."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
result = await ac.get_block_by_number(block_number, network, include_txs)
|
||||
return {"block_number": block_number, "network": network, "block": result}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Alchemy connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/balance/{address}")
|
||||
async def get_balance(address: str, network: str = "eth", block: str = "latest"):
|
||||
"""Get native token balance."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
result = await ac.get_balance(address, network, block)
|
||||
# Convert hex wei to ETH
|
||||
try:
|
||||
eth = int(result, 16) / 1e18 if result.startswith("0x") else float(result)
|
||||
except Exception:
|
||||
eth = 0
|
||||
return {"address": address, "network": network, "balance_wei": result, "balance_eth": eth}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Alchemy connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.post("/contract/call")
|
||||
async def contract_call(req: ContractCallQuery):
|
||||
"""Call a contract read function."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
result = await ac.call_contract(req.contract, req.data, req.network, req.from_address)
|
||||
return {"contract": req.contract, "network": req.network, "result": result}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Alchemy connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
# ── Health ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def alchemy_health():
|
||||
"""Alchemy connector status."""
|
||||
try:
|
||||
from app.alchemy_connector import get_alchemy_connector
|
||||
|
||||
ac = get_alchemy_connector()
|
||||
return {"status": "ok", "service": "alchemy-connector", **ac.status()}
|
||||
except ImportError:
|
||||
return {"status": "ok", "service": "alchemy-connector", "api_key_set": False}
|
||||
|
|
@ -1,356 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Auto-Alerting Pipeline for RMI Platform
|
||||
Send real-time alerts from detections to Telegram + webhooks
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
from dataclasses import asdict, dataclass
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import redis
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class AlertLevel(Enum):
|
||||
CRITICAL = "CRITICAL"
|
||||
WARNING = "WARNING"
|
||||
INFO = "INFO"
|
||||
DEBUG = "DEBUG"
|
||||
|
||||
|
||||
@dataclass
|
||||
class Alert:
|
||||
source: str
|
||||
alert_type: str
|
||||
level: str # "CRITICAL", "WARNING", "INFO", "DEBUG"
|
||||
wallet_address: str
|
||||
token_symbol: str
|
||||
message: str
|
||||
timestamp: str
|
||||
confidence: float = 0.0
|
||||
metadata: dict[str, Any] = None
|
||||
|
||||
def to_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
def to_telegram_message(self) -> str:
|
||||
"""Format alert as Telegram message"""
|
||||
level_emoji = {"CRITICAL": "🚨", "WARNING": "⚠️", "INFO": "ℹ️", "DEBUG": "🔍"} # noqa: RUF001
|
||||
return (
|
||||
f"{level_emoji.get(self.level, 'ℹ️')} *{self.alert_type.upper()}* [{self.level}]\n\n" # noqa: RUF001
|
||||
f" Wallet: `{self.wallet_address[:10]}...{self.wallet_address[-6:]}`\n"
|
||||
f" Token: `{self.token_symbol}`\n"
|
||||
f" Confidence: {self.confidence:.0%}\n\n"
|
||||
f"{self.message}\n\n"
|
||||
f"📅 {self.timestamp}"
|
||||
)
|
||||
|
||||
|
||||
# Configuration
|
||||
REDIS_HOST = os.getenv("REDIS_HOST", "172.20.0.3")
|
||||
REDIS_PORT = int(os.getenv("REDIS_PORT", "6379"))
|
||||
REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", "")
|
||||
|
||||
TELEGRAM_BOT_TOKEN = os.getenv("TELEGRAM_BOT_TOKEN", "")
|
||||
TELEGRAM_ALERTS_CHANNEL = os.getenv("TELEGRAM_ALERTS_CHANNEL", "") # -100xxx format
|
||||
|
||||
WEBHOOK_URLS = [
|
||||
os.getenv("ALERT_WEBHOOK_URL", ""),
|
||||
os.getenv("SLACK_ALERT_WEBHOOK", ""),
|
||||
]
|
||||
|
||||
# Rate limiting (max alerts per minute)
|
||||
RATE_LIMIT_MAX = 5
|
||||
RATE_LIMIT_WINDOW = 60 # seconds
|
||||
|
||||
# API Router
|
||||
router = APIRouter(prefix="/api/v1/alerts", tags=["alerting"])
|
||||
|
||||
|
||||
# Rate limit tracking via Redis
|
||||
def get_rate_limit_key() -> str:
|
||||
"""Get Redis key for rate limiting"""
|
||||
return "alerting:rate_limit:counts"
|
||||
|
||||
|
||||
def check_and_update_rate_limit() -> bool:
|
||||
"""Check rate limit and increment counter. Returns True if under limit."""
|
||||
try:
|
||||
r = redis.Redis(host=REDIS_HOST, port=REDIS_PORT, password=REDIS_PASSWORD or None, decode_responses=True)
|
||||
key = get_rate_limit_key()
|
||||
current_time = int(datetime.now().timestamp())
|
||||
window_key = f"{key}:{current_time // RATE_LIMIT_WINDOW}"
|
||||
|
||||
count = r.incr(window_key)
|
||||
r.expire(window_key, RATE_LIMIT_WINDOW * 2)
|
||||
|
||||
return not count > RATE_LIMIT_MAX
|
||||
except Exception as e:
|
||||
logger.warning(f"Rate limit check failed: {e}")
|
||||
return True # Allow on error
|
||||
|
||||
|
||||
# Send to Telegram
|
||||
async def send_to_telegram(alert: Alert) -> bool:
|
||||
"""Send alert to Telegram channel"""
|
||||
if not TELEGRAM_BOT_TOKEN or not TELEGRAM_ALERTS_CHANNEL:
|
||||
logger.info("Telegram config missing, skipping")
|
||||
return False
|
||||
|
||||
if not check_and_update_rate_limit():
|
||||
logger.info("Rate limit exceeded, alert queued")
|
||||
return False
|
||||
|
||||
try:
|
||||
import telegram
|
||||
|
||||
bot = telegram.Bot(token=TELEGRAM_BOT_TOKEN)
|
||||
message = alert.to_telegram_message()
|
||||
await bot.send_message(chat_id=TELEGRAM_ALERTS_CHANNEL, text=message, parse_mode="MarkdownV2")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.warning(f"Telegram send failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# Send to webhooks
|
||||
async def send_to_webhooks(alert: Alert) -> list[dict]:
|
||||
"""Send alert to configured webhooks"""
|
||||
results = []
|
||||
|
||||
for url in WEBHOOK_URLS:
|
||||
if not url:
|
||||
continue
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||
response = await client.post(
|
||||
url,
|
||||
json={"alert": alert.to_dict(), "timestamp": datetime.now().isoformat()},
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
results.append(
|
||||
{
|
||||
"url": url,
|
||||
"status": response.status_code,
|
||||
"success": response.status_code == 200,
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
results.append({"url": url, "status": "error", "error": str(e), "success": False})
|
||||
|
||||
return results
|
||||
|
||||
|
||||
# Alert storage in Redis
|
||||
def cache_alert(alert: Alert, ttl: int = 3600) -> bool:
|
||||
"""Cache alert in Redis for downstream processing"""
|
||||
try:
|
||||
r = redis.Redis(host=REDIS_HOST, port=REDIS_PORT, password=REDIS_PASSWORD or None, decode_responses=True)
|
||||
key = f"alerts:{alert.alert_type}:{alert.wallet_address}"
|
||||
alert_json = json.dumps(alert.to_dict())
|
||||
r.hset(
|
||||
key,
|
||||
mapping={
|
||||
"alert_json": alert_json,
|
||||
"created_at": datetime.now().isoformat(),
|
||||
"level": alert.level,
|
||||
},
|
||||
)
|
||||
r.expire(key, ttl)
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.warning(f"Cache alert failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# Process and dispatch alert
|
||||
async def process_alert(alert: Alert) -> dict[str, Any]:
|
||||
"""Process alert: send to Telegram, webhooks, cache"""
|
||||
results = {
|
||||
"telegram": False,
|
||||
"webhooks": [],
|
||||
"cached": False,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
}
|
||||
|
||||
# Send to Telegram
|
||||
if alert.level == "CRITICAL":
|
||||
results["telegram"] = await send_to_telegram(alert)
|
||||
|
||||
# Send to webhooks (always for all levels)
|
||||
if alert.level in ["CRITICAL", "WARNING"]:
|
||||
results["webhooks"] = await send_to_webhooks(alert)
|
||||
|
||||
# Cache alert
|
||||
results["cached"] = cache_alert(alert)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
# API Endpoints
|
||||
class AlertCreateRequest(BaseModel):
|
||||
source: str
|
||||
alert_type: str
|
||||
level: str = "INFO"
|
||||
wallet_address: str
|
||||
token_symbol: str
|
||||
message: str
|
||||
confidence: float = 0.0
|
||||
metadata: dict[str, Any] = None
|
||||
|
||||
|
||||
@router.post("/create")
|
||||
async def create_alert(request: AlertCreateRequest):
|
||||
"""Create and dispatch a new alert"""
|
||||
alert = Alert(
|
||||
source=request.source,
|
||||
alert_type=request.alert_type,
|
||||
level=request.level,
|
||||
wallet_address=request.wallet_address,
|
||||
token_symbol=request.token_symbol,
|
||||
message=request.message,
|
||||
timestamp=datetime.now().isoformat(),
|
||||
confidence=request.confidence,
|
||||
metadata=request.metadata or {},
|
||||
)
|
||||
|
||||
results = await process_alert(alert)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"alert_id": f"{alert.alert_type}:{alert.wallet_address}",
|
||||
"dispatch": results,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/webhook")
|
||||
async def webhook_alert(notification: dict[str, Any]):
|
||||
"""Receive external webhook and convert to internal alert"""
|
||||
source = notification.get("source", "external")
|
||||
alert_type = notification.get("alert_type", "unknown")
|
||||
level = notification.get("level", "INFO")
|
||||
|
||||
alert = Alert(
|
||||
source=source,
|
||||
alert_type=alert_type,
|
||||
level=level,
|
||||
wallet_address=notification.get("wallet_address", "unknown"),
|
||||
token_symbol=notification.get("token_symbol", "unknown"),
|
||||
message=notification.get("message", ""),
|
||||
timestamp=datetime.now().isoformat(),
|
||||
confidence=notification.get("confidence", 0.0),
|
||||
metadata=notification.get("metadata", {}),
|
||||
)
|
||||
|
||||
results = await process_alert(alert)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"alert_id": f"{alert.alert_type}:{alert.wallet_address}",
|
||||
"dispatch": results,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/status")
|
||||
async def get_alert_status():
|
||||
"""Get current alerting system status"""
|
||||
try:
|
||||
r = redis.Redis(host=REDIS_HOST, port=REDIS_PORT, password=REDIS_PASSWORD or None, decode_responses=True)
|
||||
total_alerts = r.dbsize()
|
||||
|
||||
# Count alerts by type
|
||||
alert_counts = {}
|
||||
for key in r.scan_iter("alerts:*"):
|
||||
if ":" in key:
|
||||
alert_type = key.split(":")[1]
|
||||
alert_counts[alert_type] = alert_counts.get(alert_type, 0) + 1
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"redis_connected": True,
|
||||
"total_alerts_cached": total_alerts,
|
||||
"alerts_by_type": alert_counts,
|
||||
"rate_limit": {"max_per_minute": RATE_LIMIT_MAX, "window_seconds": RATE_LIMIT_WINDOW},
|
||||
"webhooks_configured": len([w for w in WEBHOOK_URLS if w]),
|
||||
"telegram_channel": TELEGRAM_ALERTS_CHANNEL if TELEGRAM_ALERTS_CHANNEL else "not_configured",
|
||||
}
|
||||
except Exception as e:
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
|
||||
@router.get("/recent")
|
||||
async def get_recent_alerts(limit: int = 10):
|
||||
"""Get recent alerts from cache"""
|
||||
try:
|
||||
r = redis.Redis(host=REDIS_HOST, port=REDIS_PORT, password=REDIS_PASSWORD or None, decode_responses=True)
|
||||
|
||||
alerts = []
|
||||
for key in r.scan_iter("alerts:*", count=limit):
|
||||
alert_data = r.hgetall(key)
|
||||
if alert_data and "alert_json" in alert_data:
|
||||
alert = json.loads(alert_data["alert_json"])
|
||||
alerts.append(alert)
|
||||
|
||||
# Sort by timestamp (newest first)
|
||||
alerts.sort(key=lambda x: x.get("timestamp", ""), reverse=True)
|
||||
|
||||
return {"success": True, "count": len(alerts), "alerts": alerts[:limit]}
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)) from e
|
||||
|
||||
|
||||
# Background task: monitor alert queue
|
||||
async def alert_pipeline_monitor():
|
||||
"""Monitor Redis for new alerts from agents"""
|
||||
logger.info(f"[{datetime.now().isoformat()}] Alert pipeline monitor started")
|
||||
while True:
|
||||
try:
|
||||
r = redis.Redis(
|
||||
host=REDIS_HOST,
|
||||
port=REDIS_PORT,
|
||||
password=REDIS_PASSWORD or None,
|
||||
decode_responses=True,
|
||||
)
|
||||
|
||||
# Check for alerts in queue
|
||||
alert_json = r.rpop("social_monitor_queue")
|
||||
if alert_json:
|
||||
alert = json.loads(alert_json)
|
||||
logger.info(f"Processing alert from queue: {alert.get('alert_type', 'unknown')}")
|
||||
# Process alert through pipeline
|
||||
result = await process_alert(Alert(**alert))
|
||||
logger.info(f"Dispatch result: {result}")
|
||||
alert_json = r.rpop("chain_walker_queue")
|
||||
if alert_json:
|
||||
alert = json.loads(alert_json)
|
||||
logger.info(f"Processing chain-walker alert: {alert.get('alert_type', 'unknown')}")
|
||||
result = await process_alert(Alert(**alert))
|
||||
logger.info(f"Dispatch result: {result}")
|
||||
# Check syndicate detection alerts
|
||||
alert_json = r.rpop("syndicate_alerts")
|
||||
if alert_json:
|
||||
alert = json.loads(alert_json)
|
||||
logger.info(f"Processing syndicate alert: {alert.get('wallet_address', 'unknown')}")
|
||||
result = await process_alert(Alert(**alert))
|
||||
logger.info(f"Dispatch result: {result}")
|
||||
await asyncio.sleep(5) # Poll every 5 seconds
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Alert pipeline monitor error: {e}")
|
||||
await asyncio.sleep(10)
|
||||
|
||||
|
||||
def get_alert_pipeline_router():
|
||||
"""Get the alerting router"""
|
||||
return router
|
||||
|
|
@ -1,55 +0,0 @@
|
|||
"""
|
||||
Alerts Router - Token alert subscriptions
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/v1", tags=["alerts"])
|
||||
|
||||
|
||||
class AlertRequest(BaseModel):
|
||||
token_address: str
|
||||
alert_types: list[str] = Field(default=["liquidity_remove", "mint", "blacklist"])
|
||||
webhook_url: str | None = None
|
||||
|
||||
|
||||
from app.auth import get_redis, require_auth # noqa: E402
|
||||
|
||||
|
||||
def _get_redis_sync():
|
||||
"""Get Redis instance synchronously (get_redis returns redis.Redis, not async)."""
|
||||
return get_redis()
|
||||
|
||||
|
||||
@router.post("/alerts/subscribe")
|
||||
async def subscribe_alert(req: AlertRequest, user: dict[str, Any] = Depends(require_auth)):
|
||||
alert_id = f"alert:{datetime.utcnow().timestamp():.0f}"
|
||||
alert_data = {
|
||||
"id": alert_id,
|
||||
"token_address": req.token_address,
|
||||
"types": req.alert_types,
|
||||
"webhook_url": req.webhook_url,
|
||||
"created_at": datetime.utcnow().isoformat(),
|
||||
"active": True,
|
||||
}
|
||||
|
||||
# Redis fallback (RMI doesn't have full DB client)
|
||||
r = _get_redis_sync()
|
||||
r.hset("rmi:alerts", alert_id, json.dumps(alert_data))
|
||||
return alert_data
|
||||
|
||||
|
||||
@router.get("/alerts")
|
||||
async def list_alerts():
|
||||
r = _get_redis_sync()
|
||||
alerts_raw = r.hgetall("rmi:alerts") or {}
|
||||
alerts = [json.loads(v) for v in alerts_raw.values()]
|
||||
return {"alerts": alerts, "total": len(alerts)}
|
||||
|
|
@ -1,218 +0,0 @@
|
|||
"""
|
||||
Alibaba Cloud Connector - Tongyi Wanxiang AI for Image Generation.
|
||||
Generate professional graphics for cards, scorecards, marketing assets.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Alibaba Cloud Config ─────────────────────────────────────
|
||||
|
||||
DASHSCOPE_API_KEY = os.getenv("DASHSCOPE_API_KEY", "")
|
||||
DASHSCOPE_BASE_URL = "https://dashscope.aliyuncs.com/api/v1"
|
||||
|
||||
# Tongyi Wanxiang endpoints
|
||||
IMAGE_GENERATION_ENDPOINT = f"{DASHSCOPE_BASE_URL}/services/aigc/text-generation/generation"
|
||||
|
||||
|
||||
class AlibabaConnector:
|
||||
"""Alibaba Cloud AI services connector."""
|
||||
|
||||
def __init__(self):
|
||||
self.api_key = DASHSCOPE_API_KEY
|
||||
self._session = None
|
||||
|
||||
def _get_session(self):
|
||||
if self._session is None:
|
||||
self._session = httpx.AsyncClient(
|
||||
timeout=60.0,
|
||||
headers={
|
||||
"Authorization": f"Bearer {self.api_key}",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
)
|
||||
return self._session
|
||||
|
||||
async def generate_image(
|
||||
self,
|
||||
prompt: str,
|
||||
size: str = "1024x1024",
|
||||
style: str = "professional",
|
||||
negative_prompt: str | None = None,
|
||||
) -> dict:
|
||||
"""
|
||||
Generate image using Tongyi Wanxiang.
|
||||
|
||||
Args:
|
||||
prompt: Text description of image to generate
|
||||
size: Image size (e.g., "1024x1024", "1200x675")
|
||||
style: Art style ("professional", "cartoon", "realistic", etc.)
|
||||
negative_prompt: What to avoid in the image
|
||||
|
||||
Returns:
|
||||
Dict with image_url, thumbnail_url, and metadata
|
||||
"""
|
||||
if not self.api_key:
|
||||
logger.error("DASHSCOPE_API_KEY not configured")
|
||||
return {"error": "Alibaba API key not configured"}
|
||||
|
||||
# Parse size
|
||||
width, height = size.split("x")
|
||||
|
||||
# Build request
|
||||
payload = {
|
||||
"model": "wanx-v1", # Tongyi Wanxiang model
|
||||
"input": {
|
||||
"prompt": prompt,
|
||||
"negative_prompt": negative_prompt or "blurry, low quality, distorted, ugly, text, watermark",
|
||||
"size": f"{width}*{height}",
|
||||
"style": style,
|
||||
},
|
||||
"parameters": {
|
||||
"n": 1, # Number of images
|
||||
"seed": 42, # For reproducibility
|
||||
},
|
||||
}
|
||||
|
||||
try:
|
||||
session = self._get_session()
|
||||
response = await session.post(IMAGE_GENERATION_ENDPOINT, json=payload)
|
||||
|
||||
if response.status_code == 200:
|
||||
result = response.json()
|
||||
|
||||
# Extract image URLs
|
||||
images = result.get("output", {}).get("results", [])
|
||||
if images and len(images) > 0:
|
||||
return {
|
||||
"status": "success",
|
||||
"image_url": images[0].get("url"),
|
||||
"thumbnail_url": images[0].get("thumbnail_url"),
|
||||
"id": images[0].get("task_id"),
|
||||
"prompt": prompt,
|
||||
"size": size,
|
||||
"style": style,
|
||||
}
|
||||
else:
|
||||
return {"error": "No images generated", "raw": result}
|
||||
else:
|
||||
logger.error(f"Alibaba API error: {response.status_code} - {response.text[:200]}")
|
||||
return {
|
||||
"error": f"API error: {response.status_code}",
|
||||
"details": response.text[:500],
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Alibaba image generation failed: {e}")
|
||||
return {"error": str(e)}
|
||||
|
||||
async def generate_marketing_image(self, campaign_type: str, content: dict) -> dict:
|
||||
"""Generate marketing image for campaigns."""
|
||||
|
||||
prompts = {
|
||||
"launch": """
|
||||
Professional crypto platform launch announcement,
|
||||
dark theme, neon accents, "RMI Intelligence Platform" text,
|
||||
futuristic trading interface background,
|
||||
high quality, 4K, professional marketing graphic
|
||||
""",
|
||||
"feature_showcase": f"""
|
||||
Professional feature showcase graphic,
|
||||
"{content.get("feature_name", "Feature")}" prominently displayed,
|
||||
trading platform UI elements, charts, graphs,
|
||||
dark mode, neon green accents,
|
||||
clean modern design, marketing quality
|
||||
""",
|
||||
"stats_announcement": f"""
|
||||
Professional stats announcement graphic,
|
||||
"{content.get("stat_value", "1000")}" large number display,
|
||||
"{content.get("stat_label", "Users")}" label,
|
||||
crypto trading platform aesthetic,
|
||||
dark background, neon accents,
|
||||
high quality marketing graphic
|
||||
""",
|
||||
"kol_ranking": """
|
||||
Professional KOL ranking graphic,
|
||||
leaderboard style, top 10 layout,
|
||||
crypto influencer theme,
|
||||
dark mode, purple and gold accents,
|
||||
trading platform quality,
|
||||
high resolution marketing graphic
|
||||
""",
|
||||
}
|
||||
|
||||
prompt = prompts.get(campaign_type, content.get("custom_prompt", ""))
|
||||
|
||||
return await self.generate_image(
|
||||
prompt=prompt,
|
||||
size="1200x628", # Facebook/Twitter link preview size
|
||||
style="professional",
|
||||
negative_prompt="blurry, low quality, distorted, ugly, amateur, cluttered",
|
||||
)
|
||||
|
||||
async def generate_social_post_image(self, post_type: str, data: dict) -> dict:
|
||||
"""Generate image for social media posts."""
|
||||
|
||||
if post_type == "win_alert":
|
||||
prompt = f"""
|
||||
Big win celebration graphic,
|
||||
crypto trading win alert,
|
||||
"+${data.get("pnl_usd", 0):,.0f}" large display,
|
||||
green neon style,
|
||||
dark background,
|
||||
professional trading platform aesthetic,
|
||||
high quality social media graphic
|
||||
"""
|
||||
elif post_type == "loss_alert":
|
||||
prompt = f"""
|
||||
Loss porn graphic,
|
||||
crypto trading loss alert,
|
||||
"-${data.get("pnl_usd", 0):,.0f}" large display,
|
||||
red neon style,
|
||||
dark background,
|
||||
professional trading platform aesthetic,
|
||||
high quality social media graphic
|
||||
"""
|
||||
elif post_type == "rug_alert":
|
||||
prompt = """
|
||||
Rugpull warning graphic,
|
||||
crypto scam alert,
|
||||
"RUG PULL" large warning text,
|
||||
orange and red warning colors,
|
||||
dark background,
|
||||
professional security alert aesthetic,
|
||||
high quality social media graphic
|
||||
"""
|
||||
else:
|
||||
prompt = data.get("custom_prompt", "Professional crypto graphic")
|
||||
|
||||
return await self.generate_image(
|
||||
prompt=prompt,
|
||||
size="1200x675", # Twitter optimized
|
||||
style="professional",
|
||||
negative_prompt="blurry, low quality, distorted, ugly, text overlay, watermark",
|
||||
)
|
||||
|
||||
def status(self) -> dict:
|
||||
"""Check connector status."""
|
||||
return {
|
||||
"api_key_configured": bool(self.api_key),
|
||||
"api_key_prefix": self.api_key[:20] + "..." if self.api_key else "NOT SET",
|
||||
"base_url": DASHSCOPE_BASE_URL,
|
||||
"models_available": ["wanx-v1"],
|
||||
}
|
||||
|
||||
|
||||
# Singleton
|
||||
_alibaba: AlibabaConnector | None = None
|
||||
|
||||
|
||||
def get_alibaba_connector() -> AlibabaConnector:
|
||||
global _alibaba
|
||||
if _alibaba is None:
|
||||
_alibaba = AlibabaConnector()
|
||||
return _alibaba
|
||||
|
|
@ -1,322 +0,0 @@
|
|||
"""
|
||||
Alibaba Cloud DashScope Connector - Qwen Models for Content Generation.
|
||||
Supports: qwen-max, qwen-plus, qwen-turbo, qwen-coder, qwen-vl-max
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Alibaba DashScope Config ─────────────────────────────────
|
||||
|
||||
DASHSCOPE_API_KEY = os.getenv("DASHSCOPE_API_KEY", "")
|
||||
DASHSCOPE_BASE_URL = "https://dashscope.aliyuncs.com/api/v1"
|
||||
|
||||
# Available Qwen models
|
||||
QWEN_MODELS = {
|
||||
"qwen-max": {
|
||||
"context": 32000,
|
||||
"best_for": "Long-form content, detailed copy, highest quality",
|
||||
"cost": "$$",
|
||||
},
|
||||
"qwen-plus": {
|
||||
"context": 32000,
|
||||
"best_for": "Balanced quality/speed, marketing copy",
|
||||
"cost": "$",
|
||||
},
|
||||
"qwen-turbo": {
|
||||
"context": 8000,
|
||||
"best_for": "Quick drafts, social posts, fastest",
|
||||
"cost": "¢",
|
||||
},
|
||||
"qwen-coder": {
|
||||
"context": 32000,
|
||||
"best_for": "Technical docs, API guides, code",
|
||||
"cost": "$$",
|
||||
},
|
||||
"qwen-vl-max": {
|
||||
"context": 8000,
|
||||
"best_for": "Image + text, vision tasks",
|
||||
"cost": "$$$",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class AlibabaDashScopeConnector:
|
||||
"""Alibaba DashScope AI services connector."""
|
||||
|
||||
def __init__(self):
|
||||
self.api_key = DASHSCOPE_API_KEY
|
||||
self._session = None
|
||||
|
||||
def _get_session(self):
|
||||
if self._session is None:
|
||||
self._session = httpx.AsyncClient(
|
||||
timeout=120.0,
|
||||
headers={
|
||||
"Authorization": f"Bearer {self.api_key}",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
)
|
||||
return self._session
|
||||
|
||||
async def generate_text(
|
||||
self,
|
||||
prompt: str,
|
||||
model: str = "qwen-plus",
|
||||
max_tokens: int = 1000,
|
||||
temperature: float = 0.7,
|
||||
system_prompt: str | None = None,
|
||||
) -> dict:
|
||||
"""
|
||||
Generate text using Qwen models.
|
||||
|
||||
Args:
|
||||
prompt: User prompt
|
||||
model: Model name (qwen-max, qwen-plus, qwen-turbo, qwen-coder)
|
||||
max_tokens: Max tokens in response
|
||||
temperature: Creativity (0.0-1.0)
|
||||
system_prompt: System instructions
|
||||
|
||||
Returns:
|
||||
Dict with generated text and metadata
|
||||
"""
|
||||
if not self.api_key:
|
||||
logger.error("DASHSCOPE_API_KEY not configured")
|
||||
return {"error": "Alibaba API key not configured"}
|
||||
|
||||
if model not in QWEN_MODELS:
|
||||
return {"error": f"Unknown model: {model}. Available: {list(QWEN_MODELS.keys())}"}
|
||||
|
||||
# Build request
|
||||
messages = []
|
||||
if system_prompt:
|
||||
messages.append({"role": "system", "content": system_prompt})
|
||||
messages.append({"role": "user", "content": prompt})
|
||||
|
||||
payload = {
|
||||
"model": model,
|
||||
"input": {"messages": messages},
|
||||
"parameters": {
|
||||
"max_tokens": max_tokens,
|
||||
"temperature": temperature,
|
||||
"result_format": "text",
|
||||
},
|
||||
}
|
||||
|
||||
try:
|
||||
session = self._get_session()
|
||||
response = await session.post(
|
||||
f"{DASHSCOPE_BASE_URL}/services/aigc/text-generation/generation", json=payload
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
result = response.json()
|
||||
output = result.get("output", {})
|
||||
return {
|
||||
"status": "success",
|
||||
"text": output.get("text", ""),
|
||||
"model": model,
|
||||
"usage": output.get("usage", {}),
|
||||
"prompt": prompt[:100] + "...",
|
||||
}
|
||||
else:
|
||||
logger.error(f"DashScope API error: {response.status_code} - {response.text[:200]}")
|
||||
return {
|
||||
"error": f"API error: {response.status_code}",
|
||||
"details": response.text[:500],
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"DashScope text generation failed: {e}")
|
||||
return {"error": str(e)}
|
||||
|
||||
async def generate_marketing_content(self, content_type: str, topic: str, details: dict | None = None) -> dict:
|
||||
"""Generate marketing content for specific use cases."""
|
||||
|
||||
# Content type templates
|
||||
templates = {
|
||||
"blog_post": {
|
||||
"system": "You are a professional crypto marketing copywriter. Write engaging, informative blog posts.",
|
||||
"prompt": f"""Write a {details.get("word_count", 600)}-word blog post about: {topic}
|
||||
|
||||
Key points to cover:
|
||||
{chr(10).join(f"- {point}" for point in details.get("key_points", []))}
|
||||
|
||||
Tone: Professional but accessible
|
||||
Include: Call to action at the end
|
||||
Platform: RMI Intelligence Platform blog
|
||||
""",
|
||||
},
|
||||
"twitter_thread": {
|
||||
"system": "You are a crypto Twitter expert. Write engaging threads that get shares.",
|
||||
"prompt": f"""Create a Twitter thread (8-12 tweets) about: {topic}
|
||||
|
||||
Key points:
|
||||
{chr(10).join(f"- {point}" for point in details.get("key_points", []))}
|
||||
|
||||
Format:
|
||||
- Tweet 1: Hook
|
||||
- Tweets 2-10: Content
|
||||
- Final tweet: CTA
|
||||
|
||||
Include emojis, hashtags, and @cryptorugmunch tag
|
||||
Max 280 chars per tweet
|
||||
""",
|
||||
},
|
||||
"telegram_post": {
|
||||
"system": "You write engaging Telegram posts for crypto communities.",
|
||||
"prompt": f"""Write a Telegram announcement about: {topic}
|
||||
|
||||
Key points:
|
||||
{chr(10).join(f"- {point}" for point in details.get("key_points", []))}
|
||||
|
||||
Format:
|
||||
- Start with emoji headline
|
||||
- Use **bold** for emphasis
|
||||
- Include links
|
||||
- Add relevant hashtags
|
||||
|
||||
Tone: Exciting but professional
|
||||
""",
|
||||
},
|
||||
"email_newsletter": {
|
||||
"system": "You write engaging email newsletters for crypto platforms.",
|
||||
"prompt": f"""Write an email newsletter about: {topic}
|
||||
|
||||
Key points:
|
||||
{chr(10).join(f"- {point}" for point in details.get("key_points", []))}
|
||||
|
||||
Structure:
|
||||
- Subject line (5 options)
|
||||
- Opening hook
|
||||
- Main content
|
||||
- CTA
|
||||
- Sign-off
|
||||
|
||||
Tone: Friendly, professional, valuable
|
||||
Length: {details.get("word_count", 400)} words
|
||||
""",
|
||||
},
|
||||
"press_release": {
|
||||
"system": "You write professional press releases for crypto companies.",
|
||||
"prompt": f"""Write a press release about: {topic}
|
||||
|
||||
Key points:
|
||||
{chr(10).join(f"- {point}" for point in details.get("key_points", []))}
|
||||
|
||||
Format:
|
||||
- FOR IMMEDIATE RELEASE
|
||||
- Headline
|
||||
- Dateline
|
||||
- Body paragraphs
|
||||
- About RMI
|
||||
- Media contact
|
||||
|
||||
Tone: Professional, newsworthy
|
||||
Length: {details.get("word_count", 500)} words
|
||||
""",
|
||||
},
|
||||
"feature_announcement": {
|
||||
"system": "You write exciting feature announcements for crypto products.",
|
||||
"prompt": f"""Write a feature announcement for: {topic}
|
||||
|
||||
Feature details:
|
||||
{chr(10).join(f"- {point}" for point in details.get("features", []))}
|
||||
|
||||
Benefits:
|
||||
{chr(10).join(f"- {point}" for point in details.get("benefits", []))}
|
||||
|
||||
Include:
|
||||
- Exciting headline
|
||||
- What it does
|
||||
- Why it matters
|
||||
- How to use it
|
||||
- CTA
|
||||
|
||||
Tone: Exciting, clear, benefit-focused
|
||||
""",
|
||||
},
|
||||
}
|
||||
|
||||
template = templates.get(content_type)
|
||||
if not template:
|
||||
return {"error": f"Unknown content type: {content_type}"}
|
||||
|
||||
# Generate using qwen-plus by default
|
||||
model = details.get("model", "qwen-plus")
|
||||
|
||||
return await self.generate_text(
|
||||
prompt=template["prompt"],
|
||||
system_prompt=template["system"],
|
||||
model=model,
|
||||
max_tokens=details.get("max_tokens", 1500),
|
||||
temperature=details.get("temperature", 0.7),
|
||||
)
|
||||
|
||||
async def generate_variations(self, base_content: str, num_variations: int = 5, platform: str = "twitter") -> dict:
|
||||
"""Generate multiple variations of content."""
|
||||
|
||||
prompt = f"""Generate {num_variations} variations of this content for {platform}:
|
||||
|
||||
Original:
|
||||
{base_content}
|
||||
|
||||
Requirements:
|
||||
- Each variation should be unique
|
||||
- Keep the core message
|
||||
- Vary the tone slightly (some more excited, some more professional)
|
||||
- All should be high quality
|
||||
- Include relevant emojis for {platform}
|
||||
|
||||
Output format:
|
||||
Variation 1: [content]
|
||||
Variation 2: [content]
|
||||
...
|
||||
"""
|
||||
|
||||
return await self.generate_text(prompt=prompt, model="qwen-plus", max_tokens=2000, temperature=0.8)
|
||||
|
||||
async def summarize_content(self, content: str, summary_type: str = "bullet_points") -> dict:
|
||||
"""Summarize long content into different formats."""
|
||||
|
||||
summary_prompts = {
|
||||
"bullet_points": "Summarize this into 5-7 key bullet points:",
|
||||
"twitter_thread": "Convert this into a 5-tweet Twitter thread:",
|
||||
"one_liner": "Summarize this in one compelling sentence:",
|
||||
"email_blurb": "Summarize this into a 100-word email blurb:",
|
||||
}
|
||||
|
||||
prompt = f"""{summary_prompts.get(summary_type, "Summarize:")}
|
||||
|
||||
{content[:3000]} # Limit input length
|
||||
"""
|
||||
|
||||
return await self.generate_text(prompt=prompt, model="qwen-turbo", max_tokens=500, temperature=0.5)
|
||||
|
||||
def list_models(self) -> list[dict]:
|
||||
"""List available Qwen models."""
|
||||
return [{"id": model_id, **info} for model_id, info in QWEN_MODELS.items()]
|
||||
|
||||
def status(self) -> dict:
|
||||
"""Check connector status."""
|
||||
return {
|
||||
"api_key_configured": bool(self.api_key),
|
||||
"api_key_prefix": self.api_key[:20] + "..." if self.api_key else "NOT SET",
|
||||
"base_url": DASHSCOPE_BASE_URL,
|
||||
"models_available": list(QWEN_MODELS.keys()),
|
||||
}
|
||||
|
||||
|
||||
# Singleton
|
||||
_alibaba_dashscope: AlibabaDashScopeConnector | None = None
|
||||
|
||||
|
||||
def get_alibaba_dashscope_connector() -> AlibabaDashScopeConnector:
|
||||
global _alibaba_dashscope
|
||||
if _alibaba_dashscope is None:
|
||||
_alibaba_dashscope = AlibabaDashScopeConnector()
|
||||
return _alibaba_dashscope
|
||||
|
|
@ -1,577 +0,0 @@
|
|||
"""
|
||||
All Connectors Router - Wires all 20+ unwired modules into API routes.
|
||||
One file to rule them all.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/v1", tags=["connectors"])
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# BIRDEYE - Token data, trending, OHLCV
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/birdeye/token/{address}")
|
||||
async def birdeye_token(address: str, chain: str = "solana"):
|
||||
try:
|
||||
from app.birdeye_client import BirdeyeClient
|
||||
|
||||
client = BirdeyeClient()
|
||||
data = client.get_token_info(address)
|
||||
return {"address": address, "chain": chain, "data": data, "source": "birdeye"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/birdeye/trending")
|
||||
async def birdeye_trending(chain: str = "solana", limit: int = 20):
|
||||
try:
|
||||
from app.birdeye_client import BirdeyeClient
|
||||
|
||||
client = BirdeyeClient()
|
||||
tokens = client.get_trending(limit=limit)
|
||||
return {"tokens": tokens, "chain": chain, "source": "birdeye"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# ARKHAM INTELLIGENCE - Entity labeling, wallet attribution,
|
||||
# institutional tracking, sanctions screening
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/arkham/entity/{address}")
|
||||
async def arkham_entity(address: str):
|
||||
try:
|
||||
from app.arkham_connector import ArkhamClient
|
||||
|
||||
client = ArkhamClient()
|
||||
try:
|
||||
data = await client.get_entity(address)
|
||||
return {"address": address, "entity": data, "source": "arkham"}
|
||||
finally:
|
||||
await client.close()
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/arkham/labels")
|
||||
async def arkham_labels(page: int = 0, limit: int = 100):
|
||||
try:
|
||||
from app.arkham_connector import ArkhamClient
|
||||
|
||||
client = ArkhamClient()
|
||||
try:
|
||||
data = await client.get_labels(page=page, limit=limit)
|
||||
return {"labels": data, "page": page, "limit": limit, "source": "arkham"}
|
||||
finally:
|
||||
await client.close()
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/arkham/portfolio/{address}")
|
||||
async def arkham_portfolio(address: str):
|
||||
try:
|
||||
from app.arkham_connector import ArkhamClient
|
||||
|
||||
client = ArkhamClient()
|
||||
try:
|
||||
data = await client.get_portfolio(address)
|
||||
return {"address": address, "portfolio": data, "source": "arkham"}
|
||||
finally:
|
||||
await client.close()
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# BLOCKCHAIR - Multi-chain block explorer
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/blockchair/balance/{address}")
|
||||
async def blockchair_balance(address: str, chain: str = "bitcoin"):
|
||||
try:
|
||||
from app.blockchair_connector import get_address_balance
|
||||
|
||||
result = get_address_balance(address, chain)
|
||||
return {"address": address, "chain": chain, "data": result, "source": "blockchair"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/blockchair/search")
|
||||
async def blockchair_search(q: str):
|
||||
try:
|
||||
from app.blockchair_connector import search_blockchain
|
||||
|
||||
result = search_blockchain(q)
|
||||
return {"query": q, "results": result, "source": "blockchair"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# DEFILLAMA - DeFi analytics
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/defillama/tvl")
|
||||
async def defillama_tvl():
|
||||
try:
|
||||
from app.defillama_connector import get_defi_tvl
|
||||
|
||||
result = get_defi_tvl()
|
||||
return {"tvl": result, "source": "defillama"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/defillama/protocols")
|
||||
async def defillama_protocols():
|
||||
try:
|
||||
from app.defillama_connector import get_defi_protocols
|
||||
|
||||
result = get_defi_protocols()
|
||||
return {"protocols": result, "source": "defillama"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/defillama/chains")
|
||||
async def defillama_chains():
|
||||
try:
|
||||
from app.defillama_connector import get_chain_tvls
|
||||
|
||||
result = get_chain_tvls()
|
||||
return {"chains": result, "source": "defillama"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# ENTITY CLUSTERING - Wallet cluster analysis
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/entity/clusters")
|
||||
async def entity_clusters(address: str | None = None, min_size: int = 2):
|
||||
try:
|
||||
from app.entity_clustering import get_clustering_engine
|
||||
|
||||
engine = get_clustering_engine()
|
||||
if address:
|
||||
entity = engine.graph.get_entity(address)
|
||||
return {"entity": entity, "address": address}
|
||||
clusters = engine.graph.find_clusters(min_size=min_size)
|
||||
return {"clusters": clusters, "total": len(clusters)}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.post("/entity/link")
|
||||
async def entity_link(data: dict):
|
||||
try:
|
||||
from app.entity_clustering import get_clustering_engine
|
||||
|
||||
engine = get_clustering_engine()
|
||||
addr1 = data.get("address1", "")
|
||||
addr2 = data.get("address2", "")
|
||||
relationship = data.get("relationship", "related")
|
||||
if not addr1 or not addr2:
|
||||
raise HTTPException(status_code=400, detail="address1 and address2 required")
|
||||
engine.graph.link_wallets(addr1, addr2, relationship)
|
||||
return {
|
||||
"status": "linked",
|
||||
"address1": addr1,
|
||||
"address2": addr2,
|
||||
"relationship": relationship,
|
||||
}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# THREAT INTEL - Sanctions, reputation, blocklists
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/threat/reputation/{address}")
|
||||
async def threat_reputation(address: str, chain: str = "ethereum"):
|
||||
try:
|
||||
from app.threat_intel import check_wallet_reputation
|
||||
|
||||
result = check_wallet_reputation(address, chain)
|
||||
return {"address": address, "chain": chain, "reputation": result}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/threat/sanctions/{address}")
|
||||
async def threat_sanctions(address: str):
|
||||
try:
|
||||
from app.threat_intel import check_sanctions
|
||||
|
||||
result = check_sanctions(address)
|
||||
return {"address": address, "sanctions": result, "sanctioned": len(result) > 0}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.post("/threat/blocklist")
|
||||
async def threat_blocklist_add(data: dict):
|
||||
try:
|
||||
from app.threat_intel import add_to_blocklist
|
||||
|
||||
address = data.get("address", "")
|
||||
reason = data.get("reason", "")
|
||||
if not address:
|
||||
raise HTTPException(status_code=400, detail="address required")
|
||||
success = add_to_blocklist(address, reason)
|
||||
return {"address": address, "added": success, "reason": reason}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# EXCHANGE FLOW - CEX inflows/outflows
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/exchange/flow/{address}")
|
||||
async def exchange_flow(address: str, chain: str = "ethereum"):
|
||||
try:
|
||||
from app.exchange_flow_analyzer import analyze_entity_flows
|
||||
|
||||
result = analyze_entity_flows(address, chain)
|
||||
return {"address": address, "chain": chain, "flows": result}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/exchange/whales")
|
||||
async def exchange_whales(chain: str = "ethereum"):
|
||||
try:
|
||||
from app.exchange_flow_analyzer import ExchangeFlowAnalyzer
|
||||
|
||||
analyzer = ExchangeFlowAnalyzer()
|
||||
whale_movements = analyzer.detect_large_transfers(chain=chain, min_value_usd=1000000)
|
||||
return {"whale_movements": whale_movements, "chain": chain}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# CROSS-CHAIN CORRELATOR
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/crosschain/fingerprint/{address}")
|
||||
async def crosschain_fingerprint(address: str, chains: str = "ethereum,base,bsc,polygon"):
|
||||
try:
|
||||
from app.cross_chain_correlator import CrossChainCorrelator
|
||||
|
||||
correlator = CrossChainCorrelator()
|
||||
chain_list = [c.strip() for c in chains.split(",")]
|
||||
results = {}
|
||||
for chain in chain_list:
|
||||
try:
|
||||
fp = correlator.get_fingerprint(address, chain)
|
||||
results[chain] = fp
|
||||
except Exception:
|
||||
results[chain] = {"error": f"Chain {chain} unavailable"}
|
||||
return {"address": address, "fingerprints": results, "chains_checked": len(results)}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# AGENT MESH - 8 AI agents
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
AGENTS = {
|
||||
"nexus": {
|
||||
"name": "NEXUS",
|
||||
"role": "Strategic Coordinator",
|
||||
"tier": "T0",
|
||||
"triggers": ["strategize", "plan", "coordinate"],
|
||||
},
|
||||
"scout": {
|
||||
"name": "SCOUT",
|
||||
"role": "Alpha Hunter",
|
||||
"tier": "T3",
|
||||
"triggers": ["find", "scan", "hunt", "alpha"],
|
||||
},
|
||||
"tracer": {
|
||||
"name": "TRACER",
|
||||
"role": "Forensic Investigator",
|
||||
"tier": "T1",
|
||||
"triggers": ["trace", "investigate", "wallet"],
|
||||
},
|
||||
"cipher": {
|
||||
"name": "CIPHER",
|
||||
"role": "Contract Auditor",
|
||||
"tier": "T1",
|
||||
"triggers": ["audit", "security", "contract"],
|
||||
},
|
||||
"sentinel": {
|
||||
"name": "SENTINEL",
|
||||
"role": "Rug Detector",
|
||||
"tier": "T2",
|
||||
"triggers": ["monitor", "watch", "alert", "rug"],
|
||||
},
|
||||
"chronicler": {
|
||||
"name": "CHRONICLER",
|
||||
"role": "Investigative Reporter",
|
||||
"tier": "T2",
|
||||
"triggers": ["write", "document", "report"],
|
||||
},
|
||||
"forge": {
|
||||
"name": "FORGE",
|
||||
"role": "Implementation Architect",
|
||||
"tier": "T1",
|
||||
"triggers": ["code", "implement", "build"],
|
||||
},
|
||||
"relay": {
|
||||
"name": "RELAY",
|
||||
"role": "Communications Coordinator",
|
||||
"tier": "T3",
|
||||
"triggers": ["format", "relay", "dispatch"],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/agents")
|
||||
async def list_agents():
|
||||
return {"agents": AGENTS, "total": len(AGENTS)}
|
||||
|
||||
|
||||
@router.get("/agents/{agent_id}")
|
||||
async def get_agent(agent_id: str):
|
||||
agent = AGENTS.get(agent_id)
|
||||
if not agent:
|
||||
raise HTTPException(status_code=404, detail=f"Agent {agent_id} not found")
|
||||
return agent
|
||||
|
||||
|
||||
@router.post("/agents/{agent_id}/command")
|
||||
async def agent_command(agent_id: str, data: dict):
|
||||
agent = AGENTS.get(agent_id)
|
||||
if not agent:
|
||||
raise HTTPException(status_code=404, detail=f"Agent {agent_id} not found")
|
||||
command = data.get("command", "")
|
||||
return {
|
||||
"agent": agent["name"],
|
||||
"role": agent["role"],
|
||||
"command": command,
|
||||
"status": "queued",
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# MCP SERVERS - Multi-chain data gateways
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/mcp/servers")
|
||||
async def mcp_servers_list():
|
||||
return {
|
||||
"servers": {
|
||||
"dexpaprika": "Real-time DEX data for 5M+ tokens across 20+ chains",
|
||||
"solana": "Solana RPC - wallet balances, token prices, DeFi yields",
|
||||
"dexscreener": "DEX pair data, token info, market stats",
|
||||
"defillama": "DeFi TVL, protocols, yields, fees",
|
||||
"coingecko": "13K+ tokens, global stats, historical data",
|
||||
"helius": "Enhanced Solana RPC - parsed txs, webhooks",
|
||||
"goplus": "Multi-chain token security - 700K+ tokens scanned",
|
||||
"rugcheck": "Solana token safety audit",
|
||||
},
|
||||
"status": "operational",
|
||||
}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# SENTIMENT - Crypto market sentiment analysis
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/sentiment/market")
|
||||
async def sentiment_market():
|
||||
try:
|
||||
from app.ml_anomaly import AnomalyDetector
|
||||
|
||||
detector = AnomalyDetector()
|
||||
anomalies = detector.detect_market_anomalies()
|
||||
return {"anomalies": anomalies, "timestamp": datetime.now(UTC).isoformat()}
|
||||
except Exception:
|
||||
# Fallback to fear & greed
|
||||
import httpx
|
||||
|
||||
async with httpx.AsyncClient(timeout=8) as c:
|
||||
r = await c.get("https://api.alternative.me/fng/?limit=1")
|
||||
if r.status_code == 200:
|
||||
data = r.json().get("data", [{}])[0]
|
||||
return {
|
||||
"sentiment": {
|
||||
"fear_greed_index": int(data.get("value", 50)),
|
||||
"classification": data.get("value_classification", "Neutral"),
|
||||
},
|
||||
"source": "alternative.me",
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
return {"error": "Sentiment data unavailable"}
|
||||
|
||||
|
||||
@router.get("/sentiment/token/{address}")
|
||||
async def sentiment_token(address: str):
|
||||
# On-chain sentiment from buy/sell ratio
|
||||
try:
|
||||
import httpx
|
||||
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(f"https://api.dexscreener.com/latest/dex/tokens/{address}")
|
||||
if r.status_code == 200:
|
||||
pairs = r.json().get("pairs", [])
|
||||
if pairs:
|
||||
p = pairs[0]
|
||||
buys = p.get("txns", {}).get("h24", {}).get("buys", 0)
|
||||
sells = p.get("txns", {}).get("h24", {}).get("sells", 0)
|
||||
total = buys + sells
|
||||
buy_ratio = buys / total if total > 0 else 0.5
|
||||
sentiment = "bullish" if buy_ratio > 0.6 else ("bearish" if buy_ratio < 0.4 else "neutral")
|
||||
return {
|
||||
"token": address,
|
||||
"sentiment": sentiment,
|
||||
"buy_ratio": round(buy_ratio, 3),
|
||||
"buys_24h": buys,
|
||||
"sells_24h": sells,
|
||||
"source": "dexscreener",
|
||||
}
|
||||
except Exception:
|
||||
pass
|
||||
return {"token": address, "sentiment": "unknown"}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# NANSEN - Wallet labels, smart money, token flow
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/nansen/labels/{address}")
|
||||
async def nansen_labels(address: str):
|
||||
try:
|
||||
from app.nansen_connector import get_wallet_labels
|
||||
|
||||
result = get_wallet_labels(address)
|
||||
return {"address": address, "labels": result, "source": "nansen"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/nansen/smart-money")
|
||||
async def nansen_smart_money():
|
||||
try:
|
||||
from app.nansen_connector import get_smart_money
|
||||
|
||||
result = get_smart_money()
|
||||
return {"smart_money": result, "source": "nansen"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/nansen/activity/{address}")
|
||||
async def nansen_activity(address: str):
|
||||
try:
|
||||
from app.nansen_connector import get_wallet_activity
|
||||
|
||||
result = get_wallet_activity(address)
|
||||
return {"address": address, "activity": result, "source": "nansen"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# MEMPOOL - Bitcoin mempool monitoring
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/mempool/status")
|
||||
async def mempool_status():
|
||||
try:
|
||||
import httpx
|
||||
|
||||
async with httpx.AsyncClient(timeout=8) as c:
|
||||
r = await c.get("https://mempool.space/api/v1/fees/recommended")
|
||||
if r.status_code == 200:
|
||||
fees = r.json()
|
||||
r2 = await c.get("https://mempool.space/api/mempool")
|
||||
mempool = r2.json() if r2.status_code == 200 else {}
|
||||
return {
|
||||
"fees": fees,
|
||||
"mempool_tx_count": mempool.get("count", 0),
|
||||
"mempool_size_mb": round(mempool.get("vsize", 0) / 1_000_000, 2),
|
||||
"source": "mempool.space",
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
except Exception:
|
||||
pass
|
||||
return {"error": "Mempool data unavailable"}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# COINGECKO - Price data, trending, global metrics
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/coingecko/ping")
|
||||
async def coingecko_ping():
|
||||
try:
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
result = await cg.ping()
|
||||
return {"ping": result, "source": "coingecko"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/coingecko/trending")
|
||||
async def coingecko_trending():
|
||||
try:
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
result = await cg.get_trending()
|
||||
return {"trending": result, "source": "coingecko"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/coingecko/markets")
|
||||
async def coingecko_markets(vs_currency: str = "usd", per_page: int = 50):
|
||||
try:
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
result = await cg.get_market_overview(vs_currency=vs_currency, per_page=per_page)
|
||||
return {"markets": result, "vs_currency": vs_currency, "source": "coingecko"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/coingecko/price/{coin_id}")
|
||||
async def coingecko_price(coin_id: str):
|
||||
try:
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
result = await cg.get_token_price(coin_id)
|
||||
return {"coin_id": coin_id, "price": result, "source": "coingecko"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/coingecko/global")
|
||||
async def coingecko_global():
|
||||
try:
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
result = await cg.get_global_metrics()
|
||||
return {"global": result, "source": "coingecko"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
@router.get("/coingecko/coin/{coin_id}")
|
||||
async def coingecko_coin(coin_id: str):
|
||||
try:
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
result = await cg.get_token_detail(coin_id)
|
||||
return {"coin_id": coin_id, "coin": result, "source": "coingecko"}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
|
@ -1,258 +0,0 @@
|
|||
"""
|
||||
RMI Analytics API Router
|
||||
==========================
|
||||
REST API for real-time analytics and dashboard data.
|
||||
|
||||
Endpoints:
|
||||
GET /api/v1/analytics/metrics - List all metrics
|
||||
GET /api/v1/analytics/metrics/{name} - Get metric data
|
||||
POST /api/v1/analytics/metrics/{name} - Record metric
|
||||
GET /api/v1/analytics/dashboards - List dashboards
|
||||
GET /api/v1/analytics/dashboards/{id} - Get dashboard data
|
||||
POST /api/v1/analytics/dashboards - Create dashboard
|
||||
GET /api/v1/analytics/trends/{metric} - Get trend analysis
|
||||
GET /api/v1/analytics/stats - System stats
|
||||
GET /api/v1/analytics/prometheus - Prometheus export
|
||||
GET /api/v1/analytics/export/{metric} - Export metric
|
||||
GET /api/v1/analytics/realtime/{dashboard} - WebSocket-compatible data
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.admin_backend import AuditLogger, require_admin
|
||||
from app.analytics_engine import AnalyticsEngine, DashboardWidget, get_analytics_engine
|
||||
|
||||
logger = logging.getLogger("analytics_api")
|
||||
|
||||
router = APIRouter(prefix="/api/v1/analytics", tags=["analytics"])
|
||||
|
||||
# ── Models ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class RecordMetricRequest(BaseModel):
|
||||
value: float
|
||||
labels: dict[str, str] | None = None
|
||||
|
||||
|
||||
class CreateDashboardRequest(BaseModel):
|
||||
name: str
|
||||
description: str = ""
|
||||
|
||||
|
||||
class AddWidgetRequest(BaseModel):
|
||||
widget_type: str = Field(..., description="line, bar, gauge, counter, table, pie")
|
||||
title: str
|
||||
metric_name: str
|
||||
width: int = 6
|
||||
height: int = 4
|
||||
refresh_interval: int = 30
|
||||
config: dict[str, Any] | None = None
|
||||
|
||||
|
||||
# ── Helper ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _get_engine() -> AnalyticsEngine:
|
||||
return get_analytics_engine()
|
||||
|
||||
|
||||
# ── Endpoints ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/metrics")
|
||||
async def list_metrics(request: Request):
|
||||
"""List all tracked metrics."""
|
||||
await require_admin(request, "analytics.read")
|
||||
|
||||
engine = _get_engine()
|
||||
metrics = []
|
||||
for name in engine.get_metric_names():
|
||||
metric = engine.get_metric(name)
|
||||
if metric:
|
||||
metrics.append(metric.to_dict())
|
||||
|
||||
return {"metrics": metrics, "total": len(metrics)}
|
||||
|
||||
|
||||
@router.get("/metrics/{name}")
|
||||
async def get_metric(request: Request, name: str, limit: int = 100):
|
||||
"""Get metric data points."""
|
||||
await require_admin(request, "analytics.read")
|
||||
|
||||
engine = _get_engine()
|
||||
metric = engine.get_metric(name)
|
||||
if not metric:
|
||||
raise HTTPException(status_code=404, detail="Metric not found")
|
||||
|
||||
points = [{"timestamp": p.timestamp, "value": p.value, "labels": p.labels} for p in metric.points[-limit:]]
|
||||
|
||||
return {
|
||||
"name": metric.name,
|
||||
"description": metric.description,
|
||||
"unit": metric.unit,
|
||||
"latest": metric.latest(),
|
||||
"avg_1m": metric.avg(60),
|
||||
"trend": metric.trend(),
|
||||
"points": points,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/metrics/{name}")
|
||||
async def record_metric(request: Request, name: str, body: RecordMetricRequest):
|
||||
"""Record a metric data point."""
|
||||
# Allow public recording for system metrics (from middleware)
|
||||
engine = _get_engine()
|
||||
engine.record_metric(name, body.value, body.labels or {})
|
||||
return {"success": True}
|
||||
|
||||
|
||||
@router.get("/dashboards")
|
||||
async def list_dashboards(request: Request):
|
||||
"""List all dashboards."""
|
||||
await require_admin(request, "analytics.read")
|
||||
|
||||
engine = _get_engine()
|
||||
dashboards = engine.list_dashboards()
|
||||
return {
|
||||
"dashboards": [
|
||||
{
|
||||
"dashboard_id": d.dashboard_id,
|
||||
"name": d.name,
|
||||
"description": d.description,
|
||||
"widget_count": len(d.widgets),
|
||||
"is_default": d.is_default,
|
||||
}
|
||||
for d in dashboards
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@router.get("/dashboards/{dashboard_id}")
|
||||
async def get_dashboard_data(request: Request, dashboard_id: str):
|
||||
"""Get current data for a dashboard."""
|
||||
await require_admin(request, "analytics.read")
|
||||
|
||||
engine = _get_engine()
|
||||
data = engine.get_dashboard_data(dashboard_id)
|
||||
if "error" in data:
|
||||
raise HTTPException(status_code=404, detail=data["error"])
|
||||
return data
|
||||
|
||||
|
||||
@router.post("/dashboards")
|
||||
async def create_dashboard(request: Request, body: CreateDashboardRequest):
|
||||
"""Create a new dashboard."""
|
||||
auth = await require_admin(request, "analytics.read")
|
||||
admin = auth["admin"]
|
||||
|
||||
engine = _get_engine()
|
||||
dashboard = engine.create_dashboard(body.name, body.description, admin["id"])
|
||||
|
||||
await AuditLogger.log(
|
||||
admin_id=admin["id"],
|
||||
admin_email=admin["email"],
|
||||
action="dashboard.create",
|
||||
resource_type="dashboard",
|
||||
resource_id=dashboard.dashboard_id,
|
||||
ip_address=request.client.host if request.client else "",
|
||||
user_agent=request.headers.get("user-agent", ""),
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"dashboard": {"dashboard_id": dashboard.dashboard_id, "name": dashboard.name},
|
||||
}
|
||||
|
||||
|
||||
@router.post("/dashboards/{dashboard_id}/widgets")
|
||||
async def add_widget(request: Request, dashboard_id: str, body: AddWidgetRequest):
|
||||
"""Add widget to dashboard."""
|
||||
auth = await require_admin(request, "analytics.read")
|
||||
auth["admin"]
|
||||
|
||||
engine = _get_engine()
|
||||
widget = DashboardWidget(
|
||||
widget_id=f"wid_{int(time.time())}_{os.urandom(4).hex()}",
|
||||
widget_type=body.widget_type,
|
||||
title=body.title,
|
||||
metric_name=body.metric_name,
|
||||
width=body.width,
|
||||
height=body.height,
|
||||
refresh_interval=body.refresh_interval,
|
||||
config=body.config or {},
|
||||
)
|
||||
|
||||
result = engine.add_widget(dashboard_id, widget)
|
||||
if not result:
|
||||
raise HTTPException(status_code=404, detail="Dashboard not found")
|
||||
|
||||
return {"success": True, "widget": {"widget_id": widget.widget_id, "title": widget.title}}
|
||||
|
||||
|
||||
@router.get("/trends/{metric_name}")
|
||||
async def get_trends(request: Request, metric_name: str, window: int = 60):
|
||||
"""Get trend analysis for a metric."""
|
||||
await require_admin(request, "analytics.read")
|
||||
|
||||
engine = _get_engine()
|
||||
trends = engine.detect_trends(metric_name, window)
|
||||
return trends
|
||||
|
||||
|
||||
@router.get("/stats")
|
||||
async def get_stats(request: Request):
|
||||
"""Get analytics system statistics."""
|
||||
await require_admin(request, "analytics.read")
|
||||
|
||||
engine = _get_engine()
|
||||
return engine.get_system_stats()
|
||||
|
||||
|
||||
@router.get("/prometheus")
|
||||
async def prometheus_export(request: Request):
|
||||
"""Export metrics in Prometheus format."""
|
||||
# Public endpoint for Prometheus scraping
|
||||
engine = _get_engine()
|
||||
return engine.to_prometheus()
|
||||
|
||||
|
||||
@router.get("/export/{metric_name}")
|
||||
async def export_metric(
|
||||
request: Request,
|
||||
metric_name: str,
|
||||
format: str = Query("json", regex="^(json|csv)$"),
|
||||
):
|
||||
"""Export metric data."""
|
||||
await require_admin(request, "analytics.read")
|
||||
|
||||
engine = _get_engine()
|
||||
data = engine.export_metric(metric_name, format)
|
||||
if data is None:
|
||||
raise HTTPException(status_code=404, detail="Metric not found")
|
||||
|
||||
if format == "csv":
|
||||
return {"data": data, "format": "csv"}
|
||||
return data
|
||||
|
||||
|
||||
@router.get("/realtime/{dashboard_id}")
|
||||
async def realtime_data(request: Request, dashboard_id: str):
|
||||
"""Get real-time dashboard data (WebSocket-compatible)."""
|
||||
await require_admin(request, "analytics.read")
|
||||
|
||||
engine = _get_engine()
|
||||
data = engine.get_dashboard_data(dashboard_id)
|
||||
if "error" in data:
|
||||
raise HTTPException(status_code=404, detail=data["error"])
|
||||
|
||||
# Add timestamp for client sync
|
||||
data["server_time"] = time.time()
|
||||
data["refresh_interval"] = 30
|
||||
|
||||
return data
|
||||
|
|
@ -1,449 +0,0 @@
|
|||
"""
|
||||
Historical Data Storage & Analytics Module
|
||||
==========================================
|
||||
|
||||
Provides persistent storage for:
|
||||
- Transaction history
|
||||
- Entity relationships
|
||||
- Alert history
|
||||
- Analytics queries
|
||||
|
||||
Uses Redis for fast-access cache and optional PostgreSQL for long-term storage.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
import redis
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ─── PERSISTENT MODELS ────────────────────────────────────────────────
|
||||
|
||||
|
||||
class TransactionRecord(BaseModel):
|
||||
"""Represents a transaction record."""
|
||||
|
||||
tx_hash: str
|
||||
chain: str = "ethereum"
|
||||
from_address: str
|
||||
to_address: str
|
||||
value: float = 0.0
|
||||
gas_used: int = 0
|
||||
gas_price: int = 0
|
||||
block_number: int = 0
|
||||
timestamp: datetime = Field(default_factory=datetime.utcnow)
|
||||
status: int = 1 # 1 = success, 0 = failed
|
||||
function_name: str = ""
|
||||
token_transfers: list[dict[str, Any]] = Field(default_factory=list)
|
||||
|
||||
|
||||
class WalletHistory(BaseModel):
|
||||
"""Complete transaction history for a wallet."""
|
||||
|
||||
wallet_address: str
|
||||
chain: str = "ethereum"
|
||||
transactions: list[TransactionRecord] = Field(default_factory=list)
|
||||
first_seen: datetime = Field(default_factory=datetime.utcnow)
|
||||
last_seen: datetime = Field(default_factory=datetime.utcnow)
|
||||
total_tx_count: int = 0
|
||||
total_volume: float = 0.0
|
||||
unique_interactions: int = 0
|
||||
|
||||
|
||||
class EntityAlertRecord(BaseModel):
|
||||
"""Record of an alert for an entity."""
|
||||
|
||||
alert_id: str
|
||||
entity_id: str
|
||||
alert_type: str
|
||||
severity: str
|
||||
message: str
|
||||
timestamp: datetime = Field(default_factory=datetime.utcnow)
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
resolved: bool = False
|
||||
|
||||
|
||||
# ─── REDIS STORAGE ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class RedisStorage:
|
||||
"""Redis-backed storage for historical data."""
|
||||
|
||||
def __init__(self, host: str = "localhost", port: int = 6379, db: int = 0):
|
||||
self.host = host
|
||||
self.port = port
|
||||
self.db = db
|
||||
self.client = redis.Redis(host=host, port=port, db=db, decode_responses=True)
|
||||
logger.info(f"Connected to Redis at {host}:{port}/{db}")
|
||||
|
||||
def key_prefix(self, category: str, identifier: str) -> str:
|
||||
"""Generate a Redis key."""
|
||||
return f"rmi:{category}:{identifier}"
|
||||
|
||||
def save_transaction(self, tx: TransactionRecord) -> bool:
|
||||
"""Save a transaction record."""
|
||||
try:
|
||||
key = self.key_prefix("transaction", tx.tx_hash)
|
||||
self.client.setex(
|
||||
key,
|
||||
86400 * 30, # 30 days TTL
|
||||
tx.json(),
|
||||
)
|
||||
|
||||
# Index by wallet
|
||||
wallet_key = self.key_prefix("wallet", f"{tx.from_address}_{tx.chain}")
|
||||
self.client.zadd(wallet_key, {tx.tx_hash: tx.timestamp.timestamp()})
|
||||
|
||||
# Update wallet history
|
||||
self._update_wallet_history(tx.wallet_address if hasattr(tx, "wallet_address") else tx.from_address, tx)
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to save transaction: {e}")
|
||||
return False
|
||||
|
||||
def get_transaction(self, tx_hash: str) -> dict[str, Any] | None:
|
||||
"""Get a transaction record."""
|
||||
try:
|
||||
key = self.key_prefix("transaction", tx_hash)
|
||||
data = self.client.get(key)
|
||||
return json.loads(data) if data else None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get transaction: {e}")
|
||||
return None
|
||||
|
||||
def get_wallet_history(self, wallet_address: str, chain: str = "ethereum") -> WalletHistory | None:
|
||||
"""Get complete wallet history."""
|
||||
try:
|
||||
key = self.key_prefix("wallet", f"{wallet_address}_{chain}")
|
||||
|
||||
if not self.client.exists(key):
|
||||
return None
|
||||
|
||||
tx_hashes = self.client.zrange(key, 0, -1, withscores=True)
|
||||
|
||||
transactions = []
|
||||
for tx_hash, score in tx_hashes:
|
||||
tx = self.get_transaction(tx_hash)
|
||||
if tx:
|
||||
tx_record = TransactionRecord(**tx)
|
||||
tx_record.timestamp = datetime.fromtimestamp(score)
|
||||
transactions.append(tx_record)
|
||||
|
||||
# Sort by timestamp
|
||||
transactions.sort(key=lambda x: x.timestamp)
|
||||
|
||||
return WalletHistory(
|
||||
wallet_address=wallet_address,
|
||||
chain=chain,
|
||||
transactions=transactions,
|
||||
total_tx_count=len(transactions),
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get wallet history: {e}")
|
||||
return None
|
||||
|
||||
def save_alert(self, alert: EntityAlertRecord) -> bool:
|
||||
"""Save an alert record."""
|
||||
try:
|
||||
key = self.key_prefix("alert", alert.alert_id)
|
||||
self.client.setex(
|
||||
key,
|
||||
86400 * 90, # 90 days TTL
|
||||
alert.json(),
|
||||
)
|
||||
|
||||
# Index by entity
|
||||
entity_key = self.key_prefix("entity_alert", alert.entity_id)
|
||||
self.client.zadd(entity_key, {alert.alert_id: alert.timestamp.timestamp()})
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to save alert: {e}")
|
||||
return False
|
||||
|
||||
def get_entity_alerts(self, entity_id: str, limit: int = 100) -> list[dict[str, Any]]:
|
||||
"""Get alerts for an entity."""
|
||||
try:
|
||||
key = self.key_prefix("entity_alert", entity_id)
|
||||
|
||||
if not self.client.exists(key):
|
||||
return []
|
||||
|
||||
alert_ids = self.client.zrange(key, 0, limit - 1)
|
||||
|
||||
alerts = []
|
||||
for alert_id in alert_ids:
|
||||
alert = self.get_alert(alert_id)
|
||||
if alert:
|
||||
alerts.append(alert)
|
||||
|
||||
return alerts
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get entity alerts: {e}")
|
||||
return []
|
||||
|
||||
def get_alert(self, alert_id: str) -> dict[str, Any] | None:
|
||||
"""Get an alert record."""
|
||||
try:
|
||||
key = self.key_prefix("alert", alert_id)
|
||||
data = self.client.get(key)
|
||||
return json.loads(data) if data else None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get alert: {e}")
|
||||
return None
|
||||
|
||||
def save_entity_relation(self, from_entity: str, to_entity: str, relation: str):
|
||||
"""Save an entity relationship."""
|
||||
try:
|
||||
key = self.key_prefix("entity_relation", from_entity)
|
||||
self.client.sadd(key, json.dumps({"to": to_entity, "relation": relation}))
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to save entity relation: {e}")
|
||||
|
||||
def get_entity_relations(self, entity_id: str) -> list[dict[str, str]]:
|
||||
"""Get relations for an entity."""
|
||||
try:
|
||||
key = self.key_prefix("entity_relation", entity_id)
|
||||
relations = self.client.smembers(key)
|
||||
return [json.loads(r) for r in relations]
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get entity relations: {e}")
|
||||
return []
|
||||
|
||||
def save_wallet_cluster(self, cluster_id: str, members: list[str], labels: list[str]):
|
||||
"""Save a wallet cluster."""
|
||||
try:
|
||||
key = self.key_prefix("cluster", cluster_id)
|
||||
data = {
|
||||
"cluster_id": cluster_id,
|
||||
"members": members,
|
||||
"labels": labels,
|
||||
"created_at": datetime.utcnow().isoformat(),
|
||||
}
|
||||
self.client.setex(key, 86400 * 365, json.dumps(data)) # 1 year TTL
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to save cluster: {e}")
|
||||
|
||||
def get_wallet_cluster(self, cluster_id: str) -> dict[str, Any] | None:
|
||||
"""Get a wallet cluster."""
|
||||
try:
|
||||
key = self.key_prefix("cluster", cluster_id)
|
||||
data = self.client.get(key)
|
||||
return json.loads(data) if data else None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get cluster: {e}")
|
||||
return None
|
||||
|
||||
def get_or_create_wallet_history(self, wallet_address: str, chain: str = "ethereum") -> WalletHistory:
|
||||
"""Get or create wallet history."""
|
||||
history = self.get_wallet_history(wallet_address, chain)
|
||||
if history is None:
|
||||
history = WalletHistory(wallet_address=wallet_address, chain=chain)
|
||||
return history
|
||||
|
||||
def _update_wallet_history(self, wallet_address: str, tx: TransactionRecord):
|
||||
"""Update wallet history metadata."""
|
||||
history = self.get_or_create_wallet_history(wallet_address, tx.chain)
|
||||
|
||||
# Update last seen
|
||||
history.last_seen = tx.timestamp
|
||||
|
||||
# Update total volume
|
||||
history.total_volume += tx.value
|
||||
|
||||
# Update interaction count
|
||||
if tx.to_address not in [t.to_address for t in history.transactions]:
|
||||
history.unique_interactions += 1
|
||||
|
||||
# Update transaction count
|
||||
history.total_tx_count = len(history.transactions) + 1
|
||||
|
||||
|
||||
# ─── DATABASE WRAPPER ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
class AnalyticsDatabase:
|
||||
"""Database wrapper for analytics queries."""
|
||||
|
||||
def __init__(self, redis_host: str = "localhost", redis_port: int = 6379, redis_db: int = 0):
|
||||
self.redis = RedisStorage(host=redis_host, port=redis_port, db=redis_db)
|
||||
|
||||
def store_transaction(self, tx: TransactionRecord) -> bool:
|
||||
"""Store a transaction."""
|
||||
return self.redis.save_transaction(tx)
|
||||
|
||||
def store_entity_alert(self, alert: EntityAlertRecord) -> bool:
|
||||
"""Store an entity alert."""
|
||||
return self.redis.save_alert(alert)
|
||||
|
||||
def store_entity_relation(self, from_entity: str, to_entity: str, relation: str):
|
||||
"""Store an entity relationship."""
|
||||
self.redis.save_entity_relation(from_entity, to_entity, relation)
|
||||
|
||||
def store_wallet_cluster(self, cluster_id: str, members: list[str], labels: list[str]):
|
||||
"""Store a wallet cluster."""
|
||||
self.redis.save_wallet_cluster(cluster_id, members, labels)
|
||||
|
||||
def get_wallet_history(self, wallet_address: str, chain: str = "ethereum") -> WalletHistory | None:
|
||||
"""Get wallet history."""
|
||||
return self.redis.get_wallet_history(wallet_address, chain)
|
||||
|
||||
def get_entity_alerts(self, entity_id: str, limit: int = 100) -> list[dict[str, Any]]:
|
||||
"""Get entity alerts."""
|
||||
return self.redis.get_entity_alerts(entity_id, limit)
|
||||
|
||||
def get_entity_relations(self, entity_id: str) -> list[dict[str, str]]:
|
||||
"""Get entity relations."""
|
||||
return self.redis.get_entity_relations(entity_id)
|
||||
|
||||
def get_wallet_cluster(self, cluster_id: str) -> dict[str, Any] | None:
|
||||
"""Get wallet cluster."""
|
||||
return self.redis.get_wallet_cluster(cluster_id)
|
||||
|
||||
def get_transaction(self, tx_hash: str) -> dict[str, Any] | None:
|
||||
"""Get transaction by hash."""
|
||||
return self.redis.get_transaction(tx_hash)
|
||||
|
||||
# ─── ANALYTICS QUERIES ────────────────────────────────────────────
|
||||
|
||||
def get_wallet_activity_summary(self, wallet_address: str, chain: str = "ethereum") -> dict[str, Any]:
|
||||
"""Get activity summary for a wallet."""
|
||||
history = self.redis.get_wallet_history(wallet_address, chain)
|
||||
|
||||
if history is None or not history.transactions:
|
||||
return {
|
||||
"wallet_address": wallet_address,
|
||||
"chain": chain,
|
||||
"total_transactions": 0,
|
||||
"total_volume": 0,
|
||||
"first_seen": None,
|
||||
"last_seen": None,
|
||||
"unique_contracts": 0,
|
||||
}
|
||||
|
||||
return {
|
||||
"wallet_address": wallet_address,
|
||||
"chain": chain,
|
||||
"total_transactions": len(history.transactions),
|
||||
"total_volume": history.total_volume,
|
||||
"first_seen": history.first_seen.isoformat(),
|
||||
"last_seen": history.last_seen.isoformat(),
|
||||
"unique_contracts": history.unique_interactions,
|
||||
}
|
||||
|
||||
def get_wallet_similarity(self, address1: str, address2: str) -> dict[str, Any]:
|
||||
"""Calculate similarity between two wallets based on interactions."""
|
||||
history1 = self.redis.get_wallet_history(address1, "ethereum")
|
||||
history2 = self.redis.get_wallet_history(address2, "ethereum")
|
||||
|
||||
if not history1 or not history2 or not history1.transactions or not history2.transactions:
|
||||
return {"similarity": 0, "shared_contracts": [], "reason": "Insufficient data"}
|
||||
|
||||
# Get unique contract interactions
|
||||
contracts1 = {t.to_address for t in history1.transactions}
|
||||
contracts2 = {t.to_address for t in history2.transactions}
|
||||
|
||||
# Calculate Jaccard similarity
|
||||
intersection = contracts1 & contracts2
|
||||
union = contracts1 | contracts2
|
||||
|
||||
jaccard = len(intersection) / len(union) if union else 0
|
||||
|
||||
return {
|
||||
"similarity": round(jaccard, 4),
|
||||
"shared_contracts": list(intersection)[:10], # Top 10
|
||||
"total_shared": len(intersection),
|
||||
"unique_1": len(contracts1 - contracts2),
|
||||
"unique_2": len(contracts2 - contracts1),
|
||||
}
|
||||
|
||||
def get_entity_network(self, entity_id: str, depth: int = 2) -> dict[str, Any]:
|
||||
"""Get entity's network of connected entities."""
|
||||
relations = self.redis.get_entity_relations(entity_id)
|
||||
|
||||
network = {"entity_id": entity_id, "direct_relations": relations, "depth": depth}
|
||||
|
||||
# If depth > 0, get relations of related entities
|
||||
if depth > 0:
|
||||
related_entities = [r["to"] for r in relations]
|
||||
network["related_entities"] = related_entities
|
||||
|
||||
if depth >= 2:
|
||||
network["second_degree"] = []
|
||||
for related in related_entities:
|
||||
second_degree = self.redis.get_entity_relations(related)
|
||||
network["second_degree"].extend(second_degree)
|
||||
|
||||
return network
|
||||
|
||||
|
||||
# ─── SINGLETON INSTANCE ───────────────────────────────────────────────
|
||||
|
||||
_db_instance: AnalyticsDatabase | None = None
|
||||
|
||||
|
||||
def get_analytics_database(
|
||||
redis_host: str | None = None, redis_port: int | None = None, redis_db: int | None = None
|
||||
) -> AnalyticsDatabase:
|
||||
"""Get the analytics database instance."""
|
||||
global _db_instance
|
||||
|
||||
if _db_instance is None:
|
||||
_db_instance = AnalyticsDatabase(
|
||||
redis_host=redis_host or "localhost",
|
||||
redis_port=redis_port or 6379,
|
||||
redis_db=redis_db or 0,
|
||||
)
|
||||
|
||||
return _db_instance
|
||||
|
||||
|
||||
# ─── INITIAL DATA IMPORT ──────────────────────────────────────────────
|
||||
|
||||
|
||||
def initialize_analytics():
|
||||
"""Initialize analytics storage with default data."""
|
||||
get_analytics_database()
|
||||
|
||||
# Clear old data (optional - for fresh starts)
|
||||
# db.redis.client.flushdb()
|
||||
|
||||
logger.info("Analytics database initialized")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Test the analytics database
|
||||
db = get_analytics_database()
|
||||
|
||||
# Create a test transaction
|
||||
tx = TransactionRecord(
|
||||
tx_hash="0x" + "a" * 64,
|
||||
chain="ethereum",
|
||||
from_address="0x1234567890123456789012345678901234567890",
|
||||
to_address="0xabcdef1234567890abcdef1234567890abcdef12",
|
||||
value=1.5,
|
||||
gas_used=21000,
|
||||
block_number=1000000,
|
||||
function_name="transfer",
|
||||
)
|
||||
|
||||
db.store_transaction(tx)
|
||||
|
||||
# Get the transaction back
|
||||
stored = db.get_transaction(tx.tx_hash)
|
||||
print(f"Stored transaction: {stored}")
|
||||
|
||||
# Get wallet history
|
||||
history = db.get_wallet_history(tx.from_address, "ethereum")
|
||||
if history:
|
||||
print(f"Wallet history: {history.total_tx_count} transactions")
|
||||
|
||||
# Get activity summary
|
||||
summary = db.get_wallet_activity_summary(tx.from_address, "ethereum")
|
||||
print(f"Activity summary: {summary}")
|
||||
|
|
@ -1,49 +0,0 @@
|
|||
"""API versioning router for RMI Backend."""
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
|
||||
router = APIRouter(prefix="", tags=["api-versioning"])
|
||||
|
||||
DEPRECATED_PATHS: dict[str, str] = {}
|
||||
|
||||
|
||||
@router.get("/api/version")
|
||||
async def api_version():
|
||||
"""Return current API version info."""
|
||||
return {
|
||||
"versions": {"v1": {"status": "stable"}, "v2": {"status": "planned"}},
|
||||
"current": "v1",
|
||||
"docs": "/docs",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/api")
|
||||
async def api_root():
|
||||
"""API root with version links."""
|
||||
return {
|
||||
"message": "RugMunch Intelligence API",
|
||||
"versions": {"v1": "/api/v1/", "v2": "/api/v2/ (planned)"},
|
||||
"docs": {"swagger": "/docs", "redoc": "/redoc", "openapi": "/openapi.json"},
|
||||
"health": "/health",
|
||||
"metrics": "/metrics",
|
||||
}
|
||||
|
||||
|
||||
def register_deprecation(path: str, new_path: str) -> None:
|
||||
"""Register a deprecated path for sunset headers."""
|
||||
DEPRECATED_PATHS[path] = new_path
|
||||
|
||||
|
||||
def setup_deprecation_middleware(app):
|
||||
"""Add deprecation headers middleware to the FastAPI app."""
|
||||
|
||||
@app.middleware("http")
|
||||
async def _deprecation(request: Request, call_next):
|
||||
path = request.url.path
|
||||
if path in DEPRECATED_PATHS:
|
||||
response = await call_next(request)
|
||||
response.headers["Deprecation"] = "true"
|
||||
response.headers["Sunset"] = "Sat, 01 Jan 2027 00:00:00 GMT"
|
||||
response.headers["Link"] = f'<{DEPRECATED_PATHS[path]}>; rel="successor-version"'
|
||||
return response
|
||||
return await call_next(request)
|
||||
|
|
@ -1,97 +0,0 @@
|
|||
"""Cross-chain arbitrage detector. Finds price discrepancies across 112 chains."""
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
router = APIRouter(prefix="/api/v1/arbitrage", tags=["arbitrage"])
|
||||
|
||||
DEXSCREENER = "https://api.dexscreener.com/latest/dex/search"
|
||||
CHAINS = ["ethereum", "bsc", "arbitrum", "base", "polygon", "avalanche", "optimism", "solana"]
|
||||
STABLES = {"USDC", "USDT", "DAI", "BUSD", "USDD", "FRAX"}
|
||||
|
||||
|
||||
async def _get_multi_chain_price(symbol: str) -> list[dict]:
|
||||
"""Get price across all chains from DexScreener."""
|
||||
listings = []
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(f"{DEXSCREENER}?q={symbol}")
|
||||
if r.status_code == 200:
|
||||
pairs = r.json().get("pairs", [])
|
||||
for p in pairs[:50]:
|
||||
chain = p.get("chainId", "unknown")
|
||||
if chain in CHAINS or chain == "solana":
|
||||
listings.append(
|
||||
{
|
||||
"chain": chain,
|
||||
"price_usd": float(p.get("priceUsd", 0) or 0),
|
||||
"liquidity_usd": p.get("liquidity", {}).get("usd", 0) or 0,
|
||||
"dex": p.get("dexId", "?"),
|
||||
"pair": p.get("pairAddress", ""),
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
return listings
|
||||
|
||||
|
||||
@router.get("/find/{symbol}")
|
||||
async def find_arbitrage(symbol: str, min_profit_pct: float = Query(0.5, ge=0.1), max_gas_usd: float = Query(50, ge=1)):
|
||||
"""Find arbitrage opportunities for a token across chains."""
|
||||
listings = await _get_multi_chain_price(symbol)
|
||||
if len(listings) < 2:
|
||||
return {"symbol": symbol, "opportunities": [], "note": "Need at least 2 chains with liquidity"}
|
||||
|
||||
# Find min/max price
|
||||
listings.sort(key=lambda l: l["price_usd"]) # noqa: E741
|
||||
cheapest = listings[0]
|
||||
most_expensive = listings[-1]
|
||||
|
||||
spread_pct = (
|
||||
((most_expensive["price_usd"] - cheapest["price_usd"]) / cheapest["price_usd"] * 100)
|
||||
if cheapest["price_usd"] > 0
|
||||
else 0
|
||||
)
|
||||
|
||||
opportunities = []
|
||||
if spread_pct >= min_profit_pct:
|
||||
gross_profit = most_expensive["price_usd"] - cheapest["price_usd"]
|
||||
net_profit = gross_profit - max_gas_usd
|
||||
opportunities.append(
|
||||
{
|
||||
"buy_chain": cheapest["chain"],
|
||||
"buy_price": cheapest["price_usd"],
|
||||
"buy_dex": cheapest["dex"],
|
||||
"sell_chain": most_expensive["chain"],
|
||||
"sell_price": most_expensive["price_usd"],
|
||||
"sell_dex": most_expensive["dex"],
|
||||
"spread_pct": round(spread_pct, 2),
|
||||
"gross_profit_per_unit": round(gross_profit, 4),
|
||||
"estimated_gas": max_gas_usd,
|
||||
"net_profit_estimate": round(net_profit, 2),
|
||||
"viable": net_profit > 0,
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"symbol": symbol,
|
||||
"chains_with_liquidity": len(listings),
|
||||
"cheapest": {"chain": cheapest["chain"], "price": cheapest["price_usd"]},
|
||||
"most_expensive": {"chain": most_expensive["chain"], "price": most_expensive["price_usd"]},
|
||||
"max_spread_pct": round(spread_pct, 2),
|
||||
"opportunities": opportunities,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/scan")
|
||||
async def scan_arbitrage(limit: int = Query(10, le=20)):
|
||||
"""Scan popular tokens for arbitrage opportunities."""
|
||||
popular = ["ETH", "USDC", "USDT", "MATIC", "ARB", "OP", "LINK", "UNI", "AAVE", "SNX", "CRV", "LDO"]
|
||||
results = []
|
||||
for symbol in popular[:limit]:
|
||||
opps = await find_arbitrage(symbol, min_profit_pct=0.3)
|
||||
if opps.get("opportunities"):
|
||||
results.append(opps)
|
||||
|
||||
results.sort(key=lambda r: r["max_spread_pct"], reverse=True)
|
||||
return {"scanned": len(popular[:limit]), "found": len(results), "opportunities": results}
|
||||
|
|
@ -1,595 +0,0 @@
|
|||
"""
|
||||
RMI Auth Extensions - Password Reset, Profile Management, Multi-Chain Wallets
|
||||
=============================================================================
|
||||
Adds to existing /root/backend/app/auth.py:
|
||||
- Password reset via email token
|
||||
- Password change (authenticated)
|
||||
- Profile update (display name, email, avatar)
|
||||
- Multi-chain wallet linking/unlinking
|
||||
- Wallet signature verification for EVM, Solana, Tron, BTC, Monero
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel, EmailStr, Field
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(tags=["auth-ext"])
|
||||
|
||||
# ── Config ──
|
||||
JWT_SECRET = os.getenv("JWT_SECRET", "rmi-jwt-secret-change-me")
|
||||
JWT_ALGORITHM = "HS256"
|
||||
RESET_TOKEN_EXPIRY_HOURS = 24
|
||||
|
||||
|
||||
# ── Redis helper ──
|
||||
# (These will be imported or we patch auth.py directly)
|
||||
|
||||
|
||||
# ── Models ──
|
||||
class ForgotPasswordRequest(BaseModel):
|
||||
email: EmailStr
|
||||
|
||||
|
||||
class ResetPasswordRequest(BaseModel):
|
||||
token: str
|
||||
new_password: str = Field(..., min_length=8)
|
||||
|
||||
|
||||
class ChangePasswordRequest(BaseModel):
|
||||
current_password: str
|
||||
new_password: str = Field(..., min_length=8)
|
||||
|
||||
|
||||
class UpdateProfileRequest(BaseModel):
|
||||
display_name: str | None = None
|
||||
avatar_url: str | None = None
|
||||
|
||||
|
||||
class LinkWalletRequest(BaseModel):
|
||||
address: str
|
||||
chain: str
|
||||
signature: str
|
||||
message: str
|
||||
|
||||
|
||||
class UnlinkWalletRequest(BaseModel):
|
||||
address: str
|
||||
chain: str
|
||||
|
||||
|
||||
class WalletInfo(BaseModel):
|
||||
address: str
|
||||
chain: str
|
||||
added_at: str
|
||||
is_primary: bool = False
|
||||
|
||||
|
||||
class ProfileResponse(BaseModel):
|
||||
id: str
|
||||
email: str
|
||||
display_name: str
|
||||
avatar_url: str | None = None
|
||||
tier: str
|
||||
role: str
|
||||
wallets: list[WalletInfo]
|
||||
created_at: str
|
||||
xp: int = 0
|
||||
level: int = 1
|
||||
badges: list[str] = []
|
||||
|
||||
|
||||
# ── Password Reset ──
|
||||
|
||||
|
||||
@router.post("/forgot-password")
|
||||
async def forgot_password(req: ForgotPasswordRequest):
|
||||
"""Request password reset - sends email with reset token."""
|
||||
from app.auth import _get_user_by_email
|
||||
|
||||
user = _get_user_by_email(req.email)
|
||||
if not user:
|
||||
# Return success even if email not found (prevents enumeration)
|
||||
return {
|
||||
"status": "ok",
|
||||
"message": "If this email is registered, a reset link has been sent.",
|
||||
}
|
||||
|
||||
# Generate reset token
|
||||
reset_token = secrets.token_urlsafe(32)
|
||||
token_hash = hashlib.sha256(reset_token.encode()).hexdigest()
|
||||
|
||||
# Store token in Redis with expiry
|
||||
r = get_redis() # noqa: F821 -- pre-existing bug, see fix(f821) tracking issue
|
||||
r.setex(f"rmi:password_reset:{token_hash}", timedelta(hours=RESET_TOKEN_EXPIRY_HOURS), user["id"])
|
||||
|
||||
# TODO: Send actual email via your email router
|
||||
# For now, log it (replace with email service)
|
||||
reset_url = f"https://rugmunch.io/reset-password?token={reset_token}"
|
||||
logger.info(f"[PASSWORD RESET] Email={req.email} URL={reset_url}")
|
||||
|
||||
# If you have email_router.py:
|
||||
try:
|
||||
from app.email_router import send_email
|
||||
|
||||
await send_email(
|
||||
to=req.email,
|
||||
subject="RugMunch Intelligence - Password Reset",
|
||||
body=f"Click to reset your password: {reset_url}\n\nThis link expires in 24 hours.\n\nIf you didn't request this, ignore this email.",
|
||||
html=f"""
|
||||
<html><body style="font-family:sans-serif;max-width:500px;margin:40px auto;">
|
||||
<h2>Password Reset</h2>
|
||||
<p>Click below to reset your RugMunch Intelligence password:</p>
|
||||
<a href="{reset_url}" style="display:inline-block;padding:12px 24px;background:#6366f1;color:#fff;text-decoration:none;border-radius:6px;">Reset Password</a>
|
||||
<p style="color:#666;font-size:12px;margin-top:20px;">Expires in 24 hours. Didn't request this? Ignore this email.</p>
|
||||
</body></html>
|
||||
""",
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Email send failed: {e}")
|
||||
|
||||
return {"status": "ok", "message": "If this email is registered, a reset link has been sent."}
|
||||
|
||||
|
||||
@router.post("/reset-password")
|
||||
async def reset_password(req: ResetPasswordRequest):
|
||||
"""Reset password using token from email."""
|
||||
from app.auth import _get_user, _save_user, hash_password
|
||||
|
||||
token_hash = hashlib.sha256(req.token.encode()).hexdigest()
|
||||
r = get_redis() # noqa: F821 -- pre-existing bug, see fix(f821) tracking issue
|
||||
user_id = r.get(f"rmi:password_reset:{token_hash}")
|
||||
|
||||
if not user_id:
|
||||
raise HTTPException(status_code=400, detail="Invalid or expired reset token")
|
||||
|
||||
user = _get_user(user_id)
|
||||
if not user:
|
||||
raise HTTPException(status_code=400, detail="Invalid or expired reset token")
|
||||
|
||||
# Update password
|
||||
user["password_hash"] = hash_password(req.new_password)
|
||||
user["updated_at"] = datetime.utcnow().isoformat()
|
||||
_save_user(user)
|
||||
|
||||
# Delete used token
|
||||
r.delete(f"rmi:password_reset:{token_hash}")
|
||||
|
||||
# Invalidate all existing sessions for this user
|
||||
# (Optional: force re-login)
|
||||
|
||||
return {"status": "ok", "message": "Password updated successfully. Please log in again."}
|
||||
|
||||
|
||||
# ── Change Password (Authenticated) ──
|
||||
|
||||
|
||||
@router.post("/change-password")
|
||||
async def change_password(req: ChangePasswordRequest, request: Request):
|
||||
"""Change password while logged in."""
|
||||
from app.auth import _save_user, get_current_user, hash_password, verify_password
|
||||
|
||||
user = await get_current_user(request)
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
# Verify current password
|
||||
if not user.get("password_hash"):
|
||||
raise HTTPException(status_code=400, detail="No password set for this account")
|
||||
|
||||
if not verify_password(req.current_password, user["password_hash"]):
|
||||
raise HTTPException(status_code=401, detail="Current password is incorrect")
|
||||
|
||||
# Update password
|
||||
user["password_hash"] = hash_password(req.new_password)
|
||||
user["updated_at"] = datetime.utcnow().isoformat()
|
||||
_save_user(user)
|
||||
|
||||
return {"status": "ok", "message": "Password changed successfully"}
|
||||
|
||||
|
||||
# ── Profile Management ──
|
||||
|
||||
|
||||
@router.get("/profile", response_model=ProfileResponse)
|
||||
async def get_profile(request: Request):
|
||||
"""Get full user profile including linked wallets."""
|
||||
from app.auth import get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
# Get linked wallets
|
||||
r = get_redis() # noqa: F821 -- pre-existing bug, see fix(f821) tracking issue
|
||||
wallets_raw = r.hget("rmi:user_wallets", user["id"])
|
||||
wallets = json.loads(wallets_raw) if wallets_raw else []
|
||||
|
||||
return {
|
||||
"id": user["id"],
|
||||
"email": user["email"],
|
||||
"display_name": user.get("display_name", user["email"]),
|
||||
"avatar_url": user.get("avatar_url"),
|
||||
"tier": user.get("tier", "FREE"),
|
||||
"role": user.get("role", "USER"),
|
||||
"wallets": wallets,
|
||||
"created_at": user.get("created_at"),
|
||||
"xp": user.get("xp", 0),
|
||||
"level": user.get("level", 1),
|
||||
"badges": user.get("badges", []),
|
||||
}
|
||||
|
||||
|
||||
@router.patch("/profile")
|
||||
async def update_profile(req: UpdateProfileRequest, request: Request):
|
||||
"""Update user profile (display name, avatar)."""
|
||||
from app.auth import _save_user, get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
if req.display_name is not None:
|
||||
user["display_name"] = req.display_name[:50] # Max 50 chars
|
||||
if req.avatar_url is not None:
|
||||
user["avatar_url"] = req.avatar_url[:500] # Max 500 chars
|
||||
|
||||
user["updated_at"] = datetime.utcnow().isoformat()
|
||||
_save_user(user)
|
||||
|
||||
return {"status": "ok", "message": "Profile updated"}
|
||||
|
||||
|
||||
# ── Multi-Chain Wallet Management ──
|
||||
|
||||
CHAIN_CONFIG = {
|
||||
"ethereum": {"name": "Ethereum", "type": "evm", "chain_id": 1},
|
||||
"base": {"name": "Base", "type": "evm", "chain_id": 8453},
|
||||
"arbitrum": {"name": "Arbitrum", "type": "evm", "chain_id": 42161},
|
||||
"optimism": {"name": "Optimism", "type": "evm", "chain_id": 10},
|
||||
"polygon": {"name": "Polygon", "type": "evm", "chain_id": 137},
|
||||
"bsc": {"name": "BSC", "type": "evm", "chain_id": 56},
|
||||
"avalanche": {"name": "Avalanche", "type": "evm", "chain_id": 43114},
|
||||
"fantom": {"name": "Fantom", "type": "evm", "chain_id": 250},
|
||||
"solana": {"name": "Solana", "type": "solana"},
|
||||
"tron": {"name": "Tron", "type": "tron"},
|
||||
"bitcoin": {"name": "Bitcoin", "type": "btc"},
|
||||
"monero": {"name": "Monero", "type": "xmr"},
|
||||
"litecoin": {"name": "Litecoin", "type": "ltc"},
|
||||
"dogecoin": {"name": "Dogecoin", "type": "doge"},
|
||||
"ripple": {"name": "XRP", "type": "xrp"},
|
||||
}
|
||||
|
||||
|
||||
def verify_evm_signature(message: str, signature: str, address: str) -> bool:
|
||||
"""Verify EVM (Ethereum, Base, BSC, etc.) signature."""
|
||||
try:
|
||||
from eth_account import Account
|
||||
from eth_account.messages import encode_defunct
|
||||
|
||||
message_encoded = encode_defunct(text=message)
|
||||
recovered = Account.recover_message(message_encoded, signature=signature)
|
||||
return recovered.lower() == address.lower()
|
||||
except Exception as e:
|
||||
logger.warning(f"EVM signature verification failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def verify_solana_signature(message: str, signature: str, address: str) -> bool:
|
||||
"""Verify Solana signature."""
|
||||
try:
|
||||
import base58
|
||||
import nacl.signing
|
||||
|
||||
pubkey = base58.b58decode(address)
|
||||
sig = base58.b58decode(signature)
|
||||
verify_key = nacl.signing.VerifyKey(pubkey)
|
||||
verify_key.verify(message.encode(), sig)
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.warning(f"Solana signature verification failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def verify_tron_signature(message: str, signature: str, address: str) -> bool:
|
||||
"""Verify Tron signature (uses same curve as Ethereum)."""
|
||||
try:
|
||||
# Tron addresses start with T, need to convert to ETH format for verification
|
||||
from eth_account import Account
|
||||
from eth_account.messages import encode_defunct
|
||||
|
||||
# Basic Tron address validation
|
||||
if not address.startswith("T") or len(address) != 34:
|
||||
return False
|
||||
|
||||
# For Tron, we use the same ECDSA as Ethereum but address format differs
|
||||
# In production, use tronpy or proper conversion
|
||||
message_encoded = encode_defunct(text=message)
|
||||
Account.recover_message(message_encoded, signature=signature)
|
||||
|
||||
# Convert recovered ETH address to Tron base58 (simplified)
|
||||
# Full implementation needs tronpy
|
||||
return True # Stub - replace with full Tron verification
|
||||
except Exception as e:
|
||||
logger.warning(f"Tron signature verification failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def verify_btc_signature(message: str, signature: str, address: str) -> bool:
|
||||
"""Verify Bitcoin signature."""
|
||||
try:
|
||||
# Use bitcoinlib or ecdsa for full verification
|
||||
# This is a simplified stub
|
||||
return len(signature) >= 20 and len(address) >= 26
|
||||
except Exception as e:
|
||||
logger.warning(f"BTC signature verification failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def verify_monero_signature(message: str, signature: str, address: str) -> bool:
|
||||
"""Verify Monero signature."""
|
||||
try:
|
||||
# Monero uses ed25519 + ring signatures
|
||||
# Full verification requires monero-python or similar
|
||||
return len(address) >= 95 and address.startswith("4")
|
||||
except Exception as e:
|
||||
logger.warning(f"Monero signature verification failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def verify_wallet_signature_chain(message: str, signature: str, address: str, chain: str) -> bool:
|
||||
"""Route signature verification to the correct chain handler."""
|
||||
chain_lower = chain.lower()
|
||||
config = CHAIN_CONFIG.get(chain_lower)
|
||||
|
||||
if not config:
|
||||
logger.warning(f"Unknown chain for signature verification: {chain}")
|
||||
return False
|
||||
|
||||
chain_type = config["type"]
|
||||
|
||||
if chain_type == "evm":
|
||||
return verify_evm_signature(message, signature, address)
|
||||
elif chain_type == "solana":
|
||||
return verify_solana_signature(message, signature, address)
|
||||
elif chain_type == "tron":
|
||||
return verify_tron_signature(message, signature, address)
|
||||
elif chain_type == "btc":
|
||||
return verify_btc_signature(message, signature, address)
|
||||
elif chain_type == "xmr":
|
||||
return verify_monero_signature(message, signature, address)
|
||||
else:
|
||||
# Generic fallback
|
||||
return len(signature) >= 60 and len(address) >= 20
|
||||
|
||||
|
||||
@router.get("/wallet/chains")
|
||||
async def list_supported_chains():
|
||||
"""List all supported wallet chains for connection."""
|
||||
return {
|
||||
"chains": [
|
||||
{
|
||||
"id": k,
|
||||
"name": v["name"],
|
||||
"type": v["type"],
|
||||
"chain_id": v.get("chain_id"),
|
||||
}
|
||||
for k, v in CHAIN_CONFIG.items()
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@router.post("/wallet/link")
|
||||
async def link_wallet(req: LinkWalletRequest, request: Request):
|
||||
"""Link a new wallet to the user's account."""
|
||||
from app.auth import _save_user, get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
# Validate chain
|
||||
if req.chain.lower() not in CHAIN_CONFIG:
|
||||
raise HTTPException(status_code=400, detail=f"Unsupported chain: {req.chain}")
|
||||
|
||||
# Verify signature
|
||||
if not verify_wallet_signature_chain(req.message, req.signature, req.address, req.chain):
|
||||
raise HTTPException(status_code=401, detail="Invalid wallet signature")
|
||||
|
||||
# Get existing wallets
|
||||
r = get_redis() # noqa: F821 -- pre-existing bug, see fix(f821) tracking issue
|
||||
wallets_key = "rmi:user_wallets"
|
||||
wallets_raw = r.hget(wallets_key, user["id"])
|
||||
wallets = json.loads(wallets_raw) if wallets_raw else []
|
||||
|
||||
# Check if already linked
|
||||
for w in wallets:
|
||||
if w["address"].lower() == req.address.lower() and w["chain"].lower() == req.chain.lower():
|
||||
raise HTTPException(status_code=400, detail="Wallet already linked")
|
||||
|
||||
# Add wallet
|
||||
is_primary = len(wallets) == 0
|
||||
new_wallet = {
|
||||
"address": req.address,
|
||||
"chain": req.chain.lower(),
|
||||
"added_at": datetime.utcnow().isoformat(),
|
||||
"is_primary": is_primary,
|
||||
}
|
||||
wallets.append(new_wallet)
|
||||
|
||||
r.hset(wallets_key, user["id"], json.dumps(wallets))
|
||||
|
||||
# Update user's primary wallet if first
|
||||
if is_primary:
|
||||
user["wallet_address"] = req.address
|
||||
user["wallet_chain"] = req.chain.lower()
|
||||
_save_user(user)
|
||||
|
||||
return {"status": "ok", "wallet": new_wallet}
|
||||
|
||||
|
||||
@router.post("/wallet/unlink")
|
||||
async def unlink_wallet(req: UnlinkWalletRequest, request: Request):
|
||||
"""Unlink a wallet from the user's account."""
|
||||
from app.auth import _save_user, get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
r = get_redis() # noqa: F821 -- pre-existing bug, see fix(f821) tracking issue
|
||||
wallets_key = "rmi:user_wallets"
|
||||
wallets_raw = r.hget(wallets_key, user["id"])
|
||||
wallets = json.loads(wallets_raw) if wallets_raw else []
|
||||
|
||||
# Find and remove
|
||||
new_wallets = [
|
||||
w
|
||||
for w in wallets
|
||||
if not (w["address"].lower() == req.address.lower() and w["chain"].lower() == req.chain.lower())
|
||||
]
|
||||
|
||||
if len(new_wallets) == len(wallets):
|
||||
raise HTTPException(status_code=404, detail="Wallet not found")
|
||||
|
||||
# Update primary if needed
|
||||
if new_wallets and not any(w.get("is_primary") for w in new_wallets):
|
||||
new_wallets[0]["is_primary"] = True
|
||||
user["wallet_address"] = new_wallets[0]["address"]
|
||||
user["wallet_chain"] = new_wallets[0]["chain"]
|
||||
elif not new_wallets:
|
||||
user["wallet_address"] = None
|
||||
user["wallet_chain"] = None
|
||||
|
||||
r.hset(wallets_key, user["id"], json.dumps(new_wallets))
|
||||
_save_user(user)
|
||||
|
||||
return {"status": "ok", "message": "Wallet unlinked"}
|
||||
|
||||
|
||||
@router.get("/wallet/list")
|
||||
async def list_wallets(request: Request):
|
||||
"""List all wallets linked to the current user."""
|
||||
from app.auth import get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
r = get_redis() # noqa: F821 -- pre-existing bug, see fix(f821) tracking issue
|
||||
wallets_raw = r.hget("rmi:user_wallets", user["id"])
|
||||
wallets = json.loads(wallets_raw) if wallets_raw else []
|
||||
|
||||
return {"wallets": wallets}
|
||||
|
||||
|
||||
# ── Privacy Settings ──
|
||||
|
||||
|
||||
class PrivacySettings(BaseModel):
|
||||
show_email: bool = False
|
||||
show_wallets: bool = False
|
||||
show_xp_level: bool = True
|
||||
show_badges: bool = True
|
||||
allow_direct_messages: bool = True
|
||||
profile_visibility: str = "public" # public, unlisted, private
|
||||
|
||||
|
||||
@router.get("/privacy-settings")
|
||||
async def get_privacy_settings(request: Request):
|
||||
"""Get user privacy settings."""
|
||||
from app.auth import get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
settings = user.get(
|
||||
"privacy_settings",
|
||||
{
|
||||
"show_email": False,
|
||||
"show_wallets": False,
|
||||
"show_xp_level": True,
|
||||
"show_badges": True,
|
||||
"allow_direct_messages": True,
|
||||
"profile_visibility": "public",
|
||||
},
|
||||
)
|
||||
|
||||
return {"settings": settings}
|
||||
|
||||
|
||||
@router.patch("/privacy-settings")
|
||||
async def update_privacy_settings(req: PrivacySettings, request: Request):
|
||||
"""Update user privacy settings."""
|
||||
from app.auth import _save_user, get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
user["privacy_settings"] = req.model_dump()
|
||||
user["updated_at"] = datetime.utcnow().isoformat()
|
||||
_save_user(user)
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"message": "Privacy settings updated",
|
||||
"settings": user["privacy_settings"],
|
||||
}
|
||||
|
||||
|
||||
# ── Account Deletion (GDPR Compliant) ──
|
||||
|
||||
|
||||
class DeleteAccountRequest(BaseModel):
|
||||
password: str
|
||||
confirm_delete: bool = True
|
||||
|
||||
|
||||
@router.post("/delete-account")
|
||||
async def delete_account(req: DeleteAccountRequest, request: Request):
|
||||
"""
|
||||
Permanently delete user account and all associated data (GDPR compliant).
|
||||
Requires password confirmation.
|
||||
"""
|
||||
from app.auth import _delete_user, get_current_user, verify_password
|
||||
from app.core.redis import get_redis
|
||||
|
||||
user = await get_current_user(request)
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
if not req.confirm_delete:
|
||||
raise HTTPException(status_code=400, detail="You must confirm account deletion")
|
||||
|
||||
# Verify password for security
|
||||
if user.get("password_hash") and not verify_password(req.password, user["password_hash"]):
|
||||
raise HTTPException(status_code=401, detail="Incorrect password")
|
||||
|
||||
user_id = user["id"]
|
||||
email = user.get("email", "unknown")
|
||||
|
||||
# 1. Delete user from main store
|
||||
_delete_user(user_id)
|
||||
|
||||
# 2. Delete linked wallets from Redis
|
||||
r = get_redis()
|
||||
r.hdel("rmi:user_wallets", user_id)
|
||||
|
||||
# 3. Delete any password reset tokens
|
||||
# (They will expire naturally, but we can clean up if we track them)
|
||||
|
||||
# 4. Log the deletion for audit purposes (anonymized)
|
||||
logger.warning(f"[ACCOUNT DELETED] User ID={user_id[:8]}... Email={email}")
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"message": "Account permanently deleted. All associated data has been removed.",
|
||||
}
|
||||
|
|
@ -1,84 +0,0 @@
|
|||
"""Auto-Healing Infrastructure - self-diagnosing, self-repairing platform."""
|
||||
|
||||
import os
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import httpx
|
||||
import psutil
|
||||
from fastapi import APIRouter
|
||||
|
||||
router = APIRouter(prefix="/api/v1/health", tags=["auto-healing"])
|
||||
|
||||
BACKEND = os.getenv("BACKEND_URL", "http://localhost:8000")
|
||||
CHECKS = [
|
||||
("redis", "http://localhost:6379", "Redis"),
|
||||
("postgres", "http://localhost:5432", "Postgres"),
|
||||
("ollama", "http://localhost:11434/api/tags", "Ollama"),
|
||||
("backend", f"{BACKEND}/health", "RMI Backend"),
|
||||
("clickhouse", "http://localhost:8123/ping", "ClickHouse"),
|
||||
("qdrant", "http://localhost:6333/health", "Qdrant"),
|
||||
]
|
||||
|
||||
|
||||
@router.get("/full")
|
||||
async def full_health_check():
|
||||
"""Comprehensive system health check - all services."""
|
||||
results = {}
|
||||
all_healthy = True
|
||||
|
||||
async with httpx.AsyncClient(timeout=5) as c:
|
||||
for name, url, label in CHECKS:
|
||||
try:
|
||||
r = await c.get(url)
|
||||
healthy = r.status_code < 500
|
||||
except Exception:
|
||||
healthy = False
|
||||
|
||||
results[name] = {"service": label, "healthy": healthy, "last_checked": datetime.now(UTC).isoformat()}
|
||||
if not healthy:
|
||||
all_healthy = False
|
||||
|
||||
# System resources
|
||||
mem = psutil.virtual_memory()
|
||||
disk = psutil.disk_usage("/")
|
||||
|
||||
return {
|
||||
"overall": "HEALTHY" if all_healthy else "DEGRADED",
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
"services": results,
|
||||
"system": {
|
||||
"cpu_pct": psutil.cpu_percent(interval=1),
|
||||
"memory_pct": mem.percent,
|
||||
"memory_available_gb": round(mem.available / (1024**3), 1),
|
||||
"disk_pct": disk.percent,
|
||||
"disk_free_gb": round(disk.free / (1024**3), 1),
|
||||
"swap_pct": psutil.swap_memory().percent,
|
||||
},
|
||||
"alerts": _check_alerts(mem, disk),
|
||||
}
|
||||
|
||||
|
||||
def _check_alerts(mem, disk) -> list:
|
||||
"""Generate alerts for concerning metrics."""
|
||||
alerts = []
|
||||
if mem.percent > 90:
|
||||
alerts.append({"level": "CRITICAL", "msg": f"Memory usage at {mem.percent}% - consider restarting services"})
|
||||
elif mem.percent > 80:
|
||||
alerts.append({"level": "WARNING", "msg": f"Memory usage at {mem.percent}%"})
|
||||
if disk.percent > 90:
|
||||
alerts.append({"level": "CRITICAL", "msg": f"Disk usage at {disk.percent}% - prune logs immediately"})
|
||||
elif disk.percent > 80:
|
||||
alerts.append({"level": "WARNING", "msg": f"Disk usage at {disk.percent}%"})
|
||||
return alerts
|
||||
|
||||
|
||||
@router.post("/heal/{service}")
|
||||
async def heal_service(service: str):
|
||||
"""Attempt to heal a failed service."""
|
||||
if service == "backend":
|
||||
os.system("docker restart rmi-backend")
|
||||
return {"service": service, "action": "restarted", "note": "Backend restart initiated"}
|
||||
if service == "redis":
|
||||
os.system("docker restart rmi-redis")
|
||||
return {"service": service, "action": "restarted"}
|
||||
return {"service": service, "action": "unknown", "note": "Manual intervention required"}
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
"""Billing module skeleton - x402 crypto payments ready, Stripe slot for later."""
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel
|
||||
|
||||
router = APIRouter(prefix="/api/v1/billing", tags=["billing"])
|
||||
|
||||
# x402 pricing tiers (live)
|
||||
TIERS = {
|
||||
"free": {"price_usd": 0, "requests_per_day": 100, "features": ["basic_scan", "market_data"]},
|
||||
"pro": {
|
||||
"price_usd": 19.99,
|
||||
"requests_per_day": 10000,
|
||||
"features": ["deep_scan", "signals", "whale_alerts", "api_access"],
|
||||
},
|
||||
"enterprise": {
|
||||
"price_usd": 499,
|
||||
"requests_per_day": 999999,
|
||||
"features": ["all", "custom_models", "white_label", "support"],
|
||||
},
|
||||
}
|
||||
|
||||
# Stripe placeholder - ready when you add keys
|
||||
STRIPE_ENABLED = False
|
||||
|
||||
|
||||
class UpgradeRequest(BaseModel):
|
||||
user_id: str
|
||||
tier: str
|
||||
payment_method: str = "x402" # x402 | stripe
|
||||
tx_signature: str | None = None
|
||||
|
||||
|
||||
@router.get("/tiers")
|
||||
async def get_tiers():
|
||||
return {"tiers": TIERS, "payment_methods": ["x402", "stripe (coming soon)"]}
|
||||
|
||||
|
||||
@router.post("/upgrade")
|
||||
async def upgrade_tier(req: UpgradeRequest):
|
||||
if req.tier not in TIERS:
|
||||
return {"error": "Invalid tier"}
|
||||
if req.payment_method == "stripe" and not STRIPE_ENABLED:
|
||||
return {"error": "Stripe not configured yet. Use x402 crypto payments."}
|
||||
return {"status": "upgraded", "user": req.user_id, "tier": req.tier, "payment": req.payment_method}
|
||||
|
||||
|
||||
@router.get("/usage/{user_id}")
|
||||
async def get_usage(user_id: str):
|
||||
return {"user_id": user_id, "tier": "free", "requests_today": 0, "limit": 100}
|
||||
|
|
@ -1,259 +0,0 @@
|
|||
"""
|
||||
RMI Marketing Content Generator - Creates all marketing graphics with EXACT brand compliance.
|
||||
Uses detective character, purple/gold colors, circular frames.
|
||||
NO DEVIATION from brand guidelines.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Paths ────────────────────────────────────────────────────
|
||||
|
||||
CHARACTER_PATH = "/root/backend/assets/characters/detective-character.png"
|
||||
LOGO_PATH = "/root/backend/assets/logos/rugmunch-logo.jpg"
|
||||
OUTPUT_DIR = "/root/backend/assets/marketing_generated"
|
||||
|
||||
os.makedirs(OUTPUT_DIR, exist_ok=True)
|
||||
|
||||
# ── Brand Colors (EXACT - NO DEVIATION) ──────────────────────
|
||||
|
||||
BRAND = {
|
||||
"purple": "#2D1B36",
|
||||
"purple_light": "#3D2346",
|
||||
"gold": "#D4AF37",
|
||||
"gold_light": "#F1D475",
|
||||
"gold_dark": "#AA8828",
|
||||
"cyan": "#00FFFF",
|
||||
"white": "#FFFFFF",
|
||||
"green_alert": "#00FF88", # ONLY for rugpulls/scams
|
||||
"red_danger": "#FF4444", # ONLY for losses
|
||||
}
|
||||
|
||||
# ── Marketing Content Types ──────────────────────────────────
|
||||
|
||||
CONTENT_TYPES = {
|
||||
"feature_showcase": {
|
||||
"title": "Feature Showcase",
|
||||
"size": (1200, 675),
|
||||
"bg_color": BRAND["purple"],
|
||||
"text_color": BRAND["gold"],
|
||||
"include_character": True,
|
||||
},
|
||||
"stats_announcement": {
|
||||
"title": "Stats/Metrics",
|
||||
"size": (1200, 675),
|
||||
"bg_color": BRAND["purple"],
|
||||
"text_color": BRAND["gold"],
|
||||
"include_character": False,
|
||||
},
|
||||
"launch_announcement": {
|
||||
"title": "Launch Announcement",
|
||||
"size": (1200, 675),
|
||||
"bg_color": BRAND["purple"],
|
||||
"text_color": BRAND["gold"],
|
||||
"include_character": True,
|
||||
},
|
||||
"platform_overview": {
|
||||
"title": "Platform Overview",
|
||||
"size": (1920, 1080),
|
||||
"bg_color": BRAND["purple"],
|
||||
"text_color": BRAND["gold"],
|
||||
"include_character": True,
|
||||
},
|
||||
"premium_promo": {
|
||||
"title": "Premium Tier Promo",
|
||||
"size": (1080, 1080),
|
||||
"bg_color": BRAND["purple"],
|
||||
"text_color": BRAND["gold"],
|
||||
"include_character": True,
|
||||
},
|
||||
}
|
||||
|
||||
# ── Content Copy Templates ───────────────────────────────────
|
||||
|
||||
MARKETING_COPY = {
|
||||
"smart_money": {
|
||||
"headline": "SMART MONEY TRACKING",
|
||||
"subhead": "Follow The Whales, Profit Like Them",
|
||||
"body": "Track 1,000+ labeled whale wallets in real-time. See what VCs and funds buy BEFORE the pump.",
|
||||
"stats": ["1,000+ Wallets", "Real-Time Alerts", "8 Chains"],
|
||||
},
|
||||
"rug_detection": {
|
||||
"headline": "RUGPULL DETECTION",
|
||||
"subhead": "2-Minute Alert Speed",
|
||||
"body": "7-method detection engine catches rugs before you ape. 2,530+ scams tracked.",
|
||||
"stats": ["7 Methods", "2-Min Alerts", "2,530+ Scams"],
|
||||
},
|
||||
"kol_scorecards": {
|
||||
"headline": "KOL SCORECARDS",
|
||||
"subhead": "No More Fake Gurus",
|
||||
"body": "500+ influencers tracked. Verified on-chain performance. Win rate transparency.",
|
||||
"stats": ["500+ KOLs", "On-Chain Verified", "Win Rates"],
|
||||
},
|
||||
"platform_launch": {
|
||||
"headline": "RUG MUNCH INTELLIGENCE",
|
||||
"subhead": "Track Smart Money. Avoid Rugs. Find Alpha.",
|
||||
"body": "40+ features live. Real-time alerts. Professional crypto intelligence.",
|
||||
"cta": "Join Free - rugmunch.io",
|
||||
},
|
||||
"premium_tier": {
|
||||
"headline": "PREMIUM INTELLIGENCE",
|
||||
"subhead": "For Serious Traders Only",
|
||||
"body": "Smart money tracking. Insider alerts. Exchange flows. Cluster analysis.",
|
||||
"price": "$29/mo",
|
||||
"features": ["Real-Time Alerts", "Smart Money Tracking", "500+ KOLs", "API Access"],
|
||||
},
|
||||
}
|
||||
|
||||
# ── Graphics Generation Functions ────────────────────────────
|
||||
|
||||
|
||||
def create_gradient_background(size, color1, color2):
|
||||
"""Create purple gradient background."""
|
||||
img = Image.new("RGB", size, color1)
|
||||
draw = ImageDraw.Draw(img)
|
||||
|
||||
for y in range(size[1]):
|
||||
alpha = y / size[1]
|
||||
r = int(int(color1[1:3], 16) * (1 - alpha) + int(color2[1:3], 16) * alpha)
|
||||
g = int(int(color1[3:5], 16) * (1 - alpha) + int(color2[3:5], 16) * alpha)
|
||||
b = int(int(color1[5:7], 16) * (1 - alpha) + int(color2[5:7], 16) * alpha)
|
||||
draw.line([(0, y), (size[0], y)], fill=(r, g, b))
|
||||
|
||||
return img
|
||||
|
||||
|
||||
def add_circular_frame(img, color=BRAND["gold"], width=5):
|
||||
"""Add gold circular frame."""
|
||||
draw = ImageDraw.Draw(img)
|
||||
margin = 20
|
||||
draw.ellipse([margin, margin, img.size[0] - margin, img.size[1] - margin], outline=color, width=width)
|
||||
return img
|
||||
|
||||
|
||||
def add_text_centered(img, text, position, font_size, color, font_path=None):
|
||||
"""Add centered text."""
|
||||
draw = ImageDraw.Draw(img)
|
||||
|
||||
try:
|
||||
font = ImageFont.truetype(font_path or "/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf", font_size)
|
||||
except Exception:
|
||||
font = ImageFont.load_default()
|
||||
|
||||
bbox = draw.textbbox((0, 0), text, font=font)
|
||||
text_width = bbox[2] - bbox[0]
|
||||
x = position[0] - text_width // 2
|
||||
draw.text((x, position[1]), text, fill=color, font=font)
|
||||
|
||||
return img
|
||||
|
||||
|
||||
def generate_feature_graphic(feature_key: str) -> dict:
|
||||
"""Generate feature showcase graphic."""
|
||||
config = CONTENT_TYPES["feature_showcase"]
|
||||
copy = MARKETING_COPY.get(feature_key)
|
||||
|
||||
if not copy:
|
||||
return {"error": f"Unknown feature: {feature_key}"}
|
||||
|
||||
# Create background
|
||||
img = create_gradient_background(config["size"], BRAND["purple"], BRAND["purple_light"])
|
||||
|
||||
# Add circular frame
|
||||
img = add_circular_frame(img, BRAND["gold"], width=5)
|
||||
|
||||
ImageDraw.Draw(img)
|
||||
|
||||
# Add headline
|
||||
add_text_centered(img, copy["headline"], (config["size"][0] // 2, 150), 72, BRAND["gold"])
|
||||
|
||||
# Add subhead
|
||||
add_text_centered(img, copy["subhead"], (config["size"][0] // 2, 250), 48, BRAND["white"])
|
||||
|
||||
# Add body
|
||||
add_text_centered(img, copy["body"], (config["size"][0] // 2, 350), 36, BRAND["white"])
|
||||
|
||||
# Add stats
|
||||
y = 450
|
||||
for stat in copy.get("stats", []):
|
||||
add_text_centered(img, f"● {stat}", (config["size"][0] // 2, y), 32, BRAND["cyan"])
|
||||
y += 50
|
||||
|
||||
# Add watermark
|
||||
add_text_centered(img, "@cryptorugmunch", (config["size"][0] // 2, config["size"][1] - 80), 28, BRAND["gold"])
|
||||
|
||||
# Save
|
||||
filename = f"feature_{feature_key}_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.png"
|
||||
output_path = os.path.join(OUTPUT_DIR, filename)
|
||||
img.save(output_path, "PNG")
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"feature": feature_key,
|
||||
"image_path": output_path,
|
||||
"filename": filename,
|
||||
}
|
||||
|
||||
|
||||
def generate_launch_graphic() -> dict:
|
||||
"""Generate platform launch announcement."""
|
||||
config = CONTENT_TYPES["launch_announcement"]
|
||||
copy = MARKETING_COPY["platform_launch"]
|
||||
|
||||
# Create background
|
||||
img = create_gradient_background(config["size"], BRAND["purple"], BRAND["purple_light"])
|
||||
img = add_circular_frame(img, BRAND["gold"], width=5)
|
||||
|
||||
# Add text
|
||||
add_text_centered(img, copy["headline"], (config["size"][0] // 2, 200), 80, BRAND["gold"])
|
||||
add_text_centered(img, copy["subhead"], (config["size"][0] // 2, 320), 48, BRAND["white"])
|
||||
add_text_centered(img, copy["body"], (config["size"][0] // 2, 420), 36, BRAND["white"])
|
||||
add_text_centered(img, copy.get("cta", ""), (config["size"][0] // 2, 550), 42, BRAND["cyan"])
|
||||
add_text_centered(img, "@cryptorugmunch", (config["size"][0] // 2, config["size"][1] - 80), 28, BRAND["gold"])
|
||||
|
||||
# Save
|
||||
filename = f"launch_{datetime.now(UTC).strftime('%Y%m%d_%H%M%S')}.png"
|
||||
output_path = os.path.join(OUTPUT_DIR, filename)
|
||||
img.save(output_path, "PNG")
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"type": "launch",
|
||||
"image_path": output_path,
|
||||
"filename": filename,
|
||||
}
|
||||
|
||||
|
||||
def generate_all_marketing_graphics() -> list[dict]:
|
||||
"""Generate all marketing graphics."""
|
||||
results = []
|
||||
|
||||
# Feature graphics
|
||||
for feature in ["smart_money", "rug_detection", "kol_scorecards"]:
|
||||
result = generate_feature_graphic(feature)
|
||||
results.append(result)
|
||||
|
||||
# Launch graphic
|
||||
result = generate_launch_graphic()
|
||||
results.append(result)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("Generating marketing graphics with EXACT brand compliance...")
|
||||
results = generate_all_marketing_graphics()
|
||||
|
||||
print(f"\n✅ Generated {len(results)} graphics:")
|
||||
for r in results:
|
||||
if r.get("status") == "success":
|
||||
print(f" ✅ {r.get('type') or r.get('feature')}: {r.get('filename')}")
|
||||
else:
|
||||
print(f" ❌ {r.get('error')}")
|
||||
|
||||
print(f"\n📁 All graphics saved to: {OUTPUT_DIR}")
|
||||
|
|
@ -1,547 +0,0 @@
|
|||
"""
|
||||
RMI Bulk Marketing Graphics Generator - 50+ Graphics with Qwen-VL Max
|
||||
Uses Alibaba's best vision model for professional marketing graphics.
|
||||
All graphics follow EXACT brand guidelines. Uploads to Dropbox automatically.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Paths ────────────────────────────────────────────────────
|
||||
|
||||
CHARACTER_PATH = "/root/backend/assets/characters/detective-character.png"
|
||||
LOGO_PATH = "/root/backend/assets/logos/rugmunch-logo.jpg"
|
||||
OUTPUT_DIR = "/root/backend/assets/marketing_generated"
|
||||
DROPBOX_DIR = os.path.expanduser("~/Dropbox/RMI Marketing Graphics")
|
||||
|
||||
os.makedirs(OUTPUT_DIR, exist_ok=True)
|
||||
os.makedirs(DROPBOX_DIR, exist_ok=True)
|
||||
|
||||
# ── Brand Colors (EXACT - NO DEVIATION) ──────────────────────
|
||||
|
||||
BRAND = {
|
||||
"purple": "#2D1B36",
|
||||
"purple_light": "#3D2346",
|
||||
"purple_dark": "#1D0B26",
|
||||
"gold": "#D4AF37",
|
||||
"gold_light": "#F1D475",
|
||||
"gold_dark": "#AA8828",
|
||||
"cyan": "#00FFFF",
|
||||
"white": "#FFFFFF",
|
||||
"green_alert": "#00FF88",
|
||||
"red_danger": "#FF4444",
|
||||
}
|
||||
|
||||
# ── 50+ Marketing Graphics Concepts ──────────────────────────
|
||||
|
||||
GRAPHIC_CONCEPTS = [
|
||||
# Feature Showcases (12)
|
||||
{
|
||||
"type": "feature",
|
||||
"name": "smart_money",
|
||||
"headline": "SMART MONEY TRACKING",
|
||||
"subhead": "Follow The Whales",
|
||||
"stat": "1,000+ Wallets",
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"name": "rug_detection",
|
||||
"headline": "RUGPULL DETECTION",
|
||||
"subhead": "2-Minute Alerts",
|
||||
"stat": "2,530+ Scams",
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"name": "kol_scorecards",
|
||||
"headline": "KOL SCORECARDS",
|
||||
"subhead": "No Fake Gurus",
|
||||
"stat": "500+ KOLs",
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"name": "whale_alerts",
|
||||
"headline": "WHALE ALERTS",
|
||||
"subhead": "Real-Time Moves",
|
||||
"stat": "$10k+ Threshold",
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"name": "bundle_detection",
|
||||
"headline": "BUNDLE DETECTION",
|
||||
"subhead": "Jito/Flashbots",
|
||||
"stat": "5-Signal Engine",
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"name": "cluster_analysis",
|
||||
"headline": "CLUSTER ANALYSIS",
|
||||
"subhead": "Sybil Detection",
|
||||
"stat": "7 Methods",
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"name": "cross_chain",
|
||||
"headline": "CROSS-CHAIN",
|
||||
"subhead": "Multi-Chain Intel",
|
||||
"stat": "8 Chains",
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"name": "nft_intel",
|
||||
"headline": "NFT INTELLIGENCE",
|
||||
"subhead": "Wash Trading Detection",
|
||||
"stat": "Floor Tracking",
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"name": "security_scanner",
|
||||
"headline": "SECURITY SCANNER",
|
||||
"subhead": "Contract Audits",
|
||||
"stat": "Auto-Audit",
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"name": "social_sentiment",
|
||||
"headline": "SOCIAL SENTIMENT",
|
||||
"subhead": "X, Telegram, Discord",
|
||||
"stat": "Real-Time",
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"name": "meme_tracker",
|
||||
"headline": "MEME TRACKER",
|
||||
"subhead": "10,000+ Tokens",
|
||||
"stat": "5 Chains",
|
||||
},
|
||||
{
|
||||
"type": "feature",
|
||||
"name": "premium_api",
|
||||
"headline": "PREMIUM API",
|
||||
"subhead": "Developer Access",
|
||||
"stat": "Webhooks",
|
||||
},
|
||||
# Stats/Metrics (10)
|
||||
{
|
||||
"type": "stats",
|
||||
"name": "total_features",
|
||||
"stat_value": "40+",
|
||||
"stat_label": "Features Live",
|
||||
"context": "Across 11 Services",
|
||||
},
|
||||
{
|
||||
"type": "stats",
|
||||
"name": "api_endpoints",
|
||||
"stat_value": "450+",
|
||||
"stat_label": "API Endpoints",
|
||||
"context": "Ready for Integration",
|
||||
},
|
||||
{
|
||||
"type": "stats",
|
||||
"name": "scams_tracked",
|
||||
"stat_value": "2,530",
|
||||
"stat_label": "Scams Tracked",
|
||||
"context": "And Counting",
|
||||
},
|
||||
{
|
||||
"type": "stats",
|
||||
"name": "kol_tracked",
|
||||
"stat_value": "500+",
|
||||
"stat_label": "KOLs Tracked",
|
||||
"context": "Verified On-Chain",
|
||||
},
|
||||
{
|
||||
"type": "stats",
|
||||
"name": "whale_wallets",
|
||||
"stat_value": "1,000+",
|
||||
"stat_label": "Whale Wallets",
|
||||
"context": "Labeled & Tracked",
|
||||
},
|
||||
{
|
||||
"type": "stats",
|
||||
"name": "alert_speed",
|
||||
"stat_value": "2 Min",
|
||||
"stat_label": "Alert Speed",
|
||||
"context": "Rugpull Detection",
|
||||
},
|
||||
{
|
||||
"type": "stats",
|
||||
"name": "chains_supported",
|
||||
"stat_value": "8",
|
||||
"stat_label": "Chains Supported",
|
||||
"context": "Multi-Chain Intel",
|
||||
},
|
||||
{
|
||||
"type": "stats",
|
||||
"name": "users_served",
|
||||
"stat_value": "1,000+",
|
||||
"stat_label": "Users Served",
|
||||
"context": "And Growing Fast",
|
||||
},
|
||||
{
|
||||
"type": "stats",
|
||||
"name": "uptime",
|
||||
"stat_value": "99.9%",
|
||||
"stat_label": "Uptime",
|
||||
"context": "Enterprise Grade",
|
||||
},
|
||||
{
|
||||
"type": "stats",
|
||||
"name": "data_points",
|
||||
"stat_value": "10M+",
|
||||
"stat_label": "Data Points",
|
||||
"context": "Processed Daily",
|
||||
},
|
||||
# Launch Announcements (8)
|
||||
{
|
||||
"type": "launch",
|
||||
"name": "platform_live",
|
||||
"headline": "RUG MUNCH INTELLIGENCE",
|
||||
"subhead": "LIVE NOW",
|
||||
},
|
||||
{
|
||||
"type": "launch",
|
||||
"name": "premium_launch",
|
||||
"headline": "PREMIUM TIER",
|
||||
"subhead": "Now Available",
|
||||
},
|
||||
{
|
||||
"type": "launch",
|
||||
"name": "api_launch",
|
||||
"headline": "PUBLIC API",
|
||||
"subhead": "Developers Welcome",
|
||||
},
|
||||
{"type": "launch", "name": "mobile_teaser", "headline": "MOBILE APP", "subhead": "Coming Soon"},
|
||||
{"type": "launch", "name": "kol_program", "headline": "KOL PROGRAM", "subhead": "Get Verified"},
|
||||
{
|
||||
"type": "launch",
|
||||
"name": "partnership",
|
||||
"headline": "MAJOR PARTNERSHIP",
|
||||
"subhead": "Announcement",
|
||||
},
|
||||
{
|
||||
"type": "launch",
|
||||
"name": "feature_drop",
|
||||
"headline": "NEW FEATURES",
|
||||
"subhead": "10 Added This Week",
|
||||
},
|
||||
{"type": "launch", "name": "milestone", "headline": "1,000 USERS", "subhead": "Thank You!"},
|
||||
# Pricing Tiers (6)
|
||||
{
|
||||
"type": "pricing",
|
||||
"name": "free_tier",
|
||||
"tier": "FREE",
|
||||
"price": "$0",
|
||||
"features": ["Basic Alerts", "50 Calls/mo", "Community"],
|
||||
},
|
||||
{
|
||||
"type": "pricing",
|
||||
"name": "premium_tier",
|
||||
"tier": "PREMIUM",
|
||||
"price": "$29",
|
||||
"features": ["Real-Time Alerts", "Smart Money", "500+ KOLs"],
|
||||
},
|
||||
{
|
||||
"type": "pricing",
|
||||
"name": "premium_plus",
|
||||
"tier": "PREMIUM+",
|
||||
"price": "$99",
|
||||
"features": ["Insider Alerts", "API Access", "1-on-1 Support"],
|
||||
},
|
||||
{
|
||||
"type": "pricing",
|
||||
"name": "enterprise",
|
||||
"tier": "ENTERPRISE",
|
||||
"price": "Custom",
|
||||
"features": ["White-Label", "Dedicated Support", "SLA"],
|
||||
},
|
||||
{
|
||||
"type": "pricing",
|
||||
"name": "early_adopter",
|
||||
"tier": "EARLY ADOPTER",
|
||||
"price": "$19",
|
||||
"features": ["Lifetime Discount", "All Premium Features", "Badge"],
|
||||
},
|
||||
{
|
||||
"type": "pricing",
|
||||
"name": "comparison",
|
||||
"tier": "VS COMPETITORS",
|
||||
"price": "Save $200/mo",
|
||||
"features": ["Free Tier", "Better Features", "Real-Time"],
|
||||
},
|
||||
# Testimonials/Social Proof (6)
|
||||
{
|
||||
"type": "testimonial",
|
||||
"name": "user_win_1",
|
||||
"quote": "Caught a rug 2 mins before it pulled. RMI paid for itself.",
|
||||
"author": "@degen_trader",
|
||||
"title": "Premium User",
|
||||
},
|
||||
{
|
||||
"type": "testimonial",
|
||||
"name": "user_win_2",
|
||||
"quote": "Following smart money wallets changed my trading game.",
|
||||
"author": "@crypto_whale",
|
||||
"title": "Premium+ User",
|
||||
},
|
||||
{
|
||||
"type": "testimonial",
|
||||
"name": "kol_verified",
|
||||
"quote": "Finally, a platform that verifies our actual performance.",
|
||||
"author": "@alpha_caller",
|
||||
"title": "Verified KOL",
|
||||
},
|
||||
{
|
||||
"type": "testimonial",
|
||||
"name": "dev_feedback",
|
||||
"quote": "Best crypto intelligence API. Clean docs, fast response.",
|
||||
"author": "@dev_builder",
|
||||
"title": "API User",
|
||||
},
|
||||
{
|
||||
"type": "testimonial",
|
||||
"name": "community_love",
|
||||
"quote": "The Telegram community is pure alpha. No shilling.",
|
||||
"author": "@community_mod",
|
||||
"title": "Moderator",
|
||||
},
|
||||
{
|
||||
"type": "testimonial",
|
||||
"name": "enterprise_client",
|
||||
"quote": "Enterprise tier is worth every penny for our fund.",
|
||||
"author": "@fund_manager",
|
||||
"title": "Enterprise",
|
||||
},
|
||||
# Educational/How-To (8)
|
||||
{
|
||||
"type": "educational",
|
||||
"name": "how_to_start",
|
||||
"headline": "GETTING STARTED",
|
||||
"subhead": "5-Minute Setup",
|
||||
},
|
||||
{
|
||||
"type": "educational",
|
||||
"name": "smart_money_101",
|
||||
"headline": "SMART MONEY 101",
|
||||
"subhead": "Follow The Pros",
|
||||
},
|
||||
{
|
||||
"type": "educational",
|
||||
"name": "avoid_rugs",
|
||||
"headline": "AVOID RUGS",
|
||||
"subhead": "7 Red Flags",
|
||||
},
|
||||
{
|
||||
"type": "educational",
|
||||
"name": "kol_analysis",
|
||||
"headline": "ANALYZE KOLs",
|
||||
"subhead": "Check Track Records",
|
||||
},
|
||||
{
|
||||
"type": "educational",
|
||||
"name": "whale_watching",
|
||||
"headline": "WHALE WATCHING",
|
||||
"subhead": "Track Big Moves",
|
||||
},
|
||||
{
|
||||
"type": "educational",
|
||||
"name": "meme_coins",
|
||||
"headline": "MEME COINS",
|
||||
"subhead": "Find The Next 100x",
|
||||
},
|
||||
{
|
||||
"type": "educational",
|
||||
"name": "security_tips",
|
||||
"headline": "SECURITY TIPS",
|
||||
"subhead": "Stay Safe",
|
||||
},
|
||||
{
|
||||
"type": "educational",
|
||||
"name": "api_guide",
|
||||
"headline": "API GUIDE",
|
||||
"subhead": "Build On RMI",
|
||||
},
|
||||
]
|
||||
|
||||
# ── Graphics Generation Functions ────────────────────────────
|
||||
|
||||
|
||||
def create_gradient_background(size, color1, color2, direction="vertical"):
|
||||
"""Create purple gradient background."""
|
||||
img = Image.new("RGB", size, color1)
|
||||
draw = ImageDraw.Draw(img)
|
||||
|
||||
for i in range(size[1] if direction == "vertical" else size[0]):
|
||||
alpha = i / max(size[1] if direction == "vertical" else size[0], 1)
|
||||
r = int(int(color1[1:3], 16) * (1 - alpha) + int(color2[1:3], 16) * alpha)
|
||||
g = int(int(color1[3:5], 16) * (1 - alpha) + int(color2[3:5], 16) * alpha)
|
||||
b = int(int(color1[5:7], 16) * (1 - alpha) + int(color2[5:7], 16) * alpha)
|
||||
|
||||
if direction == "vertical":
|
||||
draw.line([(0, i), (size[0], i)], fill=(r, g, b))
|
||||
else:
|
||||
draw.line([(i, 0), (i, size[1])], fill=(r, g, b))
|
||||
|
||||
return img
|
||||
|
||||
|
||||
def add_circular_frame(img, color=BRAND["gold"], width=5):
|
||||
"""Add gold circular frame."""
|
||||
draw = ImageDraw.Draw(img)
|
||||
margin = 20
|
||||
draw.ellipse([margin, margin, img.size[0] - margin, img.size[1] - margin], outline=color, width=width)
|
||||
return img
|
||||
|
||||
|
||||
def add_text_centered(img, text, position, font_size, color, bold=True):
|
||||
"""Add centered text."""
|
||||
draw = ImageDraw.Draw(img)
|
||||
|
||||
try:
|
||||
font_path = (
|
||||
"/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf"
|
||||
if bold
|
||||
else "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf"
|
||||
)
|
||||
font = ImageFont.truetype(font_path, font_size)
|
||||
except Exception:
|
||||
font = ImageFont.load_default()
|
||||
|
||||
bbox = draw.textbbox((0, 0), text, font=font)
|
||||
text_width = bbox[2] - bbox[0]
|
||||
x = position[0] - text_width // 2
|
||||
draw.text((x, position[1]), text, fill=color, font=font)
|
||||
|
||||
return img
|
||||
|
||||
|
||||
def generate_graphic(concept: dict) -> dict:
|
||||
"""Generate a single marketing graphic."""
|
||||
graphic_type = concept.get("type", "feature")
|
||||
|
||||
# Determine size based on type
|
||||
if graphic_type == "pricing":
|
||||
size = (800, 1000)
|
||||
elif graphic_type == "launch":
|
||||
size = (1200, 675)
|
||||
else:
|
||||
size = (1200, 675)
|
||||
|
||||
# Create background
|
||||
img = create_gradient_background(size, BRAND["purple"], BRAND["purple_light"])
|
||||
img = add_circular_frame(img, BRAND["gold"], width=5)
|
||||
|
||||
# Add content based on type
|
||||
if graphic_type == "feature":
|
||||
add_text_centered(img, concept["headline"], (size[0] // 2, 150), 64, BRAND["gold"])
|
||||
add_text_centered(img, concept["subhead"], (size[0] // 2, 250), 42, BRAND["white"])
|
||||
add_text_centered(img, concept["stat"], (size[0] // 2, 400), 96, BRAND["cyan"])
|
||||
|
||||
elif graphic_type == "stats":
|
||||
add_text_centered(img, concept["stat_value"], (size[0] // 2, 250), 144, BRAND["gold"])
|
||||
add_text_centered(img, concept["stat_label"], (size[0] // 2, 400), 56, BRAND["white"])
|
||||
add_text_centered(img, concept["context"], (size[0] // 2, 480), 36, BRAND["cyan"])
|
||||
|
||||
elif graphic_type == "launch":
|
||||
add_text_centered(img, "🚀 " + concept["subhead"], (size[0] // 2, 150), 64, BRAND["cyan"])
|
||||
add_text_centered(img, concept["headline"], (size[0] // 2, 280), 72, BRAND["gold"])
|
||||
|
||||
elif graphic_type == "pricing":
|
||||
add_text_centered(img, concept["tier"], (size[0] // 2, 150), 64, BRAND["gold"])
|
||||
add_text_centered(
|
||||
img,
|
||||
concept["price"] + "/mo" if concept["price"] != "Custom" else concept["price"],
|
||||
(size[0] // 2, 280),
|
||||
96,
|
||||
BRAND["white"],
|
||||
)
|
||||
y = 400
|
||||
for feature in concept.get("features", []):
|
||||
add_text_centered(img, f"✓ {feature}", (size[0] // 2, y), 32, BRAND["cyan"])
|
||||
y += 50
|
||||
|
||||
elif graphic_type == "testimonial":
|
||||
add_text_centered(img, '"', (size[0] // 2, 150), 144, BRAND["gold"])
|
||||
add_text_centered(img, concept["quote"][:100], (size[0] // 2, 280), 36, BRAND["white"])
|
||||
add_text_centered(img, f"- {concept['author']}", (size[0] // 2, 450), 32, BRAND["gold"])
|
||||
add_text_centered(img, concept["title"], (size[0] // 2, 500), 28, BRAND["cyan"])
|
||||
|
||||
elif graphic_type == "educational":
|
||||
add_text_centered(img, "📚 " + concept["headline"], (size[0] // 2, 200), 64, BRAND["gold"])
|
||||
add_text_centered(img, concept["subhead"], (size[0] // 2, 320), 48, BRAND["white"])
|
||||
|
||||
# Add watermark
|
||||
add_text_centered(img, "@cryptorugmunch", (size[0] // 2, size[1] - 60), 28, BRAND["gold"])
|
||||
|
||||
# Generate filename
|
||||
timestamp = datetime.now(UTC).strftime("%Y%m%d_%H%M%S")
|
||||
filename = f"{graphic_type}_{concept['name']}_{timestamp}.png"
|
||||
output_path = os.path.join(OUTPUT_DIR, filename)
|
||||
|
||||
# Save
|
||||
img.save(output_path, "PNG")
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"type": graphic_type,
|
||||
"name": concept["name"],
|
||||
"filename": filename,
|
||||
"path": output_path,
|
||||
"size": f"{size[0]}x{size[1]}",
|
||||
}
|
||||
|
||||
|
||||
def upload_to_dropbox(local_path: str, dropbox_path: str | None = None) -> bool:
|
||||
"""Upload file to Dropbox."""
|
||||
try:
|
||||
if not dropbox_path:
|
||||
dropbox_path = os.path.join(DROPBOX_DIR, os.path.basename(local_path))
|
||||
|
||||
# Copy file to Dropbox
|
||||
import shutil
|
||||
|
||||
shutil.copy2(local_path, dropbox_path)
|
||||
|
||||
logger.info(f"Uploaded to Dropbox: {dropbox_path}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Dropbox upload failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def generate_all_graphics() -> list[dict]:
|
||||
"""Generate all 50+ marketing graphics."""
|
||||
results = []
|
||||
|
||||
print(f"Generating {len(GRAPHIC_CONCEPTS)} marketing graphics...")
|
||||
print("=" * 60)
|
||||
|
||||
for i, concept in enumerate(GRAPHIC_CONCEPTS, 1):
|
||||
result = generate_graphic(concept)
|
||||
results.append(result)
|
||||
|
||||
# Upload to Dropbox
|
||||
if result["status"] == "success":
|
||||
upload_to_dropbox(result["path"])
|
||||
print(f"[{i:3d}/{len(GRAPHIC_CONCEPTS)}] ✅ {result['type']:15} - {result['name']:25} - {result['size']}")
|
||||
else:
|
||||
print(f"[{i:3d}/{len(GRAPHIC_CONCEPTS)}] ❌ {result.get('error', 'Unknown error')}")
|
||||
|
||||
print("=" * 60)
|
||||
print(f"\n✅ Generated {len([r for r in results if r['status'] == 'success'])}/{len(GRAPHIC_CONCEPTS)} graphics")
|
||||
print(f"📁 Local: {OUTPUT_DIR}")
|
||||
print(f"☁️ Dropbox: {DROPBOX_DIR}")
|
||||
|
||||
return results
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
results = generate_all_graphics()
|
||||
|
||||
# Summary
|
||||
success = len([r for r in results if r["status"] == "success"])
|
||||
print(f"\n🎉 BULK GENERATION COMPLETE: {success} graphics created and backed up to Dropbox!")
|
||||
|
|
@ -1,876 +0,0 @@
|
|||
"""
|
||||
Supply Manipulation / Bundler Detector
|
||||
=======================================
|
||||
Detects bundled token launches where insiders control disproportionate
|
||||
supply through sniper-controlled wallet distributions.
|
||||
|
||||
Signals detected:
|
||||
- Bundled initial buys (multiple wallets funded from same source,
|
||||
buying within same block/seconds)
|
||||
- Supply concentration across linked wallets (top holders controlled
|
||||
by same entity)
|
||||
- Fund flow analysis (same funding source → multiple snipers)
|
||||
- TIMEO (This Is My Eyes Only) token distribution patterns
|
||||
- Sniper cluster detection (wallets that only buy this token)
|
||||
- Launch timing anomalies (coordinated buys in first blocks)
|
||||
- Holder overlap with known bundler addresses
|
||||
- Supply distribution entropy analysis
|
||||
|
||||
Tier : Premium ($0.08)
|
||||
Price : 80000 atoms
|
||||
Endpoint: POST /api/v1/x402-tools/bundler_detect
|
||||
"""
|
||||
|
||||
import logging
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Constants ──────────────────────────────────────────────────────
|
||||
|
||||
SOLANA_ADDR_RE = re.compile(r"^[1-9A-HJ-NP-Za-km-z]{32,44}$")
|
||||
EVM_ADDR_RE = re.compile(r"^0x[a-fA-F0-9]{40}$")
|
||||
|
||||
EVM_CHAINS = frozenset(
|
||||
{
|
||||
"ethereum",
|
||||
"bsc",
|
||||
"polygon",
|
||||
"arbitrum",
|
||||
"optimism",
|
||||
"avalanche",
|
||||
"base",
|
||||
"fantom",
|
||||
"linea",
|
||||
"zksync",
|
||||
"scroll",
|
||||
"mantle",
|
||||
}
|
||||
)
|
||||
|
||||
SUPPORTED_CHAINS = [*EVM_CHAINS, "solana"]
|
||||
|
||||
# DEX API endpoints
|
||||
DEXSCREENER_API = "https://api.dexscreener.com/latest/dex"
|
||||
|
||||
# Free Solana RPC for account info
|
||||
SOLANA_RPC = "https://api.mainnet-beta.solana.com"
|
||||
|
||||
# Birdeye public API (no key needed for basic queries)
|
||||
BIRDEYE_PUBLIC = "https://public-api.birdeye.so"
|
||||
|
||||
# Known bundler wallet addresses (publicly flagged on-chain)
|
||||
KNOWN_BUNDLER_SEEDS: set[str] = set()
|
||||
|
||||
# ── Risk Levels ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
class BundlerRisk(Enum):
|
||||
CRITICAL = "critical"
|
||||
HIGH = "high"
|
||||
MEDIUM = "medium"
|
||||
LOW = "low"
|
||||
NONE = "none"
|
||||
|
||||
|
||||
# ── Data Models ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class BundledBuy:
|
||||
"""A single suspicious buy event identified as potentially bundled."""
|
||||
|
||||
wallet: str
|
||||
amount_usd: float
|
||||
buy_block: int
|
||||
buy_timestamp: float
|
||||
tx_hash: str = ""
|
||||
funding_source: str = ""
|
||||
is_sniper: bool = False
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"wallet": self.wallet,
|
||||
"amount_usd": round(self.amount_usd, 2),
|
||||
"buy_block": self.buy_block,
|
||||
"buy_timestamp": self.buy_timestamp,
|
||||
"tx_hash": self.tx_hash,
|
||||
"funding_source": self.funding_source,
|
||||
"is_sniper": self.is_sniper,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class HolderCluster:
|
||||
"""A cluster of wallets suspected to be controlled by one entity."""
|
||||
|
||||
wallets: list[str]
|
||||
total_supply_pct: float
|
||||
funding_overlap_score: float # 0-1, how much funding sources overlap
|
||||
buy_time_similarity: float # 0-1, how clustered buys were in time
|
||||
common_funding_source: str = ""
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"wallet_count": len(self.wallets),
|
||||
"wallets": self.wallets[:20], # cap at 20 in output
|
||||
"total_supply_pct": round(self.total_supply_pct, 2),
|
||||
"funding_overlap_score": round(self.funding_overlap_score, 3),
|
||||
"buy_time_similarity": round(self.buy_time_similarity, 3),
|
||||
"common_funding_source": self.common_funding_source,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class BundlerReport:
|
||||
"""Full supply manipulation analysis result."""
|
||||
|
||||
token_address: str
|
||||
chain: str
|
||||
name: str = ""
|
||||
symbol: str = ""
|
||||
|
||||
# Core scores (0-100)
|
||||
bundler_score: float = 0.0
|
||||
supply_concentration_score: float = 0.0
|
||||
sniper_cluster_score: float = 0.0
|
||||
launch_timing_anomaly_score: float = 0.0
|
||||
fund_flow_risk_score: float = 0.0
|
||||
|
||||
# Findings
|
||||
suspected_bundled_buys: list[BundledBuy] = field(default_factory=list)
|
||||
holder_clusters: list[HolderCluster] = field(default_factory=list)
|
||||
top_10_holder_concentration: float = 0.0
|
||||
dev_hold_pct: float = 0.0
|
||||
unique_buyers_first_block: int = 0
|
||||
total_buys_first_blocks: int = 0
|
||||
buys_from_same_funding: int = 0
|
||||
estimated_unique_entities: int = 0
|
||||
|
||||
risk_label: str = "none"
|
||||
errors: list[str] = field(default_factory=list)
|
||||
raw: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"token_address": self.token_address,
|
||||
"chain": self.chain,
|
||||
"name": self.name,
|
||||
"symbol": self.symbol,
|
||||
"bundler_score": round(self.bundler_score, 1),
|
||||
"risk_label": self.risk_label,
|
||||
"signals": {
|
||||
"supply_concentration": round(self.supply_concentration_score, 1),
|
||||
"sniper_cluster": round(self.sniper_cluster_score, 1),
|
||||
"launch_timing_anomaly": round(self.launch_timing_anomaly_score, 1),
|
||||
"fund_flow_risk": round(self.fund_flow_risk_score, 1),
|
||||
},
|
||||
"suspected_bundled_buys": [b.to_dict() for b in self.suspected_bundled_buys[:50]],
|
||||
"holder_clusters": [c.to_dict() for c in self.holder_clusters[:10]],
|
||||
"top_10_holder_concentration": round(self.top_10_holder_concentration, 2),
|
||||
"dev_hold_pct": round(self.dev_hold_pct, 2),
|
||||
"unique_buyers_first_block": self.unique_buyers_first_block,
|
||||
"total_buys_first_blocks": self.total_buys_first_blocks,
|
||||
"buys_from_same_funding": self.buys_from_same_funding,
|
||||
"estimated_unique_entities": self.estimated_unique_entities,
|
||||
}
|
||||
|
||||
def summary(self) -> str:
|
||||
flags = []
|
||||
if self.top_10_holder_concentration > 80:
|
||||
flags.append(f"top10hld:{self.top_10_holder_concentration:.0f}%")
|
||||
if self.buys_from_same_funding > 3:
|
||||
flags.append(f"shared_fund:{self.buys_from_same_funding}x")
|
||||
if self.suspected_bundled_buys:
|
||||
flags.append(f"bundled:{len(self.suspected_bundled_buys)}buys")
|
||||
if self.holder_clusters:
|
||||
total_cluster_pct = sum(c.total_supply_pct for c in self.holder_clusters)
|
||||
flags.append(f"clustered:{total_cluster_pct:.0f}%")
|
||||
flag_str = f" [{', '.join(flags)}]" if flags else ""
|
||||
return (
|
||||
f"[{self.risk_label.upper()}] {self.token_address[:14]}... "
|
||||
f"({self.name}/{self.symbol}) - "
|
||||
f"Bundler score: {self.bundler_score:.0f}/100 | "
|
||||
f"{len(self.holder_clusters)} clusters | "
|
||||
f"{self.estimated_unique_entities} entities estimated"
|
||||
f"{flag_str}"
|
||||
)
|
||||
|
||||
|
||||
# ── Scoring Helpers ──────────────────────────────────────────────
|
||||
|
||||
|
||||
def _gini_coefficient(values: list[float]) -> float:
|
||||
"""Compute Gini coefficient for supply distribution (0=equal, 1=max concentration)."""
|
||||
if not values:
|
||||
return 0.0
|
||||
sorted_vals = sorted(values)
|
||||
n = len(sorted_vals)
|
||||
cumulative = 0.0
|
||||
for i, v in enumerate(sorted_vals):
|
||||
cumulative += (i + 1) * v
|
||||
gini = (2 * cumulative) / (n * sum(sorted_vals)) - (n + 1) / n
|
||||
return max(0.0, min(gini, 1.0))
|
||||
|
||||
|
||||
def _entropy(values: list[float]) -> float:
|
||||
"""Shannon entropy of a distribution (lower = more concentrated).
|
||||
Returns normalized [0, 1] where 1 = perfectly uniform, 0 = fully concentrated.
|
||||
"""
|
||||
total = sum(values)
|
||||
if total <= 0:
|
||||
return 0.0
|
||||
n = len(values)
|
||||
if n <= 1:
|
||||
return 1.0 # Single bin = trivially uniform
|
||||
raw = 0.0
|
||||
for v in values:
|
||||
p = v / total
|
||||
if p > 0:
|
||||
raw -= p * math.log2(p)
|
||||
max_entropy = math.log2(n)
|
||||
return raw / max_entropy if max_entropy > 0 else 0.0
|
||||
|
||||
|
||||
def _time_cluster_similarity(timestamps: list[float]) -> float:
|
||||
"""Score how tightly clustered timestamps are (0=spread, 1=all at once)."""
|
||||
if len(timestamps) < 2:
|
||||
return 0.0
|
||||
min_ts = min(timestamps)
|
||||
max_ts = max(timestamps)
|
||||
span = max_ts - min_ts
|
||||
if span == 0:
|
||||
return 1.0
|
||||
# If all buys happened within 60 seconds, high similarity
|
||||
if span <= 60:
|
||||
return 1.0 - (span / 60) * 0.5 # 0.5-1.0
|
||||
# If within 5 minutes, medium
|
||||
if span <= 300:
|
||||
return 0.5 - (span - 60) / (300 - 60) * 0.3 # 0.2-0.5
|
||||
return max(0.0, 0.2 - (span - 300) / 3600)
|
||||
|
||||
|
||||
def _funding_overlap(funding_sources: list[str]) -> float:
|
||||
"""Score how many wallets share the same funding source (0-1)."""
|
||||
if not funding_sources:
|
||||
return 0.0
|
||||
total = len(funding_sources)
|
||||
if total < 2:
|
||||
return 0.0
|
||||
# Count how many share a source with at least one other
|
||||
from collections import Counter
|
||||
|
||||
source_counts = Counter(funding_sources)
|
||||
shared = sum(c for c in source_counts.values() if c > 1)
|
||||
return shared / total
|
||||
|
||||
|
||||
def _label_risk(score: float) -> str:
|
||||
if score >= 75:
|
||||
return "critical"
|
||||
if score >= 50:
|
||||
return "high"
|
||||
if score >= 25:
|
||||
return "medium"
|
||||
if score > 0:
|
||||
return "low"
|
||||
return "none"
|
||||
|
||||
|
||||
# ── Core Detector ────────────────────────────────────────────────
|
||||
|
||||
|
||||
class BundlerDetector:
|
||||
"""Main detector for bundled/supply-manipulated token launches."""
|
||||
|
||||
def __init__(self, http_timeout: float = 15.0):
|
||||
self.http = httpx.AsyncClient(timeout=http_timeout)
|
||||
self._birdeye_api_key = os.environ.get("BIRDEYE_API_KEY", "")
|
||||
|
||||
async def close(self):
|
||||
await self.http.aclose()
|
||||
|
||||
# ── Public API ──────────────────────────────────────────────
|
||||
|
||||
async def scan(self, address: str, chain: str) -> BundlerReport:
|
||||
"""Full supply manipulation analysis for a token."""
|
||||
if not self._validate_address(address, chain):
|
||||
return BundlerReport(
|
||||
token_address=address,
|
||||
chain=chain,
|
||||
errors=[f"Invalid address format for chain: {chain}"],
|
||||
risk_label="error",
|
||||
)
|
||||
|
||||
chain = chain.lower()
|
||||
if chain not in SUPPORTED_CHAINS:
|
||||
return BundlerReport(
|
||||
token_address=address,
|
||||
chain=chain,
|
||||
errors=[f"Unsupported chain: {chain}"],
|
||||
risk_label="error",
|
||||
)
|
||||
|
||||
report = BundlerReport(token_address=address, chain=chain)
|
||||
|
||||
try:
|
||||
# 1. Fetch token metadata and pair info
|
||||
metadata = await self._fetch_metadata(address, chain)
|
||||
report.name = metadata.get("name", "Unknown")
|
||||
report.symbol = metadata.get("symbol", "UNKNOWN")
|
||||
report.raw["metadata"] = metadata
|
||||
|
||||
# 2. Fetch holder data
|
||||
holders = await self._fetch_holders(address, chain)
|
||||
report.raw["holders_raw"] = holders
|
||||
|
||||
if not holders:
|
||||
report.errors.append("No holder data available")
|
||||
report.risk_label = "error"
|
||||
return report
|
||||
|
||||
# 3. Compute supply concentration
|
||||
top10_pct = self._compute_top_holder_pct(holders, 10)
|
||||
report.top_10_holder_concentration = top10_pct
|
||||
report.dev_hold_pct = self._extract_dev_hold_pct(holders, metadata)
|
||||
|
||||
# 4. Fetch and analyze buys for bundling patterns
|
||||
buys = await self._fetch_buys(address, chain)
|
||||
report.raw["buys_raw"] = buys
|
||||
|
||||
# 5. Detect bundled buys (same funding source, same block)
|
||||
bundled_buys, buys_from_same_funding = self._detect_bundled_buys(buys)
|
||||
report.suspected_bundled_buys = bundled_buys
|
||||
report.buys_from_same_funding = buys_from_same_funding
|
||||
|
||||
# 6. Analyze launch timing
|
||||
timing_info = self._analyze_launch_timing(buys)
|
||||
report.unique_buyers_first_block = timing_info["unique_buyers_first_block"]
|
||||
report.total_buys_first_blocks = timing_info["total_buys_first_blocks"]
|
||||
|
||||
# 7. Cluster wallets by funding source and timing
|
||||
clusters = self._cluster_wallets(buys, holders)
|
||||
report.holder_clusters = clusters
|
||||
|
||||
# 8. Estimate unique entities
|
||||
report.estimated_unique_entities = self._estimate_entities(holders, clusters, len(bundled_buys))
|
||||
|
||||
# 9. Compute all scores
|
||||
report.supply_concentration_score = self._score_supply_concentration(holders, top10_pct)
|
||||
report.sniper_cluster_score = self._score_sniper_clusters(clusters, bundled_buys)
|
||||
report.launch_timing_anomaly_score = self._score_launch_timing(timing_info, buys, holders)
|
||||
report.fund_flow_risk_score = self._score_fund_flow(bundled_buys, buys_from_same_funding, clusters)
|
||||
|
||||
# 10. Composite bundler score
|
||||
report.bundler_score = self._compute_bundler_score(report)
|
||||
report.risk_label = _label_risk(report.bundler_score)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Bundler scan error for {address}: {e}")
|
||||
report.errors.append(str(e))
|
||||
report.risk_label = "error"
|
||||
|
||||
return report
|
||||
|
||||
async def quick_check(self, address: str, chain: str) -> dict[str, Any]:
|
||||
"""Quick supply concentration check - holder data only."""
|
||||
if not self._validate_address(address, chain):
|
||||
return {"error": f"Invalid address for chain {chain}"}
|
||||
|
||||
chain = chain.lower()
|
||||
metadata = await self._fetch_metadata(address, chain)
|
||||
holders = await self._fetch_holders(address, chain)
|
||||
|
||||
if not holders:
|
||||
return {
|
||||
"address": address,
|
||||
"chain": chain,
|
||||
"name": metadata.get("name", ""),
|
||||
"symbol": metadata.get("symbol", ""),
|
||||
"error": "No holder data available",
|
||||
}
|
||||
|
||||
top10 = self._compute_top_holder_pct(holders, 10)
|
||||
gini = _gini_coefficient([h.get("percentage", 0) for h in holders[:100]])
|
||||
|
||||
score = 0.0
|
||||
if top10 > 80:
|
||||
score += 40
|
||||
elif top10 > 60:
|
||||
score += 25
|
||||
if gini > 0.8:
|
||||
score += 30
|
||||
elif gini > 0.6:
|
||||
score += 15
|
||||
|
||||
return {
|
||||
"address": address,
|
||||
"chain": chain,
|
||||
"name": metadata.get("name", ""),
|
||||
"symbol": metadata.get("symbol", ""),
|
||||
"supply_concentration_score": min(score, 100),
|
||||
"risk_label": _label_risk(min(score, 100)),
|
||||
"top_10_holder_pct": round(top10, 2),
|
||||
"gini_coefficient": round(gini, 3),
|
||||
}
|
||||
|
||||
# ── Validation ──────────────────────────────────────────────
|
||||
|
||||
def _validate_address(self, address: str, chain: str) -> bool:
|
||||
chain = chain.lower()
|
||||
if chain == "solana":
|
||||
return bool(SOLANA_ADDR_RE.match(address))
|
||||
if chain in EVM_CHAINS:
|
||||
return bool(EVM_ADDR_RE.match(address))
|
||||
return bool(EVM_ADDR_RE.match(address) or SOLANA_ADDR_RE.match(address))
|
||||
|
||||
# ── Data Fetching ───────────────────────────────────────────
|
||||
|
||||
async def _fetch_metadata(self, address: str, chain: str) -> dict[str, Any]:
|
||||
"""Fetch token metadata from DexScreener."""
|
||||
try:
|
||||
url = f"{DEXSCREENER_API}/tokens/{address}"
|
||||
resp = await self.http.get(url, timeout=10)
|
||||
if resp.status_code != 200:
|
||||
return {}
|
||||
data = resp.json()
|
||||
pairs = data.get("pairs", [])
|
||||
if not pairs:
|
||||
return {}
|
||||
|
||||
pair = pairs[0]
|
||||
return {
|
||||
"name": pair.get("baseToken", {}).get("name", ""),
|
||||
"symbol": pair.get("baseToken", {}).get("symbol", ""),
|
||||
"decimals": pair.get("baseToken", {}).get("decimals"),
|
||||
"price_usd": pair.get("priceUsd", ""),
|
||||
"liquidity_usd": pair.get("liquidity", {}).get("usd", 0),
|
||||
"fdv": pair.get("fdv", 0),
|
||||
"pair_address": pair.get("pairAddress", ""),
|
||||
"dex": pair.get("dexId", ""),
|
||||
"url": pair.get("url", ""),
|
||||
"social": {
|
||||
"twitter": pair.get("info", {}).get("twitter", ""),
|
||||
"website": pair.get("info", {}).get("website", ""),
|
||||
"telegram": pair.get("info", {}).get("telegram", ""),
|
||||
},
|
||||
"creation_block": None, # May not be available
|
||||
}
|
||||
except Exception as e:
|
||||
logger.debug(f"Metadata fetch error: {e}")
|
||||
return {}
|
||||
|
||||
async def _fetch_holders(self, address: str, chain: str) -> list[dict[str, Any]]:
|
||||
"""Fetch top holders from Birdeye public API or Solscan."""
|
||||
try:
|
||||
if chain == "solana":
|
||||
return await self._fetch_solana_holders(address)
|
||||
# EVM chains - try Birdeye first
|
||||
return await self._fetch_evm_holders(address, chain)
|
||||
except Exception as e:
|
||||
logger.debug(f"Holder fetch error: {e}")
|
||||
return []
|
||||
|
||||
async def _fetch_solana_holders(self, address: str) -> list[dict[str, Any]]:
|
||||
"""Fetch Solana token holders via Birdeye public API."""
|
||||
try:
|
||||
url = f"{BIRDEYE_PUBLIC}/defi/holder/tokenlist?tokenAddress={address}&limit=100"
|
||||
headers = {"Accept": "application/json"}
|
||||
if self._birdeye_api_key:
|
||||
headers["X-API-KEY"] = self._birdeye_api_key
|
||||
|
||||
resp = await self.http.get(url, headers=headers, timeout=10)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
items = data.get("data", {}).get("items", [])
|
||||
return [
|
||||
{
|
||||
"address": h.get("holder", ""),
|
||||
"amount": h.get("amount", 0),
|
||||
"percentage": h.get("percent", 0),
|
||||
}
|
||||
for h in items
|
||||
]
|
||||
except Exception as e:
|
||||
logger.debug(f"Solana holder fetch error: {e}")
|
||||
|
||||
# Fallback: Solscan free API
|
||||
try:
|
||||
url = f"https://public-api.solscan.io/token/holders?tokenAddress={address}&limit=100&offset=0"
|
||||
resp = await self.http.get(url, timeout=10)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
items = data if isinstance(data, list) else data.get("data", [])
|
||||
return [
|
||||
{
|
||||
"address": h.get("owner", h.get("address", "")),
|
||||
"amount": h.get("amount", h.get("balance", 0)),
|
||||
"percentage": h.get("percentage", h.get("percent", 0)),
|
||||
}
|
||||
for h in items
|
||||
]
|
||||
except Exception as e:
|
||||
logger.debug(f"Solscan holder fallback error: {e}")
|
||||
|
||||
return []
|
||||
|
||||
async def _fetch_evm_holders(self, address: str, chain: str) -> list[dict[str, Any]]:
|
||||
"""Fetch EVM token holders via Birdeye public API."""
|
||||
try:
|
||||
url = f"{BIRDEYE_PUBLIC}/defi/holder/tokenlist?tokenAddress={address}&limit=100"
|
||||
headers = {"Accept": "application/json"}
|
||||
if self._birdeye_api_key:
|
||||
headers["X-API-KEY"] = self._birdeye_api_key
|
||||
|
||||
resp = await self.http.get(url, headers=headers, timeout=10)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
items = data.get("data", {}).get("items", [])
|
||||
return [
|
||||
{
|
||||
"address": h.get("holder", ""),
|
||||
"amount": h.get("amount", 0),
|
||||
"percentage": h.get("percent", 0),
|
||||
}
|
||||
for h in items
|
||||
]
|
||||
except Exception as e:
|
||||
logger.debug(f"EVM holder fetch error: {e}")
|
||||
|
||||
return []
|
||||
|
||||
async def _fetch_buys(self, address: str, chain: str) -> list[dict[str, Any]]:
|
||||
"""Fetch recent buy transactions for the token."""
|
||||
buys: list[dict[str, Any]] = []
|
||||
try:
|
||||
url = f"{DEXSCREENER_API}/tokens/{address}"
|
||||
resp = await self.http.get(url, timeout=10)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
pairs = data.get("pairs", [])
|
||||
for pair in pairs[:5]: # Check top 5 pairs
|
||||
txns = pair.get("txns", {})
|
||||
# Extract buys from recent transactions
|
||||
m5 = txns.get("m5", {}) or {}
|
||||
h1 = txns.get("h1", {}) or {}
|
||||
h6 = txns.get("h6", {}) or {}
|
||||
buys.append(
|
||||
{
|
||||
"type": "buy",
|
||||
"m5_buys": m5.get("buys", 0),
|
||||
"m5_sells": m5.get("sells", 0),
|
||||
"h1_buys": h1.get("buys", 0),
|
||||
"h1_sells": h1.get("sells", 0),
|
||||
"h6_buys": h6.get("buys", 0),
|
||||
"h6_sells": h6.get("sells", 0),
|
||||
"pair_address": pair.get("pairAddress", ""),
|
||||
"creation_block": None, # May not be available
|
||||
}
|
||||
)
|
||||
|
||||
# Try to get volume per tx for bundling analysis
|
||||
volume_m5 = pair.get("volume", {}).get("m5", 0) or 0
|
||||
if m5.get("buys", 0) > 0:
|
||||
avg_buy = float(volume_m5) / max(1, m5.get("buys", 1))
|
||||
buys[-1]["avg_buy_value"] = avg_buy
|
||||
except Exception as e:
|
||||
logger.debug(f"Buy fetch error: {e}")
|
||||
|
||||
return buys
|
||||
|
||||
# ── Analysis ────────────────────────────────────────────────
|
||||
|
||||
@staticmethod
|
||||
def _compute_top_holder_pct(holders: list[dict[str, Any]], top_n: int) -> float:
|
||||
"""Calculate the percentage of supply held by top N holders."""
|
||||
sorted_h = sorted(holders, key=lambda h: h.get("percentage", 0), reverse=True)
|
||||
top = sorted_h[:top_n]
|
||||
return sum(h.get("percentage", 0) for h in top if h.get("percentage") is not None)
|
||||
|
||||
@staticmethod
|
||||
def _extract_dev_hold_pct(holders: list[dict[str, Any]], metadata: dict[str, Any]) -> float:
|
||||
"""Extract developer/allocation wallet holding percentage."""
|
||||
if not holders:
|
||||
return 0.0
|
||||
return holders[0].get("percentage", 0) if holders else 0.0
|
||||
|
||||
def _detect_bundled_buys(self, buys: list[dict[str, Any]]) -> tuple[list[BundledBuy], int]:
|
||||
"""Detect buys that appear bundled (same source, time clustering)."""
|
||||
bundled: list[BundledBuy] = []
|
||||
same_funding_count = 0
|
||||
|
||||
# From aggregated transaction data, detect anomalous patterns
|
||||
for buy in buys:
|
||||
m5_buys = buy.get("m5_buys", 0)
|
||||
h1_buys = buy.get("h1_buys", 0)
|
||||
|
||||
# If buys/minute in first 5min is very high relative to later
|
||||
if m5_buys > 0 and h1_buys > 0:
|
||||
m5_rate = m5_buys / 5
|
||||
h1_rate = h1_buys / 60
|
||||
if m5_rate > h1_rate * 3 and m5_buys >= 10:
|
||||
# High initial buy concentration - suspicious
|
||||
bundled.append(
|
||||
BundledBuy(
|
||||
wallet=f"cluster:{buy.get('pair_address', '')[:12]}",
|
||||
amount_usd=0, # aggregated
|
||||
buy_block=0,
|
||||
buy_timestamp=time.time(),
|
||||
tx_hash="",
|
||||
funding_source="aggregated",
|
||||
is_sniper=True,
|
||||
)
|
||||
)
|
||||
same_funding_count += m5_buys
|
||||
|
||||
return bundled, same_funding_count
|
||||
|
||||
def _analyze_launch_timing(self, buys: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
"""Analyze launch timing for anomalous patterns."""
|
||||
result = {
|
||||
"unique_buyers_first_block": 0,
|
||||
"total_buys_first_blocks": 0,
|
||||
"buy_concentration_ratio": 0.0,
|
||||
}
|
||||
|
||||
for buy in buys:
|
||||
m5_buys = buy.get("m5_buys", 0)
|
||||
h1_buys = buy.get("h1_buys", 0)
|
||||
h6_buys = buy.get("h6_buys", 0)
|
||||
total = m5_buys + h1_buys + h6_buys
|
||||
|
||||
if total > 0:
|
||||
# What % of all buys happened in first 5 minutes?
|
||||
first_5m_pct = m5_buys / total if total > 0 else 0
|
||||
result["buy_concentration_ratio"] = max(result["buy_concentration_ratio"], first_5m_pct)
|
||||
result["total_buys_first_blocks"] += m5_buys
|
||||
# Estimate unique from m5 vs h1 ratio
|
||||
if h1_buys > 0 and m5_buys > 0:
|
||||
result["unique_buyers_first_block"] = max(
|
||||
result["unique_buyers_first_block"],
|
||||
min(m5_buys, h1_buys), # rough proxy
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
def _cluster_wallets(self, buys: list[dict[str, Any]], holders: list[dict[str, Any]]) -> list[HolderCluster]:
|
||||
"""Cluster wallets by funding overlap and timing patterns."""
|
||||
clusters: list[HolderCluster] = []
|
||||
|
||||
if not holders:
|
||||
return clusters
|
||||
|
||||
# Identify clusters based on supply concentration
|
||||
sorted_h = sorted(holders, key=lambda h: h.get("percentage", 0), reverse=True)
|
||||
|
||||
# If top 3 holders control >60%, they form a natural cluster
|
||||
top3 = sorted_h[:3]
|
||||
top3_pct = sum(h.get("percentage", 0) for h in top3 if h.get("percentage") is not None)
|
||||
if top3_pct > 60 and len(top3) >= 2:
|
||||
clusters.append(
|
||||
HolderCluster(
|
||||
wallets=[h.get("address", "") for h in top3 if h.get("address")],
|
||||
total_supply_pct=top3_pct,
|
||||
funding_overlap_score=0.7 if top3_pct > 80 else 0.5,
|
||||
buy_time_similarity=0.8 if top3_pct > 80 else 0.6,
|
||||
common_funding_source="top_holders_cluster",
|
||||
)
|
||||
)
|
||||
|
||||
# Check for wallet groupings with 5-15% each (typical bundler pattern)
|
||||
cluster_wallets: list[dict[str, Any]] = []
|
||||
cluster_pct = 0.0
|
||||
for h in sorted_h[3:]: # Skip top 3
|
||||
pct = h.get("percentage", 0)
|
||||
if pct and 2 <= pct <= 15:
|
||||
cluster_wallets.append(h)
|
||||
cluster_pct += pct
|
||||
if len(cluster_wallets) >= 5 and cluster_pct >= 15:
|
||||
break
|
||||
|
||||
if len(cluster_wallets) >= 5 and cluster_pct >= 15:
|
||||
clusters.append(
|
||||
HolderCluster(
|
||||
wallets=[h.get("address", "") for h in cluster_wallets],
|
||||
total_supply_pct=cluster_pct,
|
||||
funding_overlap_score=0.6,
|
||||
buy_time_similarity=0.7,
|
||||
common_funding_source="mid_holder_belt",
|
||||
)
|
||||
)
|
||||
|
||||
return clusters
|
||||
|
||||
@staticmethod
|
||||
def _estimate_entities(
|
||||
holders: list[dict[str, Any]],
|
||||
clusters: list[HolderCluster],
|
||||
bundled_buys_count: int,
|
||||
) -> int:
|
||||
"""Estimate number of truly independent entities behind the token."""
|
||||
total_holders = len(holders)
|
||||
|
||||
# Each cluster represents 1 entity instead of N wallets
|
||||
cluster_wallet_count = sum(len(c.wallets) for c in clusters)
|
||||
|
||||
# Reduce estimated entities by clustered wallets
|
||||
entities = max(1, total_holders - cluster_wallet_count)
|
||||
|
||||
# Further reduce if many bundled buys detected
|
||||
if bundled_buys_count > 20:
|
||||
entities = max(1, entities - bundled_buys_count // 5)
|
||||
|
||||
return entities
|
||||
|
||||
# ── Scoring ─────────────────────────────────────────────────
|
||||
|
||||
def _score_supply_concentration(self, holders: list[dict[str, Any]], top10_pct: float) -> float:
|
||||
"""Score supply distribution risk (0-100)."""
|
||||
score = 0.0
|
||||
|
||||
# Top 10 concentration
|
||||
if top10_pct >= 90:
|
||||
score += 50
|
||||
elif top10_pct >= 75:
|
||||
score += 35
|
||||
elif top10_pct >= 50:
|
||||
score += 20
|
||||
elif top10_pct >= 30:
|
||||
score += 10
|
||||
|
||||
# Gini coefficient
|
||||
amounts = [h.get("percentage", 0) for h in holders[:100] if h.get("percentage") is not None]
|
||||
gini = _gini_coefficient(amounts)
|
||||
if gini >= 0.9:
|
||||
score += 40
|
||||
elif gini >= 0.8:
|
||||
score += 30
|
||||
elif gini >= 0.6:
|
||||
score += 15
|
||||
|
||||
# Entropy (low entropy = concentrated)
|
||||
ent = _entropy(amounts)
|
||||
if ent < 0.3:
|
||||
score += 15
|
||||
elif ent < 0.5:
|
||||
score += 8
|
||||
|
||||
return min(score, 100)
|
||||
|
||||
def _score_sniper_clusters(self, clusters: list[HolderCluster], bundled_buys: list[BundledBuy]) -> float:
|
||||
"""Score sniper cluster risk (0-100)."""
|
||||
score = 0.0
|
||||
|
||||
# High-funding-overlap clusters
|
||||
high_overlap = [c for c in clusters if c.funding_overlap_score > 0.6]
|
||||
if high_overlap:
|
||||
total_pct = sum(c.total_supply_pct for c in high_overlap)
|
||||
if total_pct >= 50:
|
||||
score += 50
|
||||
elif total_pct >= 30:
|
||||
score += 35
|
||||
elif total_pct >= 15:
|
||||
score += 20
|
||||
|
||||
# Bundled buys
|
||||
if bundled_buys:
|
||||
score += min(len(bundled_buys) * 5, 30)
|
||||
|
||||
# Time clustering in clusters
|
||||
high_time = [c for c in clusters if c.buy_time_similarity > 0.7]
|
||||
if high_time:
|
||||
score += min(len(high_time) * 10, 25)
|
||||
|
||||
return min(score, 100)
|
||||
|
||||
def _score_launch_timing(
|
||||
self,
|
||||
timing_info: dict[str, Any],
|
||||
buys: list[dict[str, Any]],
|
||||
holders: list[dict[str, Any]],
|
||||
) -> float:
|
||||
"""Score launch timing anomalies (0-100)."""
|
||||
score = 0.0
|
||||
|
||||
# High buy concentration in first 5 minutes
|
||||
ratio = timing_info.get("buy_concentration_ratio", 0)
|
||||
if ratio >= 0.8:
|
||||
score += 50
|
||||
elif ratio >= 0.6:
|
||||
score += 35
|
||||
elif ratio >= 0.4:
|
||||
score += 20
|
||||
|
||||
# Very few unique buyers relative to total buys
|
||||
unique = timing_info.get("unique_buyers_first_block", 0)
|
||||
total = timing_info.get("total_buys_first_blocks", 0)
|
||||
if total > 0 and unique > 0:
|
||||
repeat_rate = total / max(1, unique)
|
||||
if repeat_rate >= 5:
|
||||
score += 30
|
||||
elif repeat_rate >= 3:
|
||||
score += 20
|
||||
|
||||
# Holder count vs buy count mismatch
|
||||
holder_count = len(holders)
|
||||
if holder_count > 0 and total > 0:
|
||||
buys_per_holder = total / holder_count
|
||||
if buys_per_holder >= 3:
|
||||
score += 15
|
||||
|
||||
return min(score, 100)
|
||||
|
||||
def _score_fund_flow(
|
||||
self,
|
||||
bundled_buys: list[BundledBuy],
|
||||
same_funding_count: int,
|
||||
clusters: list[HolderCluster],
|
||||
) -> float:
|
||||
"""Score fund flow risk (0-100)."""
|
||||
score = 0.0
|
||||
|
||||
# Same funding source buys
|
||||
if same_funding_count >= 20:
|
||||
score += 45
|
||||
elif same_funding_count >= 10:
|
||||
score += 30
|
||||
elif same_funding_count >= 5:
|
||||
score += 15
|
||||
|
||||
# Clusters with high funding overlap
|
||||
high_overlap = [c for c in clusters if c.funding_overlap_score > 0.7]
|
||||
if high_overlap:
|
||||
score += min(len(high_overlap) * 15, 30)
|
||||
|
||||
# Overall cluster funding overlap average
|
||||
if clusters:
|
||||
avg_overlap = sum(c.funding_overlap_score for c in clusters) / len(clusters)
|
||||
score += avg_overlap * 20
|
||||
|
||||
return min(score, 100)
|
||||
|
||||
def _compute_bundler_score(self, report: BundlerReport) -> float:
|
||||
"""Weighted composite bundler score."""
|
||||
weights = {
|
||||
"supply_concentration": 0.30,
|
||||
"sniper_cluster": 0.25,
|
||||
"launch_timing_anomaly": 0.20,
|
||||
"fund_flow_risk": 0.25,
|
||||
}
|
||||
|
||||
score = (
|
||||
report.supply_concentration_score * weights["supply_concentration"]
|
||||
+ report.sniper_cluster_score * weights["sniper_cluster"]
|
||||
+ report.launch_timing_anomaly_score * weights["launch_timing_anomaly"]
|
||||
+ report.fund_flow_risk_score * weights["fund_flow_risk"]
|
||||
)
|
||||
|
||||
return min(score, 100)
|
||||
|
|
@ -1,256 +0,0 @@
|
|||
"""
|
||||
Campaign Radar - Coordinated Scam Detection
|
||||
============================================
|
||||
|
||||
Detects coordinated rug pull campaigns across multiple tokens.
|
||||
Clusters tokens by deployer entity, funding source, contract similarity,
|
||||
and social signal correlation.
|
||||
|
||||
Premium feature: "4 tokens detected from same entity - coordinated rug campaign"
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger("sentinel.campaign")
|
||||
|
||||
# In-memory recent scan cache (should be Redis-backed in production)
|
||||
_recent_scans: dict[str, dict[str, Any]] = {} # "chain:address" → scan metadata
|
||||
MAX_RECENT = 500 # Keep last 500 scans for campaign detection
|
||||
|
||||
|
||||
@dataclass
|
||||
class CampaignCluster:
|
||||
"""A detected coordinated campaign."""
|
||||
|
||||
cluster_id: str
|
||||
tokens: list[dict[str, Any]] = field(default_factory=list)
|
||||
deployer_entity: str | None = None
|
||||
funding_source: str | None = None
|
||||
contract_similarity: float = 0.0 # 0-1
|
||||
social_correlation: float = 0.0 # 0-1
|
||||
risk_level: str = "unknown" # "critical"/"high"/"medium"
|
||||
estimated_victims: int = 0
|
||||
first_detected: str | None = None
|
||||
description: str = ""
|
||||
|
||||
|
||||
def record_scan(chain: str, address: str, metadata: dict[str, Any]):
|
||||
"""Record a scan for campaign correlation."""
|
||||
key = f"{chain}:{address.lower()}"
|
||||
metadata["_recorded_at"] = (
|
||||
asyncio.get_event_loop().time() if asyncio.get_event_loop().is_running() else __import__("time").time()
|
||||
)
|
||||
_recent_scans[key] = metadata
|
||||
|
||||
# Evict oldest if over capacity
|
||||
if len(_recent_scans) > MAX_RECENT:
|
||||
oldest = min(_recent_scans.keys(), key=lambda k: _recent_scans[k].get("_recorded_at", 0))
|
||||
del _recent_scans[oldest]
|
||||
|
||||
|
||||
def detect_campaigns(min_cluster_size: int = 3) -> list[CampaignCluster]:
|
||||
"""Analyze recent scans for coordinated campaigns.
|
||||
|
||||
Clusters tokens by:
|
||||
1. Same deployer entity (strongest signal)
|
||||
2. Same funding source
|
||||
3. High contract bytecode similarity
|
||||
4. Correlated social/KOL mentions
|
||||
"""
|
||||
if len(_recent_scans) < min_cluster_size:
|
||||
return []
|
||||
|
||||
scans = list(_recent_scans.values())
|
||||
campaigns = []
|
||||
|
||||
# ── Strategy 1: Same deployer entity ──
|
||||
deployer_groups = defaultdict(list)
|
||||
for scan in scans:
|
||||
deployer = _extract_deployer_entity(scan)
|
||||
if deployer:
|
||||
deployer_groups[deployer].append(scan)
|
||||
|
||||
for entity, group in deployer_groups.items():
|
||||
if len(group) >= min_cluster_size:
|
||||
campaign = CampaignCluster(
|
||||
cluster_id=f"deployer_{entity[:12]}",
|
||||
tokens=[_token_summary(s) for s in group],
|
||||
deployer_entity=entity,
|
||||
risk_level="critical" if len(group) >= 5 else "high",
|
||||
estimated_victims=sum(s.get("holder_count", 0) or 0 for s in group),
|
||||
description=f"{len(group)} tokens launched by same deployer entity {entity[:8]}...",
|
||||
)
|
||||
campaigns.append(campaign)
|
||||
|
||||
# ── Strategy 2: Same funding source ──
|
||||
funding_groups = defaultdict(list)
|
||||
for scan in scans:
|
||||
funder = _extract_funding_source(scan)
|
||||
if funder:
|
||||
funding_groups[funder].append(scan)
|
||||
|
||||
for funder, group in funding_groups.items():
|
||||
if len(group) >= min_cluster_size:
|
||||
# Avoid double-counting with deployer groups
|
||||
existing_tokens = set()
|
||||
for c in campaigns:
|
||||
for t in c.tokens:
|
||||
existing_tokens.add(f"{t.get('chain', '')}:{t.get('address', '')}")
|
||||
|
||||
new_tokens = [
|
||||
s for s in group if f"{s.get('chain', '')}:{s.get('address', '')}".lower() not in existing_tokens
|
||||
]
|
||||
if len(new_tokens) >= min_cluster_size:
|
||||
campaign = CampaignCluster(
|
||||
cluster_id=f"funder_{funder[:12]}",
|
||||
tokens=[_token_summary(s) for s in new_tokens],
|
||||
funding_source=funder,
|
||||
risk_level="high",
|
||||
estimated_victims=sum(s.get("holder_count", 0) or 0 for s in new_tokens),
|
||||
description=f"{len(new_tokens)} tokens funded from same source {funder[:8]}...",
|
||||
)
|
||||
campaigns.append(campaign)
|
||||
|
||||
# ── Strategy 3: Contract similarity ──
|
||||
similar_pairs = []
|
||||
scan_list = list(_recent_scans.values())
|
||||
for i in range(len(scan_list)):
|
||||
for j in range(i + 1, len(scan_list)):
|
||||
sim = _contract_similarity(scan_list[i], scan_list[j])
|
||||
if sim > 0.85:
|
||||
similar_pairs.append((scan_list[i], scan_list[j], sim))
|
||||
|
||||
if similar_pairs:
|
||||
# Union-find to cluster similar contracts
|
||||
clusters = _cluster_similar(similar_pairs)
|
||||
for cluster_tokens in clusters:
|
||||
if len(cluster_tokens) >= min_cluster_size:
|
||||
avg_sim = sum(p[2] for p in similar_pairs if p[0] in cluster_tokens and p[1] in cluster_tokens) / max(
|
||||
len(cluster_tokens), 1
|
||||
)
|
||||
campaign = CampaignCluster(
|
||||
cluster_id=f"contract_{hashlib.sha256(str(sorted([t.get('address', '') for t in cluster_tokens])).encode()).hexdigest()[:12]}",
|
||||
tokens=[_token_summary(s) for s in cluster_tokens],
|
||||
contract_similarity=avg_sim,
|
||||
risk_level="high" if avg_sim > 0.95 else "medium",
|
||||
estimated_victims=sum(s.get("holder_count", 0) or 0 for s in cluster_tokens),
|
||||
description=f"{len(cluster_tokens)} tokens with {avg_sim:.0%} contract similarity - likely cloned scam contracts",
|
||||
)
|
||||
campaigns.append(campaign)
|
||||
|
||||
return sorted(campaigns, key=lambda c: -len(c.tokens))
|
||||
|
||||
|
||||
def _extract_deployer_entity(scan: dict) -> str | None:
|
||||
"""Extract deployer entity ID from scan metadata."""
|
||||
free = scan.get("free", scan)
|
||||
deployer = free.get("deployer", {}) or {}
|
||||
deep = free.get("deep_deployer", {}) or {}
|
||||
|
||||
entity_id = deployer.get("entity_id") or deep.get("entity_id") or deployer.get("address")
|
||||
return entity_id
|
||||
|
||||
|
||||
def _extract_funding_source(scan: dict) -> str | None:
|
||||
"""Extract funding source from scan metadata."""
|
||||
free = scan.get("free", scan)
|
||||
funding = free.get("funding_source") or free.get("deep_deployer", {}).get("funding_source")
|
||||
return funding
|
||||
|
||||
|
||||
def _contract_similarity(scan_a: dict, scan_b: dict) -> float:
|
||||
"""Estimate contract similarity between two scans."""
|
||||
free_a = scan_a.get("free", scan_a)
|
||||
free_b = scan_b.get("free", scan_b)
|
||||
|
||||
# Bytecode hash match (strongest)
|
||||
bc_a = free_a.get("bytecode_hash") or free_a.get("contract_diff", {}).get("bytecode_hash")
|
||||
bc_b = free_b.get("bytecode_hash") or free_b.get("contract_diff", {}).get("bytecode_hash")
|
||||
if bc_a and bc_b and bc_a == bc_b:
|
||||
return 1.0
|
||||
|
||||
# Selector set Jaccard similarity
|
||||
selectors_a = set(free_a.get("selectors", []) or [])
|
||||
selectors_b = set(free_b.get("selectors", []) or [])
|
||||
if selectors_a and selectors_b:
|
||||
intersection = selectors_a & selectors_b
|
||||
union = selectors_a | selectors_b
|
||||
if union:
|
||||
return len(intersection) / len(union)
|
||||
|
||||
return 0.0
|
||||
|
||||
|
||||
def _cluster_similar(pairs: list[tuple]) -> list[list]:
|
||||
"""Union-find clustering of similar contract pairs."""
|
||||
parent = {}
|
||||
|
||||
def find(x):
|
||||
addr = x.get("address", id(x))
|
||||
if addr not in parent:
|
||||
parent[addr] = addr
|
||||
if parent[addr] != addr:
|
||||
parent[addr] = find({"address": parent[addr]})
|
||||
return parent[addr]
|
||||
|
||||
def union(a, b):
|
||||
ra, rb = find(a), find(b)
|
||||
if ra != rb:
|
||||
parent[ra] = rb
|
||||
|
||||
for a, b, _ in pairs:
|
||||
union(a, b)
|
||||
|
||||
clusters = defaultdict(list)
|
||||
for a, b, _ in pairs:
|
||||
root = find(a)
|
||||
if a not in clusters[root]:
|
||||
clusters[root].append(a)
|
||||
if b not in clusters[root]:
|
||||
clusters[root].append(b)
|
||||
|
||||
return list(clusters.values())
|
||||
|
||||
|
||||
def _token_summary(scan: dict) -> dict[str, Any]:
|
||||
"""Create a concise token summary for campaign display."""
|
||||
return {
|
||||
"address": scan.get("address") or scan.get("token_address", ""),
|
||||
"chain": scan.get("chain", ""),
|
||||
"symbol": scan.get("symbol", ""),
|
||||
"name": scan.get("name", ""),
|
||||
"safety_score": scan.get("safety_score", 50),
|
||||
"age_hours": scan.get("free", {}).get("age_hours", 0) if isinstance(scan.get("free"), dict) else 0,
|
||||
"holder_count": scan.get("free", {}).get("holders", {}).get("total", 0)
|
||||
if isinstance(scan.get("free", {}).get("holders"), dict)
|
||||
else 0,
|
||||
}
|
||||
|
||||
|
||||
def get_active_campaigns() -> dict[str, Any]:
|
||||
"""Get all currently detected campaigns."""
|
||||
campaigns = detect_campaigns()
|
||||
return {
|
||||
"status": "ok",
|
||||
"active_campaigns": len(campaigns),
|
||||
"scans_analyzed": len(_recent_scans),
|
||||
"campaigns": [
|
||||
{
|
||||
"id": c.cluster_id,
|
||||
"token_count": len(c.tokens),
|
||||
"deployer_entity": c.deployer_entity,
|
||||
"funding_source": c.funding_source,
|
||||
"contract_similarity": round(c.contract_similarity, 3),
|
||||
"risk_level": c.risk_level,
|
||||
"estimated_victims": c.estimated_victims,
|
||||
"description": c.description,
|
||||
"tokens": c.tokens[:10], # Top 10 tokens
|
||||
}
|
||||
for c in campaigns
|
||||
],
|
||||
}
|
||||
|
|
@ -1,432 +0,0 @@
|
|||
"""
|
||||
cases_investigators_api.py - Public case files + investigator profiles
|
||||
|
||||
Endpoints:
|
||||
POST /api/v1/cases - create/save a case snapshot
|
||||
GET /api/v1/cases/:id - fetch a case by id (public)
|
||||
GET /api/v1/cases - list cases by creator (auth)
|
||||
DELETE /api/v1/cases/:id - delete a case (creator only)
|
||||
GET /api/v1/investigators/:username - public investigator profile
|
||||
GET /api/v1/portfolio/features - returns tier-gated portfolio limits for current user
|
||||
POST /api/v1/subscription/addon - add an add-on to an existing subscription
|
||||
|
||||
Designed to work with localStorage fallback on the frontend (cases are
|
||||
still public-shareable URLs even before this backend is wired).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import secrets
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(tags=["cases-investigators"])
|
||||
|
||||
|
||||
# ── Storage ───────────────────────────────────────────────────
|
||||
# Cases are stored in Redis if available, else in-memory dict.
|
||||
# In production this should be Postgres/ClickHouse, but for the
|
||||
# public-share URL contract, this gives full functionality end-to-end.
|
||||
|
||||
_cases_db: dict[str, dict] = {}
|
||||
_investigators_db: dict[str, dict] = {}
|
||||
|
||||
try:
|
||||
import redis as _redis # type: ignore
|
||||
|
||||
_r = _redis.Redis(
|
||||
host=os.getenv("REDIS_HOST", "localhost"),
|
||||
port=int(os.getenv("REDIS_PORT", "6379")),
|
||||
db=0,
|
||||
decode_responses=True,
|
||||
socket_timeout=2,
|
||||
)
|
||||
_r.ping()
|
||||
REDIS_OK = True
|
||||
except Exception as e:
|
||||
REDIS_OK = False
|
||||
logger.warning(f"cases: redis unavailable, using in-memory store ({e})")
|
||||
|
||||
|
||||
def _store_case(case_id: str, data: dict) -> None:
|
||||
if REDIS_OK:
|
||||
_r.setex(f"rmi:case:{case_id}", 60 * 60 * 24 * 365, json.dumps(data))
|
||||
_cases_db[case_id] = data
|
||||
|
||||
|
||||
def _load_case(case_id: str) -> dict | None:
|
||||
if REDIS_OK:
|
||||
raw = _r.get(f"rmi:case:{case_id}")
|
||||
if raw:
|
||||
return json.loads(raw)
|
||||
return _cases_db.get(case_id)
|
||||
|
||||
|
||||
def _delete_case(case_id: str) -> bool:
|
||||
deleted = False
|
||||
if REDIS_OK:
|
||||
deleted = bool(_r.delete(f"rmi:case:{case_id}"))
|
||||
if case_id in _cases_db:
|
||||
del _cases_db[case_id]
|
||||
deleted = True
|
||||
return deleted
|
||||
|
||||
|
||||
def _store_investigator(handle: str, data: dict) -> None:
|
||||
if REDIS_OK:
|
||||
_r.setex(f"rmi:investigator:{handle}", 60 * 60 * 24, json.dumps(data))
|
||||
_investigators_db[handle] = data
|
||||
|
||||
|
||||
def _load_investigator(handle: str) -> dict | None:
|
||||
if REDIS_OK:
|
||||
raw = _r.get(f"rmi:investigator:{handle}")
|
||||
if raw:
|
||||
return json.loads(raw)
|
||||
return _investigators_db.get(handle)
|
||||
|
||||
|
||||
# ── Models ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class CaseWallet(BaseModel):
|
||||
address: str
|
||||
label: str | None = None
|
||||
pct: float | None = None
|
||||
|
||||
|
||||
class CaseCreateRequest(BaseModel):
|
||||
chain: str
|
||||
token_address: str
|
||||
title: str
|
||||
risk_score: float = Field(ge=0, le=100)
|
||||
risk_level: str = "medium"
|
||||
description: str | None = None
|
||||
key_finding: str | None = None
|
||||
findings: list[str] = []
|
||||
evidence_wallets: list[CaseWallet] = []
|
||||
tx_hashes: list[str] = []
|
||||
bubble_svg: str | None = None
|
||||
|
||||
|
||||
class AddonRequest(BaseModel):
|
||||
addon: str = Field(..., pattern="^(PORTFOLIO_PRO)$")
|
||||
period: str = Field(..., pattern="^(monthly|six_month|yearly)$")
|
||||
payment_method: str = Field("x402", pattern="^(stripe|crypto|x402)$")
|
||||
crypto_chain: str | None = None
|
||||
|
||||
|
||||
# ── Cases ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/cases")
|
||||
async def create_case(req: CaseCreateRequest, request: Request):
|
||||
"""Create a public case file. Returns the case id + public URL."""
|
||||
# Resolve creator if authed
|
||||
creator = "anonymous"
|
||||
try:
|
||||
from app.auth import get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
if user:
|
||||
creator = (
|
||||
user.get("user_metadata", {}).get("handle")
|
||||
or user.get("email", "").split("@")[0]
|
||||
or user.get("id", "anonymous")
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Derive deterministic id from payload
|
||||
payload = {
|
||||
"chain": req.chain,
|
||||
"address": req.token_address,
|
||||
"risk_score": req.risk_score,
|
||||
"findings": req.findings,
|
||||
"wallets": [w.model_dump() for w in req.evidence_wallets],
|
||||
"txs": req.tx_hashes,
|
||||
"title": req.title,
|
||||
}
|
||||
payload_sorted = json.dumps(payload, sort_keys=True)
|
||||
full_hash = hashlib.sha256(payload_sorted.encode()).hexdigest()
|
||||
case_id = f"rmi-{full_hash[:12]}"
|
||||
|
||||
full = {
|
||||
"id": case_id,
|
||||
"created_at": datetime.utcnow().isoformat() + "Z",
|
||||
"creator": creator,
|
||||
"chain": req.chain,
|
||||
"token_address": req.token_address,
|
||||
"title": req.title,
|
||||
"description": req.description,
|
||||
"risk_score": req.risk_score,
|
||||
"risk_level": req.risk_level,
|
||||
"key_finding": req.key_finding,
|
||||
"findings": req.findings,
|
||||
"evidence_wallets": [w.model_dump() for w in req.evidence_wallets],
|
||||
"tx_hashes": req.tx_hashes,
|
||||
"bubble_svg": req.bubble_svg,
|
||||
"snapshot_hash": full_hash,
|
||||
"url": f"https://rugmunch.io/case/{case_id}",
|
||||
}
|
||||
_store_case(case_id, full)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"case": full,
|
||||
"public_url": full["url"],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/cases/{case_id}")
|
||||
async def get_case(case_id: str):
|
||||
"""Public, no auth required. Returns 404 if not found."""
|
||||
data = _load_case(case_id)
|
||||
if not data:
|
||||
raise HTTPException(status_code=404, detail=f"Case {case_id} not found")
|
||||
return data
|
||||
|
||||
|
||||
@router.get("/cases")
|
||||
async def list_my_cases(request: Request, limit: int = 50):
|
||||
"""List cases created by the current user. Auth required."""
|
||||
try:
|
||||
from app.auth import get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
except Exception:
|
||||
user = None
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
creator = user.get("user_metadata", {}).get("handle") or user.get("email", "").split("@")[0] or user.get("id")
|
||||
matches = [c for c in _cases_db.values() if c.get("creator") == creator]
|
||||
matches.sort(key=lambda c: c.get("created_at", ""), reverse=True)
|
||||
return {"cases": matches[:limit], "total": len(matches)}
|
||||
|
||||
|
||||
@router.delete("/cases/{case_id}")
|
||||
async def delete_case(case_id: str, request: Request):
|
||||
"""Delete a case. Only the creator (or anon if created anonymously) can delete."""
|
||||
data = _load_case(case_id)
|
||||
if not data:
|
||||
raise HTTPException(status_code=404, detail="Case not found")
|
||||
try:
|
||||
from app.auth import get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
except Exception:
|
||||
user = None
|
||||
creator = None
|
||||
if user:
|
||||
creator = user.get("user_metadata", {}).get("handle") or user.get("email", "").split("@")[0] or user.get("id")
|
||||
if data.get("creator") != "anonymous" and data.get("creator") != creator:
|
||||
raise HTTPException(status_code=403, detail="Not your case")
|
||||
_delete_case(case_id)
|
||||
return {"success": True}
|
||||
|
||||
|
||||
# ── Investigator profiles ─────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/investigators/{username}")
|
||||
async def get_investigator(username: str):
|
||||
"""Public investigator profile. Returns synthesized profile if no record."""
|
||||
handle = username.lower()
|
||||
data = _load_investigator(handle)
|
||||
if data:
|
||||
return data
|
||||
# Synthesize a default profile so the /u/:username page always renders
|
||||
return {
|
||||
"username": handle,
|
||||
"display_name": handle.replace("_", " ").replace("-", " ").title() if handle != "anonymous" else "Anonymous",
|
||||
"bio": f"RMI investigator @{handle}",
|
||||
"joined_at": (datetime.utcnow() - timedelta(days=90)).isoformat() + "Z",
|
||||
"reputation": 0,
|
||||
"cases_filed": 0,
|
||||
"scans_run": 0,
|
||||
"accuracy": 0,
|
||||
"badges": [],
|
||||
}
|
||||
|
||||
|
||||
@router.post("/investigators/{username}")
|
||||
async def upsert_investigator(username: str, request: Request):
|
||||
"""Update or create investigator profile. Auth required."""
|
||||
try:
|
||||
from app.auth import get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
except Exception:
|
||||
user = None
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
body = await request.json()
|
||||
handle = username.lower()
|
||||
data = {
|
||||
"username": handle,
|
||||
"display_name": body.get("display_name", handle),
|
||||
"bio": body.get("bio", ""),
|
||||
"avatar_url": body.get("avatar_url"),
|
||||
"twitter": body.get("twitter"),
|
||||
"github": body.get("github"),
|
||||
"website": body.get("website"),
|
||||
"telegram": body.get("telegram"),
|
||||
"reputation": body.get("reputation", 0),
|
||||
"cases_filed": body.get("cases_filed", 0),
|
||||
"scans_run": body.get("scans_run", 0),
|
||||
"accuracy": body.get("accuracy", 0),
|
||||
"badges": body.get("badges", []),
|
||||
"updated_at": datetime.utcnow().isoformat() + "Z",
|
||||
}
|
||||
_store_investigator(handle, data)
|
||||
return {"success": True, "investigator": data}
|
||||
|
||||
|
||||
# ── Portfolio features (tier-gated limits) ───────────────────
|
||||
|
||||
_FREE_LIMITS = {"wallets": 3, "history_days": 7, "csv_export": False, "alerts": False}
|
||||
_PORTFOLIO_PRO_LIMITS = {"wallets": 50, "history_days": 30, "csv_export": True, "alerts": True}
|
||||
|
||||
|
||||
@router.get("/portfolio/features")
|
||||
async def portfolio_features(request: Request):
|
||||
"""Return tier-gated Portfolio features for the current user.
|
||||
|
||||
Resolves the user's effective tier by checking:
|
||||
1. Supabase auth → user metadata.has_portfolio_pro
|
||||
2. Their active subscription (if any)
|
||||
3. Falls back to FREE limits
|
||||
|
||||
Frontend uses this to gate wallets, history window, and exports.
|
||||
"""
|
||||
is_pro = False
|
||||
has_subscription = False
|
||||
base_tier = "FREE"
|
||||
|
||||
try:
|
||||
from app.auth import get_current_user
|
||||
|
||||
user = await get_current_user(request)
|
||||
except Exception:
|
||||
user = None
|
||||
|
||||
if user:
|
||||
# Check user metadata flag
|
||||
if user.get("user_metadata", {}).get("has_portfolio_pro"):
|
||||
is_pro = True
|
||||
# Check subscription entitlements (Redis or DB)
|
||||
# Convention: rmi:sub:{user_id} → JSON with {tier, addons: [PORTFOLIO_PRO], ...}
|
||||
try:
|
||||
if REDIS_OK:
|
||||
sub_raw = _r.get(f"rmi:sub:{user.get('id')}")
|
||||
if sub_raw:
|
||||
sub = json.loads(sub_raw)
|
||||
base_tier = sub.get("tier", "FREE")
|
||||
has_subscription = True
|
||||
if "PORTFOLIO_PRO" in sub.get("addons", []):
|
||||
is_pro = True
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
limits = _PORTFOLIO_PRO_LIMITS if is_pro else _FREE_LIMITS
|
||||
return {
|
||||
"tier": "PORTFOLIO_PRO" if is_pro else base_tier,
|
||||
"is_portfolio_pro": is_pro,
|
||||
"has_subscription": has_subscription,
|
||||
"limits": limits,
|
||||
"price_usd": 3.00,
|
||||
"price_atoms": "3000000",
|
||||
"x402_chain_id": 8453,
|
||||
"x402_pay_to": os.getenv("X402_PAY_TO", "0x1E3AC01d0fdb976179790BDD02823196A92705C9"),
|
||||
"upgrade_url": "/pricing?tier=PORTFOLIO_PRO",
|
||||
}
|
||||
|
||||
|
||||
# ── Add-on subscription ──────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/subscription/addon")
|
||||
async def add_addon(req: AddonRequest, request: Request):
|
||||
"""Add an add-on (e.g. PORTFOLIO_PRO) to an existing subscription.
|
||||
|
||||
Returns an x402 challenge for micropayment, or a stripe checkout URL
|
||||
for fiat, or a crypto payment address.
|
||||
"""
|
||||
from app.auth import get_current_user
|
||||
|
||||
try:
|
||||
user = await get_current_user(request)
|
||||
except Exception:
|
||||
user = None
|
||||
if not user:
|
||||
raise HTTPException(status_code=401, detail="Authentication required")
|
||||
|
||||
addon = req.addon.upper()
|
||||
if addon == "PORTFOLIO_PRO":
|
||||
price_usd = 3.00
|
||||
x402_price_atoms = "3000000"
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail=f"Unknown addon {addon}")
|
||||
|
||||
addon_id = secrets.token_hex(8)
|
||||
now = datetime.utcnow()
|
||||
|
||||
if req.payment_method == "x402":
|
||||
# Return EIP-3009 challenge
|
||||
return {
|
||||
"success": True,
|
||||
"addon": addon,
|
||||
"addon_id": addon_id,
|
||||
"user_id": user["id"],
|
||||
"x402": {
|
||||
"version": 2,
|
||||
"scheme": "exact",
|
||||
"network": "eip155:8453",
|
||||
"asset": "USDC",
|
||||
"amount_atoms": x402_price_atoms,
|
||||
"amount_usd": price_usd,
|
||||
"pay_to": os.getenv("X402_PAY_TO", "0x1E3AC01d0fdb976179790BDD02823196A92705C9"),
|
||||
"expires_at": (now + timedelta(minutes=10)).isoformat() + "Z",
|
||||
},
|
||||
"instructions": f"Sign EIP-3009 transfer for ${price_usd} USDC. Addon activates on settlement.",
|
||||
}
|
||||
|
||||
if req.payment_method == "crypto":
|
||||
return {
|
||||
"success": True,
|
||||
"addon": addon,
|
||||
"addon_id": addon_id,
|
||||
"user_id": user["id"],
|
||||
"payment_details": {
|
||||
"chain": req.crypto_chain,
|
||||
"amount_usd": price_usd,
|
||||
"memo": f"RMI-ADDON-{addon_id}",
|
||||
},
|
||||
}
|
||||
|
||||
# Stripe
|
||||
return {
|
||||
"success": True,
|
||||
"addon": addon,
|
||||
"addon_id": addon_id,
|
||||
"user_id": user["id"],
|
||||
"checkout_url": f"/pricing?addon={addon}&period={req.period}",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def health():
|
||||
return {
|
||||
"status": "ok",
|
||||
"redis": REDIS_OK,
|
||||
"cases_stored": len(_cases_db),
|
||||
"investigators_stored": len(_investigators_db),
|
||||
}
|
||||
|
|
@ -1,101 +0,0 @@
|
|||
"""
|
||||
Chain Data Feeder - Pre-loads wallet clustering engine with real on-chain data.
|
||||
Uses Helius (primary) with QuickNode fallback. Rate-limited + cached.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from app.chain_cache import get_chain_cache
|
||||
from app.chain_client import get_chain_client
|
||||
from app.wallet_clustering import Transaction, get_clustering_engine
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def feed_wallet_transactions(wallet: str, limit: int = 50) -> int:
|
||||
"""Fetch real transactions for a wallet and load into clustering engine.
|
||||
Returns number of transactions loaded. Cached for 5 min."""
|
||||
cache = get_chain_cache()
|
||||
cached = await cache.get("feed_wallet", wallet, limit)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
client = get_chain_client()
|
||||
engine = get_clustering_engine()
|
||||
|
||||
# Get recent signatures
|
||||
sigs = await cache.get("signatures", wallet, limit)
|
||||
if sigs is None:
|
||||
sigs = await client.get_signatures(wallet, limit=min(limit, 50))
|
||||
await cache.set("signatures", sigs, wallet, limit, ttl=120)
|
||||
|
||||
if not sigs:
|
||||
await cache.set("feed_wallet", 0, wallet, limit)
|
||||
return 0
|
||||
|
||||
# Get parsed transactions (batch)
|
||||
tx_hashes = [s.get("signature") for s in sigs[:25] if s.get("signature")]
|
||||
txs = await cache.get("transactions_batch", *tx_hashes[:5])
|
||||
if txs is None:
|
||||
txs = await client.get_transactions(tx_hashes[:25])
|
||||
await cache.set("transactions_batch", txs, *tx_hashes[:5], ttl=120)
|
||||
|
||||
count = 0
|
||||
for sig_data in sigs:
|
||||
sig = sig_data.get("signature")
|
||||
if not sig:
|
||||
continue
|
||||
ts = sig_data.get("blockTime")
|
||||
dt = datetime.fromtimestamp(ts, tz=UTC) if ts else datetime.now(UTC)
|
||||
|
||||
# Simple transaction from signature data
|
||||
tx = Transaction(
|
||||
signature=sig,
|
||||
timestamp=dt,
|
||||
from_address=wallet,
|
||||
to_address=sig_data.get("to", "unknown"),
|
||||
amount=float(sig_data.get("lamport", 0)) / 1e9 if sig_data.get("lamport") else 0,
|
||||
token="SOL",
|
||||
program="system",
|
||||
)
|
||||
engine.add_transaction(tx)
|
||||
count += 1
|
||||
|
||||
await cache.set("feed_wallet", count, wallet, limit)
|
||||
logger.info(f"Fed {count} transactions for {wallet[:12]}...")
|
||||
return count
|
||||
|
||||
|
||||
async def get_holders_for_token(token_address: str, limit: int = 20) -> list[str]:
|
||||
"""Get top token holders. Uses cached Helius token-accounts lookup."""
|
||||
cache = get_chain_cache()
|
||||
cached = await cache.get("holders", token_address, limit)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
client = get_chain_client()
|
||||
accounts = await client.get_token_accounts(token_address)
|
||||
|
||||
holders = []
|
||||
for acct in accounts[:limit]:
|
||||
info = acct.get("account", {}).get("data", {}).get("parsed", {}).get("info", {})
|
||||
owner = info.get("owner")
|
||||
if owner:
|
||||
holders.append(owner)
|
||||
|
||||
await cache.set("holders", holders, token_address, limit, ttl=300)
|
||||
return holders
|
||||
|
||||
|
||||
async def get_wallet_balance(wallet: str) -> float:
|
||||
"""Get SOL balance with caching."""
|
||||
cache = get_chain_cache()
|
||||
cached = await cache.get("balance", wallet)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
client = get_chain_client()
|
||||
balance = await client.get_balance(wallet)
|
||||
await cache.set("balance", balance, wallet, ttl=300)
|
||||
return balance
|
||||
|
|
@ -1,66 +0,0 @@
|
|||
"""
|
||||
Lightweight Redis-backed Circuit Breaker for external API calls.
|
||||
Prevents cascading failures when external services (Helius, OpenRouter, etc.) go down.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
|
||||
import redis
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _get_redis() -> redis.Redis:
|
||||
return redis.Redis(
|
||||
host=os.getenv("REDIS_HOST", "localhost"),
|
||||
port=int(os.getenv("REDIS_PORT", "6379")),
|
||||
password=os.getenv("REDIS_PASSWORD", ""),
|
||||
decode_responses=True,
|
||||
socket_timeout=2,
|
||||
)
|
||||
|
||||
|
||||
async def check_circuit(service: str) -> bool:
|
||||
"""
|
||||
Returns True if circuit is OPEN (do not call the service).
|
||||
Returns False if CLOSED or HALF_OPEN (safe to call).
|
||||
"""
|
||||
try:
|
||||
r = _get_redis()
|
||||
state = r.get(f"circuit:{service}:state")
|
||||
|
||||
if state == "OPEN":
|
||||
opened_at = float(r.get(f"circuit:{service}:opened_at") or 0)
|
||||
if time.time() - opened_at > 60: # 60 seconds cooldown
|
||||
r.set(f"circuit:{service}:state", "HALF_OPEN")
|
||||
logger.info(f"Circuit breaker for {service} moving to HALF_OPEN")
|
||||
return False
|
||||
return True # Still OPEN, block the call
|
||||
|
||||
return False # CLOSED or HALF_OPEN
|
||||
except Exception as e:
|
||||
logger.error(f"Circuit breaker check failed for {service}: {e}")
|
||||
return False # Fail open if Redis is down
|
||||
|
||||
|
||||
async def record_success(service: str):
|
||||
try:
|
||||
r = _get_redis()
|
||||
r.set(f"circuit:{service}:state", "CLOSED")
|
||||
r.delete(f"circuit:{service}:failures")
|
||||
except Exception as e:
|
||||
logger.error(f"Circuit breaker success record failed for {service}: {e}")
|
||||
|
||||
|
||||
async def record_failure(service: str):
|
||||
try:
|
||||
r = _get_redis()
|
||||
failures = r.incr(f"circuit:{service}:failures")
|
||||
if failures >= 5:
|
||||
r.set(f"circuit:{service}:state", "OPEN")
|
||||
r.set(f"circuit:{service}:opened_at", str(time.time()))
|
||||
logger.warning(f"Circuit breaker for {service} TRIPPED (OPEN) after 5 failures")
|
||||
except Exception as e:
|
||||
logger.error(f"Circuit breaker failure record failed for {service}: {e}")
|
||||
|
|
@ -1,95 +0,0 @@
|
|||
"""
|
||||
Community Badges Router - FastAPI endpoints for voting and badge queries.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.community_badges import (
|
||||
cast_vote,
|
||||
get_leaderboard,
|
||||
get_token_votes,
|
||||
get_user_badge,
|
||||
resolve_token_outcome,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/api/v1/community", tags=["community"])
|
||||
|
||||
|
||||
# ── Request models ────────────────────────────────────────────────
|
||||
|
||||
|
||||
class VoteRequest(BaseModel):
|
||||
token_address: str = Field(..., min_length=32, max_length=128)
|
||||
chain: str = Field(default="solana")
|
||||
voter_id: str = Field(..., min_length=1, max_length=256)
|
||||
vote: str = Field(..., pattern=r"^(clean|malicious)$")
|
||||
|
||||
|
||||
class ResolveRequest(BaseModel):
|
||||
token_address: str = Field(..., min_length=32, max_length=128)
|
||||
chain: str = Field(default="solana")
|
||||
outcome: str = Field(..., pattern=r"^(clean|malicious|rug_confirmed)$")
|
||||
|
||||
|
||||
# ── Endpoints ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/vote")
|
||||
async def api_cast_vote(req: VoteRequest):
|
||||
"""Cast a vote on whether a token is clean or malicious."""
|
||||
result = cast_vote(
|
||||
token_address=req.token_address,
|
||||
chain=req.chain,
|
||||
voter_id=req.voter_id,
|
||||
vote=req.vote,
|
||||
)
|
||||
if result.get("success") is False:
|
||||
raise HTTPException(status_code=400, detail=result.get("error", "Vote failed"))
|
||||
return result
|
||||
|
||||
|
||||
@router.post("/resolve")
|
||||
async def api_resolve_outcome(req: ResolveRequest):
|
||||
"""Resolve a token's outcome (admin/cron). Updates all voter badges."""
|
||||
result = resolve_token_outcome(
|
||||
token_address=req.token_address,
|
||||
chain=req.chain,
|
||||
outcome=req.outcome,
|
||||
)
|
||||
if result.get("success") is False:
|
||||
raise HTTPException(status_code=400, detail=result.get("error", "Resolve failed"))
|
||||
return result
|
||||
|
||||
|
||||
@router.get("/badge/{user_id}")
|
||||
async def api_get_badge(user_id: str):
|
||||
"""Get a user's badge profile."""
|
||||
badge = get_user_badge(user_id)
|
||||
if badge is None:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
return {
|
||||
"user_id": badge.user_id,
|
||||
"total_votes": badge.total_votes,
|
||||
"correct_votes": badge.correct_votes,
|
||||
"current_tier": badge.current_tier,
|
||||
"next_tier": badge.next_tier,
|
||||
"votes_needed_for_next": badge.votes_needed_for_next,
|
||||
"badges_earned": badge.badges_earned,
|
||||
"accuracy_pct": badge.accuracy_pct,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/votes/{chain}/{token_address:path}")
|
||||
async def api_get_token_votes(token_address: str, chain: str):
|
||||
"""Get all votes for a token."""
|
||||
result = get_token_votes(token_address, chain)
|
||||
if "error" in result:
|
||||
raise HTTPException(status_code=404, detail=result["error"])
|
||||
return result
|
||||
|
||||
|
||||
@router.get("/leaderboard")
|
||||
async def api_get_leaderboard(limit: int = Query(default=20, le=100)):
|
||||
"""Get community leaderboard (top sleuths)."""
|
||||
return {"leaderboard": get_leaderboard(limit)}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,46 +0,0 @@
|
|||
"""Content API - Ghost-powered unified feed."""
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
|
||||
router = APIRouter(prefix="/api/v1/content", tags=["content"])
|
||||
|
||||
|
||||
@router.post("/post")
|
||||
async def create_post(request: Request, data: dict):
|
||||
from app.content_platform import create_post
|
||||
|
||||
return await create_post(
|
||||
content=data.get("content", ""),
|
||||
title=data.get("title", ""),
|
||||
tags=data.get("tags", []),
|
||||
post_type=data.get("post_type", "micro"),
|
||||
status=data.get("status", "published"),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/feed")
|
||||
async def feed(request: Request, page: int = 1, limit: int = 20):
|
||||
from app.content_platform import get_feed
|
||||
|
||||
return await get_feed(page=page, limit=limit)
|
||||
|
||||
|
||||
@router.get("/posts")
|
||||
async def posts(request: Request, page: int = 1, limit: int = 20, tag: str = ""):
|
||||
from app.content_platform import get_posts
|
||||
|
||||
return await get_posts(page=page, limit=limit, tag=tag)
|
||||
|
||||
|
||||
@router.get("/profile")
|
||||
async def profile(request: Request):
|
||||
from app.content_platform import PROFILE
|
||||
|
||||
return PROFILE
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def content_health(request: Request):
|
||||
from app.content_platform import check_health
|
||||
|
||||
return await check_health()
|
||||
|
|
@ -1,82 +0,0 @@
|
|||
"""
|
||||
Cross-Token Tracking API Router - Track wallets across multiple tokens.
|
||||
Connects to /api/v1/cross-token/*
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
router = APIRouter(prefix="/api/v1/cross-token", tags=["cross-token"])
|
||||
|
||||
# Lazy import - cross_token.py has broken internal deps
|
||||
PROJECT_TOKENS = {
|
||||
"CRM": "Eme5T2s2HB7B8W4YgLG1eReQpnadEVUnQBRjaKTdBAGS",
|
||||
"SOSANA": "SoSaNaTokenAddressPlaceholder123456789",
|
||||
"PBTC": "pBTC Token Address Placeholder",
|
||||
"SHIFT_AI": "ShiftAITokenAddressPlaceholder123456",
|
||||
}
|
||||
|
||||
|
||||
class CrossTokenRequest(BaseModel):
|
||||
wallet: str
|
||||
token_addresses: list[str] | None = None
|
||||
|
||||
|
||||
@router.get("/projects")
|
||||
async def list_projects():
|
||||
"""List all tracked token projects."""
|
||||
return {"projects": PROJECT_TOKENS, "count": len(PROJECT_TOKENS)}
|
||||
|
||||
|
||||
@router.post("/affiliations")
|
||||
async def track_affiliations(req: CrossTokenRequest):
|
||||
"""Track a wallet's affiliations across all known token projects."""
|
||||
try:
|
||||
from app.cross_token import CrossTokenAffiliationTracker
|
||||
|
||||
tracker = CrossTokenAffiliationTracker()
|
||||
affiliations = await tracker.track_wallet_affiliations(wallet=req.wallet)
|
||||
return {
|
||||
"wallet": req.wallet,
|
||||
"affiliations": [
|
||||
{
|
||||
"project": a.project,
|
||||
"token_address": a.token_address,
|
||||
"balance": a.balance,
|
||||
"first_acquired": str(a.first_acquired) if a.first_acquired else None,
|
||||
"evidence_strength": a.evidence_strength,
|
||||
}
|
||||
for a in affiliations
|
||||
],
|
||||
"project_count": len(affiliations),
|
||||
}
|
||||
except ImportError as e:
|
||||
raise HTTPException(status_code=503, detail=f"Module unavailable: {e}") from e
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:500]) from e
|
||||
|
||||
|
||||
@router.get("/connections/{wallet}")
|
||||
async def get_cross_connections(wallet: str):
|
||||
"""Find cross-token connections for a wallet."""
|
||||
try:
|
||||
from app.cross_token import CrossTokenAffiliationTracker
|
||||
|
||||
tracker = CrossTokenAffiliationTracker()
|
||||
affiliations = await tracker.track_wallet_affiliations(wallet=wallet)
|
||||
multi = [a for a in affiliations if a.evidence_strength != "unverified"]
|
||||
return {
|
||||
"wallet": wallet,
|
||||
"multi_project": len(multi),
|
||||
"projects": [a.project for a in multi],
|
||||
"total_affiliations": len(affiliations),
|
||||
}
|
||||
except ImportError as e:
|
||||
raise HTTPException(status_code=503, detail=str(e)) from e
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:500]) from e
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def cross_token_health():
|
||||
return {"status": "ok", "service": "cross-token-tracker"}
|
||||
|
|
@ -1,497 +0,0 @@
|
|||
"""
|
||||
Darkroom Airdrop Admin Router
|
||||
==============================
|
||||
Admin-only endpoints for airdrop operations:
|
||||
• Snapshot creation from existing tokens
|
||||
• Team/dev token allocation with vesting
|
||||
• Anti-sniper protection configuration
|
||||
• Batch airdrop execution
|
||||
• Vesting management
|
||||
|
||||
All endpoints require X-Admin-Key header.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.airdrop_engine import (
|
||||
AirdropCampaign,
|
||||
AirdropDistributor,
|
||||
AirdropStorage,
|
||||
AntiSniperProtection,
|
||||
SnapshotEngine,
|
||||
TeamAllocation,
|
||||
create_full_token_with_protection,
|
||||
)
|
||||
from app.token_deployer import TokenDeployerFactory, get_storage
|
||||
|
||||
logger = logging.getLogger("darkroom_airdrop_admin")
|
||||
|
||||
router = APIRouter(prefix="/api/v1/admin/tokens/airdrop", tags=["darkroom-airdrop"])
|
||||
|
||||
|
||||
# ── Auth helper ───────────────────────────────────────────────
|
||||
|
||||
|
||||
async def _verify_admin(request: Request):
|
||||
admin_key = os.getenv("ADMIN_API_KEY", "dev-key-change-me")
|
||||
key = request.headers.get("X-Admin-Key", "")
|
||||
if key != admin_key:
|
||||
raise HTTPException(status_code=401, detail="Invalid admin key")
|
||||
return True
|
||||
|
||||
|
||||
# ── Request Models ────────────────────────────────────────────
|
||||
|
||||
|
||||
class SnapshotRequest(BaseModel):
|
||||
source_token: str
|
||||
source_chain: str = Field(..., description="ethereum, base, bsc, solana")
|
||||
min_holdings: str = "0"
|
||||
excluded_addresses: list[str] = Field(default_factory=list)
|
||||
block_number: int | None = None
|
||||
|
||||
|
||||
class AirdropExecuteRequest(BaseModel):
|
||||
deployment_id: str
|
||||
snapshot_id: str | None = None
|
||||
distribution_type: str = "snapshot" # snapshot, manual, team
|
||||
recipients: list[dict[str, str]] = Field(default_factory=list)
|
||||
# For manual: [{"address": "0x...", "amount": "1000", "reason": "marketing"}]
|
||||
|
||||
|
||||
class TeamAllocationRequest(BaseModel):
|
||||
deployment_id: str
|
||||
allocations: list[dict[str, Any]] = Field(
|
||||
...,
|
||||
description="""
|
||||
[
|
||||
{"address": "0x...", "role": "dev", "percent": 15, "immediate": 20,
|
||||
"vested": 80, "cliff_months": 6, "vesting_months": 24},
|
||||
{"address": "0x...", "role": "marketing", "percent": 10, "immediate": 50,
|
||||
"vested": 50, "cliff_months": 3, "vesting_months": 12}
|
||||
]
|
||||
""",
|
||||
)
|
||||
|
||||
|
||||
class AntiSniperRequest(BaseModel):
|
||||
deployment_id: str
|
||||
blacklist_addresses: list[str] = Field(default_factory=list)
|
||||
trading_delay_blocks: int = 0
|
||||
max_wallet_percent: float = 2.0
|
||||
max_tx_percent: float = 1.0
|
||||
enable_trading_at_block: int | None = None
|
||||
|
||||
|
||||
class FullLaunchRequest(BaseModel):
|
||||
chain: str
|
||||
name: str
|
||||
symbol: str
|
||||
decimals: int = 18
|
||||
initial_supply: str = "1000000000" # 1B default
|
||||
team_allocations: list[dict[str, Any]] = Field(default_factory=list)
|
||||
airdrop_source_token: str | None = None
|
||||
airdrop_source_chain: str | None = None
|
||||
anti_sniper: bool = True
|
||||
trading_delay_blocks: int = 0
|
||||
max_wallet_percent: float = 2.0
|
||||
max_tx_percent: float = 1.0
|
||||
blacklist_addresses: list[str] = Field(default_factory=list)
|
||||
# Quick team presets
|
||||
dev_percent: float | None = None
|
||||
marketing_percent: float | None = None
|
||||
treasury_percent: float | None = None
|
||||
airdrop_percent: float | None = None
|
||||
dev_address: str | None = None
|
||||
marketing_address: str | None = None
|
||||
treasury_address: str | None = None
|
||||
|
||||
|
||||
class VestingClaimRequest(BaseModel):
|
||||
schedule_id: str
|
||||
|
||||
|
||||
class EnableTradingRequest(BaseModel):
|
||||
deployment_id: str
|
||||
|
||||
|
||||
# ── Endpoints ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/snapshot")
|
||||
async def create_snapshot(request: Request, body: SnapshotRequest, _=Depends(_verify_admin)):
|
||||
"""Create a snapshot of token holders for airdrop eligibility."""
|
||||
try:
|
||||
if body.source_chain in ["ethereum", "base", "bsc"]:
|
||||
rpc = os.getenv(f"{body.source_chain.upper()}_RPC_URL", "")
|
||||
if not rpc:
|
||||
raise HTTPException(status_code=400, detail=f"No RPC configured for {body.source_chain}")
|
||||
|
||||
snapshot = await SnapshotEngine.create_evm_snapshot(
|
||||
token_address=body.source_token,
|
||||
chain=body.source_chain,
|
||||
rpc_url=rpc,
|
||||
min_holdings=body.min_holdings,
|
||||
excluded_addresses=body.excluded_addresses,
|
||||
block_number=body.block_number,
|
||||
)
|
||||
elif body.source_chain == "solana":
|
||||
rpc = os.getenv("SOLANA_RPC_URL", "")
|
||||
if not rpc:
|
||||
raise HTTPException(status_code=400, detail="No Solana RPC configured")
|
||||
|
||||
snapshot = await SnapshotEngine.create_solana_snapshot(
|
||||
token_address=body.source_token,
|
||||
rpc_url=rpc,
|
||||
min_holdings=body.min_holdings,
|
||||
excluded_addresses=body.excluded_addresses,
|
||||
)
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail=f"Unsupported chain: {body.source_chain}")
|
||||
|
||||
await AirdropStorage.save_snapshot(snapshot)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"snapshot": {
|
||||
"snapshot_id": snapshot.snapshot_id,
|
||||
"source_token": snapshot.source_token,
|
||||
"source_chain": snapshot.source_chain,
|
||||
"block_number": snapshot.block_number,
|
||||
"total_holders": snapshot.total_holders,
|
||||
"total_supply_snapshotted": snapshot.total_supply_snapshotted,
|
||||
"timestamp": snapshot.timestamp,
|
||||
},
|
||||
"holders_preview": [{"address": h.address, "amount": h.amount} for h in snapshot.holders[:10]],
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Snapshot failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Snapshot failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/execute")
|
||||
async def execute_airdrop(request: Request, body: AirdropExecuteRequest, _=Depends(_verify_admin)):
|
||||
"""Execute airdrop distribution."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(body.deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
|
||||
# Get recipients
|
||||
recipients = []
|
||||
if body.snapshot_id:
|
||||
# Load from snapshot
|
||||
snapshot_data = None
|
||||
if storage.redis:
|
||||
data = await storage.redis.get(f"airdrop_snapshot:{body.snapshot_id}")
|
||||
if data:
|
||||
snapshot_data = json.loads(data)
|
||||
|
||||
if not snapshot_data:
|
||||
raise HTTPException(status_code=404, detail="Snapshot not found")
|
||||
|
||||
from app.airdrop_engine import AirdropRecipient
|
||||
|
||||
recipients = [AirdropRecipient(**r) for r in snapshot_data.get("holders", [])]
|
||||
elif body.recipients:
|
||||
from app.airdrop_engine import AirdropRecipient
|
||||
|
||||
recipients = [AirdropRecipient(**r) for r in body.recipients]
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail="No recipients provided (need snapshot_id or recipients)")
|
||||
|
||||
# Execute based on chain
|
||||
if deployment.chain in ["ethereum", "base", "bsc"]:
|
||||
result = await AirdropDistributor.execute_evm_airdrop(deployer, deployment.contract_address, recipients)
|
||||
elif deployment.chain == "solana":
|
||||
result = await AirdropDistributor.execute_solana_airdrop(deployer, deployment.contract_address, recipients)
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail=f"Airdrop not supported for {deployment.chain}")
|
||||
|
||||
# Save campaign
|
||||
campaign = AirdropCampaign(
|
||||
campaign_id=f"airdrop_{body.deployment_id}_{int(time.time())}",
|
||||
deployment_id=body.deployment_id,
|
||||
snapshot_id=body.snapshot_id or "manual",
|
||||
chain=deployment.chain,
|
||||
status="completed" if result["failed"] == 0 else "partial",
|
||||
distribution_type=body.distribution_type,
|
||||
recipients=recipients,
|
||||
total_amount=str(sum(int(r.amount) for r in recipients)),
|
||||
distributed_amount=str(sum(int(r.amount) for r in recipients if r.claimed)),
|
||||
tx_hashes=result["tx_hashes"],
|
||||
)
|
||||
await AirdropStorage.save_campaign(campaign)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"campaign_id": campaign.campaign_id,
|
||||
"result": result,
|
||||
"total_recipients": len(recipients),
|
||||
"distributed": campaign.distributed_amount,
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Airdrop execution failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Airdrop failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/team")
|
||||
async def allocate_team(request: Request, body: TeamAllocationRequest, _=Depends(_verify_admin)):
|
||||
"""Allocate team/dev tokens with optional vesting."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(body.deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
|
||||
result = await TeamAllocation.allocate_team_tokens(deployer, deployment, body.allocations)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"deployment_id": body.deployment_id,
|
||||
"allocations": result["allocations"],
|
||||
"total_allocated": str(result["total_allocated"]),
|
||||
"tx_hashes": result["tx_hashes"],
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Team allocation failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Team allocation failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/antisniper")
|
||||
async def apply_antisniper(request: Request, body: AntiSniperRequest, _=Depends(_verify_admin)):
|
||||
"""Apply anti-sniper protection to a deployed token."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(body.deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
|
||||
result = await AntiSniperProtection.apply_protection(
|
||||
deployer,
|
||||
deployment.contract_address,
|
||||
deployment,
|
||||
blacklist_addresses=body.blacklist_addresses,
|
||||
trading_delay_blocks=body.trading_delay_blocks,
|
||||
max_wallet_percent=body.max_wallet_percent,
|
||||
max_tx_percent=body.max_tx_percent,
|
||||
)
|
||||
|
||||
# Update deployment record
|
||||
await storage.update_status(body.deployment_id, "anti_sniper_applied", {"anti_sniper": result})
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"deployment_id": body.deployment_id,
|
||||
"protection_applied": result,
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Anti-sniper failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Anti-sniper failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/launch")
|
||||
async def full_launch(request: Request, body: FullLaunchRequest, _=Depends(_verify_admin)):
|
||||
"""
|
||||
Full token launch with everything: deploy, anti-sniper, team allocation, airdrop.
|
||||
This is the one-shot launch endpoint for CRM v2 or any new token.
|
||||
"""
|
||||
try:
|
||||
# Build team allocations from quick presets if provided
|
||||
team_allocs = body.team_allocations or []
|
||||
|
||||
if body.dev_percent and body.dev_address:
|
||||
team_allocs.append(
|
||||
{
|
||||
"address": body.dev_address,
|
||||
"role": "development",
|
||||
"percent": body.dev_percent,
|
||||
"immediate": 20,
|
||||
"vested": 80,
|
||||
"cliff_months": 6,
|
||||
"vesting_months": 24,
|
||||
}
|
||||
)
|
||||
|
||||
if body.marketing_percent and body.marketing_address:
|
||||
team_allocs.append(
|
||||
{
|
||||
"address": body.marketing_address,
|
||||
"role": "marketing",
|
||||
"percent": body.marketing_percent,
|
||||
"immediate": 50,
|
||||
"vested": 50,
|
||||
"cliff_months": 3,
|
||||
"vesting_months": 12,
|
||||
}
|
||||
)
|
||||
|
||||
if body.treasury_percent and body.treasury_address:
|
||||
team_allocs.append(
|
||||
{
|
||||
"address": body.treasury_address,
|
||||
"role": "treasury",
|
||||
"percent": body.treasury_percent,
|
||||
"immediate": 0,
|
||||
"vested": 100,
|
||||
"cliff_months": 12,
|
||||
"vesting_months": 36,
|
||||
}
|
||||
)
|
||||
|
||||
result = await create_full_token_with_protection(
|
||||
chain=body.chain,
|
||||
name=body.name,
|
||||
symbol=body.symbol,
|
||||
decimals=body.decimals,
|
||||
initial_supply=body.initial_supply,
|
||||
team_allocations=team_allocs,
|
||||
airdrop_source_token=body.airdrop_source_token,
|
||||
airdrop_source_chain=body.airdrop_source_chain,
|
||||
anti_sniper=body.anti_sniper,
|
||||
trading_delay_blocks=body.trading_delay_blocks,
|
||||
max_wallet_percent=body.max_wallet_percent,
|
||||
max_tx_percent=body.max_tx_percent,
|
||||
blacklist_addresses=body.blacklist_addresses,
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"launch_complete": True,
|
||||
"deployment": result.get("deployment"),
|
||||
"anti_sniper": result.get("anti_sniper"),
|
||||
"team_allocation": result.get("team_allocation"),
|
||||
"airdrop": result.get("airdrop"),
|
||||
"errors": result.get("errors", []),
|
||||
"explorer_url": _get_explorer_url(body.chain, result["deployment"]["contract_address"])
|
||||
if result.get("deployment")
|
||||
else None,
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Full launch failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Launch failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/trading/enable")
|
||||
async def enable_trading(request: Request, body: EnableTradingRequest, _=Depends(_verify_admin)):
|
||||
"""Enable trading for a token (after anti-sniper delay)."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(body.deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
tx_hash = await deployer.set_trading_enabled(deployment.contract_address, True)
|
||||
|
||||
await storage.update_status(body.deployment_id, "trading_enabled", {"trading_enabled_tx": tx_hash})
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"tx_hash": tx_hash,
|
||||
"trading_enabled": True,
|
||||
"explorer_url": _get_explorer_url(deployment.chain, tx_hash, is_tx=True),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Enable trading failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Enable trading failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.get("/campaign/{campaign_id}")
|
||||
async def get_campaign(request: Request, campaign_id: str, _=Depends(_verify_admin)):
|
||||
"""Get airdrop campaign status."""
|
||||
try:
|
||||
campaign = await AirdropStorage.get_campaign(campaign_id)
|
||||
if not campaign:
|
||||
raise HTTPException(status_code=404, detail="Campaign not found")
|
||||
|
||||
return {
|
||||
"campaign": campaign.to_dict(),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Get campaign failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Get failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.get("/vesting/{schedule_id}")
|
||||
async def get_vesting(request: Request, schedule_id: str, _=Depends(_verify_admin)):
|
||||
"""Get vesting schedule details."""
|
||||
try:
|
||||
from app.token_deployer import get_storage
|
||||
|
||||
storage = await get_storage()
|
||||
|
||||
data = None
|
||||
if storage.redis:
|
||||
data = await storage.redis.get(f"vesting:{schedule_id}")
|
||||
|
||||
if not data:
|
||||
raise HTTPException(status_code=404, detail="Vesting schedule not found")
|
||||
|
||||
return {
|
||||
"vesting": json.loads(data),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Get vesting failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Get failed: {e!s}") from e
|
||||
|
||||
|
||||
# ── Helpers ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _get_explorer_url(chain: str, address_or_tx: str, is_tx: bool = False) -> str:
|
||||
explorers = {
|
||||
"ethereum": "https://etherscan.io",
|
||||
"base": "https://basescan.org",
|
||||
"bsc": "https://bscscan.com",
|
||||
"solana": "https://solscan.io",
|
||||
"tron": "https://tronscan.org/#",
|
||||
}
|
||||
base = explorers.get(chain, "")
|
||||
if not base:
|
||||
return ""
|
||||
|
||||
if chain == "tron":
|
||||
path = "transaction" if is_tx else "contract"
|
||||
return f"{base}/{path}/{address_or_tx}"
|
||||
|
||||
path = "tx" if is_tx else "address"
|
||||
return f"{base}/{path}/{address_or_tx}"
|
||||
|
|
@ -1,641 +0,0 @@
|
|||
"""
|
||||
Darkroom Multi-Chain Airdrop Admin Router
|
||||
==========================================
|
||||
Admin endpoints for cross-chain airdrop campaigns.
|
||||
|
||||
Supports:
|
||||
• Multi-source snapshots (Base + Solana + any chain)
|
||||
• Weighted allocation based on holdings
|
||||
• Cross-chain bonuses for multi-chain holders
|
||||
• CRM + CryptoRugMunch specific presets
|
||||
|
||||
All endpoints require X-Admin-Key header.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.multichain_airdrop import (
|
||||
CrossChainIdentityResolver,
|
||||
MultiChainAirdropManager,
|
||||
TokenSource,
|
||||
create_crm_multichain_airdrop,
|
||||
)
|
||||
from app.token_deployer import get_storage
|
||||
|
||||
logger = logging.getLogger("darkroom_multichain_admin")
|
||||
|
||||
router = APIRouter(prefix="/api/v1/admin/tokens/multichain", tags=["darkroom-multichain"])
|
||||
|
||||
|
||||
# ── Auth helper ───────────────────────────────────────────────
|
||||
|
||||
|
||||
async def _verify_admin(request: Request):
|
||||
admin_key = os.getenv("ADMIN_API_KEY", "dev-key-change-me")
|
||||
key = request.headers.get("X-Admin-Key", "")
|
||||
if key != admin_key:
|
||||
raise HTTPException(status_code=401, detail="Invalid admin key")
|
||||
return True
|
||||
|
||||
|
||||
# ── Request Models ────────────────────────────────────────────
|
||||
|
||||
|
||||
class TokenSourceConfig(BaseModel):
|
||||
chain: str
|
||||
token_address: str
|
||||
token_symbol: str
|
||||
weight_multiplier: float = 1.0
|
||||
min_holdings: str = "0"
|
||||
max_holdings: str | None = None
|
||||
|
||||
|
||||
class CreateCampaignRequest(BaseModel):
|
||||
target_chain: str
|
||||
target_token: str
|
||||
source_tokens: list[TokenSourceConfig]
|
||||
total_pool: str
|
||||
distribution_mode: str = "proportional"
|
||||
require_hold_both_chains: bool = False
|
||||
require_hold_any_chain: bool = True
|
||||
cross_chain_bonus: float = 0.0
|
||||
exclude_contracts: bool = True
|
||||
exclude_known_bots: bool = True
|
||||
min_wallet_age_days: int | None = None
|
||||
min_tx_count_30d: int | None = None
|
||||
snapshot_time_window_hours: int = 24
|
||||
|
||||
|
||||
class CRMPresetRequest(BaseModel):
|
||||
crm_base_address: str | None = None
|
||||
crm_solana_address: str
|
||||
cryptorugmunch_zora_address: str # Zora tokens are on Base
|
||||
target_chain: str = "solana"
|
||||
target_token: str = ""
|
||||
total_pool: str | None = None # Auto-calculated for 1:1
|
||||
mode: str = "one_to_one" # one_to_one, proportional, tiered
|
||||
|
||||
|
||||
class CRMv2LaunchRequest(BaseModel):
|
||||
crm_solana_address: str
|
||||
cryptorugmunch_zora_address: str
|
||||
target_chain: str = "solana"
|
||||
target_token: str = ""
|
||||
# Deploy params if target_token not provided
|
||||
deploy_new_token: bool = False
|
||||
token_name: str = "CRM Token v2"
|
||||
token_symbol: str = "CRMv2"
|
||||
decimals: int = 18
|
||||
# Anti-sniper
|
||||
anti_sniper: bool = True
|
||||
blacklist_addresses: list[str] = Field(default_factory=list)
|
||||
max_wallet_percent: float = 2.0
|
||||
max_tx_percent: float = 1.0
|
||||
# Team allocation
|
||||
dev_address: str | None = None
|
||||
dev_percent: float = 15.0
|
||||
marketing_address: str | None = None
|
||||
marketing_percent: float = 10.0
|
||||
treasury_address: str | None = None
|
||||
treasury_percent: float = 15.0
|
||||
|
||||
|
||||
class ExecuteSnapshotRequest(BaseModel):
|
||||
campaign_id: str
|
||||
use_custom_snapshot: bool = False
|
||||
custom_snapshot_format: str = "json" # json, csv, manual
|
||||
custom_snapshot_data: str | None = None # JSON string or CSV content
|
||||
custom_snapshot_file: str | None = None # Path to uploaded file
|
||||
|
||||
|
||||
class UploadSnapshotRequest(BaseModel):
|
||||
campaign_id: str
|
||||
format: str = "json" # json, csv
|
||||
data: str # JSON array or CSV content
|
||||
deduplicate: bool = True
|
||||
validate_balances: bool = False
|
||||
|
||||
|
||||
class ManualHoldersRequest(BaseModel):
|
||||
campaign_id: str
|
||||
holders: list[dict[str, str]] # [{"address": "0x...", "amount": "1000", "chain": "solana", "token": "CRM"}]
|
||||
append: bool = False # True = append to existing, False = replace
|
||||
|
||||
|
||||
class ExecuteDistributionRequest(BaseModel):
|
||||
campaign_id: str
|
||||
|
||||
|
||||
class LinkAddressesRequest(BaseModel):
|
||||
primary_address: str
|
||||
primary_chain: str
|
||||
linked_chain: str
|
||||
linked_address: str
|
||||
proof_signature: str = ""
|
||||
|
||||
|
||||
class GetCampaignRequest(BaseModel):
|
||||
campaign_id: str
|
||||
|
||||
|
||||
class TierConfig(BaseModel):
|
||||
name: str
|
||||
min_score: float
|
||||
max_score: float | None = None
|
||||
base_allocation: str
|
||||
bonus_multiplier: float = 1.0
|
||||
|
||||
|
||||
class CreateTieredCampaignRequest(BaseModel):
|
||||
target_chain: str
|
||||
target_token: str
|
||||
source_tokens: list[TokenSourceConfig]
|
||||
tiers: list[TierConfig]
|
||||
total_pool: str
|
||||
cross_chain_bonus: float = 0.0
|
||||
|
||||
|
||||
# ── Endpoints ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/campaign/create")
|
||||
async def create_campaign(request: Request, body: CreateCampaignRequest, _=Depends(_verify_admin)):
|
||||
"""Create a new multi-chain airdrop campaign."""
|
||||
try:
|
||||
sources = [
|
||||
TokenSource(
|
||||
chain=s.chain,
|
||||
token_address=s.token_address,
|
||||
token_symbol=s.token_symbol,
|
||||
weight_multiplier=s.weight_multiplier,
|
||||
min_holdings=s.min_holdings,
|
||||
max_holdings=s.max_holdings,
|
||||
)
|
||||
for s in body.source_tokens
|
||||
]
|
||||
|
||||
campaign = await MultiChainAirdropManager.create_campaign(
|
||||
target_chain=body.target_chain,
|
||||
target_token=body.target_token,
|
||||
source_tokens=sources,
|
||||
total_pool=body.total_pool,
|
||||
distribution_mode=body.distribution_mode,
|
||||
require_hold_both_chains=body.require_hold_both_chains,
|
||||
require_hold_any_chain=body.require_hold_any_chain,
|
||||
cross_chain_bonus=body.cross_chain_bonus,
|
||||
exclude_contracts=body.exclude_contracts,
|
||||
exclude_known_bots=body.exclude_known_bots,
|
||||
min_wallet_age_days=body.min_wallet_age_days,
|
||||
min_tx_count_30d=body.min_tx_count_30d,
|
||||
snapshot_time_window_hours=body.snapshot_time_window_hours,
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"campaign": campaign.to_dict(),
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Create campaign failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Create failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/campaign/crm-preset")
|
||||
async def create_crm_preset(request: Request, body: CRMPresetRequest, _=Depends(_verify_admin)):
|
||||
"""
|
||||
Create CRM v2 airdrop campaign.
|
||||
|
||||
Modes:
|
||||
- one_to_one: Holders receive exact same amount they hold (default)
|
||||
- proportional: Weighted proportional distribution
|
||||
|
||||
Default (1:1):
|
||||
- Hold $CRM on Solana → receive exact same amount of CRMv2
|
||||
- Hold CryptoRugMunch on Zora → receive exact same amount of CRMv2
|
||||
- Hold both → receive SUM of both holdings
|
||||
"""
|
||||
try:
|
||||
if body.mode == "one_to_one":
|
||||
from app.multichain_airdrop import create_crm_v2_airdrop_preset
|
||||
|
||||
campaign = await create_crm_v2_airdrop_preset(
|
||||
crm_solana_address=body.crm_solana_address,
|
||||
cryptorugmunch_zora_address=body.cryptorugmunch_zora_address,
|
||||
target_chain=body.target_chain,
|
||||
target_token=body.target_token,
|
||||
)
|
||||
rules = {
|
||||
"mode": "1:1 exact match",
|
||||
"hold_any_of": [
|
||||
"CRM on Solana (any amount) → receive exact same amount",
|
||||
"CryptoRugMunch on Zora (any amount) → receive exact same amount",
|
||||
],
|
||||
"multiple_holdings": "SUM of all holdings",
|
||||
"distribution": f"1:1 on {body.target_chain}",
|
||||
}
|
||||
else:
|
||||
# Proportional mode (legacy)
|
||||
campaign = await create_crm_multichain_airdrop(
|
||||
crm_base_address=body.crm_base_address or "",
|
||||
crm_solana_address=body.crm_solana_address,
|
||||
cryptorugmunch_solana_address=body.cryptorugmunch_zora_address,
|
||||
target_chain=body.target_chain,
|
||||
target_token=body.target_token,
|
||||
total_pool=body.total_pool or "100000000",
|
||||
)
|
||||
rules = {
|
||||
"mode": "proportional",
|
||||
"hold_any_of": [
|
||||
"CRM on Base (>= 1000, weight 2.0x)",
|
||||
"CRM on Solana (>= 1000, weight 1.5x)",
|
||||
"CryptoRugMunch on Zora (>= 1, weight 1.0x)",
|
||||
],
|
||||
"cross_chain_bonus": "10% per additional chain",
|
||||
"distribution": f"Proportional on {body.target_chain}",
|
||||
}
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"campaign": campaign.to_dict(),
|
||||
"preset": "CRM v2 Airdrop",
|
||||
"qualification_rules": rules,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"CRM preset failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Preset failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/snapshot")
|
||||
async def execute_snapshot(request: Request, body: ExecuteSnapshotRequest, _=Depends(_verify_admin)):
|
||||
"""Execute snapshot phase for a campaign."""
|
||||
try:
|
||||
# Use custom snapshot if provided
|
||||
if body.use_custom_snapshot:
|
||||
from app.multichain_airdrop import CustomSnapshotLoader
|
||||
|
||||
if body.custom_snapshot_format == "json" and body.custom_snapshot_data:
|
||||
holders = CustomSnapshotLoader.parse_json_snapshot(body.custom_snapshot_data)
|
||||
elif body.custom_snapshot_format == "csv" and body.custom_snapshot_data:
|
||||
holders = CustomSnapshotLoader.parse_csv_snapshot(body.custom_snapshot_data)
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail="Custom snapshot data required")
|
||||
|
||||
campaign = await MultiChainAirdropManager.execute_custom_snapshot(
|
||||
body.campaign_id, holders, deduplicate=True
|
||||
)
|
||||
else:
|
||||
campaign = await MultiChainAirdropManager.execute_snapshot(body.campaign_id)
|
||||
|
||||
# Build summary
|
||||
holder_preview = []
|
||||
for h in campaign.qualified_holders[:10]:
|
||||
holder_preview.append(
|
||||
{
|
||||
"address": h.primary_address,
|
||||
"chains": list(h.chain_addresses.keys()),
|
||||
"holdings": h.holdings_per_source,
|
||||
"weighted_score": h.weighted_score,
|
||||
"allocation": h.airdrop_allocation,
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"campaign_id": campaign.campaign_id,
|
||||
"status": campaign.status,
|
||||
"total_qualified": campaign.total_qualified,
|
||||
"snapshot_completed_at": campaign.snapshot_completed_at,
|
||||
"holders_preview": holder_preview,
|
||||
"total_pool": campaign.total_airdrop_pool,
|
||||
"snapshot_source": "custom" if body.use_custom_snapshot else "on-chain",
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Snapshot failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Snapshot failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/snapshot/upload")
|
||||
async def upload_snapshot(request: Request, body: UploadSnapshotRequest, _=Depends(_verify_admin)):
|
||||
"""Upload a custom snapshot file (JSON or CSV) for a campaign."""
|
||||
try:
|
||||
from app.multichain_airdrop import CustomSnapshotLoader
|
||||
|
||||
if body.format == "json":
|
||||
holders = CustomSnapshotLoader.parse_json_snapshot(body.data)
|
||||
elif body.format == "csv":
|
||||
holders = CustomSnapshotLoader.parse_csv_snapshot(body.data)
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail="Format must be 'json' or 'csv'")
|
||||
|
||||
if body.deduplicate:
|
||||
holders = CustomSnapshotLoader.deduplicate_holders(holders)
|
||||
|
||||
# Execute custom snapshot
|
||||
campaign = await MultiChainAirdropManager.execute_custom_snapshot(
|
||||
body.campaign_id,
|
||||
holders,
|
||||
deduplicate=False, # Already deduped
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"campaign_id": body.campaign_id,
|
||||
"format": body.format,
|
||||
"holders_parsed": len(holders),
|
||||
"total_qualified": campaign.total_qualified,
|
||||
"total_pool": campaign.total_airdrop_pool,
|
||||
"status": campaign.status,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Upload snapshot failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Upload failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/snapshot/manual")
|
||||
async def add_manual_holders(request: Request, body: ManualHoldersRequest, _=Depends(_verify_admin)):
|
||||
"""Add manual holders to a campaign (or replace existing)."""
|
||||
try:
|
||||
from app.multichain_airdrop import CustomSnapshotLoader
|
||||
|
||||
# Parse manual holders
|
||||
new_holders = CustomSnapshotLoader.parse_manual_holders(body.holders)
|
||||
|
||||
campaign = await MultiChainAirdropManager._get_campaign(body.campaign_id)
|
||||
if not campaign:
|
||||
raise HTTPException(status_code=404, detail="Campaign not found")
|
||||
|
||||
if body.append and campaign.qualified_holders:
|
||||
# Merge with existing
|
||||
existing = {h.primary_address: h for h in campaign.qualified_holders}
|
||||
existing.update(new_holders)
|
||||
all_holders = existing
|
||||
else:
|
||||
all_holders = new_holders
|
||||
|
||||
# Re-run snapshot with combined data
|
||||
campaign = await MultiChainAirdropManager.execute_custom_snapshot(
|
||||
body.campaign_id, all_holders, deduplicate=True
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"campaign_id": body.campaign_id,
|
||||
"holders_added": len(body.holders),
|
||||
"total_qualified": campaign.total_qualified,
|
||||
"total_pool": campaign.total_airdrop_pool,
|
||||
"status": campaign.status,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Manual holders failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/distribute")
|
||||
async def execute_distribution(request: Request, body: ExecuteDistributionRequest, _=Depends(_verify_admin)):
|
||||
"""Execute airdrop distribution to all qualified holders."""
|
||||
try:
|
||||
campaign = await MultiChainAirdropManager.execute_distribution(body.campaign_id)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"campaign_id": campaign.campaign_id,
|
||||
"status": campaign.status,
|
||||
"total_qualified": campaign.total_qualified,
|
||||
"total_distributed": campaign.total_distributed,
|
||||
"distribution_completed_at": campaign.distribution_completed_at,
|
||||
"tx_count": len(campaign.tx_hashes),
|
||||
"tx_hashes": campaign.tx_hashes[:10], # Preview first 10
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Distribution failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Distribution failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/launch")
|
||||
async def full_multichain_launch(request: Request, body: CRMv2LaunchRequest, _=Depends(_verify_admin)):
|
||||
"""
|
||||
FULL CRM v2 LAUNCH: Deploy token + airdrop in one shot.
|
||||
|
||||
If target_token is empty, deploys a new CRMv2 token first.
|
||||
Then snapshots CRM (Solana) + CryptoRugMunch (Zora) holders.
|
||||
Finally distributes 1:1 airdrop to all qualified holders.
|
||||
"""
|
||||
try:
|
||||
target_token = body.target_token
|
||||
|
||||
# 1. Deploy new token if needed
|
||||
if not target_token and body.deploy_new_token:
|
||||
from app.token_deployer import DeployParams, TokenDeployerFactory
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(body.target_chain)
|
||||
|
||||
deploy_params = DeployParams(
|
||||
chain=body.target_chain,
|
||||
name=body.token_name,
|
||||
symbol=body.token_symbol,
|
||||
decimals=body.decimals,
|
||||
initial_supply="1000000000", # 1B total
|
||||
mintable=True,
|
||||
burnable=True,
|
||||
blacklist_enabled=body.anti_sniper,
|
||||
)
|
||||
|
||||
deployment = await deployer.deploy_token(deploy_params)
|
||||
target_token = deployment.contract_address
|
||||
|
||||
# Apply anti-sniper
|
||||
if body.anti_sniper:
|
||||
from app.airdrop_engine import AntiSniperProtection
|
||||
|
||||
await AntiSniperProtection.apply_protection(
|
||||
deployer,
|
||||
target_token,
|
||||
deployment,
|
||||
blacklist_addresses=body.blacklist_addresses,
|
||||
max_wallet_percent=body.max_wallet_percent,
|
||||
max_tx_percent=body.max_tx_percent,
|
||||
)
|
||||
|
||||
# Save deployment
|
||||
storage = await get_storage()
|
||||
await storage.save(deployment)
|
||||
|
||||
# 2. Create 1:1 airdrop campaign
|
||||
from app.multichain_airdrop import create_crm_v2_airdrop_preset
|
||||
|
||||
campaign = await create_crm_v2_airdrop_preset(
|
||||
crm_solana_address=body.crm_solana_address,
|
||||
cryptorugmunch_zora_address=body.cryptorugmunch_zora_address,
|
||||
target_chain=body.target_chain,
|
||||
target_token=target_token,
|
||||
)
|
||||
|
||||
# 3. Snapshot
|
||||
campaign = await MultiChainAirdropManager.execute_snapshot(campaign.campaign_id)
|
||||
|
||||
# 4. Distribute
|
||||
if campaign.total_qualified > 0:
|
||||
campaign = await MultiChainAirdropManager.execute_distribution(campaign.campaign_id)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"campaign_id": campaign.campaign_id,
|
||||
"target_token": target_token,
|
||||
"status": campaign.status,
|
||||
"total_qualified": campaign.total_qualified,
|
||||
"total_distributed": campaign.total_distributed,
|
||||
"total_pool": campaign.total_airdrop_pool,
|
||||
"tx_count": len(campaign.tx_hashes),
|
||||
"phases": {
|
||||
"token_deployed": body.deploy_new_token and not body.target_token,
|
||||
"snapshot_complete": True,
|
||||
"distribution_complete": campaign.status in ["completed", "partial"],
|
||||
},
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Full launch failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Launch failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.get("/campaign/{campaign_id}")
|
||||
async def get_campaign(request: Request, campaign_id: str, _=Depends(_verify_admin)):
|
||||
"""Get campaign details and status."""
|
||||
try:
|
||||
campaign = await MultiChainAirdropManager._get_campaign(campaign_id)
|
||||
if not campaign:
|
||||
raise HTTPException(status_code=404, detail="Campaign not found")
|
||||
|
||||
return {
|
||||
"campaign": campaign.to_dict(),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Get campaign failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Get failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.get("/campaign/{campaign_id}/holders")
|
||||
async def get_campaign_holders(
|
||||
request: Request, campaign_id: str, limit: int = 100, offset: int = 0, _=Depends(_verify_admin)
|
||||
):
|
||||
"""Get qualified holders for a campaign."""
|
||||
try:
|
||||
campaign = await MultiChainAirdropManager._get_campaign(campaign_id)
|
||||
if not campaign:
|
||||
raise HTTPException(status_code=404, detail="Campaign not found")
|
||||
|
||||
holders = campaign.qualified_holders[offset : offset + limit]
|
||||
|
||||
return {
|
||||
"campaign_id": campaign_id,
|
||||
"total": campaign.total_qualified,
|
||||
"offset": offset,
|
||||
"limit": limit,
|
||||
"holders": [h.to_dict() for h in holders],
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Get holders failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Get failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/link-addresses")
|
||||
async def link_addresses(request: Request, body: LinkAddressesRequest, _=Depends(_verify_admin)):
|
||||
"""Manually link cross-chain addresses for a user."""
|
||||
try:
|
||||
result = await CrossChainIdentityResolver.store_linkage(
|
||||
primary_address=body.primary_address,
|
||||
primary_chain=body.primary_chain,
|
||||
linked_chain=body.linked_chain,
|
||||
linked_address=body.linked_address,
|
||||
proof_signature=body.proof_signature,
|
||||
)
|
||||
|
||||
return {
|
||||
"success": result,
|
||||
"primary": f"{body.primary_chain}:{body.primary_address}",
|
||||
"linked": f"{body.linked_chain}:{body.linked_address}",
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Link addresses failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Link failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.get("/link-addresses/{address}/{chain}")
|
||||
async def get_linked_addresses(request: Request, address: str, chain: str, _=Depends(_verify_admin)):
|
||||
"""Get linked addresses for a wallet."""
|
||||
try:
|
||||
links = await CrossChainIdentityResolver.resolve_linked_addresses(address, chain)
|
||||
|
||||
return {
|
||||
"primary": f"{chain}:{address}",
|
||||
"linked_addresses": links,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Get links failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Get failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/campaign/tiered")
|
||||
async def create_tiered_campaign(request: Request, body: CreateTieredCampaignRequest, _=Depends(_verify_admin)):
|
||||
"""Create a tiered multi-chain airdrop campaign."""
|
||||
try:
|
||||
sources = [
|
||||
TokenSource(
|
||||
chain=s.chain,
|
||||
token_address=s.token_address,
|
||||
token_symbol=s.token_symbol,
|
||||
weight_multiplier=s.weight_multiplier,
|
||||
min_holdings=s.min_holdings,
|
||||
max_holdings=s.max_holdings,
|
||||
)
|
||||
for s in body.source_tokens
|
||||
]
|
||||
|
||||
from app.multichain_airdrop import AirdropTier
|
||||
|
||||
tiers = [
|
||||
AirdropTier(
|
||||
name=t.name,
|
||||
min_score=t.min_score,
|
||||
max_score=t.max_score,
|
||||
base_allocation=t.base_allocation,
|
||||
bonus_multiplier=t.bonus_multiplier,
|
||||
)
|
||||
for t in body.tiers
|
||||
]
|
||||
|
||||
campaign = await MultiChainAirdropManager.create_campaign(
|
||||
target_chain=body.target_chain,
|
||||
target_token=body.target_token,
|
||||
source_tokens=sources,
|
||||
total_pool=body.total_pool,
|
||||
distribution_mode="tiered",
|
||||
tiers=tiers,
|
||||
cross_chain_bonus=body.cross_chain_bonus,
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"campaign": campaign.to_dict(),
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Tiered campaign failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Create failed: {e!s}") from e
|
||||
|
|
@ -1,574 +0,0 @@
|
|||
"""
|
||||
Darkroom Token Admin Router
|
||||
=============================
|
||||
Admin-only endpoints for deploying, minting, and managing tokens
|
||||
across multiple chains. Includes blacklist/anti-bot controls.
|
||||
|
||||
All endpoints require X-Admin-Key header.
|
||||
|
||||
Routes:
|
||||
POST /api/v1/admin/tokens/deploy - Deploy new token
|
||||
POST /api/v1/admin/tokens/mint - Mint additional tokens
|
||||
POST /api/v1/admin/tokens/burn - Burn tokens
|
||||
POST /api/v1/admin/tokens/transfer - Transfer ownership
|
||||
POST /api/v1/admin/tokens/renounce - Renounce ownership
|
||||
POST /api/v1/admin/tokens/blacklist - Add to blacklist
|
||||
POST /api/v1/admin/tokens/unblacklist - Remove from blacklist
|
||||
GET /api/v1/admin/tokens/list - List all deployments
|
||||
GET /api/v1/admin/tokens/{id} - Get deployment details
|
||||
GET /api/v1/admin/tokens/chains - List supported chains
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
from fastapi import APIRouter, Body, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.token_deployer import (
|
||||
DeployParams,
|
||||
TokenDeployerFactory,
|
||||
get_storage,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("darkroom_token_admin")
|
||||
|
||||
router = APIRouter(prefix="/api/v1/admin/tokens", tags=["darkroom-tokens"])
|
||||
|
||||
|
||||
# ── Auth helper ───────────────────────────────────────────────
|
||||
|
||||
|
||||
async def _verify_admin(request: Request):
|
||||
"""Verify admin access with API key."""
|
||||
admin_key = os.getenv("ADMIN_API_KEY", "dev-key-change-me")
|
||||
key = request.headers.get("X-Admin-Key", "")
|
||||
if key != admin_key:
|
||||
raise HTTPException(status_code=401, detail="Invalid admin key")
|
||||
return True
|
||||
|
||||
|
||||
# ── Request Models ──────────────────────────────────────────
|
||||
|
||||
|
||||
class DeployTokenRequest(BaseModel):
|
||||
chain: str = Field(..., description="Chain: ethereum, base, bsc, solana, tron")
|
||||
name: str = Field(..., min_length=1, max_length=50)
|
||||
symbol: str = Field(..., min_length=1, max_length=10)
|
||||
decimals: int = Field(default=18, ge=0, le=18)
|
||||
initial_supply: str = Field(default="1000000")
|
||||
max_supply: str | None = None
|
||||
mintable: bool = True
|
||||
burnable: bool = True
|
||||
pausable: bool = False
|
||||
owner_address: str | None = None
|
||||
metadata_uri: str | None = None
|
||||
# Blacklist / anti-bot
|
||||
blacklist_enabled: bool = True
|
||||
blacklist_addresses: list[str] = Field(default_factory=list)
|
||||
max_wallet_limit: str | None = None
|
||||
max_tx_limit: str | None = None
|
||||
trading_enabled: bool = True
|
||||
anti_bot_delay: int = 0
|
||||
# Advanced
|
||||
tax_rate: float | None = None
|
||||
tax_recipient: str | None = None
|
||||
deflationary: bool = False
|
||||
|
||||
|
||||
class MintRequest(BaseModel):
|
||||
deployment_id: str
|
||||
to_address: str
|
||||
amount: str
|
||||
|
||||
|
||||
class BurnRequest(BaseModel):
|
||||
deployment_id: str
|
||||
amount: str
|
||||
|
||||
|
||||
class TransferOwnershipRequest(BaseModel):
|
||||
deployment_id: str
|
||||
new_owner: str
|
||||
|
||||
|
||||
class BlacklistRequest(BaseModel):
|
||||
deployment_id: str
|
||||
address: str
|
||||
|
||||
|
||||
class SetLimitRequest(BaseModel):
|
||||
deployment_id: str
|
||||
amount: str
|
||||
|
||||
|
||||
class TradingToggleRequest(BaseModel):
|
||||
deployment_id: str
|
||||
enabled: bool
|
||||
|
||||
|
||||
# ── Endpoints ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/chains")
|
||||
async def list_chains(request: Request, _=Depends(_verify_admin)):
|
||||
"""List all supported chains and their configuration status."""
|
||||
chains = TokenDeployerFactory.list_supported_chains()
|
||||
return {
|
||||
"chains": chains,
|
||||
"total": len(chains),
|
||||
"configured": sum(1 for c in chains if c["configured"]),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/deploy")
|
||||
async def deploy_token(request: Request, body: DeployTokenRequest, _=Depends(_verify_admin)):
|
||||
"""Deploy a new token on the specified chain."""
|
||||
try:
|
||||
# Get deployer for chain
|
||||
deployer = TokenDeployerFactory.get_deployer(body.chain)
|
||||
|
||||
# Build params
|
||||
params = DeployParams(
|
||||
chain=body.chain,
|
||||
name=body.name,
|
||||
symbol=body.symbol,
|
||||
decimals=body.decimals,
|
||||
initial_supply=body.initial_supply,
|
||||
max_supply=body.max_supply,
|
||||
mintable=body.mintable,
|
||||
burnable=body.burnable,
|
||||
pausable=body.pausable,
|
||||
owner_address=body.owner_address or deployer.deployer_address,
|
||||
metadata_uri=body.metadata_uri,
|
||||
blacklist_enabled=body.blacklist_enabled,
|
||||
blacklist_addresses=body.blacklist_addresses,
|
||||
max_wallet_limit=body.max_wallet_limit,
|
||||
max_tx_limit=body.max_tx_limit,
|
||||
trading_enabled=body.trading_enabled,
|
||||
anti_bot_delay=body.anti_bot_delay,
|
||||
tax_rate=body.tax_rate,
|
||||
tax_recipient=body.tax_recipient,
|
||||
deflationary=body.deflationary,
|
||||
)
|
||||
|
||||
# Deploy
|
||||
deployment = await deployer.deploy_token(params)
|
||||
|
||||
# Apply initial blacklist if provided
|
||||
if body.blacklist_addresses and body.blacklist_enabled:
|
||||
for addr in body.blacklist_addresses:
|
||||
try:
|
||||
await deployer.blacklist_add(deployment.contract_address, addr)
|
||||
logger.info(f"Blacklisted {addr} on {deployment.contract_address}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to blacklist {addr}: {e}")
|
||||
|
||||
# Apply max limits if set
|
||||
if body.max_wallet_limit:
|
||||
try:
|
||||
await deployer.set_max_wallet(deployment.contract_address, body.max_wallet_limit)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to set max wallet: {e}")
|
||||
|
||||
if body.max_tx_limit:
|
||||
try:
|
||||
await deployer.set_max_tx(deployment.contract_address, body.max_tx_limit)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to set max tx: {e}")
|
||||
|
||||
# Save deployment record
|
||||
storage = await get_storage()
|
||||
await storage.save(deployment)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"deployment": deployment.to_dict(),
|
||||
"explorer_url": _get_explorer_url(body.chain, deployment.contract_address),
|
||||
}
|
||||
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e)) from e
|
||||
except Exception as e:
|
||||
logger.error(f"Token deployment failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Deployment failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/mint")
|
||||
async def mint_tokens(request: Request, body: MintRequest, _=Depends(_verify_admin)):
|
||||
"""Mint additional tokens for a deployed contract."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(body.deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
if deployment.status != "deployed":
|
||||
raise HTTPException(status_code=400, detail=f"Cannot mint - status is {deployment.status}")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
tx_hash = await deployer.mint_tokens(deployment.contract_address, body.to_address, body.amount)
|
||||
|
||||
await storage.update_status(body.deployment_id, "minting", {"last_mint_tx": tx_hash})
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"tx_hash": tx_hash,
|
||||
"explorer_url": _get_explorer_url(deployment.chain, tx_hash, is_tx=True),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Mint failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Mint failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/burn")
|
||||
async def burn_tokens(request: Request, body: BurnRequest, _=Depends(_verify_admin)):
|
||||
"""Burn tokens from deployer balance."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(body.deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
tx_hash = await deployer.burn_tokens(deployment.contract_address, body.amount)
|
||||
|
||||
await storage.update_status(body.deployment_id, "burned", {"last_burn_tx": tx_hash})
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"tx_hash": tx_hash,
|
||||
"explorer_url": _get_explorer_url(deployment.chain, tx_hash, is_tx=True),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Burn failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Burn failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/transfer")
|
||||
async def transfer_ownership(request: Request, body: TransferOwnershipRequest, _=Depends(_verify_admin)):
|
||||
"""Transfer contract ownership to a new address."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(body.deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
tx_hash = await deployer.transfer_ownership(deployment.contract_address, body.new_owner)
|
||||
|
||||
await storage.update_status(
|
||||
body.deployment_id,
|
||||
"ownership_transferred",
|
||||
{"new_owner": body.new_owner, "transfer_tx": tx_hash},
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"tx_hash": tx_hash,
|
||||
"new_owner": body.new_owner,
|
||||
"explorer_url": _get_explorer_url(deployment.chain, tx_hash, is_tx=True),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Ownership transfer failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Transfer failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/renounce")
|
||||
async def renounce_ownership(request: Request, body: dict = Body(...), _=Depends(_verify_admin)):
|
||||
"""Renounce contract ownership (make immutable). WARNING: Irreversible."""
|
||||
deployment_id = body.get("deployment_id", "")
|
||||
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
tx_hash = await deployer.renounce_ownership(deployment.contract_address)
|
||||
|
||||
await storage.update_status(deployment_id, "ownership_renounced", {"renounce_tx": tx_hash})
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"tx_hash": tx_hash,
|
||||
"warning": "Ownership renounced. Contract is now immutable.",
|
||||
"explorer_url": _get_explorer_url(deployment.chain, tx_hash, is_tx=True),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Renounce failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Renounce failed: {e!s}") from e
|
||||
|
||||
|
||||
# ── Blacklist / Anti-Bot Endpoints ────────────────────────────
|
||||
|
||||
|
||||
@router.post("/blacklist")
|
||||
async def blacklist_add(request: Request, body: BlacklistRequest, _=Depends(_verify_admin)):
|
||||
"""Add an address to the token blacklist."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(body.deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
tx_hash = await deployer.blacklist_add(deployment.contract_address, body.address)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"tx_hash": tx_hash,
|
||||
"address": body.address,
|
||||
"action": "blacklisted",
|
||||
"explorer_url": _get_explorer_url(deployment.chain, tx_hash, is_tx=True),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Blacklist add failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Blacklist failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/unblacklist")
|
||||
async def blacklist_remove(request: Request, body: BlacklistRequest, _=Depends(_verify_admin)):
|
||||
"""Remove an address from the token blacklist."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(body.deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
tx_hash = await deployer.blacklist_remove(deployment.contract_address, body.address)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"tx_hash": tx_hash,
|
||||
"address": body.address,
|
||||
"action": "unblacklisted",
|
||||
"explorer_url": _get_explorer_url(deployment.chain, tx_hash, is_tx=True),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Unblacklist failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Unblacklist failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.get("/blacklist/check")
|
||||
async def check_blacklist(request: Request, deployment_id: str, address: str, _=Depends(_verify_admin)):
|
||||
"""Check if an address is blacklisted."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
is_blacklisted = await deployer.is_blacklisted(deployment.contract_address, address)
|
||||
|
||||
return {
|
||||
"address": address,
|
||||
"is_blacklisted": is_blacklisted,
|
||||
"contract": deployment.contract_address,
|
||||
"chain": deployment.chain,
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Blacklist check failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Check failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/trading")
|
||||
async def set_trading(request: Request, body: TradingToggleRequest, _=Depends(_verify_admin)):
|
||||
"""Enable or disable trading for the token."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(body.deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
tx_hash = await deployer.set_trading_enabled(deployment.contract_address, body.enabled)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"tx_hash": tx_hash,
|
||||
"trading_enabled": body.enabled,
|
||||
"explorer_url": _get_explorer_url(deployment.chain, tx_hash, is_tx=True),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Trading toggle failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Toggle failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/max-wallet")
|
||||
async def set_max_wallet(request: Request, body: SetLimitRequest, _=Depends(_verify_admin)):
|
||||
"""Set maximum wallet holding limit."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(body.deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
tx_hash = await deployer.set_max_wallet(deployment.contract_address, body.amount)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"tx_hash": tx_hash,
|
||||
"max_wallet": body.amount,
|
||||
"explorer_url": _get_explorer_url(deployment.chain, tx_hash, is_tx=True),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Max wallet set failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Set failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.post("/max-tx")
|
||||
async def set_max_tx(request: Request, body: SetLimitRequest, _=Depends(_verify_admin)):
|
||||
"""Set maximum transaction limit."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(body.deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
tx_hash = await deployer.set_max_tx(deployment.contract_address, body.amount)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"tx_hash": tx_hash,
|
||||
"max_tx": body.amount,
|
||||
"explorer_url": _get_explorer_url(deployment.chain, tx_hash, is_tx=True),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Max tx set failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Set failed: {e!s}") from e
|
||||
|
||||
|
||||
# ── Query Endpoints ───────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/list")
|
||||
async def list_deployments(request: Request, chain: str | None = None, limit: int = 100, _=Depends(_verify_admin)):
|
||||
"""List all token deployments."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployments = await storage.list_all(chain=chain, limit=limit)
|
||||
|
||||
return {
|
||||
"deployments": [d.to_dict() for d in deployments],
|
||||
"total": len(deployments),
|
||||
"chain_filter": chain,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"List failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"List failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.get("/{deployment_id}")
|
||||
async def get_deployment(request: Request, deployment_id: str, _=Depends(_verify_admin)):
|
||||
"""Get deployment details by ID."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
# Get on-chain info
|
||||
try:
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
token_info = await deployer.get_token_info(deployment.contract_address)
|
||||
except Exception:
|
||||
token_info = {}
|
||||
|
||||
return {
|
||||
"deployment": deployment.to_dict(),
|
||||
"on_chain_info": token_info,
|
||||
"explorer_url": _get_explorer_url(deployment.chain, deployment.contract_address),
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Get deployment failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Get failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.get("/{deployment_id}/balance/{wallet_address}")
|
||||
async def get_balance(request: Request, deployment_id: str, wallet_address: str, _=Depends(_verify_admin)):
|
||||
"""Get token balance for a specific wallet."""
|
||||
try:
|
||||
storage = await get_storage()
|
||||
deployment = await storage.get(deployment_id)
|
||||
if not deployment:
|
||||
raise HTTPException(status_code=404, detail="Deployment not found")
|
||||
|
||||
deployer = TokenDeployerFactory.get_deployer(deployment.chain)
|
||||
balance = await deployer.get_balance(deployment.contract_address, wallet_address)
|
||||
|
||||
return {
|
||||
"wallet": wallet_address,
|
||||
"balance": balance,
|
||||
"contract": deployment.contract_address,
|
||||
"chain": deployment.chain,
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Balance check failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Balance check failed: {e!s}") from e
|
||||
|
||||
|
||||
# ── Helpers ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _get_explorer_url(chain: str, address_or_tx: str, is_tx: bool = False) -> str:
|
||||
"""Get block explorer URL for a chain."""
|
||||
explorers = {
|
||||
"ethereum": "https://etherscan.io",
|
||||
"base": "https://basescan.org",
|
||||
"bsc": "https://bscscan.com",
|
||||
"solana": "https://solscan.io",
|
||||
"tron": "https://tronscan.org/#",
|
||||
}
|
||||
base = explorers.get(chain, "")
|
||||
if not base:
|
||||
return ""
|
||||
|
||||
if chain == "tron":
|
||||
path = "transaction" if is_tx else "contract"
|
||||
return f"{base}/{path}/{address_or_tx}"
|
||||
|
||||
path = "tx" if is_tx else "address"
|
||||
return f"{base}/{path}/{address_or_tx}"
|
||||
|
|
@ -1,58 +0,0 @@
|
|||
"""#2 SSE Streaming + #3 Provider Dashboard endpoints."""
|
||||
|
||||
import os
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter
|
||||
|
||||
router = APIRouter(prefix="/api/v1/databus", tags=["databus-extras"])
|
||||
|
||||
BACKEND = os.getenv("BACKEND_URL", "http://localhost:8000")
|
||||
|
||||
|
||||
@router.get("/providers/dashboard")
|
||||
async def provider_dashboard():
|
||||
"""Real-time provider health dashboard data - feed into Grafana."""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(
|
||||
f"{BACKEND}/api/v1/databus/providers/health",
|
||||
headers={"X-RMI-Key": os.getenv("RMI_INTERNAL_KEY", "rmi-internal-2026")},
|
||||
)
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
providers = data.get("providers", data)
|
||||
|
||||
# Format for Grafana
|
||||
panels = []
|
||||
for name, health in providers.items() if isinstance(providers, dict) else []:
|
||||
panels.append(
|
||||
{
|
||||
"provider": name,
|
||||
"status": "healthy" if health.get("healthy", True) else "degraded",
|
||||
"latency_ms": health.get("avg_latency_ms", 0),
|
||||
"error_rate": health.get("error_rate", 0),
|
||||
"circuit": health.get("circuit_state", "closed"),
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"providers": panels,
|
||||
"summary": {
|
||||
"total": len(panels),
|
||||
"healthy": sum(1 for p in panels if p["status"] == "healthy"),
|
||||
"degraded": sum(1 for p in panels if p["status"] != "healthy"),
|
||||
},
|
||||
}
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {"providers": [], "note": "Provider health API unavailable - check backend"}
|
||||
|
||||
|
||||
@router.get("/queue/stats")
|
||||
async def task_queue_stats():
|
||||
"""Background task queue statistics."""
|
||||
from app.core.task_queue import get_queue_stats
|
||||
|
||||
return await get_queue_stats()
|
||||
|
|
@ -1,76 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""#10 - DataBus Public API Gateway Router. Free tier + x402 paid tier."""
|
||||
|
||||
from fastapi import APIRouter, Header, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.databus_gateway import (
|
||||
SUPPORTED_CHAINS,
|
||||
TIER_LIMITS,
|
||||
_check_rate,
|
||||
databus_cross_chain,
|
||||
databus_get,
|
||||
verify_x402_payment,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/api/v1/databus", tags=["databus-gateway"])
|
||||
|
||||
|
||||
class ChainQuery(BaseModel):
|
||||
chain: str
|
||||
endpoint: str = "overview"
|
||||
params: dict | None = None
|
||||
|
||||
|
||||
class CrossChainQuery(BaseModel):
|
||||
query: str
|
||||
chains: list[str] | None = None
|
||||
|
||||
|
||||
@router.get("/chains")
|
||||
async def list_chains():
|
||||
"""List all supported chains with tier availability."""
|
||||
return {"chains": SUPPORTED_CHAINS, "total": len(SUPPORTED_CHAINS)}
|
||||
|
||||
|
||||
@router.get("/{chain}/{endpoint:path}")
|
||||
async def query_chain(chain: str, endpoint: str, x_rmi_key: str = Header(None), x_x402_sig: str = Header(None)):
|
||||
"""Query a single DataBus chain. Free tier with rate limiting, paid tier via x402 header."""
|
||||
tier = "free"
|
||||
if x_x402_sig:
|
||||
valid = await verify_x402_payment(x_x402_sig, f"{chain}/{endpoint}")
|
||||
if valid:
|
||||
tier = "pro"
|
||||
else:
|
||||
raise HTTPException(402, "Invalid x402 payment signature")
|
||||
|
||||
limits = TIER_LIMITS[tier]
|
||||
client_id = x_rmi_key or "anonymous"
|
||||
rate_key = f"databus:{client_id}:{tier}"
|
||||
|
||||
if not _check_rate(rate_key, limits["requests_per_hour"]):
|
||||
raise HTTPException(429, f"Rate limit exceeded ({limits['requests_per_hour']}/hr for {tier} tier)")
|
||||
|
||||
if chain not in SUPPORTED_CHAINS:
|
||||
raise HTTPException(404, f"Chain '{chain}' not supported. Use /api/v1/databus/chains for full list.")
|
||||
|
||||
result = await databus_get(chain, endpoint, tier=tier)
|
||||
return {
|
||||
"chain": result.chain,
|
||||
"data": result.data,
|
||||
"source": result.source,
|
||||
"cached": result.cached,
|
||||
"latency_ms": result.latency_ms,
|
||||
"tier": tier,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/cross-chain")
|
||||
async def cross_chain_query(query: CrossChainQuery):
|
||||
"""Query multiple DataBus chains in parallel."""
|
||||
results = await databus_cross_chain(query.query, query.chains)
|
||||
return {
|
||||
"query": query.query,
|
||||
"chains_queried": list(results.keys()),
|
||||
"results": {c: {"data": r.data, "cached": r.cached} for c, r in results.items()},
|
||||
}
|
||||
|
|
@ -1,841 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
RMI Supabase Database Client
|
||||
============================
|
||||
Persistent PostgreSQL storage via Supabase with Redis caching.
|
||||
Write-through: Supabase first, then Redis.
|
||||
Read-through: Redis first, Supabase on miss.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from collections.abc import Callable
|
||||
from datetime import UTC, datetime
|
||||
from functools import wraps
|
||||
from typing import Any
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load .env with override to ensure JWT keys win over stale Docker env vars
|
||||
load_dotenv("/app/.env", override=True)
|
||||
|
||||
from pydantic import BaseModel, Field # noqa: E402
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Lazy imports to avoid import-time failures ──
|
||||
try:
|
||||
from supabase import Client, create_client
|
||||
except ImportError as _imp_err:
|
||||
create_client = None
|
||||
Client = None
|
||||
logger.warning(f"[DB] supabase client not installed: {_imp_err}")
|
||||
|
||||
try:
|
||||
import redis.asyncio as redis
|
||||
except ImportError:
|
||||
redis = None
|
||||
|
||||
# ── SQL to ensure missing tables exist ──
|
||||
# Run these in the Supabase SQL Editor if the tables are missing.
|
||||
ENSURE_TABLES_SQL = """
|
||||
-- Users / Profiles (extends auth.users for wallet-auth users)
|
||||
CREATE TABLE IF NOT EXISTS profiles (
|
||||
id TEXT PRIMARY KEY,
|
||||
email TEXT,
|
||||
wallet_address TEXT UNIQUE,
|
||||
role TEXT DEFAULT 'USER',
|
||||
tier TEXT DEFAULT 'FREE',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
xp INTEGER DEFAULT 0,
|
||||
level INTEGER DEFAULT 1,
|
||||
display_name TEXT,
|
||||
scans_remaining INTEGER DEFAULT 5,
|
||||
total_scans_used INTEGER DEFAULT 0,
|
||||
reputation_score INTEGER DEFAULT 0
|
||||
);
|
||||
ALTER TABLE profiles ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY "Profiles public read" ON profiles FOR SELECT USING (true);
|
||||
CREATE POLICY "Profiles admin write" ON profiles FOR ALL USING (
|
||||
EXISTS (SELECT 1 FROM auth.users WHERE auth.users.id = auth.uid() AND auth.users.role = 'admin')
|
||||
);
|
||||
|
||||
-- Contract Audits
|
||||
CREATE TABLE IF NOT EXISTS contract_audits (
|
||||
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
|
||||
address TEXT NOT NULL,
|
||||
chain TEXT DEFAULT 'solana',
|
||||
risk_score NUMERIC DEFAULT 0,
|
||||
findings JSONB DEFAULT '[]',
|
||||
ai_analysis JSONB DEFAULT '{}',
|
||||
audited_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
ALTER TABLE contract_audits ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY "Audits public read" ON contract_audits FOR SELECT USING (true);
|
||||
CREATE POLICY "Audits admin write" ON contract_audits FOR ALL USING (
|
||||
EXISTS (SELECT 1 FROM auth.users WHERE auth.users.id = auth.uid() AND auth.users.role = 'admin')
|
||||
);
|
||||
|
||||
-- Trenches Posts (community board)
|
||||
CREATE TABLE IF NOT EXISTS trenches_posts (
|
||||
id TEXT PRIMARY KEY,
|
||||
title TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
category TEXT DEFAULT 'discussion',
|
||||
author_id TEXT,
|
||||
upvotes INTEGER DEFAULT 0,
|
||||
comments INTEGER DEFAULT 0,
|
||||
tags TEXT[] DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
ALTER TABLE trenches_posts ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY "Posts public read" ON trenches_posts FOR SELECT USING (true);
|
||||
CREATE POLICY "Posts owner write" ON trenches_posts FOR ALL USING (
|
||||
auth.uid()::text = author_id OR
|
||||
EXISTS (SELECT 1 FROM auth.users WHERE auth.users.id = auth.uid() AND auth.users.role = 'admin')
|
||||
);
|
||||
|
||||
-- Gamification Profiles
|
||||
CREATE TABLE IF NOT EXISTS gamification_profiles (
|
||||
user_id TEXT PRIMARY KEY REFERENCES profiles(id) ON DELETE CASCADE,
|
||||
xp INTEGER DEFAULT 0,
|
||||
level INTEGER DEFAULT 1,
|
||||
badges TEXT[] DEFAULT '{}',
|
||||
scans_count INTEGER DEFAULT 0,
|
||||
posts_count INTEGER DEFAULT 0,
|
||||
updated_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
ALTER TABLE gamification_profiles ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY "Gamification public read" ON gamification_profiles FOR SELECT USING (true);
|
||||
CREATE POLICY "Gamification owner write" ON gamification_profiles FOR ALL USING (
|
||||
auth.uid()::text = user_id OR
|
||||
EXISTS (SELECT 1 FROM auth.users WHERE auth.users.id = auth.uid() AND auth.users.role = 'admin')
|
||||
);
|
||||
|
||||
-- Indexes
|
||||
CREATE INDEX IF NOT EXISTS idx_profiles_wallet ON profiles(wallet_address);
|
||||
CREATE INDEX IF NOT EXISTS idx_contract_audits_address ON contract_audits(address);
|
||||
CREATE INDEX IF NOT EXISTS idx_trenches_posts_category ON trenches_posts(category);
|
||||
CREATE INDEX IF NOT EXISTS idx_trenches_posts_created ON trenches_posts(created_at DESC);
|
||||
"""
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# RETRY UTILITIES
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
def _async_retry(max_retries: int = 3, delay: float = 0.5, backoff: float = 2.0):
|
||||
"""Simple async retry decorator for Supabase operations."""
|
||||
|
||||
def decorator(func: Callable):
|
||||
@wraps(func)
|
||||
async def wrapper(*args, **kwargs):
|
||||
last_exc = None
|
||||
wait = delay
|
||||
for attempt in range(max_retries + 1):
|
||||
try:
|
||||
return await func(*args, **kwargs)
|
||||
except Exception as exc:
|
||||
last_exc = exc
|
||||
if attempt >= max_retries:
|
||||
raise
|
||||
logger.warning(f"[DB] Retry {attempt + 1}/{max_retries} for {func.__name__}: {exc}")
|
||||
await asyncio.sleep(wait)
|
||||
wait *= backoff
|
||||
raise last_exc
|
||||
|
||||
return wrapper
|
||||
|
||||
return decorator
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# PYDANTIC MODELS
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
class User(BaseModel):
|
||||
id: str
|
||||
email: str | None = None
|
||||
wallet_address: str | None = None
|
||||
role: str = "USER"
|
||||
tier: str = "FREE"
|
||||
created_at: str | None = None
|
||||
xp: int = 0
|
||||
level: int = 1
|
||||
|
||||
|
||||
class InvestigationCase(BaseModel):
|
||||
id: str
|
||||
target: str
|
||||
type: str = "wallet"
|
||||
status: str = "open"
|
||||
evidence: list[str] = Field(default_factory=list)
|
||||
agents_assigned: list[str] = Field(default_factory=list)
|
||||
created_at: str | None = None
|
||||
updated_at: str | None = None
|
||||
risk_score: float = 0.0
|
||||
findings: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class WalletScan(BaseModel):
|
||||
id: str | None = None
|
||||
address: str
|
||||
chain: str = "solana"
|
||||
risk_score: float = 0.0
|
||||
findings: dict[str, Any] = Field(default_factory=dict)
|
||||
scanned_at: str | None = None
|
||||
user_id: str | None = None
|
||||
|
||||
|
||||
class ContractAudit(BaseModel):
|
||||
id: str | None = None
|
||||
address: str
|
||||
chain: str = "solana"
|
||||
risk_score: float = 0.0
|
||||
findings: list[dict[str, Any]] = Field(default_factory=list)
|
||||
ai_analysis: dict[str, Any] = Field(default_factory=dict)
|
||||
audited_at: str | None = None
|
||||
|
||||
|
||||
class TrenchesPost(BaseModel):
|
||||
id: str | None = None
|
||||
title: str
|
||||
content: str
|
||||
category: str = "discussion"
|
||||
author_id: str | None = None
|
||||
upvotes: int = 0
|
||||
comments: int = 0
|
||||
tags: list[str] = Field(default_factory=list)
|
||||
created_at: str | None = None
|
||||
|
||||
|
||||
class Alert(BaseModel):
|
||||
id: str | None = None
|
||||
token_address: str
|
||||
types: list[str] = Field(default_factory=list)
|
||||
webhook_url: str | None = None
|
||||
active: bool = True
|
||||
created_at: str | None = None
|
||||
|
||||
|
||||
class GamificationProfile(BaseModel):
|
||||
user_id: str
|
||||
xp: int = 0
|
||||
level: int = 1
|
||||
badges: list[str] = Field(default_factory=list)
|
||||
scans_count: int = 0
|
||||
posts_count: int = 0
|
||||
updated_at: str | None = None
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# SUPABASE CLIENT
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
class SupabaseClient:
|
||||
"""Wrapped Supabase client with connection pooling, retries, and helpers."""
|
||||
|
||||
def __init__(self, url: str | None = None, key: str | None = None):
|
||||
self.url = url or os.getenv("SUPABASE_URL", "")
|
||||
# Fall back through env var variants
|
||||
self.key = (
|
||||
key or os.getenv("SUPABASE_SERVICE_KEY") or os.getenv("SUPABASE_KEY") or os.getenv("SUPABASE_ANON_KEY", "")
|
||||
)
|
||||
self._client: Any | None = None
|
||||
|
||||
def _ensure_client(self) -> Any:
|
||||
if self._client is None:
|
||||
if create_client is None:
|
||||
raise RuntimeError("supabase package is not installed")
|
||||
if not self.url or not self.key:
|
||||
raise RuntimeError("SUPABASE_URL and SUPABASE_KEY must be set")
|
||||
self._client = create_client(self.url, self.key)
|
||||
return self._client
|
||||
|
||||
@property
|
||||
def client(self) -> Any:
|
||||
return self._ensure_client()
|
||||
|
||||
# ── Auth helpers ──
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def sign_up(self, email: str, password: str) -> dict[str, Any]:
|
||||
"""Register a new user via Supabase Auth."""
|
||||
result = self.client.auth.sign_up({"email": email, "password": password})
|
||||
return result.model_dump() if hasattr(result, "model_dump") else dict(result)
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def sign_in(self, email: str, password: str) -> dict[str, Any]:
|
||||
"""Sign in via Supabase Auth."""
|
||||
result = self.client.auth.sign_in_with_password({"email": email, "password": password})
|
||||
return result.model_dump() if hasattr(result, "model_dump") else dict(result)
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def get_user(self, token: str) -> dict[str, Any] | None:
|
||||
"""Get user by JWT token."""
|
||||
result = self.client.auth.get_user(token)
|
||||
if result and result.user:
|
||||
return {
|
||||
"id": result.user.id,
|
||||
"email": result.user.email,
|
||||
}
|
||||
return None
|
||||
|
||||
# ── Table access ──
|
||||
|
||||
def table(self, name: str) -> Any:
|
||||
return self.client.table(name)
|
||||
|
||||
# ── SQL execution (requires service role or elevated privileges) ──
|
||||
|
||||
@_async_retry(max_retries=2)
|
||||
async def execute_sql(self, sql: str) -> Any:
|
||||
"""Execute raw SQL via RPC (requires a stored procedure or service role)."""
|
||||
# Prefer using migrations; this is a fallback.
|
||||
return self.client.rpc("exec_sql", {"query": sql}).execute()
|
||||
|
||||
@_async_retry(max_retries=2)
|
||||
async def ensure_tables(self) -> None:
|
||||
"""Best-effort table creation. Requires exec_sql RPC or run SQL manually."""
|
||||
try:
|
||||
await self.execute_sql(ENSURE_TABLES_SQL)
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] ensure_tables failed (tables may already exist or RPC missing): {exc}")
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# REPOSITORIES
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
class _BaseRepo:
|
||||
def __init__(self, db: SupabaseClient):
|
||||
self.db = db
|
||||
|
||||
def _now(self) -> str:
|
||||
return datetime.now(UTC).isoformat()
|
||||
|
||||
def _to_dict(self, model: BaseModel, exclude_none: bool = True) -> dict[str, Any]:
|
||||
d = model.model_dump()
|
||||
if exclude_none:
|
||||
d = {k: v for k, v in d.items() if v is not None}
|
||||
return d
|
||||
|
||||
|
||||
class UserRepository(_BaseRepo):
|
||||
TABLE = "profiles"
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def create(self, user: User) -> dict[str, Any]:
|
||||
data = self._to_dict(user)
|
||||
if not data.get("created_at"):
|
||||
data["created_at"] = self._now()
|
||||
result = self.db.table(self.TABLE).insert(data).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def get(self, user_id: str) -> dict[str, Any] | None:
|
||||
result = self.db.table(self.TABLE).select("*").eq("id", user_id).execute()
|
||||
return result.data[0] if result.data else None
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def get_by_wallet(self, wallet_address: str) -> dict[str, Any] | None:
|
||||
result = self.db.table(self.TABLE).select("*").eq("wallet_address", wallet_address.lower()).execute()
|
||||
return result.data[0] if result.data else None
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def list(self, limit: int = 100) -> list[dict[str, Any]]:
|
||||
result = self.db.table(self.TABLE).select("*").limit(limit).execute()
|
||||
return result.data or []
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def update(self, user_id: str, updates: dict[str, Any]) -> dict[str, Any]:
|
||||
result = self.db.table(self.TABLE).update(updates).eq("id", user_id).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def upsert(self, user: User) -> dict[str, Any]:
|
||||
data = self._to_dict(user)
|
||||
if not data.get("created_at"):
|
||||
data["created_at"] = self._now()
|
||||
result = self.db.table(self.TABLE).upsert(data, on_conflict="id").execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def delete(self, user_id: str) -> bool:
|
||||
self.db.table(self.TABLE).delete().eq("id", user_id).execute()
|
||||
return True
|
||||
|
||||
|
||||
class InvestigationCaseRepository(_BaseRepo):
|
||||
TABLE = "cases"
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def create(self, case: InvestigationCase) -> dict[str, Any]:
|
||||
data = self._to_dict(case)
|
||||
if not data.get("created_at"):
|
||||
data["created_at"] = self._now()
|
||||
if not data.get("updated_at"):
|
||||
data["updated_at"] = self._now()
|
||||
result = self.db.table(self.TABLE).insert(data).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def get(self, case_id: str) -> dict[str, Any] | None:
|
||||
result = self.db.table(self.TABLE).select("*").eq("id", case_id).execute()
|
||||
return result.data[0] if result.data else None
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def list(self, limit: int = 100, status: str | None = None) -> list[dict[str, Any]]:
|
||||
query = self.db.table(self.TABLE).select("*")
|
||||
if status:
|
||||
query = query.eq("status", status)
|
||||
result = query.limit(limit).execute()
|
||||
return result.data or []
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def update(self, case_id: str, updates: dict[str, Any]) -> dict[str, Any]:
|
||||
updates["updated_at"] = self._now()
|
||||
result = self.db.table(self.TABLE).update(updates).eq("id", case_id).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def delete(self, case_id: str) -> bool:
|
||||
self.db.table(self.TABLE).delete().eq("id", case_id).execute()
|
||||
return True
|
||||
|
||||
|
||||
class WalletScanRepository(_BaseRepo):
|
||||
TABLE = "wallet_intel"
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def create(self, scan: WalletScan) -> dict[str, Any]:
|
||||
data = self._to_dict(scan)
|
||||
# Map findings to transactions JSONB for schema compatibility
|
||||
if "findings" in data:
|
||||
data["transactions"] = data.pop("findings")
|
||||
if not data.get("scanned_at"):
|
||||
data["last_seen"] = self._now()
|
||||
result = self.db.table(self.TABLE).insert(data).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def get(self, scan_id: str) -> dict[str, Any] | None:
|
||||
result = self.db.table(self.TABLE).select("*").eq("id", scan_id).execute()
|
||||
return result.data[0] if result.data else None
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def get_by_address(self, address: str) -> dict[str, Any] | None:
|
||||
result = self.db.table(self.TABLE).select("*").eq("address", address).execute()
|
||||
return result.data[0] if result.data else None
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def list(self, limit: int = 100, chain: str | None = None) -> list[dict[str, Any]]:
|
||||
query = self.db.table(self.TABLE).select("*")
|
||||
if chain:
|
||||
query = query.eq("chain", chain)
|
||||
result = query.limit(limit).execute()
|
||||
return result.data or []
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def update(self, scan_id: str, updates: dict[str, Any]) -> dict[str, Any]:
|
||||
if "findings" in updates:
|
||||
updates["transactions"] = updates.pop("findings")
|
||||
updates["updated_at"] = self._now()
|
||||
result = self.db.table(self.TABLE).update(updates).eq("id", scan_id).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def delete(self, scan_id: str) -> bool:
|
||||
self.db.table(self.TABLE).delete().eq("id", scan_id).execute()
|
||||
return True
|
||||
|
||||
|
||||
class ContractAuditRepository(_BaseRepo):
|
||||
TABLE = "contract_audits"
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def create(self, audit: ContractAudit) -> dict[str, Any]:
|
||||
data = self._to_dict(audit)
|
||||
if not data.get("audited_at"):
|
||||
data["audited_at"] = self._now()
|
||||
result = self.db.table(self.TABLE).insert(data).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def get(self, audit_id: str) -> dict[str, Any] | None:
|
||||
result = self.db.table(self.TABLE).select("*").eq("id", audit_id).execute()
|
||||
return result.data[0] if result.data else None
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def list(self, limit: int = 100, chain: str | None = None) -> list[dict[str, Any]]:
|
||||
query = self.db.table(self.TABLE).select("*")
|
||||
if chain:
|
||||
query = query.eq("chain", chain)
|
||||
result = query.limit(limit).execute()
|
||||
return result.data or []
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def update(self, audit_id: str, updates: dict[str, Any]) -> dict[str, Any]:
|
||||
result = self.db.table(self.TABLE).update(updates).eq("id", audit_id).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def delete(self, audit_id: str) -> bool:
|
||||
self.db.table(self.TABLE).delete().eq("id", audit_id).execute()
|
||||
return True
|
||||
|
||||
|
||||
class TrenchesPostRepository(_BaseRepo):
|
||||
TABLE = "trenches_posts"
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def create(self, post: TrenchesPost) -> dict[str, Any]:
|
||||
data = self._to_dict(post)
|
||||
if not data.get("created_at"):
|
||||
data["created_at"] = self._now()
|
||||
result = self.db.table(self.TABLE).insert(data).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def get(self, post_id: str) -> dict[str, Any] | None:
|
||||
result = self.db.table(self.TABLE).select("*").eq("id", post_id).execute()
|
||||
return result.data[0] if result.data else None
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def list(self, limit: int = 50, category: str | None = None) -> list[dict[str, Any]]:
|
||||
query = self.db.table(self.TABLE).select("*")
|
||||
if category and category != "all":
|
||||
query = query.eq("category", category)
|
||||
result = query.order("created_at", desc=True).limit(limit).execute()
|
||||
return result.data or []
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def update(self, post_id: str, updates: dict[str, Any]) -> dict[str, Any]:
|
||||
result = self.db.table(self.TABLE).update(updates).eq("id", post_id).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def delete(self, post_id: str) -> bool:
|
||||
self.db.table(self.TABLE).delete().eq("id", post_id).execute()
|
||||
return True
|
||||
|
||||
|
||||
class AlertRepository(_BaseRepo):
|
||||
TABLE = "alerts"
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def create(self, alert: Alert) -> dict[str, Any]:
|
||||
data = self._to_dict(alert)
|
||||
# Map types -> alert_types for schema compatibility
|
||||
if "types" in data:
|
||||
data["alert_types"] = data.pop("types")
|
||||
if not data.get("created_at"):
|
||||
data["created_at"] = self._now()
|
||||
result = self.db.table(self.TABLE).insert(data).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def get(self, alert_id: str) -> dict[str, Any] | None:
|
||||
result = self.db.table(self.TABLE).select("*").eq("id", alert_id).execute()
|
||||
return result.data[0] if result.data else None
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def list(self, limit: int = 100, active_only: bool = True) -> list[dict[str, Any]]:
|
||||
query = self.db.table(self.TABLE).select("*")
|
||||
if active_only:
|
||||
query = query.eq("active", True)
|
||||
result = query.limit(limit).execute()
|
||||
return result.data or []
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def update(self, alert_id: str, updates: dict[str, Any]) -> dict[str, Any]:
|
||||
if "types" in updates:
|
||||
updates["alert_types"] = updates.pop("types")
|
||||
result = self.db.table(self.TABLE).update(updates).eq("id", alert_id).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def delete(self, alert_id: str) -> bool:
|
||||
self.db.table(self.TABLE).delete().eq("id", alert_id).execute()
|
||||
return True
|
||||
|
||||
|
||||
class GamificationProfileRepository(_BaseRepo):
|
||||
TABLE = "gamification_profiles"
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def create(self, profile: GamificationProfile) -> dict[str, Any]:
|
||||
data = self._to_dict(profile)
|
||||
if not data.get("updated_at"):
|
||||
data["updated_at"] = self._now()
|
||||
result = self.db.table(self.TABLE).insert(data).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def get(self, user_id: str) -> dict[str, Any] | None:
|
||||
result = self.db.table(self.TABLE).select("*").eq("user_id", user_id).execute()
|
||||
return result.data[0] if result.data else None
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def list(self, limit: int = 100) -> list[dict[str, Any]]:
|
||||
result = self.db.table(self.TABLE).select("*").limit(limit).execute()
|
||||
return result.data or []
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def update(self, user_id: str, updates: dict[str, Any]) -> dict[str, Any]:
|
||||
updates["updated_at"] = self._now()
|
||||
result = self.db.table(self.TABLE).update(updates).eq("user_id", user_id).execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def upsert(self, profile: GamificationProfile) -> dict[str, Any]:
|
||||
data = self._to_dict(profile)
|
||||
if not data.get("updated_at"):
|
||||
data["updated_at"] = self._now()
|
||||
result = self.db.table(self.TABLE).upsert(data, on_conflict="user_id").execute()
|
||||
return result.data[0] if result.data else {}
|
||||
|
||||
@_async_retry(max_retries=3)
|
||||
async def delete(self, user_id: str) -> bool:
|
||||
self.db.table(self.TABLE).delete().eq("user_id", user_id).execute()
|
||||
return True
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# UNIFIED DB INTERFACE
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
class RmiDatabase:
|
||||
"""Unified interface: Supabase persistence + Redis cache."""
|
||||
|
||||
def __init__(self, supabase_url: str | None = None, supabase_key: str | None = None):
|
||||
self.db = SupabaseClient(url=supabase_url, key=supabase_key)
|
||||
self.users = UserRepository(self.db)
|
||||
self.cases = InvestigationCaseRepository(self.db)
|
||||
self.wallet_scans = WalletScanRepository(self.db)
|
||||
self.contract_audits = ContractAuditRepository(self.db)
|
||||
self.trenches_posts = TrenchesPostRepository(self.db)
|
||||
self.alerts = AlertRepository(self.db)
|
||||
self.gamification = GamificationProfileRepository(self.db)
|
||||
|
||||
async def ensure_tables(self) -> None:
|
||||
await self.db.ensure_tables()
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# REDIS CACHE HELPERS
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
_CACHE_TTL = 300 # 5 minutes default
|
||||
|
||||
|
||||
async def _get_redis() -> Any | None:
|
||||
"""Lazy Redis client (mirrors main.py pattern)."""
|
||||
if redis is None:
|
||||
return None
|
||||
try:
|
||||
import os
|
||||
|
||||
host = os.getenv("REDIS_HOST", "127.0.0.1")
|
||||
port = int(os.getenv("REDIS_PORT", "6379"))
|
||||
password = os.getenv("REDIS_PASSWORD") or None
|
||||
db = int(os.getenv("REDIS_DB", "0"))
|
||||
return redis.Redis(host=host, port=port, password=password, db=db, decode_responses=True)
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] Redis init failed: {exc}")
|
||||
return None
|
||||
|
||||
|
||||
async def cache_set(key: str, value: Any, ttl: int = _CACHE_TTL) -> None:
|
||||
r = await _get_redis()
|
||||
if r is None:
|
||||
return
|
||||
try:
|
||||
await r.set(key, json.dumps(value), ex=ttl)
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] cache_set failed: {exc}")
|
||||
|
||||
|
||||
async def cache_get(key: str) -> Any | None:
|
||||
r = await _get_redis()
|
||||
if r is None:
|
||||
return None
|
||||
try:
|
||||
raw = await r.get(key)
|
||||
return json.loads(raw) if raw else None
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] cache_get failed: {exc}")
|
||||
return None
|
||||
|
||||
|
||||
async def cache_delete(key: str) -> None:
|
||||
r = await _get_redis()
|
||||
if r is None:
|
||||
return
|
||||
try:
|
||||
await r.delete(key)
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] cache_delete failed: {exc}")
|
||||
|
||||
|
||||
async def cache_hash_set(hash_name: str, field: str, value: Any) -> None:
|
||||
r = await _get_redis()
|
||||
if r is None:
|
||||
return
|
||||
try:
|
||||
await r.hset(hash_name, field, json.dumps(value))
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] cache_hash_set failed: {exc}")
|
||||
|
||||
|
||||
async def cache_hash_get_all(hash_name: str) -> dict[str, Any]:
|
||||
r = await _get_redis()
|
||||
if r is None:
|
||||
return {}
|
||||
try:
|
||||
raw = await r.hgetall(hash_name)
|
||||
return {k: json.loads(v) for k, v in raw.items()}
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] cache_hash_get_all failed: {exc}")
|
||||
return {}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# MIGRATION HELPER
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
async def sync_redis_to_supabase() -> dict[str, int]:
|
||||
"""
|
||||
Read all data from Redis hashes and upsert into Supabase tables.
|
||||
Returns counts of synced records per table.
|
||||
"""
|
||||
counts = {"cases": 0, "trenches_posts": 0, "alerts": 0, "tasks": 0}
|
||||
r = await _get_redis()
|
||||
if r is None:
|
||||
logger.warning("[DB] Redis unavailable; nothing to migrate")
|
||||
return counts
|
||||
|
||||
db = RmiDatabase()
|
||||
|
||||
# ── Cases ──
|
||||
try:
|
||||
cases_raw = await r.hgetall("rmi:cases")
|
||||
for case_json in cases_raw.values():
|
||||
try:
|
||||
data = json.loads(case_json)
|
||||
case = InvestigationCase(
|
||||
id=data.get("id", ""),
|
||||
target=data.get("target", ""),
|
||||
type=data.get("type", "wallet"),
|
||||
status=data.get("status", "open"),
|
||||
evidence=data.get("evidence") or data.get("evidence", []),
|
||||
agents_assigned=data.get("agents_assigned") or data.get("agents_assigned", []),
|
||||
created_at=data.get("created_at") or data.get("created"),
|
||||
updated_at=data.get("updated_at") or data.get("updated") or data.get("created"),
|
||||
risk_score=float(data.get("risk_score", 0)),
|
||||
findings=data.get("findings") or {},
|
||||
)
|
||||
await db.cases.create(case)
|
||||
counts["cases"] += 1
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] Failed to migrate case: {exc}")
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] Cases migration error: {exc}")
|
||||
|
||||
# ── Trenches Posts ──
|
||||
try:
|
||||
posts_raw = await r.hgetall("rmi:trenches:posts")
|
||||
for post_json in posts_raw.values():
|
||||
try:
|
||||
data = json.loads(post_json)
|
||||
post = TrenchesPost(
|
||||
id=data.get("id"),
|
||||
title=data.get("title", ""),
|
||||
content=data.get("content", ""),
|
||||
category=data.get("category", "discussion"),
|
||||
author_id=data.get("author_id"),
|
||||
upvotes=int(data.get("upvotes", 0)),
|
||||
comments=int(data.get("comments", 0)),
|
||||
tags=data.get("tags") or [],
|
||||
created_at=data.get("created_at") or data.get("created"),
|
||||
)
|
||||
await db.trenches_posts.create(post)
|
||||
counts["trenches_posts"] += 1
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] Failed to migrate post: {exc}")
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] Posts migration error: {exc}")
|
||||
|
||||
# ── Alerts ──
|
||||
try:
|
||||
alerts_raw = await r.hgetall("rmi:alerts")
|
||||
for alert_json in alerts_raw.values():
|
||||
try:
|
||||
data = json.loads(alert_json)
|
||||
alert = Alert(
|
||||
id=data.get("id"),
|
||||
token_address=data.get("token") or data.get("token_address", ""),
|
||||
types=data.get("types") or data.get("alert_types") or [],
|
||||
webhook_url=data.get("webhook") or data.get("webhook_url"),
|
||||
active=data.get("active", True),
|
||||
created_at=data.get("created_at") or data.get("created"),
|
||||
)
|
||||
await db.alerts.create(alert)
|
||||
counts["alerts"] += 1
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] Failed to migrate alert: {exc}")
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] Alerts migration error: {exc}")
|
||||
|
||||
# ── Tasks (migrate as cases with type=task for traceability) ──
|
||||
try:
|
||||
tasks_raw = await r.hgetall("rmi:tasks")
|
||||
for task_json in tasks_raw.values():
|
||||
try:
|
||||
data = json.loads(task_json)
|
||||
case = InvestigationCase(
|
||||
id=data.get("id", ""),
|
||||
target=data.get("command", "task"),
|
||||
type="task",
|
||||
status=data.get("status", "open"),
|
||||
evidence=[],
|
||||
agents_assigned=[data.get("agent", "")] if data.get("agent") else [],
|
||||
created_at=data.get("created") or data.get("created_at"),
|
||||
updated_at=data.get("created") or data.get("created_at"),
|
||||
risk_score=0.0,
|
||||
findings={
|
||||
"context": data.get("context", {}),
|
||||
"priority": data.get("priority", "normal"),
|
||||
},
|
||||
)
|
||||
await db.cases.create(case)
|
||||
counts["tasks"] += 1
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] Failed to migrate task: {exc}")
|
||||
except Exception as exc:
|
||||
logger.warning(f"[DB] Tasks migration error: {exc}")
|
||||
|
||||
logger.info(f"[DB] Migration complete: {counts}")
|
||||
return counts
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# SINGLETON INSTANCE
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
|
||||
_rmi_db: RmiDatabase | None = None
|
||||
|
||||
|
||||
def get_db() -> RmiDatabase:
|
||||
global _rmi_db
|
||||
if _rmi_db is None:
|
||||
_rmi_db = RmiDatabase()
|
||||
return _rmi_db
|
||||
|
|
@ -1,333 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
FastAPI Endpoint for Degen Security Scanner
|
||||
Mounts at: /degen-scan/*
|
||||
Integrates with RMI backend main.py
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
# Add the app directory to path
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from app.degen_security_scanner import (
|
||||
DegenSecurityScanner,
|
||||
SecurityReport,
|
||||
scan_token,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ─── Pydantic Models ──────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class DegenScanRequest(BaseModel):
|
||||
token_address: str = Field(..., min_length=32, max_length=48, description="Token mint address")
|
||||
quick: bool = Field(False, description="Skip slow operations for faster response")
|
||||
|
||||
|
||||
class DegenScanResponse(BaseModel):
|
||||
success: bool
|
||||
token_address: str
|
||||
token_name: str
|
||||
token_symbol: str
|
||||
rug_pull_probability: int
|
||||
overall_risk: str
|
||||
confidence_level: str
|
||||
mint_authority_renounced: bool
|
||||
freeze_authority_renounced: bool
|
||||
top_10_concentration: float
|
||||
total_holders: int
|
||||
total_liquidity_usd: float
|
||||
bundler_detected: bool
|
||||
bundler_wallet_count: int
|
||||
sniper_count: int
|
||||
honeypot_detected: bool
|
||||
lp_locked: bool
|
||||
exchange_funding_pct: float
|
||||
critical_warnings: list
|
||||
recommendations: list
|
||||
full_report: dict[str, Any]
|
||||
scan_duration_ms: int
|
||||
timestamp: str
|
||||
|
||||
|
||||
class BundleCheckResponse(BaseModel):
|
||||
token_address: str
|
||||
bundler_detected: bool
|
||||
bundler_wallets: list
|
||||
bundler_pattern: str
|
||||
bundler_supply_pct: float
|
||||
confidence: float
|
||||
details: dict[str, Any]
|
||||
|
||||
|
||||
class FundingTraceResponse(BaseModel):
|
||||
token_address: str
|
||||
deployer: str
|
||||
funding_sources: list
|
||||
exchange_funding_pct: float
|
||||
organic_funding_pct: float
|
||||
mixer_detected: bool
|
||||
risk_level: str
|
||||
trace_graph: list
|
||||
|
||||
|
||||
# ─── Cache Layer ──────────────────────────────────────────────────────────────
|
||||
|
||||
_scan_cache: dict[str, dict] = {}
|
||||
CACHE_TTL_SECONDS = 300 # 5 minute cache
|
||||
|
||||
|
||||
def _get_cached(token: str) -> dict | None:
|
||||
"""Get cached scan result if still valid."""
|
||||
entry = _scan_cache.get(token)
|
||||
if entry:
|
||||
age = (datetime.utcnow() - entry["cached_at"]).total_seconds()
|
||||
if age < CACHE_TTL_SECONDS:
|
||||
return entry["data"]
|
||||
return None
|
||||
|
||||
|
||||
def _set_cache(token: str, data: dict):
|
||||
"""Cache scan result."""
|
||||
_scan_cache[token] = {
|
||||
"data": data,
|
||||
"cached_at": datetime.utcnow(),
|
||||
}
|
||||
|
||||
|
||||
# ─── Router ───────────────────────────────────────────────────────────────────
|
||||
|
||||
router = APIRouter(prefix="/degen-scan", tags=["Degen Security Scanner"])
|
||||
|
||||
|
||||
@router.post("/full", response_model=DegenScanResponse)
|
||||
async def full_degen_scan(request: DegenScanRequest):
|
||||
"""
|
||||
Run a comprehensive DEGEN security scan on any Solana token.
|
||||
|
||||
Analyzes 15+ scam indicators including:
|
||||
- Bundler detection, gas funding traces, exchange funding %
|
||||
- LP analysis, sniper metrics, insider trading detection
|
||||
- Wash trading, dev wallet forensics, authority renouncement
|
||||
- Tax detection, honeypot detection, holder concentration
|
||||
|
||||
Returns a RUG PULL PROBABILITY score (0-100).
|
||||
"""
|
||||
token = request.token_address
|
||||
|
||||
# Check cache
|
||||
cached = _get_cached(token)
|
||||
if cached:
|
||||
return DegenScanResponse(**cached)
|
||||
|
||||
try:
|
||||
report = await scan_token(token, quick=request.quick)
|
||||
|
||||
response = {
|
||||
"success": True,
|
||||
"token_address": report.token_address,
|
||||
"token_name": report.token_name,
|
||||
"token_symbol": report.token_symbol,
|
||||
"rug_pull_probability": report.rug_pull_probability,
|
||||
"overall_risk": report.overall_risk,
|
||||
"confidence_level": report.confidence_level,
|
||||
"mint_authority_renounced": report.mint_authority_renounced,
|
||||
"freeze_authority_renounced": report.freeze_authority_renounced,
|
||||
"top_10_concentration": report.top_10_concentration,
|
||||
"total_holders": report.total_holders,
|
||||
"total_liquidity_usd": report.total_liquidity_usd,
|
||||
"bundler_detected": report.bundler_detected,
|
||||
"bundler_wallet_count": len(report.bundler_wallets),
|
||||
"sniper_count": report.sniper_count,
|
||||
"honeypot_detected": report.honeypot_detected,
|
||||
"lp_locked": any(p.lp_locked_percentage > 80 for p in report.liquidity_pools),
|
||||
"exchange_funding_pct": report.exchange_funding_percentage,
|
||||
"critical_warnings": report.critical_warnings,
|
||||
"recommendations": report.recommendations,
|
||||
"full_report": report.to_dict(),
|
||||
"scan_duration_ms": report.scan_duration_ms,
|
||||
"timestamp": datetime.utcnow().isoformat(),
|
||||
}
|
||||
|
||||
_set_cache(token, response)
|
||||
return DegenScanResponse(**response)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Degen scan failed for {token}: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Scan failed: {e!s}") from e
|
||||
|
||||
|
||||
@router.get("/quick")
|
||||
async def quick_degen_scan(
|
||||
token: str = Query(..., min_length=32, max_length=48),
|
||||
):
|
||||
"""
|
||||
Quick GET endpoint for instant risk assessment.
|
||||
Perfect for Telegram bot integration.
|
||||
"""
|
||||
cached = _get_cached(token)
|
||||
if cached:
|
||||
return cached
|
||||
|
||||
report = await scan_token(token, quick=True)
|
||||
|
||||
response = {
|
||||
"success": True,
|
||||
"token": f"{report.token_name} (${report.token_symbol})",
|
||||
"rug_probability": report.rug_pull_probability,
|
||||
"risk": report.overall_risk,
|
||||
"confidence": report.confidence_level,
|
||||
"mint_renounced": "✅" if report.mint_authority_renounced else "❌",
|
||||
"freeze_renounced": "✅" if report.freeze_authority_renounced else "❌",
|
||||
"top_10_pct": report.top_10_concentration,
|
||||
"holders": report.total_holders,
|
||||
"liquidity_usd": report.total_liquidity_usd,
|
||||
"bundler": "🎯 YES" if report.bundler_detected else "No",
|
||||
"bundler_count": len(report.bundler_wallets),
|
||||
"snipers": report.sniper_count,
|
||||
"honeypot": "🍯 YES" if report.honeypot_detected else "No",
|
||||
"lp_locked": any(p.lp_locked_percentage > 80 for p in report.liquidity_pools),
|
||||
"warnings": report.critical_warnings[:5],
|
||||
"ms": report.scan_duration_ms,
|
||||
}
|
||||
|
||||
_set_cache(token, response)
|
||||
return response
|
||||
|
||||
|
||||
@router.get("/bundle-check", response_model=BundleCheckResponse)
|
||||
async def check_bundler(
|
||||
token: str = Query(..., min_length=32, max_length=48),
|
||||
):
|
||||
"""
|
||||
Dedicated bundler detection endpoint.
|
||||
Identifies coordinated launch buying patterns.
|
||||
"""
|
||||
scanner = DegenSecurityScanner()
|
||||
report = SecurityReport(token_address=token)
|
||||
await scanner._phase5_bundlers_snipers(report)
|
||||
|
||||
return BundleCheckResponse(
|
||||
token_address=token,
|
||||
bundler_detected=report.bundler_detected,
|
||||
bundler_wallets=report.bundler_wallets,
|
||||
bundler_pattern=report.bundler_pattern,
|
||||
bundler_supply_pct=report.bundler_percentage_of_supply,
|
||||
confidence=0.85 if report.bundler_detected else 0.0,
|
||||
details={
|
||||
"sniper_count": report.sniper_count,
|
||||
"avg_sniper_entry_ms": report.avg_sniper_entry_time_ms,
|
||||
"sniper_volume_sol": report.sniper_volume_sol,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/funding-trace", response_model=FundingTraceResponse)
|
||||
async def trace_funding(
|
||||
token: str = Query(..., min_length=32, max_length=48),
|
||||
):
|
||||
"""
|
||||
Trace deployer wallet funding sources.
|
||||
Reveals exchange funding % and mixer usage.
|
||||
"""
|
||||
scanner = DegenSecurityScanner()
|
||||
report = SecurityReport(token_address=token)
|
||||
await scanner._phase1_metadata(report)
|
||||
await scanner._phase4_funding(report)
|
||||
|
||||
sources = []
|
||||
if report.deployer_funding:
|
||||
df = report.deployer_funding
|
||||
sources = [
|
||||
{
|
||||
"funder": df.initial_funder,
|
||||
"tag": df.funder_tag,
|
||||
"amount_sol": df.funding_amount_sol,
|
||||
"time": df.funding_time,
|
||||
"hops_from_exchange": df.hops_from_exchange,
|
||||
"is_exchange": df.is_exchange_direct,
|
||||
}
|
||||
]
|
||||
|
||||
return FundingTraceResponse(
|
||||
token_address=token,
|
||||
deployer=report.deployer,
|
||||
funding_sources=sources,
|
||||
exchange_funding_pct=report.exchange_funding_percentage,
|
||||
organic_funding_pct=report.organic_funding_percentage,
|
||||
mixer_detected=report.mixer_funding_detected,
|
||||
risk_level=report.funding_risk,
|
||||
trace_graph=[],
|
||||
)
|
||||
|
||||
|
||||
@router.get("/compare")
|
||||
async def compare_tokens(
|
||||
tokens: str = Query(..., description="Comma-separated token addresses"),
|
||||
):
|
||||
"""
|
||||
Compare security scores across multiple tokens.
|
||||
Perfect for evaluating which of several tokens is safest.
|
||||
"""
|
||||
token_list = [t.strip() for t in tokens.split(",") if len(t.strip()) >= 32]
|
||||
if len(token_list) > 5:
|
||||
raise HTTPException(status_code=400, detail="Max 5 tokens per comparison")
|
||||
|
||||
results = []
|
||||
for token in token_list:
|
||||
try:
|
||||
report = await scan_token(token, quick=True)
|
||||
results.append(
|
||||
{
|
||||
"token": f"{report.token_name} (${report.token_symbol})",
|
||||
"address": token,
|
||||
"rug_probability": report.rug_pull_probability,
|
||||
"risk": report.overall_risk,
|
||||
"liquidity": report.total_liquidity_usd,
|
||||
"holders": report.total_holders,
|
||||
"bundler": report.bundler_detected,
|
||||
"honeypot": report.honeypot_detected,
|
||||
"mint_renounced": report.mint_authority_renounced,
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
results.append({"address": token, "error": str(e)})
|
||||
|
||||
# Sort by safest first
|
||||
results.sort(key=lambda x: x.get("rug_probability", 100))
|
||||
|
||||
return {
|
||||
"comparison": results,
|
||||
"safest": results[0] if results else None,
|
||||
"riskiest": results[-1] if len(results) > 1 else None,
|
||||
"timestamp": datetime.utcnow().isoformat(),
|
||||
}
|
||||
|
||||
|
||||
# ─── Mount Instructions ───────────────────────────────────────────────────────
|
||||
|
||||
# In your main.py, add:
|
||||
# from degen_scan_endpoint import router as degen_scan_router
|
||||
# app.include_router(degen_scan_router)
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
from fastapi import FastAPI
|
||||
|
||||
app = FastAPI(title="RMI Degen Security Scanner")
|
||||
app.include_router(router)
|
||||
|
||||
@app.get("/")
|
||||
def root():
|
||||
return {"status": "RMI Degen Scanner API", "version": "1.0.0"}
|
||||
|
||||
uvicorn.run(app, host="0.0.0.0", port=8765)
|
||||
|
|
@ -1,184 +0,0 @@
|
|||
"""#14 - Dify RMI Tool Suite. 15 crypto tools exposed as OpenAPI endpoints for Dify agents.
|
||||
Each tool is a standalone endpoint that Dify can call via API. Security: rate-limited, API-key gated, logged."""
|
||||
|
||||
import os
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
router = APIRouter(prefix="/api/v1/dify-tools", tags=["dify-tools"])
|
||||
|
||||
BACKEND = os.getenv("BACKEND_URL", "http://localhost:8000")
|
||||
RMI_KEY = os.getenv("RMI_INTERNAL_KEY", "rmi-internal-2026")
|
||||
|
||||
|
||||
# ── Tool 1: Search Crypto ──
|
||||
@router.get("/search")
|
||||
async def search_crypto(q: str, limit: int = Query(5, le=20)):
|
||||
async with httpx.AsyncClient(timeout=15) as c:
|
||||
r = await c.get(
|
||||
f"{BACKEND}/api/v1/databus/fetch/token_search",
|
||||
params={"q": q, "limit": limit},
|
||||
headers={"X-RMI-Key": RMI_KEY},
|
||||
)
|
||||
return {"query": q, "results": r.json() if r.status_code == 200 else []}
|
||||
|
||||
|
||||
# ── Tool 2: Scan Token ──
|
||||
@router.get("/scan")
|
||||
async def scan_token(address: str, chain: str = "solana"):
|
||||
async with httpx.AsyncClient(timeout=20) as c:
|
||||
r = await c.post(
|
||||
f"{BACKEND}/api/v1/token/scan",
|
||||
json={"token_address": address, "chain": chain},
|
||||
headers={"X-RMI-Key": RMI_KEY},
|
||||
)
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
return {
|
||||
"safety_score": data.get("safety_score"),
|
||||
"risk_flags": data.get("risk_flags", []),
|
||||
"symbol": data.get("symbol"),
|
||||
}
|
||||
return {"error": "scan failed"}
|
||||
|
||||
|
||||
# ── Tool 3: Get Price ──
|
||||
@router.get("/price")
|
||||
async def get_price(symbol: str, chain: str | None = None):
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(
|
||||
f"{BACKEND}/api/v1/databus/fetch/token_price", params={"symbol": symbol}, headers={"X-RMI-Key": RMI_KEY}
|
||||
)
|
||||
return r.json() if r.status_code == 200 else {"error": "price unavailable"}
|
||||
|
||||
|
||||
# ── Tool 4: Whale Alert ──
|
||||
@router.get("/whales")
|
||||
async def whale_alert(chain: str = "ethereum", threshold: float = Query(100000, ge=10000)):
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(
|
||||
f"{BACKEND}/api/v1/databus/fetch/whale_alerts", params={"chain": chain}, headers={"X-RMI-Key": RMI_KEY}
|
||||
)
|
||||
return r.json() if r.status_code == 200 else {"alerts": 0}
|
||||
|
||||
|
||||
# ── Tool 5: Market Overview ──
|
||||
@router.get("/market")
|
||||
async def market_overview():
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(f"{BACKEND}/api/v1/databus/fetch/market_overview", headers={"X-RMI-Key": RMI_KEY})
|
||||
return r.json() if r.status_code == 200 else {"error": "market data unavailable"}
|
||||
|
||||
|
||||
# ── Tool 6: Token Report ──
|
||||
@router.get("/report")
|
||||
async def token_report(address: str, chain: str = "ethereum"):
|
||||
async with httpx.AsyncClient(timeout=20) as c:
|
||||
r = await c.get(f"{BACKEND}/api/v1/token-cv/{chain}/{address}", headers={"X-RMI-Key": RMI_KEY})
|
||||
return r.json() if r.status_code == 200 else {"error": "report failed"}
|
||||
|
||||
|
||||
# ── Tool 7: Address Profile ──
|
||||
@router.get("/profile")
|
||||
async def address_profile(address: str):
|
||||
async with httpx.AsyncClient(timeout=15) as c:
|
||||
r = await c.get(f"{BACKEND}/api/v1/address-profiler/profile/{address}", headers={"X-RMI-Key": RMI_KEY})
|
||||
return r.json() if r.status_code == 200 else {"error": "profile failed"}
|
||||
|
||||
|
||||
# ── Tool 8: Chain Compare ──
|
||||
@router.get("/compare")
|
||||
async def compare_chains(symbol: str):
|
||||
async with httpx.AsyncClient(timeout=15) as c:
|
||||
r = await c.get(f"{BACKEND}/api/v1/chain-compare/token/{symbol}", headers={"X-RMI-Key": RMI_KEY})
|
||||
return r.json() if r.status_code == 200 else {"error": "compare failed"}
|
||||
|
||||
|
||||
# ── Tool 9: Predict Rug ──
|
||||
@router.get("/predict")
|
||||
async def predict_rug(address: str, chain: str = "ethereum"):
|
||||
async with httpx.AsyncClient(timeout=15) as c:
|
||||
r = await c.get(f"{BACKEND}/api/v1/death-clock/predict/{chain}/{address}", headers={"X-RMI-Key": RMI_KEY})
|
||||
return r.json() if r.status_code == 200 else {"error": "prediction failed"}
|
||||
|
||||
|
||||
# ── Tool 10: Contract Analyze ──
|
||||
@router.get("/contract")
|
||||
async def contract_analyze(address: str, chain: str = "ethereum"):
|
||||
async with httpx.AsyncClient(timeout=15) as c:
|
||||
r = await c.get(
|
||||
f"{BACKEND}/api/v1/contract-analyzer/analyze/{address}?chain={chain}", headers={"X-RMI-Key": RMI_KEY}
|
||||
)
|
||||
return r.json() if r.status_code == 200 else {"error": "analysis failed"}
|
||||
|
||||
|
||||
# ── Tool 11: Trending ──
|
||||
@router.get("/trending")
|
||||
async def trending_tokens(chain: str = "solana", limit: int = Query(10, le=25)):
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(
|
||||
f"{BACKEND}/api/v1/databus/fetch/trending",
|
||||
params={"chain": chain, "limit": limit},
|
||||
headers={"X-RMI-Key": RMI_KEY},
|
||||
)
|
||||
return r.json() if r.status_code == 200 else {"trending": []}
|
||||
|
||||
|
||||
# ── Tool 12: Fear & Greed ──
|
||||
@router.get("/fear-greed")
|
||||
async def fear_greed_index():
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(f"{BACKEND}/api/v1/databus/fetch/fear_greed", headers={"X-RMI-Key": RMI_KEY})
|
||||
return r.json() if r.status_code == 200 else {"value": 50, "classification": "Neutral"}
|
||||
|
||||
|
||||
# ── Tool 13: News ──
|
||||
@router.get("/news")
|
||||
async def news_headlines(limit: int = Query(5, le=20)):
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(f"{BACKEND}/api/v1/databus/fetch/news", params={"limit": limit}, headers={"X-RMI-Key": RMI_KEY})
|
||||
return r.json() if r.status_code == 200 else {"headlines": []}
|
||||
|
||||
|
||||
# ── Tool 14: RAG Search ──
|
||||
@router.get("/rag")
|
||||
async def search_rag(query: str, limit: int = Query(5, le=10)):
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(
|
||||
f"{BACKEND}/api/v1/rag/search", params={"q": query, "limit": limit}, headers={"X-RMI-Key": RMI_KEY}
|
||||
)
|
||||
return r.json() if r.status_code == 200 else {"results": []}
|
||||
|
||||
|
||||
# ── Tool 15: Ollama Chat (fallback for local inference) ──
|
||||
@router.get("/ollama")
|
||||
async def ollama_chat(prompt: str, model: str = "qwen2.5-coder:7b"):
|
||||
async with httpx.AsyncClient(timeout=60) as c:
|
||||
r = await c.get(f"{BACKEND}/api/v1/ollama/chat?prompt={prompt}&model={model}", headers={"X-RMI-Key": RMI_KEY})
|
||||
return r.json() if r.status_code == 200 else {"error": "ollama unavailable"}
|
||||
|
||||
|
||||
# ── Tool catalog ──
|
||||
@router.get("/catalog")
|
||||
async def tool_catalog():
|
||||
return {
|
||||
"tools": [
|
||||
{"name": "search_crypto", "description": "Search crypto tokens across 112 chains"},
|
||||
{"name": "scan_token", "description": "SENTINEL security scan"},
|
||||
{"name": "get_price", "description": "Real-time token price"},
|
||||
{"name": "whale_alert", "description": "Recent whale movements"},
|
||||
{"name": "market_overview", "description": "Market stats + fear & greed"},
|
||||
{"name": "token_report", "description": "Full token security report"},
|
||||
{"name": "address_profile", "description": "Cross-chain wallet analysis"},
|
||||
{"name": "compare_chains", "description": "Price/liquidity across chains"},
|
||||
{"name": "predict_rug", "description": "Token Death Clock prediction"},
|
||||
{"name": "contract_analyze", "description": "Smart contract function analysis"},
|
||||
{"name": "trending_tokens", "description": "What's pumping right now"},
|
||||
{"name": "fear_greed_index", "description": "Market sentiment index"},
|
||||
{"name": "news_headlines", "description": "Latest crypto news"},
|
||||
{"name": "search_rag", "description": "Search 17K+ scam documents"},
|
||||
{"name": "ollama_chat", "description": "Local AI inference (free)"},
|
||||
],
|
||||
"total": 15,
|
||||
}
|
||||
|
|
@ -1,162 +0,0 @@
|
|||
"""
|
||||
Token Discovery API Router - Multi-chain token discovery from DexScreener + GeckoTerminal.
|
||||
Connects to /api/v1/discovery/*
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from app.token_discovery import MONITORED_CHAINS, discover_tokens, scan_token_security
|
||||
|
||||
router = APIRouter(prefix="/api/v1/discovery", tags=["token-discovery"])
|
||||
|
||||
|
||||
class DiscoveryRequest(BaseModel):
|
||||
chains: list[str] | None = None
|
||||
limit: int = 20
|
||||
|
||||
|
||||
@router.post("/tokens")
|
||||
async def discover_new_tokens(req: DiscoveryRequest):
|
||||
"""Discover new tokens across chains (DexScreener + GeckoTerminal)."""
|
||||
chains = req.chains if req.chains else list(MONITORED_CHAINS.keys())
|
||||
result = await discover_tokens(chains=chains)
|
||||
total = sum(len(tokens) for tokens in result.values())
|
||||
return {"chains": list(result.keys()), "total_tokens": total, "tokens": result}
|
||||
|
||||
|
||||
@router.get("/chains")
|
||||
async def list_chains():
|
||||
"""List monitored chains."""
|
||||
return {"chains": MONITORED_CHAINS, "count": len(MONITORED_CHAINS)}
|
||||
|
||||
|
||||
@router.get("/security/{chain}/{address}")
|
||||
async def token_security(chain: str, address: str):
|
||||
"""GoPlus security scan for a token."""
|
||||
chain_id = MONITORED_CHAINS.get(chain, chain)
|
||||
result = await scan_token_security(chain_id, address)
|
||||
if not result:
|
||||
raise HTTPException(status_code=404, detail="Security scan unavailable")
|
||||
return {"chain": chain, "address": address, "security": result}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def discovery_health():
|
||||
cg_status = {}
|
||||
try:
|
||||
from app.coingecko_connector import COINGECKO_FREE_KEY, COINGECKO_PRO_KEY
|
||||
|
||||
cg_status = {
|
||||
"free_key": bool(COINGECKO_FREE_KEY),
|
||||
"pro_key": bool(COINGECKO_PRO_KEY),
|
||||
"mode": "pro" if COINGECKO_PRO_KEY else "demo",
|
||||
}
|
||||
except Exception:
|
||||
pass
|
||||
return {
|
||||
"status": "ok",
|
||||
"service": "token-discovery",
|
||||
"chains": len(MONITORED_CHAINS),
|
||||
"coingecko": cg_status,
|
||||
}
|
||||
|
||||
|
||||
# ── CoinGecko Market Data ───────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/coingecko/trending")
|
||||
async def coingecko_trending():
|
||||
"""Get trending coins from CoinGecko (updated every 30 min)."""
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
trending = await cg.get_trending()
|
||||
return {"trending": trending, "count": len(trending)}
|
||||
|
||||
|
||||
@router.get("/coingecko/markets")
|
||||
async def coingecko_markets(vs: str = "usd", limit: int = 50):
|
||||
"""Get top coins by market cap."""
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
markets = await cg.get_market_overview(vs_currency=vs, per_page=min(limit, 250))
|
||||
return {"markets": markets, "count": len(markets)}
|
||||
|
||||
|
||||
@router.get("/coingecko/global")
|
||||
async def coingecko_global():
|
||||
"""Get global crypto market metrics (total market cap, BTC dominance, etc)."""
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
metrics = await cg.get_global_metrics()
|
||||
return metrics or {"error": "unavailable"}
|
||||
|
||||
|
||||
@router.get("/coingecko/coin/{coin_id}")
|
||||
async def coingecko_coin_detail(coin_id: str):
|
||||
"""Get detailed info about a specific coin."""
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
detail = await cg.get_token_detail(coin_id)
|
||||
if not detail:
|
||||
raise HTTPException(status_code=404, detail=f"Coin '{coin_id}' not found")
|
||||
return detail
|
||||
|
||||
|
||||
@router.get("/coingecko/categories")
|
||||
async def coingecko_categories():
|
||||
"""Get coin categories with market data."""
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
categories = await cg.get_category_market_data()
|
||||
return {"categories": categories, "count": len(categories)}
|
||||
|
||||
|
||||
@router.get("/coingecko/exchanges")
|
||||
async def coingecko_exchanges(limit: int = 20):
|
||||
"""Get top exchanges by trading volume."""
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
exchanges = await cg.get_exchanges(per_page=min(limit, 100))
|
||||
return {"exchanges": exchanges, "count": len(exchanges)}
|
||||
|
||||
|
||||
@router.get("/coingecko/ohlc/{coin_id}")
|
||||
async def coingecko_ohlc(coin_id: str, vs: str = "usd", days: int = 7):
|
||||
"""Get OHLC candlestick data for a coin."""
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
data = await cg.get_ohlc(coin_id, vs_currency=vs, days=days)
|
||||
return {"coin_id": coin_id, "vs": vs, "days": days, "candles": data}
|
||||
|
||||
|
||||
@router.get("/coingecko/pools/trending")
|
||||
async def coingecko_trending_pools(chain: str = "solana"):
|
||||
"""Get trending DEX pools (GeckoTerminal)."""
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
pools = await cg.get_trending_pools(chain=chain)
|
||||
return {"chain": chain, "pools": pools, "count": len(pools)}
|
||||
|
||||
|
||||
@router.get("/coingecko/status")
|
||||
async def coingecko_status():
|
||||
"""Check CoinGecko API key status and connectivity."""
|
||||
from app.coingecko_connector import get_coingecko_connector
|
||||
|
||||
cg = get_coingecko_connector()
|
||||
ping = await cg.ping()
|
||||
key_status = await cg.api_key_status()
|
||||
return {
|
||||
"ping": ping,
|
||||
"key_status": key_status,
|
||||
"connector": cg.status(),
|
||||
}
|
||||
|
|
@ -1,135 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Email API Router - Maildir + SMTP access for all RMI email accounts.
|
||||
Maildir is mounted from host at /var/mail/vhosts.
|
||||
SMTP uses host gateway for outbound delivery.
|
||||
"""
|
||||
|
||||
import email
|
||||
import os
|
||||
import smtplib
|
||||
from email.mime.multipart import MIMEMultipart
|
||||
from email.mime.text import MIMEText
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
from pydantic import BaseModel
|
||||
|
||||
router = APIRouter(prefix="/email", tags=["email"])
|
||||
|
||||
SMTP_HOST = os.getenv("SMTP_HOST", "172.20.0.1")
|
||||
SMTP_PORT = int(os.getenv("SMTP_PORT", "25"))
|
||||
MAILDIR_BASE = "/var/mail/vhosts"
|
||||
|
||||
ACCOUNTS = {
|
||||
"admin": "admin@rugmunch.io",
|
||||
"biz": "biz@rugmunch.io",
|
||||
"security": "security@rugmunch.io",
|
||||
"hello": "hello@cryptorugmunch.com",
|
||||
"media": "media@cryptorugmunch.com",
|
||||
}
|
||||
|
||||
|
||||
class SendRequest(BaseModel):
|
||||
sender_account: str
|
||||
to: str
|
||||
subject: str
|
||||
body: str
|
||||
|
||||
|
||||
def _maildir_for(account: str) -> str:
|
||||
addr = ACCOUNTS.get(account)
|
||||
if not addr:
|
||||
raise HTTPException(404, f"Unknown account: {account}")
|
||||
local, domain = addr.split("@")
|
||||
return f"{MAILDIR_BASE}/{domain}/{local}"
|
||||
|
||||
|
||||
def _read_msg(filepath: str, msg_id: int, full: bool = False) -> dict:
|
||||
with open(filepath, "rb") as f:
|
||||
msg = email.message_from_binary_file(f)
|
||||
result = {
|
||||
"id": msg_id,
|
||||
"subject": str(msg.get("Subject", "")),
|
||||
"sender": str(msg.get("From", "")),
|
||||
"recipient": str(msg.get("To", "")),
|
||||
"date": str(msg.get("Date", "")),
|
||||
}
|
||||
if full:
|
||||
body = ""
|
||||
if msg.is_multipart():
|
||||
for part in msg.walk():
|
||||
if part.get_content_type() == "text/plain":
|
||||
payload = part.get_payload(decode=True)
|
||||
if payload:
|
||||
body = payload.decode(errors="replace")
|
||||
break
|
||||
else:
|
||||
payload = msg.get_payload(decode=True)
|
||||
if payload:
|
||||
body = payload.decode(errors="replace")
|
||||
result["body"] = body[:10000]
|
||||
return result
|
||||
|
||||
|
||||
def _list_files(maildir: str) -> list:
|
||||
files = []
|
||||
for sub in ["new", "cur"]:
|
||||
d = os.path.join(maildir, sub)
|
||||
if os.path.isdir(d):
|
||||
files.extend(os.path.join(d, f) for f in os.listdir(d))
|
||||
files.sort(key=lambda p: os.path.getmtime(p), reverse=True)
|
||||
return files
|
||||
|
||||
|
||||
@router.get("/accounts")
|
||||
def list_accounts():
|
||||
return {"accounts": dict(ACCOUNTS.items())}
|
||||
|
||||
|
||||
@router.get("/{account}/inbox")
|
||||
def list_inbox(account: str, limit: int = Query(20, le=100)):
|
||||
if account not in ACCOUNTS:
|
||||
raise HTTPException(404, f"Unknown account. Options: {list(ACCOUNTS.keys())}")
|
||||
try:
|
||||
maildir = _maildir_for(account)
|
||||
files = _list_files(maildir)[:limit]
|
||||
msgs = [_read_msg(f, i + 1) for i, f in enumerate(files)]
|
||||
return {"account": ACCOUNTS[account], "count": len(msgs), "messages": msgs}
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise HTTPException(500, str(e)) from e
|
||||
|
||||
|
||||
@router.get("/{account}/message/{msg_id}")
|
||||
def read_message(account: str, msg_id: int):
|
||||
if account not in ACCOUNTS:
|
||||
raise HTTPException(404, "Unknown account")
|
||||
try:
|
||||
maildir = _maildir_for(account)
|
||||
files = _list_files(maildir)
|
||||
if msg_id < 1 or msg_id > len(files):
|
||||
raise HTTPException(404, "Message not found")
|
||||
return _read_msg(files[msg_id - 1], msg_id, full=True)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise HTTPException(500, str(e)) from e
|
||||
|
||||
|
||||
@router.post("/send")
|
||||
def send_email(req: SendRequest):
|
||||
if req.sender_account not in ACCOUNTS:
|
||||
raise HTTPException(404, "Unknown sender account")
|
||||
sender_addr = ACCOUNTS[req.sender_account]
|
||||
msg = MIMEMultipart()
|
||||
msg["From"] = sender_addr
|
||||
msg["To"] = req.to
|
||||
msg["Subject"] = req.subject
|
||||
msg.attach(MIMEText(req.body, "plain"))
|
||||
try:
|
||||
with smtplib.SMTP(SMTP_HOST, SMTP_PORT, timeout=10) as smtp:
|
||||
smtp.send_message(msg)
|
||||
return {"status": "sent", "from": sender_addr, "to": req.to}
|
||||
except Exception as e:
|
||||
raise HTTPException(500, f"Send failed: {e}") from e
|
||||
|
|
@ -1,16 +0,0 @@
|
|||
"""Embedding Router - thin wrapper around smart rate-limited dispatcher."""
|
||||
|
||||
from app.rate_limiter import get_dispatcher
|
||||
|
||||
|
||||
async def get_router():
|
||||
return await get_dispatcher()
|
||||
|
||||
|
||||
async def embed_texts(texts, task="default"):
|
||||
return await (await get_dispatcher()).embed(texts, task)
|
||||
|
||||
|
||||
async def embed_single(text, task="default"):
|
||||
vecs, provider = await (await get_dispatcher()).embed([text], task)
|
||||
return vecs[0] if vecs else [], provider
|
||||
|
|
@ -1,396 +0,0 @@
|
|||
"""
|
||||
Entity Graph Visualization - Interactive Fund Flow for RugMaps + Scans
|
||||
=======================================================================
|
||||
|
||||
Generates interactive entity relationship graphs showing wallet connections,
|
||||
fund flows, and cluster relationships. SVG output for embedding in scan results
|
||||
and RugMaps pages.
|
||||
|
||||
Premium feature: Visual forensics that sells in screenshots.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger("entity.graph")
|
||||
|
||||
|
||||
@dataclass
|
||||
class GraphNode:
|
||||
id: str
|
||||
label: str
|
||||
node_type: str = "wallet" # "wallet", "token", "contract", "entity", "cex"
|
||||
risk_score: float = 0.0
|
||||
chain: str = ""
|
||||
size: int = 20
|
||||
color: str = "#8B5CF6"
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class GraphEdge:
|
||||
source: str
|
||||
target: str
|
||||
edge_type: str = "transfer" # "transfer", "deploy", "fund", "cluster", "counterparty"
|
||||
value_usd: float = 0.0
|
||||
tx_count: int = 1
|
||||
color: str = "#4B5563"
|
||||
width: float = 1.0
|
||||
|
||||
|
||||
def build_entity_graph(
|
||||
center_address: str,
|
||||
chain: str = "ethereum",
|
||||
transactions: list[dict] | None = None,
|
||||
labels: dict[str, Any] | None = None,
|
||||
cluster_members: list[str] | None = None,
|
||||
depth: int = 2,
|
||||
) -> dict[str, Any]:
|
||||
"""Build an entity relationship graph from wallet data.
|
||||
|
||||
Returns graph data suitable for visualization (SVG or Canvas).
|
||||
"""
|
||||
nodes: dict[str, GraphNode] = {}
|
||||
edges: list[GraphEdge] = []
|
||||
|
||||
# ── Center node ──
|
||||
center_id = f"{chain}:{center_address.lower()}"
|
||||
nodes[center_id] = GraphNode(
|
||||
id=center_id,
|
||||
label=_shorten_addr(center_address),
|
||||
node_type="wallet",
|
||||
chain=chain,
|
||||
size=30,
|
||||
color="#22D3EE", # Cyan for center
|
||||
metadata={"address": center_address, "chain": chain},
|
||||
)
|
||||
|
||||
# ── Process transactions ──
|
||||
txs = transactions or []
|
||||
counterparties = set()
|
||||
|
||||
for tx in txs[:200]: # Cap at 200 transactions
|
||||
if not isinstance(tx, dict):
|
||||
continue
|
||||
|
||||
counterparty = tx.get("counterparty") or tx.get("to") or tx.get("from")
|
||||
if not counterparty or counterparty.lower() == center_address.lower():
|
||||
continue
|
||||
|
||||
cp_id = f"{tx.get('chain', chain)}:{counterparty.lower()}"
|
||||
counterparties.add(cp_id)
|
||||
|
||||
value = float(tx.get("value_usd", 0) or 0)
|
||||
tx_type = tx.get("type", "transfer")
|
||||
is_outgoing = tx.get("direction") == "out" or tx.get("from", "").lower() == center_address.lower()
|
||||
|
||||
# Add counterparty node if new
|
||||
if cp_id not in nodes:
|
||||
# Determine node type and color
|
||||
node_type, color = _classify_node(counterparty, labels or {}, tx)
|
||||
nodes[cp_id] = GraphNode(
|
||||
id=cp_id,
|
||||
label=_shorten_addr(counterparty),
|
||||
node_type=node_type,
|
||||
chain=tx.get("chain", chain),
|
||||
size=18 if node_type == "wallet" else 22,
|
||||
color=color,
|
||||
metadata={"address": counterparty},
|
||||
)
|
||||
|
||||
# Add edge
|
||||
edge_color = "#EF4444" if is_outgoing else "#22C55E" # Red=out, Green=in
|
||||
edge_width = min(max(value / 10000, 1), 6) if value > 0 else 1
|
||||
|
||||
edges.append(
|
||||
GraphEdge(
|
||||
source=center_id if is_outgoing else cp_id,
|
||||
target=cp_id if is_outgoing else center_id,
|
||||
edge_type=tx_type,
|
||||
value_usd=value,
|
||||
tx_count=1,
|
||||
color=edge_color,
|
||||
width=edge_width,
|
||||
)
|
||||
)
|
||||
|
||||
# ── Add cluster members (if available) ──
|
||||
if cluster_members:
|
||||
cluster_color = "#F59E0B" # Amber for cluster
|
||||
for member in cluster_members[:20]:
|
||||
if member.lower() == center_address.lower():
|
||||
continue
|
||||
member_id = f"{chain}:{member.lower()}"
|
||||
if member_id not in nodes:
|
||||
nodes[member_id] = GraphNode(
|
||||
id=member_id,
|
||||
label=_shorten_addr(member),
|
||||
node_type="cluster_member",
|
||||
color=cluster_color,
|
||||
size=16,
|
||||
)
|
||||
edges.append(
|
||||
GraphEdge(
|
||||
source=center_id,
|
||||
target=member_id,
|
||||
edge_type="cluster",
|
||||
color=cluster_color,
|
||||
width=1.5,
|
||||
)
|
||||
)
|
||||
|
||||
# ── Add label-based nodes (CEX, scams, etc.) ──
|
||||
if labels:
|
||||
label_data = labels.get(center_address, labels)
|
||||
if isinstance(label_data, dict):
|
||||
label_type = label_data.get("label_type") or label_data.get("type", "")
|
||||
entity = label_data.get("entity") or label_data.get("name", "")
|
||||
|
||||
if label_type in ("cex", "exchange"):
|
||||
nodes[center_id].node_type = "cex"
|
||||
nodes[center_id].color = "#F59E0B" # Amber
|
||||
nodes[center_id].label = entity or "Exchange"
|
||||
elif label_type in ("scam", "phish", "hack"):
|
||||
nodes[center_id].node_type = "scam"
|
||||
nodes[center_id].color = "#EF4444" # Red
|
||||
nodes[center_id].risk_score = 90
|
||||
nodes[center_id].label = entity or "Scam"
|
||||
elif label_type in ("mixer", "tumbler"):
|
||||
nodes[center_id].node_type = "mixer"
|
||||
nodes[center_id].color = "#8B5CF6" # Purple
|
||||
nodes[center_id].risk_score = 70
|
||||
|
||||
# ── Risk-based coloring ──
|
||||
for node in nodes.values():
|
||||
if node.risk_score >= 80:
|
||||
node.color = "#EF4444" # Red
|
||||
elif node.risk_score >= 60:
|
||||
node.color = "#F97316" # Orange
|
||||
elif node.risk_score >= 40:
|
||||
node.color = "#EAB308" # Yellow
|
||||
|
||||
# ── Convert to graph format ──
|
||||
return {
|
||||
"center": {"id": center_id, "address": center_address, "chain": chain},
|
||||
"nodes": [
|
||||
{
|
||||
"id": n.id,
|
||||
"label": n.label,
|
||||
"type": n.node_type,
|
||||
"chain": n.chain,
|
||||
"risk_score": n.risk_score,
|
||||
"size": n.size,
|
||||
"color": n.color,
|
||||
"metadata": n.metadata,
|
||||
}
|
||||
for n in nodes.values()
|
||||
],
|
||||
"edges": [
|
||||
{
|
||||
"source": e.source,
|
||||
"target": e.target,
|
||||
"type": e.edge_type,
|
||||
"value_usd": e.value_usd,
|
||||
"tx_count": e.tx_count,
|
||||
"color": e.color,
|
||||
"width": e.width,
|
||||
}
|
||||
for e in edges
|
||||
],
|
||||
"stats": {
|
||||
"total_nodes": len(nodes),
|
||||
"total_edges": len(edges),
|
||||
"total_value_usd": sum(e.value_usd for e in edges),
|
||||
"entity_types": list({n.node_type for n in nodes.values()}),
|
||||
"risk_distribution": {
|
||||
"high": sum(1 for n in nodes.values() if n.risk_score >= 60),
|
||||
"medium": sum(1 for n in nodes.values() if 30 <= n.risk_score < 60),
|
||||
"low": sum(1 for n in nodes.values() if n.risk_score < 30),
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def generate_svg(graph_data: dict[str, Any], width: int = 800, height: int = 600) -> str:
|
||||
"""Generate an SVG visualization of the entity graph.
|
||||
|
||||
Uses a simple force-directed layout algorithm.
|
||||
"""
|
||||
import math
|
||||
|
||||
nodes = graph_data.get("nodes", [])
|
||||
edges = graph_data.get("edges", [])
|
||||
center = graph_data.get("center", {})
|
||||
center_id = center.get("id", "")
|
||||
|
||||
if not nodes:
|
||||
return '<svg xmlns="http://www.w3.org/2000/svg" width="800" height="200"><text x="400" y="100" text-anchor="middle" fill="#6B7280">No graph data</text></svg>'
|
||||
|
||||
# Simple radial layout: center in middle, others in concentric circles
|
||||
cx, cy = width // 2, height // 2
|
||||
|
||||
# Separate center from others
|
||||
other_nodes = [n for n in nodes if n["id"] != center_id]
|
||||
center_node = next((n for n in nodes if n["id"] == center_id), None)
|
||||
|
||||
# Arrange other nodes in circles
|
||||
positions = {}
|
||||
if center_node:
|
||||
positions[center_id] = (cx, cy)
|
||||
|
||||
rings = [
|
||||
(150, 0.0), # radius, rotation offset
|
||||
(250, 0.3),
|
||||
(350, 0.15),
|
||||
]
|
||||
|
||||
node_idx = 0
|
||||
for ring_radius, rotation in rings:
|
||||
nodes_in_ring = other_nodes[node_idx : node_idx + max(8, len(other_nodes) // len(rings))]
|
||||
n = len(nodes_in_ring)
|
||||
for i, node in enumerate(nodes_in_ring):
|
||||
angle = (2 * math.pi * i / max(n, 1)) + rotation
|
||||
x = cx + ring_radius * math.cos(angle)
|
||||
y = cy + ring_radius * math.sin(angle)
|
||||
positions[node["id"]] = (x, y)
|
||||
node_idx += n
|
||||
if node_idx >= len(other_nodes):
|
||||
break
|
||||
|
||||
# Build SVG
|
||||
svg_parts = [
|
||||
f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" style="background:#07070b;font-family:monospace">',
|
||||
"<defs>",
|
||||
'<filter id="glow"><feGaussianBlur stdDeviation="3" result="blur"/><feMerge><feMergeNode in="blur"/><feMergeNode in="SourceGraphic"/></feMerge></filter>',
|
||||
'<marker id="arrow-red" viewBox="0 0 10 10" refX="10" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0,0 L10,5 L0,10 Z" fill="#EF4444"/></marker>',
|
||||
'<marker id="arrow-green" viewBox="0 0 10 10" refX="10" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0,0 L10,5 L0,10 Z" fill="#22C55E"/></marker>',
|
||||
"</defs>",
|
||||
# Background grid
|
||||
'<pattern id="grid" width="40" height="40" patternUnits="userSpaceOnUse"><path d="M 40 0 L 0 0 0 40" fill="none" stroke="#1a2332" stroke-width="0.5"/></pattern>',
|
||||
f'<rect width="{width}" height="{height}" fill="url(#grid)"/>',
|
||||
]
|
||||
|
||||
# Draw edges
|
||||
for edge in edges:
|
||||
src_pos = positions.get(edge["source"])
|
||||
tgt_pos = positions.get(edge["target"])
|
||||
if not src_pos or not tgt_pos:
|
||||
continue
|
||||
|
||||
x1, y1 = src_pos
|
||||
x2, y2 = tgt_pos
|
||||
|
||||
# Curve edges slightly
|
||||
mid_x = (x1 + x2) / 2
|
||||
mid_y = (y1 + y2) / 2 - 30
|
||||
|
||||
edge_color = edge.get("color", "#4B5563")
|
||||
edge_width = edge.get("width", 1)
|
||||
marker_attr = 'marker-end="url(#arrow-red)"' if edge["type"] == "transfer" else ""
|
||||
|
||||
svg_parts.append(
|
||||
f'<path d="M{x1},{y1} Q{mid_x},{mid_y} {x2},{y2}" '
|
||||
f'stroke="{edge_color}" stroke-width="{edge_width}" fill="none" '
|
||||
f'opacity="0.4" {marker_attr}/>'
|
||||
)
|
||||
|
||||
# Draw nodes
|
||||
for node in nodes:
|
||||
pos = positions.get(node["id"])
|
||||
if not pos:
|
||||
continue
|
||||
|
||||
x, y = pos
|
||||
node_color = node.get("color", "#8B5CF6")
|
||||
node_size = node.get("size", 20)
|
||||
node_type = node.get("type", "wallet")
|
||||
|
||||
# Different shapes by type
|
||||
if node_type == "scam" or node.get("risk_score", 0) >= 80:
|
||||
# Diamond for high-risk
|
||||
svg_parts.append(
|
||||
f'<polygon points="{x},{y - node_size} {x + node_size},{y} {x},{y + node_size} {x - node_size},{y}" '
|
||||
f'fill="{node_color}" fill-opacity="0.2" stroke="{node_color}" stroke-width="2" filter="url(#glow)"/>'
|
||||
)
|
||||
elif node_type == "cex" or node_type == "exchange":
|
||||
# Square for exchanges
|
||||
svg_parts.append(
|
||||
f'<rect x="{x - node_size}" y="{y - node_size}" width="{node_size * 2}" height="{node_size * 2}" '
|
||||
f'rx="4" fill="{node_color}" fill-opacity="0.2" stroke="{node_color}" stroke-width="2"/>'
|
||||
)
|
||||
else:
|
||||
# Circle for wallets
|
||||
svg_parts.append(
|
||||
f'<circle cx="{x}" cy="{y}" r="{node_size}" '
|
||||
f'fill="{node_color}" fill-opacity="0.2" stroke="{node_color}" stroke-width="2"/>'
|
||||
)
|
||||
|
||||
# Label
|
||||
label = node.get("label", "")[:12]
|
||||
svg_parts.append(
|
||||
f'<text x="{x}" y="{y + node_size + 14}" text-anchor="middle" fill="#9CA3AF" font-size="10">{label}</text>'
|
||||
)
|
||||
|
||||
# Center node highlight
|
||||
if node["id"] == center_id:
|
||||
svg_parts.append(
|
||||
f'<circle cx="{x}" cy="{y}" r="{node_size + 4}" '
|
||||
f'fill="none" stroke="#22D3EE" stroke-width="2" stroke-dasharray="4,2" opacity="0.5"/>'
|
||||
)
|
||||
|
||||
# Legend
|
||||
legend_x = 20
|
||||
legend_y = height - 80
|
||||
legend_items = [
|
||||
("#EF4444", "High Risk/Scam"),
|
||||
("#F97316", "Medium Risk"),
|
||||
("#22C55E", "Incoming"),
|
||||
("#EF4444", "Outgoing"),
|
||||
("#F59E0B", "Exchange/CEX"),
|
||||
("#8B5CF6", "Wallet"),
|
||||
]
|
||||
for i, (color, label) in enumerate(legend_items):
|
||||
lx = legend_x + (i % 3) * 130
|
||||
ly = legend_y + (i // 3) * 20
|
||||
svg_parts.append(f'<circle cx="{lx}" cy="{ly}" r="5" fill="{color}"/>')
|
||||
svg_parts.append(f'<text x="{lx + 12}" y="{ly + 4}" fill="#6B7280" font-size="10">{label}</text>')
|
||||
|
||||
svg_parts.append("</svg>")
|
||||
return "\n".join(svg_parts)
|
||||
|
||||
|
||||
def _classify_node(address: str, labels: dict[str, Any], tx: dict) -> tuple[str, str]:
|
||||
"""Classify a counterparty node by type and color."""
|
||||
addr_lower = address.lower()
|
||||
|
||||
# Check labels
|
||||
label_info = labels.get(address, labels.get(addr_lower, {}))
|
||||
if isinstance(label_info, dict):
|
||||
label_type = label_info.get("label_type", label_info.get("type", ""))
|
||||
if label_type in ("cex", "exchange"):
|
||||
return ("cex", "#F59E0B")
|
||||
if label_type in ("scam", "phish", "hack", "exploit"):
|
||||
return ("scam", "#EF4444")
|
||||
if label_type in ("mixer", "tumbler", "tornado"):
|
||||
return ("mixer", "#8B5CF6")
|
||||
if label_type in ("defi", "protocol"):
|
||||
return ("protocol", "#06B6D4")
|
||||
|
||||
# Check transaction patterns
|
||||
tx_type = tx.get("type", "")
|
||||
if tx_type == "contract_creation" or tx_type == "deploy":
|
||||
return ("contract", "#10B981")
|
||||
if tx_type == "swap":
|
||||
return ("dex", "#06B6D4")
|
||||
|
||||
return ("wallet", "#8B5CF6")
|
||||
|
||||
|
||||
def _shorten_addr(addr: str) -> str:
|
||||
"""Shorten address for display."""
|
||||
if not addr:
|
||||
return "?"
|
||||
if len(addr) <= 12:
|
||||
return addr
|
||||
return f"{addr[:6]}...{addr[-4:]}"
|
||||
|
|
@ -1,273 +0,0 @@
|
|||
"""
|
||||
Etherscan Family API Router - EVM Block Explorers.
|
||||
Single API for: Ethereum, BSC, Polygon, Avalanche, Fantom, Arbitrum, Optimism, Base.
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/v1/etherscan", tags=["etherscan"])
|
||||
|
||||
|
||||
# ── Models ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
class AccountQuery(BaseModel):
|
||||
address: str
|
||||
network: str = "eth" # eth, bsc, polygon, avalanche, fantom, arbitrum, optimism, base
|
||||
page: int = 1
|
||||
offset: int = 100
|
||||
|
||||
|
||||
class TokenBalanceQuery(BaseModel):
|
||||
contract: str
|
||||
address: str
|
||||
network: str = "eth"
|
||||
|
||||
|
||||
class ContractQuery(BaseModel):
|
||||
contract: str
|
||||
network: str = "eth"
|
||||
|
||||
|
||||
class LogsQuery(BaseModel):
|
||||
from_block: int
|
||||
to_block: int
|
||||
address: str | None = None
|
||||
topic0: str | None = None
|
||||
network: str = "eth"
|
||||
|
||||
|
||||
# ── Account API ──────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/balance")
|
||||
async def get_balance(address: str, network: str = "eth"):
|
||||
"""Get native token balance (ETH/BNB/MATIC/etc)."""
|
||||
try:
|
||||
from app.etherscan_connector import get_etherscan_connector
|
||||
|
||||
ec = get_etherscan_connector()
|
||||
balance = await ec.get_balance(address, network)
|
||||
# Convert wei to native token
|
||||
try:
|
||||
native = int(balance) / 1e18 if balance else 0
|
||||
except Exception:
|
||||
native = 0
|
||||
return {
|
||||
"address": address,
|
||||
"network": network,
|
||||
"balance_wei": balance,
|
||||
"balance_native": native,
|
||||
}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Etherscan connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/transactions")
|
||||
async def get_transactions(address: str, network: str = "eth", page: int = 1, offset: int = 100):
|
||||
"""Get transaction list for address."""
|
||||
try:
|
||||
from app.etherscan_connector import get_etherscan_connector
|
||||
|
||||
ec = get_etherscan_connector()
|
||||
txs = await ec.get_tx_list(address, network, page=page, offset=offset)
|
||||
return {
|
||||
"address": address,
|
||||
"network": network,
|
||||
"transactions": txs[:offset],
|
||||
"count": len(txs),
|
||||
}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Etherscan connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/token-transfers")
|
||||
async def get_token_transfers(
|
||||
address: str,
|
||||
network: str = "eth",
|
||||
contract: str | None = None,
|
||||
page: int = 1,
|
||||
offset: int = 100,
|
||||
):
|
||||
"""Get ERC-20 token transfer events."""
|
||||
try:
|
||||
from app.etherscan_connector import get_etherscan_connector
|
||||
|
||||
ec = get_etherscan_connector()
|
||||
transfers = await ec.get_token_transfers(address, network, contract_address=contract, page=page, offset=offset)
|
||||
return {
|
||||
"address": address,
|
||||
"network": network,
|
||||
"transfers": transfers[:offset],
|
||||
"count": len(transfers),
|
||||
}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Etherscan connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/token-balance")
|
||||
async def get_token_balance(contract: str, address: str, network: str = "eth"):
|
||||
"""Get ERC-20 token balance for address."""
|
||||
try:
|
||||
from app.etherscan_connector import get_etherscan_connector
|
||||
|
||||
ec = get_etherscan_connector()
|
||||
balance = await ec.get_token_balance(contract, address, network)
|
||||
return {
|
||||
"contract": contract,
|
||||
"address": address,
|
||||
"network": network,
|
||||
"balance_wei": balance,
|
||||
}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Etherscan connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/nft-transfers")
|
||||
async def get_nft_transfers(address: str, network: str = "eth", page: int = 1, offset: int = 100):
|
||||
"""Get ERC-721 NFT transfer events."""
|
||||
try:
|
||||
from app.etherscan_connector import get_etherscan_connector
|
||||
|
||||
ec = get_etherscan_connector()
|
||||
transfers = await ec.get_erc721_transfers(address, network, page=page, offset=offset)
|
||||
return {
|
||||
"address": address,
|
||||
"network": network,
|
||||
"transfers": transfers[:offset],
|
||||
"count": len(transfers),
|
||||
}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Etherscan connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
# ── Contract API ─────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/contract/abi")
|
||||
async def get_contract_abi(contract: str, network: str = "eth"):
|
||||
"""Get contract ABI."""
|
||||
try:
|
||||
from app.etherscan_connector import get_etherscan_connector
|
||||
|
||||
ec = get_etherscan_connector()
|
||||
abi = await ec.get_contract_abi(contract, network)
|
||||
return {"contract": contract, "network": network, "abi": abi}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Etherscan connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/contract/source")
|
||||
async def get_contract_source(contract: str, network: str = "eth"):
|
||||
"""Get contract source code."""
|
||||
try:
|
||||
from app.etherscan_connector import get_etherscan_connector
|
||||
|
||||
ec = get_etherscan_connector()
|
||||
source = await ec.get_contract_source(contract, network)
|
||||
return {"contract": contract, "network": network, "source": source}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Etherscan connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/contract/verify")
|
||||
async def verify_contract(contract: str, network: str = "eth"):
|
||||
"""Check if contract is verified."""
|
||||
try:
|
||||
from app.etherscan_connector import get_etherscan_connector
|
||||
|
||||
ec = get_etherscan_connector()
|
||||
result = await ec.verify_contract(contract, network)
|
||||
return {"contract": contract, "network": network, **result}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Etherscan connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
# ── Transaction API ──────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/tx/{tx_hash}/status")
|
||||
async def get_tx_status(tx_hash: str, network: str = "eth"):
|
||||
"""Get transaction status (success/fail)."""
|
||||
try:
|
||||
from app.etherscan_connector import get_etherscan_connector
|
||||
|
||||
ec = get_etherscan_connector()
|
||||
status = await ec.get_transaction_status(tx_hash, network)
|
||||
return {"tx_hash": tx_hash, "network": network, "status": status}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Etherscan connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
# ── Block API ────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/block/latest")
|
||||
async def get_latest_block(network: str = "eth"):
|
||||
"""Get latest block number."""
|
||||
try:
|
||||
from app.etherscan_connector import get_etherscan_connector
|
||||
|
||||
ec = get_etherscan_connector()
|
||||
block = await ec.get_block_number(network)
|
||||
return {"network": network, "block_number": block}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Etherscan connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
# ── Logs API ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/logs")
|
||||
async def get_logs(req: LogsQuery):
|
||||
"""Get event logs (filtered by address/topic)."""
|
||||
try:
|
||||
from app.etherscan_connector import get_etherscan_connector
|
||||
|
||||
ec = get_etherscan_connector()
|
||||
logs = await ec.get_logs(req.from_block, req.to_block, req.address, req.topic0, req.network)
|
||||
return {"network": req.network, "logs": logs, "count": len(logs)}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Etherscan connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
# ── Health ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def etherscan_health():
|
||||
"""Etherscan connector status."""
|
||||
try:
|
||||
from app.etherscan_connector import get_etherscan_connector
|
||||
|
||||
ec = get_etherscan_connector()
|
||||
return {"status": "ok", "service": "etherscan-connector", **ec.status()}
|
||||
except ImportError:
|
||||
return {"status": "ok", "service": "etherscan-connector", "networks_active": 0}
|
||||
|
|
@ -1,120 +0,0 @@
|
|||
"""
|
||||
Forensics API Router - Deep contract scan + cross-chain correlation + risk reports.
|
||||
Connects to /api/v1/forensics/*
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
from pydantic import BaseModel
|
||||
|
||||
router = APIRouter(prefix="/api/v1/forensics", tags=["forensics"])
|
||||
|
||||
|
||||
class DeepScanRequest(BaseModel):
|
||||
contract_address: str
|
||||
chain: str = "base"
|
||||
|
||||
|
||||
class BatchScanRequest(BaseModel):
|
||||
contracts: list[str]
|
||||
chain: str = "base"
|
||||
|
||||
|
||||
class RiskReportRequest(BaseModel):
|
||||
token_address: str
|
||||
chain: str = "solana"
|
||||
include_graph: bool = False
|
||||
|
||||
|
||||
class ThreatCheckRequest(BaseModel):
|
||||
address: str
|
||||
chain_id: str = "1" # 1=ethereum, 56=bsc, 8453=base, solana=solana
|
||||
|
||||
|
||||
@router.post("/deep-scan")
|
||||
async def deep_scan(req: DeepScanRequest):
|
||||
"""Deep contract scan using slither/mythril."""
|
||||
try:
|
||||
from app.contract_deepscan import deep_scan_contract
|
||||
|
||||
result = deep_scan_contract(req.contract_address, chain=req.chain)
|
||||
return {"contract": req.contract_address, "chain": req.chain, "scan": result}
|
||||
except ImportError as e:
|
||||
raise HTTPException(status_code=503, detail=f"Scanner unavailable: {e}") from e
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:500]) from e
|
||||
|
||||
|
||||
@router.post("/batch-scan")
|
||||
async def batch_deep_scan(req: BatchScanRequest):
|
||||
"""Batch deep scan multiple contracts."""
|
||||
try:
|
||||
from app.contract_deepscan import batch_deep_scan
|
||||
|
||||
result = batch_deep_scan(req.contracts, chain=req.chain)
|
||||
return {"contracts": len(req.contracts), "chain": req.chain, "results": result}
|
||||
except ImportError as e:
|
||||
raise HTTPException(status_code=503, detail=str(e)) from e
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:500]) from e
|
||||
|
||||
|
||||
@router.post("/risk-report")
|
||||
async def risk_report(req: RiskReportRequest):
|
||||
"""Generate comprehensive rug risk report."""
|
||||
try:
|
||||
from app.advanced_analysis import RugRiskReport
|
||||
|
||||
report = RugRiskReport(token_address=req.token_address, chain=req.chain)
|
||||
return {"token_address": req.token_address, "chain": req.chain, "report": str(report)}
|
||||
except ImportError as e:
|
||||
raise HTTPException(status_code=503, detail=str(e)) from e
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:500]) from e
|
||||
|
||||
|
||||
@router.post("/cross-chain")
|
||||
async def cross_chain_correlate(addresses: list[str] = Query(...), chains: list[str] | None = Query(None)):
|
||||
"""Correlate wallets across chains using behavioral fingerprinting + CEX patterns."""
|
||||
try:
|
||||
from app.cross_chain_correlator import get_cross_chain_correlator
|
||||
|
||||
cc = get_cross_chain_correlator()
|
||||
profiles = await cc.correlate(addresses=addresses, chains=chains)
|
||||
return {
|
||||
"addresses_scanned": len(addresses),
|
||||
"profiles_found": len(profiles),
|
||||
"profiles": [
|
||||
{
|
||||
"entity_id": p.entity_id,
|
||||
"total_addresses": p.total_addresses,
|
||||
"chains": dict(p.chains),
|
||||
"links": len(p.links),
|
||||
"risk_score": p.risk_score,
|
||||
}
|
||||
for p in profiles
|
||||
],
|
||||
}
|
||||
except ImportError as e:
|
||||
raise HTTPException(status_code=503, detail=str(e)) from e
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:500]) from e
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def forensics_health():
|
||||
return {"status": "ok", "service": "forensics-engine"}
|
||||
|
||||
|
||||
@router.post("/threat-check")
|
||||
async def threat_check(req: ThreatCheckRequest):
|
||||
"""Multi-source threat intel check: CryptoScamDB + GoPlus + Januus (all free)."""
|
||||
try:
|
||||
from app.threat_feeds import get_threat_feeds
|
||||
|
||||
feeds = get_threat_feeds()
|
||||
result = await feeds.full_check(address=req.address, chain_id=req.chain_id)
|
||||
return result
|
||||
except ImportError as e:
|
||||
raise HTTPException(status_code=503, detail=str(e)) from e
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:500]) from e
|
||||
|
|
@ -1,175 +0,0 @@
|
|||
"""
|
||||
Ghostunnel SSL/TLS Tunnel Integration
|
||||
======================================
|
||||
|
||||
Secure SSL/TLS proxy with mutual authentication for non-TLS services.
|
||||
Features:
|
||||
- TLS termination
|
||||
- Client certificate verification
|
||||
- Proxying to TCP services
|
||||
"""
|
||||
|
||||
import logging
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class GhostunnelConfig:
|
||||
"""Ghostunnel configuration."""
|
||||
|
||||
listen: str
|
||||
target: str
|
||||
cacert: str | None = None
|
||||
cert: str | None = None
|
||||
key: str | None = None
|
||||
clientcert: str | None = None
|
||||
acceptonly: str | None = None
|
||||
status: str | None = "127.0.0.1:8080"
|
||||
|
||||
|
||||
# ─── GHOSTUNNEL MANAGER ───────────────────────────────────────────
|
||||
|
||||
|
||||
class GhostunnelManager:
|
||||
"""Manages Ghostunnel processes."""
|
||||
|
||||
def __init__(self, ghostunnel_bin: str = "ghostunnel"):
|
||||
self.ghostunnel_bin = ghostunnel_bin
|
||||
self._available = self._check_available()
|
||||
self.processes: dict[str, subprocess.Popen] = {}
|
||||
|
||||
def _check_available(self) -> bool:
|
||||
"""Check if ghostunnel is installed."""
|
||||
try:
|
||||
result = subprocess.run([self.ghostunnel_bin, "version"], capture_output=True, timeout=5)
|
||||
return result.returncode == 0
|
||||
except (subprocess.TimeoutExpired, FileNotFoundError):
|
||||
return False
|
||||
|
||||
def start_tunnel(self, config: GhostunnelConfig) -> str:
|
||||
"""
|
||||
Start a new tunnel.
|
||||
|
||||
Args:
|
||||
config: Ghostunnel configuration
|
||||
|
||||
Returns:
|
||||
Process ID of the tunnel
|
||||
"""
|
||||
if not self._available:
|
||||
return "stub_tunnel"
|
||||
|
||||
cmd = [
|
||||
self.ghostunnel_bin,
|
||||
"server",
|
||||
"--listen",
|
||||
config.listen,
|
||||
"--target",
|
||||
config.target,
|
||||
]
|
||||
|
||||
if config.cacert:
|
||||
cmd.extend(["--cacert", config.cacert])
|
||||
if config.cert:
|
||||
cmd.extend(["--cert", config.cert])
|
||||
if config.key:
|
||||
cmd.extend(["--key", config.key])
|
||||
if config.clientcert:
|
||||
cmd.extend(["--clientcert", config.clientcert])
|
||||
if config.acceptonly:
|
||||
cmd.extend(["--acceptonly", config.acceptonly])
|
||||
if config.status:
|
||||
cmd.extend(["--status", config.status])
|
||||
|
||||
try:
|
||||
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
self.processes[config.listen] = process
|
||||
return str(process.pid)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start ghostunnel: {e}")
|
||||
return ""
|
||||
|
||||
def stop_tunnel(self, listen_addr: str) -> bool:
|
||||
"""
|
||||
Stop a tunnel by listening address.
|
||||
|
||||
Args:
|
||||
listen_addr: Listening address of the tunnel
|
||||
|
||||
Returns:
|
||||
Success status
|
||||
"""
|
||||
process = self.processes.get(listen_addr)
|
||||
if process:
|
||||
process.terminate()
|
||||
try:
|
||||
process.wait(timeout=5)
|
||||
del self.processes[listen_addr]
|
||||
return True
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
del self.processes[listen_addr]
|
||||
return True
|
||||
return False
|
||||
|
||||
def stop_all(self) -> int:
|
||||
"""Stop all tunnels. Returns count of stopped processes."""
|
||||
count = 0
|
||||
for addr in list(self.processes.keys()):
|
||||
if self.stop_tunnel(addr):
|
||||
count += 1
|
||||
return count
|
||||
|
||||
def get_status(self) -> dict[str, str]:
|
||||
"""Get status of all tunnels."""
|
||||
status = {}
|
||||
for addr, process in self.processes.items():
|
||||
status[addr] = {
|
||||
"pid": process.pid,
|
||||
"alive": process.poll() is None,
|
||||
"started_at": datetime.utcnow().isoformat(),
|
||||
}
|
||||
return status
|
||||
|
||||
|
||||
# ─── GLOBAL SINGLETON ─────────────────────────────────────────────
|
||||
|
||||
_manager: GhostunnelManager | None = None
|
||||
|
||||
|
||||
def get_ghostunnel_manager() -> GhostunnelManager:
|
||||
"""Get or create Ghostunnel manager."""
|
||||
global _manager
|
||||
if _manager is None:
|
||||
_manager = GhostunnelManager()
|
||||
return _manager
|
||||
|
||||
|
||||
def start_tunnel(listen: str, target: str, **kwargs) -> str:
|
||||
"""
|
||||
Convenience function to start a tunnel.
|
||||
|
||||
Args:
|
||||
listen: Listen address (host:port)
|
||||
target: Target address (host:port)
|
||||
**kwargs: Extra config options
|
||||
"""
|
||||
config = GhostunnelConfig(listen=listen, target=target, **kwargs)
|
||||
manager = get_ghostunnel_manager()
|
||||
return manager.start_tunnel(config)
|
||||
|
||||
|
||||
def stop_tunnel(listen: str) -> bool:
|
||||
"""Convenience function to stop a tunnel."""
|
||||
manager = get_ghostunnel_manager()
|
||||
return manager.stop_tunnel(listen)
|
||||
|
||||
|
||||
def stop_all_tunnels() -> int:
|
||||
"""Convenience function to stop all tunnels."""
|
||||
manager = get_ghostunnel_manager()
|
||||
return manager.stop_all()
|
||||
|
|
@ -1,230 +0,0 @@
|
|||
"""
|
||||
RAG GitHub Feeder v2 - Declarative, Config-Driven
|
||||
==================================================
|
||||
Add any GitHub repo as a RAG data source with minimal config.
|
||||
Auto-clones, extracts markdown/solidity/text, chunks, and ingests.
|
||||
|
||||
Config format (add to SOURCES list):
|
||||
{
|
||||
"repo": "owner/repo",
|
||||
"collection": "defi_hacks|vuln_patterns|contract_audits|...",
|
||||
"category": "defi_hack|audit|scam_report|...",
|
||||
"tags": ["tag1", "tag2"],
|
||||
"file_patterns": ["*.md", "*.sol", "*.py"], # files to extract
|
||||
"max_files": 50, # limit per run
|
||||
"enabled": True,
|
||||
}
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
|
||||
import httpx
|
||||
|
||||
BACKEND = "http://localhost:8000"
|
||||
INGEST_URL = f"{BACKEND}/api/v1/rag/ingest"
|
||||
|
||||
# ═══════════════════════════════════════════════════
|
||||
# DECLARATIVE SOURCE CONFIG - add repos here
|
||||
# ═══════════════════════════════════════════════════
|
||||
SOURCES = [
|
||||
{
|
||||
"repo": "SunWeb3Sec/DeFiHackLabs",
|
||||
"collection": "defi_hacks",
|
||||
"category": "defi_hack",
|
||||
"tags": ["DeFi", "hack_reproduction", "Foundry", "education"],
|
||||
"file_patterns": ["*.md"],
|
||||
"max_files": 50,
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"repo": "TradMod/awesome-audits-checklists",
|
||||
"collection": "contract_audits",
|
||||
"category": "audit_checklist",
|
||||
"tags": ["audit", "checklist", "smart_contract", "security"],
|
||||
"file_patterns": ["*.md"],
|
||||
"max_files": 20,
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"repo": "alt-research/SolidityGuard",
|
||||
"collection": "vuln_patterns",
|
||||
"category": "vuln_pattern",
|
||||
"tags": ["OWASP", "Solidity", "security", "vulnerability"],
|
||||
"file_patterns": ["*.md", "*.sol"],
|
||||
"max_files": 40,
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"repo": "forta-network/labelled-datasets",
|
||||
"collection": "known_scams",
|
||||
"category": "scam_report",
|
||||
"tags": ["Forta", "labelled_data", "threat_intel", "web3_security"],
|
||||
"file_patterns": ["*.md", "*.json", "*.csv"],
|
||||
"max_files": 30,
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"repo": "Messi-Q/Smart-Contract-Dataset",
|
||||
"collection": "vuln_patterns",
|
||||
"category": "vuln_pattern",
|
||||
"tags": ["dataset", "smart_contract", "vulnerability", "academic"],
|
||||
"file_patterns": ["*.md", "*.sol", "*.csv"],
|
||||
"max_files": 30,
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"repo": "nurkiewicz/crypto-hall-of-shame",
|
||||
"collection": "rug_timeline",
|
||||
"category": "scam_report",
|
||||
"tags": ["scam_timeline", "historical", "hall_of_shame"],
|
||||
"file_patterns": ["*.md", "*.adoc"],
|
||||
"max_files": 5,
|
||||
"enabled": True,
|
||||
},
|
||||
]
|
||||
|
||||
CLONE_DIR = "/tmp/rag_sources"
|
||||
|
||||
|
||||
def ensure_cloned(repo: str) -> str | None:
|
||||
"""Clone repo if not already present. Returns path or None."""
|
||||
name = repo.split("/")[-1]
|
||||
path = os.path.join(CLONE_DIR, name)
|
||||
if os.path.exists(path):
|
||||
return path
|
||||
os.makedirs(CLONE_DIR, exist_ok=True)
|
||||
try:
|
||||
subprocess.run(
|
||||
["git", "clone", "--depth", "1", f"https://github.com/{repo}.git", path],
|
||||
capture_output=True,
|
||||
timeout=60,
|
||||
)
|
||||
return path if os.path.exists(path) else None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def extract_from_repo(source: dict) -> list[dict]:
|
||||
"""Extract documents from a cloned GitHub repo."""
|
||||
path = ensure_cloned(source["repo"])
|
||||
if not path:
|
||||
print(f" {source['repo']}: clone failed")
|
||||
return []
|
||||
|
||||
docs = []
|
||||
patterns = source.get("file_patterns", ["*.md"])
|
||||
max_files = source.get("max_files", 50)
|
||||
|
||||
files = []
|
||||
for pat in patterns:
|
||||
files.extend(glob.glob(f"{path}/**/{pat}", recursive=True))
|
||||
|
||||
for fpath in files[:max_files]:
|
||||
try:
|
||||
# Skip binary/large files
|
||||
size = os.path.getsize(fpath)
|
||||
if size > 500_000 or size < 50:
|
||||
continue
|
||||
|
||||
with open(fpath, errors="ignore") as f:
|
||||
content = f.read()
|
||||
|
||||
if not content.strip():
|
||||
continue
|
||||
|
||||
# Extract filename for title
|
||||
relpath = os.path.relpath(fpath, path)
|
||||
name = os.path.splitext(os.path.basename(fpath))[0]
|
||||
|
||||
# Try to extract a title from markdown heading
|
||||
title_match = re.search(r"^#\s+(.+)", content, re.MULTILINE)
|
||||
title = title_match.group(1) if title_match else name
|
||||
|
||||
# For .sol files, extract contract/function structure
|
||||
if fpath.endswith(".sol"):
|
||||
contracts = re.findall(r"(?:contract|interface|library)\s+(\w+)", content)
|
||||
funcs = re.findall(r"function\s+(\w+)", content)[:10]
|
||||
summary = f"Solidity: {', '.join(contracts[:5])}. Functions: {', '.join(funcs)}"
|
||||
text = f"Smart Contract: {title} ({relpath})\n\n{summary}\n\n{content[:3000]}"
|
||||
elif fpath.endswith(".json"):
|
||||
try:
|
||||
data = json.loads(content)
|
||||
text = f"Dataset: {title}\n\n{json.dumps(data, indent=2)[:3000]}"
|
||||
except Exception:
|
||||
text = f"Data File: {title}\n\n{content[:2000]}"
|
||||
elif fpath.endswith(".csv"):
|
||||
lines = content.split("\n")[:50]
|
||||
text = f"CSV Data: {title}\n\n" + "\n".join(lines)
|
||||
else:
|
||||
text = f"{title}\n\n{content[:3000]}"
|
||||
|
||||
docs.append(
|
||||
{
|
||||
"text": text[:4000],
|
||||
"source": source["repo"],
|
||||
"url": f"https://github.com/{source['repo']}/blob/main/{relpath}",
|
||||
"category": source["category"],
|
||||
"tags": source.get("tags", []),
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
print(f" {source['repo']}: {len(docs)} docs from {len(files[:max_files])} files")
|
||||
return docs
|
||||
|
||||
|
||||
async def ingest_repo(source: dict):
|
||||
"""Clone, extract, and ingest a single repo."""
|
||||
docs = extract_from_repo(source)
|
||||
if not docs:
|
||||
return 0
|
||||
|
||||
total = 0
|
||||
async with httpx.AsyncClient(timeout=120) as c:
|
||||
for i in range(0, len(docs), 30):
|
||||
batch = docs[i : i + 30]
|
||||
try:
|
||||
r = await c.post(
|
||||
INGEST_URL,
|
||||
json={
|
||||
"collection": source["collection"],
|
||||
"documents": batch,
|
||||
"source": source["repo"],
|
||||
"chunking": "scam_report",
|
||||
},
|
||||
)
|
||||
if r.status_code == 200:
|
||||
n = r.json().get("ingested", 0)
|
||||
total += n
|
||||
except Exception:
|
||||
pass
|
||||
return total
|
||||
|
||||
|
||||
async def ingest_all():
|
||||
"""Run all enabled sources."""
|
||||
print("=== RAG GitHub Feeder v2 ===\n")
|
||||
results = {}
|
||||
total = 0
|
||||
|
||||
for src in SOURCES:
|
||||
if not src.get("enabled", True):
|
||||
continue
|
||||
print(f"Ingesting: {src['repo']} → {src['collection']}")
|
||||
n = await ingest_repo(src)
|
||||
results[src["repo"]] = n
|
||||
total += n
|
||||
print(f" ingested: {n}\n")
|
||||
|
||||
print(f"Total: {total} docs across {len(results)} sources")
|
||||
return results
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(ingest_all())
|
||||
|
|
@ -1,480 +0,0 @@
|
|||
"""
|
||||
Human-Facing Tool Catalog API
|
||||
================================
|
||||
Rich, groupable, searchable catalog for the public web terminal and Telegram bot.
|
||||
Every tool from TOOL_PRICES + expanded_mcp_catalog is auto-exposed here.
|
||||
Adding a tool anywhere in the system? It's automatically picked up here.
|
||||
|
||||
Endpoints:
|
||||
GET /api/v1/catalog - Full catalog grouped by category + service
|
||||
GET /api/v1/catalog/search?q= - Free-text search
|
||||
GET /api/v1/catalog/featured - Curated featured tools (5 per category)
|
||||
GET /api/v1/catalog/by-trial - Grouped by trial count
|
||||
GET /api/v1/catalog/by-price - Grouped by price tier (free/$0.01/$0.05/etc)
|
||||
GET /api/v1/catalog/by-chain/{chain} - Tools for a specific chain
|
||||
GET /api/v1/catalog/{tool_id} - Single tool with full metadata
|
||||
GET /api/v1/catalog/stats - Catalog statistics
|
||||
|
||||
Trial policy tiers (per tool):
|
||||
simple: 5 free trials (sentiment, info, lookup)
|
||||
standard: 3 free trials (analysis, security scan)
|
||||
premium: 1 free trial (audit, deep scan, codegen)
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
|
||||
logger = logging.getLogger("human_catalog")
|
||||
router = APIRouter(prefix="/api/v1/catalog", tags=["human-catalog"])
|
||||
|
||||
# ════════════════════════════════════════════════════════════
|
||||
# Trial tier mapping
|
||||
# ════════════════════════════════════════════════════════════
|
||||
|
||||
# Tiers based on tool value/computational cost
|
||||
TRIAL_TIERS = {
|
||||
"free": {"trials": 5, "label": "Free", "color": "emerald", "badge": "🆓"},
|
||||
"simple": {"trials": 5, "label": "5 free trials", "color": "emerald", "badge": "🎁"},
|
||||
"standard": {"trials": 3, "label": "3 free trials", "color": "blue", "badge": "✨"},
|
||||
"premium": {"trials": 1, "label": "1 free trial", "color": "purple", "badge": "💎"},
|
||||
}
|
||||
|
||||
PRICE_TIERS = {
|
||||
"micro": {"max": 0.01, "label": "Micro", "color": "emerald"},
|
||||
"low": {"max": 0.05, "label": "Low", "color": "blue"},
|
||||
"mid": {"max": 0.15, "label": "Mid", "color": "amber"},
|
||||
"high": {"max": 0.50, "label": "High", "color": "rose"},
|
||||
"premium": {"max": 999.0, "label": "Premium", "color": "violet"},
|
||||
}
|
||||
|
||||
# Categories and their complexity (determines trial count)
|
||||
CATEGORY_TRIAL_DEFAULTS = {
|
||||
# Simple lookup/info - high trial count
|
||||
"sentiment": "simple",
|
||||
"social": "simple",
|
||||
"api": "simple",
|
||||
"research": "simple",
|
||||
# Standard analysis - medium trial count
|
||||
"analysis": "standard",
|
||||
"intelligence": "standard",
|
||||
"market": "standard",
|
||||
"monitoring": "standard",
|
||||
# Premium/heavy - low trial count
|
||||
"security": "premium",
|
||||
"forensics": "premium",
|
||||
"premium": "premium",
|
||||
"launchpad": "standard",
|
||||
"defi": "standard",
|
||||
"nft": "standard",
|
||||
"bundle": "standard",
|
||||
"mcp-external": "simple",
|
||||
}
|
||||
|
||||
|
||||
def _classify_trial_tier(tool: dict) -> str:
|
||||
"""Determine trial tier from tool metadata."""
|
||||
# Honor explicit trialFree in price_usd range
|
||||
if tool.get("trialFree") is not None:
|
||||
n = int(tool["trialFree"])
|
||||
if n >= 5:
|
||||
return "simple"
|
||||
elif n >= 3:
|
||||
return "standard"
|
||||
else:
|
||||
return "premium"
|
||||
|
||||
# Fall back to category default
|
||||
cat = tool.get("category", "analysis").lower()
|
||||
return CATEGORY_TRIAL_DEFAULTS.get(cat, "standard")
|
||||
|
||||
|
||||
def _classify_price_tier(price_usd: float) -> str:
|
||||
"""Determine price tier from USD amount."""
|
||||
for tier_name, tier_info in PRICE_TIERS.items():
|
||||
if price_usd <= tier_info["max"]:
|
||||
return tier_name
|
||||
return "premium"
|
||||
|
||||
|
||||
# ════════════════════════════════════════════════════════════
|
||||
# Cache - rebuild catalog from sources on demand
|
||||
# ════════════════════════════════════════════════════════════
|
||||
|
||||
_catalog_cache: dict[str, Any] = {}
|
||||
_catalog_cache_time: float = 0
|
||||
CATALOG_CACHE_TTL = 30 # 30 seconds - short enough to pick up new tools quickly
|
||||
|
||||
|
||||
def _get_full_catalog() -> dict[str, Any]:
|
||||
"""Pull the live catalog from all sources. Cached for CATALOG_CACHE_TTL."""
|
||||
global _catalog_cache, _catalog_cache_time
|
||||
|
||||
now = time.time()
|
||||
if _catalog_cache and (now - _catalog_cache_time) < CATALOG_CACHE_TTL:
|
||||
return _catalog_cache
|
||||
|
||||
# Source 1: x402 enforcement (TOOL_PRICES) - authoritative for our 221 native tools
|
||||
try:
|
||||
from app.routers.x402_enforcement import TOOL_PRICES
|
||||
|
||||
dict(TOOL_PRICES)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# Source 2: x402_catalog (which merges everything)
|
||||
try:
|
||||
from app.routers.x402_catalog import get_catalog
|
||||
|
||||
get_catalog_func = get_catalog
|
||||
except ImportError:
|
||||
get_catalog_func = None
|
||||
|
||||
# Source 3: expanded_mcp_catalog (the 25 new tools)
|
||||
try:
|
||||
from app.services.expanded_mcp_catalog import EXTERNAL_MCP_SERVICES, EXTERNAL_MCP_TOOLS
|
||||
|
||||
{t["id"] for t in EXTERNAL_MCP_TOOLS}
|
||||
external_services = list(EXTERNAL_MCP_SERVICES)
|
||||
except ImportError:
|
||||
external_services = []
|
||||
|
||||
# Combine: take the x402_catalog full output and enrich
|
||||
if get_catalog_func:
|
||||
try:
|
||||
merged = get_catalog_func()
|
||||
tools_list = list(merged.get("tools", []))
|
||||
except Exception:
|
||||
tools_list = []
|
||||
else:
|
||||
tools_list = []
|
||||
|
||||
# Convert each to human-friendly format
|
||||
enriched = []
|
||||
for t in tools_list:
|
||||
price_usd = float(t.get("priceUsd", 0.01) or 0.01)
|
||||
trial_tier = _classify_trial_tier(t)
|
||||
price_tier = _classify_price_tier(price_usd)
|
||||
|
||||
enriched.append(
|
||||
{
|
||||
"id": t.get("id", ""),
|
||||
"name": t.get("name", t.get("id", "?")),
|
||||
"description": t.get("description", ""),
|
||||
"category": t.get("category", "analysis"),
|
||||
"service": t.get("service", "rmi-native"),
|
||||
"source": t.get("source", "rmi-native"),
|
||||
"chains": t.get("chains", []),
|
||||
"price_usd": price_usd,
|
||||
"price_label": f"${price_usd:.2f}",
|
||||
"trial_tier": trial_tier,
|
||||
"trial_count": TRIAL_TIERS[trial_tier]["trials"],
|
||||
"trial_label": TRIAL_TIERS[trial_tier]["label"],
|
||||
"trial_badge": TRIAL_TIERS[trial_tier]["badge"],
|
||||
"price_tier": price_tier,
|
||||
"price_tier_label": PRICE_TIERS[price_tier]["label"],
|
||||
"method": t.get("method", "POST"),
|
||||
}
|
||||
)
|
||||
|
||||
# Group by category
|
||||
by_category = defaultdict(list)
|
||||
for t in enriched:
|
||||
by_category[t["category"]].append(t)
|
||||
|
||||
# Group by service
|
||||
by_service = defaultdict(list)
|
||||
for t in enriched:
|
||||
by_service[t["service"]].append(t)
|
||||
|
||||
# Group by trial tier
|
||||
by_trial = defaultdict(list)
|
||||
for t in enriched:
|
||||
by_trial[t["trial_tier"]].append(t)
|
||||
|
||||
# Group by price tier
|
||||
by_price = defaultdict(list)
|
||||
for t in enriched:
|
||||
by_price[t["price_tier"]].append(t)
|
||||
|
||||
# Build featured set - top 5 from each major category
|
||||
major_cats = ["security", "intelligence", "market", "analysis", "social", "defi"]
|
||||
featured = []
|
||||
for cat in major_cats:
|
||||
cat_tools = by_category.get(cat, [])
|
||||
featured.extend(cat_tools[:5])
|
||||
|
||||
# Sort: native first, then external
|
||||
enriched.sort(key=lambda t: (0 if t["source"] == "enforcement" else 1, t["price_usd"]))
|
||||
|
||||
result = {
|
||||
"version": "1.0.0",
|
||||
"generated_at": int(now),
|
||||
"total_tools": len(enriched),
|
||||
"total_native": sum(1 for t in enriched if t["source"] in ("enforcement", "route", "rmi-native")),
|
||||
"total_external": sum(1 for t in enriched if t["source"] in ("expanded-mcp", "mcp-router")),
|
||||
"total_services": len(by_service),
|
||||
"total_categories": len(by_category),
|
||||
"total_chains": len({c for t in enriched for c in t["chains"]}),
|
||||
"categories": sorted(by_category.keys()),
|
||||
"services": sorted(by_service.keys()),
|
||||
"tools": enriched,
|
||||
"by_category": dict(by_category.items()),
|
||||
"by_service": dict(by_service.items()),
|
||||
"by_trial_tier": dict(by_trial.items()),
|
||||
"by_price_tier": dict(by_price.items()),
|
||||
"featured": featured,
|
||||
"external_services": external_services,
|
||||
"trial_tiers": TRIAL_TIERS,
|
||||
"price_tiers": PRICE_TIERS,
|
||||
}
|
||||
|
||||
_catalog_cache = result
|
||||
_catalog_cache_time = now
|
||||
return result
|
||||
|
||||
|
||||
def invalidate_catalog_cache():
|
||||
"""Force rebuild on next request - call after adding/updating tools."""
|
||||
global _catalog_cache_time
|
||||
_catalog_cache_time = 0
|
||||
|
||||
|
||||
# ════════════════════════════════════════════════════════════
|
||||
# Endpoints
|
||||
# ════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
@router.get("")
|
||||
async def get_catalog_root(
|
||||
sort: str = Query("default", description="default, price, trial, name"),
|
||||
category: str | None = Query(None, description="Filter by category"),
|
||||
service: str | None = Query(None, description="Filter by service"),
|
||||
chain: str | None = Query(None, description="Filter by chain"),
|
||||
free_only: bool = Query(False, description="Only tools with free trials"),
|
||||
limit: int = Query(500, description="Max tools to return (0 = all)"),
|
||||
):
|
||||
"""Full human-facing catalog. Grouped + filterable + sortable."""
|
||||
cat = _get_full_catalog()
|
||||
tools = list(cat["tools"])
|
||||
|
||||
# Filters
|
||||
if category:
|
||||
tools = [t for t in tools if t["category"].lower() == category.lower()]
|
||||
if service:
|
||||
tools = [t for t in tools if t["service"].lower() == service.lower()]
|
||||
if chain:
|
||||
chain_upper = chain.upper()
|
||||
tools = [t for t in tools if chain_upper in [c.upper() for c in t["chains"]]]
|
||||
if free_only:
|
||||
tools = [t for t in tools if t["trial_count"] > 0]
|
||||
|
||||
# Sort
|
||||
if sort == "price":
|
||||
tools.sort(key=lambda t: t["price_usd"])
|
||||
elif sort == "trial":
|
||||
tools.sort(key=lambda t: -t["trial_count"])
|
||||
elif sort == "name":
|
||||
tools.sort(key=lambda t: t["name"].lower())
|
||||
elif sort == "category":
|
||||
tools.sort(key=lambda t: (t["category"], t["name"].lower()))
|
||||
|
||||
# Limit
|
||||
if limit > 0:
|
||||
tools = tools[:limit]
|
||||
|
||||
return {
|
||||
"total": len(tools),
|
||||
"total_available": cat["total_tools"],
|
||||
"version": cat["version"],
|
||||
"generated_at": cat["generated_at"],
|
||||
"filters": {
|
||||
"category": category,
|
||||
"service": service,
|
||||
"chain": chain,
|
||||
"sort": sort,
|
||||
"free_only": free_only,
|
||||
},
|
||||
"tools": tools,
|
||||
"categories": cat["categories"],
|
||||
"services": cat["services"],
|
||||
"trial_tiers": cat["trial_tiers"],
|
||||
"price_tiers": cat["price_tiers"],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/search")
|
||||
async def search_catalog(
|
||||
q: str = Query(..., min_length=2, description="Search query"),
|
||||
limit: int = Query(50, description="Max results"),
|
||||
):
|
||||
"""Free-text search across tool names, descriptions, IDs, and chains."""
|
||||
cat = _get_full_catalog()
|
||||
q_lower = q.lower()
|
||||
results = []
|
||||
|
||||
for t in cat["tools"]:
|
||||
score = 0
|
||||
# ID match (highest)
|
||||
if q_lower in t["id"].lower():
|
||||
score += 100
|
||||
# Name match
|
||||
if q_lower in t["name"].lower():
|
||||
score += 50
|
||||
# Description match
|
||||
if q_lower in t["description"].lower():
|
||||
score += 20
|
||||
# Category match
|
||||
if q_lower in t["category"].lower():
|
||||
score += 10
|
||||
# Service match
|
||||
if q_lower in t["service"].lower():
|
||||
score += 5
|
||||
# Chain match
|
||||
if any(q_lower in c.lower() for c in t["chains"]):
|
||||
score += 5
|
||||
|
||||
if score > 0:
|
||||
results.append({**t, "_score": score})
|
||||
|
||||
# Sort by score
|
||||
results.sort(key=lambda t: -t["_score"])
|
||||
# Drop score
|
||||
for r in results:
|
||||
r.pop("_score")
|
||||
|
||||
return {
|
||||
"query": q,
|
||||
"total": len(results),
|
||||
"tools": results[:limit],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/featured")
|
||||
async def get_featured():
|
||||
"""Curated featured tools - 5 from each major category."""
|
||||
cat = _get_full_catalog()
|
||||
return {
|
||||
"total": len(cat["featured"]),
|
||||
"tools": cat["featured"],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/by-trial")
|
||||
async def get_by_trial():
|
||||
"""Tools grouped by free trial tier (free/standard/premium)."""
|
||||
cat = _get_full_catalog()
|
||||
return {
|
||||
"tiers": cat["trial_tiers"],
|
||||
"groups": dict(cat["by_trial_tier"].items()),
|
||||
"counts": {k: len(v) for k, v in cat["by_trial_tier"].items()},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/by-price")
|
||||
async def get_by_price():
|
||||
"""Tools grouped by price tier (micro/low/mid/high/premium)."""
|
||||
cat = _get_full_catalog()
|
||||
return {
|
||||
"tiers": cat["price_tiers"],
|
||||
"groups": dict(cat["by_price_tier"].items()),
|
||||
"counts": {k: len(v) for k, v in cat["by_price_tier"].items()},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/by-chain/{chain}")
|
||||
async def get_by_chain(chain: str):
|
||||
"""All tools available for a specific chain."""
|
||||
cat = _get_full_catalog()
|
||||
chain_upper = chain.upper()
|
||||
tools = [t for t in cat["tools"] if chain_upper in [c.upper() for c in t["chains"]]]
|
||||
return {
|
||||
"chain": chain_upper,
|
||||
"total": len(tools),
|
||||
"tools": tools,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/by-category/{category}")
|
||||
async def get_by_category(category: str):
|
||||
"""All tools in a specific category."""
|
||||
cat = _get_full_catalog()
|
||||
tools = [t for t in cat["tools"] if t["category"].lower() == category.lower()]
|
||||
return {
|
||||
"category": category.lower(),
|
||||
"total": len(tools),
|
||||
"tools": tools,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/by-service/{service}")
|
||||
async def get_by_service(service: str):
|
||||
"""All tools from a specific service."""
|
||||
cat = _get_full_catalog()
|
||||
tools = [t for t in cat["tools"] if t["service"].lower() == service.lower()]
|
||||
return {
|
||||
"service": service.lower(),
|
||||
"total": len(tools),
|
||||
"tools": tools,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/stats")
|
||||
async def get_stats():
|
||||
"""Catalog statistics - counts, breakdown, pricing."""
|
||||
cat = _get_full_catalog()
|
||||
return {
|
||||
"version": cat["version"],
|
||||
"generated_at": cat["generated_at"],
|
||||
"total_tools": cat["total_tools"],
|
||||
"total_native": cat["total_native"],
|
||||
"total_external": cat["total_external"],
|
||||
"total_services": cat["total_services"],
|
||||
"total_categories": cat["total_categories"],
|
||||
"total_chains": cat["total_chains"],
|
||||
"by_category": {k: len(v) for k, v in cat["by_category"].items()},
|
||||
"by_service": {k: len(v) for k, v in cat["by_service"].items()},
|
||||
"by_trial_tier": {k: len(v) for k, v in cat["by_trial_tier"].items()},
|
||||
"by_price_tier": {k: len(v) for k, v in cat["by_price_tier"].items()},
|
||||
"external_services": cat["external_services"],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/external")
|
||||
async def get_external():
|
||||
"""All free open-source MCPs we've integrated, with their metadata."""
|
||||
cat = _get_full_catalog()
|
||||
return {
|
||||
"total": len(cat["external_services"]),
|
||||
"services": cat["external_services"],
|
||||
"tools": [t for t in cat["tools"] if t["source"] in ("expanded-mcp", "mcp-router")],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{tool_id}")
|
||||
async def get_tool(tool_id: str):
|
||||
"""Single tool with full metadata."""
|
||||
cat = _get_full_catalog()
|
||||
for t in cat["tools"]:
|
||||
if t["id"] == tool_id:
|
||||
return t
|
||||
raise HTTPException(status_code=404, detail=f"Tool '{tool_id}' not found")
|
||||
|
||||
|
||||
@router.post("/invalidate-cache")
|
||||
async def invalidate():
|
||||
"""Force catalog rebuild on next request. Call after adding tools."""
|
||||
invalidate_catalog_cache()
|
||||
return {"status": "ok", "message": "Cache invalidated - next request will rebuild"}
|
||||
|
||||
|
||||
# ════════════════════════════════════════════════════════════
|
||||
# Convenience: get_total_tool_count (used by other routers)
|
||||
# ════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
def get_total_tool_count() -> int:
|
||||
return _get_full_catalog().get("total_tools", 0)
|
||||
|
|
@ -1,608 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
RMI INTELLIGENCE FEED PIPELINE
|
||||
==============================
|
||||
Pulls crypto threat intelligence from RSS feeds and open APIs.
|
||||
Runs continuously, indexing scam/hack/exploit data into RAG.
|
||||
|
||||
Feeds (verified working):
|
||||
- Web3IsGoingGreat (Molly White) - incident tracker RSS
|
||||
- SlowMist - blockchain security firm (Medium)
|
||||
- PeckShield - on-chain security monitor (via Nitter)
|
||||
- Blockworks - crypto news
|
||||
- Cointelegraph Security - tagged articles
|
||||
|
||||
Additional sources:
|
||||
- Helius webhooks - new Solana tokens
|
||||
- GoPlus API - real-time token security checks
|
||||
- On-chain scanning - new token → quick risk assessment
|
||||
|
||||
Architecture:
|
||||
Prometheus → every N minutes pull RSS → extract entities → embed → store
|
||||
New token event → quick keyword scan → if suspicious → deep embed → alert
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import html
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from typing import Any
|
||||
|
||||
import feedparser
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
# CONFIGURED FEEDS
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
|
||||
FEEDS = [
|
||||
{
|
||||
"name": "web3isgoinggreat",
|
||||
"url": "https://www.web3isgoinggreat.com/feed",
|
||||
"category": "incident",
|
||||
"severity": "high",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"name": "slowmist",
|
||||
"url": "https://slowmist.medium.com/feed",
|
||||
"category": "hack_report",
|
||||
"severity": "critical",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"name": "peckshield",
|
||||
"url": "https://peckshield.medium.com/feed",
|
||||
"category": "security_alert",
|
||||
"severity": "high",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"name": "certik",
|
||||
"url": "https://certik.medium.com/feed",
|
||||
"category": "audit_report",
|
||||
"severity": "high",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"name": "immunefi",
|
||||
"url": "https://immunefi.medium.com/feed",
|
||||
"category": "bounty_report",
|
||||
"severity": "critical",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"name": "trailofbits",
|
||||
"url": "https://blog.trailofbits.com/feed/",
|
||||
"category": "security_research",
|
||||
"severity": "high",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"name": "cryptosecurity",
|
||||
"url": "https://cryptosecurity.substack.com/feed",
|
||||
"category": "security_news",
|
||||
"severity": "medium",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"name": "blockworks",
|
||||
"url": "https://blockworks.co/feed",
|
||||
"category": "crypto_news",
|
||||
"severity": "medium",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
"keywords": ["hack", "exploit", "scam", "rug", "drain", "attack", "breach", "stolen"],
|
||||
},
|
||||
{
|
||||
"name": "cointelegraph_security",
|
||||
"url": "https://cointelegraph.com/rss/tag/security",
|
||||
"category": "security_news",
|
||||
"severity": "medium",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
},
|
||||
{
|
||||
"name": "chainalysis",
|
||||
"url": "https://blog.chainalysis.com/feed/",
|
||||
"category": "forensic_report",
|
||||
"severity": "high",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
"keywords": ["scam", "ransomware", "sanction", "illicit", "money laundering"],
|
||||
},
|
||||
{
|
||||
"name": "cointelegraph",
|
||||
"url": "https://cointelegraph.com/rss",
|
||||
"category": "crypto_news",
|
||||
"severity": "low",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
"keywords": [
|
||||
"hack",
|
||||
"exploit",
|
||||
"scam",
|
||||
"rug",
|
||||
"drain",
|
||||
"attack",
|
||||
"breach",
|
||||
"stolen",
|
||||
"phishing",
|
||||
"theft",
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "decrypt",
|
||||
"url": "https://decrypt.co/feed",
|
||||
"category": "crypto_news",
|
||||
"severity": "low",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
"keywords": [
|
||||
"hack",
|
||||
"exploit",
|
||||
"scam",
|
||||
"rug",
|
||||
"drain",
|
||||
"stolen",
|
||||
"theft",
|
||||
"honeypot",
|
||||
"phishing",
|
||||
],
|
||||
},
|
||||
{
|
||||
"name": "theblock",
|
||||
"url": "https://www.theblock.co/rss.xml",
|
||||
"category": "crypto_news",
|
||||
"severity": "low",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
"keywords": ["hack", "exploit", "scam", "rug", "drain", "stolen"],
|
||||
},
|
||||
{
|
||||
"name": "bankless",
|
||||
"url": "https://www.bankless.com/feed",
|
||||
"category": "crypto_news",
|
||||
"severity": "low",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
"keywords": ["hack", "exploit", "scam", "rug", "drain", "attack"],
|
||||
},
|
||||
{
|
||||
"name": "thedefiant",
|
||||
"url": "https://thedefiant.io/feed",
|
||||
"category": "defi_news",
|
||||
"severity": "medium",
|
||||
"extractor": "rss",
|
||||
"enabled": True,
|
||||
"keywords": ["hack", "exploit", "scam", "rug", "drain", "attack", "breach"],
|
||||
},
|
||||
]
|
||||
|
||||
# Entities to extract from articles
|
||||
ENTITY_PATTERNS = {
|
||||
"eth_address": r"0x[a-fA-F0-9]{40}",
|
||||
"sol_address": r"[1-9A-HJ-NP-Za-km-z]{32,44}",
|
||||
"btc_address": r"[13][a-km-zA-HJ-NP-Z1-9]{25,34}",
|
||||
"tornado_cash": r"(?i)tornado\s*cash",
|
||||
"amount_usd": r"\$[\d,]+(?:\s*(?:million|billion|M|B|K))?",
|
||||
"protocol_name": r"(?<=the\s)(?:Uniswap|Aave|Compound|Curve|Balancer|Sushi|Pancake|Raydium|Orca|Jupiter|Marinade)(?=\s)",
|
||||
}
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
# FEED PROCESSORS
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
@dataclass
|
||||
class IntelItem:
|
||||
id: str
|
||||
title: str
|
||||
content: str
|
||||
source: str
|
||||
url: str
|
||||
published: str
|
||||
category: str
|
||||
severity: str
|
||||
entities: dict[str, list[str]] = field(default_factory=dict)
|
||||
raw: dict = field(default_factory=dict)
|
||||
|
||||
|
||||
class RSSProcessor:
|
||||
"""Process standard RSS/Atom feeds into IntelItems."""
|
||||
|
||||
@staticmethod
|
||||
async def fetch(feed_config: dict) -> list[IntelItem]:
|
||||
items = []
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=30, follow_redirects=True) as client:
|
||||
resp = await client.get(
|
||||
feed_config["url"],
|
||||
headers={"User-Agent": "RMI-ThreatIntel/1.0 (crypto security research)"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
feed = feedparser.parse(resp.text)
|
||||
|
||||
keywords = feed_config.get("keywords", [])
|
||||
|
||||
for entry in feed.entries[:20]: # Max 20 per feed per run
|
||||
title = entry.get("title", "")
|
||||
summary = entry.get("summary", entry.get("description", ""))
|
||||
content = f"{title}\n\n{html.escape(summary)[:2000]}"
|
||||
|
||||
# Keyword filter
|
||||
if keywords:
|
||||
text_lower = (title + " " + summary).lower()
|
||||
if not any(k.lower() in text_lower for k in keywords):
|
||||
continue
|
||||
|
||||
# Extract entities
|
||||
entities = RSSProcessor._extract_entities(content)
|
||||
|
||||
item_id = hashlib.sha256(f"{feed_config['name']}:{entry.get('link', title)}".encode()).hexdigest()[:16]
|
||||
|
||||
items.append(
|
||||
IntelItem(
|
||||
id=item_id,
|
||||
title=title[:300],
|
||||
content=content[:3000],
|
||||
source=feed_config["name"],
|
||||
url=entry.get("link", ""),
|
||||
published=entry.get("published", entry.get("updated", "")),
|
||||
category=feed_config["category"],
|
||||
severity=feed_config["severity"],
|
||||
entities=entities,
|
||||
raw={"feed_title": feed.get("feed", {}).get("title", "")},
|
||||
)
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"RSS fetch failed for {feed_config['name']}: {e}")
|
||||
|
||||
return items
|
||||
|
||||
@staticmethod
|
||||
def _extract_entities(text: str) -> dict[str, list[str]]:
|
||||
found = {}
|
||||
for entity_type, pattern in ENTITY_PATTERNS.items():
|
||||
matches = re.findall(pattern, text)
|
||||
if matches:
|
||||
found[entity_type] = list(set(matches))[:10]
|
||||
return found
|
||||
|
||||
|
||||
class NitterProcessor(RSSProcessor):
|
||||
"""Process Nitter (Twitter mirror) RSS feeds."""
|
||||
|
||||
@staticmethod
|
||||
async def fetch(feed_config: dict) -> list[IntelItem]:
|
||||
items = await RSSProcessor.fetch(feed_config)
|
||||
# Clean up Nitter-specific formatting
|
||||
for item in items:
|
||||
# Extract tweet content from title (format: "user: tweet text")
|
||||
if ":" in item.title:
|
||||
parts = item.title.split(":", 1)
|
||||
item.title = parts[1].strip()[:300]
|
||||
# Clean HTML entities
|
||||
item.content = html.unescape(item.content)
|
||||
return items
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
# ON-CHAIN SCANNER
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
class OnChainScanner:
|
||||
"""Scan new tokens on Solana/Ethereum for immediate risk assessment."""
|
||||
|
||||
@staticmethod
|
||||
async def scan_new_solana_tokens(limit: int = 20) -> list[IntelItem]:
|
||||
"""Scan recently deployed Solana tokens via Helius/Birdeye."""
|
||||
items = []
|
||||
try:
|
||||
# Use Birdeye new listing API or Helius webhook data
|
||||
async with httpx.AsyncClient(timeout=30) as client:
|
||||
# Birdeye trending/new tokens
|
||||
resp = await client.get(
|
||||
"https://public-api.birdeye.so/defi/tokenlist",
|
||||
params={
|
||||
"sort_by": "v24hChangePercent",
|
||||
"sort_type": "desc",
|
||||
"limit": str(limit),
|
||||
},
|
||||
headers={
|
||||
"X-API-KEY": os.getenv("BIRDEYE_API_KEY", ""),
|
||||
"User-Agent": "RMI-Scanner/1.0",
|
||||
},
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
tokens = data.get("data", {}).get("tokens", [])
|
||||
for token in tokens:
|
||||
addr = token.get("address", "")
|
||||
name = token.get("name", "")
|
||||
symbol = token.get("symbol", "")
|
||||
change = token.get("v24hChangePercent", 0)
|
||||
|
||||
# Quick risk heuristics
|
||||
risk_signals = []
|
||||
if abs(change) > 500:
|
||||
risk_signals.append("extreme_volatility")
|
||||
if not name or len(name) < 3:
|
||||
risk_signals.append("suspicious_name")
|
||||
if float(token.get("liquidity", 0)) < 5000:
|
||||
risk_signals.append("low_liquidity")
|
||||
|
||||
if risk_signals:
|
||||
item_id = hashlib.sha256(f"solana_new:{addr}".encode()).hexdigest()[:16]
|
||||
items.append(
|
||||
IntelItem(
|
||||
id=item_id,
|
||||
title=f"New token alert: {name} ({symbol})",
|
||||
content=f"Token {name} ({symbol}) on Solana. Address: {addr}. "
|
||||
f"24h change: {change}%. Risk signals: {', '.join(risk_signals)}. "
|
||||
f"Liquidity: ${token.get('liquidity', 0):,.0f}. Volume 24h: ${token.get('v24hUSD', 0):,.0f}.",
|
||||
source="onchain_scanner",
|
||||
url=f"https://birdeye.so/token/{addr}?chain=solana",
|
||||
published=datetime.now(UTC).isoformat(),
|
||||
category="new_token",
|
||||
severity="medium" if len(risk_signals) < 2 else "high",
|
||||
entities={"sol_address": [addr]},
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Solana scan failed: {e}")
|
||||
|
||||
return items
|
||||
|
||||
@staticmethod
|
||||
async def check_token_security(address: str, chain: str = "solana") -> dict[str, Any]:
|
||||
"""Check a token via GoPlus security API (free, no key needed)."""
|
||||
try:
|
||||
chain_id = {"solana": "solana", "ethereum": "1", "bsc": "56", "base": "8453"}.get(chain, chain)
|
||||
async with httpx.AsyncClient(timeout=15) as client:
|
||||
resp = await client.get(
|
||||
f"https://api.gopluslabs.io/api/v1/token_security/{chain_id}",
|
||||
params={"contract_addresses": address},
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
return data.get("result", {}).get(address.lower(), {})
|
||||
except Exception:
|
||||
pass
|
||||
return {}
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
# INTELLIGENCE PIPELINE ORCHESTRATOR
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
|
||||
|
||||
class IntelPipeline:
|
||||
"""
|
||||
Continuous intelligence ingestion pipeline.
|
||||
|
||||
Usage:
|
||||
pipeline = IntelPipeline()
|
||||
stats = await pipeline.run_cycle()
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._last_run: datetime | None = None
|
||||
self._total_ingested = 0
|
||||
self._seen_ids: set[str] = set()
|
||||
self._feed_processors = {
|
||||
"rss": RSSProcessor,
|
||||
"nitter_rss": NitterProcessor,
|
||||
}
|
||||
|
||||
async def run_cycle(self) -> dict[str, Any]:
|
||||
"""Run one full ingestion cycle across all sources."""
|
||||
from app.crypto_embeddings import get_embedder
|
||||
from app.rag_service import _get_redis
|
||||
|
||||
embedder = await get_embedder()
|
||||
r = await _get_redis()
|
||||
now = datetime.now(UTC)
|
||||
stats = {"feeds": {}, "onchain": {}, "total": 0, "skipped": 0, "errors": 0}
|
||||
|
||||
# Phase 1: RSS feeds
|
||||
for feed_config in FEEDS:
|
||||
if not feed_config.get("enabled", True):
|
||||
continue
|
||||
|
||||
try:
|
||||
processor_class = self._feed_processors.get(feed_config["extractor"], RSSProcessor)
|
||||
items = await processor_class.fetch(feed_config)
|
||||
|
||||
ingested = 0
|
||||
for item in items:
|
||||
if item.id in self._seen_ids:
|
||||
stats["skipped"] += 1
|
||||
continue
|
||||
|
||||
try:
|
||||
# Embed the content
|
||||
result = await embedder.embed_scam_pattern(
|
||||
pattern_name=item.title,
|
||||
description=item.content,
|
||||
severity=item.severity,
|
||||
)
|
||||
|
||||
# Store in Redis
|
||||
doc = {
|
||||
"id": item.id,
|
||||
"collection": "market_intel",
|
||||
"vector": result.vector,
|
||||
"dims": result.dims,
|
||||
"model": result.model,
|
||||
"metadata": {
|
||||
"source": item.source,
|
||||
"url": item.url,
|
||||
"published": item.published,
|
||||
"category": item.category,
|
||||
"severity": item.severity,
|
||||
"entities": item.entities,
|
||||
},
|
||||
"content": item.content,
|
||||
"source": item.source,
|
||||
"severity": item.severity,
|
||||
"stored_at": now.isoformat(),
|
||||
}
|
||||
await r.setex(
|
||||
f"rag:market_intel:{item.id}",
|
||||
86400 * 7, # 7-day TTL for news
|
||||
json.dumps(doc),
|
||||
)
|
||||
await r.sadd("rag:idx:market_intel", item.id)
|
||||
self._seen_ids.add(item.id)
|
||||
ingested += 1
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to ingest {item.id}: {e}")
|
||||
|
||||
stats["feeds"][feed_config["name"]] = ingested
|
||||
stats["total"] += ingested
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Feed {feed_config['name']} failed: {e}")
|
||||
stats["errors"] += 1
|
||||
stats["feeds"][feed_config["name"]] = 0
|
||||
|
||||
# Phase 2: On-chain scanning
|
||||
try:
|
||||
new_tokens = await OnChainScanner.scan_new_solana_tokens(limit=10)
|
||||
ingested = 0
|
||||
for item in new_tokens:
|
||||
if item.id in self._seen_ids:
|
||||
continue
|
||||
try:
|
||||
result = await embedder.embed_scam_pattern(
|
||||
pattern_name=item.title,
|
||||
description=item.content,
|
||||
severity=item.severity,
|
||||
)
|
||||
doc = {
|
||||
"id": item.id,
|
||||
"collection": "token_analysis",
|
||||
"vector": result.vector,
|
||||
"dims": result.dims,
|
||||
"metadata": {
|
||||
"source": item.source,
|
||||
"category": item.category,
|
||||
"severity": item.severity,
|
||||
"entities": item.entities,
|
||||
},
|
||||
"content": item.content,
|
||||
"source": item.source,
|
||||
"severity": item.severity,
|
||||
"stored_at": now.isoformat(),
|
||||
}
|
||||
await r.setex(f"rag:token_analysis:{item.id}", 86400 * 7, json.dumps(doc))
|
||||
await r.sadd("rag:idx:token_analysis", item.id)
|
||||
self._seen_ids.add(item.id)
|
||||
ingested += 1
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to ingest onchain {item.id}: {e}")
|
||||
|
||||
stats["onchain"]["new_tokens"] = ingested
|
||||
stats["total"] += ingested
|
||||
except Exception as e:
|
||||
logger.warning(f"On-chain scan failed: {e}")
|
||||
|
||||
self._last_run = now
|
||||
self._total_ingested += stats["total"]
|
||||
|
||||
# Cleanup old entries (older than 7 days)
|
||||
try:
|
||||
old_cutoff = (now - timedelta(days=7)).isoformat()
|
||||
for coll in ["market_intel", "token_analysis"]:
|
||||
all_ids = await r.smembers(f"rag:idx:{coll}")
|
||||
for did in all_ids:
|
||||
doc_raw = await r.get(f"rag:{coll}:{did}")
|
||||
if doc_raw:
|
||||
try:
|
||||
doc = json.loads(doc_raw)
|
||||
if doc.get("stored_at", "") < old_cutoff:
|
||||
await r.delete(f"rag:{coll}:{did}")
|
||||
await r.srem(f"rag:idx:{coll}", did)
|
||||
except Exception:
|
||||
pass
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"status": "completed",
|
||||
"total_ingested": stats["total"],
|
||||
"cumulative": self._total_ingested,
|
||||
"feeds": stats["feeds"],
|
||||
"onchain": stats["onchain"],
|
||||
"skipped": stats["skipped"],
|
||||
"errors": stats["errors"],
|
||||
"seen_cache_size": len(self._seen_ids),
|
||||
"timestamp": now.isoformat(),
|
||||
"next_run_in": "30 minutes",
|
||||
}
|
||||
|
||||
async def get_latest_intel(self, limit: int = 20) -> list[dict[str, Any]]:
|
||||
"""Get latest threat intelligence for frontend display."""
|
||||
from app.rag_service import _get_redis
|
||||
|
||||
r = await _get_redis()
|
||||
|
||||
results = []
|
||||
for coll in ["market_intel", "token_analysis", "known_scams"]:
|
||||
doc_ids = await r.smembers(f"rag:idx:{coll}")
|
||||
for did in list(doc_ids)[:limit]:
|
||||
raw = await r.get(f"rag:{coll}:{did}")
|
||||
if raw:
|
||||
try:
|
||||
doc = json.loads(raw)
|
||||
results.append(
|
||||
{
|
||||
"id": doc["id"],
|
||||
"title": doc.get("metadata", {}).get("name", doc.get("content", "")[:100]),
|
||||
"content": doc.get("content", "")[:500],
|
||||
"source": doc.get("source", coll),
|
||||
"severity": doc.get("severity", "medium"),
|
||||
"category": doc.get("metadata", {}).get("category", ""),
|
||||
"url": doc.get("metadata", {}).get("url", ""),
|
||||
"published": doc.get("metadata", {}).get("published", ""),
|
||||
"entities": doc.get("metadata", {}).get("entities", {}),
|
||||
"stored_at": doc.get("stored_at", ""),
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
results.sort(key=lambda x: x.get("stored_at", ""), reverse=True)
|
||||
return results[:limit]
|
||||
|
||||
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
# SINGLETON
|
||||
# ══════════════════════════════════════════════════════════════════════
|
||||
|
||||
_pipeline: IntelPipeline | None = None
|
||||
|
||||
|
||||
async def get_pipeline() -> IntelPipeline:
|
||||
global _pipeline
|
||||
if _pipeline is None:
|
||||
_pipeline = IntelPipeline()
|
||||
return _pipeline
|
||||
|
|
@ -1,64 +0,0 @@
|
|||
"""
|
||||
Universal API key loader with cascading fallbacks.
|
||||
Resolves Docker env-loading issues by checking multiple sources.
|
||||
Pattern: env var → /tmp/{name}_key.txt → /root/.secrets/sentinel_enrichment_keys.env → None
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
|
||||
def load_key(name: str) -> str:
|
||||
"""Load an API key with multiple fallback sources.
|
||||
|
||||
Checks in order:
|
||||
1. Environment variable (os.getenv)
|
||||
2. File-based fallback at /tmp/{name}_key.txt (Docker workaround)
|
||||
3. Secrets file at /root/.secrets/sentinel_enrichment_keys.env
|
||||
|
||||
Returns empty string if not found.
|
||||
"""
|
||||
# 1. Environment variable
|
||||
key = os.getenv(name, "")
|
||||
if key:
|
||||
return key
|
||||
|
||||
# 2. File-based fallback (works around Docker env sanitization)
|
||||
file_path = f"/tmp/{name.lower().replace('_api_key', '')}_key.txt"
|
||||
try:
|
||||
if os.path.exists(file_path):
|
||||
with open(file_path) as f:
|
||||
key = f.read().strip()
|
||||
if key:
|
||||
return key
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 3. Central secrets file
|
||||
secrets_path = "/root/.secrets/sentinel_enrichment_keys.env"
|
||||
try:
|
||||
if os.path.exists(secrets_path):
|
||||
with open(secrets_path) as f:
|
||||
for line in f:
|
||||
if line.startswith(f"{name}="):
|
||||
key = line.split("=", 1)[1].split("#")[0].strip()
|
||||
if key:
|
||||
return key
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return ""
|
||||
|
||||
|
||||
# Enrichment key service name → env var mapping
|
||||
SERVICE_KEYS = {
|
||||
"nansen": "NANSEN_API_KEY",
|
||||
"chainaware": "CHAIN_AWARE_API_KEY",
|
||||
"dune": "DUNE_API_KEY",
|
||||
"santiment": "SANTIMENT_API_KEY",
|
||||
"thegraph": "THEGRAPH_API_KEY",
|
||||
"defi": "DEFI_API_KEY",
|
||||
"webacy": "WEBACY_API_KEY",
|
||||
"lunarcrush": "LUNARCRUSH_API_KEY",
|
||||
"arkham": "ARKHAM_API_KEY",
|
||||
"blowfish": "BLOWFISH_API_KEY",
|
||||
}
|
||||
|
|
@ -1,176 +0,0 @@
|
|||
"""
|
||||
Liquidity Watchdog Router
|
||||
=========================
|
||||
FastAPI endpoints for monitoring liquidity lock expiries.
|
||||
|
||||
Endpoints:
|
||||
GET /api/v1/alerts/liquidity-expiring → Upcoming lock expiries within 24h
|
||||
POST /api/v1/alerts/liquidity-track → Register a token for monitoring
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from app.liquidity_watchdog import (
|
||||
check_all_locks_summary,
|
||||
check_expiring_locks,
|
||||
get_tracked_tokens,
|
||||
track_token,
|
||||
untrack_token,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/v1/alerts", tags=["liquidity-watchdog"])
|
||||
|
||||
|
||||
# ── Request/Response models ──────────────────────────────────────
|
||||
|
||||
|
||||
class LiquidityTrackRequest(BaseModel):
|
||||
token_address: str = Field(..., description="Token contract/mint address")
|
||||
chain: str = Field(..., description="Blockchain name (e.g. solana, ethereum, bsc)")
|
||||
lock_expiry_timestamp: int = Field(..., description="Unix timestamp (seconds) when lock expires")
|
||||
liquidity_amount: float = Field(0.0, description="Locked liquidity amount in USD")
|
||||
|
||||
|
||||
class LiquidityTrackResponse(BaseModel):
|
||||
status: str = "ok"
|
||||
message: str = "Token registered for liquidity expiry monitoring"
|
||||
data: dict[str, Any]
|
||||
|
||||
|
||||
class ExpiringLock(BaseModel):
|
||||
token: str
|
||||
chain: str
|
||||
lock_expiry: int
|
||||
hours_remaining: float
|
||||
liquidity_amount: float
|
||||
|
||||
|
||||
class LiquidityExpiringResponse(BaseModel):
|
||||
status: str = "ok"
|
||||
count: int = 0
|
||||
expiring: list[ExpiringLock] = []
|
||||
|
||||
|
||||
class LiquiditySummaryResponse(BaseModel):
|
||||
status: str = "ok"
|
||||
total_tracked: int = 0
|
||||
already_expired: int = 0
|
||||
expiring_soon: int = 0
|
||||
safe: int = 0
|
||||
expiring_tokens: list[ExpiringLock] = []
|
||||
|
||||
|
||||
# ── Endpoints ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/liquidity-track", response_model=LiquidityTrackResponse)
|
||||
async def post_liquidity_track(req: LiquidityTrackRequest):
|
||||
"""Register a token for liquidity lock expiry monitoring.
|
||||
|
||||
This will be called by a cron job that pings Telegram/community channels
|
||||
when locks are about to expire.
|
||||
|
||||
The token's lock expiry is checked against the current time to determine
|
||||
if it's already expired (returns a warning but still registers it).
|
||||
"""
|
||||
if not req.token_address or not req.token_address.strip():
|
||||
raise HTTPException(status_code=400, detail="token_address is required")
|
||||
if not req.chain or not req.chain.strip():
|
||||
raise HTTPException(status_code=400, detail="chain is required")
|
||||
if req.lock_expiry_timestamp <= 0:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="lock_expiry_timestamp must be a positive unix timestamp",
|
||||
)
|
||||
|
||||
data = track_token(
|
||||
token_address=req.token_address,
|
||||
chain=req.chain,
|
||||
lock_expiry_timestamp=req.lock_expiry_timestamp,
|
||||
liquidity_amount=req.liquidity_amount or 0.0,
|
||||
)
|
||||
|
||||
return LiquidityTrackResponse(
|
||||
status="ok",
|
||||
message="Token registered for liquidity expiry monitoring",
|
||||
data=data,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/liquidity-expiring", response_model=LiquidityExpiringResponse)
|
||||
async def get_liquidity_expiring(
|
||||
hours: float | None = Query(
|
||||
None,
|
||||
description="Override the expiry window in hours (default: 24). "
|
||||
"Only returns locks expiring within this many hours.",
|
||||
),
|
||||
):
|
||||
"""Get all tracked tokens with liquidity locks expiring within the default
|
||||
24-hour window (or a custom window via the `hours` query parameter).
|
||||
|
||||
Returns tokens sorted by urgency (most imminent first).
|
||||
"""
|
||||
if hours is not None and hours <= 0:
|
||||
raise HTTPException(status_code=400, detail="hours must be > 0")
|
||||
|
||||
# If a custom hours window is given, we temporarily override the module's
|
||||
# default and re-check. Since check_expiring_locks() uses a module constant,
|
||||
# we just call it once and filter further if needed.
|
||||
expiring = check_expiring_locks()
|
||||
|
||||
if hours is not None:
|
||||
expiring = [e for e in expiring if e["hours_remaining"] <= hours]
|
||||
|
||||
expiring_models = [ExpiringLock(**e) for e in expiring]
|
||||
|
||||
return LiquidityExpiringResponse(
|
||||
status="ok",
|
||||
count=len(expiring_models),
|
||||
expiring=expiring_models,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/liquidity-tracked", response_model=dict)
|
||||
async def get_liquidity_tracked():
|
||||
"""List all tokens currently being monitored for liquidity expiry."""
|
||||
tokens = get_tracked_tokens()
|
||||
return {
|
||||
"status": "ok",
|
||||
"count": len(tokens),
|
||||
"tokens": tokens,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/liquidity-summary", response_model=LiquiditySummaryResponse)
|
||||
async def get_liquidity_summary():
|
||||
"""Get a summary of all tracked liquidity locks with counts."""
|
||||
summary = check_all_locks_summary()
|
||||
return LiquiditySummaryResponse(
|
||||
status="ok",
|
||||
total_tracked=summary["total_tracked"],
|
||||
already_expired=summary["already_expired"],
|
||||
expiring_soon=summary["expiring_soon"],
|
||||
safe=summary["safe"],
|
||||
expiring_tokens=[ExpiringLock(**e) for e in summary["expiring_tokens"]],
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/liquidity-track/{chain}/{token_address}", response_model=dict)
|
||||
async def delete_liquidity_track(chain: str, token_address: str):
|
||||
"""Remove a token from liquidity expiry monitoring."""
|
||||
removed = untrack_token(token_address, chain)
|
||||
if not removed:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Token {token_address} on {chain} is not being tracked",
|
||||
)
|
||||
return {
|
||||
"status": "ok",
|
||||
"message": f"Token {token_address} on {chain} removed from monitoring",
|
||||
}
|
||||
|
|
@ -1,228 +0,0 @@
|
|||
"""
|
||||
Email Dashboard - Full email management for rugmunch.io + cryptorugmunch.com
|
||||
Access cryptorugmuncher@gmail.com via backend (no password needed).
|
||||
"""
|
||||
|
||||
import email as emaillib
|
||||
import imaplib
|
||||
import logging
|
||||
import os
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/api/v1/mail", tags=["mail"])
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# EMAIL ADDRESSES
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
EMAILS = {
|
||||
"rugmunch.io": [
|
||||
{
|
||||
"address": "team@rugmunch.io",
|
||||
"purpose": "General inquiries",
|
||||
"forward_to": "cryptorugmuncher@gmail.com",
|
||||
},
|
||||
{
|
||||
"address": "alerts@rugmunch.io",
|
||||
"purpose": "Security alerts & notifications",
|
||||
"forward_to": "cryptorugmuncher@gmail.com",
|
||||
},
|
||||
{
|
||||
"address": "intel@rugmunch.io",
|
||||
"purpose": "Intelligence reports",
|
||||
"forward_to": "cryptorugmuncher@gmail.com",
|
||||
},
|
||||
{
|
||||
"address": "support@rugmunch.io",
|
||||
"purpose": "Customer support",
|
||||
"forward_to": "cryptorugmuncher@gmail.com",
|
||||
},
|
||||
{
|
||||
"address": "admin@rugmunch.io",
|
||||
"purpose": "Admin/backend",
|
||||
"forward_to": "cryptorugmuncher@gmail.com",
|
||||
},
|
||||
{
|
||||
"address": "newsletter@rugmunch.io",
|
||||
"purpose": "Weekly/daily newsletter dispatches",
|
||||
"forward_to": "cryptorugmuncher@gmail.com",
|
||||
},
|
||||
{
|
||||
"address": "biz@rugmunch.io",
|
||||
"purpose": "Business relationships & partnerships",
|
||||
"forward_to": "cryptorugmuncher@gmail.com",
|
||||
},
|
||||
{
|
||||
"address": "bot@rugmunch.io",
|
||||
"purpose": "Bot/automation account (X/Telegram/backend alerts)",
|
||||
"forward_to": "admin@rugmunch.io",
|
||||
},
|
||||
],
|
||||
"cryptorugmunch.com": [
|
||||
{
|
||||
"address": "team@cryptorugmunch.com",
|
||||
"purpose": "Main contact",
|
||||
"forward_to": "cryptorugmuncher@gmail.com",
|
||||
},
|
||||
{
|
||||
"address": "alerts@cryptorugmunch.com",
|
||||
"purpose": "Alert system",
|
||||
"forward_to": "cryptorugmuncher@gmail.com",
|
||||
},
|
||||
{
|
||||
"address": "intel@cryptorugmunch.com",
|
||||
"purpose": "Intel feed",
|
||||
"forward_to": "cryptorugmuncher@gmail.com",
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/addresses")
|
||||
async def list_addresses():
|
||||
"""All email addresses across domains."""
|
||||
return {
|
||||
"domains": list(EMAILS.keys()),
|
||||
"addresses": EMAILS,
|
||||
"total": sum(len(v) for v in EMAILS.values()),
|
||||
"primary_inbox": "cryptorugmuncher@gmail.com",
|
||||
"provider": "gmail_smtp + listmonk_newsletter",
|
||||
}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# GMAIL INBOX - Read without password
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/inbox")
|
||||
async def check_inbox(limit: int = 10):
|
||||
"""Check cryptorugmuncher@gmail.com inbox (requires app password in env)."""
|
||||
app_password = os.getenv("GMAIL_APP_PASSWORD", "")
|
||||
if not app_password:
|
||||
return {
|
||||
"emails": [],
|
||||
"error": "GMAIL_APP_PASSWORD not set. Get one at https://myaccount.google.com/apppasswords",
|
||||
"account": "cryptorugmuncher@gmail.com",
|
||||
}
|
||||
|
||||
try:
|
||||
mail = imaplib.IMAP4_SSL("imap.gmail.com", 993)
|
||||
mail.login("cryptorugmuncher@gmail.com", app_password)
|
||||
mail.select("INBOX")
|
||||
|
||||
status, messages = mail.search(None, "ALL") # noqa: RUF059
|
||||
email_ids = messages[0].split()[-limit:] if messages[0] else []
|
||||
|
||||
emails = []
|
||||
for eid in reversed(email_ids):
|
||||
_status, msg_data = mail.fetch(eid, "(RFC822)")
|
||||
if msg_data and msg_data[0]:
|
||||
raw = emaillib.message_from_bytes(msg_data[0][1])
|
||||
emails.append(
|
||||
{
|
||||
"id": eid.decode(),
|
||||
"from": raw.get("From", ""),
|
||||
"subject": raw.get("Subject", ""),
|
||||
"date": raw.get("Date", ""),
|
||||
"snippet": _get_body(raw)[:200],
|
||||
}
|
||||
)
|
||||
|
||||
mail.close()
|
||||
mail.logout()
|
||||
|
||||
return {
|
||||
"account": "cryptorugmuncher@gmail.com",
|
||||
"total_inbox": len(email_ids),
|
||||
"showing": len(emails),
|
||||
"emails": emails,
|
||||
"checked_at": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
except Exception as e:
|
||||
return {"error": str(e), "account": "cryptorugmuncher@gmail.com"}
|
||||
|
||||
|
||||
def _get_body(msg) -> str:
|
||||
"""Extract text body from email."""
|
||||
try:
|
||||
if msg.is_multipart():
|
||||
for part in msg.walk():
|
||||
if part.get_content_type() == "text/plain":
|
||||
payload = part.get_payload(decode=True)
|
||||
if payload:
|
||||
return payload.decode(errors="ignore")
|
||||
payload = msg.get_payload(decode=True)
|
||||
if payload:
|
||||
return payload.decode(errors="ignore")
|
||||
except Exception:
|
||||
pass
|
||||
return "(no text content)"
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# SEND - From any rugmunch.io address
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.post("/send")
|
||||
async def send_mail(data: dict):
|
||||
"""Send email from any rugmunch.io/cryptorugmunch.com address via Gmail SMTP."""
|
||||
to = data.get("to", "")
|
||||
subject = data.get("subject", "")
|
||||
body = data.get("body", "")
|
||||
from_addr = data.get("from", "team@rugmunch.io")
|
||||
|
||||
if not to or not body:
|
||||
raise HTTPException(status_code=400, detail="to and body required")
|
||||
|
||||
app_password = os.getenv("GMAIL_APP_PASSWORD", "")
|
||||
if not app_password:
|
||||
return {"status": "failed", "error": "GMAIL_APP_PASSWORD not set"}
|
||||
|
||||
try:
|
||||
import smtplib
|
||||
from email.mime.multipart import MIMEMultipart
|
||||
from email.mime.text import MIMEText
|
||||
|
||||
msg = MIMEMultipart()
|
||||
msg["From"] = f"Rug Munch Intelligence <{from_addr}>"
|
||||
msg["To"] = to
|
||||
msg["Subject"] = subject
|
||||
msg["Reply-To"] = "team@rugmunch.io"
|
||||
msg.attach(MIMEText(body, "html"))
|
||||
|
||||
with smtplib.SMTP("smtp.gmail.com", 587, timeout=15) as server:
|
||||
server.starttls()
|
||||
server.login("cryptorugmuncher@gmail.com", app_password)
|
||||
server.send_message(msg)
|
||||
|
||||
return {
|
||||
"status": "sent",
|
||||
"from": from_addr,
|
||||
"to": to,
|
||||
"subject": subject,
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
except Exception as e:
|
||||
return {"status": "failed", "error": str(e)}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# DOMAINS SETUP
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
@router.get("/domains")
|
||||
async def mail_domains():
|
||||
return {
|
||||
"domains": [
|
||||
{"domain": "rugmunch.io", "status": "active", "mx": "smtp.gmail.com", "emails": 5},
|
||||
{
|
||||
"domain": "cryptorugmunch.com",
|
||||
"status": "active",
|
||||
"mx": "smtp.gmail.com",
|
||||
"emails": 3,
|
||||
},
|
||||
],
|
||||
"smtp_provider": "gmail",
|
||||
"smtp_account": "cryptorugmuncher@gmail.com",
|
||||
"newsletter": "listmonk (localhost:9001)",
|
||||
"setup_note": "Add GMAIL_APP_PASSWORD to enable full email functionality",
|
||||
}
|
||||
|
|
@ -1,380 +0,0 @@
|
|||
"""
|
||||
Marketing Content Templates - Auto-generated posts for X, Telegram, Discord.
|
||||
Professional copy for wins, losses, KOL scorecards, security alerts.
|
||||
"""
|
||||
|
||||
# ── X/Twitter Post Templates ──────────────────────────────────
|
||||
|
||||
X_TEMPLATES = {
|
||||
"big_win": """🎉 BIG WIN ALERT! 🎉
|
||||
|
||||
Wallet: {wallet_short}
|
||||
Token: {token_symbol}
|
||||
Profit: +${pnl_usd:,.0f} ({pnl_pct:.1f}%)
|
||||
|
||||
This whale called it early and rode it all the way up! 🐋
|
||||
|
||||
Track smart money: rugmunch.io/wallet/{wallet_address}
|
||||
|
||||
#Crypto #MemeCoin #SmartMoney #Trading
|
||||
|
||||
@cryptorugmunch
|
||||
""",
|
||||
"big_loss": """💀 LOSS PORN 💀
|
||||
|
||||
Wallet: {wallet_short}
|
||||
Token: {token_symbol}
|
||||
Loss: -${pnl_usd:,.0f} ({pnl_pct:.1f}%)
|
||||
|
||||
Oof. Another reminder to take profits, friends. 📉
|
||||
|
||||
Learn from their mistakes: rugmunch.io/wallet/{wallet_address}
|
||||
|
||||
#Crypto #Trading #LossPorn #DeFi
|
||||
|
||||
@cryptorugmunch
|
||||
""",
|
||||
"kol_scorecard": """📊 KOL SCORECARD: @{kol_handle}
|
||||
|
||||
Win Rate: {win_rate:.1f}%
|
||||
Total Calls: {total_calls}
|
||||
Avg PnL: {avg_pnl:.1f}%
|
||||
Followers: {follower_count:,}
|
||||
|
||||
Tier: {tier}
|
||||
|
||||
Track their calls: rugmunch.io/kol/{kol_handle}
|
||||
|
||||
#CryptoTwitter #KOL #Alpha #Trading
|
||||
|
||||
@cryptorugmunch
|
||||
""",
|
||||
"rugpull": """🚨 RUG PULL DETECTED 🚨
|
||||
|
||||
Token: {token_symbol}
|
||||
Stolen: ${amount_stolen:,.0f}
|
||||
Victims: {victims}
|
||||
Type: {rug_type}
|
||||
|
||||
Deployer: {deployer_short}
|
||||
|
||||
⚠️ STAY AWAY FROM THIS TOKEN
|
||||
|
||||
Full analysis: rugmunch.io/rug/{token_address}
|
||||
|
||||
#RugPull #CryptoScam #DeFi #StaySafe
|
||||
|
||||
@cryptorugmunch
|
||||
""",
|
||||
"whale_move": """🐋 WHALE ALERT 🐋
|
||||
|
||||
{wallet_label}
|
||||
{action} {amount:,.0f} {symbol}
|
||||
Value: ${usd_value:,.0f}
|
||||
|
||||
Destination: {destination}
|
||||
|
||||
Track this whale: rugmunch.io/whale/{wallet_address}
|
||||
|
||||
#WhaleAlert #Crypto #SmartMoney
|
||||
|
||||
@cryptorugmunch
|
||||
""",
|
||||
"smart_money": """🧠 SMART MONEY PICK 🧠
|
||||
|
||||
{wallet_label}
|
||||
Win Rate: {win_rate:.1f}%
|
||||
|
||||
Just bought: {token_symbol}
|
||||
Position: ${position_size:,.0f}
|
||||
Entry: ${entry_price}
|
||||
Current PnL: +{current_pnl:.1f}%
|
||||
|
||||
Follow their moves: rugmunch.io/wallet/{wallet_address}
|
||||
|
||||
#SmartMoney #Crypto #Alpha #Trading
|
||||
|
||||
@cryptorugmunch
|
||||
""",
|
||||
"hack_alert": """🚨 HACK ALERT 🚨
|
||||
|
||||
Protocol: {protocol_name}
|
||||
Amount: ${amount_usd:,.0f}
|
||||
Chain: {chain}
|
||||
Type: {attack_type}
|
||||
|
||||
{description}
|
||||
|
||||
Stay safe out there! 🔒
|
||||
|
||||
#CryptoSecurity #DeFi #HackAlert
|
||||
|
||||
@cryptorugmunch
|
||||
""",
|
||||
"launch_announcement": """🚀 RMI INTELLIGENCE PLATFORM IS LIVE
|
||||
|
||||
Track smart money. Avoid rugs. Find alpha.
|
||||
|
||||
✅ Real-time whale tracking
|
||||
✅ KOL scorecards
|
||||
✅ Rugpull detection
|
||||
✅ Meme intelligence
|
||||
|
||||
Join now: rugmunch.io
|
||||
|
||||
#Crypto #DeFi #Trading #Alpha
|
||||
|
||||
@cryptorugmunch
|
||||
""",
|
||||
}
|
||||
|
||||
# ── Telegram Post Templates ───────────────────────────────────
|
||||
|
||||
TELEGRAM_TEMPLATES = {
|
||||
"big_win": """🎉 *BIG WIN ALERT!* 🎉
|
||||
|
||||
*Wallet:* `{wallet_short}`
|
||||
*Token:* {token_symbol}
|
||||
*Profit:* +${pnl_usd:,.0f} (*{pnl_pct:.1f}%*)
|
||||
|
||||
This whale called it early and rode it all the way up! 🐋
|
||||
|
||||
📊 *Track smart money:* rugmunch.io/wallet/{wallet_address}
|
||||
|
||||
#Crypto #MemeCoin #SmartMoney #Trading
|
||||
""",
|
||||
"big_loss": """💀 *LOSS PORN* 💀
|
||||
|
||||
*Wallet:* `{wallet_short}`
|
||||
*Token:* {token_symbol}
|
||||
*Loss:* -${pnl_usd:,.0f} (*{pnl_pct:.1f}%)
|
||||
|
||||
Oof. Another reminder to take profits, friends. 📉
|
||||
|
||||
📚 *Learn from their mistakes:* rugmunch.io/wallet/{wallet_address}
|
||||
|
||||
#Crypto #Trading #LossPorn #DeFi
|
||||
""",
|
||||
"kol_scorecard": """📊 *KOL SCORECARD: @{kol_handle}*
|
||||
|
||||
*Win Rate:* {win_rate:.1f}%
|
||||
*Total Calls:* {total_calls}
|
||||
*Avg PnL:* {avg_pnl:.1f}%
|
||||
*Followers:* {follower_count:,}
|
||||
|
||||
*Tier:* {tier}
|
||||
|
||||
📈 *Track their calls:* rugmunch.io/kol/{kol_handle}
|
||||
|
||||
#CryptoTwitter #KOL #Alpha #Trading
|
||||
""",
|
||||
"rugpull": """🚨 *RUG PULL DETECTED* 🚨
|
||||
|
||||
*Token:* {token_symbol}
|
||||
*Stolen:* ${amount_stolen:,.0f}
|
||||
*Victims:* {victims}
|
||||
*Type:* {rug_type}
|
||||
|
||||
*Deployer:* `{deployer_short}`
|
||||
|
||||
⚠️ *STAY AWAY FROM THIS TOKEN*
|
||||
|
||||
🔍 *Full analysis:* rugmunch.io/rug/{token_address}
|
||||
|
||||
#RugPull #CryptoScam #DeFi #StaySafe
|
||||
""",
|
||||
"whale_move": """🐋 *WHALE ALERT* 🐋
|
||||
|
||||
*{wallet_label}*
|
||||
*{action}* {amount:,.0f} {symbol}
|
||||
*Value:* ${usd_value:,.0f}
|
||||
|
||||
*Destination:* {destination}
|
||||
|
||||
📊 *Track this whale:* rugmunch.io/whale/{wallet_address}
|
||||
|
||||
#WhaleAlert #Crypto #SmartMoney
|
||||
""",
|
||||
"smart_money": """🧠 *SMART MONEY PICK* 🧠
|
||||
|
||||
*{wallet_label}*
|
||||
*Win Rate:* {win_rate:.1f}%
|
||||
|
||||
Just bought: *{token_symbol}*
|
||||
*Position:* ${position_size:,.0f}
|
||||
*Entry:* ${entry_price}
|
||||
*Current PnL:* +{current_pnl:.1f}%
|
||||
|
||||
📈 *Follow their moves:* rugmunch.io/wallet/{wallet_address}
|
||||
|
||||
#SmartMoney #Crypto #Alpha #Trading
|
||||
""",
|
||||
"daily_roundup": """📊 *DAILY INTELLIGENCE ROUNDUP*
|
||||
|
||||
📅 {date}
|
||||
|
||||
🎯 *Top Wins:*
|
||||
{top_wins}
|
||||
|
||||
💀 *Top Losses:*
|
||||
{top_losses}
|
||||
|
||||
🐋 *Notable Whale Moves:*
|
||||
{whale_moves}
|
||||
|
||||
🚨 *Security Alerts:*
|
||||
{security_alerts}
|
||||
|
||||
Full report: rugmunch.io/daily/{date}
|
||||
|
||||
#Crypto #DailyRoundup #Intelligence
|
||||
""",
|
||||
"weekly_kol_rankings": """📊 *WEEKLY KOL RANKINGS*
|
||||
|
||||
Week {week_number}
|
||||
|
||||
🥇 *#1:* @{kol_1} - {win_rate_1:.1f}% win rate
|
||||
🥈 *#2:* @{kol_2} - {win_rate_2:.1f}% win rate
|
||||
🥉 *#3:* @{kol_3} - {win_rate_3:.1f}% win rate
|
||||
|
||||
See full leaderboard: rugmunch.io/kol/leaderboard
|
||||
|
||||
#KOL #CryptoTwitter #Rankings
|
||||
""",
|
||||
}
|
||||
|
||||
# ── Discord Post Templates ────────────────────────────────────
|
||||
|
||||
DISCORD_TEMPLATES = {
|
||||
"big_win": """
|
||||
🎉 **BIG WIN ALERT!** 🎉
|
||||
|
||||
**Wallet:** `{wallet_short}`
|
||||
**Token:** {token_symbol}
|
||||
**Profit:** +${pnl_usd:,.0f} (**{pnl_pct:.1f}%**)
|
||||
|
||||
This whale called it early and rode it all the way up! 🐋
|
||||
|
||||
📊 **Track smart money:** https://rugmunch.io/wallet/{wallet_address}
|
||||
""",
|
||||
"big_loss": """
|
||||
💀 **LOSS PORN** 💀
|
||||
|
||||
**Wallet:** `{wallet_short}`
|
||||
**Token:** {token_symbol}
|
||||
**Loss:** -${pnl_usd:,.0f} (**{pnl_pct:.1f}%**)
|
||||
|
||||
Oof. Another reminder to take profits, friends. 📉
|
||||
|
||||
📚 **Learn from their mistakes:** https://rugmunch.io/wallet/{wallet_address}
|
||||
""",
|
||||
"kol_scorecard": """
|
||||
📊 **KOL SCORECARD: @{kol_handle}**
|
||||
|
||||
**Win Rate:** {win_rate:.1f}%
|
||||
**Total Calls:** {total_calls}
|
||||
**Avg PnL:** {avg_pnl:.1f}%
|
||||
**Followers:** {follower_count:,}
|
||||
|
||||
**Tier:** {tier}
|
||||
|
||||
📈 **Track their calls:** https://rugmunch.io/kol/{kol_handle}
|
||||
""",
|
||||
"rugpull": """
|
||||
🚨 **RUG PULL DETECTED** 🚨
|
||||
|
||||
**Token:** {token_symbol}
|
||||
**Stolen:** ${amount_stolen:,.0f}
|
||||
**Victims:** {victims}
|
||||
**Type:** {rug_type}
|
||||
|
||||
**Deployer:** `{deployer_short}`
|
||||
|
||||
⚠️ **STAY AWAY FROM THIS TOKEN**
|
||||
|
||||
🔍 **Full analysis:** https://rugmunch.io/rug/{token_address}
|
||||
|
||||
@everyone Stay safe!
|
||||
""",
|
||||
"announcement": """
|
||||
📢 **RMI ANNOUNCEMENT**
|
||||
|
||||
{message}
|
||||
|
||||
🔗 **Learn more:** https://rugmunch.io
|
||||
|
||||
{reactions}
|
||||
""",
|
||||
}
|
||||
|
||||
# ── Content Generation Functions ──────────────────────────────
|
||||
|
||||
|
||||
def format_wallet_short(address: str) -> str:
|
||||
"""Format wallet address for display."""
|
||||
if len(address) >= 14:
|
||||
return f"{address[:8]}...{address[-6:]}"
|
||||
return address
|
||||
|
||||
|
||||
def generate_x_post(template_name: str, data: dict) -> str:
|
||||
"""Generate X/Twitter post from template."""
|
||||
template = X_TEMPLATES.get(template_name, "")
|
||||
|
||||
# Add wallet_short if wallet_address provided
|
||||
if "wallet_address" in data and "wallet_short" not in data:
|
||||
data["wallet_short"] = format_wallet_short(data["wallet_address"])
|
||||
if "deployer_address" in data and "deployer_short" not in data:
|
||||
data["deployer_short"] = format_wallet_short(data["deployer_address"])
|
||||
|
||||
# Format numbers
|
||||
for key, value in data.items():
|
||||
if isinstance(value, float):
|
||||
data[key] = round(value, 2)
|
||||
|
||||
return template.format(**data)
|
||||
|
||||
|
||||
def generate_telegram_post(template_name: str, data: dict) -> str:
|
||||
"""Generate Telegram post from template (Markdown)."""
|
||||
template = TELEGRAM_TEMPLATES.get(template_name, "")
|
||||
|
||||
# Add wallet_short if wallet_address provided
|
||||
if "wallet_address" in data and "wallet_short" not in data:
|
||||
data["wallet_short"] = format_wallet_short(data["wallet_address"])
|
||||
if "deployer_address" in data and "deployer_short" not in data:
|
||||
data["deployer_short"] = format_wallet_short(data["deployer_address"])
|
||||
|
||||
# Format numbers
|
||||
for key, value in data.items():
|
||||
if isinstance(value, float):
|
||||
data[key] = round(value, 2)
|
||||
|
||||
return template.format(**data)
|
||||
|
||||
|
||||
def generate_discord_post(template_name: str, data: dict) -> str:
|
||||
"""Generate Discord post from template."""
|
||||
template = DISCORD_TEMPLATES.get(template_name, "")
|
||||
|
||||
# Add wallet_short if wallet_address provided
|
||||
if "wallet_address" in data and "wallet_short" not in data:
|
||||
data["wallet_short"] = format_wallet_short(data["wallet_address"])
|
||||
if "deployer_address" in data and "deployer_short" not in data:
|
||||
data["deployer_short"] = format_wallet_short(data["deployer_address"])
|
||||
|
||||
# Format numbers
|
||||
for key, value in data.items():
|
||||
if isinstance(value, float):
|
||||
data[key] = round(value, 2)
|
||||
|
||||
return template.format(**data)
|
||||
|
||||
|
||||
def generate_all_posts(template_name: str, data: dict) -> dict:
|
||||
"""Generate posts for all platforms."""
|
||||
return {
|
||||
"x": generate_x_post(template_name, data),
|
||||
"telegram": generate_telegram_post(template_name, data),
|
||||
"discord": generate_discord_post(template_name, data),
|
||||
}
|
||||
|
|
@ -1,161 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""#17 - MBAL 10M-Label Market Maker. Public searchable database of the MBAL dataset
|
||||
(10M labeled addresses). Free search, paid API. Kaggle: cryptorugmuncher."""
|
||||
|
||||
import contextlib
|
||||
import os
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Query
|
||||
from pydantic import BaseModel
|
||||
|
||||
router = APIRouter(prefix="/api/v1/mbal", tags=["mbal-market-maker"])
|
||||
|
||||
MBAL_DB = os.environ.get("MBAL_DB", str(Path.home() / "rmi/backend/data/mbal.db"))
|
||||
|
||||
|
||||
def _get_db() -> sqlite3.Connection | None:
|
||||
"""Open MBAL database connection."""
|
||||
db_path = Path(MBAL_DB)
|
||||
if not db_path.exists():
|
||||
return None
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
|
||||
@router.get("/search/{address}")
|
||||
async def search_mbal(address: str):
|
||||
"""Search MBAL database for an address. Returns label if found."""
|
||||
conn = _get_db()
|
||||
if not conn:
|
||||
return {
|
||||
"address": address,
|
||||
"found": False,
|
||||
"database": "unavailable",
|
||||
"note": "MBAL database not found. Visit: https://www.kaggle.com/datasets/cryptorugmuncher/mbal-10m-labeled-addresses",
|
||||
}
|
||||
|
||||
try:
|
||||
cursor = conn.execute(
|
||||
"SELECT address, label, category, confidence, source FROM addresses WHERE address = ? LIMIT 1",
|
||||
(address.lower(),),
|
||||
)
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
return {
|
||||
"address": address,
|
||||
"found": True,
|
||||
"label": row["label"],
|
||||
"category": row["category"],
|
||||
"confidence": row["confidence"],
|
||||
"source": row["source"],
|
||||
"dataset": "MBAL (Multi-chain Blockchain Address Labels)",
|
||||
}
|
||||
return {"address": address, "found": False, "database": "available"}
|
||||
except sqlite3.Error:
|
||||
return {"address": address, "found": False, "database": "error"}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
@router.get("/stats")
|
||||
async def mbal_stats():
|
||||
"""MBAL dataset statistics."""
|
||||
conn = _get_db()
|
||||
stats = {
|
||||
"dataset": "MBAL - Multi-chain Blockchain Address Labels",
|
||||
"total_addresses": 0,
|
||||
"categories": [],
|
||||
"chains_covered": [],
|
||||
"kaggle_url": "https://www.kaggle.com/datasets/cryptorugmuncher/mbal-10m-labeled-addresses",
|
||||
"database_available": conn is not None,
|
||||
}
|
||||
if conn:
|
||||
try:
|
||||
stats["total_addresses"] = conn.execute("SELECT COUNT(*) FROM addresses").fetchone()[0]
|
||||
stats["categories"] = [
|
||||
r[0]
|
||||
for r in conn.execute(
|
||||
"SELECT DISTINCT category FROM addresses WHERE category IS NOT NULL LIMIT 20"
|
||||
).fetchall()
|
||||
]
|
||||
with contextlib.suppress(sqlite3.Error):
|
||||
stats["chains_covered"] = [
|
||||
r[0]
|
||||
for r in conn.execute(
|
||||
"SELECT DISTINCT chain FROM addresses WHERE chain IS NOT NULL LIMIT 20"
|
||||
).fetchall()
|
||||
]
|
||||
except sqlite3.Error:
|
||||
pass
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
class BulkSearchRequest(BaseModel):
|
||||
addresses: list[str]
|
||||
max_results: int = 100
|
||||
|
||||
|
||||
@router.post("/bulk-search")
|
||||
async def bulk_search_mbal(request: BulkSearchRequest):
|
||||
"""Search multiple addresses in MBAL. Paid tier (x402)."""
|
||||
conn = _get_db()
|
||||
if not conn:
|
||||
raise HTTPException(503, "MBAL database unavailable")
|
||||
|
||||
results: list[dict] = []
|
||||
try:
|
||||
for addr in request.addresses[: request.max_results]:
|
||||
cursor = conn.execute(
|
||||
"SELECT address, label, category, confidence FROM addresses WHERE address = ? LIMIT 1", (addr.lower(),)
|
||||
)
|
||||
row = cursor.fetchone()
|
||||
if row:
|
||||
results.append(
|
||||
{
|
||||
"address": addr,
|
||||
"found": True,
|
||||
"label": row["label"],
|
||||
"category": row["category"],
|
||||
"confidence": row["confidence"],
|
||||
}
|
||||
)
|
||||
else:
|
||||
results.append({"address": addr, "found": False})
|
||||
except sqlite3.Error:
|
||||
raise HTTPException(500, "Database query error") from None
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
return {
|
||||
"searched": len(request.addresses[: request.max_results]),
|
||||
"found": sum(1 for r in results if r.get("found")),
|
||||
"results": results,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/category/{category}")
|
||||
async def search_by_category(category: str, limit: int = Query(20, le=100)):
|
||||
"""Search MBAL addresses by category (e.g., 'exchange', 'scam', 'mixer')."""
|
||||
conn = _get_db()
|
||||
if not conn:
|
||||
raise HTTPException(503, "MBAL database unavailable")
|
||||
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"SELECT address, label, category, confidence FROM addresses WHERE category = ? LIMIT ?", (category, limit)
|
||||
).fetchall()
|
||||
results = [
|
||||
{"address": r["address"], "label": r["label"], "category": r["category"], "confidence": r["confidence"]}
|
||||
for r in rows
|
||||
]
|
||||
return {"category": category, "count": len(results), "addresses": results}
|
||||
except sqlite3.Error:
|
||||
raise HTTPException(500, "Database query error") from None
|
||||
finally:
|
||||
conn.close()
|
||||
|
|
@ -1,63 +0,0 @@
|
|||
"""Local MCP Proxy Router - free local MCP server proxy.
|
||||
NOTE: Auth via X-Admin-Key is disabled due to Docker pycache issue.
|
||||
Secure via nginx IP whitelist or firewall in production.
|
||||
Frontend already requires adminKey in React Query hooks (enabled: !!adminKey)."""
|
||||
|
||||
from fastapi import APIRouter, Query
|
||||
from pydantic import BaseModel
|
||||
|
||||
from .mcp_proxy import SERVER_REGISTRY, call_local_mcp, get_local_mcp_tools
|
||||
|
||||
router = APIRouter(prefix="/api/v1/mcp-local", tags=["mcp-local"])
|
||||
|
||||
|
||||
class MCPCallRequest(BaseModel):
|
||||
server: str
|
||||
tool: str
|
||||
params: dict = {}
|
||||
|
||||
|
||||
@router.get("/tools")
|
||||
async def list_local_mcp_tools():
|
||||
tools = get_local_mcp_tools()
|
||||
servers = [
|
||||
{"id": s, "name": c["name"], "tool_count": len(c["tools"]), "tools": c["tools"]}
|
||||
for s, c in SERVER_REGISTRY.items()
|
||||
]
|
||||
return {
|
||||
"total_servers": len(servers),
|
||||
"total_tools": len(tools),
|
||||
"servers": servers,
|
||||
"tools": tools,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/servers")
|
||||
async def list_local_mcp_servers():
|
||||
return {
|
||||
"servers": [{"id": s, "name": c["name"], "tool_count": len(c["tools"])} for s, c in SERVER_REGISTRY.items()]
|
||||
}
|
||||
|
||||
|
||||
@router.post("/call")
|
||||
async def call_local_mcp_tool(req: MCPCallRequest):
|
||||
result = await call_local_mcp(req.server, req.tool, req.params)
|
||||
return {"server": req.server, "tool": req.tool, "params": req.params, **result}
|
||||
|
||||
|
||||
@router.get("/call/{server}/{tool}")
|
||||
async def call_local_mcp_tool_get(
|
||||
server: str,
|
||||
tool: str,
|
||||
symbol: str = Query(None),
|
||||
address: str = Query(None),
|
||||
chain: str = Query("ethereum"),
|
||||
):
|
||||
params = {}
|
||||
if symbol:
|
||||
params["symbol"] = symbol
|
||||
if address:
|
||||
params["address"] = address
|
||||
params["chain"] = chain
|
||||
result = await call_local_mcp(server, tool, params)
|
||||
return {"server": server, "tool": tool, "params": params, **result}
|
||||
|
|
@ -1,325 +0,0 @@
|
|||
"""
|
||||
Local MCP Server Proxy - makes free local MCP servers accessible via REST API.
|
||||
Reads tool definitions from MCP server manifests and proxies tool calls.
|
||||
|
||||
Local servers run via stdio - we spawn them on demand with a process cache.
|
||||
All data is FREE to call (no API keys, just local compute/RPC queries).
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
SERVER_REGISTRY = {
|
||||
"feargreed": {
|
||||
"name": "Crypto Fear & Greed Index",
|
||||
"path": "/root/.hermes/mcp-servers/crypto-feargreed-mcp",
|
||||
"command": ["python3", "main.py"],
|
||||
"tools": [
|
||||
{
|
||||
"id": "feargreed_get_index",
|
||||
"name": "Get Fear & Greed Index",
|
||||
"description": "Current crypto Fear & Greed Index value and classification",
|
||||
},
|
||||
{
|
||||
"id": "feargreed_get_historical",
|
||||
"name": "Get Historical Fear & Greed",
|
||||
"description": "Historical Fear & Greed values for a date range",
|
||||
},
|
||||
],
|
||||
},
|
||||
"crypto-news": {
|
||||
"name": "Crypto News API",
|
||||
"path": "/root/.hermes/mcp-servers/crypto-news-api",
|
||||
"command": ["node", "dist/index.js"],
|
||||
"tools": [
|
||||
{
|
||||
"id": "news_get_latest",
|
||||
"name": "Get Latest News",
|
||||
"description": "Real-time crypto news from 12+ sources with AI sentiment",
|
||||
},
|
||||
{
|
||||
"id": "news_search",
|
||||
"name": "Search News",
|
||||
"description": "Search crypto news by keyword, source, or date range",
|
||||
},
|
||||
{
|
||||
"id": "news_sentiment",
|
||||
"name": "News Sentiment",
|
||||
"description": "Aggregated market sentiment from news sources",
|
||||
},
|
||||
],
|
||||
},
|
||||
"indicators": {
|
||||
"name": "Crypto Indicators",
|
||||
"path": "/root/.hermes/mcp-servers/crypto-indicators-mcp",
|
||||
"command": ["node", "dist/index.js"],
|
||||
"tools": [
|
||||
{
|
||||
"id": "indicators_rsi",
|
||||
"name": "RSI Indicator",
|
||||
"description": "Relative Strength Index for any token",
|
||||
},
|
||||
{
|
||||
"id": "indicators_macd",
|
||||
"name": "MACD Indicator",
|
||||
"description": "Moving Average Convergence Divergence",
|
||||
},
|
||||
{
|
||||
"id": "indicators_bollinger",
|
||||
"name": "Bollinger Bands",
|
||||
"description": "Bollinger Bands volatility indicator",
|
||||
},
|
||||
{"id": "indicators_ema", "name": "EMA", "description": "Exponential Moving Average"},
|
||||
{"id": "indicators_sma", "name": "SMA", "description": "Simple Moving Average"},
|
||||
],
|
||||
},
|
||||
"evmscope": {
|
||||
"name": "EVMScope Intelligence",
|
||||
"path": "/root/.hermes/mcp-servers/evmscope",
|
||||
"command": ["node", "dist/index.js"],
|
||||
"tools": [
|
||||
{
|
||||
"id": "evmscope_token_price",
|
||||
"name": "Token Price",
|
||||
"description": "Current token price across chains",
|
||||
},
|
||||
{
|
||||
"id": "evmscope_token_info",
|
||||
"name": "Token Info",
|
||||
"description": "Token metadata - name, symbol, decimals, supply",
|
||||
},
|
||||
{
|
||||
"id": "evmscope_token_holders",
|
||||
"name": "Token Holders",
|
||||
"description": "Token holder distribution and top holders",
|
||||
},
|
||||
{
|
||||
"id": "evmscope_honeypot",
|
||||
"name": "Honeypot Check",
|
||||
"description": "Check if a token is a honeypot scam",
|
||||
},
|
||||
{
|
||||
"id": "evmscope_whale_movements",
|
||||
"name": "Whale Movements",
|
||||
"description": "Recent large transactions for a token",
|
||||
},
|
||||
{
|
||||
"id": "evmscope_gas_price",
|
||||
"name": "Gas Price",
|
||||
"description": "Current gas prices across chains",
|
||||
},
|
||||
{
|
||||
"id": "evmscope_portfolio",
|
||||
"name": "Portfolio",
|
||||
"description": "Wallet portfolio with balances across chains",
|
||||
},
|
||||
{
|
||||
"id": "evmscope_bridge_routes",
|
||||
"name": "Bridge Routes",
|
||||
"description": "Optimal bridge routes between chains",
|
||||
},
|
||||
{
|
||||
"id": "evmscope_yield_rates",
|
||||
"name": "Yield Rates",
|
||||
"description": "Current DeFi yield rates across protocols",
|
||||
},
|
||||
{
|
||||
"id": "evmscope_swap_quote",
|
||||
"name": "Swap Quote",
|
||||
"description": "Best swap quote across DEX aggregators",
|
||||
},
|
||||
{
|
||||
"id": "evmscope_nft_info",
|
||||
"name": "NFT Info",
|
||||
"description": "NFT metadata and collection info",
|
||||
},
|
||||
{
|
||||
"id": "evmscope_contract_abi",
|
||||
"name": "Contract ABI",
|
||||
"description": "Fetch verified contract ABI",
|
||||
},
|
||||
{
|
||||
"id": "evmscope_decode_tx",
|
||||
"name": "Decode Transaction",
|
||||
"description": "Decode raw transaction data",
|
||||
},
|
||||
],
|
||||
},
|
||||
"polymarket": {
|
||||
"name": "Polymarket Data",
|
||||
"path": "/root/.hermes/mcp-servers/graph-polymarket-mcp",
|
||||
"command": ["node", "dist/index.js"],
|
||||
"tools": [
|
||||
{
|
||||
"id": "polymarket_active_markets",
|
||||
"name": "Active Markets",
|
||||
"description": "Currently active prediction markets",
|
||||
},
|
||||
{
|
||||
"id": "polymarket_market_detail",
|
||||
"name": "Market Detail",
|
||||
"description": "Detailed market data - prices, volume, liquidity",
|
||||
},
|
||||
{
|
||||
"id": "polymarket_orderbook",
|
||||
"name": "Orderbook",
|
||||
"description": "Live orderbook for a market",
|
||||
},
|
||||
{
|
||||
"id": "polymarket_open_interest",
|
||||
"name": "Open Interest",
|
||||
"description": "Open interest across markets",
|
||||
},
|
||||
{
|
||||
"id": "polymarket_traders",
|
||||
"name": "Top Traders",
|
||||
"description": "Top traders by volume/profit",
|
||||
},
|
||||
],
|
||||
},
|
||||
"prediction-markets": {
|
||||
"name": "Prediction Markets",
|
||||
"path": "/root/.hermes/mcp-servers/prediction-market-mcp",
|
||||
"command": ["node", "dist/index.js"],
|
||||
"tools": [
|
||||
{
|
||||
"id": "predmkt_polymarket",
|
||||
"name": "Polymarket Data",
|
||||
"description": "Polymarket markets, prices, and events",
|
||||
},
|
||||
{
|
||||
"id": "predmkt_kalshi",
|
||||
"name": "Kalshi Data",
|
||||
"description": "Kalshi prediction market data",
|
||||
},
|
||||
{
|
||||
"id": "predmkt_predictit",
|
||||
"name": "PredictIt Data",
|
||||
"description": "PredictIt market data",
|
||||
},
|
||||
],
|
||||
},
|
||||
"web3-research": {
|
||||
"name": "Web3 Research",
|
||||
"path": "/root/.hermes/mcp-servers/web3-research-mcp",
|
||||
"command": ["node", "dist/index.js"],
|
||||
"tools": [
|
||||
{
|
||||
"id": "research_coingecko",
|
||||
"name": "CoinGecko Research",
|
||||
"description": "Deep CoinGecko data - prices, markets, exchanges",
|
||||
},
|
||||
{
|
||||
"id": "research_defillama",
|
||||
"name": "DeFiLlama Research",
|
||||
"description": "Protocol TVL, yields, chain metrics from DeFiLlama",
|
||||
},
|
||||
{
|
||||
"id": "research_combined",
|
||||
"name": "Combined Research",
|
||||
"description": "Multi-source research combining CoinGecko + DeFiLlama",
|
||||
},
|
||||
],
|
||||
},
|
||||
"jupiter": {
|
||||
"name": "Jupiter DEX",
|
||||
"path": "/root/.hermes/mcp-servers/jupiter-mcp",
|
||||
"command": ["node", "dist/index.js"],
|
||||
"tools": [
|
||||
{
|
||||
"id": "jupiter_quote",
|
||||
"name": "Swap Quote",
|
||||
"description": "Best swap route and price quote on Solana",
|
||||
},
|
||||
{
|
||||
"id": "jupiter_price",
|
||||
"name": "Token Price",
|
||||
"description": "Current token price via Jupiter aggregator",
|
||||
},
|
||||
{
|
||||
"id": "jupiter_token_list",
|
||||
"name": "Token List",
|
||||
"description": "All verified tokens on Jupiter",
|
||||
},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
# Process cache - keep server processes alive for 5 min
|
||||
_proc_cache: dict[str, tuple[subprocess.Popen, float]] = {}
|
||||
_CACHE_TTL = 300 # 5 minutes
|
||||
|
||||
|
||||
async def _get_or_spawn(server_id: str):
|
||||
"""Get or spawn an MCP server process."""
|
||||
if server_id not in SERVER_REGISTRY:
|
||||
return None
|
||||
|
||||
now = time.time()
|
||||
if server_id in _proc_cache:
|
||||
proc, created = _proc_cache[server_id]
|
||||
if now - created < _CACHE_TTL and proc.poll() is None:
|
||||
return proc
|
||||
|
||||
cfg = SERVER_REGISTRY[server_id]
|
||||
try:
|
||||
proc = subprocess.Popen(
|
||||
cfg["command"],
|
||||
cwd=cfg["path"],
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
)
|
||||
_proc_cache[server_id] = (proc, now)
|
||||
# Give it a moment to initialize
|
||||
await asyncio.sleep(1)
|
||||
return proc
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
async def call_local_mcp(server_id: str, tool_name: str, params: dict | None = None) -> dict:
|
||||
"""Call a tool on a local MCP server."""
|
||||
proc = await _get_or_spawn(server_id)
|
||||
if not proc:
|
||||
return {"error": f"Server '{server_id}' not available"}
|
||||
|
||||
# MCP JSON-RPC call
|
||||
request = {
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1,
|
||||
"method": "tools/call",
|
||||
"params": {
|
||||
"name": tool_name,
|
||||
"arguments": params or {},
|
||||
},
|
||||
}
|
||||
|
||||
try:
|
||||
proc.stdin.write((json.dumps(request) + "\n").encode())
|
||||
proc.stdin.flush()
|
||||
# Read response line
|
||||
line = proc.stdout.readline()
|
||||
result = json.loads(line.decode())
|
||||
if "error" in result:
|
||||
return {"error": result["error"]}
|
||||
return {"result": result.get("result", result)}
|
||||
except Exception as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
def get_local_mcp_tools() -> list[dict]:
|
||||
"""Get all available local MCP tools."""
|
||||
tools = []
|
||||
for server_id, cfg in SERVER_REGISTRY.items():
|
||||
for tool in cfg["tools"]:
|
||||
tools.append(
|
||||
{
|
||||
"server": server_id,
|
||||
"server_name": cfg["name"],
|
||||
**tool,
|
||||
}
|
||||
)
|
||||
return tools
|
||||
|
|
@ -1,397 +0,0 @@
|
|||
"""
|
||||
MCP Router - Receives tool execution requests from Cloudflare X402 Workers.
|
||||
Exposes /mcp/tools (catalog) and /mcp/execute (tool execution).
|
||||
This is what the worker calls when x402 payment is verified.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/mcp", tags=["mcp-router"])
|
||||
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
# TOOL CATALOG - What the worker fetches on startup
|
||||
# ═══════════════════════════════════════════════════════════
|
||||
TOOLS = {
|
||||
# Security
|
||||
"honeypot_check": {
|
||||
"name": "Honeypot Detector",
|
||||
"endpoint": "/api/v1/contract/audit",
|
||||
"method": "POST",
|
||||
"category": "security",
|
||||
"price": "$0.05",
|
||||
},
|
||||
"rug_pull_predictor": {
|
||||
"name": "Rug Pull Predictor",
|
||||
"endpoint": "/api/v1/contract/audit",
|
||||
"method": "POST",
|
||||
"category": "security",
|
||||
"price": "$0.10",
|
||||
},
|
||||
"rugshield": {
|
||||
"name": "Rug Shield",
|
||||
"endpoint": "/api/v1/security/scan",
|
||||
"method": "POST",
|
||||
"category": "security",
|
||||
"price": "$0.02",
|
||||
},
|
||||
"audit": {
|
||||
"name": "Smart Contract Audit",
|
||||
"endpoint": "/api/v1/contract/audit",
|
||||
"method": "POST",
|
||||
"category": "security",
|
||||
"price": "$0.05",
|
||||
},
|
||||
"mev_protection": {
|
||||
"name": "MEV Protection Check",
|
||||
"endpoint": "/api/v1/tools/cast/contract-info",
|
||||
"method": "GET",
|
||||
"category": "security",
|
||||
"price": "$0.08",
|
||||
},
|
||||
"bridge_security": {
|
||||
"name": "Bridge Security Monitor",
|
||||
"endpoint": "/api/v1/defillama/chains",
|
||||
"method": "GET",
|
||||
"category": "security",
|
||||
"price": "$0.08",
|
||||
},
|
||||
"profile_flip": {
|
||||
"name": "Profile Flip Detector",
|
||||
"endpoint": "/api/v1/token/scan",
|
||||
"method": "POST",
|
||||
"category": "security",
|
||||
"price": "$0.03",
|
||||
},
|
||||
"fresh_pair": {
|
||||
"name": "Fresh Pair Scanner",
|
||||
"endpoint": "/api/v1/tokens/new",
|
||||
"method": "GET",
|
||||
"category": "security",
|
||||
"price": "$0.03",
|
||||
},
|
||||
"clone_detect": {
|
||||
"name": "Clone Detector",
|
||||
"endpoint": "/api/v1/token/scan",
|
||||
"method": "POST",
|
||||
"category": "security",
|
||||
"price": "$0.02",
|
||||
},
|
||||
"urlcheck": {
|
||||
"name": "URL Safety Check",
|
||||
"endpoint": "/api/v1/security/scan",
|
||||
"method": "POST",
|
||||
"category": "security",
|
||||
"price": "$0.01",
|
||||
},
|
||||
# Intelligence
|
||||
"whale": {
|
||||
"name": "Whale Wallet Decoder",
|
||||
"endpoint": "/api/v1/helius/whale-profile",
|
||||
"method": "POST",
|
||||
"category": "intelligence",
|
||||
"price": "$0.15",
|
||||
},
|
||||
"whale_scan": {
|
||||
"name": "Whale Scanner",
|
||||
"endpoint": "/api/v1/helius/whale-scan",
|
||||
"method": "POST",
|
||||
"category": "intelligence",
|
||||
"price": "$0.03",
|
||||
},
|
||||
"whale_profile": {
|
||||
"name": "Whale Profile",
|
||||
"endpoint": "/api/v1/helius/whale-profile",
|
||||
"method": "POST",
|
||||
"category": "intelligence",
|
||||
"price": "$0.05",
|
||||
},
|
||||
"smartmoney": {
|
||||
"name": "Smart Money Tracker",
|
||||
"endpoint": "/api/v1/smart-money",
|
||||
"method": "GET",
|
||||
"category": "intelligence",
|
||||
"price": "$0.05",
|
||||
},
|
||||
"cluster": {
|
||||
"name": "Wallet Cluster Analysis",
|
||||
"endpoint": "/api/v1/entity/clusters",
|
||||
"method": "GET",
|
||||
"category": "intelligence",
|
||||
"price": "$0.05",
|
||||
},
|
||||
"insider": {
|
||||
"name": "Insider Trading Detection",
|
||||
"endpoint": "/api/v1/threat/reputation/0x0",
|
||||
"method": "GET",
|
||||
"category": "intelligence",
|
||||
"price": "$0.10",
|
||||
},
|
||||
"sniper_detect": {
|
||||
"name": "Sniper Detector",
|
||||
"endpoint": "/api/v1/helius/sniper-detect",
|
||||
"method": "POST",
|
||||
"category": "intelligence",
|
||||
"price": "$0.08",
|
||||
},
|
||||
"syndicate_scan": {
|
||||
"name": "Syndicate Scanner",
|
||||
"endpoint": "/api/v1/helius/syndicate/scan",
|
||||
"method": "GET",
|
||||
"category": "intelligence",
|
||||
"price": "$0.08",
|
||||
},
|
||||
"copy_trade_finder": {
|
||||
"name": "Copy Trade Finder",
|
||||
"endpoint": "/api/v1/whales/top",
|
||||
"method": "GET",
|
||||
"category": "intelligence",
|
||||
"price": "$0.10",
|
||||
},
|
||||
"liquidity_flow": {
|
||||
"name": "Liquidity Flow Tracker",
|
||||
"endpoint": "/api/v1/exchange/whales",
|
||||
"method": "GET",
|
||||
"category": "intelligence",
|
||||
"price": "$0.08",
|
||||
},
|
||||
# Market
|
||||
"pulse": {
|
||||
"name": "Market Pulse",
|
||||
"endpoint": "/api/v1/intelligence/dashboard",
|
||||
"method": "GET",
|
||||
"category": "market",
|
||||
"price": "$0.01",
|
||||
},
|
||||
"market_overview": {
|
||||
"name": "Market Overview",
|
||||
"endpoint": "/api/v1/markets/ccxt",
|
||||
"method": "GET",
|
||||
"category": "market",
|
||||
"price": "$0.05",
|
||||
},
|
||||
"chain_health": {
|
||||
"name": "Chain Health",
|
||||
"endpoint": "/api/v1/defillama/chains",
|
||||
"method": "GET",
|
||||
"category": "market",
|
||||
"price": "$0.05",
|
||||
},
|
||||
"defi_yield_scanner": {
|
||||
"name": "DeFi Yield Scanner",
|
||||
"endpoint": "/api/v1/defillama/protocols",
|
||||
"method": "GET",
|
||||
"category": "market",
|
||||
"price": "$0.08",
|
||||
},
|
||||
"gas_forecast": {
|
||||
"name": "Gas Forecaster",
|
||||
"endpoint": "/api/v1/mempool/status",
|
||||
"method": "GET",
|
||||
"category": "market",
|
||||
"price": "$0.05",
|
||||
},
|
||||
# Analysis
|
||||
"wallet": {
|
||||
"name": "Wallet Analysis",
|
||||
"endpoint": "/api/v1/wallet/multichain/{address}",
|
||||
"method": "GET",
|
||||
"category": "analysis",
|
||||
"price": "$0.05",
|
||||
},
|
||||
"forensics": {
|
||||
"name": "Token Forensics",
|
||||
"endpoint": "/api/v1/wallet/{address}/analysis",
|
||||
"method": "GET",
|
||||
"category": "analysis",
|
||||
"price": "$0.10",
|
||||
},
|
||||
"token_deep_dive": {
|
||||
"name": "Token Deep Dive",
|
||||
"endpoint": "/api/v1/birdeye/token/{address}",
|
||||
"method": "GET",
|
||||
"category": "analysis",
|
||||
"price": "$0.10",
|
||||
},
|
||||
"portfolio_tracker": {
|
||||
"name": "Portfolio Tracker",
|
||||
"endpoint": "/api/v1/wallet/pnl/{address}",
|
||||
"method": "GET",
|
||||
"category": "analysis",
|
||||
"price": "$0.10",
|
||||
},
|
||||
"token_comparison": {
|
||||
"name": "Token Comparison",
|
||||
"endpoint": "/api/v1/token/scan",
|
||||
"method": "POST",
|
||||
"category": "analysis",
|
||||
"price": "$0.08",
|
||||
},
|
||||
"nft_wash_detector": {
|
||||
"name": "NFT Wash Detector",
|
||||
"endpoint": "/api/v1/threat/reputation/0x0",
|
||||
"method": "GET",
|
||||
"category": "analysis",
|
||||
"price": "$0.10",
|
||||
},
|
||||
# Launch
|
||||
"launch": {
|
||||
"name": "Token Launch Analysis",
|
||||
"endpoint": "/api/v1/tokens/new",
|
||||
"method": "GET",
|
||||
"category": "launchpad",
|
||||
"price": "$0.03",
|
||||
},
|
||||
"launch_intel": {
|
||||
"name": "Launch Intelligence",
|
||||
"endpoint": "/api/v1/tokens/trending",
|
||||
"method": "GET",
|
||||
"category": "launchpad",
|
||||
"price": "$0.05",
|
||||
},
|
||||
"sniper_alert": {
|
||||
"name": "Sniper Alert",
|
||||
"endpoint": "/api/v1/tokens/new",
|
||||
"method": "GET",
|
||||
"category": "launchpad",
|
||||
"price": "$0.05",
|
||||
},
|
||||
"airdrop_finder": {
|
||||
"name": "Airdrop Finder",
|
||||
"endpoint": "/api/v1/wallet/pnl/{address}",
|
||||
"method": "GET",
|
||||
"category": "intelligence",
|
||||
"price": "$0.05",
|
||||
},
|
||||
# Social
|
||||
"sentiment": {
|
||||
"name": "Sentiment Analysis",
|
||||
"endpoint": "/api/v1/sentiment/market",
|
||||
"method": "GET",
|
||||
"category": "social",
|
||||
"price": "$0.03",
|
||||
},
|
||||
"social_signal": {
|
||||
"name": "Social Signal Analyzer",
|
||||
"endpoint": "/api/v1/sentiment/token/{address}",
|
||||
"method": "GET",
|
||||
"category": "social",
|
||||
"price": "$0.10",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _build_tools_catalog() -> dict:
|
||||
"""Merge hardcoded TOOLS with dynamic TOOL_PRICES, preferring dynamic data."""
|
||||
merged = dict(TOOLS) # Start with hardcoded
|
||||
try:
|
||||
from app.routers.x402_enforcement import TOOL_PRICES
|
||||
|
||||
# Tool ID → internal endpoint mapping
|
||||
_ENDPOINT_MAP = {
|
||||
"forensic_valuation": "/api/v1/x402-tools/forensic_valuation",
|
||||
"osint_identity_hunt": "/api/v1/x402-tools/osint_identity_hunt",
|
||||
"investigation_report": "/api/v1/x402-tools/investigation_report",
|
||||
"forensic_pack": "/api/v1/x402-tools/forensic_pack",
|
||||
"catalog": "/api/v1/x402-tools/catalog",
|
||||
"smart_money_alpha": "/api/v1/x402-tools/smart_money_alpha",
|
||||
"meme_vibe_score": "/api/v1/x402-tools/meme_vibe_score",
|
||||
"mcp-proxy": "/api/v1/x402-tools/mcp-proxy",
|
||||
"human-execute": "/api/v1/x402-tools/human-execute",
|
||||
}
|
||||
for tool_id, pricing in TOOL_PRICES.items():
|
||||
if tool_id not in merged:
|
||||
endpoint = _ENDPOINT_MAP.get(tool_id, f"/api/v1/x402-tools/{tool_id}")
|
||||
merged[tool_id] = {
|
||||
"name": pricing.get("description", tool_id.replace("_", " ").title()),
|
||||
"endpoint": endpoint,
|
||||
"method": "POST",
|
||||
"category": pricing.get("category", "analysis"),
|
||||
"price": f"${pricing.get('price_usd', 0.01):.2f}",
|
||||
}
|
||||
else:
|
||||
# Update existing entries with latest pricing/category from TOOL_PRICES
|
||||
if tool_id in TOOL_PRICES:
|
||||
merged[tool_id]["category"] = TOOL_PRICES[tool_id].get(
|
||||
"category", merged[tool_id].get("category", "analysis")
|
||||
)
|
||||
merged[tool_id]["price"] = f"${TOOL_PRICES[tool_id].get('price_usd', 0.01):.2f}"
|
||||
except Exception as e:
|
||||
logger.warning(f"mcp_router: could not merge TOOL_PRICES: {e}")
|
||||
return merged
|
||||
|
||||
|
||||
@router.get("/tools")
|
||||
async def mcp_tools():
|
||||
"""Return tool catalog for Cloudflare Worker to cache."""
|
||||
catalog = _build_tools_catalog()
|
||||
return {
|
||||
"tools": catalog,
|
||||
"total": len(catalog),
|
||||
"categories": list({t["category"] for t in catalog.values()}),
|
||||
"updated_at": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/execute/{tool_id}")
|
||||
async def mcp_execute(tool_id: str, request: Request):
|
||||
"""Execute a tool - called by Cloudflare Worker after x402 payment verified."""
|
||||
tool = TOOLS.get(tool_id)
|
||||
if not tool:
|
||||
raise HTTPException(status_code=404, detail=f"Tool {tool_id} not found")
|
||||
|
||||
try:
|
||||
body = await request.json() if request.headers.get("content-type") == "application/json" else {}
|
||||
except Exception:
|
||||
body = {}
|
||||
|
||||
# Forward to the actual backend endpoint
|
||||
endpoint = tool["endpoint"]
|
||||
method = tool["method"]
|
||||
|
||||
# Replace path parameters from body
|
||||
for key in ["address", "token", "chain"]:
|
||||
if f"{{{key}}}" in endpoint and key in body:
|
||||
endpoint = endpoint.replace(f"{{{key}}}", body[key])
|
||||
|
||||
# Internal calls bypass auth - add internal API key header
|
||||
internal_headers = {}
|
||||
auth_token = os.getenv("RMI_AUTH_TOKEN", "")
|
||||
if auth_token:
|
||||
internal_headers["X-API-Key"] = auth_token
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=30) as c:
|
||||
if method == "GET":
|
||||
# Build query params from body
|
||||
params = {k: v for k, v in body.items() if k not in ["address", "token"] and f"{{{k}}}" not in endpoint}
|
||||
r = await c.get(f"http://127.0.0.1:8000{endpoint}", params=params, headers=internal_headers)
|
||||
else:
|
||||
r = await c.post(f"http://127.0.0.1:8000{endpoint}", json=body, headers=internal_headers)
|
||||
|
||||
if r.status_code == 200:
|
||||
return {
|
||||
"tool": tool_id,
|
||||
"status": "success",
|
||||
"data": r.json(),
|
||||
"executed_at": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
return {
|
||||
"tool": tool_id,
|
||||
"status": "error",
|
||||
"error": f"Backend returned {r.status_code}",
|
||||
"executed_at": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
except Exception as e:
|
||||
return {"tool": tool_id, "status": "error", "error": str(e)}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def mcp_health():
|
||||
catalog = _build_tools_catalog()
|
||||
return {"status": "healthy", "tools": len(catalog), "timestamp": datetime.now(UTC).isoformat()}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,116 +0,0 @@
|
|||
"""Content Moderation Pipeline - AI-powered spam/scam/NSFW detection for user content."""
|
||||
|
||||
import os
|
||||
import re
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel
|
||||
|
||||
router = APIRouter(prefix="/api/v1/moderation", tags=["moderation"])
|
||||
|
||||
OLLAMA = os.getenv("OLLAMA_HOST", "http://localhost:11434")
|
||||
|
||||
BLOCKED_PATTERNS = [
|
||||
(r"(?i)(buy|sell|trade).*\b(signal|call)\b", "trading_signal"),
|
||||
(r"(?i)(\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b)", "ip_address"),
|
||||
(r"(?i)\b(0x[a-fA-F0-9]{40})\b.*\b(private.?key|seed.?phrase|mnemonic|password)\b", "wallet_phishing"),
|
||||
(r"(?i)(airdrop|giveaway|free.*token).*\b(claim|connect|verify)\b", "scam_airdrop"),
|
||||
(r"(https?://(?!rugmunch\.io|polymarket\.com|dexscreener\.com)[^\s]+)", "external_link"),
|
||||
]
|
||||
|
||||
|
||||
class ModerationRequest(BaseModel):
|
||||
text: str
|
||||
user_id: str = "anonymous"
|
||||
context: str = "comment" # comment, post, review, chat
|
||||
|
||||
|
||||
class ModerationResult(BaseModel):
|
||||
approved: bool
|
||||
risk_score: int # 0-100
|
||||
flags: list[str]
|
||||
reason: str = ""
|
||||
|
||||
|
||||
async def _ai_classify(text: str) -> dict:
|
||||
"""Use Ollama to classify content."""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=15) as c:
|
||||
r = await c.post(
|
||||
f"{OLLAMA}/api/generate",
|
||||
json={
|
||||
"model": "qwen2.5-coder:7b",
|
||||
"prompt": f"Classify this crypto-related content as SAFE, SPAM, SCAM, or NSFW. Answer with one word only.\n\nContent: {text[:500]}\n\nClassification:",
|
||||
"stream": False,
|
||||
"options": {"num_predict": 5, "temperature": 0.1},
|
||||
},
|
||||
)
|
||||
if r.status_code == 200:
|
||||
response = r.json().get("response", "").strip().upper()
|
||||
return {"ai_verdict": response, "ai_used": True}
|
||||
except Exception:
|
||||
pass
|
||||
return {"ai_verdict": "UNKNOWN", "ai_used": False}
|
||||
|
||||
|
||||
@router.post("/check")
|
||||
async def moderate_content(req: ModerationRequest):
|
||||
"""Check content for spam, scams, NSFW, and policy violations."""
|
||||
flags = []
|
||||
risk = 0
|
||||
|
||||
# Pattern-based detection
|
||||
for pattern, flag_type in BLOCKED_PATTERNS:
|
||||
if re.search(pattern, req.text):
|
||||
flags.append(flag_type)
|
||||
risk += 25
|
||||
|
||||
# Length checks
|
||||
if len(req.text) < 5:
|
||||
flags.append("too_short")
|
||||
risk += 10
|
||||
if len(req.text) > 10000:
|
||||
flags.append("too_long")
|
||||
risk += 5
|
||||
|
||||
# AI classification
|
||||
ai = await _ai_classify(req.text)
|
||||
ai_verdict = ai["ai_verdict"]
|
||||
if "SCAM" in ai_verdict:
|
||||
flags.append("ai_scam")
|
||||
risk += 40
|
||||
elif "SPAM" in ai_verdict:
|
||||
flags.append("ai_spam")
|
||||
risk += 25
|
||||
elif "NSFW" in ai_verdict:
|
||||
flags.append("ai_nsfw")
|
||||
risk += 50
|
||||
|
||||
approved = risk < 40
|
||||
reason = "Content approved" if approved else f"Flagged: {', '.join(flags)}"
|
||||
|
||||
return {
|
||||
"approved": approved,
|
||||
"risk_score": min(100, risk),
|
||||
"flags": flags,
|
||||
"reason": reason,
|
||||
"ai_classification": ai_verdict,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/stats")
|
||||
async def moderation_stats():
|
||||
return {
|
||||
"patterns_checked": len(BLOCKED_PATTERNS),
|
||||
"ai_model": "qwen2.5-coder:7b",
|
||||
"categories": [
|
||||
"trading_signal",
|
||||
"wallet_phishing",
|
||||
"scam_airdrop",
|
||||
"external_link",
|
||||
"ai_scam",
|
||||
"ai_spam",
|
||||
"ai_nsfw",
|
||||
],
|
||||
}
|
||||
|
|
@ -1,290 +0,0 @@
|
|||
"""
|
||||
Moralis API Router - Wallet auth (SIWE/SIWS) + Data endpoints.
|
||||
Key 1: Data API (wallets, tokens, NFTs, whale tracking, streams)
|
||||
Key 2: Auth API (Sign-In With Ethereum/Solana - Phantom, MetaMask, etc.)
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/v1/moralis", tags=["moralis"])
|
||||
|
||||
|
||||
# ── Models ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
class AuthChallengeRequest(BaseModel):
|
||||
address: str
|
||||
chain: str = "eth" # eth, polygon, bsc, avalanche, arbitrum, optimism, base
|
||||
domain: str = "rugmunch.io"
|
||||
|
||||
|
||||
class SolanaChallengeRequest(BaseModel):
|
||||
address: str
|
||||
domain: str = "rugmunch.io"
|
||||
|
||||
|
||||
class VerifySignatureRequest(BaseModel):
|
||||
message: str
|
||||
signature: str
|
||||
|
||||
|
||||
class WalletQuery(BaseModel):
|
||||
address: str
|
||||
chain: str = "eth"
|
||||
limit: int = 20
|
||||
|
||||
|
||||
class StreamCreateRequest(BaseModel):
|
||||
webhook_url: str
|
||||
description: str
|
||||
chains: list[str] = ["0x1"]
|
||||
address: str | None = None
|
||||
topic0: list[str] | None = None
|
||||
|
||||
|
||||
# ── Auth Endpoints (Key 2 - Wallet Login) ────────────────────
|
||||
|
||||
|
||||
@router.post("/auth/challenge/evm")
|
||||
async def evm_challenge(req: AuthChallengeRequest):
|
||||
"""Request EVM Sign-In challenge (MetaMask, etc)."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
result = await mc.request_evm_challenge(
|
||||
address=req.address,
|
||||
chain=req.chain,
|
||||
domain=req.domain,
|
||||
)
|
||||
if result:
|
||||
return {"status": "ok", "challenge": result}
|
||||
raise HTTPException(status_code=502, detail="Moralis challenge failed")
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
|
||||
|
||||
@router.post("/auth/challenge/solana")
|
||||
async def solana_challenge(req: SolanaChallengeRequest):
|
||||
"""Request Solana Sign-In challenge (Phantom, etc)."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
result = await mc.request_solana_challenge(
|
||||
address=req.address,
|
||||
domain=req.domain,
|
||||
)
|
||||
if result:
|
||||
return {"status": "ok", "challenge": result}
|
||||
raise HTTPException(status_code=502, detail="Moralis challenge failed")
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
|
||||
|
||||
@router.post("/auth/verify/evm")
|
||||
async def verify_evm(req: VerifySignatureRequest):
|
||||
"""Verify EVM wallet signature - returns JWT token for authenticated session."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
result = await mc.verify_evm_signature(
|
||||
message=req.message,
|
||||
signature=req.signature,
|
||||
)
|
||||
if result:
|
||||
return {"status": "ok", "auth": result}
|
||||
raise HTTPException(status_code=401, detail="Signature verification failed")
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
|
||||
|
||||
@router.post("/auth/verify/solana")
|
||||
async def verify_solana(req: VerifySignatureRequest):
|
||||
"""Verify Solana wallet signature - returns JWT token for authenticated session."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
result = await mc.verify_solana_signature(
|
||||
message=req.message,
|
||||
signature=req.signature,
|
||||
)
|
||||
if result:
|
||||
return {"status": "ok", "auth": result}
|
||||
raise HTTPException(status_code=401, detail="Signature verification failed")
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
|
||||
|
||||
# ── Data Endpoints (Key 1 - Wallet Intelligence) ─────────────
|
||||
|
||||
|
||||
@router.post("/wallet/tokens")
|
||||
async def wallet_tokens(req: WalletQuery):
|
||||
"""Get ERC-20 tokens held by wallet."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
tokens = await mc.get_wallet_tokens(req.address, req.chain)
|
||||
return {"address": req.address, "chain": req.chain, "tokens": tokens[: req.limit]}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.post("/wallet/nfts")
|
||||
async def wallet_nfts(req: WalletQuery):
|
||||
"""Get NFTs held by wallet."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
nfts = await mc.get_wallet_nfts(req.address, req.chain, req.limit)
|
||||
return {"address": req.address, "chain": req.chain, "nfts": nfts}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.post("/wallet/balance")
|
||||
async def wallet_balance(req: WalletQuery):
|
||||
"""Get native balance (ETH/MATIC/BNB) for wallet."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
balance = await mc.get_wallet_native_balance(req.address, req.chain)
|
||||
return {"address": req.address, "chain": req.chain, "balance": balance}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.post("/wallet/transfers")
|
||||
async def wallet_transfers(req: WalletQuery):
|
||||
"""Get token transfer history (whale tracking)."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
transfers = await mc.get_wallet_token_transfers(req.address, req.chain, req.limit)
|
||||
return {"address": req.address, "chain": req.chain, "transfers": transfers}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/token/{token_address}/price")
|
||||
async def token_price(token_address: str, chain: str = "eth"):
|
||||
"""Get token price in USD."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
price = await mc.get_token_price(token_address, chain)
|
||||
return {"token": token_address, "chain": chain, "price": price}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/token/{token_address}/metadata")
|
||||
async def token_metadata(token_address: str, chain: str = "eth"):
|
||||
"""Get token metadata (name, symbol, decimals, logo)."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
metadata = await mc.get_token_metadata(token_address, chain)
|
||||
return {"token": token_address, "chain": chain, "metadata": metadata}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.get("/token/{token_address}/holders")
|
||||
async def token_holders(token_address: str, chain: str = "eth", limit: int = 20):
|
||||
"""Get top token holders (EVM chains)."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
holders = await mc.get_token_holders(token_address, chain, limit)
|
||||
return {"token": token_address, "chain": chain, "holders": holders}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
# ── Streams Management ───────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/streams")
|
||||
async def list_streams():
|
||||
"""List all Moralis streams (EVM webhooks)."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
streams = await mc.list_streams()
|
||||
return {"streams": streams}
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=str(e)[:200]) from e
|
||||
|
||||
|
||||
@router.post("/streams/create")
|
||||
async def create_stream(req: StreamCreateRequest):
|
||||
"""Create a Moralis stream (webhook for EVM events)."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
stream = await mc.create_stream(
|
||||
webhook_url=req.webhook_url,
|
||||
description=req.description,
|
||||
chains=req.chains,
|
||||
address=req.address,
|
||||
topic0=req.topic0,
|
||||
)
|
||||
if stream:
|
||||
return {"status": "created", "stream": stream}
|
||||
raise HTTPException(status_code=502, detail="Stream creation failed")
|
||||
except ImportError:
|
||||
raise HTTPException(status_code=503, detail="Moralis connector not available") from None
|
||||
|
||||
|
||||
# ── Health ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def moralis_health():
|
||||
"""Moralis connector status."""
|
||||
try:
|
||||
from app.moralis_connector import get_moralis_connector
|
||||
|
||||
mc = get_moralis_connector()
|
||||
return {"status": "ok", "service": "moralis-connector", **mc.status()}
|
||||
except ImportError:
|
||||
return {
|
||||
"status": "ok",
|
||||
"service": "moralis-connector",
|
||||
"data_api": False,
|
||||
"auth_api": False,
|
||||
}
|
||||
|
|
@ -1,65 +0,0 @@
|
|||
"""NFT Wash Trading Detector - detects fake volume in NFT collections."""
|
||||
|
||||
import os
|
||||
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
router = APIRouter(prefix="/api/v1/nft-detector", tags=["nft-detector"])
|
||||
|
||||
BACKEND = os.getenv("BACKEND_URL", "http://localhost:8000")
|
||||
RMI_KEY = os.getenv("RMI_INTERNAL_KEY", "rmi-internal-2026")
|
||||
|
||||
|
||||
@router.get("/check/{collection_address}")
|
||||
async def check_nft_wash(collection_address: str, chain: str = Query("ethereum")):
|
||||
"""Check NFT collection for wash trading patterns."""
|
||||
|
||||
# Use DataBus volume authenticity engine with NFT-specific heuristics
|
||||
score = 85 # Placeholder - real implementation queries DataBus
|
||||
|
||||
# NFT-specific wash trading patterns
|
||||
patterns = []
|
||||
patterns.append({"pattern": "Self-trades", "description": "Same wallet buying and selling", "detected": False})
|
||||
patterns.append({"pattern": "Round-trip", "description": "A→B→A within minutes", "detected": False})
|
||||
patterns.append({"pattern": "Bid stuffing", "description": "Fake bids to inflate floor price", "detected": False})
|
||||
patterns.append(
|
||||
{"pattern": "Wash pool", "description": "Group of wallets trading among themselves", "detected": False}
|
||||
)
|
||||
|
||||
if score < 30:
|
||||
risk = "CRITICAL"
|
||||
emoji = "🔴"
|
||||
elif score < 60:
|
||||
risk = "HIGH"
|
||||
emoji = "🟡"
|
||||
elif score < 80:
|
||||
risk = "MEDIUM"
|
||||
emoji = "⚪"
|
||||
else:
|
||||
risk = "LOW"
|
||||
emoji = "🟢"
|
||||
|
||||
return {
|
||||
"collection": collection_address,
|
||||
"chain": chain,
|
||||
"authenticity_score": score,
|
||||
"risk": risk,
|
||||
"emoji": emoji,
|
||||
"wash_patterns": patterns,
|
||||
"recommendation": "Likely authentic trading"
|
||||
if score >= 80
|
||||
else "Suspicious activity detected - verify before buying",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/floor-check/{collection_address}")
|
||||
async def check_floor_manipulation(collection_address: str, chain: str = Query("ethereum")):
|
||||
"""Check if NFT floor price is being manipulated."""
|
||||
return {
|
||||
"collection": collection_address,
|
||||
"floor_price_eth": 0.0,
|
||||
"bid_ask_spread_pct": 0,
|
||||
"suspicious_bids_24h": 0,
|
||||
"manipulation_risk": "LOW",
|
||||
"note": "NFT data providers integration pending",
|
||||
}
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
"""Stub for ollama_api router - local dev fallback."""
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
router = APIRouter(prefix="/api/v1/ollama", tags=["ollama"])
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def health():
|
||||
return {"status": "stub"}
|
||||
|
|
@ -1,567 +0,0 @@
|
|||
"""
|
||||
RMI Persistent State Layer
|
||||
===========================
|
||||
User watchlists, portfolios, saved scans, and alert history.
|
||||
Redis-backed for speed, with JSON serialization.
|
||||
|
||||
Endpoints:
|
||||
POST /api/v1/state/watchlist - Add address to watchlist
|
||||
DELETE /api/v1/state/watchlist/{id} - Remove from watchlist
|
||||
GET /api/v1/state/watchlist - List watched addresses
|
||||
POST /api/v1/state/portfolio - Add portfolio entry
|
||||
DELETE /api/v1/state/portfolio/{id} - Remove portfolio entry
|
||||
GET /api/v1/state/portfolio - Get portfolio summary
|
||||
GET /api/v1/state/portfolio/{address} - Get single holding details
|
||||
POST /api/v1/state/scan - Save a scan configuration
|
||||
GET /api/v1/state/scan/{id} - Get saved scan
|
||||
GET /api/v1/state/scan - List saved scans
|
||||
DELETE /api/v1/state/scan/{id} - Delete saved scan
|
||||
GET /api/v1/state/alerts - Get alert history
|
||||
GET /api/v1/state/alerts/unread - Get unread alerts
|
||||
POST /api/v1/state/alerts/{id}/read - Mark alert as read
|
||||
GET /api/v1/state/stats - User state statistics
|
||||
|
||||
Identity: Uses X-RMI-Dev-Key header for API key users, or
|
||||
X-Wallet-Identity header for verified wallet users.
|
||||
Falls back to fingerprint-based identity.
|
||||
|
||||
Author: RMI Development
|
||||
Date: 2026-06-05
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
import uuid
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
|
||||
from app.core.redis import get_redis
|
||||
|
||||
logger = logging.getLogger("persistent_state")
|
||||
|
||||
router = APIRouter(prefix="/api/v1/state", tags=["persistent-state"])
|
||||
|
||||
|
||||
# ── Identity Helper ───────────────────────────────────────────────
|
||||
|
||||
|
||||
def get_user_identity(request: Request) -> str:
|
||||
"""Get a stable user identity from the request."""
|
||||
# Priority 1: Developer API key
|
||||
dev_key = request.headers.get("X-RMI-Dev-Key", "") or request.headers.get("x-rmi-dev-key", "")
|
||||
if dev_key:
|
||||
import hashlib
|
||||
|
||||
return f"dev:{hashlib.sha256(dev_key.encode()).hexdigest()[:16]}"
|
||||
|
||||
# Priority 2: Wallet identity
|
||||
wallet_id = request.headers.get("X-Wallet-Identity", "") or request.headers.get("x-wallet-identity", "")
|
||||
if wallet_id:
|
||||
return f"wallet:{wallet_id}"
|
||||
|
||||
# Priority 3: Fingerprint
|
||||
fp = request.headers.get("x-fingerprint", "") or request.headers.get("X-Fingerprint", "")
|
||||
if fp:
|
||||
return f"fp:{fp[:16]}"
|
||||
|
||||
# Priority 4: IP address
|
||||
client_ip = request.client.host if request.client else "unknown"
|
||||
return f"ip:{client_ip}"
|
||||
|
||||
|
||||
# ── Redis Helper ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def add_to_watchlist(entry: WatchlistEntry, request: Request): # noqa: F821 -- pre-existing bug, see fix(f821) tracking issue
|
||||
"""Add an address to the user's watchlist."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(status_code=500, content={"error": "redis_unavailable"})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
watch_id = f"watch:{uuid.uuid4().hex[:12]}"
|
||||
|
||||
watch_data = {
|
||||
"id": watch_id,
|
||||
"address": entry.address.lower(),
|
||||
"chain": entry.chain,
|
||||
"label": entry.label or entry.address[:8],
|
||||
"alert_types": entry.alert_types,
|
||||
"notes": entry.notes,
|
||||
"added_at": datetime.now(UTC).isoformat(),
|
||||
"status": "active",
|
||||
}
|
||||
|
||||
# Store in sorted set (by added time)
|
||||
r.zadd(
|
||||
f"rmi:state:{user_id}:watchlist",
|
||||
{json.dumps(watch_data): time.time()},
|
||||
)
|
||||
|
||||
# Also index by address for quick lookup
|
||||
r.sadd(
|
||||
f"rmi:state:{user_id}:watchlist:addresses",
|
||||
f"{entry.chain}:{entry.address.lower()}",
|
||||
)
|
||||
|
||||
r.incr("rmi:state:stats:watchlist_total")
|
||||
|
||||
return JSONResponse(
|
||||
status_code=201,
|
||||
content={
|
||||
"success": True,
|
||||
"watch_id": watch_id,
|
||||
"address": entry.address,
|
||||
"chain": entry.chain,
|
||||
"message": f"Added {entry.address} to watchlist",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/watchlist/{watch_id}")
|
||||
async def remove_from_watchlist(watch_id: str, request: Request):
|
||||
"""Remove an address from the watchlist."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(status_code=500, content={"error": "redis_unavailable"})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
|
||||
# Find and remove the entry
|
||||
entries = r.zrange(f"rmi:state:{user_id}:watchlist", 0, -1)
|
||||
for entry_json in entries:
|
||||
entry = json.loads(entry_json)
|
||||
if entry.get("id") == watch_id:
|
||||
r.zrem(f"rmi:state:{user_id}:watchlist", entry_json)
|
||||
addr_key = f"{entry.get('chain', '')}:{entry.get('address', '')}"
|
||||
r.srem(f"rmi:state:{user_id}:watchlist:addresses", addr_key)
|
||||
return JSONResponse(content={"success": True, "message": f"Removed {entry.get('address')}"})
|
||||
|
||||
return JSONResponse(status_code=404, content={"error": "Watchlist entry not found"})
|
||||
|
||||
|
||||
@router.get("/watchlist")
|
||||
async def get_watchlist(request: Request):
|
||||
"""Get the user's watchlist."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(content={"watchlist": [], "count": 0})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
entries = r.zrange(f"rmi:state:{user_id}:watchlist", 0, -1)
|
||||
|
||||
watchlist = []
|
||||
for entry_json in entries:
|
||||
entry = json.loads(entry_json)
|
||||
watchlist.append(entry)
|
||||
|
||||
return JSONResponse(
|
||||
content={
|
||||
"watchlist": watchlist,
|
||||
"count": len(watchlist),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# ── Portfolio ─────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/portfolio")
|
||||
async def add_portfolio_entry(entry: PortfolioEntry, request: Request): # noqa: F821 -- pre-existing bug, see fix(f821) tracking issue
|
||||
"""Add a holding to the user's portfolio."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(status_code=500, content={"error": "redis_unavailable"})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
holding_id = f"holding:{uuid.uuid4().hex[:12]}"
|
||||
|
||||
holding_data = {
|
||||
"id": holding_id,
|
||||
"address": entry.address.lower(),
|
||||
"chain": entry.chain,
|
||||
"amount": entry.amount,
|
||||
"cost_basis": entry.cost_basis,
|
||||
"label": entry.label or entry.address[:8],
|
||||
"notes": entry.notes,
|
||||
"added_at": datetime.now(UTC).isoformat(),
|
||||
"status": "active",
|
||||
}
|
||||
|
||||
r.zadd(
|
||||
f"rmi:state:{user_id}:portfolio",
|
||||
{json.dumps(holding_data): time.time()},
|
||||
)
|
||||
|
||||
return JSONResponse(
|
||||
status_code=201,
|
||||
content={
|
||||
"success": True,
|
||||
"holding_id": holding_id,
|
||||
"address": entry.address,
|
||||
"message": f"Added {entry.address} to portfolio",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/portfolio/{holding_id}")
|
||||
async def remove_portfolio_entry(holding_id: str, request: Request):
|
||||
"""Remove a holding from the portfolio."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(status_code=500, content={"error": "redis_unavailable"})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
|
||||
entries = r.zrange(f"rmi:state:{user_id}:portfolio", 0, -1)
|
||||
for entry_json in entries:
|
||||
entry = json.loads(entry_json)
|
||||
if entry.get("id") == holding_id:
|
||||
r.zrem(f"rmi:state:{user_id}:portfolio", entry_json)
|
||||
return JSONResponse(content={"success": True, "message": f"Removed {entry.get('address')}"})
|
||||
|
||||
return JSONResponse(status_code=404, content={"error": "Portfolio entry not found"})
|
||||
|
||||
|
||||
@router.get("/portfolio")
|
||||
async def get_portfolio(request: Request):
|
||||
"""Get the user's portfolio summary."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(content={"portfolio": [], "total_value": 0})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
entries = r.zrange(f"rmi:state:{user_id}:portfolio", 0, -1)
|
||||
|
||||
portfolio = []
|
||||
total_cost = 0
|
||||
for entry_json in entries:
|
||||
entry = json.loads(entry_json)
|
||||
portfolio.append(entry)
|
||||
total_cost += entry.get("cost_basis", 0) * entry.get("amount", 0)
|
||||
|
||||
return JSONResponse(
|
||||
content={
|
||||
"portfolio": portfolio,
|
||||
"count": len(portfolio),
|
||||
"total_cost_basis": round(total_cost, 2),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@router.get("/portfolio/{address}")
|
||||
async def get_portfolio_holding(address: str, request: Request):
|
||||
"""Get details for a specific portfolio holding."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(status_code=500, content={"error": "redis_unavailable"})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
entries = r.zrange(f"rmi:state:{user_id}:portfolio", 0, -1)
|
||||
|
||||
for entry_json in entries:
|
||||
entry = json.loads(entry_json)
|
||||
if entry.get("address", "").lower() == address.lower():
|
||||
return JSONResponse(content={"holding": entry})
|
||||
|
||||
return JSONResponse(status_code=404, content={"error": "Holding not found"})
|
||||
|
||||
|
||||
# ── Saved Scans ───────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/scan")
|
||||
async def save_scan(scan: SavedScan, request: Request): # noqa: F821 -- pre-existing bug, see fix(f821) tracking issue
|
||||
"""Save a scan configuration for later use."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(status_code=500, content={"error": "redis_unavailable"})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
scan_id = f"scan:{uuid.uuid4().hex[:12]}"
|
||||
|
||||
scan_data = {
|
||||
"id": scan_id,
|
||||
"name": scan.name,
|
||||
"tool": scan.tool,
|
||||
"params": scan.params,
|
||||
"chain": scan.chain,
|
||||
"schedule": scan.schedule,
|
||||
"webhook_url": scan.webhook_url,
|
||||
"created_at": datetime.now(UTC).isoformat(),
|
||||
"last_run": None,
|
||||
"run_count": 0,
|
||||
"status": "active",
|
||||
}
|
||||
|
||||
r.set(
|
||||
f"rmi:state:{user_id}:scan:{scan_id}",
|
||||
json.dumps(scan_data),
|
||||
)
|
||||
|
||||
# Add to scan index
|
||||
r.zadd(
|
||||
f"rmi:state:{user_id}:scans",
|
||||
{scan_id: time.time()},
|
||||
)
|
||||
|
||||
return JSONResponse(
|
||||
status_code=201,
|
||||
content={
|
||||
"success": True,
|
||||
"scan_id": scan_id,
|
||||
"name": scan.name,
|
||||
"message": f"Saved scan '{scan.name}'",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/scan/{scan_id}")
|
||||
async def get_saved_scan(scan_id: str, request: Request):
|
||||
"""Get a saved scan configuration."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(status_code=500, content={"error": "redis_unavailable"})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
scan_data = r.get(f"rmi:state:{user_id}:scan:{scan_id}")
|
||||
|
||||
if not scan_data:
|
||||
return JSONResponse(status_code=404, content={"error": "Scan not found"})
|
||||
|
||||
return JSONResponse(content={"scan": json.loads(scan_data)})
|
||||
|
||||
|
||||
@router.get("/scan")
|
||||
async def list_saved_scans(request: Request):
|
||||
"""List all saved scans for the user."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(content={"scans": [], "count": 0})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
scan_ids = r.zrange(f"rmi:state:{user_id}:scans", 0, -1)
|
||||
|
||||
scans = []
|
||||
for scan_id in scan_ids:
|
||||
scan_data = r.get(f"rmi:state:{user_id}:scan:{scan_id}")
|
||||
if scan_data:
|
||||
scans.append(json.loads(scan_data))
|
||||
|
||||
return JSONResponse(
|
||||
content={
|
||||
"scans": scans,
|
||||
"count": len(scans),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/scan/{scan_id}")
|
||||
async def delete_saved_scan(scan_id: str, request: Request):
|
||||
"""Delete a saved scan."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(status_code=500, content={"error": "redis_unavailable"})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
|
||||
# Check ownership
|
||||
scan_data = r.get(f"rmi:state:{user_id}:scan:{scan_id}")
|
||||
if not scan_data:
|
||||
return JSONResponse(status_code=404, content={"error": "Scan not found"})
|
||||
|
||||
r.delete(f"rmi:state:{user_id}:scan:{scan_id}")
|
||||
r.zrem(f"rmi:state:{user_id}:scans", scan_id)
|
||||
|
||||
return JSONResponse(content={"success": True, "message": "Scan deleted"})
|
||||
|
||||
|
||||
# ── Alert History ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/alerts")
|
||||
async def get_alert_history(
|
||||
request: Request,
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
status: str | None = None,
|
||||
):
|
||||
"""Get alert history for the user."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(content={"alerts": [], "count": 0})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
|
||||
# Alerts stored as a list (newest first)
|
||||
alerts = []
|
||||
start = offset
|
||||
end = offset + limit - 1
|
||||
|
||||
alert_keys = r.lrange(f"rmi:state:{user_id}:alerts", start, end)
|
||||
for alert_json in alert_keys:
|
||||
alert = json.loads(alert_json)
|
||||
if status is None or alert.get("status") == status:
|
||||
alerts.append(alert)
|
||||
|
||||
total = r.llen(f"rmi:state:{user_id}:alerts")
|
||||
unread = r.scard(f"rmi:state:{user_id}:alerts:unread")
|
||||
|
||||
return JSONResponse(
|
||||
content={
|
||||
"alerts": alerts,
|
||||
"total": total,
|
||||
"unread": unread,
|
||||
"limit": limit,
|
||||
"offset": offset,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@router.get("/alerts/unread")
|
||||
async def get_unread_alerts(request: Request):
|
||||
"""Get unread alerts."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(content={"alerts": [], "count": 0})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
unread_ids = r.smembers(f"rmi:state:{user_id}:alerts:unread")
|
||||
|
||||
alerts = []
|
||||
for alert_id in list(unread_ids)[:50]: # Limit to 50
|
||||
alert_json = r.get(f"rmi:state:{user_id}:alert:{alert_id}")
|
||||
if alert_json:
|
||||
alerts.append(json.loads(alert_json))
|
||||
|
||||
return JSONResponse(
|
||||
content={
|
||||
"alerts": alerts,
|
||||
"count": len(alerts),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@router.post("/alerts/{alert_id}/read")
|
||||
async def mark_alert_read(alert_id: str, request: Request):
|
||||
"""Mark an alert as read."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(status_code=500, content={"error": "redis_unavailable"})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
|
||||
# Remove from unread set
|
||||
r.srem(f"rmi:state:{user_id}:alerts:unread", alert_id)
|
||||
|
||||
# Update status in alert data
|
||||
alert_json = r.get(f"rmi:state:{user_id}:alert:{alert_id}")
|
||||
if alert_json:
|
||||
alert = json.loads(alert_json)
|
||||
alert["status"] = "read"
|
||||
alert["read_at"] = datetime.now(UTC).isoformat()
|
||||
r.set(f"rmi:state:{user_id}:alert:{alert_id}", json.dumps(alert))
|
||||
|
||||
return JSONResponse(content={"success": True})
|
||||
|
||||
|
||||
@router.post("/alerts/mark-all-read")
|
||||
async def mark_all_alerts_read(request: Request):
|
||||
"""Mark all alerts as read."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(status_code=500, content={"error": "redis_unavailable"})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
r.delete(f"rmi:state:{user_id}:alerts:unread")
|
||||
|
||||
return JSONResponse(content={"success": True, "message": "All alerts marked as read"})
|
||||
|
||||
|
||||
# Helper function to push alerts (called by webhook/cron systems)
|
||||
def push_alert(
|
||||
user_id: str,
|
||||
alert_type: str,
|
||||
address: str,
|
||||
message: str,
|
||||
severity: str = "medium",
|
||||
data: dict | None = None,
|
||||
source: str = "rmi_scanner",
|
||||
):
|
||||
"""Push an alert to a user's alert history.
|
||||
|
||||
Called by webhook handlers, cron jobs, and scanner systems.
|
||||
"""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return
|
||||
|
||||
alert_id = f"alert:{uuid.uuid4().hex[:12]}"
|
||||
|
||||
alert_data = {
|
||||
"id": alert_id,
|
||||
"type": alert_type,
|
||||
"address": address,
|
||||
"message": message,
|
||||
"severity": severity,
|
||||
"data": data or {},
|
||||
"source": source,
|
||||
"created_at": datetime.now(UTC).isoformat(),
|
||||
"status": "unread",
|
||||
}
|
||||
|
||||
# Add to alert history (newest first)
|
||||
r.lpush(
|
||||
f"rmi:state:{user_id}:alerts",
|
||||
json.dumps(alert_data),
|
||||
)
|
||||
|
||||
# Trim to last 1000 alerts
|
||||
r.ltrim(f"rmi:state:{user_id}:alerts", 0, 999)
|
||||
|
||||
# Add to unread set
|
||||
r.sadd(f"rmi:state:{user_id}:alerts:unread", alert_id)
|
||||
|
||||
# Store individual alert for retrieval
|
||||
r.set(
|
||||
f"rmi:state:{user_id}:alert:{alert_id}",
|
||||
json.dumps(alert_data),
|
||||
ex=86400 * 30, # 30 day TTL
|
||||
)
|
||||
|
||||
# Update stats
|
||||
r.incr("rmi:state:stats:alerts_total")
|
||||
r.incr(f"rmi:state:stats:alerts:{alert_type}")
|
||||
|
||||
|
||||
# ── User State Statistics ─────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/stats")
|
||||
async def get_user_stats(request: Request):
|
||||
"""Get user state statistics."""
|
||||
r = get_redis()
|
||||
if not r:
|
||||
return JSONResponse(status_code=500, content={"error": "redis_unavailable"})
|
||||
|
||||
user_id = get_user_identity(request)
|
||||
|
||||
watchlist_count = r.zcard(f"rmi:state:{user_id}:watchlist")
|
||||
portfolio_count = r.zcard(f"rmi:state:{user_id}:portfolio")
|
||||
scan_count = r.zcard(f"rmi:state:{user_id}:scans")
|
||||
alerts_total = r.llen(f"rmi:state:{user_id}:alerts")
|
||||
alerts_unread = r.scard(f"rmi:state:{user_id}:alerts:unread")
|
||||
|
||||
return JSONResponse(
|
||||
content={
|
||||
"user_id": user_id,
|
||||
"watchlist": watchlist_count,
|
||||
"portfolio": portfolio_count,
|
||||
"saved_scans": scan_count,
|
||||
"alerts_total": alerts_total,
|
||||
"alerts_unread": alerts_unread,
|
||||
}
|
||||
)
|
||||
|
|
@ -1,255 +0,0 @@
|
|||
"""Prediction Market Monitor - Polymarket + Kalshi + Manifold tracker.
|
||||
Premium feature: wallet-level tracking, insider pattern detection, P&L analytics."""
|
||||
|
||||
import json
|
||||
import os
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Query
|
||||
|
||||
router = APIRouter(prefix="/api/v1/prediction-markets", tags=["prediction-markets"])
|
||||
|
||||
POLYMARKET = "https://clob.polymarket.com"
|
||||
BACKEND = os.getenv("BACKEND_URL", "http://localhost:8000")
|
||||
RMI_KEY = os.getenv("RMI_INTERNAL_KEY", "rmi-internal-2026")
|
||||
|
||||
# Tracked wallet registry (Redis in prod, in-memory for now)
|
||||
_tracked_wallets: dict[str, dict] = {}
|
||||
_insider_flags: list[dict] = []
|
||||
|
||||
# ═══════════════════════════════════════════
|
||||
# MARKET DATA
|
||||
# ═══════════════════════════════════════════
|
||||
|
||||
|
||||
@router.get("/markets")
|
||||
async def get_markets(
|
||||
category: str = Query("crypto", description="crypto|politics|sports|all"), limit: int = Query(10, le=50)
|
||||
):
|
||||
"""Get active prediction markets from Polymarket."""
|
||||
markets = []
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(f"{POLYMARKET}/markets", params={"limit": limit, "closed": "false"})
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
for m in data if isinstance(data, list) else data.get("data", []):
|
||||
if category == "all" or category in (m.get("category", "") or "").lower():
|
||||
markets.append(
|
||||
{
|
||||
"id": m.get("id"),
|
||||
"question": m.get("question"),
|
||||
"volume_usd": m.get("volumeNum", 0),
|
||||
"liquidity": m.get("liquidityNum", 0),
|
||||
"end_date": m.get("endDateIso"),
|
||||
"outcomes": m.get("outcomes", []),
|
||||
"outcome_prices": json.loads(m.get("outcomePrices", "[]")),
|
||||
"category": m.get("category"),
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Also get from DataBus if available
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(f"{BACKEND}/api/v1/databus/fetch/prediction_markets", headers={"X-RMI-Key": RMI_KEY})
|
||||
if r.status_code == 200:
|
||||
db_markets = r.json().get("data", r.json()).get("markets", [])
|
||||
markets.extend(db_markets)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
markets.sort(key=lambda m: m.get("volume_usd", 0), reverse=True)
|
||||
return {"markets": markets[:limit], "source": "polymarket+databus", "count": len(markets[:limit])}
|
||||
|
||||
|
||||
@router.get("/market/{market_id}")
|
||||
async def get_market_detail(market_id: str):
|
||||
"""Get detailed market info + recent trades."""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(f"{POLYMARKET}/markets/{market_id}")
|
||||
if r.status_code == 200:
|
||||
market = r.json()
|
||||
return {
|
||||
"id": market.get("id"),
|
||||
"question": market.get("question"),
|
||||
"description": market.get("description"),
|
||||
"volume_usd": market.get("volumeNum", 0),
|
||||
"liquidity": market.get("liquidityNum", 0),
|
||||
"end_date": market.get("endDateIso"),
|
||||
"outcomes": market.get("outcomes", []),
|
||||
"outcome_prices": json.loads(market.get("outcomePrices", "[]")),
|
||||
}
|
||||
except Exception:
|
||||
pass
|
||||
return {"error": "Market not found"}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════
|
||||
# WALLET TRACKING (PREMIUM)
|
||||
# ═══════════════════════════════════════════
|
||||
|
||||
|
||||
@router.post("/track-wallet")
|
||||
async def track_wallet(wallet_address: str, label: str = "", user_id: str = ""):
|
||||
"""Start tracking a wallet's prediction market activity. Premium feature."""
|
||||
wid = f"{user_id}:{wallet_address}" if user_id else wallet_address
|
||||
_tracked_wallets[wid] = {
|
||||
"wallet": wallet_address,
|
||||
"label": label,
|
||||
"user_id": user_id,
|
||||
"added_at": datetime.now(UTC).isoformat(),
|
||||
"positions": [],
|
||||
"pnl_usd": 0,
|
||||
"win_rate": 0,
|
||||
"total_trades": 0,
|
||||
}
|
||||
return {"tracked": wid, "wallet": wallet_address, "status": "monitoring"}
|
||||
|
||||
|
||||
@router.get("/wallet/{wallet_address}")
|
||||
async def get_wallet_profile(wallet_address: str):
|
||||
"""Get a tracked wallet's prediction market profile."""
|
||||
# Find wallet in tracked registry
|
||||
wallet_data = None
|
||||
for wid, data in _tracked_wallets.items():
|
||||
if wallet_address in wid:
|
||||
wallet_data = data
|
||||
break
|
||||
|
||||
if not wallet_data:
|
||||
# Try to fetch from DataBus
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10) as c:
|
||||
r = await c.get(
|
||||
f"{BACKEND}/api/v1/databus/fetch/entity_intel?address={wallet_address}",
|
||||
headers={"X-RMI-Key": RMI_KEY},
|
||||
)
|
||||
if r.status_code == 200:
|
||||
wallet_data = r.json().get("data", r.json())
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if not wallet_data:
|
||||
return {
|
||||
"wallet": wallet_address,
|
||||
"tracked": False,
|
||||
"note": "Not tracked. Use POST /api/v1/prediction-markets/track-wallet",
|
||||
}
|
||||
|
||||
# Check for insider patterns
|
||||
insider_risk = _analyze_insider_patterns(wallet_address, wallet_data)
|
||||
|
||||
return {
|
||||
"wallet": wallet_address,
|
||||
"tracked": True,
|
||||
"label": wallet_data.get("label", ""),
|
||||
"positions": wallet_data.get("positions", []),
|
||||
"pnl_usd": wallet_data.get("pnl_usd", 0),
|
||||
"win_rate": wallet_data.get("win_rate", 0),
|
||||
"total_trades": wallet_data.get("total_trades", 0),
|
||||
"insider_risk": insider_risk,
|
||||
}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════
|
||||
# INSIDER DETECTION (PREMIUM)
|
||||
# ═══════════════════════════════════════════
|
||||
|
||||
|
||||
def _analyze_insider_patterns(wallet: str, data: dict) -> dict:
|
||||
"""Analyze wallet for insider trading patterns."""
|
||||
risk = 0
|
||||
flags = []
|
||||
|
||||
trades = data.get("positions", [])
|
||||
if not trades:
|
||||
return {"risk": "unknown", "score": 0, "flags": []}
|
||||
|
||||
# Pattern 1: Trades right before major price moves
|
||||
pre_move_trades = 0
|
||||
for trade in trades:
|
||||
price_before = trade.get("price_before_move", trade.get("avg_price", 0))
|
||||
price_after = trade.get("price_after_move", trade.get("current_price", 0))
|
||||
if price_before > 0 and abs(price_after - price_before) / price_before > 0.3:
|
||||
pre_move_trades += 1
|
||||
|
||||
if pre_move_trades > 3:
|
||||
risk += 30
|
||||
flags.append(f"{pre_move_trades} trades before >30% price moves")
|
||||
|
||||
# Pattern 2: High win rate (>80%) with significant volume
|
||||
win_rate = data.get("win_rate", 0)
|
||||
total_trades = data.get("total_trades", 0)
|
||||
volume = data.get("pnl_usd", 0)
|
||||
|
||||
if win_rate > 80 and total_trades > 20 and volume > 10000:
|
||||
risk += 25
|
||||
flags.append(f"{win_rate}% win rate across {total_trades} trades - statistically anomalous")
|
||||
|
||||
# Pattern 3: Trades on markets with low public interest
|
||||
low_interest_trades = sum(1 for t in trades if t.get("market_volume", 0) < 1000)
|
||||
if low_interest_trades > 5 and win_rate > 70:
|
||||
risk += 20
|
||||
flags.append(f"{low_interest_trades} trades on low-volume markets with {win_rate}% win rate")
|
||||
|
||||
# Pattern 4: Consistent last-minute entries before resolution
|
||||
last_minute = sum(1 for t in trades if t.get("hours_before_resolution", 999) < 1)
|
||||
if last_minute > 3:
|
||||
risk += 15
|
||||
flags.append(f"{last_minute} trades within 1 hour of resolution - possible info leak")
|
||||
|
||||
if risk >= 60:
|
||||
level = "CRITICAL"
|
||||
elif risk >= 35:
|
||||
level = "HIGH"
|
||||
elif risk >= 15:
|
||||
level = "MEDIUM"
|
||||
else:
|
||||
level = "LOW"
|
||||
|
||||
return {"risk": level, "score": min(100, risk), "flags": flags}
|
||||
|
||||
|
||||
@router.get("/insider-leaderboard")
|
||||
async def insider_leaderboard(min_risk: int = Query(35, le=100), limit: int = Query(10, le=25)):
|
||||
"""Get wallets flagged for potential insider trading. Premium feature."""
|
||||
flagged = []
|
||||
for wid, data in list(_tracked_wallets.items())[:50]:
|
||||
analysis = _analyze_insider_patterns(wid, data)
|
||||
if analysis["score"] >= min_risk:
|
||||
flagged.append(
|
||||
{
|
||||
"wallet": data["wallet"],
|
||||
"label": data.get("label", "Unlabeled"),
|
||||
"insider_risk": analysis["risk"],
|
||||
"risk_score": analysis["score"],
|
||||
"flags": analysis["flags"],
|
||||
"pnl_usd": data.get("pnl_usd", 0),
|
||||
"win_rate": data.get("win_rate", 0),
|
||||
}
|
||||
)
|
||||
|
||||
flagged.sort(key=lambda w: w["risk_score"], reverse=True)
|
||||
return {"flagged_wallets": flagged[:limit], "total_flagged": len(flagged)}
|
||||
|
||||
|
||||
@router.get("/market-snapshot")
|
||||
async def market_snapshot():
|
||||
"""Quick overview of prediction markets + insider activity."""
|
||||
markets = (await get_markets("all", 5))["markets"]
|
||||
top_markets = [{"question": m["question"][:80], "volume_usd": m["volume_usd"]} for m in markets[:5]]
|
||||
|
||||
insider_count = len([w for w in _tracked_wallets.values() if w.get("insider_flag", False)])
|
||||
|
||||
return {
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
"active_markets": len(markets),
|
||||
"top_markets": top_markets,
|
||||
"tracked_wallets": len(_tracked_wallets),
|
||||
"insider_flags": insider_count,
|
||||
"source": "Polymarket + DataBus",
|
||||
}
|
||||
|
|
@ -1,634 +0,0 @@
|
|||
"""
|
||||
Price Consensus Engine - Multi-Source Aggregation with MAD Outlier Detection.
|
||||
|
||||
Queries 7+ price sources in parallel, applies Median Absolute Deviation (MAD)
|
||||
outlier filtering (z-score > 3 = outlier), and computes a weighted mean price
|
||||
using source reliability scores.
|
||||
|
||||
Sources: DexScreener, GeckoTerminal, Jupiter (Solana), DIA, CoinGecko,
|
||||
CryptoCompare, Coinpaprika - all free tier, no paid keys required.
|
||||
|
||||
Depends on: httpx, numpy (for median/percentile), optional env keys.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import numpy as np
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Source Reliability Scores (0.0-1.0, higher = more trusted) ─────────────
|
||||
|
||||
# These are initial weights based on historical accuracy, API stability,
|
||||
# and data freshness. They can be adjusted via _source_stats over time.
|
||||
DEFAULT_SOURCE_WEIGHTS = {
|
||||
"dexscreener": 0.90, # Direct DEX data, excellent for on-chain tokens
|
||||
"geckoterminal": 0.92, # CoinGecko's DEX aggregator, very reliable
|
||||
"jupiter": 0.88, # Solana's primary aggregator, excellent for Solana
|
||||
"dia": 0.85, # Oracle-grade data, transparent methodology
|
||||
"coingecko": 0.88, # CEX + DEX aggregation, broad coverage
|
||||
"cryptocompare": 0.82, # Institutional-grade, slower updates on microcaps
|
||||
"coinpaprika": 0.78, # Good coverage, slightly less reliable on low-cap
|
||||
"birdeye": 0.86, # Good Solana coverage, needs API key
|
||||
}
|
||||
|
||||
# ── Data Classes ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class PriceSource:
|
||||
"""A single price data provider."""
|
||||
|
||||
name: str
|
||||
weight: float # Reliability score 0-1
|
||||
fetcher: Any = None # Async callable: (address, chain) → Optional[float]
|
||||
last_price: float = 0.0
|
||||
last_latency: float = 0.0
|
||||
error_count: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class PriceConsensus:
|
||||
"""Result of multi-source price consensus."""
|
||||
|
||||
price: float | None = None
|
||||
confidence: float = 0.0 # 0-100%
|
||||
sources_used: list[str] = field(default_factory=list)
|
||||
outlier_sources: list[str] = field(default_factory=list)
|
||||
failed_sources: list[str] = field(default_factory=list)
|
||||
individual_prices: dict[str, float] = field(default_factory=dict)
|
||||
median: float | None = None
|
||||
mad: float | None = None
|
||||
std_dev: float | None = None
|
||||
spread_pct: float | None = None # (max-min)/median * 100
|
||||
|
||||
@property
|
||||
def is_reliable(self) -> bool:
|
||||
return self.confidence >= 60.0 and self.price is not None
|
||||
|
||||
|
||||
# ── Price Consensus Engine ─────────────────────────────────────────────────
|
||||
|
||||
|
||||
class PriceConsensusEngine:
|
||||
"""Multi-source price aggregation with MAD-based outlier rejection.
|
||||
|
||||
Fetches from all configured sources in parallel, removes statistical
|
||||
outliers (z-score > 3 using Median Absolute Deviation), and computes
|
||||
a weighted mean of the remaining prices. Falls back gracefully if
|
||||
fewer than 2 sources respond.
|
||||
"""
|
||||
|
||||
# Timeout per source fetch
|
||||
PER_SOURCE_TIMEOUT = 10.0
|
||||
|
||||
# If a source fails this many times consecutively, lower its effective weight
|
||||
MAX_CONSECUTIVE_ERRORS = 5
|
||||
|
||||
def __init__(self):
|
||||
self._sources: dict[str, PriceSource] = {}
|
||||
self._lock = asyncio.Lock()
|
||||
self._setup_sources()
|
||||
|
||||
def _setup_sources(self):
|
||||
"""Register all price sources with their fetcher callables."""
|
||||
sources = [
|
||||
("dexscreener", self._fetch_dexscreener, DEFAULT_SOURCE_WEIGHTS["dexscreener"]),
|
||||
("geckoterminal", self._fetch_geckoterminal, DEFAULT_SOURCE_WEIGHTS["geckoterminal"]),
|
||||
("jupiter", self._fetch_jupiter, DEFAULT_SOURCE_WEIGHTS["jupiter"]),
|
||||
("dia", self._fetch_dia, DEFAULT_SOURCE_WEIGHTS["dia"]),
|
||||
("coingecko", self._fetch_coingecko, DEFAULT_SOURCE_WEIGHTS["coingecko"]),
|
||||
("cryptocompare", self._fetch_cryptocompare, DEFAULT_SOURCE_WEIGHTS["cryptocompare"]),
|
||||
("coinpaprika", self._fetch_coinpaprika, DEFAULT_SOURCE_WEIGHTS["coinpaprika"]),
|
||||
]
|
||||
|
||||
# Birdeye if key is available
|
||||
birdeye_key = os.getenv("BIRDEYE_API_KEY", "")
|
||||
if birdeye_key and birdeye_key != "your_birdeye_key_here":
|
||||
sources.append(("birdeye", self._fetch_birdeye, DEFAULT_SOURCE_WEIGHTS["birdeye"]))
|
||||
|
||||
for name, fetcher, weight in sources:
|
||||
self._sources[name] = PriceSource(name=name, weight=weight, fetcher=fetcher)
|
||||
|
||||
logger.info(f"PriceConsensusEngine: {len(self._sources)} sources registered: {list(self._sources.keys())}")
|
||||
|
||||
# ── Source Fetchers ──────────────────────────────────────────────────
|
||||
|
||||
async def _fetch_dexscreener(self, address: str, chain: str) -> float | None:
|
||||
"""DexScreener free API - no key required."""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=self.PER_SOURCE_TIMEOUT) as client:
|
||||
r = await client.get(
|
||||
f"https://api.dexscreener.com/latest/dex/tokens/{address}",
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
pairs = data.get("pairs", [])
|
||||
if pairs:
|
||||
# Find the pair with highest liquidity
|
||||
best = max(pairs, key=lambda p: float(p.get("liquidity", {}).get("usd", 0) or 0))
|
||||
price = best.get("priceUsd")
|
||||
if price:
|
||||
return float(price)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.debug(f"DexScreener fetch error: {e}")
|
||||
return None
|
||||
|
||||
async def _fetch_geckoterminal(self, address: str, chain: str) -> float | None:
|
||||
"""GeckoTerminal free API - no key required."""
|
||||
network = self._chain_to_gecko_network(chain)
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=self.PER_SOURCE_TIMEOUT) as client:
|
||||
r = await client.get(
|
||||
f"https://api.geckoterminal.com/api/v2/networks/{network}/tokens/{address}",
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
token_data = data.get("data", {})
|
||||
attrs = token_data.get("attributes", {})
|
||||
price = attrs.get("price_usd")
|
||||
if price:
|
||||
return float(price)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.debug(f"GeckoTerminal fetch error: {e}")
|
||||
return None
|
||||
|
||||
async def _fetch_jupiter(self, address: str, chain: str) -> float | None:
|
||||
"""Jupiter price API - free, Solana only."""
|
||||
if chain.lower() not in ("solana", "sol"):
|
||||
return None
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=self.PER_SOURCE_TIMEOUT) as client:
|
||||
r = await client.get(
|
||||
f"https://price.jup.ag/v6/price?ids={address}",
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
token_data = data.get("data", {}).get(address)
|
||||
if token_data:
|
||||
price = token_data.get("price")
|
||||
if price:
|
||||
return float(price)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.debug(f"Jupiter fetch error: {e}")
|
||||
return None
|
||||
|
||||
async def _fetch_dia(self, address: str, chain: str) -> float | None:
|
||||
"""DIA oracle price feed - free, no key."""
|
||||
dia_chain = self._chain_to_dia_chain(chain)
|
||||
if not dia_chain:
|
||||
return None
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=self.PER_SOURCE_TIMEOUT) as client:
|
||||
r = await client.get(
|
||||
f"https://api.diadata.org/v1/assetQuotation/{dia_chain}/{address}",
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
price = data.get("Price")
|
||||
if price:
|
||||
return float(price)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.debug(f"DIA fetch error: {e}")
|
||||
return None
|
||||
|
||||
async def _fetch_coingecko(self, address: str, chain: str) -> float | None:
|
||||
"""CoinGecko token price by contract - free tier."""
|
||||
cg_chain = self._chain_to_coingecko_platform(chain)
|
||||
if not cg_chain:
|
||||
return None
|
||||
api_key = os.getenv("COINGECKO_API_KEY", "")
|
||||
headers = {"Accept": "application/json"}
|
||||
if api_key:
|
||||
headers["x-cg-demo-api-key"] = api_key
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=self.PER_SOURCE_TIMEOUT) as client:
|
||||
r = await client.get(
|
||||
f"https://api.coingecko.com/api/v3/simple/token_price/{cg_chain}",
|
||||
params={
|
||||
"contract_addresses": address,
|
||||
"vs_currencies": "usd",
|
||||
},
|
||||
headers=headers,
|
||||
)
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
price = data.get(address.lower(), {}).get("usd")
|
||||
if price:
|
||||
return float(price)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.debug(f"CoinGecko fetch error: {e}")
|
||||
return None
|
||||
|
||||
async def _fetch_cryptocompare(self, address: str, chain: str) -> float | None:
|
||||
"""CryptoCompare price API - free tier."""
|
||||
api_key = os.getenv("CRYPTOCOMPARE_API_KEY", "")
|
||||
headers = {"Accept": "application/json"}
|
||||
if api_key:
|
||||
headers["authorization"] = f"Apikey {api_key}"
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=self.PER_SOURCE_TIMEOUT) as client:
|
||||
r = await client.get(
|
||||
"https://min-api.cryptocompare.com/data/price",
|
||||
params={
|
||||
"fsym": address,
|
||||
"tsyms": "USD",
|
||||
},
|
||||
headers=headers,
|
||||
)
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
price = data.get("USD")
|
||||
if price:
|
||||
return float(price)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.debug(f"CryptoCompare fetch error: {e}")
|
||||
return None
|
||||
|
||||
async def _fetch_coinpaprika(self, address: str, chain: str) -> float | None:
|
||||
"""Coinpaprika free API - no key required."""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=self.PER_SOURCE_TIMEOUT) as client:
|
||||
# Try by contract address lookup
|
||||
r = await client.get(
|
||||
f"https://api.coinpaprika.com/v1/contracts/{chain}/{address}",
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
coin_id = data.get("id")
|
||||
if coin_id:
|
||||
# Get ticker for this coin
|
||||
r2 = await client.get(
|
||||
f"https://api.coinpaprika.com/v1/tickers/{coin_id}",
|
||||
headers={"Accept": "application/json"},
|
||||
)
|
||||
if r2.status_code == 200:
|
||||
ticker = r2.json()
|
||||
price = ticker.get("quotes", {}).get("USD", {}).get("price")
|
||||
if price:
|
||||
return float(price)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.debug(f"Coinpaprika fetch error: {e}")
|
||||
return None
|
||||
|
||||
async def _fetch_birdeye(self, address: str, chain: str) -> float | None:
|
||||
"""Birdeye price API - requires BIRDEYE_API_KEY."""
|
||||
api_key = os.getenv("BIRDEYE_API_KEY", "")
|
||||
if not api_key:
|
||||
return None
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=self.PER_SOURCE_TIMEOUT) as client:
|
||||
r = await client.get(
|
||||
"https://public-api.birdeye.so/defi/price",
|
||||
params={"address": address},
|
||||
headers={
|
||||
"X-API-KEY": api_key,
|
||||
"accept": "application/json",
|
||||
},
|
||||
)
|
||||
if r.status_code == 200:
|
||||
data = r.json()
|
||||
price = data.get("data", {}).get("value")
|
||||
if price:
|
||||
return float(price)
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.debug(f"Birdeye fetch error: {e}")
|
||||
return None
|
||||
|
||||
# ── Chain Name Normalization ──────────────────────────────────────────
|
||||
|
||||
@staticmethod
|
||||
def _chain_to_gecko_network(chain: str) -> str:
|
||||
mapping = {
|
||||
"solana": "solana",
|
||||
"sol": "solana",
|
||||
"ethereum": "eth",
|
||||
"eth": "eth",
|
||||
"1": "eth",
|
||||
"base": "base",
|
||||
"8453": "base",
|
||||
"bsc": "bsc",
|
||||
"56": "bsc",
|
||||
"bnb": "bsc",
|
||||
"arbitrum": "arbitrum",
|
||||
"42161": "arbitrum",
|
||||
"polygon": "polygon_pos",
|
||||
"137": "polygon_pos",
|
||||
"matic": "polygon_pos",
|
||||
"optimism": "optimism",
|
||||
"10": "optimism",
|
||||
"avalanche": "avax",
|
||||
"43114": "avax",
|
||||
"fantom": "fantom",
|
||||
"250": "fantom",
|
||||
}
|
||||
return mapping.get(chain.lower(), chain.lower())
|
||||
|
||||
@staticmethod
|
||||
def _chain_to_dia_chain(chain: str) -> str | None:
|
||||
mapping = {
|
||||
"solana": "Solana",
|
||||
"sol": "Solana",
|
||||
"ethereum": "Ethereum",
|
||||
"eth": "Ethereum",
|
||||
"1": "Ethereum",
|
||||
"base": "Base",
|
||||
"8453": "Base",
|
||||
"bsc": "BSC",
|
||||
"56": "BSC",
|
||||
"bnb": "BSC",
|
||||
"arbitrum": "Arbitrum",
|
||||
"42161": "Arbitrum",
|
||||
"polygon": "Polygon",
|
||||
"137": "Polygon",
|
||||
"optimism": "Optimism",
|
||||
"10": "Optimism",
|
||||
}
|
||||
return mapping.get(chain.lower())
|
||||
|
||||
@staticmethod
|
||||
def _chain_to_coingecko_platform(chain: str) -> str | None:
|
||||
mapping = {
|
||||
"solana": "solana",
|
||||
"sol": "solana",
|
||||
"ethereum": "ethereum",
|
||||
"eth": "ethereum",
|
||||
"1": "ethereum",
|
||||
"base": "base",
|
||||
"8453": "base",
|
||||
"bsc": "binance-smart-chain",
|
||||
"56": "binance-smart-chain",
|
||||
"bnb": "binance-smart-chain",
|
||||
"arbitrum": "arbitrum-one",
|
||||
"42161": "arbitrum-one",
|
||||
"polygon": "polygon-pos",
|
||||
"137": "polygon-pos",
|
||||
"matic": "polygon-pos",
|
||||
"optimism": "optimistic-ethereum",
|
||||
"10": "optimistic-ethereum",
|
||||
"avalanche": "avalanche",
|
||||
"43114": "avalanche",
|
||||
"fantom": "fantom",
|
||||
"250": "fantom",
|
||||
}
|
||||
return mapping.get(chain.lower())
|
||||
|
||||
# ── Core Consensus Logic ──────────────────────────────────────────────
|
||||
|
||||
async def get_consensus_price(
|
||||
self,
|
||||
token_address: str,
|
||||
chain: str = "solana",
|
||||
) -> PriceConsensus:
|
||||
"""Fetch prices from all sources and compute consensus.
|
||||
|
||||
Args:
|
||||
token_address: Token contract address / mint
|
||||
chain: Blockchain identifier (solana, ethereum, base, etc.)
|
||||
|
||||
Returns:
|
||||
PriceConsensus with consensus price, confidence, and breakdown.
|
||||
"""
|
||||
if not self._sources:
|
||||
return PriceConsensus(
|
||||
price=None,
|
||||
confidence=0.0,
|
||||
failed_sources=["no_sources_configured"],
|
||||
)
|
||||
|
||||
# Fire all source fetchers in parallel
|
||||
tasks = []
|
||||
source_names = []
|
||||
for name, source in self._sources.items():
|
||||
tasks.append(source.fetcher(token_address, chain))
|
||||
source_names.append(name)
|
||||
|
||||
start = time.monotonic()
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
# Collect successful prices and track failures
|
||||
prices: dict[str, float] = {}
|
||||
failed: list[str] = []
|
||||
|
||||
for name, result in zip(source_names, results, strict=False):
|
||||
if isinstance(result, Exception):
|
||||
logger.debug(f"Source {name} exception: {result}")
|
||||
failed.append(name)
|
||||
async with self._lock:
|
||||
if name in self._sources:
|
||||
self._sources[name].error_count += 1
|
||||
elif result is not None and isinstance(result, (int, float)):
|
||||
if result > 0:
|
||||
prices[name] = float(result)
|
||||
latency = time.monotonic() - start
|
||||
async with self._lock:
|
||||
if name in self._sources:
|
||||
self._sources[name].last_price = float(result)
|
||||
self._sources[name].last_latency = latency
|
||||
self._sources[name].error_count = 0
|
||||
else:
|
||||
failed.append(name)
|
||||
|
||||
# If no sources returned a price, return null consensus
|
||||
if not prices:
|
||||
logger.warning(f"No price sources responded for {token_address} on {chain}")
|
||||
return PriceConsensus(
|
||||
price=None,
|
||||
confidence=0.0,
|
||||
failed_sources=failed,
|
||||
)
|
||||
|
||||
price_values = list(prices.values())
|
||||
price_names = list(prices.keys())
|
||||
|
||||
# Single source: return it but with low confidence
|
||||
if len(price_values) == 1:
|
||||
return PriceConsensus(
|
||||
price=price_values[0],
|
||||
confidence=30.0,
|
||||
sources_used=price_names,
|
||||
failed_sources=failed,
|
||||
individual_prices=prices,
|
||||
median=price_values[0],
|
||||
)
|
||||
|
||||
# ── MAD-based Outlier Detection ─────────────────────────────────
|
||||
|
||||
arr = np.array(price_values)
|
||||
median = float(np.median(arr))
|
||||
mad = float(np.median(np.abs(arr - median)))
|
||||
|
||||
# If MAD is zero (all prices identical), no outliers
|
||||
if mad == 0:
|
||||
weighted_avg = self._weighted_mean(prices)
|
||||
return PriceConsensus(
|
||||
price=weighted_avg,
|
||||
confidence=95.0 if len(price_values) >= 3 else 70.0,
|
||||
sources_used=price_names,
|
||||
outlier_sources=[],
|
||||
failed_sources=failed,
|
||||
individual_prices=prices,
|
||||
median=median,
|
||||
mad=0.0,
|
||||
std_dev=0.0,
|
||||
spread_pct=0.0,
|
||||
)
|
||||
|
||||
# Compute modified z-scores using MAD
|
||||
# z_i = 0.6745 * (x_i - median) / MAD
|
||||
z_scores = 0.6745 * (arr - median) / mad
|
||||
|
||||
# Outlier threshold: |z| > 3 (very conservative - classic threshold)
|
||||
inliers_mask = np.abs(z_scores) <= 3.0
|
||||
outliers_mask = ~inliers_mask
|
||||
|
||||
inlier_prices = {
|
||||
name: price
|
||||
for name, price, is_inlier in zip(price_names, price_values, inliers_mask, strict=False)
|
||||
if is_inlier
|
||||
}
|
||||
outlier_names = [
|
||||
name
|
||||
for name, price, is_outlier in zip(price_names, price_values, outliers_mask, strict=False)
|
||||
if is_outlier
|
||||
]
|
||||
|
||||
# If all prices are outliers, fall back to all with low confidence
|
||||
if not inlier_prices:
|
||||
logger.warning(f"All prices flagged as outliers for {token_address} - using all with low confidence")
|
||||
weighted_avg = self._weighted_mean(prices)
|
||||
return PriceConsensus(
|
||||
price=weighted_avg,
|
||||
confidence=10.0,
|
||||
sources_used=price_names,
|
||||
outlier_sources=[],
|
||||
failed_sources=failed,
|
||||
individual_prices=prices,
|
||||
median=median,
|
||||
mad=float(mad),
|
||||
std_dev=float(np.std(arr)),
|
||||
spread_pct=self._spread_pct(price_values),
|
||||
)
|
||||
|
||||
# Compute weighted mean of inliers
|
||||
consensus_price = self._weighted_mean(inlier_prices)
|
||||
|
||||
# Confidence calculation
|
||||
total_sources = len(self._sources)
|
||||
inlier_count = len(inlier_prices)
|
||||
responder_count = len(price_values)
|
||||
|
||||
# Base confidence from inlier agreement ratio
|
||||
if inlier_count >= 3:
|
||||
agreement_ratio = inlier_count / responder_count
|
||||
confidence = agreement_ratio * 85.0 + 10.0 # 70-95 range
|
||||
elif inlier_count == 2:
|
||||
confidence = 55.0
|
||||
else:
|
||||
confidence = 35.0
|
||||
|
||||
# Penalize if we had many failures
|
||||
failure_penalty = (len(failed) / max(total_sources, 1)) * 20.0
|
||||
confidence = max(5.0, confidence - failure_penalty)
|
||||
|
||||
# Bonus for low spread among inliers
|
||||
inlier_values = list(inlier_prices.values())
|
||||
if len(inlier_values) >= 2:
|
||||
spread = self._spread_pct(inlier_values)
|
||||
if spread is not None and spread < 2.0:
|
||||
confidence = min(100.0, confidence + 10.0)
|
||||
|
||||
return PriceConsensus(
|
||||
price=round(consensus_price, 12),
|
||||
confidence=round(confidence, 1),
|
||||
sources_used=list(inlier_prices.keys()),
|
||||
outlier_sources=outlier_names,
|
||||
failed_sources=failed,
|
||||
individual_prices=prices,
|
||||
median=round(median, 12),
|
||||
mad=round(float(mad), 12) if mad else None,
|
||||
std_dev=round(float(np.std(arr)), 12),
|
||||
spread_pct=self._spread_pct(price_values),
|
||||
)
|
||||
|
||||
# ── Helpers ───────────────────────────────────────────────────────────
|
||||
|
||||
def _weighted_mean(self, prices: dict[str, float]) -> float:
|
||||
"""Weighted mean using source reliability weights, adjusted by error history."""
|
||||
if not prices:
|
||||
return 0.0
|
||||
total_weight = 0.0
|
||||
weighted_sum = 0.0
|
||||
for name, price in prices.items():
|
||||
source = self._sources.get(name)
|
||||
if source:
|
||||
# Reduce weight if source has errors
|
||||
error_penalty = min(0.5, source.error_count * 0.1)
|
||||
weight = source.weight * (1.0 - error_penalty)
|
||||
else:
|
||||
weight = 0.5
|
||||
weighted_sum += price * weight
|
||||
total_weight += weight
|
||||
return weighted_sum / total_weight if total_weight > 0 else 0.0
|
||||
|
||||
@staticmethod
|
||||
def _spread_pct(values: list[float]) -> float | None:
|
||||
"""(max - min) / median * 100. Lower = more consensus."""
|
||||
if len(values) < 2:
|
||||
return None
|
||||
arr = np.array(values)
|
||||
median = float(np.median(arr))
|
||||
if median == 0:
|
||||
return None
|
||||
return round(float((arr.max() - arr.min()) / median * 100), 2)
|
||||
|
||||
# ── Stats ─────────────────────────────────────────────────────────────
|
||||
|
||||
async def stats(self) -> dict[str, Any]:
|
||||
"""Return per-source stats and aggregate metrics."""
|
||||
source_stats = {}
|
||||
async with self._lock:
|
||||
for name, src in self._sources.items():
|
||||
source_stats[name] = {
|
||||
"weight": src.weight,
|
||||
"effective_weight": round(src.weight * (1.0 - min(0.5, src.error_count * 0.1)), 3),
|
||||
"last_price": src.last_price,
|
||||
"last_latency": round(src.last_latency, 3),
|
||||
"error_count": src.error_count,
|
||||
}
|
||||
return {
|
||||
"total_sources": len(self._sources),
|
||||
"sources": source_stats,
|
||||
}
|
||||
|
||||
|
||||
# ── Singleton ─────────────────────────────────────────────────────────────
|
||||
|
||||
_price_engine: PriceConsensusEngine | None = None
|
||||
|
||||
|
||||
def get_price_consensus() -> PriceConsensusEngine:
|
||||
"""Get the global PriceConsensusEngine singleton."""
|
||||
global _price_engine
|
||||
if _price_engine is None:
|
||||
_price_engine = PriceConsensusEngine()
|
||||
return _price_engine
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,96 +0,0 @@
|
|||
from fastapi import APIRouter
|
||||
|
||||
router = APIRouter(prefix="/api/v1/protection", tags=["protection"])
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def check_health() -> dict:
|
||||
"""Check health of all protection modules.
|
||||
|
||||
Returns:
|
||||
{"status": "ok|degraded|down", "modules": {...}}
|
||||
"""
|
||||
status = "ok"
|
||||
modules = {
|
||||
"rag": {"status": "ok"},
|
||||
"scanner": {"status": "ok"},
|
||||
"blocklist": {"status": "ok"},
|
||||
"wallet_labels": {"status": "ok"},
|
||||
"entity_intel": {"status": "ok"},
|
||||
}
|
||||
|
||||
# Check RAG
|
||||
try:
|
||||
from app.rag_service import search_similar
|
||||
|
||||
await search_similar("health_check", "known_scams", limit=1, min_similarity=0.1)
|
||||
except Exception as e:
|
||||
modules["rag"]["status"] = "down"
|
||||
modules["rag"]["error"] = str(e)
|
||||
status = "degraded"
|
||||
|
||||
# Check scanner
|
||||
try:
|
||||
from app.degen_security_scanner import DegenSecurityScanner
|
||||
|
||||
scanner = DegenSecurityScanner()
|
||||
await scanner.scan_token("health_check", "solana")
|
||||
except Exception as e:
|
||||
modules["scanner"]["status"] = "down"
|
||||
modules["scanner"]["error"] = str(e)
|
||||
status = "degraded"
|
||||
|
||||
# Check blocklist
|
||||
try:
|
||||
await get_blocklist() # noqa: F821 -- pre-existing bug, see fix(f821) tracking issue
|
||||
except Exception as e:
|
||||
modules["blocklist"]["status"] = "down"
|
||||
modules["blocklist"]["error"] = str(e)
|
||||
status = "degraded"
|
||||
|
||||
# Check wallet labels
|
||||
try:
|
||||
from app.wallet_label_loader import lookup_wallet_label
|
||||
|
||||
lookup_wallet_label("health_check", "solana")
|
||||
except Exception as e:
|
||||
modules["wallet_labels"]["status"] = "down"
|
||||
modules["wallet_labels"]["error"] = str(e)
|
||||
status = "degraded"
|
||||
|
||||
# Check entity intel
|
||||
try:
|
||||
from app.entity_intel import get_entity_intel
|
||||
|
||||
await get_entity_intel("health_check", "solana")
|
||||
except Exception as e:
|
||||
modules["entity_intel"]["status"] = "down"
|
||||
modules["entity_intel"]["error"] = str(e)
|
||||
status = "degraded"
|
||||
|
||||
return {"status": status, "modules": modules}
|
||||
|
||||
|
||||
async def check_url(url: str) -> dict:
|
||||
"""Stub for URL checking."""
|
||||
return {"safe": True, "url": url}
|
||||
|
||||
|
||||
async def check_wallet(address: str, chain: str = "solana") -> dict:
|
||||
"""Stub for wallet checking."""
|
||||
return {"safe": True, "address": address, "chain": chain}
|
||||
|
||||
|
||||
async def check_token(address: str, chain: str = "solana") -> dict:
|
||||
"""Stub for token checking."""
|
||||
return {"safe": True, "address": address, "chain": chain, "risk_score": 0, "flags": []}
|
||||
|
||||
|
||||
async def get_blocklist_domains() -> dict:
|
||||
"""Stub for getting blocklist domains."""
|
||||
return {"domains": [], "last_updated": "2024-01-01T00:00:00Z"}
|
||||
|
||||
|
||||
async def get_protection_health() -> dict:
|
||||
"""Stub for getting protection health."""
|
||||
return {"status": "ok", "modules": {"protection": "ok"}}
|
||||
|
|
@ -1,123 +0,0 @@
|
|||
# Protection API endpoints for RMI
|
||||
# Core detection API that all protection modules call
|
||||
#
|
||||
# NOTE: This module is a stub. The underlying services (RAGService class,
|
||||
# Scanner, WalletLabels, EntityIntel, Blocklist) have not been implemented
|
||||
# as importable classes yet. The rag_service module uses standalone async
|
||||
# functions, not a RAGService class. We wrap imports in try/except so the
|
||||
# app doesn't crash on startup.
|
||||
|
||||
import logging
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Attempt imports - these are stubs and may not exist
|
||||
try:
|
||||
from app.rag_service import detect_scam_patterns, search_similar
|
||||
except ImportError:
|
||||
detect_scam_patterns = None
|
||||
search_similar = None
|
||||
|
||||
try:
|
||||
from app.url_scam_detector import check_url_safety
|
||||
except ImportError:
|
||||
check_url_safety = None
|
||||
|
||||
try:
|
||||
from app.wallet_persona import analyze_wallet
|
||||
except ImportError:
|
||||
analyze_wallet = None
|
||||
|
||||
router = APIRouter(prefix="/api/v1/protect", tags=["protection"])
|
||||
|
||||
|
||||
class UrlCheckRequest(BaseModel):
|
||||
url: str
|
||||
|
||||
|
||||
class WalletCheckRequest(BaseModel):
|
||||
address: str
|
||||
chain: str
|
||||
|
||||
|
||||
class TokenCheckRequest(BaseModel):
|
||||
address: str
|
||||
chain: str
|
||||
|
||||
|
||||
@router.post("/check-url")
|
||||
async def check_url(request: UrlCheckRequest):
|
||||
"""Check URL safety against scam patterns and blocklist"""
|
||||
try:
|
||||
# Use detect_scam_patterns from rag_service (standalone function)
|
||||
if detect_scam_patterns:
|
||||
rag_result = await detect_scam_patterns(request.url)
|
||||
if rag_result and rag_result.get("is_scam"):
|
||||
return {
|
||||
"safe": False,
|
||||
"risk": rag_result.get("risk_level", "high"),
|
||||
"reason": rag_result.get("reason", "Known scam pattern"),
|
||||
"source": "rag",
|
||||
}
|
||||
|
||||
# Use URL scam detector if available
|
||||
if check_url_safety:
|
||||
scan_result = await check_url_safety(request.url)
|
||||
if scan_result and not scan_result.get("safe", True):
|
||||
return {
|
||||
"safe": False,
|
||||
"risk": scan_result.get("risk", "high"),
|
||||
"reason": scan_result.get("reason", "Suspicious URL"),
|
||||
"source": "scanner",
|
||||
}
|
||||
|
||||
return {"safe": True, "risk": "low", "reason": "No known risks found", "source": "scanner"}
|
||||
except Exception as e:
|
||||
logger.error(f"URL check failed: {e}")
|
||||
return {"safe": True, "error": "service_unavailable", "reason": str(e)}
|
||||
|
||||
|
||||
@router.post("/check-wallet")
|
||||
async def check_wallet(request: WalletCheckRequest):
|
||||
"""Check wallet safety against labels and analysis"""
|
||||
try:
|
||||
if analyze_wallet:
|
||||
result = await analyze_wallet(request.address, request.chain)
|
||||
if result and result.get("risk_score", 0) > 70:
|
||||
return {
|
||||
"safe": False,
|
||||
"risk": "high",
|
||||
"reason": result.get("risk_reasons", ["High risk wallet"]),
|
||||
"source": "analysis",
|
||||
}
|
||||
|
||||
return {"safe": True, "risk": "low", "reason": "No known risks found", "source": "analysis"}
|
||||
except Exception as e:
|
||||
logger.error(f"Wallet check failed: {e}")
|
||||
return {"safe": True, "error": "service_unavailable", "reason": str(e)}
|
||||
|
||||
|
||||
@router.post("/check-token")
|
||||
async def check_token(request: TokenCheckRequest):
|
||||
"""Check token safety - delegates to x402-tools risk_scan for full analysis"""
|
||||
return {
|
||||
"safe": True,
|
||||
"note": "Use /api/v1/x402-tools/risk_scan for full token security analysis",
|
||||
"address": request.address,
|
||||
"chain": request.chain,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def protection_health():
|
||||
"""Health check for protection services"""
|
||||
services = {
|
||||
"rag_scam_detection": detect_scam_patterns is not None,
|
||||
"url_safety_check": check_url_safety is not None,
|
||||
"wallet_analysis": analyze_wallet is not None,
|
||||
}
|
||||
all_ok = all(services.values())
|
||||
return {"status": "ok" if all_ok else "partial", "services": services}
|
||||
|
|
@ -1,22 +0,0 @@
|
|||
import logging
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger("protection_router")
|
||||
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
status: str = "ok"
|
||||
error: str = None
|
||||
rag: dict = None
|
||||
blocklist: dict = None
|
||||
scanner: dict = None
|
||||
wallet: dict = None
|
||||
entity: dict = None
|
||||
|
||||
|
||||
@router.get("/health", response_model=HealthResponse)
|
||||
async def get_health():
|
||||
return HealthResponse(status="ok")
|
||||
|
|
@ -1,125 +0,0 @@
|
|||
"""
|
||||
Provider Health Dashboard + Credit Burn Tracking
|
||||
=================================================
|
||||
Live status of all 10 embedding providers + enterprise credit burn rate.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
|
||||
from app.rate_limiter import get_dispatcher
|
||||
|
||||
logger = logging.getLogger("health.dashboard")
|
||||
|
||||
|
||||
async def provider_health() -> dict:
|
||||
"""Live health status of all embedding providers."""
|
||||
d = await get_dispatcher()
|
||||
rl = await d.tracker.stats()
|
||||
|
||||
providers = []
|
||||
for p in rl["providers"]:
|
||||
# Determine health status
|
||||
day_pct = p["day_pct"]
|
||||
if not p["available"]:
|
||||
status = "exhausted" if day_pct >= 100 else "rate_limited"
|
||||
elif day_pct > 80:
|
||||
status = "warning"
|
||||
elif day_pct > 50:
|
||||
status = "active_warm"
|
||||
else:
|
||||
status = "healthy"
|
||||
|
||||
providers.append(
|
||||
{
|
||||
"id": p["id"],
|
||||
"name": p["name"],
|
||||
"status": status,
|
||||
"day_used": p["day_used"],
|
||||
"day_limit": p["day_limit"],
|
||||
"day_pct": p["day_pct"],
|
||||
"minute_used": p["minute_used"],
|
||||
"minute_limit": p["minute_limit"],
|
||||
"total_calls": p["total"],
|
||||
"free": p["free"],
|
||||
}
|
||||
)
|
||||
|
||||
healthy = sum(1 for p in providers if p["status"] == "healthy")
|
||||
warning = sum(1 for p in providers if p["status"] == "warning")
|
||||
degraded = sum(1 for p in providers if p["status"] in ("active_warm", "rate_limited"))
|
||||
exhausted = sum(1 for p in providers if p["status"] == "exhausted")
|
||||
|
||||
return {
|
||||
"summary": {
|
||||
"total": len(providers),
|
||||
"healthy": healthy,
|
||||
"warning": warning,
|
||||
"degraded": degraded,
|
||||
"exhausted": exhausted,
|
||||
},
|
||||
"providers": providers,
|
||||
"cache": d.cache.stats(),
|
||||
"timestamp": time.time(),
|
||||
}
|
||||
|
||||
|
||||
async def credit_burn() -> dict:
|
||||
"""Enterprise credit burn tracking - daily rate, projected exhaustion."""
|
||||
d = await get_dispatcher()
|
||||
rl = await d.tracker.stats()
|
||||
|
||||
# Calculate cost estimates
|
||||
# Vertex AI: $0.000025 per 1K chars (~$0.000025 per embedding call)
|
||||
# Gemini AI Studio: free up to 1,500/day per key
|
||||
# OpenRouter: free with credits
|
||||
# Mistral: free (1B tokens/month)
|
||||
|
||||
total_calls_today = 0
|
||||
total_free_limit = 0
|
||||
provider_usage = []
|
||||
|
||||
for p in rl["providers"]:
|
||||
total_calls_today += p["day_used"]
|
||||
total_free_limit += p["day_limit"] if p["free"] else 0
|
||||
|
||||
# Estimate cost (only Vertex AI burns credits from enterprise trial)
|
||||
cost_est = 0
|
||||
if p["id"] == "vertex_ai":
|
||||
# $0.000025 per embedding call (approximate)
|
||||
cost_est = round(p["day_used"] * 0.000025, 6)
|
||||
|
||||
if p["day_used"] > 0:
|
||||
provider_usage.append(
|
||||
{
|
||||
"name": p["name"],
|
||||
"calls_today": p["day_used"],
|
||||
"estimated_cost_usd": cost_est,
|
||||
"free": p["free"],
|
||||
}
|
||||
)
|
||||
|
||||
# $300 trial, assume 90 days started ~June 1, 2026
|
||||
trial_start = time.mktime(time.strptime("2026-06-01", "%Y-%m-%d"))
|
||||
trial_end = trial_start + 90 * 86400
|
||||
days_remaining = max(0, (trial_end - time.time()) / 86400)
|
||||
|
||||
# Daily burn rate estimate
|
||||
daily_cost = sum(p["estimated_cost_usd"] for p in provider_usage)
|
||||
|
||||
return {
|
||||
"trial": {
|
||||
"credits": 300,
|
||||
"days_remaining": round(days_remaining, 0),
|
||||
"exhaustion_date": "2026-08-30" if days_remaining > 0 else "now",
|
||||
},
|
||||
"today": {
|
||||
"total_calls": total_calls_today,
|
||||
"free_limit": total_free_limit,
|
||||
"utilization_pct": round(total_calls_today / max(total_free_limit, 1) * 100, 2),
|
||||
"estimated_cost_usd": daily_cost,
|
||||
"daily_burn_rate": f"${daily_cost:.6f}",
|
||||
},
|
||||
"provider_usage": provider_usage,
|
||||
"note": "Gemini, Mistral, NVIDIA, and OpenRouter are FREE. Only Vertex AI burns enterprise credits.",
|
||||
}
|
||||
|
|
@ -1,125 +0,0 @@
|
|||
"""
|
||||
RAG optimization endpoints - included via APIRouter for clean git history.
|
||||
Confidence scoring, Solidity chunking, deployer reputation, cross-chain,
|
||||
multi-modal, investigation narratives, graph RAG, streaming, email.
|
||||
"""
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Request
|
||||
|
||||
router = APIRouter(prefix="/api/v1", tags=["rag-optimization"])
|
||||
|
||||
|
||||
@router.post("/rag/confidence")
|
||||
async def rag_confidence_score(request: Request, data: dict):
|
||||
from app.confidence import score_confidence
|
||||
|
||||
return score_confidence(
|
||||
data.get("results", []),
|
||||
query=data.get("query", ""),
|
||||
entity_matches=data.get("entity_matches", []),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/rag/chunk-solidity")
|
||||
async def rag_chunk_solidity(request: Request, data: dict):
|
||||
source = data.get("source", "")
|
||||
if not source:
|
||||
return {"error": "source is required"}
|
||||
from app.solidity_chunker import chunk_solidity_ast
|
||||
|
||||
chunks = chunk_solidity_ast(source)
|
||||
return {"chunks": chunks, "count": len(chunks)}
|
||||
|
||||
|
||||
@router.post("/scanner/deployer-risk")
|
||||
async def scanner_deployer_risk(request: Request, data: dict):
|
||||
from app.deployer_reputation import score_deployer_risk
|
||||
|
||||
return score_deployer_risk(
|
||||
address=data.get("address", ""),
|
||||
deployment_count=data.get("deployment_count", 0),
|
||||
mixer_funded=data.get("mixer_funded", False),
|
||||
source_verified=data.get("source_verified", False),
|
||||
code_similarity=data.get("code_similarity", 0.0),
|
||||
rapid_deploy=data.get("rapid_deploy", False),
|
||||
funded_by_scammer=data.get("funded_by_scammer", False),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/rag/cross-chain")
|
||||
async def rag_cross_chain(request: Request, data: dict):
|
||||
from app.cross_chain import resolve_cross_chain_identity
|
||||
|
||||
return resolve_cross_chain_identity(
|
||||
address=data.get("address", ""),
|
||||
chain=data.get("chain", "ethereum"),
|
||||
funding_sources=data.get("funding_sources", []),
|
||||
label_hints=data.get("label_hints", []),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/rag/image-analysis")
|
||||
async def rag_image_analysis(request: Request, data: dict):
|
||||
from app.multimodal_rag import describe_image
|
||||
|
||||
return {
|
||||
"task": data.get("task", "describe"),
|
||||
"description": await describe_image(image_url=data.get("image_url", ""), task=data.get("task", "describe")),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/rag/logo-check")
|
||||
async def rag_logo_check(request: Request, data: dict):
|
||||
url = data.get("image_url", "")
|
||||
if not url:
|
||||
return {"error": "image_url is required"}
|
||||
r = await httpx.AsyncClient(timeout=15).get(url)
|
||||
from app.multimodal_rag import compute_image_hash, get_logo_db
|
||||
|
||||
h = compute_image_hash(r.content)
|
||||
return {**get_logo_db().check_theft(h), "computed_hash": h}
|
||||
|
||||
|
||||
@router.post("/rag/investigate")
|
||||
async def rag_investigate_endpoint(request: Request, data: dict):
|
||||
q = data.get("query", "")
|
||||
if not q:
|
||||
return {"error": "query is required"}
|
||||
from app.investigation_narratives import investigate
|
||||
|
||||
return await investigate(query=q, investigation_type=data.get("type", "auto"), max_hops=data.get("max_hops", 5))
|
||||
|
||||
|
||||
@router.get("/rag/graph-communities")
|
||||
async def rag_graph_communities(request: Request):
|
||||
from app.graph_rag import graph_rag_search
|
||||
|
||||
return await graph_rag_search(query="")
|
||||
|
||||
|
||||
@router.post("/rag/watch")
|
||||
async def rag_create_watch(request: Request, data: dict):
|
||||
from app.streaming_rag import create_watch
|
||||
|
||||
return await create_watch(token_address=data.get("token_address", ""), chain=data.get("chain", "ethereum"))
|
||||
|
||||
|
||||
@router.get("/rag/watches")
|
||||
async def rag_list_watches(request: Request):
|
||||
from app.streaming_rag import list_active_watches
|
||||
|
||||
return await list_active_watches()
|
||||
|
||||
|
||||
@router.get("/email/accounts")
|
||||
async def email_accounts(request: Request):
|
||||
return {
|
||||
"accounts": [
|
||||
{"id": "gmail", "email": "cryptorugmuncher@gmail.com", "type": "gmail"},
|
||||
{"id": "rmi", "email": "admin@rugmunch.io", "type": "gmail"},
|
||||
{"id": "rmi-local", "email": "admin@rugmunch.io", "type": "local"},
|
||||
{"id": "biz", "email": "biz@rugmunch.io", "type": "local"},
|
||||
],
|
||||
"webmail_url": "https://webmail.rugmunch.io/",
|
||||
}
|
||||
|
|
@ -1,413 +0,0 @@
|
|||
"""
|
||||
RAGAS Evaluation Pipeline - Weekly RAG quality assessment.
|
||||
=============================================================
|
||||
Evaluates RAG system against golden test set using RAGAS metrics:
|
||||
- faithfulness: is the answer grounded in retrieved context?
|
||||
- answer_relevancy: does the answer address the question?
|
||||
- context_precision: are retrieved chunks relevant?
|
||||
- context_recall: are all relevant chunks retrieved?
|
||||
|
||||
Golden test set: 20 known crypto scam/intel queries with expected answers.
|
||||
Runs weekly. Alerts on regression > 10% from baseline.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import httpx
|
||||
|
||||
BACKEND = "http://localhost:8000"
|
||||
RAG_SEARCH = f"{BACKEND}/api/v1/rag/search"
|
||||
|
||||
# ═══════════════════════════════════════════════════
|
||||
# GOLDEN TEST SET - 20 queries with expected answers
|
||||
# ═══════════════════════════════════════════════════
|
||||
GOLDEN_TESTS = [
|
||||
{
|
||||
"query": "What are the most common DeFi hack techniques?",
|
||||
"collection": "defi_hacks",
|
||||
"expected_terms": [
|
||||
"private key",
|
||||
"flash loan",
|
||||
"oracle manipulation",
|
||||
"reentrancy",
|
||||
"access control",
|
||||
"supply chain",
|
||||
],
|
||||
"min_results": 3,
|
||||
},
|
||||
{
|
||||
"query": "How much was stolen in the Bybit hack?",
|
||||
"collection": "defi_hacks",
|
||||
"expected_terms": ["bybit", "1.4", "billion", "1.46"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What is a honeypot scam in crypto?",
|
||||
"collection": "scam_patterns",
|
||||
"expected_terms": ["honeypot", "sell", "restriction", "cannot sell", "maxTx"],
|
||||
"min_results": 2,
|
||||
},
|
||||
{
|
||||
"query": "What are the signs of a rug pull?",
|
||||
"collection": "rug_timeline",
|
||||
"expected_terms": ["liquidity", "remove", "lp", "supply", "concentration", "fresh wallet"],
|
||||
"min_results": 2,
|
||||
},
|
||||
{
|
||||
"query": "How do crypto money launderers operate?",
|
||||
"collection": "transaction_patterns",
|
||||
"expected_terms": ["chain hopping", "mixer", "peel chain", "cex", "cross chain"],
|
||||
"min_results": 2,
|
||||
},
|
||||
{
|
||||
"query": "What did the Chainalysis 2025 crime report find?",
|
||||
"collection": "crime_reports",
|
||||
"expected_terms": ["40.9", "billion", "illicit", "stablecoin", "63%", "stolen"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What are the top smart contract vulnerabilities?",
|
||||
"collection": "vuln_patterns",
|
||||
"expected_terms": ["access control", "reentrancy", "oracle", "private key", "flash loan"],
|
||||
"min_results": 2,
|
||||
},
|
||||
{
|
||||
"query": "How much did North Korean hackers steal in 2024?",
|
||||
"collection": "crime_reports",
|
||||
"expected_terms": ["north korea", "1.34", "billion", "61%"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What happened with the Squid Game token?",
|
||||
"collection": "rug_timeline",
|
||||
"expected_terms": ["squid", "game", "honeypot", "couldn't sell", "3.38"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What is a bundle attack in crypto?",
|
||||
"collection": "transaction_patterns",
|
||||
"expected_terms": ["bundle", "sniper", "mempool", "same block", "coordinated"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "How does wash trading work in crypto?",
|
||||
"collection": "transaction_patterns",
|
||||
"expected_terms": ["wash trading", "circular", "volume", "inflation", "same entity"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What is the biggest DeFi hack ever?",
|
||||
"collection": "defi_hacks",
|
||||
"expected_terms": ["bybit", "ronin", "poly network", "billion"],
|
||||
"min_results": 2,
|
||||
},
|
||||
{
|
||||
"query": "What are pig butchering scams?",
|
||||
"collection": "crime_reports",
|
||||
"expected_terms": ["pig butchering", "romance", "investment", "scam"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "How do pump and dump schemes work in crypto?",
|
||||
"collection": "transaction_patterns",
|
||||
"expected_terms": ["pump", "dump", "insider", "accumulation", "fomo", "social"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What did TRM Labs report about crypto crime in 2025?",
|
||||
"collection": "crime_reports",
|
||||
"expected_terms": ["158", "billion", "illicit", "A7A5", "russia", "sanctions"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What percentage of bug bounty programs find critical bugs?",
|
||||
"collection": "vuln_patterns",
|
||||
"expected_terms": ["93.9%", "critical", "bug bounty", "immunefi"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What is the SafeMoon scam?",
|
||||
"collection": "rug_timeline",
|
||||
"expected_terms": ["safemoon", "liquidity", "drain", "arrested", "fraud"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "How are crypto scams using AI?",
|
||||
"collection": "crime_reports",
|
||||
"expected_terms": ["AI", "deepfake", "personalized", "KYC", "bypass"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What is a flash loan attack?",
|
||||
"collection": "vuln_patterns",
|
||||
"expected_terms": ["flash loan", "uncollateralized", "manipulate", "arbitrage"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What are the signs of a honeypot token contract?",
|
||||
"collection": "scam_patterns",
|
||||
"expected_terms": ["honeypot", "sell", "restriction", "maxTx", "trading", "owner"],
|
||||
"min_results": 2,
|
||||
},
|
||||
# ── Expanded tests (30 new) ──
|
||||
{
|
||||
"query": "What is the OWASP Smart Contract Top 10?",
|
||||
"collection": "vuln_patterns",
|
||||
"expected_terms": ["access control", "business logic", "oracle", "reentrancy", "proxy"],
|
||||
"min_results": 3,
|
||||
},
|
||||
{
|
||||
"query": "How was the Ronin Bridge hacked?",
|
||||
"collection": "defi_hacks",
|
||||
"expected_terms": ["ronin", "bridge", "validator", "private key"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What are common smart contract audit checklist items?",
|
||||
"collection": "contract_audits",
|
||||
"expected_terms": ["access control", "overflow", "reentrancy", "oracle", "validation"],
|
||||
"min_results": 2,
|
||||
},
|
||||
{
|
||||
"query": "How does a flash loan attack work?",
|
||||
"collection": "defi_hacks",
|
||||
"expected_terms": ["flash loan", "uncollateralized", "single transaction", "arbitrage"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What happened in the OneCoin scam?",
|
||||
"collection": "rug_timeline",
|
||||
"expected_terms": ["onecoin", "ponzi", "4 billion", "bitcoin"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What is a private key compromise attack?",
|
||||
"collection": "defi_hacks",
|
||||
"expected_terms": ["private key", "compromise", "hot wallet", "phishing"],
|
||||
"min_results": 2,
|
||||
},
|
||||
{
|
||||
"query": "How do cross-chain bridge hacks work?",
|
||||
"collection": "defi_hacks",
|
||||
"expected_terms": ["bridge", "cross chain", "validator", "message"],
|
||||
"min_results": 2,
|
||||
},
|
||||
{
|
||||
"query": "What are the top smart contract vulnerabilities in 2025?",
|
||||
"collection": "vuln_patterns",
|
||||
"expected_terms": ["access control", "reentrancy", "oracle", "overflow", "proxy"],
|
||||
"min_results": 2,
|
||||
},
|
||||
{
|
||||
"query": "How did the Squid Game token scam work?",
|
||||
"collection": "rug_timeline",
|
||||
"expected_terms": ["squid", "game", "honeypot", "sell", "2,861"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What is a supply chain attack in crypto?",
|
||||
"collection": "vuln_patterns",
|
||||
"expected_terms": ["supply chain", "ads", "power", "bybit", "compromise"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "How much crypto was stolen in 2024?",
|
||||
"collection": "crime_reports",
|
||||
"expected_terms": ["2.2", "billion", "stolen", "40.9"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What is the GMX V1 vulnerability?",
|
||||
"collection": "defi_hacks",
|
||||
"expected_terms": ["gmx", "reentrancy", "glp", "arbitrum"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "How does Tornado Cash work in laundering?",
|
||||
"collection": "transaction_patterns",
|
||||
"expected_terms": ["tornado", "mixer", "launder", "privacy"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What is an access control vulnerability?",
|
||||
"collection": "vuln_patterns",
|
||||
"expected_terms": ["access control", "unauthorized", "privileged", "owner"],
|
||||
"min_results": 2,
|
||||
},
|
||||
{
|
||||
"query": "How did BitConnect scam investors?",
|
||||
"collection": "rug_timeline",
|
||||
"expected_terms": ["bitconnect", "ponzi", "40%", "2 billion"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What happened with the Poly Network hack?",
|
||||
"collection": "defi_hacks",
|
||||
"expected_terms": ["poly network", "610", "cross chain", "white hat"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "How do North Korean hackers steal crypto?",
|
||||
"collection": "crime_reports",
|
||||
"expected_terms": ["north korea", "lazarus", "1.34", "61%", "IT workers"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What is a proxy upgrade vulnerability?",
|
||||
"collection": "vuln_patterns",
|
||||
"expected_terms": ["proxy", "upgrade", "implementation", "initialize"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "How does the SENTINEL scanner detect scams?",
|
||||
"collection": "known_scams",
|
||||
"expected_terms": ["scam", "detect", "honeypot", "rug"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What are common DeFi money laundering patterns?",
|
||||
"collection": "transaction_patterns",
|
||||
"expected_terms": ["peel chain", "mixer", "chain hop", "cex"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "How did the Euler Finance hack happen?",
|
||||
"collection": "defi_hacks",
|
||||
"expected_terms": ["euler", "flash loan", "197", "donate"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What percentage of stolen crypto is from private key compromises?",
|
||||
"collection": "crime_reports",
|
||||
"expected_terms": ["43.8%", "private key", "stolen", "compromise"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "How do oracle manipulation attacks work?",
|
||||
"collection": "vuln_patterns",
|
||||
"expected_terms": ["oracle", "price", "manipulation", "flash loan", "twap"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What is a reentrancy attack in Solidity?",
|
||||
"collection": "vuln_patterns",
|
||||
"expected_terms": ["reentrancy", "callback", "withdraw", "state"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "How did SafeMoon defraud investors?",
|
||||
"collection": "rug_timeline",
|
||||
"expected_terms": ["safemoon", "liquidity", "drain", "arrest"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What are the biggest DeFi hacks by amount lost?",
|
||||
"collection": "defi_hacks",
|
||||
"expected_terms": ["bybit", "ronin", "poly", "billion", "million"],
|
||||
"min_results": 3,
|
||||
},
|
||||
{
|
||||
"query": "How does a sniper bot attack tokens at launch?",
|
||||
"collection": "transaction_patterns",
|
||||
"expected_terms": ["sniper", "mempool", "same block", "gas"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What is business logic vulnerability in smart contracts?",
|
||||
"collection": "vuln_patterns",
|
||||
"expected_terms": ["business logic", "design", "lending", "amm", "reward"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "How did the Thodex exchange scam work?",
|
||||
"collection": "rug_timeline",
|
||||
"expected_terms": ["thodex", "exchange", "2 billion", "turkey"],
|
||||
"min_results": 1,
|
||||
},
|
||||
{
|
||||
"query": "What are the most exploited chains for DeFi hacks?",
|
||||
"collection": "defi_hacks",
|
||||
"expected_terms": ["ethereum", "bsc", "polygon", "arbitrum", "solana"],
|
||||
"min_results": 2,
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
async def evaluate_single(test: dict) -> dict:
|
||||
"""Run a single test query and score it."""
|
||||
query = test["query"]
|
||||
collection = test["collection"]
|
||||
expected_terms = [t.lower() for t in test["expected_terms"]]
|
||||
min_results = test["min_results"]
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=30) as c:
|
||||
r = await c.get(
|
||||
RAG_SEARCH,
|
||||
params={
|
||||
"q": query,
|
||||
"collection": collection,
|
||||
"limit": 10,
|
||||
},
|
||||
)
|
||||
if r.status_code != 200:
|
||||
return {"query": query, "error": f"HTTP {r.status_code}", "score": 0}
|
||||
|
||||
data = r.json()
|
||||
results = data.get("results", [])
|
||||
total = data.get("total", 0)
|
||||
|
||||
# Score: context_precision - how many expected terms appear in results?
|
||||
all_text = " ".join([res.get("content", res.get("text", "")) for res in results]).lower()
|
||||
terms_found = sum(1 for t in expected_terms if t in all_text)
|
||||
precision = terms_found / len(expected_terms) if expected_terms else 0
|
||||
|
||||
# Score: context_recall - did we get enough results?
|
||||
recall = min(total / min_results, 1.0) if min_results > 0 else 1.0
|
||||
|
||||
# Combined score
|
||||
score = precision * 0.6 + recall * 0.4
|
||||
|
||||
return {
|
||||
"query": query[:60],
|
||||
"collection": collection,
|
||||
"results_found": total,
|
||||
"min_expected": min_results,
|
||||
"terms_matched": f"{terms_found}/{len(expected_terms)}",
|
||||
"precision": round(precision, 3),
|
||||
"recall": round(recall, 3),
|
||||
"score": round(score, 3),
|
||||
}
|
||||
except Exception as e:
|
||||
return {"query": query[:60], "error": str(e)[:200], "score": 0}
|
||||
|
||||
|
||||
async def run_evaluation() -> dict:
|
||||
"""Run full evaluation against golden test set."""
|
||||
results = []
|
||||
for test in GOLDEN_TESTS:
|
||||
result = await evaluate_single(test)
|
||||
results.append(result)
|
||||
|
||||
scores = [r["score"] for r in results if "score" in r]
|
||||
avg_score = sum(scores) / len(scores) if scores else 0
|
||||
passing = sum(1 for s in scores if s >= 0.5)
|
||||
failing = sum(1 for s in scores if s < 0.3)
|
||||
|
||||
return {
|
||||
"test_count": len(GOLDEN_TESTS),
|
||||
"tests_run": len(results),
|
||||
"average_score": round(avg_score, 3),
|
||||
"passing": passing,
|
||||
"failing": failing,
|
||||
"pass_rate": round(passing / len(scores), 3) if scores else 0,
|
||||
"results": results,
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
result = asyncio.run(run_evaluation())
|
||||
print(json.dumps(result, indent=2))
|
||||
|
|
@ -1,242 +0,0 @@
|
|||
"""
|
||||
RAG Feedback Loop - Scanner results feed back into RAG
|
||||
======================================================
|
||||
|
||||
When the SENTINEL scanner confirms a token is a scam/honeypot/rug,
|
||||
this module adjusts the RAG document weights so similar patterns
|
||||
get higher priority in future searches.
|
||||
|
||||
Flow:
|
||||
Scanner reports scam → Feedback endpoint called
|
||||
→ Find matching RAG documents (by address, pattern, chain)
|
||||
→ Boost their weight in Redis metadata
|
||||
→ FAISS index marked for rebuild on next cycle
|
||||
→ Similar documents get +weight from confirmed patterns
|
||||
|
||||
Also tracks false positives:
|
||||
Scanner clears a token → Penalize matching RAG docs
|
||||
→ Reduce weight → fewer false alarms
|
||||
|
||||
This creates a LEARNING LOOP: the more the scanner runs,
|
||||
the smarter the RAG gets.
|
||||
"""
|
||||
|
||||
import contextlib
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger("rag.feedback")
|
||||
|
||||
# Weight adjustment constants
|
||||
SCAM_CONFIRMED_BOOST = 0.3 # +30% weight for confirmed scam matches
|
||||
FALSE_POSITIVE_PENALTY = -0.2 # -20% weight for false positives
|
||||
SCANNER_HIT_BOOST = 0.05 # +5% per scanner hit on a document
|
||||
MAX_WEIGHT = 3.0 # Cap at 3x original weight
|
||||
MIN_WEIGHT = 0.1 # Floor at 10% original
|
||||
|
||||
|
||||
async def record_scanner_result(
|
||||
address: str,
|
||||
chain: str,
|
||||
verdict: str, # "scam", "honeypot", "rug", "safe", "unknown"
|
||||
confidence: float, # 0.0 - 1.0
|
||||
flags: list | None = None,
|
||||
token_name: str = "",
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Record a scanner verdict and adjust RAG weights accordingly.
|
||||
|
||||
Called by SENTINEL scanner after each token scan.
|
||||
"""
|
||||
import os as _os
|
||||
|
||||
import redis.asyncio as aioredis
|
||||
|
||||
r = aioredis.Redis(
|
||||
host=_os.getenv("REDIS_HOST", "rmi-redis"),
|
||||
port=int(_os.getenv("REDIS_PORT", "6379")),
|
||||
password=_os.getenv("REDIS_PASSWORD", ""),
|
||||
db=int(_os.getenv("REDIS_DB", "0")),
|
||||
socket_connect_timeout=3,
|
||||
socket_timeout=3,
|
||||
decode_responses=True,
|
||||
)
|
||||
|
||||
adjustments = {"boosted": 0, "penalized": 0, "errors": 0}
|
||||
|
||||
try:
|
||||
is_scam = verdict in ("scam", "honeypot", "rug", "critical")
|
||||
adjustment = SCAM_CONFIRMED_BOOST if is_scam else FALSE_POSITIVE_PENALTY if verdict == "safe" else 0
|
||||
|
||||
if adjustment == 0:
|
||||
await r.aclose()
|
||||
return {"status": "no_action", "verdict": verdict}
|
||||
|
||||
# Find RAG documents matching this address
|
||||
# Search by address in known_scams collection
|
||||
doc_ids = await r.smembers(f"rag:entity:address:{address.lower()}")
|
||||
|
||||
if not doc_ids:
|
||||
# Try fuzzy - search for partial address in content
|
||||
# Use Redis scan for efficiency
|
||||
cursor = 0
|
||||
pattern = "rag:known_scams:*"
|
||||
while True:
|
||||
cursor, keys = await r.scan(cursor, match=pattern, count=100)
|
||||
for key in keys:
|
||||
try:
|
||||
doc = await r.get(key)
|
||||
if doc and address.lower() in doc.lower():
|
||||
doc_id = key.split(":")[-1]
|
||||
doc_ids.add(doc_id)
|
||||
except Exception:
|
||||
pass
|
||||
if cursor == 0:
|
||||
break
|
||||
|
||||
# Also find documents with similar scam patterns (by flags)
|
||||
if is_scam and flags:
|
||||
cursor = 0
|
||||
pattern = "rag:known_scams:*"
|
||||
while True:
|
||||
cursor, keys = await r.scan(cursor, match=pattern, count=100)
|
||||
for key in keys:
|
||||
try:
|
||||
doc = await r.get(key)
|
||||
if doc:
|
||||
doc_lower = doc.lower()
|
||||
matching_flags = sum(1 for f in flags if f.lower() in doc_lower)
|
||||
if matching_flags >= 2: # At least 2 flags match
|
||||
doc_id = key.split(":")[-1]
|
||||
doc_ids.add(doc_id)
|
||||
except Exception:
|
||||
pass
|
||||
if cursor == 0:
|
||||
break
|
||||
|
||||
# Apply weight adjustments
|
||||
for doc_id in doc_ids:
|
||||
try:
|
||||
# Read current weight
|
||||
weight_key = f"rag:weight:{doc_id}"
|
||||
current_weight = await r.get(weight_key)
|
||||
current = float(current_weight) if current_weight else 1.0
|
||||
|
||||
# Adjust
|
||||
new_weight = current + (adjustment * confidence)
|
||||
new_weight = max(MIN_WEIGHT, min(MAX_WEIGHT, new_weight))
|
||||
|
||||
await r.set(weight_key, str(new_weight))
|
||||
|
||||
if adjustment > 0:
|
||||
adjustments["boosted"] += 1
|
||||
else:
|
||||
adjustments["penalized"] += 1
|
||||
|
||||
except Exception as e:
|
||||
adjustments["errors"] += 1
|
||||
logger.debug(f"Weight adjustment error for {doc_id}: {e}")
|
||||
|
||||
# Record scanner hit count for analytics
|
||||
if doc_ids:
|
||||
hit_key = f"rag:scanner_hits:{address.lower()}"
|
||||
await r.incr(hit_key)
|
||||
await r.expire(hit_key, 86400 * 30) # 30 day TTL
|
||||
|
||||
# Mark FAISS for rebuild on next firehose cycle
|
||||
if adjustments["boosted"] + adjustments["penalized"] > 0:
|
||||
await r.set("rag:faiss:dirty", "1")
|
||||
|
||||
logger.info(
|
||||
f"RAG feedback: {verdict} for {address} → "
|
||||
f"+{adjustments['boosted']} boosted, "
|
||||
f"-{adjustments['penalized']} penalized"
|
||||
)
|
||||
|
||||
await r.aclose()
|
||||
return {
|
||||
"status": "ok",
|
||||
"verdict": verdict,
|
||||
"adjustment": adjustment,
|
||||
"documents_adjusted": adjustments["boosted"] + adjustments["penalized"],
|
||||
"boosted": adjustments["boosted"],
|
||||
"penalized": adjustments["penalized"],
|
||||
"faiss_marked_dirty": adjustments["boosted"] + adjustments["penalized"] > 0,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"RAG feedback error: {e}")
|
||||
with contextlib.suppress(Exception):
|
||||
await r.aclose()
|
||||
return {"status": "error", "detail": str(e)}
|
||||
|
||||
|
||||
async def get_document_weight(doc_id: str) -> float:
|
||||
"""Get the current learned weight for a RAG document."""
|
||||
import os as _os
|
||||
|
||||
import redis.asyncio as aioredis
|
||||
|
||||
try:
|
||||
r = aioredis.Redis(
|
||||
host=_os.getenv("REDIS_HOST", "rmi-redis"),
|
||||
port=int(_os.getenv("REDIS_PORT", "6379")),
|
||||
password=_os.getenv("REDIS_PASSWORD", ""),
|
||||
db=int(_os.getenv("REDIS_DB", "0")),
|
||||
socket_connect_timeout=2,
|
||||
socket_timeout=2,
|
||||
decode_responses=True,
|
||||
)
|
||||
weight = await r.get(f"rag:weight:{doc_id}")
|
||||
await r.aclose()
|
||||
return float(weight) if weight else 1.0
|
||||
except Exception:
|
||||
return 1.0
|
||||
|
||||
|
||||
async def get_feedback_stats() -> dict[str, Any]:
|
||||
"""Get feedback loop statistics."""
|
||||
import os as _os
|
||||
|
||||
import redis.asyncio as aioredis
|
||||
|
||||
try:
|
||||
r = aioredis.Redis(
|
||||
host=_os.getenv("REDIS_HOST", "rmi-redis"),
|
||||
port=int(_os.getenv("REDIS_PORT", "6379")),
|
||||
password=_os.getenv("REDIS_PASSWORD", ""),
|
||||
db=int(_os.getenv("REDIS_DB", "0")),
|
||||
socket_connect_timeout=2,
|
||||
socket_timeout=2,
|
||||
decode_responses=True,
|
||||
)
|
||||
|
||||
# Count weight-adjusted documents
|
||||
weight_keys = 0
|
||||
total_weight = 0.0
|
||||
cursor = 0
|
||||
while True:
|
||||
cursor, keys = await r.scan(cursor, match="rag:weight:*", count=500)
|
||||
for key in keys:
|
||||
try:
|
||||
w = await r.get(key)
|
||||
if w:
|
||||
weight_keys += 1
|
||||
total_weight += float(w)
|
||||
except Exception:
|
||||
pass
|
||||
if cursor == 0:
|
||||
break
|
||||
|
||||
avg_weight = total_weight / weight_keys if weight_keys > 0 else 1.0
|
||||
faiss_dirty = await r.get("rag:faiss:dirty") == "1"
|
||||
|
||||
await r.aclose()
|
||||
return {
|
||||
"documents_with_weights": weight_keys,
|
||||
"average_weight": round(avg_weight, 3),
|
||||
"faiss_needs_rebuild": faiss_dirty,
|
||||
"weight_range": f"{MIN_WEIGHT} - {MAX_WEIGHT}",
|
||||
}
|
||||
except Exception as e:
|
||||
return {"status": "error", "detail": str(e)}
|
||||
|
|
@ -1,877 +0,0 @@
|
|||
"""
|
||||
RAG Firehose - Continuous Intelligence Ingestion Engine
|
||||
========================================================
|
||||
|
||||
Self-feeding RAG pipeline that continuously pulls, filters, and ingests
|
||||
crypto intelligence from multiple sources at different cadences.
|
||||
|
||||
Architecture:
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ FIREHOSE ENGINE │
|
||||
│ │
|
||||
│ Hourly (news/social) Daily (scams/wallets) Weekly │
|
||||
│ │ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌─────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ News RSS │ │ Etherscan│ │ FAISS │ │
|
||||
│ │ CT Rundn │ │ Chainab. │ │ rebuild │ │
|
||||
│ │ Social │ │ Rekt DB │ │ RAGAS │ │
|
||||
│ │ Sentiment│ │ Solana │ │ eval │ │
|
||||
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
|
||||
│ │ │ │ │
|
||||
│ └──────────────────────┼─────────────────────┘ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────────────────┐ │
|
||||
│ │ SMART INGESTION PIPELINE │ │
|
||||
│ │ Filter → Dedup → Extract │ │
|
||||
│ │ → Classify → Embed → Store │ │
|
||||
│ └────────────┬───────────────┘ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────────────────┐ │
|
||||
│ │ RAG COLLECTIONS │ │
|
||||
│ │ known_scams, news_articles │ │
|
||||
│ │ forensic_reports, etc. │ │
|
||||
│ └────────────┬───────────────┘ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────────────────┐ │
|
||||
│ │ FEEDBACK LOOP │ │
|
||||
│ │ Scanner hits → boost docs │ │
|
||||
│ │ False positives → penalize │ │
|
||||
│ └────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────┘
|
||||
|
||||
Feed Cadences:
|
||||
- 15min: CT rundown, social sentiment, scam alerts (high-urgency)
|
||||
- 1hr: News RSS (200+ feeds), market brief, fear & greed
|
||||
- 6hr: X/Twitter profiles, KOL tracking, prediction markets
|
||||
- 24hr: Etherscan labels, Solana registry, chainabuse, rekt DB
|
||||
- 72hr: FAISS index rebuild, BM25 rebuild, RAGAS evaluation
|
||||
- 168hr: Full pattern extraction from confirmed scams, quality audit
|
||||
|
||||
Smart Ingestion:
|
||||
- Content hash dedup (Redis) - never ingest the same doc twice
|
||||
- Quality scoring - skip low-signal content (<30 score)
|
||||
- Entity extraction - pull addresses, chains, tokens, protocols
|
||||
- Auto-classification - route to correct collection
|
||||
- Batch embedding with rate limiting - never overload embedder
|
||||
- Per-collection size caps - auto-evict oldest on overflow
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger("rag.firehose")
|
||||
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
# Configuration
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
|
||||
RAG_API = "http://localhost:8000/api/v1/rag"
|
||||
|
||||
# Per-collection size caps (auto-evict oldest on overflow)
|
||||
COLLECTION_CAPS = {
|
||||
"known_scams": 50000, # scam addresses - keep forever, large
|
||||
"scam_patterns": 5000, # curated patterns - small, high quality
|
||||
"forensic_reports": 10000, # hack reports - medium
|
||||
"contract_audits": 5000, # code audits - medium
|
||||
"wallet_profiles": 100000, # labeled wallets - large
|
||||
"news_articles": 20000, # news - rolling window
|
||||
"market_intel": 5000, # market data - medium
|
||||
"token_analysis": 50000, # token data - large
|
||||
"transaction_patterns": 10000, # on-chain patterns - medium
|
||||
"social_sentiment": 10000, # social data - rolling
|
||||
"general": 10000, # misc - catch-all
|
||||
}
|
||||
|
||||
# Quality thresholds (skip docs below this score)
|
||||
MIN_QUALITY_SCORE = 30 # 0-100
|
||||
|
||||
# Rate limits (docs per minute per collection)
|
||||
RATE_LIMITS = {
|
||||
"known_scams": 60,
|
||||
"news_articles": 30,
|
||||
"social_sentiment": 20,
|
||||
"default": 30,
|
||||
}
|
||||
|
||||
# Content hash TTL in Redis (7 days for news, 30 for scams)
|
||||
HASH_TTL = {
|
||||
"news_articles": 604800, # 7 days
|
||||
"social_sentiment": 604800, # 7 days
|
||||
"known_scams": 2592000, # 30 days
|
||||
"default": 1209600, # 14 days
|
||||
}
|
||||
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
# Data Structures
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class FeedSource:
|
||||
"""A data source the firehose pulls from."""
|
||||
|
||||
name: str
|
||||
collection: str
|
||||
cadence_seconds: int
|
||||
fetch_fn: Any = field(repr=False)
|
||||
enabled: bool = True
|
||||
description: str = ""
|
||||
last_run: float = 0
|
||||
runs: int = 0
|
||||
docs_ingested: int = 0
|
||||
docs_skipped: int = 0
|
||||
errors: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class FirehoseStats:
|
||||
"""Current firehose statistics."""
|
||||
|
||||
running: bool = False
|
||||
uptime_seconds: float = 0
|
||||
total_sources: int = 0
|
||||
total_ingested: int = 0
|
||||
total_skipped: int = 0
|
||||
total_errors: int = 0
|
||||
sources: dict[str, dict] = field(default_factory=dict)
|
||||
last_activity: float = 0
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
# Smart Ingestion Pipeline
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class IngestionPipeline:
|
||||
"""Filters, deduplicates, and enriches documents before RAG storage."""
|
||||
|
||||
def __init__(self, client: httpx.AsyncClient):
|
||||
self.client = client
|
||||
self._rate_trackers: dict[str, list[float]] = {} # collection → recent ingest timestamps
|
||||
|
||||
def _check_rate(self, collection: str) -> bool:
|
||||
"""Return True if we're under the rate limit for this collection."""
|
||||
limit = RATE_LIMITS.get(collection, RATE_LIMITS["default"])
|
||||
now = time.monotonic()
|
||||
times = self._rate_trackers.setdefault(collection, [])
|
||||
# Remove timestamps older than 60s
|
||||
times[:] = [t for t in times if now - t < 60]
|
||||
return len(times) < limit
|
||||
|
||||
def _record_ingest(self, collection: str):
|
||||
"""Record an ingestion for rate limiting."""
|
||||
self._rate_trackers.setdefault(collection, []).append(time.monotonic())
|
||||
|
||||
def make_content_hash(self, content: str) -> str:
|
||||
"""Deterministic hash for dedup."""
|
||||
return hashlib.sha256(content.encode()).hexdigest()[:16]
|
||||
|
||||
async def is_duplicate(self, content_hash: str, collection: str) -> bool:
|
||||
"""Check if content hash already exists in Redis dedup set."""
|
||||
try:
|
||||
resp = await self.client.get(
|
||||
f"{RAG_API}/dedup-check",
|
||||
params={"hash": content_hash, "collection": collection},
|
||||
timeout=5,
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
return resp.json().get("exists", False)
|
||||
except Exception:
|
||||
pass
|
||||
return False
|
||||
|
||||
def score_quality(self, content: str, metadata: dict) -> int:
|
||||
"""Score document quality 0-100. Skip low-signal content."""
|
||||
score = 50 # Start neutral
|
||||
|
||||
# Length bonus (substantial content is better)
|
||||
if len(content) > 500:
|
||||
score += 15
|
||||
elif len(content) > 200:
|
||||
score += 10
|
||||
elif len(content) < 50:
|
||||
score -= 20
|
||||
|
||||
# Entity richness (more entities = more useful)
|
||||
entity_count = 0
|
||||
if metadata.get("address"):
|
||||
entity_count += 1
|
||||
if metadata.get("chain"):
|
||||
entity_count += 1
|
||||
if metadata.get("token"):
|
||||
entity_count += 1
|
||||
if metadata.get("protocol"):
|
||||
entity_count += 1
|
||||
score += entity_count * 5
|
||||
|
||||
# Source authority
|
||||
high_authority = {"etherscan", "chainabuse", "rekt", "certik", "slowmist", "peckshield"}
|
||||
if metadata.get("source", "").lower() in high_authority:
|
||||
score += 20
|
||||
|
||||
# Severity boost (critical scams are more valuable)
|
||||
if metadata.get("severity") == "critical":
|
||||
score += 15
|
||||
elif metadata.get("severity") == "high":
|
||||
score += 10
|
||||
|
||||
# Penalize duplicates of very similar content
|
||||
if metadata.get("is_variant"):
|
||||
score -= 10
|
||||
|
||||
return max(0, min(100, score))
|
||||
|
||||
def extract_entities(self, content: str) -> dict:
|
||||
"""Extract addresses, chains, tokens from content."""
|
||||
import re
|
||||
|
||||
entities = {"addresses": [], "chains": [], "tokens": [], "protocols": []}
|
||||
|
||||
# EVM addresses (0x...)
|
||||
evm_addrs = re.findall(r"0x[a-fA-F0-9]{40}", content)
|
||||
entities["addresses"].extend(evm_addrs[:10])
|
||||
|
||||
# Solana addresses (base58, 32-44 chars)
|
||||
sol_addrs = re.findall(r"[1-9A-HJ-NP-Za-km-z]{32,44}", content)
|
||||
entities["addresses"].extend([a for a in sol_addrs[:10] if a not in entities["addresses"]])
|
||||
|
||||
# Known chains
|
||||
known_chains = [
|
||||
"ethereum",
|
||||
"bsc",
|
||||
"polygon",
|
||||
"arbitrum",
|
||||
"optimism",
|
||||
"avalanche",
|
||||
"solana",
|
||||
"base",
|
||||
"fantom",
|
||||
"gnosis",
|
||||
"celo",
|
||||
"zksync",
|
||||
"linea",
|
||||
"scroll",
|
||||
"mantle",
|
||||
"sui",
|
||||
"aptos",
|
||||
"near",
|
||||
"tron",
|
||||
"bitcoin",
|
||||
]
|
||||
for chain in known_chains:
|
||||
if chain.lower() in content.lower():
|
||||
entities["chains"].append(chain)
|
||||
|
||||
# Token symbols ($TOKEN)
|
||||
tokens = re.findall(r"\$([A-Z]{2,10})", content)
|
||||
entities["tokens"].extend(tokens[:10])
|
||||
|
||||
# Known protocols
|
||||
known_protocols = [
|
||||
"uniswap",
|
||||
"aave",
|
||||
"curve",
|
||||
"balancer",
|
||||
"sushi",
|
||||
"pancake",
|
||||
"raydium",
|
||||
"jupiter",
|
||||
"orca",
|
||||
"marinade",
|
||||
"lido",
|
||||
"eigenlayer",
|
||||
"compound",
|
||||
"maker",
|
||||
"yearn",
|
||||
"convex",
|
||||
]
|
||||
for proto in known_protocols:
|
||||
if proto.lower() in content.lower():
|
||||
entities["protocols"].append(proto)
|
||||
|
||||
return entities
|
||||
|
||||
async def ingest(self, collection: str, content: str, metadata: dict, doc_id: str | None = None) -> dict[str, Any]:
|
||||
"""Run the full pipeline: dedup → quality → extract → classify → store."""
|
||||
|
||||
# 1. Hash and dedup
|
||||
content_hash = self.make_content_hash(content)
|
||||
if await self.is_duplicate(content_hash, collection):
|
||||
return {"status": "duplicate", "hash": content_hash}
|
||||
|
||||
# 2. Quality filter
|
||||
quality = self.score_quality(content, metadata)
|
||||
if quality < MIN_QUALITY_SCORE:
|
||||
return {"status": "skipped", "reason": "low_quality", "score": quality}
|
||||
|
||||
# 3. Entity extraction
|
||||
entities = self.extract_entities(content)
|
||||
metadata.update(
|
||||
{
|
||||
"entities": entities,
|
||||
"quality_score": quality,
|
||||
"content_hash": content_hash,
|
||||
"ingested_at": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
)
|
||||
|
||||
# 4. Rate limit check
|
||||
if not self._check_rate(collection):
|
||||
return {"status": "rate_limited", "collection": collection}
|
||||
|
||||
# 5. Ingest into RAG
|
||||
try:
|
||||
payload = {
|
||||
"collection": collection,
|
||||
"content": content,
|
||||
"metadata": metadata,
|
||||
}
|
||||
if doc_id:
|
||||
payload["doc_id"] = doc_id
|
||||
|
||||
resp = await self.client.post(f"{RAG_API}/ingest", json=payload, timeout=30)
|
||||
self._record_ingest(collection)
|
||||
|
||||
if resp.status_code == 200:
|
||||
result = resp.json()
|
||||
# Track in dedup set
|
||||
asyncio.create_task(self._mark_ingested(content_hash, collection))
|
||||
return {
|
||||
"status": "ingested",
|
||||
"id": result.get("id"),
|
||||
"quality": quality,
|
||||
"entities": entities,
|
||||
}
|
||||
else:
|
||||
return {"status": "error", "code": resp.status_code, "detail": resp.text[:200]}
|
||||
except Exception as e:
|
||||
return {"status": "error", "detail": str(e)[:200]}
|
||||
|
||||
async def _mark_ingested(self, content_hash: str, collection: str):
|
||||
"""Mark content hash in Redis dedup set."""
|
||||
try:
|
||||
ttl = HASH_TTL.get(collection, HASH_TTL["default"])
|
||||
await self.client.post(
|
||||
f"{RAG_API}/dedup-mark",
|
||||
json={"hash": content_hash, "collection": collection, "ttl": ttl},
|
||||
timeout=5,
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
async def ingest_batch(self, docs: list[dict]) -> dict[str, int]:
|
||||
"""Ingest multiple documents with rate limiting."""
|
||||
stats = {"ingested": 0, "duplicate": 0, "skipped": 0, "errors": 0}
|
||||
|
||||
for doc in docs:
|
||||
collection = doc.get("collection", "general")
|
||||
content = doc.get("content", "")
|
||||
metadata = doc.get("metadata", {})
|
||||
doc_id = doc.get("doc_id")
|
||||
|
||||
result = await self.ingest(collection, content, metadata, doc_id)
|
||||
status = result.get("status", "error")
|
||||
if status == "ingested":
|
||||
stats["ingested"] += 1
|
||||
elif status == "duplicate":
|
||||
stats["duplicate"] += 1
|
||||
elif status == "skipped":
|
||||
stats["skipped"] += 1
|
||||
else:
|
||||
stats["errors"] += 1
|
||||
|
||||
# Small delay between docs to avoid overwhelming embedder
|
||||
await asyncio.sleep(0.05)
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
# Feed Sources - Pull Functions
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class FeedSources:
|
||||
"""All data sources the firehose can pull from."""
|
||||
|
||||
def __init__(self, client: httpx.AsyncClient):
|
||||
self.client = client
|
||||
|
||||
# ── Hourly: News ──
|
||||
|
||||
async def pull_news_rss(self) -> list[dict]:
|
||||
"""Pull latest news from DataBus news provider (200+ RSS feeds)."""
|
||||
try:
|
||||
resp = await self.client.get(
|
||||
"http://localhost:8000/api/v1/databus/fetch/news", params={"limit": 30}, timeout=30
|
||||
)
|
||||
if resp.status_code != 200:
|
||||
return []
|
||||
data = resp.json()
|
||||
articles = data.get("articles", data.get("data", []))
|
||||
|
||||
docs = []
|
||||
for article in (articles if isinstance(articles, list) else [])[:20]:
|
||||
title = article.get("title", "")
|
||||
desc = article.get("description", article.get("summary", ""))
|
||||
if not title:
|
||||
continue
|
||||
content = f"News: {title}. {desc}"
|
||||
docs.append(
|
||||
{
|
||||
"collection": "news_articles",
|
||||
"content": content[:2000],
|
||||
"metadata": {
|
||||
"source": article.get("source", "news_rss"),
|
||||
"url": article.get("url", ""),
|
||||
"published": article.get("published_at", article.get("date", "")),
|
||||
"category": article.get("category", "crypto"),
|
||||
"title": title,
|
||||
},
|
||||
}
|
||||
)
|
||||
return docs
|
||||
except Exception as e:
|
||||
logger.warning(f"News RSS pull failed: {e}")
|
||||
return []
|
||||
|
||||
async def pull_ct_rundown(self) -> list[dict]:
|
||||
"""Pull CT Rundown stories."""
|
||||
try:
|
||||
resp = await self.client.get(
|
||||
"http://localhost:8000/api/v1/databus/fetch/ct_rundown",
|
||||
params={"limit": 10},
|
||||
timeout=30,
|
||||
)
|
||||
if resp.status_code != 200:
|
||||
return []
|
||||
data = resp.json()
|
||||
stories = data.get("stories", data.get("data", []))
|
||||
|
||||
docs = []
|
||||
for story in (stories if isinstance(stories, list) else [])[:5]:
|
||||
content = f"CT: {story.get('title', '')} - {story.get('summary', '')}"
|
||||
docs.append(
|
||||
{
|
||||
"collection": "news_articles",
|
||||
"content": content[:1500],
|
||||
"metadata": {
|
||||
"source": "ct_rundown",
|
||||
"category": "crypto_twitter",
|
||||
"handle": story.get("handle", ""),
|
||||
"engagement": story.get("engagement", 0),
|
||||
},
|
||||
}
|
||||
)
|
||||
return docs
|
||||
except Exception as e:
|
||||
logger.warning(f"CT rundown pull failed: {e}")
|
||||
return []
|
||||
|
||||
async def pull_social_sentiment(self) -> list[dict]:
|
||||
"""Pull social sentiment and scam alerts."""
|
||||
docs = []
|
||||
try:
|
||||
# Scam monitor
|
||||
resp = await self.client.get("http://localhost:8000/api/v1/databus/fetch/scam_monitor", timeout=20)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
alerts = data.get("alerts", data.get("data", []))
|
||||
for alert in (alerts if isinstance(alerts, list) else [])[:10]:
|
||||
docs.append(
|
||||
{
|
||||
"collection": "social_sentiment",
|
||||
"content": f"Scam alert: {alert.get('title', '')} - {alert.get('description', '')}",
|
||||
"metadata": {
|
||||
"source": "scam_monitor",
|
||||
"severity": alert.get("severity", "medium"),
|
||||
"token": alert.get("token_address", ""),
|
||||
"chain": alert.get("chain", ""),
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
# Social metrics
|
||||
resp2 = await self.client.get("http://localhost:8000/api/v1/databus/fetch/social_metrics", timeout=20)
|
||||
if resp2.status_code == 200:
|
||||
data2 = resp2.json()
|
||||
metrics = data2.get("metrics", data2.get("data", {}))
|
||||
if metrics:
|
||||
docs.append(
|
||||
{
|
||||
"collection": "social_sentiment",
|
||||
"content": f"Social metrics: Sentiment {metrics.get('sentiment', '?')}, "
|
||||
f"Trending: {metrics.get('trending_topics', [])}",
|
||||
"metadata": {"source": "social_metrics", "type": "daily_summary"},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Social pull failed: {e}")
|
||||
|
||||
return docs
|
||||
|
||||
# ── Daily: Scam Databases ──
|
||||
|
||||
async def pull_etherscan_labels(self) -> list[dict]:
|
||||
"""Pull etherscan labeled addresses (via existing CSV)."""
|
||||
docs = []
|
||||
csv_path = os.path.join(os.path.dirname(__file__), "..", "data", "etherscan_phish_hack.csv")
|
||||
if not os.path.exists(csv_path):
|
||||
return docs
|
||||
|
||||
import csv
|
||||
|
||||
try:
|
||||
with open(csv_path, newline="", encoding="utf-8") as f:
|
||||
rows = list(csv.DictReader(f))
|
||||
|
||||
for row in rows:
|
||||
addr = row.get("address", "").strip()
|
||||
if not addr:
|
||||
continue
|
||||
content = (
|
||||
f"Etherscan label: {addr} on {row.get('chain', 'Ethereum')}. "
|
||||
f"Tag: {row.get('name_tag', '')}. Type: {row.get('label_type', 'scam')}."
|
||||
)
|
||||
docs.append(
|
||||
{
|
||||
"collection": "known_scams",
|
||||
"content": content,
|
||||
"metadata": {
|
||||
"address": addr.lower(),
|
||||
"chain": (row.get("chain", "Ethereum") or "Ethereum").lower(),
|
||||
"label_type": row.get("label_type", "scam"),
|
||||
"source": "etherscan",
|
||||
"severity": "critical" if "phish" in row.get("label_type", "").lower() else "high",
|
||||
},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Etherscan pull failed: {e}")
|
||||
|
||||
return docs
|
||||
|
||||
async def pull_solana_scams(self) -> list[dict]:
|
||||
"""Pull Solana token registry flagged tokens."""
|
||||
docs = []
|
||||
try:
|
||||
resp = await self.client.get(
|
||||
"https://raw.githubusercontent.com/solana-labs/token-list/main/src/tokens/solana.tokenlist.json",
|
||||
timeout=30,
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
for token in data.get("tokens", []):
|
||||
tags = [t.lower() for t in token.get("tags", [])]
|
||||
if any(kw in str(tags) for kw in ["scam", "spam", "fake"]):
|
||||
docs.append(
|
||||
{
|
||||
"collection": "known_scams",
|
||||
"content": f"Solana scam token: {token.get('name', '')} ({token.get('symbol', '')}) "
|
||||
f"at {token.get('address', '')}. Tags: {tags}.",
|
||||
"metadata": {
|
||||
"address": token.get("address", ""),
|
||||
"name": token.get("name", ""),
|
||||
"symbol": token.get("symbol", ""),
|
||||
"chain": "solana",
|
||||
"source": "solana_token_registry",
|
||||
"tags": tags,
|
||||
"severity": "high",
|
||||
},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Solana pull failed: {e}")
|
||||
|
||||
return docs
|
||||
|
||||
async def pull_prediction_markets(self) -> list[dict]:
|
||||
"""Pull Polymarket prediction data for market intel."""
|
||||
docs = []
|
||||
try:
|
||||
resp = await self.client.get("http://localhost:8000/api/v1/databus/fetch/prediction_markets", timeout=20)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
markets = data.get("markets", data.get("data", []))
|
||||
for m in (markets if isinstance(markets, list) else [])[:10]:
|
||||
docs.append(
|
||||
{
|
||||
"collection": "market_intel",
|
||||
"content": f"Prediction market: {m.get('question', '')} - "
|
||||
f"YES: {m.get('yes_price', '?')} NO: {m.get('no_price', '?')}",
|
||||
"metadata": {"source": "polymarket", "type": "prediction"},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Prediction market pull failed: {e}")
|
||||
|
||||
return docs
|
||||
|
||||
# ── Weekly: Pattern Extraction ──
|
||||
|
||||
async def extract_scam_patterns(self) -> list[dict]:
|
||||
"""Extract common patterns from confirmed scam documents."""
|
||||
docs = []
|
||||
try:
|
||||
# Search for confirmed scams
|
||||
resp = await self.client.get(
|
||||
f"{RAG_API}/search",
|
||||
params={
|
||||
"q": "rug pull honeypot scam confirmed",
|
||||
"collection": "known_scams",
|
||||
"limit": 50,
|
||||
},
|
||||
timeout=30,
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
results = data.get("results", [])
|
||||
if len(results) >= 10:
|
||||
# Create a meta-pattern document
|
||||
content_parts = []
|
||||
for r in results[:20]:
|
||||
c = r.get("content", "")[:200]
|
||||
if c:
|
||||
content_parts.append(c)
|
||||
|
||||
combined = "Common scam patterns observed: " + " | ".join(content_parts)
|
||||
docs.append(
|
||||
{
|
||||
"collection": "scam_patterns",
|
||||
"content": combined[:5000],
|
||||
"metadata": {
|
||||
"source": "pattern_extraction",
|
||||
"pattern_count": len(results),
|
||||
"extracted_at": datetime.now(UTC).isoformat(),
|
||||
},
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Pattern extraction failed: {e}")
|
||||
|
||||
return docs
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
# Firehose Engine
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class FirehoseEngine:
|
||||
"""The central continuous ingestion engine."""
|
||||
|
||||
def __init__(self):
|
||||
self._running = False
|
||||
self._task: asyncio.Task | None = None
|
||||
self._client: httpx.AsyncClient | None = None
|
||||
self._pipeline: IngestionPipeline | None = None
|
||||
self._feeds: FeedSources | None = None
|
||||
self._sources: list[FeedSource] = []
|
||||
self._start_time: float = 0
|
||||
self.stats = FirehoseStats()
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
async def start(self):
|
||||
"""Start the firehose engine."""
|
||||
if self._running:
|
||||
logger.info("Firehose already running")
|
||||
return
|
||||
|
||||
self._client = httpx.AsyncClient(timeout=30, limits=httpx.Limits(max_connections=20))
|
||||
self._pipeline = IngestionPipeline(self._client)
|
||||
self._feeds = FeedSources(self._client)
|
||||
self._start_time = time.monotonic()
|
||||
|
||||
# Define all feed sources with cadences
|
||||
self._sources = [
|
||||
# ── 15-minute cadence: High urgency ──
|
||||
FeedSource(
|
||||
"ct_rundown",
|
||||
"news_articles",
|
||||
900,
|
||||
self._feeds.pull_ct_rundown,
|
||||
True,
|
||||
"CT Rundown stories",
|
||||
),
|
||||
FeedSource(
|
||||
"social_sentiment",
|
||||
"social_sentiment",
|
||||
900,
|
||||
self._feeds.pull_social_sentiment,
|
||||
True,
|
||||
"Social sentiment and scam alerts",
|
||||
),
|
||||
# ── 1-hour cadence: News ──
|
||||
FeedSource(
|
||||
"news_rss",
|
||||
"news_articles",
|
||||
3600,
|
||||
self._feeds.pull_news_rss,
|
||||
True,
|
||||
"200+ RSS crypto news feeds",
|
||||
),
|
||||
# ── 6-hour cadence: Market data ──
|
||||
FeedSource(
|
||||
"prediction_markets",
|
||||
"market_intel",
|
||||
21600,
|
||||
self._feeds.pull_prediction_markets,
|
||||
True,
|
||||
"Polymarket predictions",
|
||||
),
|
||||
# ── 24-hour cadence: Scam databases ──
|
||||
FeedSource(
|
||||
"etherscan_labels",
|
||||
"known_scams",
|
||||
86400,
|
||||
self._feeds.pull_etherscan_labels,
|
||||
True,
|
||||
"Etherscan phish/hack labeled addresses",
|
||||
),
|
||||
FeedSource(
|
||||
"solana_scams",
|
||||
"known_scams",
|
||||
86400,
|
||||
self._feeds.pull_solana_scams,
|
||||
True,
|
||||
"Solana token registry flagged tokens",
|
||||
),
|
||||
# ── 72-hour cadence: Pattern extraction ──
|
||||
FeedSource(
|
||||
"scam_patterns",
|
||||
"scam_patterns",
|
||||
259200,
|
||||
self._feeds.extract_scam_patterns,
|
||||
True,
|
||||
"Extract common patterns from confirmed scams",
|
||||
),
|
||||
]
|
||||
|
||||
self.stats.total_sources = len(self._sources)
|
||||
self._running = True
|
||||
self._task = asyncio.create_task(self._run_loop())
|
||||
logger.info(f"Firehose started with {len(self._sources)} sources")
|
||||
|
||||
async def stop(self):
|
||||
"""Stop the firehose engine."""
|
||||
self._running = False
|
||||
if self._task:
|
||||
self._task.cancel()
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
await self._task
|
||||
if self._client:
|
||||
await self._client.aclose()
|
||||
logger.info("Firehose stopped")
|
||||
|
||||
async def _run_loop(self):
|
||||
"""Main firehose loop - checks sources and runs those due."""
|
||||
logger.info("Firehose loop started")
|
||||
|
||||
while self._running:
|
||||
now = time.monotonic()
|
||||
|
||||
for source in self._sources:
|
||||
if not source.enabled:
|
||||
continue
|
||||
if now - source.last_run < source.cadence_seconds:
|
||||
continue
|
||||
|
||||
# Run this source
|
||||
source.last_run = now
|
||||
source.runs += 1
|
||||
self.stats.last_activity = now
|
||||
|
||||
try:
|
||||
logger.debug(f"Firehose: pulling {source.name}")
|
||||
docs = await source.fetch_fn()
|
||||
|
||||
if docs:
|
||||
stats = await self._pipeline.ingest_batch(docs)
|
||||
source.docs_ingested += stats["ingested"]
|
||||
source.docs_skipped += stats["duplicate"] + stats["skipped"]
|
||||
source.errors += stats["errors"]
|
||||
|
||||
self.stats.total_ingested += stats["ingested"]
|
||||
self.stats.total_skipped += stats["duplicate"] + stats["skipped"]
|
||||
self.stats.total_errors += stats["errors"]
|
||||
|
||||
logger.info(
|
||||
f"Firehose {source.name}: +{stats['ingested']} new, "
|
||||
f"{stats['duplicate']} dup, {stats['skipped']} skip, "
|
||||
f"{stats['errors']} err "
|
||||
f"(total: {source.docs_ingested})"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Firehose {source.name} failed: {e}")
|
||||
source.errors += 1
|
||||
self.stats.total_errors += 1
|
||||
|
||||
# Update source stats
|
||||
async with self._lock:
|
||||
self.stats.sources = {
|
||||
s.name: {
|
||||
"collection": s.collection,
|
||||
"cadence_min": s.cadence_seconds // 60,
|
||||
"runs": s.runs,
|
||||
"ingested": s.docs_ingested,
|
||||
"skipped": s.docs_skipped,
|
||||
"errors": s.errors,
|
||||
"last_run_ago": int(now - s.last_run) if s.last_run else -1,
|
||||
}
|
||||
for s in self._sources
|
||||
}
|
||||
|
||||
# Check every 30 seconds
|
||||
await asyncio.sleep(30)
|
||||
|
||||
async def feed_now(self, source_name: str) -> dict:
|
||||
"""Manually trigger a specific feed source immediately."""
|
||||
for source in self._sources:
|
||||
if source.name == source_name:
|
||||
try:
|
||||
docs = await source.fetch_fn()
|
||||
stats = await self._pipeline.ingest_batch(docs)
|
||||
source.runs += 1
|
||||
source.docs_ingested += stats["ingested"]
|
||||
source.last_run = time.monotonic()
|
||||
self.stats.total_ingested += stats["ingested"]
|
||||
return {"source": source_name, "docs_fetched": len(docs), **stats}
|
||||
except Exception as e:
|
||||
return {"source": source_name, "error": str(e)}
|
||||
return {"error": f"Source '{source_name}' not found"}
|
||||
|
||||
def get_status(self) -> dict:
|
||||
"""Get current firehose status."""
|
||||
return {
|
||||
"running": self._running,
|
||||
"uptime_seconds": int(time.monotonic() - self._start_time) if self._start_time else 0,
|
||||
"total_sources": self.stats.total_sources,
|
||||
"total_ingested": self.stats.total_ingested,
|
||||
"total_skipped": self.stats.total_skipped,
|
||||
"total_errors": self.stats.total_errors,
|
||||
"sources": self.stats.sources,
|
||||
}
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
# Singleton
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
|
||||
_firehose: FirehoseEngine | None = None
|
||||
|
||||
|
||||
def get_firehose() -> FirehoseEngine:
|
||||
global _firehose
|
||||
if _firehose is None:
|
||||
_firehose = FirehoseEngine()
|
||||
return _firehose
|
||||
|
|
@ -1,415 +0,0 @@
|
|||
"""
|
||||
RAG Historical Scam Ingestion - Rekt DB, Chainabuse, TRM, Elliptic
|
||||
==================================================================
|
||||
Ingests historical crypto scam and hack data into RAG collections.
|
||||
Sources: Rekt DB (3K+ DeFi hacks), Chainabuse (scam reports),
|
||||
TRM Labs (annual crime report), Elliptic (scam report),
|
||||
SlowMist (hacked archive), Immunefi (bug bounties), CertiK (audits).
|
||||
|
||||
Collections fed:
|
||||
- defi_hacks: Rekt DB, SlowMist, Rekt News
|
||||
- rug_timeline: Chainabuse, SENTINEL
|
||||
- vuln_patterns: Immunefi, CertiK
|
||||
- crime_reports: TRM, Elliptic, Chainalysis
|
||||
- forensic_reports: All exploit post-mortems
|
||||
|
||||
Runs: weekly (Sunday 2am) for historical sources.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger("rag.historical")
|
||||
|
||||
# ── Config ──
|
||||
BACKEND_URL = os.getenv("BACKEND_URL", "http://localhost:8000")
|
||||
RAG_INGEST_URL = f"{BACKEND_URL}/api/v1/rag/ingest"
|
||||
REQUEST_TIMEOUT = 30
|
||||
|
||||
# ── Source Definitions ──
|
||||
SOURCES = {
|
||||
"rekt_db": {
|
||||
"name": "Rekt Database",
|
||||
"url": "https://de.fi/rekt-database",
|
||||
"collection": "defi_hacks",
|
||||
"content_type": "scam_report",
|
||||
"description": "3,000+ DeFi hacks since 2020 - the most comprehensive exploit database",
|
||||
"cadence": "weekly",
|
||||
},
|
||||
"chainabuse": {
|
||||
"name": "Chainabuse",
|
||||
"url": "https://chainabuse.com",
|
||||
"collection": "rug_timeline",
|
||||
"content_type": "scam_report",
|
||||
"description": "Community-reported scam addresses with narratives",
|
||||
"cadence": "weekly",
|
||||
},
|
||||
"slowmist_hacked": {
|
||||
"name": "SlowMist Hacked Archive",
|
||||
"url": "https://slowmist.com/en/#hacked",
|
||||
"collection": "defi_hacks",
|
||||
"content_type": "scam_report",
|
||||
"description": "Detailed exploit analysis from SlowMist security team",
|
||||
"cadence": "weekly",
|
||||
},
|
||||
"rekt_news": {
|
||||
"name": "Rekt News",
|
||||
"url": "https://rekt.news",
|
||||
"collection": "defi_hacks",
|
||||
"content_type": "scam_report",
|
||||
"description": "In-depth DeFi exploit post-mortems",
|
||||
"cadence": "weekly",
|
||||
},
|
||||
"immunefi": {
|
||||
"name": "Immunefi Bug Bounties",
|
||||
"url": "https://immunefi.com",
|
||||
"collection": "vuln_patterns",
|
||||
"content_type": "contract_code",
|
||||
"description": "Bug bounty reports - real vulnerability patterns",
|
||||
"cadence": "weekly",
|
||||
},
|
||||
"certik": {
|
||||
"name": "CertiK Audit Findings",
|
||||
"url": "https://certik.com",
|
||||
"collection": "vuln_patterns",
|
||||
"content_type": "contract_code",
|
||||
"description": "Professional smart contract audit findings",
|
||||
"cadence": "weekly",
|
||||
},
|
||||
"trm_crime_report": {
|
||||
"name": "TRM Labs Crypto Crime Report",
|
||||
"url": "https://www.trmlabs.com/reports-and-whitepapers/2026-crypto-crime-report",
|
||||
"collection": "crime_reports",
|
||||
"content_type": "annual_report",
|
||||
"description": "Annual crypto crime typologies and trends - $158B illicit volume in 2025",
|
||||
"cadence": "yearly",
|
||||
},
|
||||
"elliptic_scams": {
|
||||
"name": "Elliptic State of Crypto Scams",
|
||||
"url": "https://info.elliptic.co/hubfs/The%20state%20of%20crypto%20scams%202025/",
|
||||
"collection": "crime_reports",
|
||||
"content_type": "annual_report",
|
||||
"description": "Annual crypto scam typologies and case studies",
|
||||
"cadence": "yearly",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# ── Scraper Functions ──
|
||||
|
||||
|
||||
async def scrape_rekt_db() -> list[dict]:
|
||||
"""
|
||||
Scrape Rekt DB for historical DeFi hacks.
|
||||
de.fi/rekt-database lists 3,000+ exploits with:
|
||||
- Project name, date, amount lost, chain, vulnerability type
|
||||
- Links to post-mortems on rekt.news
|
||||
"""
|
||||
docs = []
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
|
||||
# Try the API endpoint first
|
||||
resp = await client.get("https://de.fi/api/rekt")
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
for item in data[:500]: # Limit per run
|
||||
docs.append(
|
||||
{
|
||||
"text": f"DeFi Hack: {item.get('name', 'Unknown')} - "
|
||||
f"${item.get('amount', 0):,.0f} lost on {item.get('chain', 'Unknown')}. "
|
||||
f"Vulnerability: {item.get('vulnerability', 'Unknown')}. "
|
||||
f"Date: {item.get('date', 'Unknown')}. "
|
||||
f"Details: {item.get('description', '')}",
|
||||
"source": "rekt_db",
|
||||
"url": item.get("url", "https://de.fi/rekt-database"),
|
||||
"category": "defi_hack",
|
||||
"date": item.get("date", ""),
|
||||
"chain": item.get("chain", ""),
|
||||
"tags": [
|
||||
item.get("vulnerability", ""),
|
||||
item.get("chain", ""),
|
||||
"defi_hack",
|
||||
],
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Rekt DB scrape failed: {e}")
|
||||
|
||||
logger.info(f"Rekt DB: {len(docs)} hacks scraped")
|
||||
return docs
|
||||
|
||||
|
||||
async def scrape_chainabuse() -> list[dict]:
|
||||
"""
|
||||
Scrape Chainabuse for scam reports with addresses.
|
||||
chainabuse.com has community-reported scams with:
|
||||
- Address, chain, scam type, description, reporter
|
||||
"""
|
||||
docs = []
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
|
||||
# Chainabuse has a public reports page
|
||||
resp = await client.get(
|
||||
"https://chainabuse.com/api/reports",
|
||||
params={"limit": 200, "sort": "newest"},
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
for item in data.get("reports", []):
|
||||
address = item.get("address", "")
|
||||
docs.append(
|
||||
{
|
||||
"text": f"Scam Report: {item.get('title', 'Unknown')} - "
|
||||
f"Address {address} on {item.get('chain', 'Unknown')}. "
|
||||
f"Type: {item.get('scam_type', 'Unknown')}. "
|
||||
f"Description: {item.get('description', '')}. "
|
||||
f"Reported by: {item.get('reporter', 'Anonymous')}",
|
||||
"source": "chainabuse",
|
||||
"url": f"https://chainabuse.com/report/{item.get('id', '')}",
|
||||
"category": "scam_report",
|
||||
"date": item.get("created_at", ""),
|
||||
"chain": item.get("chain", ""),
|
||||
"tags": [
|
||||
item.get("scam_type", ""),
|
||||
item.get("chain", ""),
|
||||
"scam_report",
|
||||
],
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Chainabuse scrape failed: {e}")
|
||||
|
||||
logger.info(f"Chainabuse: {len(docs)} reports scraped")
|
||||
return docs
|
||||
|
||||
|
||||
async def scrape_slowmist() -> list[dict]:
|
||||
"""Scrape SlowMist Hacked Archive for detailed exploit analysis."""
|
||||
docs = []
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
|
||||
resp = await client.get("https://slowmist.com/en/api/hacked")
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
for item in data[:200]:
|
||||
docs.append(
|
||||
{
|
||||
"text": f"SlowMist Analysis: {item.get('title', 'Unknown')} - "
|
||||
f"${item.get('amount', 0):,.0f} lost. "
|
||||
f"Root cause: {item.get('root_cause', 'Unknown')}. "
|
||||
f"Attack vector: {item.get('attack_vector', 'Unknown')}. "
|
||||
f"Recommendations: {item.get('recommendations', '')}",
|
||||
"source": "slowmist",
|
||||
"url": item.get("url", "https://slowmist.com"),
|
||||
"category": "defi_hack",
|
||||
"date": item.get("date", ""),
|
||||
"chain": item.get("chain", ""),
|
||||
"tags": [
|
||||
item.get("root_cause", ""),
|
||||
item.get("attack_vector", ""),
|
||||
"defi_hack",
|
||||
"slowmist",
|
||||
],
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"SlowMist scrape failed: {e}")
|
||||
|
||||
logger.info(f"SlowMist: {len(docs)} analyses scraped")
|
||||
return docs
|
||||
|
||||
|
||||
async def scrape_rekt_news() -> list[dict]:
|
||||
"""Scrape Rekt News for in-depth exploit post-mortems."""
|
||||
docs = []
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
|
||||
# Rekt News has an RSS feed
|
||||
resp = await client.get("https://rekt.news/feed.xml")
|
||||
if resp.status_code == 200:
|
||||
import defusedxml.ElementTree as ET
|
||||
|
||||
root = ET.fromstring(resp.text)
|
||||
for item in root.findall(".//item")[:50]:
|
||||
title = item.findtext("title", "")
|
||||
description = item.findtext("description", "")
|
||||
link = item.findtext("link", "")
|
||||
pub_date = item.findtext("pubDate", "")
|
||||
|
||||
# Extract amount from title (e.g., "$10M Rekt")
|
||||
amount_match = re.search(r"\$(\d+(?:\.\d+)?[MB]?)", title)
|
||||
amount = amount_match.group(0) if amount_match else "Unknown"
|
||||
|
||||
docs.append(
|
||||
{
|
||||
"text": f"Rekt News: {title} - {amount} lost. {description}",
|
||||
"source": "rekt_news",
|
||||
"url": link,
|
||||
"category": "defi_hack",
|
||||
"date": pub_date,
|
||||
"tags": ["defi_hack", "rekt_news", "post_mortem"],
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Rekt News scrape failed: {e}")
|
||||
|
||||
logger.info(f"Rekt News: {len(docs)} articles scraped")
|
||||
return docs
|
||||
|
||||
|
||||
async def ingest_trm_report() -> list[dict]:
|
||||
"""
|
||||
Ingest TRM Labs 2026 Crypto Crime Report.
|
||||
Key findings: $158B illicit volume, $2.87B stolen in 150 hacks,
|
||||
Russia-linked sanctions evasion via A7A5 stablecoin.
|
||||
"""
|
||||
docs = []
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
|
||||
resp = await client.get("https://www.trmlabs.com/reports-and-whitepapers/2026-crypto-crime-report")
|
||||
if resp.status_code == 200:
|
||||
# Extract key findings from the page
|
||||
text = resp.text
|
||||
|
||||
# Parse key sections
|
||||
sections = re.split(r"<h[23][^>]*>", text)
|
||||
for section in sections:
|
||||
# Strip HTML tags
|
||||
clean = re.sub(r"<[^>]+>", " ", section)
|
||||
clean = re.sub(r"\s+", " ", clean).strip()
|
||||
if len(clean) > 100:
|
||||
docs.append(
|
||||
{
|
||||
"text": f"TRM 2026 Crypto Crime Report: {clean[:2000]}",
|
||||
"source": "trm_labs",
|
||||
"url": "https://www.trmlabs.com/reports-and-whitepapers/2026-crypto-crime-report",
|
||||
"category": "crime_report",
|
||||
"date": "2026-01-28",
|
||||
"tags": [
|
||||
"crime_report",
|
||||
"trm_labs",
|
||||
"annual_2026",
|
||||
"sanctions",
|
||||
"hacks",
|
||||
],
|
||||
}
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"TRM report ingest failed: {e}")
|
||||
|
||||
logger.info(f"TRM Report: {len(docs)} sections ingested")
|
||||
return docs
|
||||
|
||||
|
||||
# ── Master Ingestion Orchestrator ──
|
||||
|
||||
|
||||
async def ingest_historical_source(source_id: str) -> dict:
|
||||
"""Ingest a single historical source into RAG."""
|
||||
source = SOURCES.get(source_id)
|
||||
if not source:
|
||||
return {"error": f"Unknown source: {source_id}"}
|
||||
|
||||
logger.info(f"Ingesting {source['name']} → {source['collection']}")
|
||||
|
||||
# Scrape
|
||||
scrapers = {
|
||||
"rekt_db": scrape_rekt_db,
|
||||
"chainabuse": scrape_chainabuse,
|
||||
"slowmist_hacked": scrape_slowmist,
|
||||
"rekt_news": scrape_rekt_news,
|
||||
"trm_crime_report": ingest_trm_report,
|
||||
}
|
||||
|
||||
scraper = scrapers.get(source_id)
|
||||
if not scraper:
|
||||
return {"error": f"No scraper for {source_id}", "source": source_id}
|
||||
|
||||
try:
|
||||
docs = await scraper()
|
||||
except Exception as e:
|
||||
return {"error": str(e), "source": source_id, "docs_scraped": 0}
|
||||
|
||||
if not docs:
|
||||
return {"source": source_id, "docs_scraped": 0, "status": "empty"}
|
||||
|
||||
# Chunk and dedup
|
||||
from app.rag_chunking import chunk_batch
|
||||
|
||||
chunks = chunk_batch(docs, source["content_type"])
|
||||
|
||||
# Ingest into backend
|
||||
ingested = 0
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60) as client:
|
||||
# Batch in groups of 50
|
||||
for i in range(0, len(chunks), 50):
|
||||
batch = chunks[i : i + 50]
|
||||
resp = await client.post(
|
||||
RAG_INGEST_URL,
|
||||
json={
|
||||
"documents": batch,
|
||||
"collection": source["collection"],
|
||||
"source": source_id,
|
||||
"chunking": source["content_type"],
|
||||
},
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
result = resp.json()
|
||||
ingested += result.get("ingested", len(batch))
|
||||
else:
|
||||
logger.warning(f"Ingest batch failed for {source_id}: HTTP {resp.status_code}")
|
||||
except Exception as e:
|
||||
logger.error(f"Ingest HTTP failed for {source_id}: {e}")
|
||||
|
||||
return {
|
||||
"source": source_id,
|
||||
"name": source["name"],
|
||||
"collection": source["collection"],
|
||||
"docs_scraped": len(docs),
|
||||
"chunks_created": len(chunks),
|
||||
"ingested": ingested,
|
||||
"status": "ok" if ingested > 0 else "partial",
|
||||
}
|
||||
|
||||
|
||||
async def ingest_all_historical() -> dict:
|
||||
"""Run full historical ingestion across all sources."""
|
||||
results = {}
|
||||
total_ingested = 0
|
||||
|
||||
for source_id in SOURCES:
|
||||
logger.info(f"Starting historical ingest: {source_id}")
|
||||
try:
|
||||
result = await ingest_historical_source(source_id)
|
||||
results[source_id] = result
|
||||
total_ingested += result.get("ingested", 0)
|
||||
except Exception as e:
|
||||
results[source_id] = {"error": str(e), "source": source_id}
|
||||
logger.error(f"Historical ingest failed for {source_id}: {e}")
|
||||
|
||||
return {
|
||||
"status": "complete",
|
||||
"total_ingested": total_ingested,
|
||||
"sources": results,
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
# ── CLI Entry Point ──
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
if len(sys.argv) > 1:
|
||||
source = sys.argv[1]
|
||||
result = asyncio.run(ingest_historical_source(source))
|
||||
print(json.dumps(result, indent=2))
|
||||
else:
|
||||
result = asyncio.run(ingest_all_historical())
|
||||
print(json.dumps(result, indent=2))
|
||||
|
|
@ -1,387 +0,0 @@
|
|||
"""
|
||||
RAG Permanence v2 - Cloudflare R2 cold storage via REST API.
|
||||
Hot (Redis) → Warm (local 7-day cache) → Cold (R2 permanent).
|
||||
|
||||
R2 free tier: 10GB storage, 10M Class A ops/month, zero egress.
|
||||
Uses CF REST API with Bearer token - no S3 credentials needed.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# CONFIG
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
R2_ACCOUNT_ID = os.getenv("R2_ACCOUNT_ID", "8f9bd9165c1250b426c66dc1967deefd")
|
||||
R2_BUCKET = os.getenv("R2_BUCKET", "rag-backup")
|
||||
R2_API_TOKEN = os.getenv("R2_API_TOKEN", "")
|
||||
R2_BASE = f"https://api.cloudflare.com/client/v4/accounts/{R2_ACCOUNT_ID}/r2/buckets/{R2_BUCKET}"
|
||||
|
||||
LOCAL_CACHE = Path(os.getenv("RAG_LOCAL_CACHE", "/data/rag-storage"))
|
||||
LOCAL_RETENTION_DAYS = 7
|
||||
EMBEDDING_VERSION = os.getenv("EMBEDDING_MODEL_VERSION", "v1")
|
||||
|
||||
LOCAL_CACHE.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def _headers():
|
||||
return {"Authorization": f"Bearer {R2_API_TOKEN}", "Content-Type": "application/json"}
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# R2 REST API OPERATIONS
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
async def _r2_put(key: str, data: bytes, content_type: str = "application/json") -> bool:
|
||||
"""Upload an object to R2 via REST API."""
|
||||
if not R2_API_TOKEN:
|
||||
logger.error("R2_API_TOKEN not set")
|
||||
return False
|
||||
url = f"{R2_BASE}/objects/{key}"
|
||||
async with httpx.AsyncClient(timeout=30) as client:
|
||||
try:
|
||||
resp = await client.put(
|
||||
url,
|
||||
content=data,
|
||||
headers={"Authorization": f"Bearer {R2_API_TOKEN}", "Content-Type": content_type},
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
return True
|
||||
logger.error(f"R2 PUT {key}: {resp.status_code} {resp.text[:200]}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"R2 PUT {key}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
async def _r2_get(key: str) -> bytes | None:
|
||||
"""Download an object from R2 via REST API."""
|
||||
if not R2_API_TOKEN:
|
||||
return None
|
||||
url = f"{R2_BASE}/objects/{key}"
|
||||
async with httpx.AsyncClient(timeout=30) as client:
|
||||
try:
|
||||
resp = await client.get(url, headers=_headers())
|
||||
if resp.status_code == 200:
|
||||
return resp.content
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"R2 GET {key}: {e}")
|
||||
return None
|
||||
|
||||
|
||||
async def _r2_list(prefix: str = "") -> list:
|
||||
"""List objects in R2 bucket."""
|
||||
if not R2_API_TOKEN:
|
||||
return []
|
||||
url = f"{R2_BASE}/objects"
|
||||
if prefix:
|
||||
url += f"?prefix={prefix}"
|
||||
async with httpx.AsyncClient(timeout=30) as client:
|
||||
try:
|
||||
resp = await client.get(url, headers=_headers())
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
return data.get("result", [])
|
||||
return []
|
||||
except Exception as e:
|
||||
logger.error(f"R2 LIST: {e}")
|
||||
return []
|
||||
|
||||
|
||||
async def _r2_delete(key: str) -> bool:
|
||||
"""Delete an object from R2."""
|
||||
if not R2_API_TOKEN:
|
||||
return False
|
||||
url = f"{R2_BASE}/objects/{key}"
|
||||
async with httpx.AsyncClient(timeout=30) as client:
|
||||
try:
|
||||
resp = await client.delete(url, headers=_headers())
|
||||
return resp.status_code in (200, 204, 404)
|
||||
except Exception as e:
|
||||
logger.error(f"R2 DELETE {key}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
# RAG PERSISTENCE OPERATIONS
|
||||
# ═══════════════════════════════════════════════════════════════
|
||||
async def snapshot_all():
|
||||
"""Snapshot all RAG collections to R2. Returns per-collection status."""
|
||||
results = {}
|
||||
ts = datetime.now(UTC).strftime("%Y%m%d_%H%M%S")
|
||||
|
||||
# Get collections from rag_service (Redis-backed)
|
||||
try:
|
||||
from app.rag_service import COLLECTIONS, _get_redis
|
||||
|
||||
redis = await _get_redis()
|
||||
has_redis = True
|
||||
except Exception:
|
||||
COLLECTIONS = []
|
||||
has_redis = False
|
||||
|
||||
for name in COLLECTIONS:
|
||||
try:
|
||||
# Get document IDs from Redis
|
||||
if not has_redis:
|
||||
results[name] = {"collection": name, "status": "skipped", "count": 0}
|
||||
continue
|
||||
|
||||
doc_ids = await redis.smembers(f"rag:idx:{name}")
|
||||
count = len(doc_ids) if doc_ids else 0
|
||||
if count == 0:
|
||||
results[name] = {"collection": name, "status": "empty", "count": 0}
|
||||
continue
|
||||
|
||||
# Collect all documents for this collection
|
||||
# Redis stores docs as strings at rag:{collection}:{id}, NOT as hashes
|
||||
items = []
|
||||
for doc_id in doc_ids:
|
||||
try:
|
||||
raw = await redis.get(f"rag:{name}:{doc_id}")
|
||||
if raw:
|
||||
doc = json.loads(raw) if isinstance(raw, str) else raw
|
||||
items.append(doc)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Upload in chunks if payload is large (R2 REST limit ~5GB but we chunk at 5MB)
|
||||
MAX_CHUNK_SIZE = 5 * 1024 * 1024 # 5MB per chunk
|
||||
all_items = items
|
||||
chunk_idx = 0
|
||||
total_uploaded = 0
|
||||
|
||||
while all_items:
|
||||
# Build chunk - add items until we exceed size limit
|
||||
chunk_items = []
|
||||
chunk_size = 0
|
||||
while all_items and chunk_size < MAX_CHUNK_SIZE:
|
||||
item = all_items.pop(0)
|
||||
item_json = json.dumps(item, default=str)
|
||||
if chunk_size + len(item_json.encode()) > MAX_CHUNK_SIZE and chunk_items:
|
||||
# Put it back and upload this chunk
|
||||
all_items.insert(0, item)
|
||||
break
|
||||
chunk_items.append(item)
|
||||
chunk_size += len(item_json.encode())
|
||||
|
||||
payload = json.dumps(
|
||||
{
|
||||
"collection": name,
|
||||
"version": EMBEDDING_VERSION,
|
||||
"timestamp": ts,
|
||||
"chunk": chunk_idx,
|
||||
"count": len(chunk_items),
|
||||
"items": chunk_items,
|
||||
},
|
||||
default=str,
|
||||
).encode()
|
||||
|
||||
key = f"rag_snapshots/{name}/{ts}_chunk{chunk_idx}.json"
|
||||
ok = await _r2_put(key, payload)
|
||||
total_uploaded += len(chunk_items)
|
||||
|
||||
if not ok and chunk_idx == 0:
|
||||
# First chunk failed - report failure
|
||||
results[name] = {
|
||||
"collection": name,
|
||||
"status": "failed",
|
||||
"count": count,
|
||||
"key": key,
|
||||
"size_bytes": len(payload),
|
||||
"chunks_uploaded": chunk_idx,
|
||||
}
|
||||
break
|
||||
chunk_idx += 1
|
||||
|
||||
# Save latest pointer (only if at least one chunk succeeded)
|
||||
if chunk_idx > 0 or count == 0:
|
||||
await _r2_put(
|
||||
f"rag_snapshots/{name}/latest.json",
|
||||
json.dumps(
|
||||
{
|
||||
"key": f"rag_snapshots/{name}/{ts}",
|
||||
"timestamp": ts,
|
||||
"count": count,
|
||||
"version": EMBEDDING_VERSION,
|
||||
"chunks": chunk_idx,
|
||||
}
|
||||
).encode(),
|
||||
)
|
||||
|
||||
results[name] = {
|
||||
"collection": name,
|
||||
"status": "uploaded" if count > 0 else "empty",
|
||||
"count": count,
|
||||
"key": f"rag_snapshots/{name}/{ts}",
|
||||
"size_bytes": sum(len(json.dumps(i, default=str).encode()) for i in items),
|
||||
"chunks": chunk_idx,
|
||||
}
|
||||
except Exception as e:
|
||||
results[name] = {"collection": name, "error": str(e)[:200]}
|
||||
|
||||
# Save local metadata
|
||||
meta = {
|
||||
"last_snapshot": ts,
|
||||
"results": {k: v.get("status", "error") for k, v in results.items()},
|
||||
}
|
||||
(LOCAL_CACHE / "snapshot_meta.json").write_text(json.dumps(meta, indent=2))
|
||||
|
||||
return results
|
||||
|
||||
|
||||
async def restore_all():
|
||||
"""Restore all RAG collections from latest R2 snapshots."""
|
||||
# Get collections and Redis connection
|
||||
try:
|
||||
from app.rag_service import COLLECTIONS, _get_redis
|
||||
|
||||
redis = await _get_redis()
|
||||
except Exception:
|
||||
return {"error": "Cannot connect to RAG service"}
|
||||
|
||||
results = {}
|
||||
|
||||
for name in COLLECTIONS:
|
||||
try:
|
||||
# Get latest pointer
|
||||
latest_raw = await _r2_get(f"rag_snapshots/{name}/latest.json")
|
||||
if not latest_raw:
|
||||
results[name] = {"collection": name, "status": "no_snapshot"}
|
||||
continue
|
||||
|
||||
latest = json.loads(latest_raw)
|
||||
key = latest["key"]
|
||||
|
||||
# Download the snapshot
|
||||
data_raw = await _r2_get(key)
|
||||
if not data_raw:
|
||||
results[name] = {"collection": name, "status": "download_failed"}
|
||||
continue
|
||||
|
||||
snapshot = json.loads(data_raw)
|
||||
items = snapshot.get("items", [])
|
||||
snapshot_ts = latest.get("timestamp", "")
|
||||
|
||||
# Handle chunked snapshots - download all chunks
|
||||
total_chunks = latest.get("chunks", 1)
|
||||
if total_chunks > 1:
|
||||
for ci in range(1, total_chunks):
|
||||
chunk_key = f"rag_snapshots/{name}/{snapshot_ts}_chunk{ci}.json"
|
||||
chunk_raw = await _r2_get(chunk_key)
|
||||
if chunk_raw:
|
||||
chunk_data = json.loads(chunk_raw)
|
||||
items.extend(chunk_data.get("items", []))
|
||||
|
||||
# Restore to Redis: set each doc as string and add to index set
|
||||
restored = 0
|
||||
for item in items:
|
||||
try:
|
||||
doc_id = item.get("id", "")
|
||||
if not doc_id:
|
||||
continue
|
||||
# Store as JSON string (matches rag_service format)
|
||||
await redis.set(f"rag:{name}:{doc_id}", json.dumps(item, default=str))
|
||||
await redis.sadd(f"rag:idx:{name}", doc_id)
|
||||
restored += 1
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
results[name] = {
|
||||
"collection": name,
|
||||
"status": "restored",
|
||||
"count": restored,
|
||||
"total_in_snapshot": len(items),
|
||||
"snapshot_key": key,
|
||||
}
|
||||
except Exception as e:
|
||||
results[name] = {"collection": name, "error": str(e)[:200]}
|
||||
|
||||
return results
|
||||
|
||||
|
||||
async def nightly_cycle():
|
||||
"""Full nightly cycle: snapshot to R2, clean local cache."""
|
||||
snap = await snapshot_all()
|
||||
ok = sum(1 for v in snap.values() if v.get("status") in ("uploaded", "empty"))
|
||||
total = len(snap)
|
||||
|
||||
# Clean local cache older than retention period
|
||||
cleaned = 0
|
||||
cutoff = datetime.now(UTC) - timedelta(days=LOCAL_RETENTION_DAYS)
|
||||
for f in LOCAL_CACHE.glob("*.json"):
|
||||
try:
|
||||
if datetime.fromtimestamp(f.stat().st_mtime, tz=UTC) < cutoff:
|
||||
f.unlink()
|
||||
cleaned += 1
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"snapshot_results": {k: v.get("status", "error") for k, v in snap.items()},
|
||||
"collections_ok": ok,
|
||||
"collections_total": total,
|
||||
"local_cache_cleaned": cleaned,
|
||||
}
|
||||
|
||||
|
||||
async def r2_stats():
|
||||
"""Get R2 usage + local cache stats."""
|
||||
# Get collections and Redis connection
|
||||
try:
|
||||
from app.rag_service import COLLECTIONS, _get_redis
|
||||
|
||||
redis = await _get_redis()
|
||||
has_redis = True
|
||||
except Exception:
|
||||
COLLECTIONS = []
|
||||
has_redis = False
|
||||
|
||||
# R2 bucket stats
|
||||
objects = await _r2_list("rag_snapshots/")
|
||||
total_size = 0
|
||||
snapshot_count = 0
|
||||
for obj in objects:
|
||||
size = int(obj.get("size", 0))
|
||||
total_size += size
|
||||
snapshot_count += 1
|
||||
|
||||
# Local cache stats
|
||||
local_files = list(LOCAL_CACHE.glob("*.json"))
|
||||
local_size = sum(f.stat().st_size for f in local_files) if local_files else 0
|
||||
|
||||
# Collection stats (from Redis)
|
||||
coll_info = {}
|
||||
if has_redis:
|
||||
for name in COLLECTIONS:
|
||||
try:
|
||||
count = await redis.scard(f"rag:idx:{name}")
|
||||
coll_info[name] = count
|
||||
except Exception:
|
||||
coll_info[name] = 0
|
||||
|
||||
return {
|
||||
"r2": {
|
||||
"bucket": R2_BUCKET,
|
||||
"endpoint": R2_BASE,
|
||||
"snapshot_count": snapshot_count,
|
||||
"total_size_bytes": total_size,
|
||||
"total_size_mb": round(total_size / 1024 / 1024, 2),
|
||||
"has_token": bool(R2_API_TOKEN),
|
||||
},
|
||||
"local": {
|
||||
"path": str(LOCAL_CACHE),
|
||||
"file_count": len(local_files),
|
||||
"size_bytes": local_size,
|
||||
"retention_days": LOCAL_RETENTION_DAYS,
|
||||
},
|
||||
"collections": coll_info,
|
||||
"model_version": EMBEDDING_VERSION,
|
||||
}
|
||||
|
|
@ -1,253 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
RAG Tool Interface for OpenClaw Agents (2026 Agentic RAG Approach)
|
||||
- Exposes RAG as a tool callable via Hermes subagents
|
||||
- Supports reflection loops and multi-hop retrieval
|
||||
- Integrates with OpenClaw tool schema
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
# Add RAG lib to path
|
||||
sys.path.insert(0, str(Path.home() / ".hermes" / "scripts"))
|
||||
from rag_system_v2 import RMIRAGSystem
|
||||
|
||||
|
||||
class RagToolInterface:
|
||||
"""2026 Agentic RAG tool wrapper."""
|
||||
|
||||
def __init__(self):
|
||||
self.rag = RMIRAGSystem()
|
||||
self.max_iterations = 3 # Prevent infinite loops
|
||||
|
||||
def query(
|
||||
self,
|
||||
query: str,
|
||||
namespace: str | None = None,
|
||||
n_results: int = 5,
|
||||
min_confidence: float = 0.7,
|
||||
) -> dict[str, Any]:
|
||||
"""
|
||||
Query RAG and return results with reflection.
|
||||
|
||||
Implements 2026 Agentic RAG pattern:
|
||||
1. Initial retrieval
|
||||
2. Self-evaluation (confidence check)
|
||||
3. Refinement if needed
|
||||
4. Evidence-weighted synthesis
|
||||
"""
|
||||
# Step 1: Initial retrieval
|
||||
results = self.rag.query(query, n=n_results, source_filter=namespace)
|
||||
|
||||
# Convert ChromaDB distance to confidence score
|
||||
def distance_to_confidence(distance: float) -> float:
|
||||
"""ChromaDB distance → confidence (lower distance = higher confidence)."""
|
||||
if distance is None:
|
||||
return 0.5
|
||||
return max(0.0, 1.0 - distance)
|
||||
|
||||
# Convert results to standard format
|
||||
formatted_results = []
|
||||
for r in results:
|
||||
conf = distance_to_confidence(r.get("distance"))
|
||||
if conf >= min_confidence:
|
||||
formatted_results.append(
|
||||
{
|
||||
"text": r["text"],
|
||||
"metadata": r["metadata"],
|
||||
"confidence": round(conf, 3),
|
||||
"source": r["metadata"].get("source", "unknown"),
|
||||
}
|
||||
)
|
||||
|
||||
# Step 2: Self-evaluation
|
||||
confidence_score = len(formatted_results) / n_results # heuristic
|
||||
|
||||
output = {
|
||||
"query": query,
|
||||
"results": formatted_results,
|
||||
"confidence": round(confidence_score, 3),
|
||||
"n_retrieved": len(formatted_results),
|
||||
"metadata": self.rag.get_stats(),
|
||||
}
|
||||
|
||||
# Step 3: Reflection loop if confidence low
|
||||
if confidence_score < 0.6:
|
||||
output["reflection"] = {
|
||||
"step": 1,
|
||||
"original_confidence": confidence_score,
|
||||
"threshold": 0.6,
|
||||
"refined_query": f"additional context for: {query}",
|
||||
"action": "refining",
|
||||
}
|
||||
# Try refinement (single additional query)
|
||||
refined_results = self.rag.query(output["reflection"]["refined_query"], n=n_results)
|
||||
for r in refined_results:
|
||||
conf = distance_to_confidence(r.get("distance"))
|
||||
if conf >= min_confidence:
|
||||
formatted_results.append(
|
||||
{
|
||||
"text": r["text"],
|
||||
"metadata": r["metadata"],
|
||||
"confidence": round(conf, 3),
|
||||
"source": r["metadata"].get("source", "unknown"),
|
||||
}
|
||||
)
|
||||
output["results"] = formatted_results
|
||||
output["n_refined"] = len(refined_results)
|
||||
|
||||
return output
|
||||
|
||||
def multi_hop_query(self, question: str, hops: list[dict[str, str]]) -> dict[str, Any]:
|
||||
"""
|
||||
Multi-hop retrieval: chain multiple queries.
|
||||
|
||||
Example:
|
||||
hops = [
|
||||
{"purpose": "finding token patterns", "query": "rug pull token patterns"},
|
||||
{"purpose": "scam indicators", "query": " scam indicators from recent rugs"}
|
||||
]
|
||||
"""
|
||||
all_results = []
|
||||
|
||||
for hop in hops:
|
||||
hop_results = self.query(hop["query"])
|
||||
hop_result_list = [
|
||||
{"text": r["text"], "source": r["source"], "hop": hop["purpose"]} for r in hop_results["results"]
|
||||
]
|
||||
all_results.extend(hop_result_list)
|
||||
|
||||
return {
|
||||
"question": question,
|
||||
"hops": len(hops),
|
||||
"results": all_results,
|
||||
"total_results": len(all_results),
|
||||
}
|
||||
|
||||
def reflection_loop(self, original_query: str, max_iterations: int = 3) -> dict[str, Any]:
|
||||
"""
|
||||
Self-correcting retrieval with iteration budgets.
|
||||
|
||||
Stops when:
|
||||
- Confidence threshold reached (0.8)
|
||||
- Max iterations exceeded
|
||||
- No new results found
|
||||
"""
|
||||
iterations = []
|
||||
current_query = original_query
|
||||
all_results = []
|
||||
|
||||
for i in range(max_iterations):
|
||||
result = self.query(current_query, n_results=5)
|
||||
|
||||
iterations.append(
|
||||
{
|
||||
"step": i,
|
||||
"query": current_query,
|
||||
"results_count": len(result["results"]),
|
||||
"confidence": result["confidence"],
|
||||
}
|
||||
)
|
||||
|
||||
# Add new results
|
||||
all_results.extend(result["results"])
|
||||
|
||||
# Check stop conditions
|
||||
if result["confidence"] >= 0.8:
|
||||
iterations[-1]["reason"] = "confidence_threshold_reached"
|
||||
break
|
||||
|
||||
if len(result["results"]) == 0:
|
||||
iterations[-1]["reason"] = "no_new_results"
|
||||
break
|
||||
|
||||
# Refine query for next iteration
|
||||
current_query = f"details about: {current_query}"
|
||||
|
||||
return {
|
||||
"original_query": original_query,
|
||||
"iterations": iterations,
|
||||
"final_results": all_results,
|
||||
}
|
||||
|
||||
def get_tool_schema(self) -> dict[str, Any]:
|
||||
"""Return OpenAI-compatible tool schema for agents."""
|
||||
return {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "rag_query",
|
||||
"description": (
|
||||
"Query RugMunch Intelligence RAG for crypto security patterns, "
|
||||
"rug pull indicators, wallet risk scores, and market intelligence."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": 'Search query for RAG (e.g., "rug pull patterns", "wallet risk indicators")',
|
||||
},
|
||||
"namespace": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"scan_results",
|
||||
"alerts",
|
||||
"content",
|
||||
"market_data",
|
||||
"onchain",
|
||||
"social_feed",
|
||||
"news",
|
||||
],
|
||||
"default": None,
|
||||
"description": "Optional source filter to narrow search",
|
||||
},
|
||||
"n_results": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"maximum": 20,
|
||||
"default": 5,
|
||||
"description": "Number of results to retrieve",
|
||||
},
|
||||
},
|
||||
"required": ["query"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# ============================================================
|
||||
# CLI INTERFACE
|
||||
# ============================================================
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description="RAG Tool Interface for OpenClaw Agents")
|
||||
parser.add_argument("command", choices=["query", "reflection", "multi-hop", "schema"])
|
||||
parser.add_argument("--query", type=str, help="Query text")
|
||||
parser.add_argument("--namespace", type=str, help="Source namespace filter")
|
||||
parser.add_argument("--n", type=int, default=5, help="Number of results")
|
||||
parser.add_argument("--hops", type=str, help="Multi-hop config (JSON)")
|
||||
parser.add_argument("--max-iter", type=int, default=3, help="Max reflection iterations")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
rag = RagToolInterface()
|
||||
|
||||
if args.command == "query":
|
||||
result = rag.query(args.query, namespace=args.namespace, n_results=args.n)
|
||||
print(json.dumps(result, indent=2))
|
||||
|
||||
elif args.command == "reflection":
|
||||
result = rag.reflection_loop(args.query, max_iterations=args.max_iter)
|
||||
print(json.dumps(result, indent=2))
|
||||
|
||||
elif args.command == "multi-hop":
|
||||
hops = json.loads(args.hops)
|
||||
result = rag.multi_hop_query(args.query, hops)
|
||||
print(json.dumps(result, indent=2))
|
||||
|
||||
elif args.command == "schema":
|
||||
print(json.dumps(rag.get_tool_schema(), indent=2))
|
||||
|
|
@ -1,179 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
"""#20 - Cross-Chain Portfolio Rebalance Bot. Users set automated rebalancing rules.
|
||||
Executes via x402, monitors via DataBus. "Keep 50% USDC on Solana, 30% ETH on Arbitrum." """
|
||||
|
||||
import os
|
||||
from datetime import UTC, datetime
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, BackgroundTasks, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
router = APIRouter(prefix="/api/v1/rebalance", tags=["portfolio-rebalance"])
|
||||
|
||||
DATABUS = os.environ.get("DATABUS_URL", "http://localhost:8000/api/v1/databus")
|
||||
BACKEND = os.environ.get("BACKEND_URL", "http://localhost:8000")
|
||||
|
||||
# In-memory store for rebalancing rules (Redis/DB in prod)
|
||||
_rebalance_rules: dict[str, dict] = {}
|
||||
|
||||
|
||||
class RebalanceRule(BaseModel):
|
||||
user_id: str
|
||||
name: str
|
||||
allocations: list[dict] # [{"chain": "solana", "token": "USDC", "target_pct": 50}, ...]
|
||||
rebalance_threshold_pct: float = 5.0 # Rebalance if any allocation off by >5%
|
||||
max_slippage_pct: float = 1.0
|
||||
enabled: bool = True
|
||||
|
||||
|
||||
@router.post("/rules")
|
||||
async def create_rebalance_rule(rule: RebalanceRule):
|
||||
"""Create an automated portfolio rebalancing rule."""
|
||||
rule_id = f"{rule.user_id}:{rule.name}"
|
||||
_rebalance_rules[rule_id] = {
|
||||
"id": rule_id,
|
||||
"user_id": rule.user_id,
|
||||
"name": rule.name,
|
||||
"allocations": rule.allocations,
|
||||
"rebalance_threshold_pct": rule.rebalance_threshold_pct,
|
||||
"max_slippage_pct": rule.max_slippage_pct,
|
||||
"enabled": rule.enabled,
|
||||
"created_at": datetime.now(UTC).isoformat(),
|
||||
"last_rebalanced": None,
|
||||
}
|
||||
return {"id": rule_id, "status": "created", "allocations": rule.allocations}
|
||||
|
||||
|
||||
@router.get("/rules/{user_id}")
|
||||
async def list_rules(user_id: str):
|
||||
"""List all rebalancing rules for a user."""
|
||||
user_rules = [r for rid, r in _rebalance_rules.items() if r["user_id"] == user_id]
|
||||
return {"rules": user_rules, "count": len(user_rules)}
|
||||
|
||||
|
||||
@router.delete("/rules/{rule_id}")
|
||||
async def delete_rule(rule_id: str):
|
||||
"""Delete a rebalancing rule."""
|
||||
_rebalance_rules.pop(rule_id, None)
|
||||
return {"id": rule_id, "status": "deleted"}
|
||||
|
||||
|
||||
@router.get("/simulate/{user_id}")
|
||||
async def simulate_rebalance(user_id: str):
|
||||
"""Simulate rebalancing - check what would change without executing."""
|
||||
user_rules = [r for rid, r in _rebalance_rules.items() if r["user_id"] == user_id and r["enabled"]]
|
||||
if not user_rules:
|
||||
return {"simulations": [], "note": "No enabled rules found"}
|
||||
|
||||
simulations = []
|
||||
for rule in user_rules[:5]:
|
||||
current: list[dict] = []
|
||||
# Fetch current balances for each allocation
|
||||
async with httpx.AsyncClient(timeout=15) as client:
|
||||
for alloc in rule["allocations"]:
|
||||
try:
|
||||
resp = await client.get(
|
||||
f"{DATABUS}/{alloc['chain']}/balance/{user_id}", params={"token": alloc.get("token", "native")}
|
||||
)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json().get("data", {})
|
||||
current.append(
|
||||
{
|
||||
"chain": alloc["chain"],
|
||||
"token": alloc.get("token", "native"),
|
||||
"balance_usd": data.get("value_usd", 0) or 0,
|
||||
"current_pct": 0,
|
||||
"target_pct": alloc["target_pct"],
|
||||
"difference_pct": 0,
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
current.append(
|
||||
{
|
||||
"chain": alloc["chain"],
|
||||
"token": alloc.get("token", "native"),
|
||||
"balance_usd": 0,
|
||||
"target_pct": alloc["target_pct"],
|
||||
"error": "unavailable",
|
||||
}
|
||||
)
|
||||
|
||||
# Calculate percentages and differences
|
||||
total = sum(a["balance_usd"] for a in current) or 1
|
||||
needs_rebalance = False
|
||||
for a in current:
|
||||
a["current_pct"] = round((a["balance_usd"] / total) * 100, 1)
|
||||
a["difference_pct"] = round(a["current_pct"] - a["target_pct"], 1)
|
||||
if abs(a["difference_pct"]) > rule["rebalance_threshold_pct"]:
|
||||
needs_rebalance = True
|
||||
|
||||
simulations.append(
|
||||
{
|
||||
"rule": rule["name"],
|
||||
"needs_rebalance": needs_rebalance,
|
||||
"total_value_usd": round(total, 2),
|
||||
"current_allocations": current,
|
||||
"rebalance_threshold_pct": rule["rebalance_threshold_pct"],
|
||||
}
|
||||
)
|
||||
|
||||
return {"simulations": simulations}
|
||||
|
||||
|
||||
@router.post("/execute/{rule_id}")
|
||||
async def execute_rebalance(rule_id: str, background_tasks: BackgroundTasks):
|
||||
"""Execute a rebalancing operation (simulation only - no real execution without x402)."""
|
||||
rule = _rebalance_rules.get(rule_id)
|
||||
if not rule:
|
||||
raise HTTPException(404, "Rule not found")
|
||||
|
||||
if not rule["enabled"]:
|
||||
raise HTTPException(400, "Rule is disabled")
|
||||
|
||||
# Simulation mode - return what WOULD be executed
|
||||
async with httpx.AsyncClient(timeout=15) as client:
|
||||
trades_needed: list[dict] = []
|
||||
current_values: list[dict] = []
|
||||
for alloc in rule["allocations"]:
|
||||
try:
|
||||
resp = await client.get(f"{DATABUS}/{alloc['chain']}/balance/{rule['user_id']}")
|
||||
if resp.status_code == 200:
|
||||
data = resp.json().get("data", {})
|
||||
current_values.append(
|
||||
{
|
||||
"chain": alloc["chain"],
|
||||
"token": alloc.get("token", "native"),
|
||||
"value_usd": data.get("value_usd", 0) or 0,
|
||||
"target_pct": alloc["target_pct"],
|
||||
}
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
total = sum(a["value_usd"] for a in current_values) or 1
|
||||
for a in current_values:
|
||||
current_pct = (a["value_usd"] / total) * 100
|
||||
diff_pct = current_pct - a["target_pct"]
|
||||
diff_usd = total * (diff_pct / 100)
|
||||
if abs(diff_pct) > rule["rebalance_threshold_pct"]:
|
||||
trades_needed.append(
|
||||
{
|
||||
"chain": a["chain"],
|
||||
"token": a["token"],
|
||||
"action": "SELL" if diff_pct > 0 else "BUY",
|
||||
"amount_usd": round(abs(diff_usd), 2),
|
||||
"current_pct": round(current_pct, 1),
|
||||
"target_pct": a["target_pct"],
|
||||
}
|
||||
)
|
||||
|
||||
rule["last_rebalanced"] = datetime.now(UTC).isoformat()
|
||||
|
||||
return {
|
||||
"rule_id": rule_id,
|
||||
"status": "simulated",
|
||||
"total_value_usd": round(total, 2),
|
||||
"trades_needed": trades_needed,
|
||||
"note": "Simulation only. Real execution requires x402 payment authorization.",
|
||||
}
|
||||
|
|
@ -1,353 +0,0 @@
|
|||
"""
|
||||
RMI TUI Dashboard - Terminal User Interface
|
||||
===========================================
|
||||
|
||||
Interactive CLI dashboard for RMI platform.
|
||||
Features:
|
||||
- System status display
|
||||
- Menu-driven navigation
|
||||
- Quick commands for common actions
|
||||
- Live backend status monitoring
|
||||
"""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from app.core.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
# Terminal colors
|
||||
class Colors:
|
||||
RED = "\033[0;31m"
|
||||
GREEN = "\033[0;32m"
|
||||
YELLOW = "\033[1;33m"
|
||||
BLUE = "\033[0;34m"
|
||||
PURPLE = "\033[0;35m"
|
||||
CYAN = "\033[0;36m"
|
||||
WHITE = "\033[0;37m"
|
||||
BOLD = "\033[1m"
|
||||
RESET = "\033[0m"
|
||||
|
||||
|
||||
# ─── UI UTILITIES ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
def clear_screen():
|
||||
"""Clear terminal screen."""
|
||||
os.system("clear" if os.name == "posix" else "cls")
|
||||
|
||||
|
||||
def print_header(title: str):
|
||||
"""Print centered header."""
|
||||
width = 60
|
||||
logger.info(f"\n{Colors.BLUE}{'=' * width}{Colors.RESET}")
|
||||
print(
|
||||
f"{Colors.BLUE}║{Colors.RESET}{Colors.BOLD}{title.center(width - 2)}{Colors.RESET}{Colors.BLUE}║{Colors.RESET}"
|
||||
)
|
||||
logger.info(f"{Colors.BLUE}{'=' * width}{Colors.RESET}\n")
|
||||
def print_box(title: str, content: str, color=Colors.WHITE):
|
||||
"""Print content in a box."""
|
||||
lines = content.split("\n")
|
||||
max_len = max(len(l) for l in lines) if lines else 0 # noqa: E741
|
||||
width = max_len + 4
|
||||
|
||||
logger.info(f"\n{color}┌{'─' * width}┐{Colors.RESET}")
|
||||
logger.info(f"{color}│{Colors.RESET} {Colors.BOLD}{title.center(width - 2)}{Colors.RESET} {color}│{Colors.RESET}")
|
||||
logger.info(f"{color}├{'─' * width}┤{Colors.RESET}")
|
||||
for line in lines:
|
||||
logger.info(f"{color}│{Colors.RESET} {line.ljust(width - 2)} {color}│{Colors.RESET}")
|
||||
logger.info(f"{color}└{'─' * width}┘{Colors.RESET}\n")
|
||||
# ─── STATUS MONITOR ────────────────────────────────────────────────
|
||||
|
||||
|
||||
def get_backend_status(url: str = "http://127.0.0.1:8010") -> dict[str, Any]:
|
||||
"""Get backend health status."""
|
||||
try:
|
||||
response = httpx.get(f"{url}/health", timeout=2)
|
||||
if response.status_code == 200:
|
||||
return response.json()
|
||||
except Exception:
|
||||
pass
|
||||
return {"error": "Backend not reachable"}
|
||||
|
||||
|
||||
def get_redis_status() -> dict[str, Any]:
|
||||
"""Get Redis connection status."""
|
||||
try:
|
||||
import redis
|
||||
|
||||
r = redis.Redis(host="localhost", port=6379, db=0, socket_timeout=2)
|
||||
if r.ping():
|
||||
info = r.info()
|
||||
return {
|
||||
"connected": True,
|
||||
"version": info.get("redis_version", "unknown"),
|
||||
"memory_used": info.get("used_memory_human", "unknown"),
|
||||
"connections": info.get("connected_clients", 0),
|
||||
}
|
||||
except Exception:
|
||||
pass
|
||||
return {"error": "Redis not reachable"}
|
||||
|
||||
|
||||
# ─── DASHBOARD UI ──────────────────────────────────────────────────
|
||||
|
||||
|
||||
class Dashboard:
|
||||
"""Interactive TUI Dashboard."""
|
||||
|
||||
def __init__(self):
|
||||
self.running = True
|
||||
self.current_view = "main"
|
||||
|
||||
def display_main(self):
|
||||
"""Display main menu."""
|
||||
clear_screen()
|
||||
|
||||
# Header
|
||||
logger.info(f"{Colors.CYAN}{Colors.BOLD}")
|
||||
logger.info("╔════════════════════════════════════════════════════════════╗")
|
||||
logger.info("║ RMI INTELLIGENCE PLATFORM v2.0.0 ║")
|
||||
logger.info("╚════════════════════════════════════════════════════════════╝")
|
||||
logger.info(f"{Colors.RESET}")
|
||||
# Status
|
||||
backend = get_backend_status()
|
||||
redis = get_redis_status()
|
||||
|
||||
backend_status = (
|
||||
f"{Colors.GREEN}● Online{Colors.RESET}"
|
||||
if backend.get("status") == "ok"
|
||||
else f"{Colors.RED}● Offline{Colors.RESET}"
|
||||
)
|
||||
redis_status = (
|
||||
f"{Colors.GREEN}● Connected{Colors.RESET}"
|
||||
if redis.get("connected")
|
||||
else f"{Colors.RED}● Disconnected{Colors.RESET}"
|
||||
)
|
||||
|
||||
logger.info(f"{Colors.YELLOW}System Status:{Colors.RESET}")
|
||||
logger.info(f" Backend: {backend_status} | Redis: {redis_status}")
|
||||
if backend.get("status") == "ok":
|
||||
logger.info(f" Service: {backend.get('service', 'unknown')}")
|
||||
logger.info(f" Version: {backend.get('version', 'unknown')}")
|
||||
logger.info(f" Timestamp: {backend.get('timestamp', 'N/A')}")
|
||||
logger.info(f"\n{Colors.YELLOW}Available Ports:{Colors.RESET}")
|
||||
logger.info(" 8010 - Main Backend API")
|
||||
logger.info(" 6379 - Redis Server")
|
||||
logger.info(" 22 - SSH (if enabled)")
|
||||
# Features
|
||||
logger.info(f"\n{Colors.YELLOW}Built-in Features:{Colors.RESET}")
|
||||
features = [
|
||||
("[1] Server Control", "Start/stop/restart backend server"),
|
||||
("[2] Auth System", "User authentication & OAuth management"),
|
||||
("[3] Security Intel", "Threat intel & contract scanning"),
|
||||
("[4] x402 Micropayments", "Micropayment enforcement & tools"),
|
||||
("[5] Chat Interface", "Direct chat with RMI assistant"),
|
||||
("[6] Dashboard TUI", "Interactive terminal dashboard"),
|
||||
("[7] Network Status", "Check all service connectivity"),
|
||||
("[0] Quit", "Exit to shell"),
|
||||
]
|
||||
|
||||
for key, (name, desc) in enumerate(features):
|
||||
logger.info(f" {Colors.GREEN}{key}{Colors.RESET}. {Colors.CYAN}{name:<20}{Colors.RESET} - {desc}")
|
||||
logger.info(f"\n{Colors.PURPLE}Use 'help' for more information{Colors.RESET}\n")
|
||||
def display_server_control(self):
|
||||
"""Display server control menu."""
|
||||
print_header("Server Control")
|
||||
|
||||
pid = self._get_backend_pid()
|
||||
status = "Running" if pid else "Stopped"
|
||||
|
||||
logger.info(f" Backend Status: {Colors.GREEN if pid else Colors.RED}{status}{Colors.RESET}")
|
||||
if pid:
|
||||
logger.info(f" PID: {pid}")
|
||||
logger.info("\n Actions:")
|
||||
logger.info(f" {Colors.GREEN}1{Colors.RESET}. Start Backend")
|
||||
logger.info(f" {Colors.GREEN}2{Colors.RESET}. Stop Backend")
|
||||
logger.info(f" {Colors.GREEN}3{Colors.RESET}. Restart Backend")
|
||||
logger.info(f" {Colors.GREEN}0{Colors.RESET}. Back")
|
||||
choice = input("\n Select: ").strip()
|
||||
|
||||
if choice == "1":
|
||||
self._start_backend()
|
||||
elif choice == "2":
|
||||
self._stop_backend()
|
||||
elif choice == "3":
|
||||
self._restart_backend()
|
||||
|
||||
def _get_backend_pid(self) -> int | None:
|
||||
"""Get backend process ID."""
|
||||
try:
|
||||
result = subprocess.run(["pgrep", "-f", "uvicorn main:app"], capture_output=True, text=True, timeout=5)
|
||||
if result.stdout.strip():
|
||||
return int(result.stdout.strip())
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
def _start_backend(self):
|
||||
"""Start backend server."""
|
||||
logger.info("\n{Colors.YELLOW}Starting backend...{Colors.RESET}")
|
||||
try:
|
||||
subprocess.Popen(
|
||||
["python3", "-m", "uvicorn", "main:app", "--host", "127.0.0.1", "--port", "8010"],
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
start_new_session=True,
|
||||
)
|
||||
time.sleep(3)
|
||||
logger.info(f"{Colors.GREEN}Backend started!{Colors.RESET}")
|
||||
except Exception as e:
|
||||
logger.warning(f"{Colors.RED}Error starting backend: {e}{Colors.RESET}")
|
||||
def _stop_backend(self):
|
||||
"""Stop backend server."""
|
||||
pid = self._get_backend_pid()
|
||||
if pid:
|
||||
try:
|
||||
os.kill(pid, 15)
|
||||
logger.info(f"{Colors.GREEN}Backend stopped (PID: {pid}){Colors.RESET}")
|
||||
except Exception as e:
|
||||
logger.warning(f"{Colors.RED}Error stopping backend: {e}{Colors.RESET}")
|
||||
def _restart_backend(self):
|
||||
"""Restart backend server."""
|
||||
self._stop_backend()
|
||||
time.sleep(1)
|
||||
self._start_backend()
|
||||
|
||||
def display_auth_system(self):
|
||||
"""Display auth system info."""
|
||||
print_header("Authentication System")
|
||||
|
||||
logger.info(f" {Colors.CYAN}OAuth Providers:{Colors.RESET}")
|
||||
logger.info(" • Email + Password")
|
||||
logger.info(" • Google")
|
||||
logger.info(" • Telegram")
|
||||
logger.info(" • GitHub")
|
||||
logger.info(" • Wallet Signature (EIP-4361)")
|
||||
logger.info(f"\n {Colors.CYAN}Features:{Colors.RESET}")
|
||||
logger.info(" • Session management (JWT)")
|
||||
logger.info(" • Rate limiting per user")
|
||||
logger.info(" • x402 micropayment enforcement")
|
||||
logger.info(f"\n {Colors.CYAN}Endpoints:{Colors.RESET}")
|
||||
logger.info(" POST /api/v1/auth/register - Register new user")
|
||||
logger.info(" POST /api/v1/auth/login - Login user")
|
||||
logger.info(" GET /api/v1/auth/status - Get current session")
|
||||
input(f"\n{Colors.PURPLE}Press Enter to return...{Colors.RESET}")
|
||||
|
||||
def display_security_intel(self):
|
||||
"""Display security intel info."""
|
||||
print_header("Security Intelligence")
|
||||
|
||||
logger.info(f" {Colors.CYAN}Modules:{Colors.RESET}")
|
||||
logger.info(" • Slither - Static analysis")
|
||||
logger.info(" • Mythril - Symbolic execution")
|
||||
logger.info(" • OFAC Sanctions - Blocklist")
|
||||
logger.info(" • Threat Intel - Risk scoring")
|
||||
logger.info(f"\n {Colors.CYAN}Contract Scanning:{Colors.RESET}")
|
||||
logger.info(" • Reentrancy detection")
|
||||
logger.info(" • Integer overflow/underflow")
|
||||
logger.info(" • Unchecked calls")
|
||||
logger.info(" • DAO patterns")
|
||||
input(f"\n{Colors.PURPLE}Press Enter to return...{Colors.RESET}")
|
||||
|
||||
def display_x402_menu(self):
|
||||
"""Display x402 micropayments info."""
|
||||
print_header("x402 Micropayments")
|
||||
|
||||
logger.info(f" {Colors.CYAN}Features:{Colors.RESET}")
|
||||
logger.info(" • Micropayment enforcement")
|
||||
logger.info(" • Rate-based pricing")
|
||||
logger.info(" • Forensic analysis")
|
||||
logger.info(" • Redis-backed tracking")
|
||||
logger.info(f"\n {Colors.CYAN}Endpoints:{Colors.RESET}")
|
||||
logger.info(" GET /api/v1/x402/tools - Available tools catalog")
|
||||
logger.info(" GET /api/v1/x402/status - Payment status")
|
||||
logger.info(" POST /api/v1/x402/enforce - Enforce micropayment")
|
||||
input(f"\n{Colors.PURPLE}Press Enter to return...{Colors.RESET}")
|
||||
|
||||
def display_chat(self):
|
||||
"""Launch chat interface."""
|
||||
print_header("Chat Interface")
|
||||
logger.info("\n{Colors.YELLOW}Direct chat with RMI assistant{Colors.RESET}")
|
||||
logger.info("\n{Colors.RED}Note:{Colors.RESET} This launches the system shell")
|
||||
input(f"\n{Colors.PURPLE}Press Enter to return...{Colors.RESET}")
|
||||
|
||||
def display_network_status(self):
|
||||
"""Display network connectivity status."""
|
||||
print_header("Network Status")
|
||||
|
||||
tests = [
|
||||
("Local Backend", "http://127.0.0.1:8010", "/health"),
|
||||
("Redis", "localhost", None),
|
||||
("Ethereum RPC", "https://ethereum-rpc.publicnode.com", None),
|
||||
("Base RPC", "https://base-rpc.publicnode.com", None),
|
||||
]
|
||||
|
||||
for name, url, path in tests:
|
||||
try:
|
||||
full_url = f"{url}{path}" if path else url
|
||||
response = httpx.get(full_url, timeout=3)
|
||||
status = f"{Colors.GREEN}✓ OK{Colors.RESET} ({response.status_code})"
|
||||
except Exception:
|
||||
status = f"{Colors.RED}✗ Failed{Colors.RESET}"
|
||||
|
||||
logger.info(f" {name:<20} {status}")
|
||||
logger.info()
|
||||
input(f"{Colors.PURPLE}Press Enter to return...{Colors.RESET}")
|
||||
|
||||
def run(self):
|
||||
"""Main dashboard loop."""
|
||||
while self.running:
|
||||
self.display_main()
|
||||
|
||||
choice = input("Select an option: ").strip().lower()
|
||||
|
||||
if choice == "0":
|
||||
self.running = False
|
||||
elif choice == "1":
|
||||
self.display_server_control()
|
||||
elif choice == "2":
|
||||
self.display_auth_system()
|
||||
elif choice == "3":
|
||||
self.display_security_intel()
|
||||
elif choice == "4":
|
||||
self.display_x402_menu()
|
||||
elif choice == "5":
|
||||
self.display_chat()
|
||||
elif choice == "6":
|
||||
logger.info("{TUI already running}")
|
||||
elif choice == "7":
|
||||
self.display_network_status()
|
||||
elif choice in ["help", "?"]:
|
||||
logger.info(f"{Colors.YELLOW}Commands:{Colors.RESET}")
|
||||
logger.info(" help - Show this help")
|
||||
logger.info(" clear - Clear screen")
|
||||
logger.info(" quit - Exit dashboard")
|
||||
elif choice == "clear":
|
||||
clear_screen()
|
||||
elif choice in ["quit", "exit"]:
|
||||
self.running = False
|
||||
else:
|
||||
logger.info(f"{Colors.RED}Invalid option{Colors.RESET}")
|
||||
# ─── SINGLETON LAUNCHER ────────────────────────────────────────────
|
||||
|
||||
|
||||
def launch_dashboard():
|
||||
"""Launch the interactive dashboard."""
|
||||
try:
|
||||
dashboard = Dashboard()
|
||||
dashboard.run()
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.warning(f"{Colors.RED}Dashboard error: {e}{Colors.RESET}")
|
||||
# ─── CLI ENTRY POINT ───────────────────────────────────────────────
|
||||
|
||||
if __name__ == "__main__":
|
||||
launch_dashboard()
|
||||
|
|
@ -1,543 +0,0 @@
|
|||
"""
|
||||
Multi-Layer Cache Manager - L1 In-Memory + L2 Redis with Smart TTLs.
|
||||
|
||||
Provides a two-level caching strategy:
|
||||
L1: In-memory dict (fast, per-process, bounded size)
|
||||
L2: Redis (shared across processes/workers, persistent optional)
|
||||
|
||||
If REDIS_URL env var is not set, falls back to L1-only mode.
|
||||
Key format: {data_type}:{chain}:{address} for easy pattern invalidation.
|
||||
|
||||
Smart TTLs per data type (seconds):
|
||||
token_meta=86400 (24h), deployer=0 (permanent), holders=300 (5min),
|
||||
price=10, liquidity=30, social=900 (15min), rpc=60, block=15.
|
||||
|
||||
Depends on: redis (>=5.0) for L2; falls back gracefully if not installed.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from collections import OrderedDict
|
||||
from collections.abc import Awaitable, Callable
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Smart TTLs per Data Type ───────────────────────────────────────────────
|
||||
|
||||
# fmt: off
|
||||
SMART_TTLS: dict[str, int] = {
|
||||
"token_meta": 86400, # 24 hours - token name/symbol/logo rarely change
|
||||
"deployer": 0, # Permanent (0 = never expire in L1, Redis TTL omitted)
|
||||
"contract_code": 3600, # 1 hour - contract code is immutable but may redeploy
|
||||
"holders": 300, # 5 minutes - holder count changes frequently
|
||||
"holder_list": 120, # 2 minutes - individual holder data can shift
|
||||
"price": 10, # 10 seconds - price is highly volatile
|
||||
"liquidity": 30, # 30 seconds - liquidity changes but slower than price
|
||||
"volume": 30, # 30 seconds - same reasoning
|
||||
"market_cap": 60, # 1 minute - derived from price x supply
|
||||
"social": 900, # 15 minutes - social media data updates slowly
|
||||
"metadata": 3600, # 1 hour - token metadata is semi-static
|
||||
"rpc": 60, # 1 minute - generic RPC responses
|
||||
"rpc_account": 30, # 30 seconds - account state (balance, etc.)
|
||||
"rpc_block": 15, # 15 seconds - block data is fresh
|
||||
"rpc_tx": 60, # 1 minute - confirmed transactions
|
||||
"scanner": 300, # 5 minutes - scanner results (risk scores, etc.)
|
||||
"entity": 3600, # 1 hour - entity labeling by address
|
||||
"news": 600, # 10 minutes - news articles
|
||||
"trending": 300, # 5 minutes - trending tokens
|
||||
"default": 120, # 2 minutes - catch-all
|
||||
}
|
||||
# fmt: on
|
||||
|
||||
|
||||
# ── Data Classes ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class CacheEntry:
|
||||
"""A cached value with metadata."""
|
||||
|
||||
value: Any
|
||||
created_at: float = field(default_factory=time.monotonic)
|
||||
ttl: float = 0.0 # 0 = permanent (no expiry)
|
||||
access_count: int = 0
|
||||
size_bytes: int = 0
|
||||
|
||||
@property
|
||||
def is_expired(self) -> bool:
|
||||
if self.ttl <= 0:
|
||||
return False
|
||||
return time.monotonic() - self.created_at > self.ttl
|
||||
|
||||
@property
|
||||
def age_seconds(self) -> float:
|
||||
return time.monotonic() - self.created_at
|
||||
|
||||
|
||||
@dataclass
|
||||
class CacheStats:
|
||||
"""Aggregate cache performance stats."""
|
||||
|
||||
hit_count: int = 0
|
||||
miss_count: int = 0
|
||||
total_calls: int = 0
|
||||
l1_size: int = 0
|
||||
l2_size: int = 0
|
||||
l1_hits: int = 0
|
||||
l2_hits: int = 0
|
||||
invalidations: int = 0
|
||||
|
||||
@property
|
||||
def hit_rate(self) -> float:
|
||||
if self.total_calls == 0:
|
||||
return 0.0
|
||||
return self.hit_count / self.total_calls
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"hit_count": self.hit_count,
|
||||
"miss_count": self.miss_count,
|
||||
"total_calls": self.total_calls,
|
||||
"hit_rate": round(self.hit_rate * 100, 1),
|
||||
"l1_size": self.l1_size,
|
||||
"l2_size": self.l2_size,
|
||||
"l1_hits": self.l1_hits,
|
||||
"l2_hits": self.l2_hits,
|
||||
"invalidations": self.invalidations,
|
||||
}
|
||||
|
||||
|
||||
# ── Cache Key Builder ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def build_key(data_type: str, chain: str, address: str) -> str:
|
||||
"""Build a standardized cache key.
|
||||
|
||||
Format: {data_type}:{chain}:{address}
|
||||
All segments are lowercased. Addresses longer than 80 chars are truncated
|
||||
and hashed for readability.
|
||||
|
||||
Examples:
|
||||
build_key("price", "solana", "EPjFWdd5AufqSSqeM2qN1xzybapC8G5wUGxvZq2moaomYR")
|
||||
→ "price:solana:epjfwd..."
|
||||
"""
|
||||
address = str(address).strip().lower()
|
||||
chain = str(chain).strip().lower()
|
||||
data_type = str(data_type).strip().lower()
|
||||
|
||||
if len(address) > 80:
|
||||
# Hash long addresses (e.g., complex contract addresses)
|
||||
short = hashlib.md5(address.encode()).hexdigest()[:16]
|
||||
address = f"{address[:40]}..{short}"
|
||||
|
||||
return f"{data_type}:{chain}:{address}"
|
||||
|
||||
|
||||
def get_ttl(data_type: str, override: int | None = None) -> int:
|
||||
"""Get TTL for a data type, with optional override."""
|
||||
if override is not None:
|
||||
return override
|
||||
return SMART_TTLS.get(data_type.lower(), SMART_TTLS["default"])
|
||||
|
||||
|
||||
# ── Cache Implementation ────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class RpcCache:
|
||||
"""Two-layer cache: L1 in-memory (OrderedDict, LRU eviction) + L2 Redis.
|
||||
|
||||
Usage:
|
||||
cache = RpcCache()
|
||||
result = await cache.get_or_fetch(
|
||||
"price", "solana", token_address,
|
||||
factory_fn=lambda: fetch_price(token_address),
|
||||
)
|
||||
"""
|
||||
|
||||
# Max entries in L1 memory cache (prevents unbounded growth)
|
||||
MAX_L1_SIZE = 10000
|
||||
|
||||
# Redis key prefix for namespacing
|
||||
REDIS_PREFIX = "rmi:cache:"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
l1_max_size: int = 10000,
|
||||
redis_url: str | None = None,
|
||||
):
|
||||
"""Initialize the cache.
|
||||
|
||||
Args:
|
||||
l1_max_size: Maximum entries in L1 (memory) cache
|
||||
redis_url: Redis connection URL. If None, reads REDIS_URL env var.
|
||||
Set to empty string or False to force L1-only mode.
|
||||
"""
|
||||
self._l1: OrderedDict[str, CacheEntry] = OrderedDict()
|
||||
self._l1_max = l1_max_size
|
||||
self._lock = asyncio.Lock()
|
||||
self._stats = CacheStats()
|
||||
|
||||
# Redis L2 setup
|
||||
self._redis = None
|
||||
self._redis_available = False
|
||||
|
||||
_url = redis_url if redis_url is not None else os.getenv("REDIS_URL", "")
|
||||
if _url and _url.strip():
|
||||
try:
|
||||
import redis.asyncio as redis_asyncio
|
||||
|
||||
self._redis = redis_asyncio.from_url(
|
||||
_url.strip(),
|
||||
decode_responses=False, # We handle serialization ourselves
|
||||
socket_connect_timeout=3.0,
|
||||
socket_timeout=2.0,
|
||||
retry_on_timeout=True,
|
||||
health_check_interval=30,
|
||||
)
|
||||
self._redis_available = True
|
||||
logger.info(f"RpcCache: L2 Redis connected (prefix={self.REDIS_PREFIX})")
|
||||
except ImportError:
|
||||
logger.warning("RpcCache: redis package not installed, L2 disabled")
|
||||
except Exception as e:
|
||||
logger.warning(f"RpcCache: Redis connection failed ({e}), L2 disabled")
|
||||
else:
|
||||
logger.info("RpcCache: No REDIS_URL set, L1-only mode")
|
||||
|
||||
# ── Public API ──────────────────────────────────────────────────────
|
||||
|
||||
async def get(self, data_type: str, chain: str, address: str) -> Any | None:
|
||||
"""Retrieve a cached value. Returns None on miss or expiry.
|
||||
|
||||
Checks L1 first, then L2 (Redis), then returns None.
|
||||
On L2 hit, promotes the value into L1.
|
||||
"""
|
||||
key = build_key(data_type, chain, address)
|
||||
ttl = get_ttl(data_type)
|
||||
|
||||
# 1. Check L1 (in-memory)
|
||||
async with self._lock:
|
||||
entry = self._l1.get(key)
|
||||
if entry is not None:
|
||||
if entry.is_expired:
|
||||
del self._l1[key]
|
||||
self._stats.l1_size = len(self._l1)
|
||||
else:
|
||||
entry.access_count += 1
|
||||
self._l1.move_to_end(key)
|
||||
self._stats.hit_count += 1
|
||||
self._stats.total_calls += 1
|
||||
self._stats.l1_hits += 1
|
||||
self._stats.l1_size = len(self._l1)
|
||||
return entry.value
|
||||
|
||||
# 2. Check L2 (Redis)
|
||||
if self._redis_available and self._redis:
|
||||
try:
|
||||
redis_key = f"{self.REDIS_PREFIX}{key}"
|
||||
raw = await self._redis.get(redis_key)
|
||||
if raw is not None:
|
||||
value = self._deserialize(raw)
|
||||
# Promote to L1
|
||||
async with self._lock:
|
||||
self._stats.hit_count += 1
|
||||
self._stats.total_calls += 1
|
||||
self._stats.l2_hits += 1
|
||||
self._l1[key] = CacheEntry(value=value, ttl=float(ttl))
|
||||
self._l1.move_to_end(key)
|
||||
self._stats.l1_size = len(self._l1)
|
||||
self._evict_l1_if_needed()
|
||||
return value
|
||||
except Exception as e:
|
||||
logger.debug(f"Redis get error for {key}: {e}")
|
||||
|
||||
# 3. Miss
|
||||
async with self._lock:
|
||||
self._stats.miss_count += 1
|
||||
self._stats.total_calls += 1
|
||||
return None
|
||||
|
||||
async def set(
|
||||
self,
|
||||
data_type: str,
|
||||
chain: str,
|
||||
address: str,
|
||||
value: Any,
|
||||
ttl_override: int | None = None,
|
||||
) -> None:
|
||||
"""Store a value in both L1 and L2 caches.
|
||||
|
||||
Args:
|
||||
data_type: Type of data (see SMART_TTLS keys)
|
||||
chain: Blockchain identifier
|
||||
address: Address or identifier
|
||||
value: Any JSON-serializable value
|
||||
ttl_override: Override the default TTL for this data type
|
||||
"""
|
||||
key = build_key(data_type, chain, address)
|
||||
ttl = get_ttl(data_type, ttl_override)
|
||||
|
||||
# Serialize for size estimation
|
||||
raw = self._serialize(value)
|
||||
size = len(raw)
|
||||
|
||||
entry = CacheEntry(value=value, ttl=float(ttl), size_bytes=size)
|
||||
|
||||
# Store in L1
|
||||
async with self._lock:
|
||||
self._l1[key] = entry
|
||||
self._l1.move_to_end(key)
|
||||
self._stats.l1_size = len(self._l1)
|
||||
self._evict_l1_if_needed()
|
||||
|
||||
# Store in L2
|
||||
if self._redis_available and self._redis:
|
||||
try:
|
||||
redis_key = f"{self.REDIS_PREFIX}{key}"
|
||||
if ttl > 0:
|
||||
await self._redis.setex(redis_key, ttl, raw)
|
||||
else:
|
||||
# Permanent - no expiry
|
||||
await self._redis.set(redis_key, raw)
|
||||
except Exception as e:
|
||||
logger.debug(f"Redis set error for {key}: {e}")
|
||||
|
||||
async def get_or_fetch(
|
||||
self,
|
||||
data_type: str,
|
||||
chain: str,
|
||||
address: str,
|
||||
factory_fn: Callable[[], Awaitable[Any]],
|
||||
ttl_override: int | None = None,
|
||||
) -> Any:
|
||||
"""Get cached value or fetch fresh via factory function.
|
||||
|
||||
This is the primary convenience method - it combines get() + set()
|
||||
in a single call. If the value is cached and not expired, returns it
|
||||
immediately. Otherwise, calls factory_fn(), caches the result, and
|
||||
returns it.
|
||||
|
||||
Args:
|
||||
data_type: Type of data (e.g., 'price', 'token_meta')
|
||||
chain: Blockchain identifier
|
||||
address: Address/identifier
|
||||
factory_fn: Async callable that returns the value
|
||||
ttl_override: Override TTL for this entry
|
||||
|
||||
Returns:
|
||||
The cached or freshly fetched value, or None if fetch failed.
|
||||
"""
|
||||
# Try cache first
|
||||
cached = await self.get(data_type, chain, address)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
# Fetch fresh
|
||||
try:
|
||||
value = await factory_fn()
|
||||
if value is not None:
|
||||
await self.set(data_type, chain, address, value, ttl_override)
|
||||
return value
|
||||
except Exception as e:
|
||||
logger.warning(f"Factory fetch failed for {data_type}:{chain}:{address}: {e}")
|
||||
return None
|
||||
|
||||
async def invalidate(self, key_pattern: str) -> int:
|
||||
"""Remove all cached entries matching a key pattern.
|
||||
|
||||
For L1, uses simple string matching (fnmatch-style with *):
|
||||
- "price:*:*" - invalidates all price entries
|
||||
- "*:solana:*" - invalidates all Solana entries
|
||||
- "price:solana:*" - invalidates a specific token's price
|
||||
|
||||
For L2, uses Redis SCAN + pattern matching.
|
||||
|
||||
Returns number of entries invalidated (L1 + L2 count).
|
||||
"""
|
||||
# Convert glob-like pattern to regex for L1
|
||||
regex_pattern = re.escape(key_pattern).replace(r"\*", ".*")
|
||||
regex = re.compile(f"^{regex_pattern}$")
|
||||
|
||||
count = 0
|
||||
|
||||
# L1 invalidation
|
||||
async with self._lock:
|
||||
keys_to_remove = [k for k in self._l1 if regex.match(k)]
|
||||
for k in keys_to_remove:
|
||||
del self._l1[k]
|
||||
count += 1
|
||||
self._stats.l1_size = len(self._l1)
|
||||
|
||||
# L2 invalidation
|
||||
if self._redis_available and self._redis:
|
||||
try:
|
||||
redis_pattern = f"{self.REDIS_PREFIX}{key_pattern}"
|
||||
cursor = 0
|
||||
l2_count = 0
|
||||
while True:
|
||||
cursor, keys = await self._redis.scan(
|
||||
cursor=cursor,
|
||||
match=redis_pattern,
|
||||
count=100,
|
||||
)
|
||||
if keys:
|
||||
await self._redis.delete(*keys)
|
||||
l2_count += len(keys)
|
||||
if cursor == 0:
|
||||
break
|
||||
count += l2_count
|
||||
logger.debug(f"Invalidated {l2_count} L2 entries matching '{key_pattern}'")
|
||||
except Exception as e:
|
||||
logger.warning(f"Redis invalidation error for pattern '{key_pattern}': {e}")
|
||||
|
||||
async with self._lock:
|
||||
self._stats.invalidations += count
|
||||
logger.info(f"Cache invalidation: {count} entries removed (pattern='{key_pattern}')")
|
||||
return count
|
||||
|
||||
async def invalidate_address(
|
||||
self,
|
||||
data_type: str,
|
||||
chain: str,
|
||||
address: str,
|
||||
) -> bool:
|
||||
"""Invalidate a specific cache key."""
|
||||
key = build_key(data_type, chain, address)
|
||||
|
||||
found = False
|
||||
|
||||
async with self._lock:
|
||||
if key in self._l1:
|
||||
del self._l1[key]
|
||||
self._stats.l1_size = len(self._l1)
|
||||
self._stats.invalidations += 1
|
||||
found = True
|
||||
|
||||
if self._redis_available and self._redis:
|
||||
try:
|
||||
redis_key = f"{self.REDIS_PREFIX}{key}"
|
||||
deleted = await self._redis.delete(redis_key)
|
||||
if deleted:
|
||||
found = True
|
||||
except Exception as e:
|
||||
logger.debug(f"Redis delete error for {key}: {e}")
|
||||
|
||||
return found
|
||||
|
||||
async def clear(self) -> int:
|
||||
"""Clear all entries from both L1 and L2 caches.
|
||||
|
||||
Returns total number of entries cleared.
|
||||
"""
|
||||
count = 0
|
||||
|
||||
async with self._lock:
|
||||
count = len(self._l1)
|
||||
self._l1.clear()
|
||||
self._stats.l1_size = 0
|
||||
|
||||
if self._redis_available and self._redis:
|
||||
try:
|
||||
keys = await self._redis.keys(f"{self.REDIS_PREFIX}*")
|
||||
if keys:
|
||||
await self._redis.delete(*keys)
|
||||
count += len(keys)
|
||||
except Exception as e:
|
||||
logger.warning(f"Redis clear error: {e}")
|
||||
|
||||
logger.info(f"Cache cleared: {count} entries removed")
|
||||
return count
|
||||
|
||||
# ── Stats ────────────────────────────────────────────────────────────
|
||||
|
||||
async def stats(self) -> dict[str, Any]:
|
||||
"""Return cache performance statistics."""
|
||||
async with self._lock:
|
||||
result = self._stats.to_dict()
|
||||
# Add Redis L2 info
|
||||
if self._redis_available and self._redis:
|
||||
try:
|
||||
key_count = await self._redis.dbsize()
|
||||
result["l2_size"] = key_count
|
||||
except Exception:
|
||||
result["l2_size"] = -1
|
||||
result["redis_available"] = self._redis_available
|
||||
result["l1_max"] = self._l1_max
|
||||
return result
|
||||
|
||||
# ── Lifecycle ─────────────────────────────────────────────────────────
|
||||
|
||||
async def close(self):
|
||||
"""Close Redis connection if open."""
|
||||
if self._redis:
|
||||
try:
|
||||
await self._redis.aclose()
|
||||
self._redis = None
|
||||
self._redis_available = False
|
||||
logger.info("RpcCache: Redis connection closed")
|
||||
except Exception as e:
|
||||
logger.debug(f"Redis close error: {e}")
|
||||
|
||||
# ── Internal Helpers ──────────────────────────────────────────────────
|
||||
|
||||
def _serialize(self, value: Any) -> bytes:
|
||||
"""Serialize a value to bytes for Redis storage. Uses JSON."""
|
||||
import json
|
||||
|
||||
try:
|
||||
# Use a custom encoder for special types
|
||||
return json.dumps(value, default=str, ensure_ascii=False).encode("utf-8")
|
||||
except (TypeError, ValueError) as e:
|
||||
logger.warning(f"Serialization fallback for {type(value)}: {e}")
|
||||
return json.dumps({"__serialized__": str(value)}).encode("utf-8")
|
||||
|
||||
def _deserialize(self, raw: bytes) -> Any:
|
||||
"""Deserialize bytes from Redis back to Python object."""
|
||||
import json
|
||||
|
||||
try:
|
||||
return json.loads(raw.decode("utf-8"))
|
||||
except (json.JSONDecodeError, UnicodeDecodeError) as e:
|
||||
logger.warning(f"Deserialization error: {e}")
|
||||
return None
|
||||
|
||||
def _evict_l1_if_needed(self):
|
||||
"""Evict oldest entries from L1 if over capacity (LRU eviction)."""
|
||||
# Called within self._lock
|
||||
while len(self._l1) > self._l1_max:
|
||||
self._l1.popitem(last=False) # Remove oldest (first inserted)
|
||||
|
||||
# ── Health Check ──────────────────────────────────────────────────────
|
||||
|
||||
async def health(self) -> dict[str, Any]:
|
||||
"""Quick health check."""
|
||||
status = {
|
||||
"l1_ok": True,
|
||||
"l1_entries": len(self._l1),
|
||||
"l2_ok": False,
|
||||
}
|
||||
if self._redis_available and self._redis:
|
||||
try:
|
||||
await self._redis.ping()
|
||||
status["l2_ok"] = True
|
||||
except Exception:
|
||||
status["l2_ok"] = False
|
||||
return status
|
||||
|
||||
|
||||
# ── Singleton ──────────────────────────────────────────────────────────────
|
||||
|
||||
_cache_instance: RpcCache | None = None
|
||||
|
||||
|
||||
def get_rpc_cache() -> RpcCache:
|
||||
"""Get the global RpcCache singleton."""
|
||||
global _cache_instance
|
||||
if _cache_instance is None:
|
||||
_cache_instance = RpcCache()
|
||||
return _cache_instance
|
||||
|
|
@ -1,168 +0,0 @@
|
|||
"""
|
||||
Safe Module Importer - Handles stub fallbacks for security_intel router
|
||||
========================================================================
|
||||
|
||||
This module wraps security feature imports with graceful fallbacks to stubs
|
||||
if dependencies aren't available, preventing full router import failures.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def safe_import(name: str, fallback_to_stub: bool = True) -> Any | None:
|
||||
"""
|
||||
Safely import a module, optionally falling back to stub implementation.
|
||||
|
||||
Args:
|
||||
name: Module path (e.g., 'web3' or 'app.contract_deepscan')
|
||||
fallback_to_stub: If True, returns stub module on ImportError
|
||||
|
||||
Returns:
|
||||
Imported module, stub fallback, or None
|
||||
"""
|
||||
try:
|
||||
module = __import__(name, fromlist=[""])
|
||||
logger.info(f"✓ Loaded: {name}")
|
||||
return module
|
||||
except ImportError:
|
||||
if fallback_to_stub:
|
||||
logger.warning(f"⚠ {name} not available, using stub fallback")
|
||||
return None
|
||||
raise
|
||||
|
||||
|
||||
# ─── STUB CLASSES FOR CRITICAL DEPENDENCIES ───────────────────────
|
||||
|
||||
|
||||
class StubContractDeepscan:
|
||||
"""Stub contract scanning module."""
|
||||
|
||||
def deep_scan_contract(self, address: str, chain: str = "base") -> dict[str, Any]:
|
||||
return {
|
||||
"address": address,
|
||||
"chain": chain,
|
||||
"scanned": False,
|
||||
"vulnerabilities": [],
|
||||
"checks": [],
|
||||
"status": "stub_mode",
|
||||
}
|
||||
|
||||
def batch_deep_scan(self, addresses: list[str], chain: str = "base") -> dict[str, Any]:
|
||||
results = [self.deep_scan_contract(addr, chain) for addr in addresses]
|
||||
return {"contracts": results, "total": len(results)}
|
||||
|
||||
|
||||
class StubCrossChainCorrelator:
|
||||
"""Stub cross-chain correlation module."""
|
||||
|
||||
class ChainFingerprint:
|
||||
def __init__(self, address: str, chain: str):
|
||||
self.address = address
|
||||
self.chain = chain
|
||||
|
||||
def get_hash(self) -> str:
|
||||
return f"{self.chain}:{self.address}"
|
||||
|
||||
class CrossChainCorrelator:
|
||||
def link_wallets(self, addresses: list[str], chains: list[str]) -> dict[str, Any]:
|
||||
return {"linked": True, "wallets": addresses, "chains": chains}
|
||||
|
||||
def find_correlations(self, address: str, limit: int = 10) -> dict[str, Any]:
|
||||
return {"address": address, "correlations": [], "count": 0}
|
||||
|
||||
|
||||
class StubCryptoGuard:
|
||||
"""Stub crypto guard module."""
|
||||
|
||||
def check_wallet_reputation(self, address: str) -> dict[str, Any]:
|
||||
return {"address": address, "reputation": "unknown", "risk_score": 0, "flags": []}
|
||||
|
||||
def rate_limit_by_tier(self, tier: str) -> dict[str, int]:
|
||||
tiers = {"FREE": {"requests_per_hour": 10, "requests_per_day": 100}}
|
||||
return tiers.get(tier, {"requests_per_hour": 100, "requests_per_day": 1000})
|
||||
|
||||
def require_crypto_auth(self, request: Any) -> dict[str, Any]:
|
||||
return {"authenticated": True}
|
||||
|
||||
|
||||
# ─── STUB INSTANCES ───────────────────────────────────────────────
|
||||
|
||||
_stub_deepscan = StubContractDeepscan()
|
||||
_stub_correlator = StubCrossChainCorrelator()
|
||||
_stub_correlator_corr = _stub_correlator.CrossChainCorrelator()
|
||||
_stub_crypto_guard = StubCryptoGuard()
|
||||
|
||||
|
||||
# ─── FALLBACK LOADERS ─────────────────────────────────────────────
|
||||
|
||||
|
||||
def get_contract_deepscan():
|
||||
"""Load contract deepscan with fallback to stub."""
|
||||
try:
|
||||
from app.contract_deepscan import batch_deep_scan, deep_scan_contract
|
||||
|
||||
return deep_scan_contract, batch_deep_scan
|
||||
except ImportError:
|
||||
logger.warning("contract_deepscan not available, using stub")
|
||||
return (_stub_deepscan.deep_scan_contract, _stub_deepscan.batch_deep_scan)
|
||||
|
||||
|
||||
def get_cross_chain_correlator():
|
||||
"""Load cross-chain correlator with fallback to stub."""
|
||||
try:
|
||||
from app.cross_chain_correlator import CrossChainCorrelator
|
||||
|
||||
return CrossChainCorrelator
|
||||
except ImportError:
|
||||
logger.warning("cross_chain_correlator not available, using stub")
|
||||
return _stub_correlator_corr
|
||||
|
||||
|
||||
def get_crypto_guard():
|
||||
"""Load crypto guard with fallback to stub."""
|
||||
try:
|
||||
from app.crypto_guard import (
|
||||
check_wallet_reputation,
|
||||
rate_limit_by_tier,
|
||||
)
|
||||
|
||||
return check_wallet_reputation, rate_limit_by_tier
|
||||
except ImportError:
|
||||
logger.warning("crypto_guard not available, using stub")
|
||||
return (_stub_crypto_guard.check_wallet_reputation, _stub_crypto_guard.rate_limit_by_tier)
|
||||
|
||||
|
||||
def get_sentinel_manager():
|
||||
"""Load mempool sentinel with fallback to stub."""
|
||||
try:
|
||||
from app.mempool_sentinel import SentinelManager
|
||||
|
||||
return SentinelManager
|
||||
except ImportError:
|
||||
logger.warning("mempool_sentinel not available, using stub")
|
||||
return lambda: None
|
||||
|
||||
|
||||
def get_wallet_anomaly_detector():
|
||||
"""Load wallet anomaly detector with fallback to stub."""
|
||||
try:
|
||||
from app.ml_anomaly import WalletAnomalyDetector
|
||||
|
||||
return WalletAnomalyDetector
|
||||
except ImportError:
|
||||
logger.warning("ml_anomaly not available, using stub")
|
||||
return lambda: None
|
||||
|
||||
|
||||
def get_token_anomaly_detector():
|
||||
"""Load token anomaly detector with fallback to stub."""
|
||||
try:
|
||||
from app.ml_anomaly import TokenMetricAnomalyDetector
|
||||
|
||||
return TokenMetricAnomalyDetector
|
||||
except ImportError:
|
||||
logger.warning("ml_anomaly not available, using stub")
|
||||
return lambda: None
|
||||
|
|
@ -1,386 +0,0 @@
|
|||
"""
|
||||
SENTINEL AI - Self-Training Scam Classifier
|
||||
============================================
|
||||
|
||||
Turns 5,000+ historical token scans into a self-improving ML model.
|
||||
Every new scan feeds back into training. The model gets smarter over time.
|
||||
|
||||
Architecture:
|
||||
Historical scans → Feature extraction → XGBoost training → Model on disk
|
||||
New scan → Feature extraction → Model prediction → AI risk score (0-100)
|
||||
Confirmed scam → Weight boost → Retrain trigger
|
||||
|
||||
Features extracted from all 45 enrichments + market data + SENTINEL modules.
|
||||
The model learns which COMBINATIONS of signals predict scams, catching
|
||||
patterns that static rules miss entirely.
|
||||
|
||||
Premium feature: "AI-Powered Risk Score" - ML confidence alongside rules-based score.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import pickle
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
logger = logging.getLogger("sentinel.ai")
|
||||
|
||||
MODEL_DIR = os.path.join(os.path.dirname(__file__), "..", "data", "models")
|
||||
MODEL_PATH = os.path.join(MODEL_DIR, "scam_classifier_xgb.pkl")
|
||||
FEATURE_NAMES_PATH = os.path.join(MODEL_DIR, "scam_classifier_features.json")
|
||||
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
# Feature Extraction - 80+ features from enrichment data
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def extract_features(scan_result: dict[str, Any]) -> dict[str, float]:
|
||||
"""Extract ML features from a scan result. Returns dict of feature_name → value."""
|
||||
f = {}
|
||||
free = scan_result.get("free", scan_result) if isinstance(scan_result, dict) else {}
|
||||
|
||||
# ── Market data features (16 features) ──
|
||||
f["price_usd"] = float(free.get("price_usd", 0) or 0)
|
||||
f["volume_24h"] = float(free.get("volume_24h", 0) or 0)
|
||||
f["liquidity_usd"] = float(free.get("liquidity_usd", 0) or 0)
|
||||
f["market_cap"] = float(free.get("market_cap", 0) or 0)
|
||||
f["age_hours"] = float(free.get("age_hours", 0) or 0)
|
||||
f["price_change_5m"] = float(free.get("price_change_5m", 0) or 0)
|
||||
f["price_change_1h"] = float(free.get("price_change_1h", 0) or 0)
|
||||
f["price_change_24h"] = float(free.get("price_change_24h", 0) or 0)
|
||||
f["tx_count"] = float(free.get("tx_count", 0) or 0)
|
||||
f["volume_to_liquidity_ratio"] = f["volume_24h"] / max(f["liquidity_usd"], 1)
|
||||
f["market_cap_to_liquidity_ratio"] = f["market_cap"] / max(f["liquidity_usd"], 1)
|
||||
f["has_price_data"] = 1.0 if f["price_usd"] > 0 else 0.0
|
||||
f["has_liquidity"] = 1.0 if f["liquidity_usd"] > 0 else 0.0
|
||||
f["is_very_new"] = 1.0 if f["age_hours"] < 1 else 0.0
|
||||
f["is_new"] = 1.0 if f["age_hours"] < 24 else 0.0
|
||||
f["price_volatility"] = abs(f["price_change_1h"]) + abs(f["price_change_24h"])
|
||||
|
||||
# ── Holder features (8 features) ──
|
||||
holders = free.get("holders", {}) or {}
|
||||
if isinstance(holders, dict):
|
||||
f["holder_count"] = float(holders.get("total", holders.get("count", 0)) or 0)
|
||||
f["top10_pct"] = float(holders.get("top_10_percentage", holders.get("top10_pct", 0)) or 0)
|
||||
f["top1_pct"] = float(holders.get("top_1_percentage", holders.get("top1_pct", 0)) or 0)
|
||||
f["holder_concentration_high"] = 1.0 if f["top10_pct"] > 80 else 0.0
|
||||
f["holder_concentration_critical"] = 1.0 if f["top10_pct"] > 95 else 0.0
|
||||
f["has_holder_data"] = 1.0 if f["holder_count"] > 0 else 0.0
|
||||
f["few_holders"] = 1.0 if 0 < f["holder_count"] < 20 else 0.0
|
||||
f["many_holders"] = 1.0 if f["holder_count"] > 500 else 0.0
|
||||
|
||||
# ── Honeypot / trade simulation (8 features) ──
|
||||
sim = free.get("simulation", {}) or {}
|
||||
if isinstance(sim, dict):
|
||||
f["honeypot_detected"] = 1.0 if sim.get("is_honeypot") or sim.get("honeypot") else 0.0
|
||||
f["buy_success"] = 1.0 if sim.get("buy_success", True) else 0.0
|
||||
f["sell_success"] = 1.0 if sim.get("sell_success", True) else 0.0
|
||||
f["buy_tax_pct"] = float(sim.get("buy_tax_pct", 0) or 0)
|
||||
f["sell_tax_pct"] = float(sim.get("sell_tax_pct", 0) or 0)
|
||||
f["tax_high"] = 1.0 if f["sell_tax_pct"] > 10 else 0.0
|
||||
f["tax_extreme"] = 1.0 if f["sell_tax_pct"] > 50 else 0.0
|
||||
f["has_simulation"] = 1.0 if sim else 0.0
|
||||
|
||||
# ── Contract / authority features (6 features) ──
|
||||
contract = free.get("contract_verification", {}) or {}
|
||||
if isinstance(contract, dict):
|
||||
f["contract_verified"] = 1.0 if contract.get("verified") else 0.0
|
||||
f["contract_unverified"] = 1.0 if not contract.get("verified") else 0.0
|
||||
f["is_proxy"] = 1.0 if contract.get("is_proxy") or contract.get("proxy") else 0.0
|
||||
f["mint_authority_renounced"] = 1.0 if free.get("mint_authority") == "renounced" else 0.0
|
||||
f["freeze_authority_exists"] = 1.0 if free.get("freeze_authority") else 0.0
|
||||
f["has_contract_data"] = 1.0 if contract else 0.0
|
||||
|
||||
# ── Deployer features (8 features) ──
|
||||
deployer = free.get("deployer", {}) or {}
|
||||
deep = free.get("deep_deployer", {}) or {}
|
||||
if isinstance(deployer, dict) or isinstance(deep, dict):
|
||||
d = {**deployer, **deep} if isinstance(deep, dict) else deployer
|
||||
f["deployer_known"] = 1.0 if d.get("address") or deployer.get("address") else 0.0
|
||||
f["deployer_risk_score"] = float(d.get("risk_score", 0) or 0)
|
||||
f["deployer_scam_count"] = float(d.get("scam_count", 0) or 0)
|
||||
f["deployer_high_risk"] = 1.0 if f["deployer_risk_score"] > 70 else 0.0
|
||||
f["deployer_critical"] = 1.0 if f["deployer_risk_score"] > 90 else 0.0
|
||||
f["multi_chain_deployer"] = 1.0 if free.get("cross_chain", {}).get("chain_count", 0) > 1 else 0.0
|
||||
f["deployer_chain_count"] = float(
|
||||
free.get("cross_chain", {}).get("chain_count", 1) if isinstance(free.get("cross_chain"), dict) else 1
|
||||
)
|
||||
f["has_deployer_data"] = 1.0 if f["deployer_known"] else 0.0
|
||||
|
||||
# ── LP / liquidity lock (5 features) ──
|
||||
lp = free.get("lp_lock_multi", {}) or {}
|
||||
if isinstance(lp, dict):
|
||||
f["lp_locked"] = 1.0 if lp.get("locked") or lp.get("is_locked") else 0.0
|
||||
f["lp_lock_unconfirmed"] = 1.0 if not f["lp_locked"] and f["liquidity_usd"] > 0 else 0.0
|
||||
f["lp_lock_pct"] = float(lp.get("lock_percentage", lp.get("locked_pct", 0)) or 0)
|
||||
f["lp_has_data"] = 1.0 if lp else 0.0
|
||||
f["no_lp"] = 1.0 if f["liquidity_usd"] == 0 else 0.0
|
||||
|
||||
# ── External API signals (12 features) ──
|
||||
# Birdeye
|
||||
birdeye = free.get("birdeye", {}) or {}
|
||||
f["birdeye_honeypot"] = 1.0 if isinstance(birdeye, dict) and birdeye.get("honeypot") else 0.0
|
||||
f["birdeye_rugpull"] = 1.0 if isinstance(birdeye, dict) and birdeye.get("rugpull") else 0.0
|
||||
|
||||
# Honeypot.is
|
||||
hp = free.get("honeypot_is", {}) or {}
|
||||
f["honeypot_is_detected"] = 1.0 if isinstance(hp, dict) and hp.get("is_honeypot") else 0.0
|
||||
|
||||
# Token Sniffer
|
||||
ts = free.get("token_sniffer", {}) or {}
|
||||
f["tokensniffer_scam"] = 1.0 if isinstance(ts, dict) and ts.get("is_scam") else 0.0
|
||||
f["tokensniffer_score"] = float(ts.get("score", 0) or 0) if isinstance(ts, dict) else 0.0
|
||||
|
||||
# ChainAware AI
|
||||
ca = free.get("chainaware_ai", {}) or {}
|
||||
f["chainaware_rug_risk"] = float(ca.get("rug_probability", 0) or 0) if isinstance(ca, dict) else 0.0
|
||||
|
||||
# GoPlus
|
||||
gp = free.get("goplus", {}) or {}
|
||||
f["goplus_honeypot"] = 1.0 if isinstance(gp, dict) and gp.get("is_honeypot") else 0.0
|
||||
|
||||
# De.Fi
|
||||
df = free.get("defi_scanner", {}) or {}
|
||||
f["defi_honeypot"] = 1.0 if isinstance(df, dict) and df.get("honeypot") else 0.0
|
||||
f["defi_ai_score"] = float(df.get("aiScore", 50) or 50) if isinstance(df, dict) else 50.0
|
||||
|
||||
# Copycat
|
||||
cc = free.get("copycat_check", {}) or {}
|
||||
f["is_copycat"] = 1.0 if isinstance(cc, dict) and cc.get("is_copycat") else 0.0
|
||||
|
||||
# Volume anomaly
|
||||
va = free.get("volume_anomaly", {}) or {}
|
||||
f["volume_manipulation"] = 1.0 if isinstance(va, dict) and va.get("manipulation_detected") else 0.0
|
||||
|
||||
# ── RAG / scam database features (6 features) ──
|
||||
rag = free.get("rag_scam_check", {}) or {}
|
||||
if isinstance(rag, dict):
|
||||
f["rag_scam_match"] = 1.0 if rag.get("match_found") or rag.get("is_scam") else 0.0
|
||||
f["rag_similarity"] = float(rag.get("similarity", 0) or 0)
|
||||
f["rag_similarity_high"] = 1.0 if f["rag_similarity"] > 0.8 else 0.0
|
||||
f["rag_matches_count"] = float(rag.get("match_count", 0) or 0)
|
||||
f["known_scam_address"] = 1.0 if rag.get("is_known_scam") else 0.0
|
||||
f["has_rag_data"] = 1.0 if rag else 0.0
|
||||
|
||||
# ── Social / sentiment (5 features) ──
|
||||
sentiment = free.get("sentiment", {}) or {}
|
||||
f["social_volume"] = float(sentiment.get("mention_count", 0) or 0) if isinstance(sentiment, dict) else 0.0
|
||||
f["sentiment_score"] = float(sentiment.get("score", 0) or 0) if isinstance(sentiment, dict) else 0.0
|
||||
santiment = free.get("santiment", {}) or {}
|
||||
f["santiment_hype"] = 1.0 if isinstance(santiment, dict) and santiment.get("hype_detected") else 0.0
|
||||
f["santiment_viral_low_vol"] = 1.0 if isinstance(santiment, dict) and santiment.get("viral_low_volume") else 0.0
|
||||
f["has_social_data"] = 1.0 if sentiment or santiment else 0.0
|
||||
|
||||
# ── Chain/network features (4 features) ──
|
||||
chain = scan_result.get("chain", "").lower()
|
||||
f["chain_solana"] = 1.0 if chain == "solana" else 0.0
|
||||
f["chain_ethereum"] = 1.0 if chain == "ethereum" else 0.0
|
||||
f["chain_bsc"] = 1.0 if chain == "bsc" else 0.0
|
||||
f["chain_base"] = 1.0 if chain == "base" else 0.0
|
||||
|
||||
# ── Nansen / Arkham (4 features) ──
|
||||
nansen = free.get("nansen", {}) or {}
|
||||
f["nansen_low_holders"] = 1.0 if isinstance(nansen, dict) and nansen.get("low_holders") else 0.0
|
||||
f["nansen_net_outflow"] = 1.0 if isinstance(nansen, dict) and nansen.get("net_outflow") else 0.0
|
||||
arkham = free.get("arkham", {}) or {}
|
||||
f["arkham_scam_entity"] = 1.0 if isinstance(arkham, dict) and arkham.get("scam_entity") else 0.0
|
||||
f["has_premium_data"] = 1.0 if nansen or arkham else 0.0
|
||||
|
||||
return f
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
# Model Training
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class ScamClassifier:
|
||||
"""XGBoost classifier trained on historical scan data."""
|
||||
|
||||
def __init__(self):
|
||||
self.model = None
|
||||
self.feature_names: list[str] = []
|
||||
self.training_samples: int = 0
|
||||
self.accuracy: float = 0.0
|
||||
self.last_trained: str | None = None
|
||||
self._load()
|
||||
|
||||
def _load(self):
|
||||
"""Load model from disk if available."""
|
||||
if os.path.exists(MODEL_PATH):
|
||||
try:
|
||||
with open(MODEL_PATH, "rb") as f:
|
||||
self.model = pickle.load(f)
|
||||
if os.path.exists(FEATURE_NAMES_PATH):
|
||||
with open(FEATURE_NAMES_PATH) as f:
|
||||
meta = json.load(f)
|
||||
self.feature_names = meta.get("features", [])
|
||||
self.training_samples = meta.get("samples", 0)
|
||||
self.accuracy = meta.get("accuracy", 0.0)
|
||||
self.last_trained = meta.get("trained_at")
|
||||
logger.info(
|
||||
f"Loaded scam classifier: {self.training_samples} samples, "
|
||||
f"{len(self.feature_names)} features, {self.accuracy:.1%} accuracy"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to load classifier: {e}")
|
||||
self.model = None
|
||||
|
||||
def _save(self):
|
||||
"""Persist model and metadata to disk."""
|
||||
os.makedirs(MODEL_DIR, exist_ok=True)
|
||||
with open(MODEL_PATH, "wb") as f:
|
||||
pickle.dump(self.model, f)
|
||||
with open(FEATURE_NAMES_PATH, "w") as f:
|
||||
json.dump(
|
||||
{
|
||||
"features": self.feature_names,
|
||||
"samples": self.training_samples,
|
||||
"accuracy": self.accuracy,
|
||||
"trained_at": datetime.now(UTC).isoformat(),
|
||||
},
|
||||
f,
|
||||
)
|
||||
logger.info(f"Saved classifier: {self.training_samples} samples, {self.accuracy:.1%} accuracy")
|
||||
|
||||
def train(self, scans: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
"""Train on historical scan data. Each scan must have 'is_scam' label."""
|
||||
try:
|
||||
from xgboost import XGBClassifier
|
||||
except ImportError:
|
||||
logger.warning("XGBoost not installed. Install with: pip install xgboost")
|
||||
return {"status": "error", "error": "xgboost not installed"}
|
||||
|
||||
# Extract features and labels
|
||||
X_list = []
|
||||
y_list = []
|
||||
|
||||
for scan in scans:
|
||||
features = extract_features(scan)
|
||||
is_scam = scan.get("is_scam", False) or scan.get("verdict") == "scam"
|
||||
|
||||
if not X_list: # First sample - record feature names
|
||||
self.feature_names = sorted(features.keys())
|
||||
|
||||
# Build feature vector in consistent order
|
||||
row = [features.get(name, 0.0) for name in self.feature_names]
|
||||
X_list.append(row)
|
||||
y_list.append(1 if is_scam else 0)
|
||||
|
||||
if len(X_list) < 10:
|
||||
return {"status": "error", "error": f"Need at least 10 samples, got {len(X_list)}"}
|
||||
|
||||
if sum(y_list) < 2:
|
||||
return {"status": "error", "error": "Need at least 2 scam samples for training"}
|
||||
|
||||
X = np.array(X_list, dtype=np.float32)
|
||||
y = np.array(y_list, dtype=np.int32)
|
||||
|
||||
# Handle NaN/Inf
|
||||
X = np.nan_to_num(X, nan=0.0, posinf=0.0, neginf=0.0)
|
||||
|
||||
# Train
|
||||
self.model = XGBClassifier(
|
||||
n_estimators=100,
|
||||
max_depth=4,
|
||||
learning_rate=0.1,
|
||||
subsample=0.8,
|
||||
colsample_bytree=0.8,
|
||||
scale_pos_weight=(len(y) - sum(y)) / max(sum(y), 1), # Handle class imbalance
|
||||
random_state=42,
|
||||
use_label_encoder=False,
|
||||
eval_metric="logloss",
|
||||
)
|
||||
self.model.fit(X, y)
|
||||
|
||||
# Evaluate
|
||||
train_pred = self.model.predict(X)
|
||||
self.accuracy = float((train_pred == y).mean())
|
||||
self.training_samples = len(X_list)
|
||||
self.last_trained = datetime.now(UTC).isoformat()
|
||||
|
||||
# Feature importance
|
||||
importance = {}
|
||||
if hasattr(self.model, "feature_importances_"):
|
||||
for name, imp in zip(self.feature_names, self.model.feature_importances_, strict=False):
|
||||
importance[name] = round(float(imp), 4)
|
||||
|
||||
self._save()
|
||||
|
||||
return {
|
||||
"status": "trained",
|
||||
"samples": self.training_samples,
|
||||
"scam_samples": int(sum(y)),
|
||||
"features": len(self.feature_names),
|
||||
"accuracy": round(self.accuracy, 3),
|
||||
"feature_importance": dict(sorted(importance.items(), key=lambda x: -x[1])[:15]),
|
||||
}
|
||||
|
||||
def predict(self, scan_result: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Predict scam probability for a new scan."""
|
||||
if self.model is None:
|
||||
return {"status": "no_model", "probability": None, "confidence": 0}
|
||||
|
||||
features = extract_features(scan_result)
|
||||
row = np.array([[features.get(name, 0.0) for name in self.feature_names]], dtype=np.float32)
|
||||
row = np.nan_to_num(row, nan=0.0, posinf=0.0, neginf=0.0)
|
||||
|
||||
try:
|
||||
proba = self.model.predict_proba(row)[0]
|
||||
scam_prob = float(proba[1]) if len(proba) > 1 else float(proba[0])
|
||||
|
||||
# Confidence based on training data quality
|
||||
confidence = min(self.accuracy * 100, 95) if self.accuracy > 0 else 50
|
||||
|
||||
# Get top contributing features
|
||||
contributions = {}
|
||||
if hasattr(self.model, "feature_importances_"):
|
||||
for name, imp in zip(self.feature_names, self.model.feature_importances_, strict=False):
|
||||
if features.get(name, 0) != 0 and imp > 0.01:
|
||||
contributions[name] = {
|
||||
"value": features.get(name, 0),
|
||||
"importance": round(float(imp), 3),
|
||||
}
|
||||
|
||||
# Risk level
|
||||
if scam_prob > 0.8:
|
||||
risk_level = "critical"
|
||||
elif scam_prob > 0.6:
|
||||
risk_level = "high"
|
||||
elif scam_prob > 0.4:
|
||||
risk_level = "medium"
|
||||
elif scam_prob > 0.2:
|
||||
risk_level = "low"
|
||||
else:
|
||||
risk_level = "safe"
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"probability": round(scam_prob, 4),
|
||||
"risk_level": risk_level,
|
||||
"ai_risk_score": round(scam_prob * 100),
|
||||
"confidence": round(confidence, 1),
|
||||
"model_samples": self.training_samples,
|
||||
"model_accuracy": round(self.accuracy, 3),
|
||||
"top_signals": dict(sorted(contributions.items(), key=lambda x: -x[1]["importance"])[:8]),
|
||||
}
|
||||
except Exception as e:
|
||||
logger.warning(f"Prediction failed: {e}")
|
||||
return {"status": "error", "probability": None, "error": str(e)}
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
# Singleton
|
||||
# ──────────────────────────────────────────────────────────────
|
||||
|
||||
_classifier: ScamClassifier | None = None
|
||||
|
||||
|
||||
def get_classifier() -> ScamClassifier:
|
||||
global _classifier
|
||||
if _classifier is None:
|
||||
_classifier = ScamClassifier()
|
||||
return _classifier
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue