rmi-backend/DESIGN.md

# RMI Backend — 2026 Architecture Design

## Why this exists

The current backend (`/root/backend/`) has grown organically:
- `main.py` is 10,305 lines
- 124 router files flat in `app/routers/`
- 14 RAG modules scattered at top of `app/`
- `token_scanner.py` is 4,109 lines
- `x402_tools.py` is 5,817 lines
- Cross-cutting concerns (redis, auth, errors) duplicated across modules
- Domain logic entangled with FastAPI
- No tests, no type safety, no clear boundaries

15 mechanical refactor tasks would patch symptoms. This design fixes the architecture.

## Target Layout

```
/root/backend/
├── pyproject.toml              uv + ruff + mypy + pytest config (single source)
├── .pre-commit-config.yaml     ruff + mypy + size cap + gitleaks
├── Dockerfile
├── alembic/                    async migrations
│
├── app/
│   ├── main.py                 <100 lines: app factory + lifespan + middleware ONLY
│   ├── config.py               pydantic-settings, env loading
│   │
│   ├── core/                   cross-cutting, NO business logic
│   │   ├── logging.py          structlog JSON + correlation ID
│   │   ├── errors.py           AppError hierarchy + FastAPI handlers
│   │   ├── redis.py            async client + get_redis() Depends()
│   │   ├── db.py               async SQLAlchemy session
│   │   ├── auth.py             JWT decode + role guards
│   │   ├── lifespan.py         startup/shutdown
│   │   ├── middleware.py       CORS, rate limit, correlation ID
│   │   ├── websocket.py        WS connection manager
│   │   ├── tracing.py          OpenTelemetry + Langfuse v4 init
│   │   ├── http.py             async httpx client
│   │   └── pagination.py       cursor-based
│   │
│   ├── api/                    HTTP transport, thin routes
│   │   ├── deps.py             shared Depends (current_user, redis, etc)
│   │   ├── v1/
│   │   │   ├── public/         no auth  — scanner, wallet, token, pricing, health
│   │   │   ├── auth/           JWT      — portfolio, alerts, intel, profile
│   │   │   ├── admin/          admin    — users, system, ops
│   │   │   ├── x402/           paid     — tools, tokens, wallets, defi, security
│   │   │   └── mcp/            MCP      — tools.py
│   │   └── ws/                 WebSocket
│   │       └── alerts.py
│   │
│   ├── domain/                 pure business logic, NO FastAPI imports
│   │   ├── scanner/            core + honeypot + rugcheck + holders + contract + deployer + models + service
│   │   ├── wallet/             analyzer + labels + behavior + models + service
│   │   ├── token/              discovery + supply + models + service
│   │   ├── rag/                embeddings + chunking + search + ingest + firehose + feedback + agentic + evaluation + tracing + router + permanence + models + service
│   │   ├── x402/               facilitator + tokens + enforcement + settlement + models + service
│   │   ├── intel/              feeds + narratives + graph + models + service
│   │   ├── scam/               classifier + patterns + models + service
│   │   ├── databus/            client + chains(96) + models + service
│   │   └── bulletin/           board + models + service
│   │
│   ├── infra/                  external integrations
│   │   ├── ollama.py
│   │   ├── langfuse.py
│   │   ├── vector_store.py
│   │   ├── chains/             evm + solana + bitcoin + base + ...
│   │   ├── apis/               coingecko + etherscan + birdeye + goplus + ...
│   │   └── providers/          ollama + openrouter + huggingface + ...
│   │
│   └── workers/                background jobs (separate from API)
│       ├── firehose.py
│       ├── scanner_queue.py
│       ├── ingest_cron.py
│       └── cleanup.py
│
└── tests/
    ├── conftest.py
    ├── unit/domain/
    └── integration/api/v1/
```

## Key Design Principles

1. **STRICT LAYERING.** `api → domain → infra`. Never reverse. Domain knows nothing about HTTP.
2. **ONE SOURCE OF TRUTH for cross-cutting.** redis/auth/errors/logging live in `core/` exactly once. Routes import, never redefine.
3. **HARD SIZE CAP.** 500 lines per file. Enforced in pre-commit. No 4,109-line `token_scanner.py` ever again.
4. **THIN ROUTES.** Routes parse → call service → return. No business logic in HTTP layer.
5. **DOMAIN = PURE PYTHON.** `domain/scanner/` can be unit tested without spinning up FastAPI. This is the test that proves the architecture.
6. **WORKERS SEPARATED.** Background jobs don't pollute the API. firehose, scanner_queue, ingest_cron live in `workers/`.
7. **PYDANTIC V2 EVERYWHERE.** Every domain has `models.py`. No `dict` types crossing boundaries.
8. **ASYNC-ONLY.** No sync I/O in handlers. Same shape for the whole codebase.
9. **OBSERVABILITY BY DEFAULT.** structlog JSON + correlation ID + OTel + Langfuse in `core/tracing.py`. Every endpoint instrumented without opt-in.
10. **STRANGLER FIG MIGRATION.** New skeleton co-exists with old code. Old `main.py` keeps importing the old routers. New routes added alongside. Per-domain cutover, not big-bang.

## Migration Order

| Order | Domain | Why |
|-------|--------|-----|
| 0 | `rag_engine` shim | unblock prod crash, temp until `app/rag/` lands |
| 1 | `core/` | foundation everyone depends on |
| 2 | `infra/` | external integrations domain depends on |
| 3 | `alerts` | smallest, well-bounded, has WS + JWT + redis — proves full pattern |
| 4 | `wallet` | high-value, used by frontend |
| 5 | `token` | high-value |
| 6 | `scanner` | biggest (4,109 lines), do last when pattern is mature |
| 7 | `x402` | payment system, critical, mature pattern by then |
| 8 | `intel`, `scam`, `databus`, `bulletin` | long tail |
| 9 | `rag` consolidation (was 14 files) | last because it's the most coupled |

## What Ships This Pass (Foundation)

1. Fix crash — `rag_engine` re-export shim, backend healthy
2. `pyproject.toml` — uv + ruff + mypy strict + pytest
3. `.pre-commit-config.yaml` — ruff + mypy + size cap (500) + gitleaks
4. `app/core/` — 11 modules, each <200 lines
5. `app/api/v1/__init__.py` — router aggregator that still imports OLD routers (zero breakage)
6. `app/main.py` — rewritten to ~100 lines, calls lifespan + middleware from `core/`, mounts new aggregator
7. Verify: backend boots, all 757 routes respond, health 200, no import errors
8. Commit + deploy

## What Does NOT Ship This Pass

- Migrating alerts/wallet/token/scanner to new `domain/`. That's Phase 2.
- The 15 mechanical refactors. Replaced with the layered architecture.
- Deleting old code. Strangler fig — old stays until domain is migrated.

## Phase 2: Alerts Vertical Slice (proves the pattern)

After foundation lands, migrate `alerts` end-to-end as the reference:

```
app/domain/alerts/
├── models.py      # Alert, AlertRule, Notification — Pydantic v2
├── repository.py  # async SQLAlchemy queries
├── service.py     # business logic, pure Python
└── broadcaster.py # WebSocket broadcast helper

app/api/v1/auth/alerts.py  # thin route: parse → call service → return
```

This proves the pattern works: domain is pure Python, route is <100 lines, can be unit tested without HTTP.

When alerts is shipped and verified in prod, the same pattern is applied to wallet, token, scanner, etc.