docs(state): update STATUS.md and ARCHITECTURE.md for Phase 4 progress

This commit is contained in:
Crypto Rug Munch 2026-07-07 04:47:09 +07:00
parent 4686cb3cfd
commit 757eefd227
2 changed files with 61 additions and 22 deletions

View file

@ -1,10 +1,11 @@
# RMI Backend — Architecture # RMI Backend — Architecture
**Status:** canonical **Status:** canonical
**Last updated:** 2026-07-03 **Last updated:** 2026-07-07
## Overview ## Overview
FastAPI monolith serving crypto risk intelligence via HTTP, Telegram, and MCP. FastAPI monolith serving crypto risk intelligence via HTTP, Telegram, and MCP.
Domains are consolidated under `app/domains/` per AUDIT-2026-Q3 Phase 4.
## High-level flow ## High-level flow
``` ```
@ -12,9 +13,9 @@ Client (Web / Telegram / AI Agent)
nginx → rmi-backend container (:8000) nginx → rmi-backend container (:8000)
FastAPI routers FastAPI routers (mounted by app/mount.py)
Services / scanners / DataBus providers app/domains/<domain>/service.py → app/infra/ + app/domains/<domain>/repository.py
Postgres / Redis / Qdrant / Neo4j / ClickHouse Postgres / Redis / Qdrant / Neo4j / ClickHouse
``` ```
@ -22,35 +23,42 @@ Postgres / Redis / Qdrant / Neo4j / ClickHouse
## Components ## Components
| Path | Purpose | | Path | Purpose |
|---|---| |---|---|
| `app/routers/` | HTTP route handlers (thin) | | `app/factory.py` | FastAPI app factory (lifespan, middleware, error handlers, routers) |
| `app/scanners/` | Token risk scanners | | `app/mount.py` | Single source of truth for router mounting |
| `app/databus/` | 30+ third-party data providers | | `app/core/` | Cross-cutting concerns (logging, errors, redis, http, auth, config) |
| `app/telegram_bot/` | Telegram bot command handlers | | `app/api/v1/` | Thin HTTP transport layer (public, auth, admin, x402, mcp) |
| `app/domains/` | Business logic domains: auth, billing, databus, news, reports, scanner, scanners, telegram, threat, token, tokens, wallet, x402 |
| `app/domains/scanners/` | Token risk scanners (IP — will move to rmi-ip in Phase 6) |
| `app/domains/databus/` | 30+ third-party data provider integrations |
| `app/domains/telegram/rugmunchbot/` | Telegram bot command handlers |
| `app/mcp/` | MCP manifest and tool manager | | `app/mcp/` | MCP manifest and tool manager |
| `app/facilitators/` | x402 payment verification | | `app/infra/` | External integrations (ollama, langfuse, vector stores, chain clients) |
| `app/wallet_memory/` | Wallet label + clustering subsystem | | `app/wallet_memory/` | Wallet label + clustering subsystem (IP — will move to rmi-ip in Phase 6) |
| `app/routers/` | Legacy HTTP route handlers (being migrated to app/domains/ or app/api/v1/) |
## Storage ## Storage
| Store | Use | Container / Host | | Store | Use | Container / Host |
|---|---|---| |---|---|---|
| Postgres | Primary OLTP, Ponder indexer data | `rmi-postgres` | | Postgres | Primary OLTP, Ponder indexer data | `rmi-postgres` |
| Redis | Cache, Celery queue | `rmi-redis` | | Redis | Cache, Celery queue | `rmi-redis` |
| Qdrant | Vector embeddings | `rmi-qdrant` (down) | | Qdrant | Vector embeddings | `rmi-qdrant` |
| Neo4j | Wallet graph / labels | `rmi-neo4j` | | Neo4j | Wallet graph / labels | `rmi-neo4j` |
| ClickHouse | Analytics, 39M address labels | `rmi-clickhouse` | | ClickHouse | Analytics, 39M address labels | `rmi-clickhouse` |
## Interfaces ## Interfaces
| Endpoint | Purpose | | Endpoint | Purpose |
|---|---| |---|---|
| `GET /health` | Health check (currently unhealthy due to Redis) | | `GET /health` | Health check (degraded — ETH RPC endpoint missing) |
| `GET /metrics` | Prometheus metrics |
| `GET /docs` | Swagger/OpenAPI docs | | `GET /docs` | Swagger/OpenAPI docs |
| `POST /scanner/scan` | Live token scan | | `POST /scanner/scan` | Live token scan |
| `POST /mcp/...` | MCP tool endpoints | | `POST /mcp/...` | MCP tool endpoints |
## Current issues ## Current issues
- `x402_tools.py` (212 KB) and `x402_enforcement.py` (108 KB) violate the 500-line limit and should become separate services or split by domain. - `app/routers/x402_tools.py` (212 KB) and `app/routers/x402_enforcement.py` (108 KB) violate the 500-line limit and should become separate services or split by domain.
- `sqlite3` is used in 5 files for local state; should migrate to Postgres/aiosqlite. - `sqlite3` is used in 5 files for local state; should migrate to Postgres/aiosqlite.
- F-string logs and `print()` statements prevent structured log aggregation. - F-string logs and `print()` statements prevent structured log aggregation.
- `mypy` has a structural issue with `app/domains/auth/` that needs resolution.
## Deployment ## Deployment
```bash ```bash

View file

@ -1,7 +1,7 @@
# STATUS -- rmi-backend # STATUS -- rmi-backend
**Owner:** Rug Munch Media LLC Engineering **Owner:** Rug Munch Media LLC Engineering
**Last updated:** 2026-07-06 -- Phase 1 of [AUDIT-2026-Q3.md](AUDIT-2026-Q3.md) complete. **Last updated:** 2026-07-07 -- Phase 4 domain consolidation in progress; P4.1-P4.8 + P5.1 complete.
## What This Repo Is ## What This Repo Is
@ -13,7 +13,7 @@ Stack: Python 3.12 / FastAPI / SQLAlchemy / async / Playwright / Pydantic v2 / R
## Current Status ## Current Status
**Phase 1 (development hygiene) complete.** Phase 2 (delete dead code) starts next per [AUDIT-2026-Q3.md](AUDIT-2026-Q3.md). **Phase 4 (domain consolidation) is active.** Foundation (Phase 1) is solid, dead-router wiring (Phase 2) is partial, god-file splitting (Phase 3) is deferred, and standardization (Phase 5) has begun with CI trustworthiness and mypy config cleanup.
### Phase 1 -- DONE ### Phase 1 -- DONE
@ -30,16 +30,44 @@ Committed to `main`:
- `92a01ff` + `5294983` -- `AppError` hierarchy in `app/core/errors.py`, wired through `error_handlers.py` - `92a01ff` + `5294983` -- `AppError` hierarchy in `app/core/errors.py`, wired through `error_handlers.py`
- `cd02714` + `2fb571e` -- author amend (canonical identity: `cryptorugmunch <admin@rugmunch.io>`) - `cd02714` + `2fb571e` -- author amend (canonical identity: `cryptorugmunch <admin@rugmunch.io>`)
### What Phase 1 DID NOT do (deferred) ### Phase 2 -- IN PROGRESS (partial)
Rather than deleting 148 routers, we validated and mounted the useful ones in `app/mount.py` Wave 1-3 (2026-07-07). Dead routers remain in tree and will be archived in a later pass.
### Phase 3 -- NOT STARTED
God-files (`x402_tools.py`, `x402_enforcement.py`, `auth.py`, `databus/providers.py`, `telegram_bot/bot.py`, `wallet_manager_v2.py`) still await splitting.
### Phase 4 -- IN PROGRESS
Domain consolidation (AUDIT-2026-Q3 P4.x) — committed to `main`:
- `dca458e` -- P4.1 billing (`app/billing/` + `app/facilitators/``app/domains/billing/`)
- `7109a16` -- P4.2 wallet (`app/wallet/``app/domains/wallet/`)
- `948e41c` -- P4.3 auth (`app/auth/``app/domains/auth/`)
- `cab9740` -- P4.4 tokens (`app/tokens/``app/domains/tokens/`)
- `ed5b830` -- P4.5 databus (`providers/` + `_generated/``app/domains/databus/`)
- `56075cf` -- P4.6 telegram (`rugmunchbot/``app/domains/telegram/rugmunchbot/`)
- `3b7ef42` -- P4.7 rename `app/domain/``app/domains/` + consolidate
- `7cced4e` + `4686cb3` -- P4.8 scanners (`app/scanners/``app/domains/scanners/`)
Remaining Phase 4:
- Finish databus root migration (`app/databus/` engine → `app/domains/databus/`)
- Finish telegram migration (`app/telegram_bot/``app/domains/telegram/`)
- Clean up deprecated shim dirs (`app/auth/`, `app/wallet/`, `app/billing/`, `app/facilitators/`, `app/scanners/`, `app/domain/`)
- Consolidate bulletin, intelligence, markets, admin, referral, MCP domains
### Phase 5 -- STARTED
- `d666ad2` -- P5.1 `HEALTH_CHECK_DURATION` + `test_factory_has_minimum_routes`
- `4686cb3` -- mypy config fixes (`mypy.ini` regex, `disallow_any_explicit` typo, `app.domains.*` module pattern)
- Still open: alembic, real coverage gate, structured logging, Sentry wiring, Prometheus middleware
### What Phase 1-4 DID NOT do (deferred)
- ~2K ruff warnings in legacy code (Phase 5) - ~2K ruff warnings in legacy code (Phase 5)
- Real coverage 8.17% -> 80% gate (Phase 5, slowest part) - Real coverage 8.17% -> 80% gate (Phase 5)
- `mypy.ini` line 8 parse error + `disallow_any_express_imports` typo (Phase 5)
- Alembic migrations (still MISSING; Phase 5) - Alembic migrations (still MISSING; Phase 5)
- 285 dead modules (Phase 2)
- 148 unmounted routers (Phase 2)
- God-files split (Phase 3) - God-files split (Phase 3)
- Domain consolidation (Phase 4)
### Open Issues ### Open Issues
@ -68,8 +96,11 @@ e404e90 feat(rmi-backend,audit): add app/main.py entrypoint delegating to factor
- `gitleaks detect` reports 28 findings -- all **public ERC-20/Solana token contract addresses** (USDC, WETH, MATIC, BONK, DAI, plus cryptoscam-address dataset). NOT secrets. Suppressed via `.gitleaksignore` + `.gitleaks.toml` `[allowlist]`. No rotation needed. - `gitleaks detect` reports 28 findings -- all **public ERC-20/Solana token contract addresses** (USDC, WETH, MATIC, BONK, DAI, plus cryptoscam-address dataset). NOT secrets. Suppressed via `.gitleaksignore` + `.gitleaks.toml` `[allowlist]`. No rotation needed.
- All secrets live in gopass under `rmi/contacts/admin_recovery` etc. - All secrets live in gopass under `rmi/contacts/admin_recovery` etc.
## Next: Phase 2 (week 2) ## Next: Phase 4 completion
Archive **148 unmounted routers** + **~140 dead `app/*.py` files** to `app/_archive/legacy_2026_07/`. Goal: ~30% LOC reduction without changing any current behavior. 1. Finish `app/databus/` engine migration to `app/domains/databus/`.
2. Finish `app/telegram_bot/` migration to `app/domains/telegram/rugmunchbot/`.
3. Remove deprecated shim directories (`app/auth/`, `app/wallet/`, `app/billing/`, `app/facilitators/`, `app/scanners/`, `app/domain/`).
4. Consolidate bulletin, intelligence, markets, admin, referral, MCP into `app/domains/`.
See [AUDIT-2026-Q3.md](AUDIT-2026-Q3.md) for the full 6-phase plan. See [AUDIT-2026-Q3.md](AUDIT-2026-Q3.md) for the full 6-phase plan.