From 757eefd227461ac25edca2c8209e8553886ad1dc Mon Sep 17 00:00:00 2001 From: cryptorugmunch Date: Tue, 7 Jul 2026 04:47:09 +0700 Subject: [PATCH] docs(state): update STATUS.md and ARCHITECTURE.md for Phase 4 progress --- ARCHITECTURE.md | 32 +++++++++++++++++++------------ STATUS.md | 51 +++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 61 insertions(+), 22 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index be1609b..d339444 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,10 +1,11 @@ # RMI Backend — Architecture **Status:** canonical -**Last updated:** 2026-07-03 +**Last updated:** 2026-07-07 ## Overview FastAPI monolith serving crypto risk intelligence via HTTP, Telegram, and MCP. +Domains are consolidated under `app/domains/` per AUDIT-2026-Q3 Phase 4. ## High-level flow ``` @@ -12,9 +13,9 @@ Client (Web / Telegram / AI Agent) ↓ nginx → rmi-backend container (:8000) ↓ -FastAPI routers +FastAPI routers (mounted by app/mount.py) ↓ -Services / scanners / DataBus providers +app/domains//service.py → app/infra/ + app/domains//repository.py ↓ Postgres / Redis / Qdrant / Neo4j / ClickHouse ``` @@ -22,35 +23,42 @@ Postgres / Redis / Qdrant / Neo4j / ClickHouse ## Components | Path | Purpose | |---|---| -| `app/routers/` | HTTP route handlers (thin) | -| `app/scanners/` | Token risk scanners | -| `app/databus/` | 30+ third-party data providers | -| `app/telegram_bot/` | Telegram bot command handlers | +| `app/factory.py` | FastAPI app factory (lifespan, middleware, error handlers, routers) | +| `app/mount.py` | Single source of truth for router mounting | +| `app/core/` | Cross-cutting concerns (logging, errors, redis, http, auth, config) | +| `app/api/v1/` | Thin HTTP transport layer (public, auth, admin, x402, mcp) | +| `app/domains/` | Business logic domains: auth, billing, databus, news, reports, scanner, scanners, telegram, threat, token, tokens, wallet, x402 | +| `app/domains/scanners/` | Token risk scanners (IP — will move to rmi-ip in Phase 6) | +| `app/domains/databus/` | 30+ third-party data provider integrations | +| `app/domains/telegram/rugmunchbot/` | Telegram bot command handlers | | `app/mcp/` | MCP manifest and tool manager | -| `app/facilitators/` | x402 payment verification | -| `app/wallet_memory/` | Wallet label + clustering subsystem | +| `app/infra/` | External integrations (ollama, langfuse, vector stores, chain clients) | +| `app/wallet_memory/` | Wallet label + clustering subsystem (IP — will move to rmi-ip in Phase 6) | +| `app/routers/` | Legacy HTTP route handlers (being migrated to app/domains/ or app/api/v1/) | ## Storage | Store | Use | Container / Host | |---|---|---| | Postgres | Primary OLTP, Ponder indexer data | `rmi-postgres` | | Redis | Cache, Celery queue | `rmi-redis` | -| Qdrant | Vector embeddings | `rmi-qdrant` (down) | +| Qdrant | Vector embeddings | `rmi-qdrant` | | Neo4j | Wallet graph / labels | `rmi-neo4j` | | ClickHouse | Analytics, 39M address labels | `rmi-clickhouse` | ## Interfaces | Endpoint | Purpose | |---|---| -| `GET /health` | Health check (currently unhealthy due to Redis) | +| `GET /health` | Health check (degraded — ETH RPC endpoint missing) | +| `GET /metrics` | Prometheus metrics | | `GET /docs` | Swagger/OpenAPI docs | | `POST /scanner/scan` | Live token scan | | `POST /mcp/...` | MCP tool endpoints | ## Current issues -- `x402_tools.py` (212 KB) and `x402_enforcement.py` (108 KB) violate the 500-line limit and should become separate services or split by domain. +- `app/routers/x402_tools.py` (212 KB) and `app/routers/x402_enforcement.py` (108 KB) violate the 500-line limit and should become separate services or split by domain. - `sqlite3` is used in 5 files for local state; should migrate to Postgres/aiosqlite. - F-string logs and `print()` statements prevent structured log aggregation. +- `mypy` has a structural issue with `app/domains/auth/` that needs resolution. ## Deployment ```bash diff --git a/STATUS.md b/STATUS.md index 0377364..5272908 100644 --- a/STATUS.md +++ b/STATUS.md @@ -1,7 +1,7 @@ # STATUS -- rmi-backend **Owner:** Rug Munch Media LLC Engineering -**Last updated:** 2026-07-06 -- Phase 1 of [AUDIT-2026-Q3.md](AUDIT-2026-Q3.md) complete. +**Last updated:** 2026-07-07 -- Phase 4 domain consolidation in progress; P4.1-P4.8 + P5.1 complete. ## What This Repo Is @@ -13,7 +13,7 @@ Stack: Python 3.12 / FastAPI / SQLAlchemy / async / Playwright / Pydantic v2 / R ## Current Status -**Phase 1 (development hygiene) complete.** Phase 2 (delete dead code) starts next per [AUDIT-2026-Q3.md](AUDIT-2026-Q3.md). +**Phase 4 (domain consolidation) is active.** Foundation (Phase 1) is solid, dead-router wiring (Phase 2) is partial, god-file splitting (Phase 3) is deferred, and standardization (Phase 5) has begun with CI trustworthiness and mypy config cleanup. ### Phase 1 -- DONE @@ -30,16 +30,44 @@ Committed to `main`: - `92a01ff` + `5294983` -- `AppError` hierarchy in `app/core/errors.py`, wired through `error_handlers.py` - `cd02714` + `2fb571e` -- author amend (canonical identity: `cryptorugmunch `) -### What Phase 1 DID NOT do (deferred) +### Phase 2 -- IN PROGRESS (partial) + +Rather than deleting 148 routers, we validated and mounted the useful ones in `app/mount.py` Wave 1-3 (2026-07-07). Dead routers remain in tree and will be archived in a later pass. + +### Phase 3 -- NOT STARTED + +God-files (`x402_tools.py`, `x402_enforcement.py`, `auth.py`, `databus/providers.py`, `telegram_bot/bot.py`, `wallet_manager_v2.py`) still await splitting. + +### Phase 4 -- IN PROGRESS + +Domain consolidation (AUDIT-2026-Q3 P4.x) — committed to `main`: +- `dca458e` -- P4.1 billing (`app/billing/` + `app/facilitators/` → `app/domains/billing/`) +- `7109a16` -- P4.2 wallet (`app/wallet/` → `app/domains/wallet/`) +- `948e41c` -- P4.3 auth (`app/auth/` → `app/domains/auth/`) +- `cab9740` -- P4.4 tokens (`app/tokens/` → `app/domains/tokens/`) +- `ed5b830` -- P4.5 databus (`providers/` + `_generated/` → `app/domains/databus/`) +- `56075cf` -- P4.6 telegram (`rugmunchbot/` → `app/domains/telegram/rugmunchbot/`) +- `3b7ef42` -- P4.7 rename `app/domain/` → `app/domains/` + consolidate +- `7cced4e` + `4686cb3` -- P4.8 scanners (`app/scanners/` → `app/domains/scanners/`) + +Remaining Phase 4: +- Finish databus root migration (`app/databus/` engine → `app/domains/databus/`) +- Finish telegram migration (`app/telegram_bot/` → `app/domains/telegram/`) +- Clean up deprecated shim dirs (`app/auth/`, `app/wallet/`, `app/billing/`, `app/facilitators/`, `app/scanners/`, `app/domain/`) +- Consolidate bulletin, intelligence, markets, admin, referral, MCP domains + +### Phase 5 -- STARTED + +- `d666ad2` -- P5.1 `HEALTH_CHECK_DURATION` + `test_factory_has_minimum_routes` +- `4686cb3` -- mypy config fixes (`mypy.ini` regex, `disallow_any_explicit` typo, `app.domains.*` module pattern) +- Still open: alembic, real coverage gate, structured logging, Sentry wiring, Prometheus middleware + +### What Phase 1-4 DID NOT do (deferred) - ~2K ruff warnings in legacy code (Phase 5) -- Real coverage 8.17% -> 80% gate (Phase 5, slowest part) -- `mypy.ini` line 8 parse error + `disallow_any_express_imports` typo (Phase 5) +- Real coverage 8.17% -> 80% gate (Phase 5) - Alembic migrations (still MISSING; Phase 5) -- 285 dead modules (Phase 2) -- 148 unmounted routers (Phase 2) - God-files split (Phase 3) -- Domain consolidation (Phase 4) ### Open Issues @@ -68,8 +96,11 @@ e404e90 feat(rmi-backend,audit): add app/main.py entrypoint delegating to factor - `gitleaks detect` reports 28 findings -- all **public ERC-20/Solana token contract addresses** (USDC, WETH, MATIC, BONK, DAI, plus cryptoscam-address dataset). NOT secrets. Suppressed via `.gitleaksignore` + `.gitleaks.toml` `[allowlist]`. No rotation needed. - All secrets live in gopass under `rmi/contacts/admin_recovery` etc. -## Next: Phase 2 (week 2) +## Next: Phase 4 completion -Archive **148 unmounted routers** + **~140 dead `app/*.py` files** to `app/_archive/legacy_2026_07/`. Goal: ~30% LOC reduction without changing any current behavior. +1. Finish `app/databus/` engine migration to `app/domains/databus/`. +2. Finish `app/telegram_bot/` migration to `app/domains/telegram/rugmunchbot/`. +3. Remove deprecated shim directories (`app/auth/`, `app/wallet/`, `app/billing/`, `app/facilitators/`, `app/scanners/`, `app/domain/`). +4. Consolidate bulletin, intelligence, markets, admin, referral, MCP into `app/domains/`. See [AUDIT-2026-Q3.md](AUDIT-2026-Q3.md) for the full 6-phase plan.