Commit graph

28 commits

Author SHA1 Message Date
opencode
5294983084 refactor(rmi-backend,audit): wire error_handlers.py to AppError at import time (P1.10 wiring)
Some checks failed
CI / build (push) Failing after 2s
Phase 1 of AUDIT-2026-Q3.md item P1.10 wiring.

Previously _register_apperror did:
    try:
        from app.core.errors import AppError
        @app.exception_handler(AppError)
        async def _app_error(...): ...
    except Exception as exc:
        log.info("error_handler_skipped name=AppError err=%s", exc)

That meant: if AppError was missing (it was — see previous commit), the
import silently failed and the app booted with ZERO typed-error handling.
All 2,532 except Exception: blocks would surface to clients as opaque 500s.

This change:
- Imports AppError (+ 4 commonly-raised subclasses) at module top so a
  missing class is a hard ImportError at import time.
- Drops the try/except wrapper around _register_apperror.
- Renames the inner handlers to handle_app_error / handle_generic_exception
  for clarity.
- handle_app_error returns exc.to_dict() (code, message, context) +
  path + type, status_code = exc.status_code. The old code referenced
  exc.message which would AttributeError since AppError stores message
  via super().__init__, not as a self attribute.
- Drops the no-op try/except/pass in _register_validation (was a stub;
  Phase 3 of the audit will wire RequestValidationError -> RMI ValidationError).

Verified via FastAPI TestClient:
  GET /a    AuthError       -> 401  {code: auth.unauthorized, ...}
  GET /r    RateLimitError  -> 429  {code: ratelimit.exceeded, ...}
  GET /v    ValidationError -> 400  {code: validation.failed, ...}
  GET /n    NotFoundError   -> 404  {code: not_found, ...}
  GET /u    UpstreamError   -> 502  {code: upstream.failed, ...}
  GET /x    RuntimeError    -> 500  {error: internal_error, ...}  (generic handler)
  GET /404                  -> 404  {error: not_found, hint, ...}  (404 handler)

Refs AUDIT-2026-Q3.md P1.10.
2026-07-06 17:58:08 +02:00
opencode
92a01ffba0 feat(rmi-backend,audit): AppError hierarchy in app/core/errors.py (P1.10)
Some checks failed
CI / build (push) Failing after 2s
Phase 1 of AUDIT-2026-Q3.md item P1.10.

Codebase had 2,532 except Exception: blocks and zero AppError class.
Bare exceptions made debugging in production impossible.

This change adds a strict typed hierarchy on top of the existing
register_error_handlers() (which app/lifespan.py imports):

  AppError (base, status 500, code internal_error)
    AuthError        401  auth.unauthorized
    RateLimitError   429  ratelimit.exceeded
    ValidationError  400  validation.failed    (RMI-level, not pydantic)
    NotFoundError    404  not_found
    UpstreamError    502  upstream.failed

Each carries status_code (HTTP), code (machine-readable), and a
context dict that handlers forward to Sentry without code changes.

register_error_handlers() also now installs an AppError handler that
returns exc.to_dict() with a request_id, in addition to the existing
StarletteHTTPException / ValueError / Exception handlers.

Refs AUDIT-2026-Q3.md P1.10.
2026-07-06 17:56:36 +02:00
opencode
da2696a264 build(rmi-backend,audit): declare app/ as installable Python package
Some checks failed
CI / build (push) Failing after 2s
Phase 1 of AUDIT-2026-Q3.md item P1.3.

33 of 35 unit test files fail pytest collection without PYTHONPATH=.
Because pyproject.toml did not declare app/ as a package, pip install -e
".[dev]" did not register app as importable. After this commit:
- pip install -e .[dev] installs app as a real package
- pytest tests/ --collect-only works from any cwd without PYTHONPATH=
- New devs can follow README.md verbatim (no PYTHONPATH= env var)

Used [tool.setuptools.packages.find] with include = ["app*"] and
namespaces = false to auto-discover every sub-package that has __init__.py
(414 sub-packages). Sub-packages without __init__.py are namespace
packages and remain out of scope for Phase 1.3 (tracked for Phase 3):
  app/middleware, app/telegram_bot, app/routers, app/domain/threat,
  app/api/v1/rag, app/protection_api, app/services, app/mcp/tools.

Verified: from /tmp without PYTHONPATH, "import app" works and
"pytest tests/ --collect-only" reports 765 tests collected. The 8
remaining collection errors are pre-existing missing optional deps
(aiohttp in app/profile_flip_detector, app/bridge_health_monitor;
numpy in app/rag/ann_index) — unrelated to this fix.
2026-07-06 17:54:29 +02:00
opencode
9c62549b50 fix(rmi-backend,audit): jwt_secret is required, fail-fast in prod (P1.5)
Some checks failed
CI / build (push) Failing after 2s
Previously: jwt_secret defaulted to dev-secret-CHANGE-ME so a missing
env var in production would silently boot with a known-public dev HMAC
key — an instant auth bypass for any service verifying JWTs.

Now:
- Field(...) with no default → missing env raises ValidationError at boot
- min_length=32 enforces minimum entropy (rejects dev placeholder too)
- field_validator rejects the literal dev-secret-CHANGE-ME in ENVIRONMENT=prod
  as belt-and-suspenders in case min_length is loosened later

Verified behavior:
- no env, no .env                 -> ValidationError: Field required
- ENVIRONMENT=prod, no JWT_SECRET -> ValidationError: Field required
- prod, JWT_SECRET=dev-secret-... -> ValidationError: string_too_short
- prod, JWT_SECRET=short          -> ValidationError: string_too_short
- prod, JWT_SECRET=<64 hex>       -> OK, len=64

.env.example updated to show the placeholder + generation hint.

Refs AUDIT-2026-Q3.md P1.5.
2026-07-06 17:53:49 +02:00
opencode
f4d42768a2 chore(rmi-backend,audit): delete ruff.toml — pyproject.toml [tool.ruff] is canonical
Phase 1 of AUDIT-2026-Q3.md item P1.8.

ruff.toml was overriding pyproject.toml [tool.ruff] silently:
  ruff.toml:    line-length 120, ignored SIM102/RUF006/B008
  pyproject.toml: line-length 100, security rules enabled (canonical)

Two configs contradicting each other silently is the worst-case
configuration — neither developer nor CI notice, and the strict
[tool.ruff] security rules (S) never run.

Deleted ruff.toml. Added defensive .gitignore entry. ruff now reads
only pyproject.toml. Verified with ruff check --show-settings:
  linter.line_length = 100
  linter.pycodestyle.max_line_length = 100

ruff.toml moved to /tmp/rmi-backend-archive-2026-07/ruff.toml.removed-2026-07-06.
2026-07-06 17:53:03 +02:00
opencode
e0d0ae3bfd chore(rmi-backend,audit): gitignore main.py.bak
Phase 1 of AUDIT-2026-Q3.md item P1.4.

main.py.bak was a 374KB legacy leftover from the old 220-file app/
monolith. Its removal from the working tree was completed in e404e90
(feat(rmi-backend,audit): add app/main.py entrypoint delegating to
factory) in this audit session — that commit deleted main.py.bak and
added app/main.py in a single change.

This commit adds the defensive .gitignore entry so any future
main.py.bak / similar leftover files do not get re-tracked.

main.py.bak archive kept at:
  /tmp/rmi-backend-archive-2026-07/main.py.bak.removed-2026-07-06

Note: historical commit bde2f3a still contains 4 mainnet token
addresses for gitleaks; full rewrite with git-filter-repo is scheduled
for Phase 1.11 of AUDIT-2026-Q3.md.
2026-07-06 17:52:51 +02:00
opencode
c1d157ac79 test(rmi-backend,audit): add tests __init__.py for package discovery
Some checks failed
CI / build (push) Failing after 2s
Phase 1 of AUDIT-2026-Q3.md item P1.2.

tests/ was not a Python package, so downstream imports
(`from tests.unit.x import y`, testcontainers config loading,
pytest --import-mode=importlib) failed without PYTHONPATH=.
Creating __init__.py in tests/, tests/unit/, tests/integration/
makes the suite a proper package. pytest collection already worked
via rootdir discovery (886 tests collected either way), but
*importing* test modules will now work.

Also seeds tests/integration/conftest.py with a Phase-1 stub
fixture. Phase 3 (testcontainers) replaces it.

Full fix lands in P1.3 (setuptools.packages = ["app"] +
tools.setuptools.package-data in pyproject.toml) — without that,
`pip install -e .` does not expose the app package.
2026-07-06 17:50:36 +02:00
opencode
e404e90c1a feat(rmi-backend,audit): add app/main.py entrypoint delegating to factory
Phase 1 of AUDIT-2026-Q3.md item P1.1.

app/main.py was referenced by Makefile, AGENTS.md, CONTRIBUTING.md,
and external `uvicorn app.main:app` invocations, but the file did not
exist (refactored to app/factory.py in a prior session — the docs were
never updated). This restores the canonical entrypoint.
2026-07-06 17:49:25 +02:00
opencode
1c815c07e4 docs(audit): add 2026-Q3 audit + 6-phase production-grade roadmap (master ref for rmi-backend/frontend)
Some checks failed
CI / build (push) Failing after 3s
Comprehensive audit covering:
- 649 files, 241k LOC, 21 god-files over 1000 LOC
- 148 unmounted routers (21 mounted, 164 defined)
- 11 circular import cycles
- 0 alembic migrations
- 2532 except-Exception swallows, 0 AppError class
- 60% coverage gate unenforced (actual 8.17%)
- GitHub CI has || true on every job (decorative)
- Pre-commit configured but not installed
- 28 gitleaks findings (incl main.py.bak with mainnet addresses)
- 285 dead modules
- 7 hallucinated files in RMI_SYSTEM_MAP.md

Plan: 6 phases, 5-6 weeks, 1-2 engineers:
1. Stop the bleeding (1 week)
2. Delete dead code (1 week)
3. Split god-files (2 weeks)
4. Consolidate scattered domains (1.5 weeks)
5. Standardize + harden (1 week)
6. Open-source pivot to 3 repos (1 week, parallel with 5): rmi-core (MIT) / rmi-ip (BSL 1.1) / rmi-pro (private)

Open-source vs proprietary split recommendation included for each module category.
2026-07-06 17:44:58 +02:00
862fd05e08 Merge pull request 'chore: lint cleanup 1175→0 + auth.py fix + drop passlib' (#3) from chore/lint-debt-cleanup into main
Some checks failed
CI / build (push) Failing after 3s
2026-07-06 16:47:11 +02:00
opencode
1171a35d9f fix(rmi-backend): drop passlib — route backup codes through direct bcrypt
Some checks failed
CI / build (pull_request) Failing after 2s
The reported symptom (AttributeError: module 'bcrypt' has no attribute
'__about__') was misleading. passlib 1.7.4 (the latest released version
— the project is unmaintained) actually traps that AttributeError with
a bare `except:` and continues. The real failure occurs later in
`passlib.handlers.bcrypt._finalize_backend_mixin`:

  detect_wrap_bug -> verify(secret, bug_hash)
    where secret = (b"0123456789"*26)[:255]   # 255-byte test secret

bcrypt 4.0+ removed silent 72-byte truncation and now raises
`ValueError: password cannot be longer than 72 bytes`, breaking the
backend probe and crashing every `pwd_context.hash()/verify()` call.

Three options were considered:

  A. Upgrade passlib — impossible, 1.7.4 is the latest PyPI release.
  B. Pin bcrypt<4.0  — conflicts with chromadb 1.1.1 (>=4.0.1 required),
     even though chromadb's bcrypt usage is optional and still works
     at runtime in practice.
  C. Shim bcrypt.__about__ — doesn't fix the actual truncation error.

Chosen: refactor app/auth.py to use bcrypt directly for backup-code
hashing too (the main `hash_password`/`verify_password` already did).
Drops the passlib dependency entirely, restoring bcrypt 5.x for chromadb
compatibility and eliminating the unmaintained passlib pin.

Verified:
  * hash_password / verify_password round-trip OK
  * _hash_backup_code / _verify_backup_code round-trip OK on
    freshly generated 8-digit backup codes
  * `pytest tests/unit` -> 717 passed, 32 warnings, 60 subtests passed
2026-07-06 16:17:47 +02:00
opencode
6b29227131 fix(rmi-backend): define missing User helpers + import CryptContext (fixes 23 F821 in app/auth.py) 2026-07-06 15:55:14 +02:00
opencode
c762564d40 style(rmi-backend): complete lint cleanup — 1175→0 ruff errors
- Fix 71 invalid-syntax files (class-body newline-broken assignments)
- Add from/None chain to 307 B904 raise-without-from sites
- Add B008 ignore to ruff.toml (already in pyproject.toml)
- Noqa F401 on __init__.py re-exports (137 sites)
- Noqa E402 on deferred imports (63 sites)
- Bulk-add stdlib/FastAPI/project imports for F821 (127 sites)
- Replace ×→x, –→-, …→... in docstrings (4093 chars)
- Manual refactor of 5 SIM103/SIM116 patterns

Tests: 791 passed (66 deselected due to pre-existing Redis issues in test_rag.py)
Co-authored-by: opencode <opencode@rugmunch.io>
2026-07-06 15:43:20 +02:00
ca9bdce365 docs(rmi-backend): add LINT-CLEANUP-REPORT.md tracking fix 2026-07-06 15:43:09 +02:00
5c6b797087 Merge pull request 'chore(rmi-backend): apply fleet-template standards (editorconfig, gitattributes, IP fixes, CI gate)' (#2) from chore/fleet-template-standards into main
Some checks failed
CI / build (push) Failing after 2s
2026-07-06 14:21:45 +02:00
07313ecac1 chore(rmi-backend): apply fleet-template standards — add .editorconfig + .gitattributes, fix stale IPs, tighten qdrant-audit CI gate
Some checks failed
CI / build (pull_request) Failing after 3s
- Add .editorconfig (Python 4-space, default 2, LF, UTF-8) per fleet-template
- Add .gitattributes (LF normalization, lockfile binary, generated paths)
- Fix AGENTS.md:35 — netcup (100.100.18.18) → Talos (100.104.130.92)
- Mark TODO.md:5 done — Qdrant stale IP (fixed in 3c6b295)
- Remove continue-on-error: true from qdrant-cleanup job (inner step has || \, no behavior change)

Refs: standards/GAPS.md#4 (Make CI gates authoritative), standards/GAPS.md#9 (Repo tooling parity), standards/GAPS.md#12 (Documentation contradictions)
2026-07-06 14:18:34 +02:00
1e71d4ae4e Merge pull request 'security(gitleaks): ignore scammer Telegram tokens in CryptoScamDB data' (#1) from security/gitleaks-ignore-scam-tokens into main
Some checks failed
CI / build (push) Failing after 2s
2026-07-03 17:42:12 +02:00
880389f37b security(gitleaks): ignore scammer Telegram tokens in CryptoScamDB data
Some checks failed
CI / build (pull_request) Failing after 2s
2026-07-03 17:40:55 +02:00
3c6b29563f fix(health): use fresh httpx client per qdrant check to avoid stale connection pool
Some checks failed
CI / build (push) Failing after 10s
2026-07-03 17:07:20 +02:00
01cb548f8f fix(health): use env vars for qdrant/clickhouse/minio/reth instead of localhost
Some checks are pending
CI / build (push) Waiting to run
2026-07-03 16:43:30 +02:00
bbb9a9291d docs: rewrite README and add PRODUCT, ARCHITECTURE, ROADMAP, TODO per governance framework 2026-07-03 15:56:26 +02:00
b2cdfce4cd security(status_page): remove hardcoded ClickHouse password fallback
Some checks failed
CI / build (push) Failing after 2s
2026-07-02 22:37:41 +02:00
f932ac4e1e security(rmi-backend): remove hardcoded API keys, env-reference instead
Gitleaks flagged 4 production secrets in rmi-backend source:

  app/caching_shield/solana_tracker.py:    st_ZMzXzdUI54TXQPx5E6JkG
  app/databus/arkham_ws.py:                ws_Z5x09Rcr_1780418740765322917
  app/routers/webhooks_router.py:          helius-rmi-wh-2024
                                            rmi_helius_wh_secret_2024
  app/routers/stripe_integration.py:       pk_test_51Tn13MAXseReicQtM...

Each replaced with os.getenv() reads. Placeholder env lines added to
/srv/rmi-infra/.env.secrets. Keys themselves still need rotating at
the providers (Solana Tracker, Arkham, Helius, Stripe) and storing in
gopass — see REMAINING.md.

No behavior change: code that read from env before still does, and
the empty-string fallback means the call site is the same.

Note: this commit scrubs the keys from the working tree. They remain
in git history. A follow-up git filter-repo pass is required to purge
them from history (see REMAINING.md). After that, all clones and
external remotes must be force-updated.
2026-07-02 21:34:33 +02:00
root
f9f977de87 Merge branch 'feat/issue-forms' 2026-07-02 15:11:00 +02:00
root
3c7d1638f2 chore: add Makefile + .env.example for rmi-backend
Makefile: standard targets (install/dev/build/lint/format/check/
typecheck/test/test-all/security/ci/clean) matching fleet convention.

.env.example: documents all env vars used by the backend:
- Runtime, Database, Auth, CORS, Rate limiting
- AI providers (Ollama, OpenRouter, DeepSeek, HuggingFace)
- Langfuse observability
- External APIs (CoinGecko, Etherscan, Birdeye, GoPlus, Basescan)
- Supabase, Telegram bot, Scan limits, Apify, RAG
2026-07-02 14:52:19 +02:00
root
aee1c048b6 ci: add .forgejo/workflows/ci.yml for rmi-backend
Python CI: uv setup, ruff lint+format, mypy (warn), pytest (non-integration).
Runs on docker-x64 runner on Talos.
44 tests exist, now they will run in CI.
2026-07-02 14:46:32 +02:00
root
30c3c2f4d8 docs(issues): add YAML issue forms (bug, feature) + config
- .forgejo/ISSUE_TEMPLATE/bug.yml
- .forgejo/ISSUE_TEMPLATE/feature.yml
- .forgejo/ISSUE_TEMPLATE/config.yml: blank_issues_enabled=false,
  contact_links to discussions (Q&A, Feature Requests, Announcements)

Discussions per repo land in a separate follow-up commit.
2026-07-02 00:06:39 +02:00
bde2f3a97d merge: chore/cleanup-remove-bloat-and-secrets into main 2026-07-02 01:24:22 +07:00