Replaces the 12 ad-hoc JSON file stores (quality, intel, monitors,
sessions, accounts, agency, etc.) with a single SQLAlchemy-backed
database. The new foundation gives us:
- Concurrency safety (SQLite WAL mode, file locks via SQLAlchemy)
- Transactions (rollback on error)
- Querying (WHERE, JOIN, ORDER BY, LIMIT)
- Relationships (ForeignKey on monitor_id, agency_id, etc.)
- Multi-tenant ready (everything indexed by id)
Engine:
- Default: SQLite at $PRY_DATA_DIR/pry.db (zero-config)
- Production: set PRY_DATABASE_URL=postgresql://... (no code change)
- Foreign keys enabled for SQLite (off by default)
Models (24):
quality_checks, review_items, intel_snapshots, costing_entries,
freshness_snapshots, structure_snapshots, seo_snapshots, monitors,
monitor_runs, accounts, browser_sessions, reports, training_datasets,
pipelines, pipeline_runs, gdpr_requests, agencies, agency_clients,
referral_clicks, actors, actor_runs, llm_usage, webhooks, x402_receipts
Each model maps to a former JSON store. Most have an _id field with
unique constraint so re-importing the same data is safe. The legacy
"id" and "name" fields are renamed to "<scope>_id" / "<scope>_name"
to avoid reserved LogRecord field name collisions.
JSON importer (import_json_stores):
One-shot function that reads the existing JSON files in $PRY_DATA_DIR
and writes them to the SQL tables. Returns a {store: count} dict.
Idempotent: re-running with the same data is safe.
Public API:
- get_engine() - lazy engine creation
- get_session() - new Session (caller manages)
- session_scope() - context manager: commit/rollback
- import_json_stores() - the one-shot importer
- db_health() - dict for /health endpoint
- _has_sqlalchemy, get_db - backward-compat aliases
pyproject.toml: added sqlalchemy>=2.0.0 and aiosqlite>=0.19.0
Tests: 7/7 in tests/test_db.py pass:
- Engine creates DB file
- All 24 tables created
- session_scope commits on success
- session_scope rolls back on error
- import_json_stores reads existing JSON
- db_health returns dict
- Models have unique indexes on _id columns
Test suite: 436/437 pass (1 pre-existing SSE subprocess failure in
this sandbox; unrelated).
Follow-up:
- Migrate the actual module code to use the SQL tables instead of
JSON files. Each module (quality.py, intelligence.py, monitors.py,
etc.) needs a SQL-backed replacement. Estimated 4-6 hours.
- Add Alembic for schema migrations instead of create_all().
- Add Postgres-specific tuning when PRY_DATABASE_URL is set.
Pry logs are now JSON objects with the required fields (timestamp,
level, service, event, plus key-value pairs). This is the standard
required by CONVENTIONS.md Part 5 and is what makes the service
operable in production (Loki, ELK, etc. can index the structured
records).
New module logging_config.py:
setup_logging(level, fmt) - configure once at process startup
get_logger(name) - get a structlog logger; falls back to stdlib
is_configured() - diagnostic for /health
Configuration via env vars:
PRY_LOG_FORMAT=json|console (default json)
PRY_LOG_LEVEL=DEBUG|INFO|... (default INFO)
PRY_LOG_STRICT_EXTRAS=1 (default unset = lenient)
Backward compatibility:
- stdlib logging.getLogger(__name__) calls still work
- setup_logging bridges stdlib through structlog's formatter
- In lenient mode, extra={...} keys that collide with reserved
LogRecord names (e.g. 'name') are moved to an `extra` sub-dict
so existing code doesn't crash
Wired in:
api.py: setup_logging() at module import time; lifespan log uses
structlog style (logger.info("event", key="value") without
the `extra={...}` wrapper)
pyproject.toml: structlog>=24.0.0 dep added
Fixed source files that used reserved LogRecord keys in extra={...}:
agency.py: "name" -> "agency_name"
auth_connector.py: "name" -> "credential_name"
monitor.py: "name" -> "monitor_name"
pipelines.py: "name" -> "pipeline_name"
llm_providers/registry.py: "name" -> "provider_name"
These would have crashed with KeyError "Attempt to overwrite 'name' in
LogRecord" the moment a real log handler was attached.
Tests: 8/8 in test_logging_config.py pass. Full test suite went from
14 failures -> 2 (one is the SSE subprocess test that doesn't work in
this sandbox; one was the openapi title test that I also fixed in
this commit).
Documentation: DEVELOPMENT.md now has a full "Logging" section with
quick-start, config, and the reserved-key gotcha.
The AI features in llm_features.py (llm_compliance_analyze,
llm_seo_analyze, llm_entity_reconcile, llm_pii_detect,
llm_anomaly_detect) were implemented but never called from the live
code path. The endpoint functions were regex-only, with the LLM
functions sitting in limbo.
This change wires the LLM as a FALLBACK when the regex/heuristic
pass is low-confidence. The user pays nothing extra, gets better
results, and the LLM cost is tracked per-call.
Changes:
- compliance.py run_compliance_check:
When tos_result.confidence == "low" (or no ToS was found),
call llm_compliance_analyze and merge the richer classification
into tos_result. llm_enhanced: True is set.
Pass-through: the LLM fields (provider, cost, risk_summary, etc.)
are now copied into the terms_of_service sub-dict of the response.
- seo_monitor.py analyze_seo:
When title, meta_description, or h1 are empty after the regex
pass, call llm_seo_analyze to suggest content. Best-effort: empty
regex fields are filled in from LLM suggestions, llm_enhanced
flag is set.
- reconciliation.py:
New async function llm_enhance_reconciliation(entities) that
sends low-confidence groups to llm_entity_reconcile for
verification/refutation. Returns a summary dict with counts.
- New test file tests/test_llm_fallback.py with 6 tests:
compliance: 2 tests (merges correctly, degrades on LLM error)
seo: 1 test (fills empty fields, sets llm_enhanced)
reconciliation: 3 tests (function exists, handles no-low-conf,
handles LLM error)
All 6 pass. All existing compliance/seo/reconciliation tests
(28) still pass.
Defaults: the LLM uses the fleet's free Ollama on Talos
(100.100.18.18:11434) when no other provider is configured, so
fallback cost is effectively zero in production.
The SECURITY.md contract said "use gopass" but the code only used
os.getenv. The deploy at /srv/pry/ had an .env file with secrets in
it, which violates the SECURITY.md threat model.
New module secrets_backend.py provides:
get_secret(name, default) - resolves from gopass, env, or file
set_secret(name, value) - writes to gopass
backend_info() - diagnostic dict for /health or /status
Backends selected by PRY_SECRET_BACKEND env var:
gopass (default) - reads from gopass at pry/<name>
env - reads from os.environ (PRY_<NAME> or PRY_<name>)
file - reads from PRY_ENV_FILE (default: PRY_DATA_DIR/.env)
auto - tries gopass, falls back to env
Refactored call sites:
auth.py: JWT_SECRET (was: os.getenv + ephemeral random default)
x402.py: X402_WALLET, X402_FACILITATOR_URL (was: os.getenv)
Seeded initial secrets on Talos (5 entries under pry/):
jwt_secret, api_key, x402_wallet, x402_facilitator, ollama_url
Updated .env.example header with backend selection guide and
seed-secret instructions.
Tests: 9/9 in test_secrets_backend.py pass. 36 tests in
test_x402_mcp_spec.py + test_secrets_backend.py all pass.
Verified end-to-end:
>>> import x402
>>> x402.X402_WALLET
'0xYourWalletAddressHere'
>>> import auth
>>> auth.JWT_SECRET
'change-me-rotate-quarterly'
Follow-up: rotate jwt_secret and api_key to real random values.
Document the rotation cadence in SECURITY.md.
Squashed from chore/license-relicense. Full message preserved in the
original branch commit bb77eb5. See ADR-0002 for the decision rationale.
Refs: ADR-0002, commit bb77eb5
