Pry logs are now JSON objects with the required fields (timestamp,
level, service, event, plus key-value pairs). This is the standard
required by CONVENTIONS.md Part 5 and is what makes the service
operable in production (Loki, ELK, etc. can index the structured
records).
New module logging_config.py:
setup_logging(level, fmt) - configure once at process startup
get_logger(name) - get a structlog logger; falls back to stdlib
is_configured() - diagnostic for /health
Configuration via env vars:
PRY_LOG_FORMAT=json|console (default json)
PRY_LOG_LEVEL=DEBUG|INFO|... (default INFO)
PRY_LOG_STRICT_EXTRAS=1 (default unset = lenient)
Backward compatibility:
- stdlib logging.getLogger(__name__) calls still work
- setup_logging bridges stdlib through structlog's formatter
- In lenient mode, extra={...} keys that collide with reserved
LogRecord names (e.g. 'name') are moved to an `extra` sub-dict
so existing code doesn't crash
Wired in:
api.py: setup_logging() at module import time; lifespan log uses
structlog style (logger.info("event", key="value") without
the `extra={...}` wrapper)
pyproject.toml: structlog>=24.0.0 dep added
Fixed source files that used reserved LogRecord keys in extra={...}:
agency.py: "name" -> "agency_name"
auth_connector.py: "name" -> "credential_name"
monitor.py: "name" -> "monitor_name"
pipelines.py: "name" -> "pipeline_name"
llm_providers/registry.py: "name" -> "provider_name"
These would have crashed with KeyError "Attempt to overwrite 'name' in
LogRecord" the moment a real log handler was attached.
Tests: 8/8 in test_logging_config.py pass. Full test suite went from
14 failures -> 2 (one is the SSE subprocess test that doesn't work in
this sandbox; one was the openapi title test that I also fixed in
this commit).
Documentation: DEVELOPMENT.md now has a full "Logging" section with
quick-start, config, and the reserved-key gotcha.
The AI features in llm_features.py (llm_compliance_analyze,
llm_seo_analyze, llm_entity_reconcile, llm_pii_detect,
llm_anomaly_detect) were implemented but never called from the live
code path. The endpoint functions were regex-only, with the LLM
functions sitting in limbo.
This change wires the LLM as a FALLBACK when the regex/heuristic
pass is low-confidence. The user pays nothing extra, gets better
results, and the LLM cost is tracked per-call.
Changes:
- compliance.py run_compliance_check:
When tos_result.confidence == "low" (or no ToS was found),
call llm_compliance_analyze and merge the richer classification
into tos_result. llm_enhanced: True is set.
Pass-through: the LLM fields (provider, cost, risk_summary, etc.)
are now copied into the terms_of_service sub-dict of the response.
- seo_monitor.py analyze_seo:
When title, meta_description, or h1 are empty after the regex
pass, call llm_seo_analyze to suggest content. Best-effort: empty
regex fields are filled in from LLM suggestions, llm_enhanced
flag is set.
- reconciliation.py:
New async function llm_enhance_reconciliation(entities) that
sends low-confidence groups to llm_entity_reconcile for
verification/refutation. Returns a summary dict with counts.
- New test file tests/test_llm_fallback.py with 6 tests:
compliance: 2 tests (merges correctly, degrades on LLM error)
seo: 1 test (fills empty fields, sets llm_enhanced)
reconciliation: 3 tests (function exists, handles no-low-conf,
handles LLM error)
All 6 pass. All existing compliance/seo/reconciliation tests
(28) still pass.
Defaults: the LLM uses the fleet's free Ollama on Talos
(100.100.18.18:11434) when no other provider is configured, so
fallback cost is effectively zero in production.
The SECURITY.md contract said "use gopass" but the code only used
os.getenv. The deploy at /srv/pry/ had an .env file with secrets in
it, which violates the SECURITY.md threat model.
New module secrets_backend.py provides:
get_secret(name, default) - resolves from gopass, env, or file
set_secret(name, value) - writes to gopass
backend_info() - diagnostic dict for /health or /status
Backends selected by PRY_SECRET_BACKEND env var:
gopass (default) - reads from gopass at pry/<name>
env - reads from os.environ (PRY_<NAME> or PRY_<name>)
file - reads from PRY_ENV_FILE (default: PRY_DATA_DIR/.env)
auto - tries gopass, falls back to env
Refactored call sites:
auth.py: JWT_SECRET (was: os.getenv + ephemeral random default)
x402.py: X402_WALLET, X402_FACILITATOR_URL (was: os.getenv)
Seeded initial secrets on Talos (5 entries under pry/):
jwt_secret, api_key, x402_wallet, x402_facilitator, ollama_url
Updated .env.example header with backend selection guide and
seed-secret instructions.
Tests: 9/9 in test_secrets_backend.py pass. 36 tests in
test_x402_mcp_spec.py + test_secrets_backend.py all pass.
Verified end-to-end:
>>> import x402
>>> x402.X402_WALLET
'0xYourWalletAddressHere'
>>> import auth
>>> auth.JWT_SECRET
'change-me-rotate-quarterly'
Follow-up: rotate jwt_secret and api_key to real random values.
Document the rotation cadence in SECURITY.md.
Each module did:
X_DIR = Path(os.path.expanduser("~/.pry/x"))
After:
from paths import PRY_DATA_DIR
X_DIR = PRY_DATA_DIR / "x"
The module-level Path construction is preserved, so the rest of the
code is unchanged. PRY_DATA_DIR is read once at import (overridable via
the env var of the same name).
Verified:
- 407 tests collect (was 5 collection errors from a misplaced import)
- 83 sampled tests pass (intelligence, proxy_manager, x402, agency,
gdpr, referrals, marketplace, api)
- 0 remaining hardcoded ~/.pry references in .py files
Follow-up: paths.py adds subdir(name) helper for new code that wants
auto-mkdir; existing modules still call .mkdir(exist_ok=True) themselves
to preserve the eager-init behavior they had before.
The data root was hardcoded as Path(os.path.expanduser("~/.pry")) in
25+ modules, making it impossible to point Pry at a different data
directory (production systemd, Docker volumes, CI scratch, tests).
Changes:
- New module paths.py: single source of truth
PRY_DATA_DIR: Path # read once at import, overridable via env var
subdir(name) -> Path # mkdir+return helper
ensure_data_dir() -> Path # eager init
- 25 modules: replace
X_DIR = Path(os.path.expanduser("~/.pry/x"))
with
X_DIR = PRY_DATA_DIR / "x"
(plus the import; total 53 changes across 26 files)
- .env.example: document PRY_DATA_DIR with examples
- Verified:
- 407 tests collect (was 5 collection errors before fix)
- 83 sampled tests pass
- 0 remaining hardcoded ~/.pry references in py files
The deploy at /srv/pry/ had a thin proxy_referrals.py with 5 curated
proxy provider affiliate entries (Bright Data, Oxylabs, Smartproxy,
IPRoyal, Webshare). The repo was missing this file, so deploy and repo
were out of sync.
Changes:
- Add proxy_referrals.py (MIT) with the 5-provider curated catalog
- proxy_manager.py: import PROVIDER_REFERRALS, add 4 helper methods:
get_proxy_referral(tag)
get_proxy_referral_url(tag)
list_proxy_referrals()
get_proxy_referral_summary() - per-tier breakdown
- api.py: expose 2 new endpoints
GET /v1/proxy/referrals - full catalog + summary
GET /v1/proxy/referrals/{tag} - single provider
- 12/12 existing proxy_manager tests still pass
- Total routes: 195 -> 197
The previous CI had a single build job with mypy marked as
continue-on-error, and no secret scanning or security audit. The
pre-commit config has gitleaks + bandit but they were never enforced
in CI.
Changes:
- Split monolithic build job into focused jobs: lint, typecheck, test,
secrets (gitleaks), security (bandit)
- Promote mypy to a real required job (not continue-on-error)
- Add gitleaks scan (v8.24.0) against the full git history
- Add bandit scan at high+medium severity (low severity is too noisy
for an OSS project)
- Each job has its own container to keep failures isolated
The committed openapi.json was a 293-line sample covering only 11 paths
and 7 tags. The actual FastAPI app exposes 183 paths across 50 tags.
This was making API documentation, SDK generation, and any external
integration effectively guess at the real surface.
Re-generate via:
python3 -c "import json; from api import app; print(json.dumps(app.openapi(), indent=2))" > openapi.json
Going forward, regenerate before each release. Add a Makefile target.
Squashed from chore/license-relicense. Full message preserved in the
original branch commit bb77eb5. See ADR-0002 for the decision rationale.
Refs: ADR-0002, commit bb77eb5
Python CI: uv setup, ruff lint+format, mypy (warn), pytest (non-integration).
Runs on docker-x64 runner on Talos.
55 tests exist, now they will run in CI.
- .forgejo/ISSUE_TEMPLATE/bug.yml
- .forgejo/ISSUE_TEMPLATE/feature.yml
- .forgejo/ISSUE_TEMPLATE/config.yml: blank_issues_enabled=false,
contact_links to discussions (Q&A, Feature Requests, Announcements)
Discussions per repo land in a separate follow-up commit.
