cryptorugmunch 117001006f feat(logging): add structlog + JSON logging (CONVENTIONS.md Part 5)

Pry logs are now JSON objects with the required fields (timestamp,
level, service, event, plus key-value pairs). This is the standard
required by CONVENTIONS.md Part 5 and is what makes the service
operable in production (Loki, ELK, etc. can index the structured
records).

New module logging_config.py:
  setup_logging(level, fmt) - configure once at process startup
  get_logger(name)         - get a structlog logger; falls back to stdlib
  is_configured()          - diagnostic for /health

Configuration via env vars:
  PRY_LOG_FORMAT=json|console   (default json)
  PRY_LOG_LEVEL=DEBUG|INFO|...  (default INFO)
  PRY_LOG_STRICT_EXTRAS=1       (default unset = lenient)

Backward compatibility:
  - stdlib logging.getLogger(__name__) calls still work
  - setup_logging bridges stdlib through structlog's formatter
  - In lenient mode, extra={...} keys that collide with reserved
    LogRecord names (e.g. 'name') are moved to an `extra` sub-dict
    so existing code doesn't crash

Wired in:
  api.py: setup_logging() at module import time; lifespan log uses
          structlog style (logger.info("event", key="value") without
          the `extra={...}` wrapper)
  pyproject.toml: structlog>=24.0.0 dep added

Fixed source files that used reserved LogRecord keys in extra={...}:
  agency.py:        "name" -> "agency_name"
  auth_connector.py: "name" -> "credential_name"
  monitor.py:       "name" -> "monitor_name"
  pipelines.py:     "name" -> "pipeline_name"
  llm_providers/registry.py: "name" -> "provider_name"
These would have crashed with KeyError "Attempt to overwrite 'name' in
LogRecord" the moment a real log handler was attached.

Tests: 8/8 in test_logging_config.py pass. Full test suite went from
14 failures -> 2 (one is the SSE subprocess test that doesn't work in
this sandbox; one was the openapi title test that I also fixed in
this commit).

Documentation: DEVELOPMENT.md now has a full "Logging" section with
quick-start, config, and the reserved-key gotcha.

2026-07-02 20:55:41 +02:00

4.1 KiB

Raw Blame History

DEVELOPMENT.md — PryScraper

Dev workflow. Install, code, test, commit, PR.

Setup

Prerequisites

Python 3.12+ / Node 20+
gopass for secrets
mise for tool version mgmt (or manual)
pre-commit for hooks

Install

git clone https://git.rugmunch.io/RugMunchMedia/pryscraper.git
cd pryscraper
make install
pre-commit install

Environment

# Required env vars (loaded from gopass on deploy, .env locally)
# See .env.example
cp .env.example .env
$EDITOR .env  # fill in test values

Workflow

1. Create a branch

git checkout -b feat/my-feature
# or fix/my-bug, docs/my-doc, chore/my-chore

2. Make changes

Write code
Add tests
Update docs (AGENTS.md, ARCHITECTURE.md, STATUS.md)

3. Run pre-commit locally

make lint
make test
make typecheck
make security

4. Commit (conventional)

make commit   # interactive
# or:
git commit -m "feat(scope): add new feature"

Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore, ops, security

5. Push + PR

git push -u origin feat/my-feature
# Open PR on forgejo: https://git.rugmunch.io/RugMunchMedia/pryscraper/pulls/new

6. Wait for CI

All checks must pass
Review by CODEOWNERS
Squash-merge to main
Auto-deploys to Talos (via forgejo webhook)

Daily End-of-Day

make status    # show what's changed
fleet-commit   # commit helper with checklist

Code Style

Python: ruff (lint + format), mypy strict
TypeScript: eslint + prettier
Shell: shellcheck
Markdown: vale

See standards/CONVENTIONS.md.

Logging

Pry uses structlog for structured (JSON) logging per CONVENTIONS.md Part 5. Every log line is a JSON object with the required fields: timestamp, level, service, event, plus any key-value pairs you pass to the logger.

Quick start

from logging_config import setup_logging, get_logger

# Call once at process startup. api.py does this at module import time.
setup_logging()

log = get_logger(__name__)
log.info("scrape_started", url=url, mode="stealth")
log.warning("rate_limited", host=host, retry_after=retry)
log.error("scrape_failed", url=url, error=str(e))

Output (JSON, one record per line):

{"url": "https://...", "mode": "stealth", "event": "scrape_started", "level": "info", "timestamp": "2026-07-02T18:34:19.567377Z", "service": "pry"}

Configuration

Set via environment variables:

Variable	Default	Description
`PRY_LOG_FORMAT`	`json`	`json` (production) or `console` (local dev, colored)
`PRY_LOG_LEVEL`	`INFO`	`DEBUG`, `INFO`, `WARNING`, `ERROR`
`PRY_LOG_STRICT_EXTRAS`	(unset)	`1` to fail-fast on `extra={...}` keys that collide with reserved LogRecord names (default is lenient — reserved keys get moved to an `extra` sub-dict)

Backward compatibility

Code that uses stdlib logging.getLogger(__name__) keeps working. The setup_logging() call bridges stdlib through structlog's formatter, so logger.warning("msg", extra={"foo": 1}) produces the same JSON shape as get_logger(__name__).warning("msg", foo=1).

Reserved key gotcha

logger.warning("event", extra={"name": "x"}) will crash with KeyError: "Attempt to overwrite 'name' in LogRecord" in strict mode, because name is a reserved stdlib field. In lenient mode (default) the key gets moved to an extra sub-dict. Prefer the structlog style (log.warning("event", name="x")) which doesn't have this issue.

Where setup_logging() is called

api.py at module import time (lifespan startup also calls it for safety)
cli.py should call it before any logger usage (TODO)

Adding to a new module

from logging_config import get_logger  # or just `import logging; log = logging.getLogger(__name__)`
log = get_logger(__name__)

def my_function(x: int) -> int:
    log.info("my_function_called", x=x)
    return x * 2

4.1 KiB Raw Blame History