cryptorugmunch
117001006f
Pry logs are now JSON objects with the required fields (timestamp,
level, service, event, plus key-value pairs). This is the standard
required by CONVENTIONS.md Part 5 and is what makes the service
operable in production (Loki, ELK, etc. can index the structured
records).
New module logging_config.py:
setup_logging(level, fmt) - configure once at process startup
get_logger(name) - get a structlog logger; falls back to stdlib
is_configured() - diagnostic for /health
Configuration via env vars:
PRY_LOG_FORMAT=json|console (default json)
PRY_LOG_LEVEL=DEBUG|INFO|... (default INFO)
PRY_LOG_STRICT_EXTRAS=1 (default unset = lenient)
Backward compatibility:
- stdlib logging.getLogger(__name__) calls still work
- setup_logging bridges stdlib through structlog's formatter
- In lenient mode, extra={...} keys that collide with reserved
LogRecord names (e.g. 'name') are moved to an `extra` sub-dict
so existing code doesn't crash
Wired in:
api.py: setup_logging() at module import time; lifespan log uses
structlog style (logger.info("event", key="value") without
the `extra={...}` wrapper)
pyproject.toml: structlog>=24.0.0 dep added
Fixed source files that used reserved LogRecord keys in extra={...}:
agency.py: "name" -> "agency_name"
auth_connector.py: "name" -> "credential_name"
monitor.py: "name" -> "monitor_name"
pipelines.py: "name" -> "pipeline_name"
llm_providers/registry.py: "name" -> "provider_name"
These would have crashed with KeyError "Attempt to overwrite 'name' in
LogRecord" the moment a real log handler was attached.
Tests: 8/8 in test_logging_config.py pass. Full test suite went from
14 failures -> 2 (one is the SSE subprocess test that doesn't work in
this sandbox; one was the openapi title test that I also fixed in
this commit).
Documentation: DEVELOPMENT.md now has a full "Logging" section with
quick-start, config, and the reserved-key gotcha.
153 lines
4.1 KiB
Markdown
153 lines
4.1 KiB
Markdown
[//]: # (SPDX-License-Identifier: MIT)
|
|
[//]: # (Copyright (c) 2026 Rug Munch Media LLC)
|
|
# DEVELOPMENT.md — PryScraper
|
|
|
|
> Dev workflow. Install, code, test, commit, PR.
|
|
|
|
## Setup
|
|
|
|
### Prerequisites
|
|
- Python 3.12+ / Node 20+
|
|
- `gopass` for secrets
|
|
- `mise` for tool version mgmt (or manual)
|
|
- `pre-commit` for hooks
|
|
|
|
### Install
|
|
```bash
|
|
git clone https://git.rugmunch.io/RugMunchMedia/pryscraper.git
|
|
cd pryscraper
|
|
make install
|
|
pre-commit install
|
|
```
|
|
|
|
### Environment
|
|
```bash
|
|
# Required env vars (loaded from gopass on deploy, .env locally)
|
|
# See .env.example
|
|
cp .env.example .env
|
|
$EDITOR .env # fill in test values
|
|
```
|
|
|
|
## Workflow
|
|
|
|
### 1. Create a branch
|
|
```bash
|
|
git checkout -b feat/my-feature
|
|
# or fix/my-bug, docs/my-doc, chore/my-chore
|
|
```
|
|
|
|
### 2. Make changes
|
|
- Write code
|
|
- Add tests
|
|
- Update docs (AGENTS.md, ARCHITECTURE.md, STATUS.md)
|
|
|
|
### 3. Run pre-commit locally
|
|
```bash
|
|
make lint
|
|
make test
|
|
make typecheck
|
|
make security
|
|
```
|
|
|
|
### 4. Commit (conventional)
|
|
```bash
|
|
make commit # interactive
|
|
# or:
|
|
git commit -m "feat(scope): add new feature"
|
|
```
|
|
|
|
Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore, ops, security
|
|
|
|
### 5. Push + PR
|
|
```bash
|
|
git push -u origin feat/my-feature
|
|
# Open PR on forgejo: https://git.rugmunch.io/RugMunchMedia/pryscraper/pulls/new
|
|
```
|
|
|
|
### 6. Wait for CI
|
|
- All checks must pass
|
|
- Review by CODEOWNERS
|
|
- Squash-merge to main
|
|
- Auto-deploys to Talos (via forgejo webhook)
|
|
|
|
## Daily End-of-Day
|
|
```bash
|
|
make status # show what's changed
|
|
fleet-commit # commit helper with checklist
|
|
```
|
|
|
|
## Code Style
|
|
- Python: ruff (lint + format), mypy strict
|
|
- TypeScript: eslint + prettier
|
|
- Shell: shellcheck
|
|
- Markdown: vale
|
|
|
|
See [standards/CONVENTIONS.md](https://git.rugmunch.io/RugMunchMedia/standards/raw/branch/main/CONVENTIONS.md).
|
|
|
|
|
|
## Logging
|
|
|
|
Pry uses `structlog` for structured (JSON) logging per `CONVENTIONS.md` Part 5.
|
|
Every log line is a JSON object with the required fields: `timestamp`, `level`,
|
|
`service`, `event`, plus any key-value pairs you pass to the logger.
|
|
|
|
### Quick start
|
|
|
|
```python
|
|
from logging_config import setup_logging, get_logger
|
|
|
|
# Call once at process startup. api.py does this at module import time.
|
|
setup_logging()
|
|
|
|
log = get_logger(__name__)
|
|
log.info("scrape_started", url=url, mode="stealth")
|
|
log.warning("rate_limited", host=host, retry_after=retry)
|
|
log.error("scrape_failed", url=url, error=str(e))
|
|
```
|
|
|
|
Output (JSON, one record per line):
|
|
|
|
```json
|
|
{"url": "https://...", "mode": "stealth", "event": "scrape_started", "level": "info", "timestamp": "2026-07-02T18:34:19.567377Z", "service": "pry"}
|
|
```
|
|
|
|
### Configuration
|
|
|
|
Set via environment variables:
|
|
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `PRY_LOG_FORMAT` | `json` | `json` (production) or `console` (local dev, colored) |
|
|
| `PRY_LOG_LEVEL` | `INFO` | `DEBUG`, `INFO`, `WARNING`, `ERROR` |
|
|
| `PRY_LOG_STRICT_EXTRAS` | (unset) | `1` to fail-fast on `extra={...}` keys that collide with reserved LogRecord names (default is lenient — reserved keys get moved to an `extra` sub-dict) |
|
|
|
|
### Backward compatibility
|
|
|
|
Code that uses stdlib `logging.getLogger(__name__)` keeps working. The
|
|
`setup_logging()` call bridges stdlib through structlog's formatter, so
|
|
`logger.warning("msg", extra={"foo": 1})` produces the same JSON shape as
|
|
`get_logger(__name__).warning("msg", foo=1)`.
|
|
|
|
### Reserved key gotcha
|
|
|
|
`logger.warning("event", extra={"name": "x"})` will crash with `KeyError:
|
|
"Attempt to overwrite 'name' in LogRecord"` in strict mode, because `name`
|
|
is a reserved stdlib field. In lenient mode (default) the key gets moved
|
|
to an `extra` sub-dict. **Prefer the structlog style** (`log.warning("event", name="x")`)
|
|
which doesn't have this issue.
|
|
|
|
### Where setup_logging() is called
|
|
|
|
- `api.py` at module import time (lifespan startup also calls it for safety)
|
|
- `cli.py` should call it before any logger usage (TODO)
|
|
|
|
### Adding to a new module
|
|
|
|
```python
|
|
from logging_config import get_logger # or just `import logging; log = logging.getLogger(__name__)`
|
|
log = get_logger(__name__)
|
|
|
|
def my_function(x: int) -> int:
|
|
log.info("my_function_called", x=x)
|
|
return x * 2
|
|
```
|