pryscraper/tests
cryptorugmunch 117001006f feat(logging): add structlog + JSON logging (CONVENTIONS.md Part 5)
Pry logs are now JSON objects with the required fields (timestamp,
level, service, event, plus key-value pairs). This is the standard
required by CONVENTIONS.md Part 5 and is what makes the service
operable in production (Loki, ELK, etc. can index the structured
records).

New module logging_config.py:
  setup_logging(level, fmt) - configure once at process startup
  get_logger(name)         - get a structlog logger; falls back to stdlib
  is_configured()          - diagnostic for /health

Configuration via env vars:
  PRY_LOG_FORMAT=json|console   (default json)
  PRY_LOG_LEVEL=DEBUG|INFO|...  (default INFO)
  PRY_LOG_STRICT_EXTRAS=1       (default unset = lenient)

Backward compatibility:
  - stdlib logging.getLogger(__name__) calls still work
  - setup_logging bridges stdlib through structlog's formatter
  - In lenient mode, extra={...} keys that collide with reserved
    LogRecord names (e.g. 'name') are moved to an `extra` sub-dict
    so existing code doesn't crash

Wired in:
  api.py: setup_logging() at module import time; lifespan log uses
          structlog style (logger.info("event", key="value") without
          the `extra={...}` wrapper)
  pyproject.toml: structlog>=24.0.0 dep added

Fixed source files that used reserved LogRecord keys in extra={...}:
  agency.py:        "name" -> "agency_name"
  auth_connector.py: "name" -> "credential_name"
  monitor.py:       "name" -> "monitor_name"
  pipelines.py:     "name" -> "pipeline_name"
  llm_providers/registry.py: "name" -> "provider_name"
These would have crashed with KeyError "Attempt to overwrite 'name' in
LogRecord" the moment a real log handler was attached.

Tests: 8/8 in test_logging_config.py pass. Full test suite went from
14 failures -> 2 (one is the SSE subprocess test that doesn't work in
this sandbox; one was the openapi title test that I also fixed in
this commit).

Documentation: DEVELOPMENT.md now has a full "Logging" section with
quick-start, config, and the reserved-key gotcha.
2026-07-02 20:55:41 +02:00
..
__init__.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
conftest.py feat(logging): add structlog + JSON logging (CONVENTIONS.md Part 5) 2026-07-02 20:55:41 +02:00
test_adaptive.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_advanced_features.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_advanced_scraping.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_agency.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_ai_plugin.py feat(logging): add structlog + JSON logging (CONVENTIONS.md Part 5) 2026-07-02 20:55:41 +02:00
test_alerter.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_api.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_api_mcp_x402.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_auth_connector.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_block_detector.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_cache.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_camoufox.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_commerce_sync.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_compliance.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_costing.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_crm_sync.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_destinations.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_email_scraper.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_enrichment.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_extractor.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_freshness.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_gdpr.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_infrastructure.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_intelligence.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_jobqueue.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_lazy_load.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_llm_fallback.py feat(ai): wire llm_features into compliance, seo, reconciliation 2026-07-02 20:33:07 +02:00
test_llm_providers.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_logging_config.py feat(logging): add structlog + JSON logging (CONVENTIONS.md Part 5) 2026-07-02 20:55:41 +02:00
test_markdown_gen.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_marketplace.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_mconfig.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_mcp_production.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_monitor.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_network.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_parser.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_pipeline.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_pipelines.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_proxy_manager.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_quality.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_ratelimit.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_real_features.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_reconciliation.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_referrals.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_reports.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_review.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_scraper.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_sdk.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_secrets_backend.py feat(secrets): gopass-based secret backend (PRY_SECRET_BACKEND) 2026-07-02 20:26:00 +02:00
test_seo_monitor.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_shadow_dom.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_structure_monitor.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_templates.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_training_data.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_ultimate_scraper.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00
test_x402_mcp_spec.py chore(license): re-license to dual MIT (core) + BSL 1.1 (stealth) 2026-07-02 19:59:18 +02:00