chore(lint): auto-fix 253 of 283 ruff issues (F401, I001, E402, RUF100, UP037, SIM105)
Mass ruff auto-fix:
- ruff check --fix: 109 issues fixed (F401 unused imports,
I001 unsorted imports, UP037 quoted annotations, SIM105
suppressible exception, RUF100 unused-noqa)
- ruff check --fix --unsafe-fixes: 22 additional issues
- ruff format: 70 files reformatted
- Manual pass: fix 16 misplaced import httpx lines
- Manual pass: fix remaining E402 (import-after-docstring)
Result: 283 errors -> 30 errors.
The remaining 30 are real issues that need manual review:
5 F401 unused-import (likely auto-generated stubs)
5 F821 undefined-name (real bugs in code that references
redis/pydantic/LLMRegistry without imports)
3 BLE001 (the compliance LLM fallback is intentional; the
other two are real)
3 RUF012 mutable-class-default
3 SIM105, 3 SIM117, 2 E722, 2 E741
1 B007, 1 B025, 1 E402, 1 RUF200 (pyproject.toml issue)
Tests: 436/437 pass (1 pre-existing SSE sandbox failure).
format check + import sort: now clean.
make ci: still gated on the 30 remaining real issues.
Follow-up: triage the 30 issues file-by-file.
This commit is contained in:
parent
e60a62a07a
commit
a7c30b12cd
85 changed files with 2374 additions and 1071 deletions
14
scraper.py
14
scraper.py
|
|
@ -217,9 +217,12 @@ class PryScraper:
|
|||
if not self.playwright_available:
|
||||
try:
|
||||
import subprocess
|
||||
|
||||
r = subprocess.run(
|
||||
["python3", "-m", "playwright", "install", "--dry-run", "chromium"],
|
||||
capture_output=True, text=True, timeout=10
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10,
|
||||
)
|
||||
if "chromium" in r.stdout:
|
||||
self.playwright_available = True
|
||||
|
|
@ -258,13 +261,11 @@ class PryScraper:
|
|||
if indicator.lower() in html_lower:
|
||||
return True
|
||||
# Check for very short / challenge-looking pages
|
||||
if (
|
||||
return bool(
|
||||
len(html) < 2000
|
||||
and "document" in html_lower
|
||||
and ("var " in html_lower or "function" in html_lower)
|
||||
):
|
||||
return True
|
||||
return False
|
||||
)
|
||||
|
||||
def _quality_score(self, content: str) -> int:
|
||||
"""Score content quality 0-100. Low score triggers retry with better tier."""
|
||||
|
|
@ -288,6 +289,7 @@ class PryScraper:
|
|||
# Use UltimateScraper for the 10-tier fetching
|
||||
if self._ultimate is None:
|
||||
from ultimate_scraper import UltimateScraper
|
||||
|
||||
self._ultimate = UltimateScraper()
|
||||
|
||||
result = await self._ultimate.scrape(url, opts)
|
||||
|
|
@ -563,5 +565,3 @@ class PryScraper:
|
|||
if urlparse(link).netloc == base_domain:
|
||||
links.add(link.split("#")[0].rstrip("/"))
|
||||
return sorted(links)[: opts.get("limit", 50)]
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue