{ "sha": "09ee870a83354273e10b4bdf2f4ad02f5e98a715", "message": "Per-model reasoning profiles, DeepSeek V4 Pro slot, freshness/observability overhaul\n\nMODEL_PROFILES (pipeline/providers.py) is the single source of truth for\nmodel ids, reasoning controls and sampling:\n- nemotron-3-ultra: chat_template_kwargs enable_thinking + medium_effort\n- nemotron-3-super: reasoning_effort low (default reasoning ate the whole\n 8192-token budget in live runs and truncated every draft)\n- deepseek-ai/deepseek-v4-pro: reasoning_effort high; wired at chain #3\n (not in this key's catalog yet - 404 falls through until NVIDIA opens it)\n- qwen3.5: enable_thinking + thinking-mode sampling (0.6/0.95/20/0.0)\n- glm-5.2: provider-default reasoning, no invented fields on the wire\n- mistral-medium-3.5: reasoning_effort high (last fallback, quality first)\n- gemini-3.6/3.5-flash: thinkingConfig.thinkingLevel medium, temp 0.1\n\nReasoning traces are now provably separated from final answers: Gemini\nthought-parts and NIM reasoning_content never reach the JSON parser,\npublished data or logs. Malformed JSON gets ONE no-thinking format-repair\ncall (syntax only; content failures still fail over per policy). NIM read\ntimeout raised to 180s (550B models regularly exceeded 120s).\n\nFreshness/observability per the editorial ops spec:\n- NEWS_MAX_AGE_HOURS in common.py (default 120, validated 24-336,\n warning + safe fallback on bad values) replaces the 48h constant\n- future-dated items dropped beyond 6h clock skew, before any AI call\n- items without a source date carry dateInferred; the prompt shows them\n as first-seen stamps, never as publication times\n- articles record sourcePublishedUtc (newest real source date); the site\n displays it instead of generation time, with m/d labels for older news\n- title-echo summaries no longer count as material (item_has_material)\n- dedupe ranks material-bearing groups above title-only ones so thin rows\n cannot crowd real stories out of the cap; per-run skip-reason stats\n- write logs ai_calls per model; CI scans build output for secret-like\n strings and reasoning traces before deploy\n\nTests: 51 provider payload/parsing checks (all mocked), 121 write checks,\n6 dedupe suites incl. freshness window, future guard and material ranking.\n\nCo-Authored-By: Claude Fable 5 ", "author": { "login": "avwire", "name": "avwire", "email": "" }, "committer": { "login": "avwire", "name": "avwire", "email": "" }, "created_at": "2026-07-26T18:15:19Z", "files": [ { "filename": ".github/workflows/hourly.yml", "previous_filename": "", "status": "modified", "additions": 14, "deletions": 1, "changes": 15 }, { "filename": "README.md", "previous_filename": "", "status": "modified", "additions": 1, "deletions": 1, "changes": 2 }, { "filename": "pipeline/CONTRACTS.md", "previous_filename": "", "status": "modified", "additions": 4, "deletions": 0, "changes": 4 }, { "filename": "pipeline/build.py", "previous_filename": "", "status": "modified", "additions": 14, "deletions": 2, "changes": 16 }, { "filename": "pipeline/common.py", "previous_filename": "", "status": "modified", "additions": 69, "deletions": 0, "changes": 69 }, { "filename": "pipeline/dedupe.py", "previous_filename": "", "status": "modified", "additions": 41, "deletions": 18, "changes": 59 }, { "filename": "pipeline/fetch.py", "previous_filename": "", "status": "modified", "additions": 10, "deletions": 2, "changes": 12 }, { "filename": "pipeline/providers.py", "previous_filename": "", "status": "modified", "additions": 216, "deletions": 43, "changes": 259 }, { "filename": "pipeline/write.py", "previous_filename": "", "status": "modified", "additions": 34, "deletions": 21, "changes": 55 }, { "filename": "tests/fixtures/dedupe/raw/faa.json", "previous_filename": "", "status": "modified", "additions": 2, "deletions": 2, "changes": 4 }, { "filename": "tests/test_dedupe.py", "previous_filename": "", "status": "modified", "additions": 112, "deletions": 8, "changes": 120 }, { "filename": "tests/test_providers_payloads.py", "previous_filename": "", "status": "added", "additions": 282, "deletions": 0, "changes": 282 }, { "filename": "tests/test_write_offline.py", "previous_filename": "", "status": "modified", "additions": 17, "deletions": 1, "changes": 18 } ], "diff": "@@ -49,13 +49,26 @@ jobs:\\n AVWIRE_GEMINI_MODEL: ${{ vars.AVWIRE_GEMINI_MODEL }}\\n AVWIRE_NVIDIA_MODEL: ${{ vars.AVWIRE_NVIDIA_MODEL }}\\n AVWIRE_PROVIDER_ORDER: ${{ vars.AVWIRE_PROVIDER_ORDER }}\\n- AVWIRE_MAX_AGE_HOURS: ${{ vars.AVWIRE_MAX_AGE_HOURS }}\\n+ # freshness window (hours); code default 120, validated 24-336\\n+ NEWS_MAX_AGE_HOURS: ${{ vars.NEWS_MAX_AGE_HOURS }}\\n run: |\\n python pipeline/fetch.py\\n python pipeline/dedupe.py\\n python pipeline/write.py\\n python pipeline/build.py\\n \\n+ - name: Scan build output for secrets and reasoning traces\\n+ run: |\\n+ if grep -rInE \"nvapi-[A-Za-z0-9_-]{16,}|AIza[0-9A-Za-z_-]{20,}|sk-ant-[A-Za-z0-9_-]{16,}|Bearer [A-Za-z0-9_./+-]{20,}\" site/ ; then\\n+ echo \"::error::secret-like string found in build output\"\\n+ exit 1\\n+ fi\\n+ if grep -rIl \"reasoning_content\\|\" site/ ; then\\n+ echo \"::error::reasoning trace found in build output\"\\n+ exit 1\\n+ fi\\n+ echo \"scan: build output clean\"\\n+\\n # Upload BEFORE the data commit so the deploy never depends on git push.\\n - uses: actions/upload-pages-artifact@v3\\n with:\\n@@ -79,7 +79,7 @@ python -m http.server -d site 8000\\n | `AVWIRE_MODEL` | `claude-opus-5` | Anthropic 模型 |\\n | `AVWIRE_GEMINI_MODEL` | `gemini-2.5-flash` | Gemini 模型 |\\n | `AVWIRE_NVIDIA_MODEL` | `deepseek-ai/deepseek-v3.1` | NVIDIA NIM 模型 |\\n-| `AVWIRE_MAX_AGE_HOURS` | `48` | 新聞新鮮度窗口（小時）；週末官方來源停更時可放寬 |\\n+| `NEWS_MAX_AGE_HOURS` | `120` | 新聞新鮮度窗口（小時），驗證範圍 24–336，非法值回退預設並警告 |\\n | `AEROAPI_KEY` | — | FlightAware AeroAPI（選用統計） |\\n | `AVWIRE_BASE_PATH` | `/avwire` | 站台子路徑（Pages 專案站台） |\\n \\n@@ -109,6 +109,10 @@ absent when nothing new that hour)\\n \"zh\": {\"title\": \"...\", \"summary\": \"<=80字\", \"body\": [\"段落1\", \"段落2\"]},\\n \"en\": {\"title\": \"...\", \"summary\": \"...\", \"body\": [\"para1\", \"para2\"]},\\n \"sources\": [{\"name\": \"FAA — Statement\", \"url\": \"https://...\"}], // >=1, REQUIRED\\n+ \"sourcePublishedUtc\": \"2026-07-23T04:00Z\", // OPTIONAL newest source-provided\\n+ // publish time (never an inferred\\n+ // stamp); the site displays this,\\n+ // publishedUtc orders the archive\\n \"writer\": \"gemini:gemini-2.5-flash\", // OPTIONAL provenance \"provider:model\";\\n // absent on pre-multi-provider articles\\n \"facts\": [ // OPTIONAL machine-verified evidence:\\n@@ -309,15 +309,27 @@ def side(own, other):\\n image = str(raw[\"image\"]) if raw.get(\"image\") else None\\n if image and not _safe_web_url(image):\\n image = None\\n- tpe = dt.astimezone(TPE)\\n+ # Display the SOURCE's newest publication time when the write stage\\n+ # recorded one; the generation time (dt) is used for ordering only, so\\n+ # a days-old official release is never presented as breaking news.\\n+ display_dt = dt\\n+ if raw.get(\"sourcePublishedUtc\"):\\n+ try:\\n+ display_dt = parse_iso(str(raw[\"sourcePublishedUtc\"]))\\n+ except (ValueError, TypeError):\\n+ pass\\n+ tpe = display_dt.astimezone(TPE)\\n+ today_tpe = now_utc().astimezone(TPE).date()\\n+ time_label = (tpe.strftime(\"%H:%M\") if tpe.date() == today_tpe\\n+ else tpe.strftime(\"%m/%d\"))\\n return {\\n \"id\": art_id,\\n \"dt\": dt,\\n \"cat\": cat,\\n \"tag_class\": TAGCLS.get(cat, \"tag-neutral\"),\\n \"image\": image,\\n \"source\": str(raw.get(\"primarySource\") or (sources[0][\"name\"] if sources else \"—\")),\\n- \"time\": tpe.strftime(\"%H:%M\"),\\n+ \"time\": time_label,\\n \"meta_ts\": tpe.strftime(\"%Y-%m-%d %H:%M\") + \" UTC+8\",\\n \"zh\": side(zh, en),\\n \"en\": side(en, zh),\\n@@ -42,6 +42,75 @@\\n # Articles shown on the home page feed.\\n HOME_FEED_LIMIT = 30\\n \\n+\\n+def _parse_max_age(value, default: int = 120,\\n+ low: int = 24, high: int = 336) -> int:\\n+ \"\"\"Validate the freshness-window setting (hours).\\n+\\n+ Non-integers fall back to the default with a warning; values outside\\n+ [low, high] are treated as configuration errors and also fall back, so\\n+ a bad setting can never admit unbounded-age news.\\n+ \"\"\"\\n+ if value in (None, \"\"):\\n+ return default\\n+ try:\\n+ hours = int(str(value).strip())\\n+ except (TypeError, ValueError):\\n+ print(f\"config: NEWS_MAX_AGE_HOURS={value!r} is not an integer; \"\\n+ f\"using {default}\")\\n+ return default\\n+ if not low <= hours <= high:\\n+ print(f\"config: NEWS_MAX_AGE_HOURS={hours} outside [{low}, {high}]; \"\\n+ f\"using {default}\")\\n+ return default\\n+ return hours\\n+\\n+\\n+# Freshness window for the dedupe stage. Official sources go quiet on\\n+# weekends, so the default reaches back 5 days; seen.json's 21-day memory\\n+# still prevents any story from running twice. Single source of truth -\\n+# do not read the env var anywhere else. (AVWIRE_MAX_AGE_HOURS is the\\n+# deprecated alias.)\\n+NEWS_MAX_AGE_HOURS = _parse_max_age(\\n+ os.environ.get(\"NEWS_MAX_AGE_HOURS\")\\n+ or os.environ.get(\"AVWIRE_MAX_AGE_HOURS\"))\\n+\\n+# Items stamped this far in the future are treated as source clock/parse\\n+# errors and dropped before any AI call; smaller offsets are tolerated as\\n+# timezone skew.\\n+FUTURE_SKEW_HOURS = 6\\n+\\n+# --- source-material completeness (editorial spec section 2) -------------\\n+# An item contributes usable evidence only when its summary adds real text\\n+# beyond the headline; a group needs at least one such item to be worth an\\n+# LLM call. Single source of truth for dedupe.py and write.py.\\n+MIN_EFFECTIVE_SUMMARY_CHARS = 25\\n+\\n+_SQUASH_RE = re.compile(r\"\\s+\")\\n+\\n+\\n+def squash_text(text) -> str:\\n+ \"\"\"Whitespace-collapsed, case-folded text for verbatim comparisons.\"\"\"\\n+ return _SQUASH_RE.sub(\" \", str(text or \"\")).strip().casefold()\\n+\\n+\\n+def item_has_material(item: dict) -> bool:\\n+ if not isinstance(item, dict):\\n+ return False\\n+ summary = squash_text(item.get(\"summary\"))\\n+ if not summary:\\n+ return False\\n+ title = squash_text(item.get(\"title\"))\\n+ if title and title in summary:\\n+ # A summary that merely repeats the headline is not evidence.\\n+ summary = summary.replace(title, \"\", 1)\\n+ return len(summary.strip()) >= MIN_EFFECTIVE_SUMMARY_CHARS\\n+\\n+\\n+def group_has_material(group: dict) -> bool:\\n+ return any(item_has_material(item)\\n+ for item in (group.get(\"items\") or []))\\n+\\n # Master source registry. `key` is the stable identifier used for\\n # data/raw/.json. `fmt` is what we display on the sources page.\\n # `endpoint` is what fetch.py actually requests; `kind`:\\n@@ -9,15 +9,17 @@\\n \"\"\"\\n from __future__ import annotations\\n \\n-import os\\n import re\\n from datetime import datetime, timedelta, timezone\\n from difflib import SequenceMatcher\\n \\n from common import (\\n DATA_DIR,\\n+ FUTURE_SKEW_HOURS,\\n+ NEWS_MAX_AGE_HOURS,\\n RAW_DIR,\\n SOURCES,\\n+ group_has_material,\\n iso_minute,\\n load_json,\\n norm_url,\\n@@ -28,13 +30,9 @@\\n \\n MAX_GROUPS = 12 # emitted per run; bounds LLM cost in write.py\\n MAX_ITEMS_PER_GROUP = 5\\n-# Freshness window. Official sources go quiet on weekends, so the repo\\n-# variable AVWIRE_MAX_AGE_HOURS can widen it (e.g. 120) without a deploy;\\n-# the 21-day seen.json memory still prevents any story from running twice.\\n-try:\\n- MAX_ITEM_AGE_HOURS = int(os.environ.get(\"AVWIRE_MAX_AGE_HOURS\") or 48)\\n-except ValueError:\\n- MAX_ITEM_AGE_HOURS = 48\\n+# Freshness window: single source of truth in common.NEWS_MAX_AGE_HOURS\\n+# (env NEWS_MAX_AGE_HOURS, validated, default 120).\\n+MAX_ITEM_AGE_HOURS = NEWS_MAX_AGE_HOURS\\n SEEN_TITLE_DAYS = 21 # only seen.json titles this recent are compared\\n SEEN_TITLE_SIM = 0.75 # vs seen.json titles -> drop as already covered\\n GROUP_TITLE_SIM = 0.60 # between fresh items -> same story group\\n@@ -111,23 +109,31 @@ def _load_seen(now: datetime) -> tuple[set[str], list[str]]:\\n \\n \\n def _is_fresh(\\n- item: dict, now: datetime, seen_urls: set[str], seen_titles: list[str]\\n+ item: dict, now: datetime, seen_urls: set[str], seen_titles: list[str],\\n+ stats: dict,\\n ) -> bool:\\n title = str(item.get(\"title\") or \"\").strip()\\n if not title:\\n+ stats[\"skipped_untitled\"] += 1\\n return False\\n url = str(item.get(\"url\") or \"\").strip()\\n if url and norm_url(url) in seen_urls:\\n+ stats[\"skipped_seen_url\"] += 1\\n return False\\n published = _published(item)\\n- if published is not None and now - published > timedelta(\\n- hours=MAX_ITEM_AGE_HOURS\\n- ):\\n- return False\\n+ if published is not None:\\n+ if now - published > timedelta(hours=MAX_ITEM_AGE_HOURS):\\n+ stats[\"skipped_too_old\"] += 1\\n+ return False\\n+ if published - now > timedelta(hours=FUTURE_SKEW_HOURS):\\n+ # Beyond timezone/parse skew: a source clock error, not news.\\n+ stats[\"skipped_future_date\"] += 1\\n+ return False\\n normalized = norm_title(title)\\n if normalized:\\n for seen_title in seen_titles:\\n if _similar(normalized, seen_title) >= SEEN_TITLE_SIM:\\n+ stats[\"skipped_seen_title\"] += 1\\n return False\\n return True\\n \\n@@ -188,19 +194,28 @@ def main() -> None:\\n now = now_utc()\\n raw_items = _load_raw_items()\\n seen_urls, seen_titles = _load_seen(now)\\n+ stats = {key: 0 for key in (\\n+ \"skipped_untitled\", \"skipped_seen_url\", \"skipped_too_old\",\\n+ \"skipped_future_date\", \"skipped_seen_title\")}\\n fresh = [\\n item\\n for item in raw_items\\n- if _is_fresh(item, now, seen_urls, seen_titles)\\n+ if _is_fresh(item, now, seen_urls, seen_titles, stats)\\n ]\\n \\n- ranked: list[tuple[bool, datetime, list[dict]]] = []\\n+ # Rank: cross-source coverage first, then groups that actually carry\\n+ # summary material (title-only groups cannot survive the evidence rules,\\n+ # so they must never crowd real stories out of the MAX_GROUPS cap),\\n+ # then recency.\\n+ ranked: list[tuple[bool, bool, datetime, list[dict]]] = []\\n for members in _group(fresh):\\n members = _best_first(members)[:MAX_ITEMS_PER_GROUP]\\n newest = max((_published(m) or _EPOCH for m in members), default=_EPOCH)\\n multi = len({m[\"sourceKey\"] for m in members}) > 1\\n- ranked.append((multi, newest, members))\\n- ranked.sort(key=lambda entry: (entry[0], entry[1]), reverse=True)\\n+ material = group_has_material({\"items\": members})\\n+ ranked.append((multi, material, newest, members))\\n+ ranked.sort(key=lambda entry: (entry[0], entry[1], entry[2]), reverse=True)\\n+ material_groups = sum(1 for entry in ranked if entry[1])\\n \\n stamp = now.strftime(\"%Y%m%d-%H%M\")\\n payload = {\\n@@ -211,7 +226,7 @@ def main() -> None:\\n \"primarySource\": members[0][\"source\"],\\n \"items\": members,\\n }\\n- for n, (_multi, _newest, members) in enumerate(\\n+ for n, (_multi, _material, _newest, members) in enumerate(\\n ranked[:MAX_GROUPS], start=1\\n )\\n ],\\n@@ -221,6 +236,14 @@ def main() -> None:\\n f\"dedupe: {len(raw_items)} raw items -> {len(fresh)} fresh \"\\n f\"-> {len(payload['groups'])} groups\"\\n )\\n+ print(\\n+ \"dedupe stats: window={}h sources={} {} \"\\n+ \"groups_with_material={}/{}\".format(\\n+ MAX_ITEM_AGE_HOURS, len(SOURCES),\\n+ \" \".join(f\"{k}={v}\" for k, v in stats.items()),\\n+ min(material_groups, MAX_GROUPS), len(payload[\"groups\"]),\\n+ )\\n+ )\\n \\n \\n if __name__ == \"__main__\":\\n@@ -434,17 +434,25 @@ def _finalize(raw_items: list[dict], fetched_iso: str,\\n items = []\\n for item in raw_items[:MAX_ITEMS_PER_SOURCE]:\\n published = item.get(\"published\")\\n+ inferred = False\\n if published:\\n published_iso = iso_minute(published)\\n else:\\n+ # No source-provided date: stamp first-seen time, stable across\\n+ # fetches, and FLAG it so downstream never presents the stamp\\n+ # as a real publication time.\\n published_iso = prev_published.get(norm_url(item[\"url\"]), fetched_iso)\\n- items.append({\\n+ inferred = True\\n+ row = {\\n \"title\": _clip(item[\"title\"], TITLE_MAX),\\n \"url\": item[\"url\"],\\n \"publishedUtc\": published_iso,\\n \"summary\": _clip(_collapse(item.get(\"summary\"))),\\n \"image\": item.get(\"image\"),\\n- })\\n+ }\\n+ if inferred:\\n+ row[\"dateInferred\"] = True\\n+ items.append(row)\\n return items\\n \\n \\n@@ -46,7 +46,86 @@\\n anthropic = None\\n \\n DEFAULT_ORDER = \"anthropic,gemini,nvidia\"\\n-HTTP_TIMEOUT = (10, 120) # (connect, read) seconds\\n+# 550B-class NIM models regularly need >120s on the free tier.\\n+HTTP_TIMEOUT = (10, 180) # (connect, read) seconds\\n+\\n+# --------------------------------------------------------------------------\\n+# Centralized per-model configuration - the single source of truth for\\n+# model ids, reasoning controls and sampling. Do NOT scatter these fields\\n+# into request builders. Each profile has:\\n+# payload / generationConfig extras merged into normal verification calls\\n+# repair reasoning overrides for the no-thinking \"format repair\" call\\n+# (JSON syntax only - content failures NEVER take this path)\\n+# Only fields the respective endpoint documents are sent; models without a\\n+# profile fall back to conservative generic settings.\\n+NVIDIA_DEFAULT_MODEL = \"nvidia/nemotron-3-ultra-550b-a55b\"\\n+GEMINI_DEFAULT_MODEL = \"gemini-2.5-flash\"\\n+\\n+MODEL_PROFILES: dict = {\\n+ \"nvidia/nemotron-3-ultra-550b-a55b\": {\\n+ # Medium thinking via the official Nemotron chat-template kwargs.\\n+ \"payload\": {\"temperature\": 1.0, \"top_p\": 0.95,\\n+ \"chat_template_kwargs\": {\"enable_thinking\": True,\\n+ \"medium_effort\": True}},\\n+ \"repair\": {\"chat_template_kwargs\": {\"enable_thinking\": False}},\\n+ },\\n+ \"nvidia/nemotron-3-super-120b-a12b\": {\\n+ # Low effort via the official top-level field: the default full\\n+ # reasoning ate the whole 8192-token budget in live runs.\\n+ \"payload\": {\"temperature\": 1.0, \"top_p\": 0.95,\\n+ \"reasoning_effort\": \"low\"},\\n+ \"repair\": {\"reasoning_effort\": \"none\"},\\n+ },\\n+ \"deepseek-ai/deepseek-v4-pro\": {\\n+ \"payload\": {\"temperature\": 1.0, \"top_p\": 0.95,\\n+ \"reasoning_effort\": \"high\"},\\n+ \"repair\": {\"reasoning_effort\": \"none\"},\\n+ },\\n+ \"qwen/qwen3.5-397b-a17b\": {\\n+ # NIM exposes thinking on/off only for Qwen; thinking-mode sampling\\n+ # per the model card.\\n+ \"payload\": {\"temperature\": 0.6, \"top_p\": 0.95, \"top_k\": 20,\\n+ \"presence_penalty\": 0.0,\\n+ \"chat_template_kwargs\": {\"enable_thinking\": True}},\\n+ \"repair\": {\"chat_template_kwargs\": {\"enable_thinking\": False}},\\n+ },\\n+ \"z-ai/glm-5.2\": {\\n+ # The public NIM schema for GLM exposes no reasoning control;\\n+ # provider-default reasoning is used deliberately. The marker below\\n+ # is internal documentation and is never sent on the wire.\\n+ \"reasoningMode\": \"provider_default\",\\n+ \"payload\": {\"temperature\": 0.1},\\n+ \"repair\": {},\\n+ },\\n+ \"mistralai/mistral-medium-3.5-128b\": {\\n+ # Last fallback: rarely called, quality over latency -> high.\\n+ # NIM offers none|high only for this model.\\n+ \"payload\": {\"temperature\": 0.7, \"reasoning_effort\": \"high\"},\\n+ \"repair\": {\"reasoning_effort\": \"none\"},\\n+ },\\n+ \"gemini-3.6-flash\": {\\n+ \"generationConfig\": {\"temperature\": 0.1, \"maxOutputTokens\": 8192,\\n+ \"thinkingConfig\": {\"thinkingLevel\": \"medium\"}},\\n+ \"repairGenerationConfig\": {\\n+ \"temperature\": 0.1, \"maxOutputTokens\": 8192,\\n+ \"thinkingConfig\": {\"thinkingLevel\": \"minimal\"}},\\n+ },\\n+ \"gemini-3.5-flash\": {\\n+ \"generationConfig\": {\"temperature\": 0.1, \"maxOutputTokens\": 8192,\\n+ \"thinkingConfig\": {\"thinkingLevel\": \"medium\"}},\\n+ \"repairGenerationConfig\": {\\n+ \"temperature\": 0.1, \"maxOutputTokens\": 8192,\\n+ \"thinkingConfig\": {\"thinkingLevel\": \"minimal\"}},\\n+ },\\n+}\\n+\\n+REPAIR_SYSTEM_PROMPT = (\\n+ \"You are a JSON syntax fixer. The user message is a single JSON object \"\\n+ \"with broken syntax (quoting, escaping, commas, brackets or field \"\\n+ \"arrangement). Return ONLY the corrected JSON object. Do not add, \"\\n+ \"remove, reorder facts or change ANY content, field name or value - \"\\n+ \"repair the syntax only.\"\\n+)\\n \\n _THINK_RE = re.compile(r\".*?\", re.DOTALL)\\n _FENCE_RE = re.compile(r\"^\\s*```(?:json)?\\s*|\\s*```\\s*$\")\\n@@ -156,30 +235,38 @@ class GeminiProvider:\\n \\n def __init__(self, model=None) -> None:\\n self.model = (model or os.environ.get(\"AVWIRE_GEMINI_MODEL\")\\n- or \"gemini-2.5-flash\")\\n+ or GEMINI_DEFAULT_MODEL)\\n self.label = f\"{self.name}:{self.model}\"\\n \\n def available(self) -> bool:\\n return bool(os.environ.get(\"GEMINI_API_KEY\"))\\n \\n- def draft(self, system_prompt: str, user_prompt: str, schema: dict):\\n+ def _generation_config(self, schema: dict, repair: bool) -> dict:\\n+ profile = MODEL_PROFILES.get(self.model, {})\\n+ key = \"repairGenerationConfig\" if repair else \"generationConfig\"\\n+ config = dict(profile.get(key) or {})\\n+ config[\"responseMimeType\"] = \"application/json\"\\n+ config[\"responseJsonSchema\"] = schema\\n+ return config\\n+\\n+ def _post(self, payload: dict):\\n url = (\"https://generativelanguage.googleapis.com/v1beta/models/\"\\n f\"{self.model}:generateContent\")\\n- payload = {\\n- \"systemInstruction\": {\"parts\": [{\"text\": system_prompt}]},\\n- \"contents\": [{\"role\": \"user\", \"parts\": [{\"text\": user_prompt}]}],\\n- \"generationConfig\": {\\n- \"responseMimeType\": \"application/json\",\\n- \"responseJsonSchema\": schema,\\n- },\\n- }\\n try:\\n- response = requests.post(\\n+ return requests.post(\\n url, json=payload, timeout=HTTP_TIMEOUT,\\n headers={\"x-goog-api-key\": os.environ[\"GEMINI_API_KEY\"]},\\n )\\n except requests.RequestException as exc:\\n raise ProviderError(type(exc).__name__) from exc\\n+\\n+ def draft(self, system_prompt: str, user_prompt: str, schema: dict):\\n+ payload = {\\n+ \"systemInstruction\": {\"parts\": [{\"text\": system_prompt}]},\\n+ \"contents\": [{\"role\": \"user\", \"parts\": [{\"text\": user_prompt}]}],\\n+ \"generationConfig\": self._generation_config(schema, repair=False),\\n+ }\\n+ response = self._post(payload)\\n if response.status_code in (401, 403):\\n raise ProviderAuthError(f\"HTTP {response.status_code}\")\\n if response.status_code == 400 and (\\n@@ -197,6 +284,19 @@ def draft(self, system_prompt: str, user_prompt: str, schema: dict):\\n if response.status_code >= 400:\\n raise ProviderError(\\n f\"HTTP {response.status_code}: {_body_snippet(response)}\")\\n+ text = self._final_text(response)\\n+ if text is None:\\n+ return None\\n+ try:\\n+ return extract_json(text)\\n+ except ValueError as exc:\\n+ return self._repair(text, schema, exc)\\n+\\n+ def _final_text(self, response):\\n+ \"\"\"Final-answer text only. Returns None for genuine content blocks;\\n+ raises ProviderError on technical failures. Thought parts (the\\n+ reasoning trace) are excluded and never reach the JSON parser, the\\n+ published data or the logs.\"\"\"\\n data = response.json()\\n feedback = data.get(\"promptFeedback\") or {}\\n if feedback.get(\"blockReason\"):\\n@@ -217,46 +317,72 @@ def draft(self, system_prompt: str, user_prompt: str, schema: dict):\\n # the failover loop try the next provider.\\n raise ProviderError(f\"finishReason {finish}\")\\n parts = (candidate.get(\"content\") or {}).get(\"parts\") or []\\n- text = \"\".join(p.get(\"text\", \"\") for p in parts)\\n+ text = \"\".join(\\n+ p.get(\"text\", \"\") for p in parts\\n+ if isinstance(p, dict) and not p.get(\"thought\"))\\n if not text.strip():\\n raise ProviderError(\"empty response text\")\\n+ return text\\n+\\n+ def _repair(self, broken: str, schema: dict, exc) -> dict:\\n+ \"\"\"One minimal-thinking retry that fixes JSON SYNTAX only.\\n+\\n+ Content problems (missing quotes, unsupported claims) never take\\n+ this path - they surface after parsing and are handled by the\\n+ failover policy in write.py.\\n+ \"\"\"\\n+ print(f\"write: {self.label} returned malformed JSON ({exc}); \"\\n+ \"attempting format repair\")\\n+ payload = {\\n+ \"systemInstruction\": {\"parts\": [{\"text\": REPAIR_SYSTEM_PROMPT}]},\\n+ \"contents\": [{\"role\": \"user\", \"parts\": [{\"text\": broken}]}],\\n+ \"generationConfig\": self._generation_config(schema, repair=True),\\n+ }\\n+ response = self._post(payload)\\n+ if response.status_code >= 400:\\n+ raise ProviderError(\\n+ f\"format repair HTTP {response.status_code}\")\\n+ text = self._final_text(response)\\n+ if not text:\\n+ raise ProviderError(\"format repair returned no text\")\\n try:\\n return extract_json(text)\\n- except ValueError as exc:\\n- raise ProviderError(f\"bad JSON from model: {exc}\") from exc\\n+ except ValueError as repair_exc:\\n+ raise ProviderError(\\n+ f\"bad JSON even after format repair: {repair_exc}\"\\n+ ) from repair_exc\\n \\n \\n class NvidiaProvider:\\n name = \"nvidia\"\\n \\n def __init__(self, model=None) -> None:\\n self.model = (model or os.environ.get(\"AVWIRE_NVIDIA_MODEL\")\\n- or \"deepseek-ai/deepseek-v3.1\")\\n+ or NVIDIA_DEFAULT_MODEL)\\n self.label = f\"{self.name}:{self.model}\"\\n \\n def available(self) -> bool:\\n return bool(os.environ.get(\"NVIDIA_API_KEY\"))\\n \\n- def draft(self, system_prompt: str, user_prompt: str, schema: dict):\\n- # No guided decoding on NIM across all models: enforce JSON by prompt\\n- # and recover it with extract_json; the caller validates the shape.\\n- instruction = (\\n- \"\\n\\nReturn ONLY one JSON object (no markdown fences, no prose) \"\\n- \"conforming exactly to this JSON Schema:\\n\"\\n- + json.dumps(schema, ensure_ascii=False)\\n- )\\n+ def _payload_extras(self, repair: bool) -> dict:\\n+ profile = MODEL_PROFILES.get(self.model, {})\\n+ extras = dict(profile.get(\"payload\") or {\"temperature\": 0.2})\\n+ if repair:\\n+ # Repair overrides replace the reasoning control wholesale;\\n+ # sampling stays as configured for the model.\\n+ extras.update(profile.get(\"repair\") or {})\\n+ return extras\\n+\\n+ def _post(self, messages: list, repair: bool):\\n payload = {\\n \"model\": self.model,\\n- \"messages\": [\\n- {\"role\": \"system\", \"content\": system_prompt},\\n- {\"role\": \"user\", \"content\": user_prompt + instruction},\\n- ],\\n- \"temperature\": 0.2,\\n+ \"messages\": messages,\\n \"max_tokens\": 8192,\\n \"stream\": False,\\n+ **self._payload_extras(repair),\\n }\\n try:\\n- response = requests.post(\\n+ return requests.post(\\n \"https://integrate.api.nvidia.com/v1/chat/completions\",\\n json=payload, timeout=HTTP_TIMEOUT,\\n headers={\\n@@ -267,18 +393,11 @@ def draft(self, system_prompt: str, user_prompt: str, schema: dict):\\n )\\n except requests.RequestException as exc:\\n raise ProviderError(type(exc).__name__) from exc\\n- if response.status_code in (401, 403):\\n- raise ProviderAuthError(f\"HTTP {response.status_code}\")\\n- if response.status_code in (402, 429):\\n- raise ProviderQuotaError(\\n- f\"HTTP {response.status_code} (credits exhausted?)\")\\n- if response.status_code == 404:\\n- raise ProviderError(\\n- f\"model not found: {self.model} \"\\n- \"(set repo variable AVWIRE_NVIDIA_MODEL)\")\\n- if response.status_code >= 400:\\n- raise ProviderError(\\n- f\"HTTP {response.status_code}: {_body_snippet(response)}\")\\n+\\n+ def _final_text(self, response):\\n+ \"\"\"Final-answer text only; message.reasoning_content (the reasoning\\n+ trace some NIM models emit) is never read, parsed, published or\\n+ logged. Returns None for content blocks; raises on failures.\"\"\"\\n data = response.json()\\n choices = data.get(\"choices\") or []\\n if not choices:\\n@@ -295,10 +414,64 @@ def draft(self, system_prompt: str, user_prompt: str, schema: dict):\\n text = (choice.get(\"message\") or {}).get(\"content\") or \"\"\\n if not text.strip():\\n raise ProviderError(\"empty response content\")\\n+ return text\\n+\\n+ def _check_status(self, response) -> None:\\n+ if response.status_code in (401, 403):\\n+ raise ProviderAuthError(f\"HTTP {response.status_code}\")\\n+ if response.status_code in (402, 429):\\n+ raise ProviderQuotaError(\\n+ f\"HTTP {response.status_code} (credits exhausted?)\")\\n+ if response.status_code == 404:\\n+ raise ProviderError(\\n+ f\"model not found: {self.model} \"\\n+ \"(not in this key's catalog yet?)\")\\n+ if response.status_code >= 400:\\n+ raise ProviderError(\\n+ f\"HTTP {response.status_code}: {_body_snippet(response)}\")\\n+\\n+ def draft(self, system_prompt: str, user_prompt: str, schema: dict):\\n+ # No guided decoding on NIM across all models: enforce JSON by prompt\\n+ # and recover it with extract_json; the caller validates the shape.\\n+ instruction = (\\n+ \"\\n\\nReturn ONLY one JSON object (no markdown fences, no prose) \"\\n+ \"conforming exactly to this JSON Schema:\\n\"\\n+ + json.dumps(schema, ensure_ascii=False)\\n+ )\\n+ response = self._post(\\n+ [{\"role\": \"system\", \"content\": system_prompt},\\n+ {\"role\": \"user\", \"content\": user_prompt + instruction}],\\n+ repair=False,\\n+ )\\n+ self._check_status(response)\\n+ text = self._final_text(response)\\n+ if text is None:\\n+ return None\\n try:\\n return extract_json(text)\\n except ValueError as exc:\\n- raise ProviderError(f\"bad JSON from model: {exc}\") from exc\\n+ return self._repair(text, exc)\\n+\\n+ def _repair(self, broken: str, exc) -> dict:\\n+ \"\"\"One no-thinking retry that fixes JSON SYNTAX only; content\\n+ failures never take this path (see GeminiProvider._repair).\"\"\"\\n+ print(f\"write: {self.label} returned malformed JSON ({exc}); \"\\n+ \"attempting format repair\")\\n+ response = self._post(\\n+ [{\"role\": \"system\", \"content\": REPAIR_SYSTEM_PROMPT},\\n+ {\"role\": \"user\", \"content\": broken}],\\n+ repair=True,\\n+ )\\n+ self._check_status(response)\\n+ text = self._final_text(response)\\n+ if not text:\\n+ raise ProviderError(\"format repair returned no text\")\\n+ try:\\n+ return extract_json(text)\\n+ except ValueError as repair_exc:\\n+ raise ProviderError(\\n+ f\"bad JSON even after format repair: {repair_exc}\"\\n+ ) from repair_exc\\n \\n \\n _REGISTRY = {\\n@@ -35,13 +35,15 @@\\n DATA_DIR,\\n MAX_FLASHES,\\n RAW_DIR,\\n+ group_has_material,\\n iso_minute,\\n load_json,\\n norm_url,\\n now_utc,\\n parse_iso,\\n save_json,\\n slugify,\\n+ squash_text,\\n )\\n from providers import (\\n ProviderAuthError,\\n@@ -362,7 +364,12 @@ def group_prompt(group: dict) -> str:\\n for idx, item in enumerate(group.get(\"items\", []), 1):\\n title = str(item.get(\"title\") or \"\")[:300] # defensive prompt-cost cap\\n lines.append(f\"{idx}. [{item.get('source', '?')}] {title}\")\\n- lines.append(f\" published: {item.get('publishedUtc', '?')}\")\\n+ if item.get(\"dateInferred\"):\\n+ # Never present our first-seen stamp as a publication time.\\n+ lines.append(f\" first seen: {item.get('publishedUtc', '?')} \"\\n+ \"(the source provides no publish date)\")\\n+ else:\\n+ lines.append(f\" published: {item.get('publishedUtc', '?')}\")\\n summary = str(item.get(\"summary\") or \"\").strip()\\n if summary:\\n lines.append(f\" summary: {summary}\")\\n@@ -469,26 +476,17 @@ def validate_draft(draft):\\n return None\\n \\n \\n-MIN_SUMMARY_CHARS = 30\\n-\\n-\\n-def has_material(group: dict) -> bool:\\n- \"\"\"Code-level source-completeness check (editorial spec section 2).\\n-\\n- A group whose items carry no real summary text (title-only RSS rows)\\n- cannot survive the evidence rules, so it is consumed without spending\\n- an API call.\\n- \"\"\"\\n- return any(\\n- len(_squash(item.get(\"summary\") or \"\")) >= MIN_SUMMARY_CHARS\\n- for item in group.get(\"items\", []) if isinstance(item, dict))\\n+# Code-level source-completeness check (editorial spec section 2): groups\\n+# without real summary text are consumed without spending an API call.\\n+# Single source of truth lives in common.group_has_material.\\n+has_material = group_has_material\\n \\n \\n class DraftInvalid(Exception):\\n \"\"\"Draft failed validation or quote verification on every allowed try.\"\"\"\\n \\n \\n-def _validated_draft(provider, group: dict, tries: int):\\n+def _validated_draft(provider, group: dict, tries: int, ai_calls=None):\\n \"\"\"Call `provider` up to `tries` times until a draft survives validation\\n and quote verification.\\n \\n@@ -498,6 +496,8 @@ def _validated_draft(provider, group: dict, tries: int):\\n policy retries validation failures, never transport failures.\\n \"\"\"\\n for attempt in range(tries):\\n+ if ai_calls is not None:\\n+ ai_calls[provider.label] = ai_calls.get(provider.label, 0) + 1\\n candidate = draft_group(provider, group)\\n if candidate is None:\\n return None, None\\n@@ -516,11 +516,7 @@ def _validated_draft(provider, group: dict, tries: int):\\n raise DraftInvalid\\n \\n \\n-_WS_RE = re.compile(r\"\\s+\")\\n-\\n-\\n-def _squash(text) -> str:\\n- return _WS_RE.sub(\" \", str(text)).strip().casefold()\\n+_squash = squash_text # shared with common.item_has_material\\n \\n \\n def verify_facts(draft: dict, group: dict, label: str) -> list:\\n@@ -594,6 +590,19 @@ def build_article(draft: dict, group: dict, now, used_ids: set, writer=None,\\n \"en\": draft.get(\"en\", {}),\\n \"sources\": sources,\\n }\\n+ source_dates = []\\n+ for item in items:\\n+ if item.get(\"dateInferred\"):\\n+ continue\\n+ try:\\n+ source_dates.append(parse_iso(str(item.get(\"publishedUtc\"))))\\n+ except (TypeError, ValueError):\\n+ continue\\n+ if source_dates:\\n+ # The source's own newest publication time - the site displays this,\\n+ # never the generation time, so multi-day-old releases are not\\n+ # presented as breaking news.\\n+ article[\"sourcePublishedUtc\"] = iso_minute(max(source_dates))\\n if writer:\\n article[\"writer\"] = writer # provenance: \"provider:model\"\\n if facts is not None:\\n@@ -776,6 +785,7 @@ def main() -> None:\\n queued_entries, queued_groups, queued_ids = [], [], set()\\n rejected_groups, rejected_ids = [], set()\\n dead_auth: set = set()\\n+ ai_calls: dict = {}\\n skipped = 0\\n \\n # --- 1. publish entries a human approved in data/review.json ---------\\n@@ -839,7 +849,7 @@ def main() -> None:\\n tries = 2 if provider is providers[0] else 1\\n try:\\n candidate, facts = _validated_draft(provider, group,\\n- tries)\\n+ tries, ai_calls)\\n except DraftInvalid:\\n continue # try the next provider in the chain\\n except ProviderAuthError as exc:\\n@@ -948,6 +958,9 @@ def main() -> None:\\n f\"{len(queued_entries)} for review, rejected \"\\n f\"{len(rejected_groups)}, skipped {skipped}, \"\\n f\"pending {remaining_count}\")\\n+ calls_note = \" \".join(f\"{label}={n}\" for label, n in ai_calls.items()) \\\\n+ or \"none\"\\n+ print(f\"write stats: ai_calls {calls_note}\")\\n if providers and dead_auth and dead_auth == {p.name for p in providers}:\\n # Every configured platform failed authentication: turn the Actions\\n # run red instead of silently green.\\n@@ -21,8 +21,8 @@\\n {\\n \"title\": \"FAA archives old advisory circular\",\\n \"url\": \"https://www.faa.gov/newsroom/old-advisory\",\\n- \"publishedUtc\": \"{{H-72}}\",\\n- \"summary\": \"Stale item, three days old; must be dropped.\",\\n+ \"publishedUtc\": \"{{H-130}}\",\\n+ \"summary\": \"Stale item, older than the 120h window; must be dropped.\",\\n \"image\": null\\n },\\n {\\n@@ -5,9 +5,10 @@\\n py tests\\\\test_dedupe.py\\n \\n Fixtures under tests/fixtures/dedupe/ use {{H-n}} / {{D-n}} placeholders\\n-(n hours / days before now) so the 48-hour and 21-day windows never go\\n-stale. Each test materializes the fixtures into a temp AVWIRE_DATA_DIR\\n-and runs dedupe.py as a subprocess, never touching the real data/ dir.\\n+(n hours / days before now) so the freshness (default 120 h) and 21-day\\n+windows never go stale. Each test materializes the fixtures into a temp\\n+AVWIRE_DATA_DIR and runs dedupe.py as a subprocess, never touching the\\n+real data/ dir.\\n \"\"\"\\n from __future__ import annotations\\n \\n@@ -51,8 +52,12 @@ def _materialize_fixtures(data_dir: Path) -> None:\\n dst.write_text(_render(src.read_text(encoding=\"utf-8\")), encoding=\"utf-8\")\\n \\n \\n-def _run_dedupe(data_dir: Path) -> str:\\n+def _run_dedupe(data_dir: Path, extra_env: dict | None = None) -> str:\\n env = dict(os.environ, AVWIRE_DATA_DIR=str(data_dir))\\n+ env.pop(\"NEWS_MAX_AGE_HOURS\", None)\\n+ env.pop(\"AVWIRE_MAX_AGE_HOURS\", None)\\n+ if extra_env:\\n+ env.update(extra_env)\\n proc = subprocess.run(\\n [sys.executable, str(DEDUPE_SCRIPT)],\\n cwd=REPO,\\n@@ -66,6 +71,15 @@ def _run_dedupe(data_dir: Path) -> str:\\n return proc.stdout.strip()\\n \\n \\n+def _summary_line(stdout: str) -> str:\\n+ \"\"\"First stdout line: 'dedupe: N raw items -> N fresh -> N groups'.\\n+ (Config warnings may precede it; the stats line follows it.)\"\"\"\\n+ for line in stdout.splitlines():\\n+ if line.startswith(\"dedupe:\"):\\n+ return line\\n+ return stdout\\n+\\n+\\n def _load_pending(data_dir: Path) -> dict:\\n with open(data_dir / \"pending.json\", encoding=\"utf-8\") as fh:\\n return json.load(fh)\\n@@ -95,7 +109,13 @@ def test_filtering_grouping_and_ranking() -> None:\\n # 15 raw (easa ok=false items are carried forward, not skipped)\\n # -> 11 fresh (dropped: seen URL, seen title, stale, empty title)\\n # -> 5 groups.\\n- assert stdout == \"dedupe: 15 raw items -> 11 fresh -> 5 groups\", stdout\\n+ assert _summary_line(stdout) == \\\\n+ \"dedupe: 15 raw items -> 11 fresh -> 5 groups\", stdout\\n+ assert \"skipped_too_old=1\" in stdout, stdout\\n+ assert \"skipped_seen_url=1\" in stdout, stdout\\n+ assert \"skipped_seen_title=1\" in stdout, stdout\\n+ assert \"skipped_untitled=1\" in stdout, stdout\\n+ assert \"window=120h\" in stdout, stdout\\n \\n # dedupe must never write seen.json.\\n assert (data_dir / \"seen.json\").read_bytes() == seen_before\\n@@ -114,7 +134,7 @@ def test_filtering_grouping_and_ranking() -> None:\\n all_titles = [i[\"title\"] for g in groups for i in g[\"items\"]]\\n for gone in (\\n \"FAA statement on winter operations\", # seen URL\\n- \"FAA archives old advisory circular\", # stale (72 h)\\n+ \"FAA archives old advisory circular\", # stale (130 h)\\n \"Boeing delivers 100th 787 to Emirates!\", # seen title, 2 days ago\\n \" \", # empty title\\n ):\\n@@ -207,7 +227,8 @@ def test_group_cap_and_recency_order() -> None:\\n )\\n # No seen.json at all: dedupe must cope with it missing.\\n stdout = _run_dedupe(data_dir)\\n- assert stdout == \"dedupe: 15 raw items -> 15 fresh -> 12 groups\", stdout\\n+ assert _summary_line(stdout) == \\\\n+ \"dedupe: 15 raw items -> 15 fresh -> 12 groups\", stdout\\n \\n groups = _load_pending(data_dir)[\"groups\"]\\n assert len(groups) == dedupe.MAX_GROUPS == 12\\n@@ -221,17 +242,100 @@ def test_empty_data_dir() -> None:\\n with tempfile.TemporaryDirectory(prefix=\"avwire-dedupe-\") as tmp:\\n data_dir = Path(tmp)\\n stdout = _run_dedupe(data_dir)\\n- assert stdout == \"dedupe: 0 raw items -> 0 fresh -> 0 groups\", stdout\\n+ assert _summary_line(stdout) == \\\\n+ \"dedupe: 0 raw items -> 0 fresh -> 0 groups\", stdout\\n pending = _load_pending(data_dir)\\n assert pending[\"groups\"] == []\\n \\n \\n+def _write_snapshot(data_dir: Path, key: str, items: list) -> None:\\n+ raw_dir = data_dir / \"raw\"\\n+ raw_dir.mkdir(parents=True, exist_ok=True)\\n+ (raw_dir / f\"{key}.json\").write_text(\\n+ json.dumps({\"source\": key, \"fetchedUtc\": _stamp(hours=0), \"ok\": True,\\n+ \"error\": None, \"items\": items}, ensure_ascii=False),\\n+ encoding=\"utf-8\")\\n+\\n+\\n+def _mk_item(title: str, url: str, hours_ago: float, summary: str) -> dict:\\n+ dt = datetime.now(timezone.utc) - timedelta(hours=hours_ago)\\n+ return {\"title\": title, \"url\": url,\\n+ \"publishedUtc\": dt.strftime(\"%Y-%m-%dT%H:%MZ\"),\\n+ \"summary\": summary, \"image\": None}\\n+\\n+\\n+def test_freshness_window_and_future_guard() -> None:\\n+ \"\"\"120h default window (env-tunable, validated) + future-date guard.\"\"\"\\n+ body = \"A perfectly usable summary sentence with plenty of real text.\"\\n+ items = [\\n+ _mk_item(\"Future item within clock skew\", \"https://www.faa.gov/n/f2\",\\n+ -2, body), # +2h -> tolerated\\n+ _mk_item(\"Future item beyond clock skew\", \"https://www.faa.gov/n/f8\",\\n+ -8, body), # +8h -> source clock error, dropped\\n+ _mk_item(\"Hundred hour old official notice\", \"https://www.faa.gov/n/o1\",\\n+ 100, body), # kept by the 120h default\\n+ _mk_item(\"Ancient item far outside window\", \"https://www.faa.gov/n/o2\",\\n+ 130, body), # dropped\\n+ ]\\n+ with tempfile.TemporaryDirectory(prefix=\"avwire-dedupe-\") as tmp:\\n+ data_dir = Path(tmp)\\n+ _write_snapshot(data_dir, \"faa\", items)\\n+\\n+ stdout = _run_dedupe(data_dir)\\n+ assert _summary_line(stdout) == \\\\n+ \"dedupe: 4 raw items -> 2 fresh -> 2 groups\", stdout\\n+ assert \"skipped_future_date=1\" in stdout, stdout\\n+ assert \"skipped_too_old=1\" in stdout, stdout\\n+ titles = [g[\"items\"][0][\"title\"]\\n+ for g in _load_pending(data_dir)[\"groups\"]]\\n+ assert \"Future item within clock skew\" in titles\\n+ assert \"Hundred hour old official notice\" in titles\\n+\\n+ # Narrower window via env: the 100h item now falls out too.\\n+ stdout = _run_dedupe(data_dir, {\"NEWS_MAX_AGE_HOURS\": \"72\"})\\n+ assert _summary_line(stdout) == \\\\n+ \"dedupe: 4 raw items -> 1 fresh -> 1 groups\", stdout\\n+ assert \"window=72h\" in stdout, stdout\\n+\\n+ # Invalid values fall back to the safe default with a warning.\\n+ for bad in (\"abc\", \"12\", \"999\"):\\n+ stdout = _run_dedupe(data_dir, {\"NEWS_MAX_AGE_HOURS\": bad})\\n+ assert \"config: NEWS_MAX_AGE_HOURS\" in stdout, (bad, stdout)\\n+ assert \"window=120h\" in stdout, (bad, stdout)\\n+\\n+\\n+def test_material_groups_rank_first() -> None:\\n+ \"\"\"Title-only groups must not crowd real stories out of the cap.\"\"\"\\n+ body = \"A real summary carrying enough evidence text for the model.\"\\n+ items = [\\n+ _mk_item(\"Newest but title only headline item\",\\n+ \"https://www.faa.gov/n/thin\", 1, \"\"),\\n+ _mk_item(\"Older official notice with a full summary\",\\n+ \"https://www.faa.gov/n/full\", 30, body),\\n+ _mk_item(\"Summary that just repeats its own headline\",\\n+ \"https://www.faa.gov/n/echo\", 2,\\n+ \"Summary that just repeats its own headline\"),\\n+ ]\\n+ with tempfile.TemporaryDirectory(prefix=\"avwire-dedupe-\") as tmp:\\n+ data_dir = Path(tmp)\\n+ _write_snapshot(data_dir, \"faa\", items)\\n+ stdout = _run_dedupe(data_dir)\\n+ assert _summary_line(stdout) == \\\\n+ \"dedupe: 3 raw items -> 3 fresh -> 3 groups\", stdout\\n+ groups = _load_pending(data_dir)[\"groups\"]\\n+ assert groups[0][\"items\"][0][\"title\"] == \\\\n+ \"Older official notice with a full summary\", groups\\n+ assert \"groups_with_material=1/3\" in stdout, stdout\\n+\\n+\\n def main() -> None:\\n tests = [\\n test_norm_title_and_tiebreak,\\n test_filtering_grouping_and_ranking,\\n test_group_cap_and_recency_order,\\n test_empty_data_dir,\\n+ test_freshness_window_and_future_guard,\\n+ test_material_groups_rank_first,\\n ]\\n for test in tests:\\n test()\\n@@ -0,0 +1,282 @@\\n+\"\"\"Offline tests for pipeline/providers.py request payloads and parsing.\\n+\\n+Run from the repo root: py tests\\\\test_providers_payloads.py\\n+\\n+requests.post is monkeypatched to capture exactly what each model profile\\n+sends and to feed canned responses back - no network, no API keys, no\\n+NVIDIA/Gemini quota. Verifies:\\n+ - per-model reasoning/sampling fields (MODEL_PROFILES)\\n+ - the no-thinking format-repair path (JSON syntax only)\\n+ - reasoning traces (reasoning_content / thought parts) never reach the\\n+ JSON parser or the returned draft\\n+\"\"\"\\n+from __future__ import annotations\\n+\\n+import json\\n+import os\\n+import sys\\n+from pathlib import Path\\n+\\n+ROOT = Path(__file__).resolve().parent.parent\\n+sys.path.insert(0, str(ROOT / \"pipeline\"))\\n+os.environ.setdefault(\"NVIDIA_API_KEY\", \"test-key-not-real\")\\n+os.environ.setdefault(\"GEMINI_API_KEY\", \"test-key-not-real\")\\n+\\n+import providers # noqa: E402\\n+\\n+_pass = 0\\n+_fail = 0\\n+\\n+\\n+def check(cond, label):\\n+ global _pass, _fail\\n+ if cond:\\n+ _pass += 1\\n+ else:\\n+ _fail += 1\\n+ print(f\"FAIL: {label}\")\\n+\\n+\\n+class FakeResponse:\\n+ def __init__(self, payload, status=200):\\n+ self.status_code = status\\n+ self._payload = payload\\n+ self.text = json.dumps(payload)\\n+\\n+ def json(self):\\n+ return self._payload\\n+\\n+\\n+def nvidia_ok(content, reasoning=\"LEAKED-REASONING-TRACE {not json\"):\\n+ return FakeResponse({\"choices\": [{\\n+ \"finish_reason\": \"stop\",\\n+ \"message\": {\"content\": content, \"reasoning_content\": reasoning},\\n+ }]})\\n+\\n+\\n+def gemini_ok(parts):\\n+ return FakeResponse({\"candidates\": [{\\n+ \"finishReason\": \"STOP\", \"content\": {\"parts\": parts},\\n+ }]})\\n+\\n+\\n+def with_post(fake_post, fn):\\n+ original = providers.requests.post\\n+ providers.requests.post = fake_post\\n+ try:\\n+ return fn()\\n+ finally:\\n+ providers.requests.post = original\\n+\\n+\\n+SCHEMA = {\"type\": \"object\"}\\n+GOOD = '{\"ok\": 1}'\\n+\\n+\\n+def test_nvidia_model_profiles():\\n+ \"\"\"Each NIM model sends exactly its documented reasoning controls.\"\"\"\\n+ reasoning_keys = (\"reasoning_effort\", \"chat_template_kwargs\", \"thinking\",\\n+ \"enable_thinking\", \"thinking_level\", \"reasoningMode\")\\n+ cases = {\\n+ \"nvidia/nemotron-3-ultra-550b-a55b\": dict(\\n+ ctk={\"enable_thinking\": True, \"medium_effort\": True},\\n+ temperature=1.0, top_p=0.95),\\n+ \"nvidia/nemotron-3-super-120b-a12b\": dict(\\n+ effort=\"low\", temperature=1.0, top_p=0.95),\\n+ \"deepseek-ai/deepseek-v4-pro\": dict(\\n+ effort=\"high\", temperature=1.0, top_p=0.95),\\n+ \"qwen/qwen3.5-397b-a17b\": dict(\\n+ ctk={\"enable_thinking\": True}, temperature=0.6, top_p=0.95,\\n+ top_k=20, presence_penalty=0.0),\\n+ \"z-ai/glm-5.2\": dict(temperature=0.1), # provider-default reasoning\\n+ \"mistralai/mistral-medium-3.5-128b\": dict(\\n+ effort=\"high\", temperature=0.7),\\n+ }\\n+ for model, want in cases.items():\\n+ captured = []\\n+\\n+ def fake_post(url, json=None, timeout=None, headers=None):\\n+ captured.append(json)\\n+ return nvidia_ok(GOOD)\\n+\\n+ provider = providers.NvidiaProvider(model)\\n+ result = with_post(fake_post,\\n+ lambda: provider.draft(\"sys\", \"user\", SCHEMA))\\n+ check(result == {\"ok\": 1}, f\"{model}: draft parsed\")\\n+ p = captured[0]\\n+ check(p[\"model\"] == model and p[\"max_tokens\"] == 8192\\n+ and p[\"stream\"] is False, f\"{model}: base payload\")\\n+ check(p.get(\"temperature\") == want[\"temperature\"],\\n+ f\"{model}: temperature {p.get('temperature')}\")\\n+ if \"top_p\" in want:\\n+ check(p.get(\"top_p\") == want[\"top_p\"], f\"{model}: top_p\")\\n+ if \"top_k\" in want:\\n+ check(p.get(\"top_k\") == want[\"top_k\"]\\n+ and p.get(\"presence_penalty\") == want[\"presence_penalty\"],\\n+ f\"{model}: qwen thinking-mode sampling\")\\n+ if \"ctk\" in want:\\n+ check(p.get(\"chat_template_kwargs\") == want[\"ctk\"]\\n+ and \"reasoning_effort\" not in p,\\n+ f\"{model}: chat_template_kwargs {p.get('chat_template_kwargs')}\")\\n+ elif \"effort\" in want:\\n+ check(p.get(\"reasoning_effort\") == want[\"effort\"]\\n+ and \"chat_template_kwargs\" not in p,\\n+ f\"{model}: reasoning_effort {p.get('reasoning_effort')}\")\\n+ else: # GLM: no reasoning field of any kind on the wire\\n+ check(all(k not in p for k in reasoning_keys),\\n+ f\"{model}: no invented reasoning fields, got {sorted(p)}\")\\n+ print(\"test_nvidia_model_profiles: done\")\\n+\\n+\\n+def test_nvidia_reasoning_trace_never_parsed():\\n+ \"\"\"reasoning_content is ignored; empty final content is a failure.\"\"\"\\n+ def fake_post(url, json=None, timeout=None, headers=None):\\n+ return nvidia_ok(GOOD, reasoning='{\"leak\": true}')\\n+\\n+ provider = providers.NvidiaProvider(\"nvidia/nemotron-3-ultra-550b-a55b\")\\n+ result = with_post(fake_post,\\n+ lambda: provider.draft(\"sys\", \"user\", SCHEMA))\\n+ check(result == {\"ok\": 1} and \"leak\" not in result,\\n+ \"final content parsed, reasoning trace ignored\")\\n+\\n+ def fake_post_empty(url, json=None, timeout=None, headers=None):\\n+ return nvidia_ok(\"\", reasoning=GOOD)\\n+\\n+ try:\\n+ with_post(fake_post_empty,\\n+ lambda: provider.draft(\"sys\", \"user\", SCHEMA))\\n+ check(False, \"empty final content must raise, not fall back to trace\")\\n+ except providers.ProviderError:\\n+ check(True, \"empty final content raises ProviderError\")\\n+ print(\"test_nvidia_reasoning_trace_never_parsed: done\")\\n+\\n+\\n+def test_nvidia_format_repair():\\n+ \"\"\"Broken JSON syntax triggers ONE no-thinking repair call.\"\"\"\\n+ calls = []\\n+\\n+ def fake_post(url, json=None, timeout=None, headers=None):\\n+ calls.append(json)\\n+ if len(calls) == 1:\\n+ return nvidia_ok('{\"ok\": 1,,}') # syntax error\\n+ return nvidia_ok(GOOD)\\n+\\n+ provider = providers.NvidiaProvider(\"nvidia/nemotron-3-super-120b-a12b\")\\n+ result = with_post(fake_post,\\n+ lambda: provider.draft(\"sys\", \"user\", SCHEMA))\\n+ check(result == {\"ok\": 1}, \"repair produced a parseable draft\")\\n+ check(len(calls) == 2, f\"exactly one repair call, got {len(calls)}\")\\n+ repair = calls[1]\\n+ check(repair[\"messages\"][0][\"content\"] == providers.REPAIR_SYSTEM_PROMPT,\\n+ \"repair uses the syntax-only system prompt\")\\n+ check(repair.get(\"reasoning_effort\") == \"none\",\\n+ f\"repair disables reasoning, got {repair.get('reasoning_effort')}\")\\n+ check(calls[0].get(\"reasoning_effort\") == \"low\",\\n+ \"normal call keeps its configured reasoning\")\\n+ print(\"test_nvidia_format_repair: done\")\\n+\\n+\\n+def test_gemini_model_profiles():\\n+ \"\"\"Gemini 3.6/3.5 send thinkingLevel medium; unknown models send none.\"\"\"\\n+ for model in (\"gemini-3.6-flash\", \"gemini-3.5-flash\"):\\n+ captured = []\\n+\\n+ def fake_post(url, json=None, timeout=None, headers=None):\\n+ captured.append(json)\\n+ return gemini_ok([{\"text\": GOOD}])\\n+\\n+ provider = providers.GeminiProvider(model)\\n+ result = with_post(fake_post,\\n+ lambda: provider.draft(\"sys\", \"user\", SCHEMA))\\n+ check(result == {\"ok\": 1}, f\"{model}: draft parsed\")\\n+ config = captured[0][\"generationConfig\"]\\n+ check(config.get(\"thinkingConfig\") == {\"thinkingLevel\": \"medium\"},\\n+ f\"{model}: thinkingLevel medium\")\\n+ check(config.get(\"temperature\") == 0.1\\n+ and config.get(\"maxOutputTokens\") == 8192,\\n+ f\"{model}: sampling per profile\")\\n+ check(config.get(\"responseMimeType\") == \"application/json\"\\n+ and config.get(\"responseJsonSchema\") == SCHEMA,\\n+ f\"{model}: structured output config kept\")\\n+\\n+ captured = []\\n+\\n+ def fake_post(url, json=None, timeout=None, headers=None):\\n+ captured.append(json)\\n+ return gemini_ok([{\"text\": GOOD}])\\n+\\n+ provider = providers.GeminiProvider(\"gemini-2.5-flash\")\\n+ with_post(fake_post, lambda: provider.draft(\"sys\", \"user\", SCHEMA))\\n+ check(\"thinkingConfig\" not in captured[0][\"generationConfig\"],\\n+ \"unprofiled Gemini model sends no thinkingConfig\")\\n+ print(\"test_gemini_model_profiles: done\")\\n+\\n+\\n+def test_gemini_thought_parts_excluded():\\n+ \"\"\"Thought parts (reasoning trace) never reach the parser or output.\"\"\"\\n+ def fake_post(url, json=None, timeout=None, headers=None):\\n+ return gemini_ok([\\n+ {\"thought\": True, \"text\": '{\"leak\": \"reasoning JSON\"}'},\\n+ {\"text\": GOOD},\\n+ ])\\n+\\n+ provider = providers.GeminiProvider(\"gemini-3.6-flash\")\\n+ result = with_post(fake_post,\\n+ lambda: provider.draft(\"sys\", \"user\", SCHEMA))\\n+ check(result == {\"ok\": 1} and \"leak\" not in result,\\n+ \"only final (non-thought) parts are parsed\")\\n+\\n+ def fake_post_only_thought(url, json=None, timeout=None, headers=None):\\n+ return gemini_ok([{\"thought\": True, \"text\": GOOD}])\\n+\\n+ try:\\n+ with_post(fake_post_only_thought,\\n+ lambda: provider.draft(\"sys\", \"user\", SCHEMA))\\n+ check(False, \"thought-only response must raise\")\\n+ except providers.ProviderError:\\n+ check(True, \"thought-only response raises ProviderError\")\\n+ print(\"test_gemini_thought_parts_excluded: done\")\\n+\\n+\\n+def test_gemini_format_repair():\\n+ calls = []\\n+\\n+ def fake_post(url, json=None, timeout=None, headers=None):\\n+ calls.append(json)\\n+ if len(calls) == 1:\\n+ return gemini_ok([{\"text\": '{\"ok\": 1'}]) # truncated syntax\\n+ return gemini_ok([{\"text\": GOOD}])\\n+\\n+ provider = providers.GeminiProvider(\"gemini-3.6-flash\")\\n+ result = with_post(fake_post,\\n+ lambda: provider.draft(\"sys\", \"user\", SCHEMA))\\n+ check(result == {\"ok\": 1}, \"gemini repair produced a parseable draft\")\\n+ check(len(calls) == 2, f\"exactly one repair call, got {len(calls)}\")\\n+ repair_config = calls[1][\"generationConfig\"]\\n+ check(repair_config.get(\"thinkingConfig\") == {\"thinkingLevel\": \"minimal\"},\\n+ \"repair drops to minimal thinking\")\\n+ check(calls[1][\"systemInstruction\"][\"parts\"][0][\"text\"]\\n+ == providers.REPAIR_SYSTEM_PROMPT,\\n+ \"repair uses the syntax-only system prompt\")\\n+ print(\"test_gemini_format_repair: done\")\\n+\\n+\\n+def main():\\n+ tests = [\\n+ test_nvidia_model_profiles,\\n+ test_nvidia_reasoning_trace_never_parsed,\\n+ test_nvidia_format_repair,\\n+ test_gemini_model_profiles,\\n+ test_gemini_thought_parts_excluded,\\n+ test_gemini_format_repair,\\n+ ]\\n+ for test in tests:\\n+ test()\\n+ print(f\"{_pass} checks passed, {_fail} failed\")\\n+ if _fail:\\n+ sys.exit(1)\\n+ print(\"OK\")\\n+\\n+\\n+if __name__ == \"__main__\":\\n+ main()\\n@@ -218,6 +218,8 @@ def fake_draft(client, group):\\n check(biz.get(\"eventStatus\") == \"confirmed\", \"eventStatus stored\")\\n check(biz.get(\"entities\", {}).get(\"organizations\")\\n == [\"International Air Transport Association\"], \"entities stored\")\\n+ check(biz.get(\"sourcePublishedUtc\") == \"2026-07-24T09:00Z\",\\n+ f\"source publish time recorded, got {biz.get('sourcePublishedUtc')}\")\\n \\n # --- the safety story sits in the review queue, not on the site --------\\n review = load(DATA / \"review.json\")\\n@@ -309,6 +311,9 @@ def fake_draft(client, group):\\n \"investigation\"],\\n \"risk flags carried onto the published article\")\\n check(len(safety.get(\"facts\", [])) == 2, \"verified facts carried over\")\\n+ check(safety.get(\"sourcePublishedUtc\") == \"2026-07-25T23:40Z\",\\n+ f\"newest source publish time kept through approval, \"\\n+ f\"got {safety.get('sourcePublishedUtc')}\")\\n check(load(DATA / \"review.json\") == [], \"review queue emptied after approval\")\\n flashes = load(DATA / \"flashes.json\")\\n check(flashes[0][\"articleId\"] == safety[\"id\"] and flashes[0][\"hot\"] is True,\\n@@ -483,6 +488,17 @@ def test_extract_json_and_validate_draft():\\n broken[\"eventStatus\"] = \"finalized\"\\n check(\"eventStatus\" in (write.validate_draft(broken) or \"\"),\\n \"unknown eventStatus rejected\")\\n+\\n+ ihm = common.item_has_material\\n+ check(ihm({\"title\": \"T\", \"summary\": \"A real sentence with plenty of \"\\n+ \"actual evidence text.\"}),\\n+ \"real summary counts as material\")\\n+ check(not ihm({\"title\": \"FAA issues new rule on flight time limits\",\\n+ \"summary\": \"FAA issues new rule on flight time limits\"}),\\n+ \"summary that merely repeats the headline is NOT material\")\\n+ check(not ihm({\"title\": \"T\", \"summary\": \"\"}), \"empty summary rejected\")\\n+ check(not ihm({\"title\": \"T\", \"summary\": \"too short\"}),\\n+ \"sub-threshold summary rejected\")\\n broken = json.loads(json.dumps(DRAFT_BIZ))\\n broken[\"cat\"] = \"sports\"\\n check(\"cat\" in (write.validate_draft(broken) or \"\"), \"bad cat rejected\")\\n@@ -563,7 +579,7 @@ def test_model_chain_and_routing_policy():\\n check(labels == [\"nvidia:nvidia/nemotron-3-ultra-550b-a55b\",\\n \"nvidia:nvidia/nemotron-3-super-120b-a12b\",\\n \"gemini:gemini-3-flash-preview\",\\n- \"nvidia:deepseek-ai/deepseek-v3.1\"],\\n+ f\"nvidia:{providers.NVIDIA_DEFAULT_MODEL}\"],\\n f\"order tokens honored in order, got {labels}\")\\n \\n # --- platform auth death skips same-platform fallback models ----------" }