From 5dfd2ad8a9c0c6fa7ce4b4881f1e2b3783c31d36 Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Wed, 9 Sep 2026 09:25:35 +0700 Subject: [PATCH] docs(skills): document fetch fallback chain and pinned target post Add a reader-proxy tier after defuddle, stop skipping URLs that only fail a plain classifier fetch, let a session pin an earlier draft as the target post, and tighten Substack image candidate handling (dedupe, stable numbering, reuse a resolved source post across a batch). --- .agents/skills/mt-add-image/SKILL.md | 2 +- .agents/skills/mt-add-post/SKILL.md | 2 +- .agents/skills/mt-add-video/SKILL.md | 2 +- .agents/skills/mt-webfetch/SKILL.md | 2 +- .claude/skills/mt-add-image/SKILL.md | 19 ++++++--- .claude/skills/mt-add-post/SKILL.md | 10 ++--- .claude/skills/mt-add-tags/SKILL.md | 2 +- .claude/skills/mt-add-url/SKILL.md | 2 +- .../references/newsletter-post-mechanics.md | 14 +++++-- .claude/skills/mt-add-video/SKILL.md | 6 +-- .claude/skills/mt-webfetch/SKILL.md | 40 ++++++++++++------- AGENTS.md | 6 +-- 12 files changed, 68 insertions(+), 39 deletions(-) diff --git a/.agents/skills/mt-add-image/SKILL.md b/.agents/skills/mt-add-image/SKILL.md index 36a5ab7..9b18d1d 100644 --- a/.agents/skills/mt-add-image/SKILL.md +++ b/.agents/skills/mt-add-image/SKILL.md @@ -1,6 +1,6 @@ --- name: mt-add-image -description: Add an image URL to today's Hugo newsletter Bonus Images section with a detected or user-provided label. Use directly for a known image URL or when dispatched by mt-add-url. +description: Add an image URL to the target Hugo newsletter post's Bonus Images section with a detected or user-provided label. Use directly for a known image URL or when dispatched by mt-add-url. --- # Add a newsletter image diff --git a/.agents/skills/mt-add-post/SKILL.md b/.agents/skills/mt-add-post/SKILL.md index 350b11a..be61879 100644 --- a/.agents/skills/mt-add-post/SKILL.md +++ b/.agents/skills/mt-add-post/SKILL.md @@ -1,6 +1,6 @@ --- name: mt-add-post -description: Add an article or blog URL to today's Hugo newsletter with its original title and a Vietnamese summary. Use directly for a known article URL or when dispatched by mt-add-url. +description: Add an article or blog URL to the target Hugo newsletter post with its original title and a Vietnamese summary. Use directly for a known article URL or when dispatched by mt-add-url. --- # Add a newsletter article diff --git a/.agents/skills/mt-add-video/SKILL.md b/.agents/skills/mt-add-video/SKILL.md index 80f2e93..130c6f5 100644 --- a/.agents/skills/mt-add-video/SKILL.md +++ b/.agents/skills/mt-add-video/SKILL.md @@ -1,6 +1,6 @@ --- name: mt-add-video -description: Add a YouTube URL to today's Hugo newsletter Bonus Videos section with a Vietnamese title and summary. Use directly for a known YouTube URL or when dispatched by mt-add-url. +description: Add a YouTube URL to the target Hugo newsletter post's Bonus Videos section with a Vietnamese title and summary. Use directly for a known YouTube URL or when dispatched by mt-add-url. --- # Add a newsletter video diff --git a/.agents/skills/mt-webfetch/SKILL.md b/.agents/skills/mt-webfetch/SKILL.md index 46d79d7..aae3031 100644 --- a/.agents/skills/mt-webfetch/SKILL.md +++ b/.agents/skills/mt-webfetch/SKILL.md @@ -1,6 +1,6 @@ --- name: mt-webfetch -description: Fetch public web content through the defuddle fallback only after the built-in fetch is blocked or returns unusable content. Do not use for paywalls, login walls, or as the first fetch attempt. +description: Fetch public web content through the fallback chain (defuddle, then a reader proxy) only after the built-in fetch is blocked or returns unusable content. Do not use for paywalls, login walls, or as the first fetch attempt. --- # Fetch blocked newsletter sources diff --git a/.claude/skills/mt-add-image/SKILL.md b/.claude/skills/mt-add-image/SKILL.md index 3d3237b..e156260 100644 --- a/.claude/skills/mt-add-image/SKILL.md +++ b/.claude/skills/mt-add-image/SKILL.md @@ -1,11 +1,11 @@ --- name: mt-add-image -description: 'Image handler for the Hugo blog newsletter. Adds an image URL to today''s newsletter Bonus → Images with a detected label. For Substack-hosted images it finds the source post (ByteByteGo and other configured publications) via RSS then sitemap, using the figure caption or post title as the label; for other sites it is best-effort; if no label is found it asks you to type one. Normally invoked by the mt-add-url meta skill after classification, but can be used directly for a known image URL.' +description: 'Image handler for the Hugo blog newsletter. Adds an image URL to the target newsletter post''s Bonus → Images with a detected label. For Substack-hosted images it finds the source post (ByteByteGo and other configured publications) via RSS then sitemap, using the figure caption or post title as the label; for other sites it is best-effort; if no label is found it asks you to type one. Normally invoked by the mt-add-url meta skill after classification, but can be used directly for a known image URL.' --- ## Overview -`mt-add-image` is the **image handler**: given an image URL, it resolves a human-readable **label** and inserts `![label](image_url)` into today's newsletter **Bonus → Images**. It does not classify/route — that is `mt-add-url`'s job. +`mt-add-image` is the **image handler**: given an image URL, it resolves a human-readable **label** and inserts `![label](image_url)` into the target newsletter post's **Bonus → Images** (today's post unless the user pinned another one). It does not classify/route — that is `mt-add-url`'s job. Shared scripts: `scripts/newsletter/`. Shared procedure: `../mt-add-url/references/newsletter-post-mechanics.md` — **follow it** for post find/create, Bonus insertion, and language rules. @@ -32,11 +32,14 @@ go run ./scripts/newsletter add-url "" # expect route:image; skip if dupl (When dispatched by `mt-add-url`, that check already ran — don't repeat it.) ### 2a. Substack image (`isSubstack: true` with `uuid`) -Find the source post: + +**First check this session.** If an earlier image in this session already resolved to a source post, check whether this image belongs to the same post before crawling again — a batch of images pasted together usually comes from one post. Reuse that post's `candidates` list directly; the deep crawl is slow and re-running it per image wastes minutes. + +Otherwise, find the source post: ```bash go run ./scripts/newsletter find-substack-post --uuid ``` -- `found: false` → retry with the deeper sitemap crawl (slower — fetches posts ~3 months back, capped at 40 fetches total across all publications; warn the user it may take a while): +- `found: false` → retry with the deeper sitemap crawl. The quick RSS pass only covers the newest posts, so a miss is the normal result for anything older than the current feed window — go straight to `--deep` rather than treating the miss as a dead end. It is slower (fetches posts ~3 months back, capped at 40 fetches total across all publications), so warn the user it may take a while: ```bash go run ./scripts/newsletter find-substack-post --uuid --deep ``` @@ -52,6 +55,11 @@ ByteByteGo doesn't attach captions to images (verified: empty `alt`, no `" ``` @@ -34,8 +34,15 @@ Use this skill only after a WebFetch attempt returned one of: ``` WebFetch(url: "https://defuddle.md/", prompt: "") ``` -3. Parse the returned markdown (has YAML frontmatter with title/description/etc.) -4. If defuddle also returns empty or an error, stop and report failure to user — do not keep retrying. +3. **Tier 2 — reader proxy.** If tier 1 returns an error (commonly `502 / empty body`) or a body with no usable content, try a reader proxy through WebFetch: + ``` + WebFetch(url: "https://r.jina.ai/", prompt: "") + ``` + Tier 1 and tier 2 fail independently — a site blocking one often still serves the other, so always attempt tier 2 before giving up. +4. Parse the returned markdown (tier 1 has YAML frontmatter with title/description/etc.) +5. If every tier fails, stop and report which tiers were tried and what each returned — do not keep retrying. + +Attempt each tier at most once. The whole chain is: built-in WebFetch → defuddle → reader proxy → report failure. ## How defuddle works @@ -49,12 +56,14 @@ The response is plain markdown. Use it directly when summarizing / extracting co ## Failure modes and exit -Give up after one retry. If defuddle returns: -- HTTP 4xx/5xx → report "both WebFetch and defuddle failed to fetch " and move on -- Empty markdown body → same -- Only frontmatter with no body → report as inaccessible +Give up once every tier has been tried once. Treat these as a tier failure and move to the next tier: +- HTTP 4xx/5xx +- Empty markdown body +- Only frontmatter with no body -Never loop. Never retry more than once. +When the last tier fails, report it plainly — name each tier and its result (e.g. "WebFetch 403, defuddle 502, reader proxy empty") so the user can decide whether to paste the text or supply another source. If a fetcher fails repeatedly across sessions for the same host family, say so: that is a signal to reorder or extend the chain, not to keep retrying. + +Never loop. Never retry a tier more than once. ## Security policy @@ -70,7 +79,10 @@ Never loop. Never retry more than once. User wanted to extract content from https://example.com/article WebFetch returned: "Request failed with status code 403" → Trigger mt-webfetch -→ go run ./scripts/newsletter fetch-via-defuddle "https://example.com/article" -→ Parse markdown output -→ Summarize as usual +→ Tier 1: go run ./scripts/newsletter fetch-via-defuddle "https://example.com/article" + → success: parse markdown output, summarize as usual + → "502 / empty body": continue +→ Tier 2: WebFetch(url: "https://r.jina.ai/https://example.com/article", prompt: ...) + → success: parse markdown output, summarize as usual + → failure: report "WebFetch 403, defuddle 502, reader proxy failed" and stop ``` diff --git a/AGENTS.md b/AGENTS.md index a8037f3..cdf2137 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,7 +47,7 @@ go run ./scripts/newsletter find-newsletter-number # next newsletter numbe go run ./scripts/newsletter list-existing-tags # existing tag frequencies go run ./scripts/newsletter detect-image-source "" # detect Substack image + uuid go run ./scripts/newsletter find-substack-post --uuid -go run ./scripts/newsletter fetch-via-defuddle "" # fallback fetch (defuddle proxy) +go run ./scripts/newsletter fetch-via-defuddle "" # fallback fetch (tier 1 of the fetch chain) ``` These are shared by all three tools — no tool-specific copies. @@ -56,7 +56,7 @@ These are shared by all three tools — no tool-specific copies. ## Newsletter Workflow Routing -The newsletter workflow adds URLs (articles, YouTube videos, images) to today's newsletter post and manages tags. How it surfaces depends on the tool: +The newsletter workflow adds URLs (articles, YouTube videos, images) to the target newsletter post — today's post, or an earlier unpublished draft the user pins for the session — and manages tags. How it surfaces depends on the tool: - **Claude Code / OpenCode** — invoke skills (both read `.claude/skills//SKILL.md` natively): - `mt-add-url` — meta dispatcher: classifies each URL, auto-invokes the right handler. **Default entry for adding URLs.** @@ -64,7 +64,7 @@ The newsletter workflow adds URLs (articles, YouTube videos, images) to today's - `mt-add-video` — YouTube link → Bonus → Videos - `mt-add-image` — image → Bonus → Images (labels Substack images via source-post lookup) - `mt-add-tags` — add/update tags in post frontmatter - - `mt-webfetch` — fallback web fetcher (defuddle proxy); use only when built-in WebFetch is blocked + - `mt-webfetch` — fallback web fetch chain (defuddle, then a reader proxy); use only when built-in WebFetch is blocked - **Codex** — discovers the repository-scoped adapters in `.agents/skills/`. Ask it to add a URL for implicit routing or invoke `$mt-add-url` explicitly. `mt-add-url` dispatches `article` / `youtube` / `image`; other types (direct video files, documents, unknown) prompt the user to add or extend a handler.