tiennm99/miti99
1
0
Fork 0
mirror of https://github.com/tiennm99/miti99.git synced 2026-09-11 16:18:38 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(skills): document fetch fallback chain and pinned target post

Browse Source 
Add a reader-proxy tier after defuddle, stop skipping URLs that only fail a
plain classifier fetch, let a session pin an earlier draft as the target post,
and tighten Substack image candidate handling (dedupe, stable numbering,
reuse a resolved source post across a batch).
This commit is contained in:
tiennm99
2026-09-09 09:25:35 +07:00
parent 2b2826a0cb
commit 5dfd2ad8a9
12 changed files with 68 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/mt-add-image/SKILL.md
+1 -1
View File
@@ -1,6 +1,6 @@
---
name: mt-add-image
description: Add an image URL to today's Hugo newsletter Bonus Images section with a detected or user-provided label. Use directly for a known image URL or when dispatched by mt-add-url.
description: Add an image URL to the target Hugo newsletter post's Bonus Images section with a detected or user-provided label. Use directly for a known image URL or when dispatched by mt-add-url.
---
# Add a newsletter image
.agents/skills/mt-add-post/SKILL.md
+1 -1
View File
@@ -1,6 +1,6 @@
---
name: mt-add-post
description: Add an article or blog URL to today's Hugo newsletter with its original title and a Vietnamese summary. Use directly for a known article URL or when dispatched by mt-add-url.
description: Add an article or blog URL to the target Hugo newsletter post with its original title and a Vietnamese summary. Use directly for a known article URL or when dispatched by mt-add-url.
---
# Add a newsletter article
.agents/skills/mt-add-video/SKILL.md
+1 -1
View File
@@ -1,6 +1,6 @@
---
name: mt-add-video
description: Add a YouTube URL to today's Hugo newsletter Bonus Videos section with a Vietnamese title and summary. Use directly for a known YouTube URL or when dispatched by mt-add-url.
description: Add a YouTube URL to the target Hugo newsletter post's Bonus Videos section with a Vietnamese title and summary. Use directly for a known YouTube URL or when dispatched by mt-add-url.
---
# Add a newsletter video
.agents/skills/mt-webfetch/SKILL.md
+1 -1
View File
@@ -1,6 +1,6 @@
---
name: mt-webfetch
description: Fetch public web content through the defuddle fallback only after the built-in fetch is blocked or returns unusable content. Do not use for paywalls, login walls, or as the first fetch attempt.
description: Fetch public web content through the fallback chain (defuddle, then a reader proxy) only after the built-in fetch is blocked or returns unusable content. Do not use for paywalls, login walls, or as the first fetch attempt.
---
# Fetch blocked newsletter sources
.claude/skills/mt-add-image/SKILL.md
+14 -5
View File
@@ -1,11 +1,11 @@
---
name: mt-add-image
description: 'Image handler for the Hugo blog newsletter. Adds an image URL to today''s newsletter Bonus → Images with a detected label. For Substack-hosted images it finds the source post (ByteByteGo and other configured publications) via RSS then sitemap, using the figure caption or post title as the label; for other sites it is best-effort; if no label is found it asks you to type one. Normally invoked by the mt-add-url meta skill after classification, but can be used directly for a known image URL.'
description: 'Image handler for the Hugo blog newsletter. Adds an image URL to the target newsletter post''s Bonus → Images with a detected label. For Substack-hosted images it finds the source post (ByteByteGo and other configured publications) via RSS then sitemap, using the figure caption or post title as the label; for other sites it is best-effort; if no label is found it asks you to type one. Normally invoked by the mt-add-url meta skill after classification, but can be used directly for a known image URL.'
---
## Overview
`mt-add-image` is the **image handler**: given an image URL, it resolves a human-readable **label** and inserts `![label](image_url)` into today's newsletter **Bonus → Images**. It does not classify/route — that is `mt-add-url`'s job.
`mt-add-image` is the **image handler**: given an image URL, it resolves a human-readable **label** and inserts `![label](image_url)` into the target newsletter post's **Bonus → Images** (today's post unless the user pinned another one). It does not classify/route — that is `mt-add-url`'s job.
Shared scripts: `scripts/newsletter/`. Shared procedure: `../mt-add-url/references/newsletter-post-mechanics.md` — **follow it** for post find/create, Bonus insertion, and language rules.
@@ -32,11 +32,14 @@ go run ./scripts/newsletter add-url "<url>" # expect route:image; skip if dupl
(When dispatched by `mt-add-url`, that check already ran — don't repeat it.)
### 2a. Substack image (`isSubstack: true` with `uuid`)
Find the source post:
**First check this session.** If an earlier image in this session already resolved to a source post, check whether this image belongs to the same post before crawling again — a batch of images pasted together usually comes from one post. Reuse that post's `candidates` list directly; the deep crawl is slow and re-running it per image wastes minutes.
Otherwise, find the source post:
```bash
go run ./scripts/newsletter find-substack-post --uuid <uuid>
```
- `found: false` → retry with the deeper sitemap crawl (slower — fetches posts ~3 months back, capped at 40 fetches total across all publications; warn the user it may take a while):
- `found: false` → retry with the deeper sitemap crawl. The quick RSS pass only covers the newest posts, so a miss is the normal result for anything older than the current feed window — go straight to `--deep` rather than treating the miss as a dead end. It is slower (fetches posts ~3 months back, capped at 40 fetches total across all publications), so warn the user it may take a while:
```bash
go run ./scripts/newsletter find-substack-post --uuid <uuid> --deep
```
@@ -52,6 +55,11 @@ ByteByteGo doesn't attach captions to images (verified: empty `alt`, no `<figcap
- Always include a "Type my own" escape.
3. Label = the user's pick (or the typed value). Preserve its original language, wording, capitalization, and punctuation; do not translate unless the user explicitly asks.
**Presenting candidates — rules that prevent mis-labelling:**
- **Dedupe first.** The list can contain the same title twice, once with literal escape sequences (e.g. `\u2019` instead of `'`). Show each title once, in its readable form.
- **Number once, never renumber.** Assign numbers on the first image from a post and keep those exact numbers for every later image from that same post. Mark already-used titles as used instead of dropping them and shifting the rest — a renumbered list makes the user's "2" ambiguous.
- **Confirm an ambiguous reply.** If a bare number could refer to two different titles (e.g. because a list was filtered), ask which one before writing. Do not guess.
### 2b. Non-Substack image (`isSubstack: false`)
Best-effort only (no reverse-image-search): if you happen to know the page that contains the image, WebFetch it and read the OpenGraph title / nearby caption. Usually there's no containing page from just an image URL → go to step 3.
@@ -63,7 +71,7 @@ If no label was detected, use `AskUserQuestion`:
If a Substack image wasn't found and the user tells you which publication it's from, offer to append that host to `scripts/newsletter/config/substack-publications.json`, then retry step 2a. This grows coverage for next time.
### 5. Insert into Bonus → Images
Follow `../mt-add-url/references/newsletter-post-mechanics.md` to find/create today's post, then add under **Images**:
Follow `../mt-add-url/references/newsletter-post-mechanics.md` to resolve/create the target post, then add under **Images**:
```markdown
**Images:**
![label](clean_image_url)
@@ -82,5 +90,6 @@ Create `### Bonus` / `**Images:**` if missing.
- [ ] Source detected (substack + uuid, or non-substack)
- [ ] Label confirmed (caption if present, else user picked from candidates, else typed)
- [ ] Candidates shown deduped, with numbering kept stable across images from the same post
- [ ] Entry under Bonus → **Images** (section/subsection created if missing)
- [ ] Label preserves original source language, wording, capitalization, and punctuation unless the user requested translation
.claude/skills/mt-add-post/SKILL.md
+5 -5
View File
@@ -1,13 +1,13 @@
---
name: mt-add-post
description: 'Article handler for the Hugo blog newsletter. Adds a single article/blog URL to today''s newsletter as a main-content entry with the original source title and a Vietnamese summary. Normally invoked by the mt-add-url meta skill after classification, but can be used directly for a known article URL.'
description: 'Article handler for the Hugo blog newsletter. Adds a single article/blog URL to the target newsletter post as a main-content entry with the original source title and a Vietnamese summary. Normally invoked by the mt-add-url meta skill after classification, but can be used directly for a known article URL.'
---
## Overview
`mt-add-post` is the **article handler**: given a clean article/blog URL, it extracts the source title and content, writes a Vietnamese summary, and inserts it into today's newsletter main content. It does **not** classify or route URLs — that is `mt-add-url`'s job. For YouTube/images/other types, use `mt-add-url`.
`mt-add-post` is the **article handler**: given a clean article/blog URL, it extracts the source title and content, writes a Vietnamese summary, and inserts it into the target newsletter post's main content (today's post unless the user pinned another one). It does **not** classify or route URLs — that is `mt-add-url`'s job. For YouTube/images/other types, use `mt-add-url`.
Shared scripts: `scripts/newsletter/`. Shared procedure: `../mt-add-url/references/newsletter-post-mechanics.md` (find/create today's post, newsletter numbering, section insertion, language rules) — **follow it** for all post mechanics.
Shared scripts: `scripts/newsletter/`. Shared procedure: `../mt-add-url/references/newsletter-post-mechanics.md` (resolve/create the target post, newsletter numbering, section insertion, language rules) — **follow it** for all post mechanics.
## Input
@@ -19,8 +19,8 @@ Trust `route: article`; skip if `duplicate` or not `accessible`.
## Workflow
1. **Post mechanics** — follow `../mt-add-url/references/newsletter-post-mechanics.md` to find/create today's post and (if new) get the newsletter number + apply the template.
2. **Extract** the original article title and main content. Preserve the source title exactly enough to remain recognizable: do not translate/localize it; keep source-language wording, capitalization, punctuation, and proper nouns from metadata (`og:title`, page title, or fetcher frontmatter).
1. **Post mechanics** — follow `../mt-add-url/references/newsletter-post-mechanics.md` to resolve/create the target post and (if new) get the newsletter number + apply the template.
2. **Extract** the original article title and main content. Try `WebFetch` first; if the host blocks it (403, Cloudflare challenge, empty body), work through the `mt-webfetch` fallback chain before giving up. A `accessible: false` from the router is not by itself a reason to skip — many public tech blogs block plain fetches but serve the fallback fetchers fine. Preserve the source title exactly enough to remain recognizable: do not translate/localize it; keep source-language wording, capitalization, punctuation, and proper nouns from metadata (`og:title`, page title, or fetcher frontmatter).
3. **Summarize in Vietnamese** — 1-2 paragraphs, max 300 words, professional tone for junior developers. Optionally add 3-5 key points.
4. **Write** the main-content block and insert it **before** the `### Bonus` section (or append at end of file if there is no Bonus yet — do not create an empty Bonus):
.claude/skills/mt-add-tags/SKILL.md
+1 -1
View File
@@ -17,7 +17,7 @@ This skill analyzes Hugo blog post content and generates a relevant list of tags
### 1. Find the target post
If the user specifies a post (date, path, or title), use that. Otherwise, use today's post:
If the user specifies a post (date, path, or title), use that. If the session has been working on a pinned post (an unpublished draft from an earlier day), use that one. Otherwise, use today's post:
```
content/post/YYYY/MM/DD/index.md (today's date in Asia/Ho_Chi_Minh UTC+7)
```
.claude/skills/mt-add-url/SKILL.md
+1 -1
View File
@@ -29,8 +29,8 @@ Output (JSON): `{ original_url, clean_url, http_status, accessible, duplicate, r
### 2. Skip non-actionable URLs
- `accessible: false` → skip, note in report.
- `duplicate: true` → skip, note in report (already in a newsletter).
- `accessible: false` → **not an automatic skip.** The classifier does a plain fetch, so a bot-blocked host (403, Cloudflare challenge) reports `accessible: false` even when the page is public and the fallback fetchers can read it. Dispatch on `route` as normal and let the handler's fetch chain decide; only report the URL as skipped when every fetcher in `mt-webfetch` has failed. A `404`/dead URL is a genuine skip.
### 3. Dispatch on route
.claude/skills/mt-add-url/references/newsletter-post-mechanics.md
+11 -3
View File
@@ -9,18 +9,26 @@ All shared scripts live in `scripts/newsletter/`.
- Timezone: Asia/Ho_Chi_Minh (UTC+7)
- Content path: `content/post/YYYY/MM/DD/index.md`
## 1. Find / create today's post
## 1. Find / create the target post
Get current date in `YYYY-MM-DD` (UTC+7). Check `content/post/YYYY/MM/DD/index.md`:
The target defaults to today's date but the user can pin a different one.
**Target date resolution:**
- Default: current date in `YYYY-MM-DD` (UTC+7).
- **Pinned target:** when the user names a post or date to keep working on — typically an unpublished draft started on an earlier day ("keep adding to the 7/9 post") — that post is the target for the rest of the session, including across a date rollover mid-session. Do not silently start a new post for the new day; if a pin might have lapsed, ask before creating one.
Check `content/post/YYYY/MM/DD/index.md` for the resolved target date:
- **Exists** → update this file.
- **Missing** → create it (new newsletter number; template below). Create directories as needed.
The newsletter number always comes from the target post, not from today's date.
## 2. Newsletter number
```bash
go run ./scripts/newsletter find-newsletter-number
```
Searches backwards from today for the most recent newsletter and returns the next number. Only needed when **creating** a new post.
Searches backwards from today for the most recent newsletter and returns the next number. Only needed when **creating** a new post — when the target post already exists, read its number from its own `title`.
## 2a. Terminology
.claude/skills/mt-add-video/SKILL.md
+3 -3
View File
@@ -1,11 +1,11 @@
---
name: mt-add-video
description: 'YouTube video handler for the Hugo blog newsletter. Adds a YouTube link to today''s newsletter Bonus → Videos section as a Vietnamese title link plus a 1-2 sentence Vietnamese summary. Normally invoked by the mt-add-url meta skill after classification, but can be used directly for a known YouTube URL. Handles YouTube links only (not direct video files).'
description: 'YouTube video handler for the Hugo blog newsletter. Adds a YouTube link to the target newsletter post''s Bonus → Videos section as a Vietnamese title link plus a 1-2 sentence Vietnamese summary. Normally invoked by the mt-add-url meta skill after classification, but can be used directly for a known YouTube URL. Handles YouTube links only (not direct video files).'
---
## Overview
`mt-add-video` is the **YouTube handler**: given a YouTube link, it adds the video to today's newsletter **Bonus → Videos** as a Vietnamese title link with a 1-2 sentence Vietnamese summary. It does **not** classify or route — that is `mt-add-url`'s job.
`mt-add-video` is the **YouTube handler**: given a YouTube link, it adds the video to the target newsletter post's **Bonus → Videos** (today's post unless the user pinned another one) as a Vietnamese title link with a 1-2 sentence Vietnamese summary. It does **not** classify or route — that is `mt-add-url`'s job.
Scope: **YouTube links only** (`watch`, `youtu.be`, `shorts`). Direct video files (`.mp4` etc.) are not handled here — they go through `mt-add-url`'s fallback.
@@ -32,7 +32,7 @@ A clean YouTube URL (passed by `mt-add-url`, or given directly).
- Worst case, derive a single sentence from the title.
Keep it to 1-2 sentences (KISS).
4. **Post mechanics** — follow `../mt-add-url/references/newsletter-post-mechanics.md` to find/create today's post and locate the Bonus section.
4. **Post mechanics** — follow `../mt-add-url/references/newsletter-post-mechanics.md` to resolve/create the target post and locate the Bonus section.
5. **Insert under Bonus → Videos:**
```markdown
.claude/skills/mt-webfetch/SKILL.md
+26 -14
View File
@@ -1,6 +1,6 @@
---
name: mt-webfetch
description: "Fallback web content fetcher using defuddle.md as a proxy. Use ONLY when the built-in WebFetch tool has already failed with 403 Forbidden, bot detection, Cloudflare challenge, empty content, or similarly blocked response. Defuddle fetches the page server-side from a different IP and returns clean markdown with YAML frontmatter. Do NOT use as a first-choice fetcher — try WebFetch first. Does NOT bypass paywalls, login walls, or pages that require JavaScript execution."
description: "Fallback web content fetchers for pages that blocked the built-in fetch. Use ONLY when the built-in WebFetch tool has already failed with 403 Forbidden, bot detection, Cloudflare challenge, empty content, or similarly blocked response. Tries defuddle.md first, then a reader proxy — both fetch the page server-side from a different IP and return clean markdown. Do NOT use as a first-choice fetcher — try WebFetch first. Does NOT bypass paywalls, login walls, or pages that require JavaScript execution."
---
## Scope
@@ -9,9 +9,9 @@ This skill handles: fetching public web pages that blocked WebFetch due to bot-d
This skill does NOT handle:
- Paywalled or login-gated content
- Pages requiring client-side JavaScript execution (defuddle's hosted service does HTTP fetch, not headless rendering)
- Pages requiring client-side JavaScript execution (these proxies do HTTP fetch, not headless rendering)
- URLs that return 404 / are actually dead
- Sites that also block defuddle.md's outbound IP
- Sites that block every fetcher in the chain below
If WebFetch succeeded, do not use this skill.
@@ -26,7 +26,7 @@ Use this skill only after a WebFetch attempt returned one of:
## Workflow
1. Confirm WebFetch already failed on the target URL
2. Run the fetch script:
2. **Tier 1 — defuddle.** Run the fetch script:
```bash
go run ./scripts/newsletter fetch-via-defuddle "<target_url>"
```
@@ -34,8 +34,15 @@ Use this skill only after a WebFetch attempt returned one of:
```
WebFetch(url: "https://defuddle.md/<target_url>", prompt: "<extraction prompt>")
```
3. Parse the returned markdown (has YAML frontmatter with title/description/etc.)
4. If defuddle also returns empty or an error, stop and report failure to user — do not keep retrying.
3. **Tier 2 — reader proxy.** If tier 1 returns an error (commonly `502 / empty body`) or a body with no usable content, try a reader proxy through WebFetch:
```
WebFetch(url: "https://r.jina.ai/<target_url>", prompt: "<extraction prompt>")
```
Tier 1 and tier 2 fail independently — a site blocking one often still serves the other, so always attempt tier 2 before giving up.
4. Parse the returned markdown (tier 1 has YAML frontmatter with title/description/etc.)
5. If every tier fails, stop and report which tiers were tried and what each returned — do not keep retrying.
Attempt each tier at most once. The whole chain is: built-in WebFetch → defuddle → reader proxy → report failure.
## How defuddle works
@@ -49,12 +56,14 @@ The response is plain markdown. Use it directly when summarizing / extracting co
## Failure modes and exit
Give up after one retry. If defuddle returns:
- HTTP 4xx/5xx → report "both WebFetch and defuddle failed to fetch <url>" and move on
- Empty markdown body → same
- Only frontmatter with no body → report as inaccessible
Give up once every tier has been tried once. Treat these as a tier failure and move to the next tier:
- HTTP 4xx/5xx
- Empty markdown body
- Only frontmatter with no body
Never loop. Never retry more than once.
When the last tier fails, report it plainly — name each tier and its result (e.g. "WebFetch 403, defuddle 502, reader proxy empty") so the user can decide whether to paste the text or supply another source. If a fetcher fails repeatedly across sessions for the same host family, say so: that is a signal to reorder or extend the chain, not to keep retrying.
Never loop. Never retry a tier more than once.
## Security policy
@@ -70,7 +79,10 @@ Never loop. Never retry more than once.
User wanted to extract content from https://example.com/article
WebFetch returned: "Request failed with status code 403"
→ Trigger mt-webfetch
→ go run ./scripts/newsletter fetch-via-defuddle "https://example.com/article"
→ Parse markdown output
→ Summarize as usual
→ Tier 1: go run ./scripts/newsletter fetch-via-defuddle "https://example.com/article"
→ success: parse markdown output, summarize as usual
→ "502 / empty body": continue
→ Tier 2: WebFetch(url: "https://r.jina.ai/https://example.com/article", prompt: ...)
→ success: parse markdown output, summarize as usual
→ failure: report "WebFetch 403, defuddle 502, reader proxy failed" and stop
```
AGENTS.md
+3 -3
View File
@@ -47,7 +47,7 @@ go run ./scripts/newsletter find-newsletter-number # next newsletter numbe
go run ./scripts/newsletter list-existing-tags # existing tag frequencies
go run ./scripts/newsletter detect-image-source "<url>" # detect Substack image + uuid
go run ./scripts/newsletter find-substack-post --uuid <uuid>
go run ./scripts/newsletter fetch-via-defuddle "<url>" # fallback fetch (defuddle proxy)
go run ./scripts/newsletter fetch-via-defuddle "<url>" # fallback fetch (tier 1 of the fetch chain)
```
These are shared by all three tools — no tool-specific copies.
@@ -56,7 +56,7 @@ These are shared by all three tools — no tool-specific copies.
## Newsletter Workflow Routing
The newsletter workflow adds URLs (articles, YouTube videos, images) to today's newsletter post and manages tags. How it surfaces depends on the tool:
The newsletter workflow adds URLs (articles, YouTube videos, images) to the target newsletter post — today's post, or an earlier unpublished draft the user pins for the session — and manages tags. How it surfaces depends on the tool:
- **Claude Code / OpenCode** — invoke skills (both read `.claude/skills/<name>/SKILL.md` natively):
- `mt-add-url` — meta dispatcher: classifies each URL, auto-invokes the right handler. **Default entry for adding URLs.**
@@ -64,7 +64,7 @@ The newsletter workflow adds URLs (articles, YouTube videos, images) to today's
- `mt-add-video` — YouTube link → Bonus → Videos
- `mt-add-image` — image → Bonus → Images (labels Substack images via source-post lookup)
- `mt-add-tags` — add/update tags in post frontmatter
- `mt-webfetch` — fallback web fetcher (defuddle proxy); use only when built-in WebFetch is blocked
- `mt-webfetch` — fallback web fetch chain (defuddle, then a reader proxy); use only when built-in WebFetch is blocked
- **Codex** — discovers the repository-scoped adapters in `.agents/skills/`. Ask it to add a URL for implicit routing or invoke `$mt-add-url` explicitly.
`mt-add-url` dispatches `article` / `youtube` / `image`; other types (direct video files, documents, unknown) prompt the user to add or extend a handler.