mirror of
https://github.com/tiennm99/miti99.git
synced 2026-09-19 22:22:42 +00:00
refactor(skills): invoke the Node engine and centralise the command reference
Switches every call site to `node scripts/newsletter` and, in the same pass, fixes three structural problems in the skill layer. The seven subcommands were spelled out in five files; they are now enumerated once in docs/newsletter/engine-commands.md, and every other mention is a link except where a skill inlines the one or two commands it actually invokes. The shared post mechanics move out of mt-add-url's directory into docs/newsletter/, so no skill owns another skill's documentation and both runtimes reach it by a repository-relative path. The Codex adapters become symlinks to their canonical counterparts, which removes the parallel frontmatter that could drift. Two skills are renamed for a uniform mt-<verb>-<object> scheme: mt-add-post becomes mt-add-article, since it adds an article to a post, and mt-webfetch becomes mt-fetch-url. The other four already conformed. Note that $mt-add-post and mt-webfetch no longer resolve; no alias directories are left behind, because two names for one skill is the duplication this change removes. Trigger wording in the frontmatter descriptions is left alone apart from the renamed tokens and the corrected fetch-tier summary: those strings are how the runtimes decide whether to invoke a skill. Setup documentation now states the npm ci prerequisite. The engine has dependencies, so unlike its predecessor it does not run on a bare clone.
This commit is contained in:
+1
@@ -0,0 +1 @@
|
||||
../../../.claude/skills/mt-add-article/SKILL.md
|
||||
@@ -1,10 +0,0 @@
|
||||
---
|
||||
name: mt-add-image
|
||||
description: Add an image URL to the target Hugo newsletter post's Bonus Images section with a detected or user-provided label. Use directly for a known image URL or when dispatched by mt-add-url.
|
||||
---
|
||||
|
||||
# Add a newsletter image
|
||||
|
||||
Read and follow `../../../.claude/skills/mt-add-image/SKILL.md` completely.
|
||||
|
||||
Treat that file as the canonical workflow and resolve its relative paths from its own directory.
|
||||
+1
@@ -0,0 +1 @@
|
||||
../../../.claude/skills/mt-add-image/SKILL.md
|
||||
@@ -1,10 +0,0 @@
|
||||
---
|
||||
name: mt-add-post
|
||||
description: Add an article or blog URL to the target Hugo newsletter post with its original title and a Vietnamese summary. Use directly for a known article URL or when dispatched by mt-add-url.
|
||||
---
|
||||
|
||||
# Add a newsletter article
|
||||
|
||||
Read and follow `../../../.claude/skills/mt-add-post/SKILL.md` completely.
|
||||
|
||||
Treat that file as the canonical workflow and resolve its relative paths from its own directory.
|
||||
@@ -1,10 +0,0 @@
|
||||
---
|
||||
name: mt-add-tags
|
||||
description: Add or update Hugo post tags after analyzing the post. Use when the user asks to tag a post or before committing a post with empty or generic-only tags.
|
||||
---
|
||||
|
||||
# Add newsletter tags
|
||||
|
||||
Read and follow `../../../.claude/skills/mt-add-tags/SKILL.md` completely.
|
||||
|
||||
Treat that file as the canonical workflow and resolve its relative paths from its own directory.
|
||||
+1
@@ -0,0 +1 @@
|
||||
../../../.claude/skills/mt-add-tags/SKILL.md
|
||||
@@ -1,10 +0,0 @@
|
||||
---
|
||||
name: mt-add-url
|
||||
description: Meta entry for adding one or more URLs to the Hugo blog newsletter. Classifies each URL and dispatches articles, YouTube videos, and images to the matching handler. Use whenever the user asks to add URLs to a newsletter.
|
||||
---
|
||||
|
||||
# Add newsletter URLs
|
||||
|
||||
Read and follow `../../../.claude/skills/mt-add-url/SKILL.md` completely.
|
||||
|
||||
Treat that file as the canonical workflow. Resolve every relative path it references from its own directory, including `references/newsletter-post-mechanics.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
../../../.claude/skills/mt-add-url/SKILL.md
|
||||
@@ -1,10 +0,0 @@
|
||||
---
|
||||
name: mt-add-video
|
||||
description: Add a YouTube URL to the target Hugo newsletter post's Bonus Videos section with a Vietnamese title and summary. Use directly for a known YouTube URL or when dispatched by mt-add-url.
|
||||
---
|
||||
|
||||
# Add a newsletter video
|
||||
|
||||
Read and follow `../../../.claude/skills/mt-add-video/SKILL.md` completely.
|
||||
|
||||
Treat that file as the canonical workflow and resolve its relative paths from its own directory.
|
||||
+1
@@ -0,0 +1 @@
|
||||
../../../.claude/skills/mt-add-video/SKILL.md
|
||||
+1
@@ -0,0 +1 @@
|
||||
../../../.claude/skills/mt-fetch-url/SKILL.md
|
||||
@@ -1,10 +0,0 @@
|
||||
---
|
||||
name: mt-webfetch
|
||||
description: Fetch public web content through the fallback chain (defuddle, then a reader proxy) only after the built-in fetch is blocked or returns unusable content. Do not use for paywalls, login walls, or as the first fetch attempt.
|
||||
---
|
||||
|
||||
# Fetch blocked newsletter sources
|
||||
|
||||
Read and follow `../../../.claude/skills/mt-webfetch/SKILL.md` completely.
|
||||
|
||||
Treat that file as the canonical workflow and resolve its relative paths from its own directory.
|
||||
@@ -1,26 +1,26 @@
|
||||
---
|
||||
name: mt-add-post
|
||||
name: mt-add-article
|
||||
description: 'Article handler for the Hugo blog newsletter. Adds a single article/blog URL to the target newsletter post as a main-content entry with the original source title and a Vietnamese summary. Normally invoked by the mt-add-url meta skill after classification, but can be used directly for a known article URL.'
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
`mt-add-post` is the **article handler**: given a clean article/blog URL, it extracts the source title and content, writes a Vietnamese summary, and inserts it into the target newsletter post's main content (today's post unless the user pinned another one). It does **not** classify or route URLs — that is `mt-add-url`'s job. For YouTube/images/other types, use `mt-add-url`.
|
||||
`mt-add-article` is the **article handler**: given a clean article/blog URL, it extracts the source title and content, writes a Vietnamese summary, and inserts it into the target newsletter post's main content (today's post unless the user pinned another one). It does **not** classify or route URLs — that is `mt-add-url`'s job. For YouTube/images/other types, use `mt-add-url`.
|
||||
|
||||
Shared scripts: `scripts/newsletter/`. Shared procedure: `../mt-add-url/references/newsletter-post-mechanics.md` (resolve/create the target post, newsletter numbering, section insertion, language rules) — **follow it** for all post mechanics.
|
||||
Shared scripts: `scripts/newsletter/`. Shared procedure: `docs/newsletter/post-mechanics.md` (resolve/create the target post, newsletter numbering, section insertion, language rules) — **follow it** for all post mechanics.
|
||||
|
||||
## Input
|
||||
|
||||
A clean article URL (passed by `mt-add-url`, or given directly). If a raw URL is provided directly, you may run the classifier to clean/dedup it first:
|
||||
```bash
|
||||
go run ./scripts/newsletter add-url "<url>"
|
||||
node scripts/newsletter add-url "<url>"
|
||||
```
|
||||
Trust `route: article`; skip if `duplicate` or not `accessible`.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Post mechanics** — follow `../mt-add-url/references/newsletter-post-mechanics.md` to resolve/create the target post and (if new) get the newsletter number + apply the template.
|
||||
2. **Extract** the original article title and main content. Try `WebFetch` first; if the host blocks it (403, Cloudflare challenge, empty body), work through the `mt-webfetch` fallback chain before giving up. A `accessible: false` from the router is not by itself a reason to skip — many public tech blogs block plain fetches but serve the fallback fetchers fine. Preserve the source title exactly enough to remain recognizable: do not translate/localize it; keep source-language wording, capitalization, punctuation, and proper nouns from metadata (`og:title`, page title, or fetcher frontmatter).
|
||||
1. **Post mechanics** — follow `docs/newsletter/post-mechanics.md` to resolve/create the target post and (if new) get the newsletter number + apply the template.
|
||||
2. **Extract** the original article title and main content. Try `WebFetch` first; if the host blocks it (403, Cloudflare challenge, empty body), work through the `mt-fetch-url` fallback chain before giving up. A `accessible: false` from the router is not by itself a reason to skip — many public tech blogs block plain fetches but serve the fallback fetchers fine. Preserve the source title exactly enough to remain recognizable: do not translate/localize it; keep source-language wording, capitalization, punctuation, and proper nouns from metadata (`og:title`, page title, or fetcher frontmatter).
|
||||
3. **Summarize in Vietnamese** — 1-2 paragraphs, max 300 words, professional tone for junior developers. Summary paragraphs only; do not add a key-points bullet list.
|
||||
4. **Write** the main-content block and insert it **before** the `### Bonus` section (or append at end of file if there is no Bonus yet — do not create an empty Bonus):
|
||||
|
||||
@@ -7,7 +7,7 @@ description: 'Image handler for the Hugo blog newsletter. Adds an image URL to t
|
||||
|
||||
`mt-add-image` is the **image handler**: given an image URL, it resolves a human-readable **label** and inserts `` into the target newsletter post's **Bonus → Images** (today's post unless the user pinned another one). It does not classify/route — that is `mt-add-url`'s job.
|
||||
|
||||
Shared scripts: `scripts/newsletter/`. Shared procedure: `../mt-add-url/references/newsletter-post-mechanics.md` — **follow it** for post find/create, Bonus insertion, and language rules.
|
||||
Shared scripts: `scripts/newsletter/`. Shared procedure: `docs/newsletter/post-mechanics.md` — **follow it** for post find/create, Bonus insertion, and language rules.
|
||||
|
||||
**Label priority:** figure caption → source post title → your typed input.
|
||||
|
||||
@@ -21,13 +21,13 @@ A clean image URL (passed by `mt-add-url`, or given directly).
|
||||
|
||||
### 1. Detect source
|
||||
```bash
|
||||
go run ./scripts/newsletter detect-image-source "<url>"
|
||||
node scripts/newsletter detect-image-source "<url>"
|
||||
```
|
||||
→ `{ original_url, clean_url, isSubstack, uuid?, innerUrl? }`.
|
||||
|
||||
When invoked **directly** (not via `mt-add-url`), first run the router to get accessibility + duplicate status and skip accordingly:
|
||||
```bash
|
||||
go run ./scripts/newsletter add-url "<url>" # expect route:image; skip if duplicate/!accessible
|
||||
node scripts/newsletter add-url "<url>" # expect route:image; skip if duplicate/!accessible
|
||||
```
|
||||
(When dispatched by `mt-add-url`, that check already ran — don't repeat it.)
|
||||
|
||||
@@ -37,11 +37,11 @@ go run ./scripts/newsletter add-url "<url>" # expect route:image; skip if dupl
|
||||
|
||||
Otherwise, find the source post:
|
||||
```bash
|
||||
go run ./scripts/newsletter find-substack-post --uuid <uuid>
|
||||
node scripts/newsletter find-substack-post --uuid <uuid>
|
||||
```
|
||||
- `found: false` → retry with the deeper sitemap crawl. The quick RSS pass only covers the newest posts, so a miss is the normal result for anything older than the current feed window — go straight to `--deep` rather than treating the miss as a dead end. It is slower (fetches posts ~3 months back, capped at 40 fetches total across all publications), so warn the user it may take a while:
|
||||
```bash
|
||||
go run ./scripts/newsletter find-substack-post --uuid <uuid> --deep
|
||||
node scripts/newsletter find-substack-post --uuid <uuid> --deep
|
||||
```
|
||||
On a miss the result reports `scanned` (posts fetched), `budget` (the 40-fetch cap), and `cutoff` (oldest date looked at) — mention how far back it looked.
|
||||
- `found: false` after `--deep` → no source post; go to step 3 (ask) and/or step 4 (add publication).
|
||||
@@ -71,7 +71,7 @@ If no label was detected, use `AskUserQuestion`:
|
||||
If a Substack image wasn't found and the user tells you which publication it's from, offer to append that host to `scripts/newsletter/config/substack-publications.json`, then retry step 2a. This grows coverage for next time.
|
||||
|
||||
### 5. Insert into Bonus → Images
|
||||
Follow `../mt-add-url/references/newsletter-post-mechanics.md` to resolve/create the target post, then add under **Images**:
|
||||
Follow `docs/newsletter/post-mechanics.md` to resolve/create the target post, then add under **Images**:
|
||||
```markdown
|
||||
**Images:**
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@ Read the full post body. Identify:
|
||||
|
||||
Before generating, run:
|
||||
```bash
|
||||
go run ./scripts/newsletter list-existing-tags
|
||||
node scripts/newsletter list-existing-tags
|
||||
```
|
||||
When a proposed tag matches an existing one case-insensitively, use the existing casing.
|
||||
-->
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
---
|
||||
name: mt-add-url
|
||||
description: 'Meta entry for adding URLs to the Hugo blog newsletter. Use whenever the user provides one or more URLs to add to their newsletter (articles, YouTube videos, images, etc.). Classifies each URL and auto-dispatches to the right handler skill (mt-add-post for articles, mt-add-video for YouTube, mt-add-image for images). For unsupported types it asks the user how to proceed. This is the default entry point for newsletter URL processing.'
|
||||
description: 'Meta entry for adding URLs to the Hugo blog newsletter. Use whenever the user provides one or more URLs to add to their newsletter (articles, YouTube videos, images, etc.). Classifies each URL and auto-dispatches to the right handler skill (mt-add-article for articles, mt-add-video for YouTube, mt-add-image for images). For unsupported types it asks the user how to proceed. This is the default entry point for newsletter URL processing.'
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
`mt-add-url` is the **meta dispatcher**: it classifies each URL and auto-invokes the matching handler skill. Handlers (`mt-add-post`, `mt-add-video`, `mt-add-image`) own the actual content writing. Shared scripts live in `scripts/newsletter/`; shared post mechanics in `references/newsletter-post-mechanics.md`.
|
||||
`mt-add-url` is the **meta dispatcher**: it classifies each URL and auto-invokes the matching handler skill. Handlers (`mt-add-article`, `mt-add-video`, `mt-add-image`) own the actual content writing. The shared engine lives in `scripts/newsletter/` — see [docs/newsletter/engine-commands.md](../../../docs/newsletter/engine-commands.md) for every command it offers, and `docs/newsletter/post-mechanics.md` for shared post mechanics.
|
||||
|
||||
**Supported routes (this version):**
|
||||
- `article` → `mt-add-post`
|
||||
- `article` → `mt-add-article`
|
||||
- `youtube` → `mt-add-video`
|
||||
- `image` → `mt-add-image`
|
||||
|
||||
@@ -20,7 +20,7 @@ Everything else (direct `video` file, `document`, or anything unrecognized) is *
|
||||
|
||||
For every URL the user provides:
|
||||
```bash
|
||||
go run ./scripts/newsletter add-url "<url>"
|
||||
node scripts/newsletter add-url "<url>"
|
||||
```
|
||||
Output (JSON): `{ original_url, clean_url, http_status, accessible, duplicate, route, title?, author? }`.
|
||||
|
||||
@@ -30,13 +30,13 @@ Output (JSON): `{ original_url, clean_url, http_status, accessible, duplicate, r
|
||||
### 2. Skip non-actionable URLs
|
||||
|
||||
- `duplicate: true` → skip, note in report (already in a newsletter).
|
||||
- `accessible: false` → **not an automatic skip.** The classifier does a plain fetch, so a bot-blocked host (403, Cloudflare challenge) reports `accessible: false` even when the page is public and the fallback fetchers can read it. Dispatch on `route` as normal and let the handler's fetch chain decide; only report the URL as skipped when every fetcher in `mt-webfetch` has failed. A `404`/dead URL is a genuine skip.
|
||||
- `accessible: false` → **not an automatic skip.** The classifier does a plain fetch, so a bot-blocked host (403, Cloudflare challenge) reports `accessible: false` even when the page is public and the fallback fetchers can read it. Dispatch on `route` as normal and let the handler's fetch chain decide; only report the URL as skipped when every fetcher in `mt-fetch-url` has failed. A `404`/dead URL is a genuine skip.
|
||||
|
||||
### 3. Dispatch on route
|
||||
|
||||
| route | Action |
|
||||
|-------|--------|
|
||||
| `article` | Invoke the **`mt-add-post`** skill, passing `clean_url` |
|
||||
| `article` | Invoke the **`mt-add-article`** skill, passing `clean_url` |
|
||||
| `youtube` | Invoke the **`mt-add-video`** skill, passing `clean_url` |
|
||||
| `image` | Invoke the **`mt-add-image`** skill, passing `clean_url` |
|
||||
| `video` (direct file) / `document` / anything else | **Fallback** — see step 4 |
|
||||
@@ -62,7 +62,7 @@ Act on the user's choice. If they choose add/update, proceed to design that skil
|
||||
Close with the target post's TL;DR tally, then the per-URL detail. Read the tally from the post itself so it reflects everything the post now holds, not just this batch:
|
||||
|
||||
```bash
|
||||
go run ./scripts/newsletter post-stats content/post/YYYY/MM/DD/index.md
|
||||
node scripts/newsletter post-stats content/post/YYYY/MM/DD/index.md
|
||||
```
|
||||
|
||||
Aggregate across all URLs:
|
||||
@@ -73,7 +73,7 @@ Aggregate across all URLs:
|
||||
(omit zero counts; documents too when present)
|
||||
|
||||
✅ Dispatched: [count]
|
||||
- [count] → mt-add-post (articles)
|
||||
- [count] → mt-add-article (articles)
|
||||
- [count] → mt-add-video (YouTube)
|
||||
- [count] → mt-add-image (images)
|
||||
|
||||
@@ -84,9 +84,9 @@ Aggregate across all URLs:
|
||||
- [url] (route: [route]): [user decision]
|
||||
```
|
||||
|
||||
The tally is report-only — never write it into `index.md`. See *Post tally* in `references/newsletter-post-mechanics.md`.
|
||||
The tally is report-only — never write it into `index.md`. See *Post tally* in `docs/newsletter/post-mechanics.md`.
|
||||
|
||||
## Notes
|
||||
|
||||
- Handlers (`mt-add-post`, `mt-add-video`, `mt-add-image`) remain directly invocable for single-purpose use, but `mt-add-url` is the normal entry point when a user pastes a URL.
|
||||
- Shared mechanics (numbering, post find/create, Bonus insertion, language rules) are defined once in `references/newsletter-post-mechanics.md`; handlers reference it.
|
||||
- Handlers (`mt-add-article`, `mt-add-video`, `mt-add-image`) remain directly invocable for single-purpose use, but `mt-add-url` is the normal entry point when a user pastes a URL.
|
||||
- Shared mechanics (numbering, post find/create, Bonus insertion, language rules) are defined once in `docs/newsletter/post-mechanics.md`; handlers reference it.
|
||||
|
||||
@@ -9,7 +9,7 @@ description: 'YouTube video handler for the Hugo blog newsletter. Adds a YouTube
|
||||
|
||||
Scope: **YouTube links only** (`watch`, `youtu.be`, `shorts`). Direct video files (`.mp4` etc.) are not handled here — they go through `mt-add-url`'s fallback.
|
||||
|
||||
Shared scripts: `scripts/newsletter/`. Shared procedure: `../mt-add-url/references/newsletter-post-mechanics.md` — **follow it** for post find/create, numbering, Bonus insertion, and language rules.
|
||||
Shared scripts: `scripts/newsletter/`. Shared procedure: `docs/newsletter/post-mechanics.md` — **follow it** for post find/create, numbering, Bonus insertion, and language rules.
|
||||
|
||||
## Input
|
||||
|
||||
@@ -19,7 +19,7 @@ A clean YouTube URL (passed by `mt-add-url`, or given directly).
|
||||
|
||||
1. **Classify / fetch title** — run the router to get the canonical URL + title:
|
||||
```bash
|
||||
go run ./scripts/newsletter add-url "<url>"
|
||||
node scripts/newsletter add-url "<url>"
|
||||
```
|
||||
Confirm `route: youtube`; skip if `duplicate` or not `accessible`. Use the returned `clean_url` (canonical `watch?v=ID`) and `title`.
|
||||
- If `title` is missing (oEmbed failed), fetch the title via WebFetch on the watch URL.
|
||||
@@ -32,7 +32,7 @@ A clean YouTube URL (passed by `mt-add-url`, or given directly).
|
||||
- Worst case, derive a single sentence from the title.
|
||||
Keep it to 1-2 sentences (KISS).
|
||||
|
||||
4. **Post mechanics** — follow `../mt-add-url/references/newsletter-post-mechanics.md` to resolve/create the target post and locate the Bonus section.
|
||||
4. **Post mechanics** — follow `docs/newsletter/post-mechanics.md` to resolve/create the target post and locate the Bonus section.
|
||||
|
||||
5. **Insert under Bonus → Videos:**
|
||||
```markdown
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: mt-webfetch
|
||||
description: "Fallback web content fetchers for pages that blocked the built-in fetch. Use ONLY when the built-in WebFetch tool has already failed with 403 Forbidden, bot detection, Cloudflare challenge, empty content, or similarly blocked response. Tries defuddle.md first, then a reader proxy — both fetch the page server-side from a different IP and return clean markdown. Do NOT use as a first-choice fetcher — try WebFetch first. Does NOT bypass paywalls, login walls, or pages that require JavaScript execution."
|
||||
name: mt-fetch-url
|
||||
description: "Fallback web content fetchers for pages that blocked the built-in fetch. Use ONLY when the built-in WebFetch tool has already failed with 403 Forbidden, bot detection, Cloudflare challenge, empty content, or similarly blocked response. Tries local defuddle extraction, then the defuddle.md proxy, then a reader proxy — the proxies fetch the page server-side from a different IP and return clean markdown. Do NOT use as a first-choice fetcher — try WebFetch first. Does NOT bypass paywalls, login walls, or pages that require JavaScript execution."
|
||||
---
|
||||
|
||||
## Scope
|
||||
@@ -26,29 +26,40 @@ Use this skill only after a WebFetch attempt returned one of:
|
||||
## Workflow
|
||||
|
||||
1. Confirm WebFetch already failed on the target URL
|
||||
2. **Tier 1 — defuddle.** Run the fetch script:
|
||||
2. **Tier 1 — defuddle.** Run the fetch command:
|
||||
```bash
|
||||
go run ./scripts/newsletter fetch-via-defuddle "<target_url>"
|
||||
node scripts/newsletter fetch-via-defuddle "<target_url>"
|
||||
```
|
||||
Alternatively, use WebFetch with the defuddle-prefixed URL:
|
||||
```
|
||||
WebFetch(url: "https://defuddle.md/<target_url>", prompt: "<extraction prompt>")
|
||||
```
|
||||
3. **Tier 2 — reader proxy.** If tier 1 returns an error (commonly `502 / empty body`) or a body with no usable content, try a reader proxy through WebFetch:
|
||||
It is itself two-stage: it extracts locally first, and falls back to the
|
||||
`https://defuddle.md/<target_url>` proxy when local extraction fails or
|
||||
returns nothing. Stderr names which stage failed. A single invocation covers
|
||||
both — do not run it twice.
|
||||
3. **Tier 2 — reader proxy.** If tier 1 exits nonzero or returns a body with no usable content, try a reader proxy through WebFetch:
|
||||
```
|
||||
WebFetch(url: "https://r.jina.ai/<target_url>", prompt: "<extraction prompt>")
|
||||
```
|
||||
Tier 1 and tier 2 fail independently — a site blocking one often still serves the other, so always attempt tier 2 before giving up.
|
||||
4. Parse the returned markdown (tier 1 has YAML frontmatter with title/description/etc.)
|
||||
4. Parse the returned markdown — both tier-1 stages emit YAML frontmatter with title/description/etc. ahead of the body
|
||||
5. If every tier fails, stop and report which tiers were tried and what each returned — do not keep retrying.
|
||||
|
||||
Attempt each tier at most once. The whole chain is: built-in WebFetch → defuddle → reader proxy → report failure.
|
||||
Attempt each tier at most once. The whole chain is: built-in WebFetch → local defuddle → defuddle.md → reader proxy → report failure.
|
||||
|
||||
## How defuddle works
|
||||
|
||||
- URL pattern: `https://defuddle.md/<target_url>` (target URL appended as path, works with or without scheme)
|
||||
- Returns: Markdown body with YAML frontmatter containing metadata (title, author, description, site name)
|
||||
- Server-side HTTP fetch from defuddle's IP + extraction via Defuddle library (clean main-content extraction)
|
||||
- **Local stage** — the engine fetches the page itself and runs the Defuddle
|
||||
library in-process, so the chain does not depend on a third-party service
|
||||
being up. Returns YAML frontmatter (title, author, description, site,
|
||||
published, source) followed by the markdown body. An extraction that looks
|
||||
like a bot challenge rather than the page counts as a failure here, so the
|
||||
proxy stage still runs.
|
||||
- **Proxy stage** — `https://defuddle.md/<target_url>` (target URL appended as
|
||||
path, works with or without scheme). Returns the same shape: YAML frontmatter
|
||||
(title, author, description, site name) plus a markdown body. This stage fetches from
|
||||
defuddle's IP, which is what matters when *this* machine's IP is the blocked
|
||||
one — that is why it is kept behind the local stage rather than replaced by it.
|
||||
|
||||
Exit codes: `0` content returned, `1` both stages failed, `2` bad arguments. See
|
||||
[docs/newsletter/engine-commands.md](../../../docs/newsletter/engine-commands.md).
|
||||
|
||||
## Output handling
|
||||
|
||||
@@ -61,7 +72,7 @@ Give up once every tier has been tried once. Treat these as a tier failure and m
|
||||
- Empty markdown body
|
||||
- Only frontmatter with no body
|
||||
|
||||
When the last tier fails, report it plainly — name each tier and its result (e.g. "WebFetch 403, defuddle 502, reader proxy empty") so the user can decide whether to paste the text or supply another source. If a fetcher fails repeatedly across sessions for the same host family, say so: that is a signal to reorder or extend the chain, not to keep retrying.
|
||||
When the last tier fails, report it plainly — name each tier and its result (e.g. "WebFetch 403, defuddle exit 1, reader proxy empty") so the user can decide whether to paste the text or supply another source. If a fetcher fails repeatedly across sessions for the same host family, say so: that is a signal to reorder or extend the chain, not to keep retrying.
|
||||
|
||||
Never loop. Never retry a tier more than once.
|
||||
|
||||
@@ -78,10 +89,10 @@ Never loop. Never retry a tier more than once.
|
||||
```
|
||||
User wanted to extract content from https://example.com/article
|
||||
WebFetch returned: "Request failed with status code 403"
|
||||
→ Trigger mt-webfetch
|
||||
→ Tier 1: go run ./scripts/newsletter fetch-via-defuddle "https://example.com/article"
|
||||
→ success: parse markdown output, summarize as usual
|
||||
→ "502 / empty body": continue
|
||||
→ Trigger mt-fetch-url
|
||||
→ Tier 1: node scripts/newsletter fetch-via-defuddle "https://example.com/article"
|
||||
→ success (exit 0): parse markdown output, summarize as usual
|
||||
→ exit 1 after both stages: continue
|
||||
→ Tier 2: WebFetch(url: "https://r.jina.ai/https://example.com/article", prompt: ...)
|
||||
→ success: parse markdown output, summarize as usual
|
||||
→ failure: report "WebFetch 403, defuddle 502, reader proxy failed" and stop
|
||||
@@ -39,19 +39,20 @@ The site will be available at `http://localhost:1313`
|
||||
|
||||
## Shared Engine
|
||||
|
||||
The portable newsletter engine lives in **`scripts/newsletter/`** (Go, stdlib only, no deps — one binary, one subcommand per task) and is invoked from the repo root with `go run`:
|
||||
The portable newsletter engine lives in **`scripts/newsletter/`** (Node ESM, one
|
||||
module per subcommand) and is invoked from the repo root:
|
||||
|
||||
```bash
|
||||
go run ./scripts/newsletter add-url "<url>" # classify + dedup a URL → JSON route
|
||||
go run ./scripts/newsletter find-newsletter-number # next newsletter number
|
||||
go run ./scripts/newsletter list-existing-tags # existing tag frequencies
|
||||
go run ./scripts/newsletter detect-image-source "<url>" # detect Substack image + uuid
|
||||
go run ./scripts/newsletter find-substack-post --uuid <uuid>
|
||||
go run ./scripts/newsletter fetch-via-defuddle "<url>" # fallback fetch (tier 1 of the fetch chain)
|
||||
go run ./scripts/newsletter post-stats <post-path> # count a post's articles/images/videos/documents
|
||||
node scripts/newsletter <command> [args]
|
||||
```
|
||||
|
||||
These are shared by all three tools — no tool-specific copies.
|
||||
The seven subcommands, their arguments, output shapes and exit codes are
|
||||
documented once in **[docs/newsletter/engine-commands.md](docs/newsletter/engine-commands.md)**.
|
||||
|
||||
The engine has npm dependencies, so a fresh clone needs `npm ci` from the repo
|
||||
root once before any skill will run.
|
||||
|
||||
The engine is shared by all three tools — no tool-specific copies.
|
||||
|
||||
---
|
||||
|
||||
@@ -61,27 +62,27 @@ The newsletter workflow adds URLs (articles, YouTube videos, images) to the targ
|
||||
|
||||
- **Claude Code / OpenCode** — invoke skills (both read `.claude/skills/<name>/SKILL.md` natively):
|
||||
- `mt-add-url` — meta dispatcher: classifies each URL, auto-invokes the right handler. **Default entry for adding URLs.**
|
||||
- `mt-add-post` — article/blog URL → newsletter main content
|
||||
- `mt-add-article` — article/blog URL → newsletter main content
|
||||
- `mt-add-video` — YouTube link → Bonus → Videos
|
||||
- `mt-add-image` — image → Bonus → Images (labels Substack images via source-post lookup)
|
||||
- `mt-add-tags` — add/update tags in post frontmatter
|
||||
- `mt-webfetch` — fallback web fetch chain (defuddle, then a reader proxy); use only when built-in WebFetch is blocked
|
||||
- `mt-fetch-url` — fallback web fetch chain (local defuddle, the defuddle.md proxy, then a reader proxy); use only when built-in WebFetch is blocked
|
||||
- **Codex** — discovers the repository-scoped adapters in `.agents/skills/`. Ask it to add a URL for implicit routing or invoke `$mt-add-url` explicitly.
|
||||
|
||||
`mt-add-url` dispatches `article` / `youtube` / `image`; other types (direct video files, documents, unknown) prompt the user to add or extend a handler.
|
||||
|
||||
Canonical skill implementations live in `.claude/skills/`; Codex adapters in `.agents/skills/` reference them so behavior stays in one place.
|
||||
Canonical skill implementations live in `.claude/skills/`; `.agents/skills/` symlinks to them so there is only ever one copy to edit.
|
||||
|
||||
### Using Codex
|
||||
|
||||
Codex automatically discovers checked-in skills from `.agents/skills/`. No install or copy step is required. Start Codex at the repository root, then either describe the task normally or explicitly mention a skill:
|
||||
|
||||
- `$mt-add-url <url>` — classify and dispatch one or more URLs
|
||||
- `$mt-add-post <url>` — add an article directly
|
||||
- `$mt-add-article <url>` — add an article directly
|
||||
- `$mt-add-video <url>` — add a YouTube video directly
|
||||
- `$mt-add-image <url>` — add an image directly
|
||||
- `$mt-add-tags [post]` — add or update tags
|
||||
- `$mt-webfetch <url>` — fallback after the built-in fetch fails
|
||||
- `$mt-fetch-url <url>` — fallback after the built-in fetch fails
|
||||
|
||||
Codex detects skill changes automatically; restart Codex if an update does not appear.
|
||||
|
||||
|
||||
@@ -39,7 +39,7 @@ The Stack theme is pulled in as a git submodule under `themes/hugo-theme-stack/`
|
||||
|
||||
## Working with AI tools
|
||||
|
||||
This repo runs from Claude Code, OpenCode, or Codex off one shared engine (`scripts/newsletter/`, Go via `go run`) and one instruction file (`AGENTS.md`). Repository-scoped Codex skills use the official `.agents/skills/` format; no installer is required. Setup, invocation per tool, and how to pick one and remove the rest: see [docs/multi-tool-usage.md](docs/multi-tool-usage.md).
|
||||
This repo runs from Claude Code, OpenCode, or Codex off one shared engine (`scripts/newsletter/`, Node — run `npm ci` once, then `node scripts/newsletter <command>`; see [docs/newsletter/engine-commands.md](docs/newsletter/engine-commands.md)) and one instruction file (`AGENTS.md`). Repository-scoped Codex skills use the official `.agents/skills/` format; no installer is required. Setup, invocation per tool, and how to pick one and remove the rest: see [docs/multi-tool-usage.md](docs/multi-tool-usage.md).
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -1,29 +1,29 @@
|
||||
# Multi-tool usage (Claude Code · OpenCode · Codex)
|
||||
|
||||
This repo is usable from three AI coding tools off **one shared engine**. The newsletter engine lives in `scripts/newsletter/` (Go, stdlib only) and every tool calls it as `go run ./scripts/newsletter <command>` from the repo root. Project instructions live once in `AGENTS.md`.
|
||||
This repo is usable from three AI coding tools off **one shared engine**. The newsletter engine lives in `scripts/newsletter/` (Node ESM) and every tool calls it as `node scripts/newsletter <command>` from the repo root — the commands are documented once in [newsletter/engine-commands.md](newsletter/engine-commands.md). Project instructions live once in `AGENTS.md`.
|
||||
|
||||
## Per-tool setup & invocation
|
||||
|
||||
| Tool | Instruction file | How the workflow surfaces | Setup | Invocation example |
|
||||
|------|------------------|---------------------------|-------|--------------------|
|
||||
| **Claude Code** | `CLAUDE.md` → imports `AGENTS.md` | Native skill auto-dispatch from `.claude/skills/` | None (works as-is) | Paste a URL; `mt-add-url` auto-runs |
|
||||
| **OpenCode** | `AGENTS.md` (auto-read) | `.claude/skills/` auto-discovered via the `skill` tool, governed by `opencode.json` | None beyond `opencode.json` (committed) | Ask to add a URL; pick/`skill` `mt-add-url` |
|
||||
| **Codex** | `AGENTS.md` (auto-read) | Repository skills discovered from `.agents/skills/` | None (works as-is) | Ask to add a URL or invoke `$mt-add-url` |
|
||||
| **Claude Code** | `CLAUDE.md` → imports `AGENTS.md` | Native skill auto-dispatch from `.claude/skills/` | `npm ci` once | Paste a URL; `mt-add-url` auto-runs |
|
||||
| **OpenCode** | `AGENTS.md` (auto-read) | `.claude/skills/` auto-discovered via the `skill` tool, governed by `opencode.json` | `npm ci` once; `opencode.json` is committed | Ask to add a URL; pick/`skill` `mt-add-url` |
|
||||
| **Codex** | `AGENTS.md` (auto-read) | Repository skills discovered from `.agents/skills/` | `npm ci` once | Ask to add a URL or invoke `$mt-add-url` |
|
||||
|
||||
**Shared engine:** all three call `go run ./scripts/newsletter <command>` from repo root — no per-tool script copies.
|
||||
**Shared engine:** all three call `node scripts/newsletter <command>` from repo root — no per-tool script copies. It has npm dependencies, so run `npm ci` once after cloning.
|
||||
|
||||
### Notes per tool
|
||||
|
||||
- **Claude Code** — `CLAUDE.md` is a thin file that imports `AGENTS.md` (`@AGENTS.md`); all instruction content lives in `AGENTS.md`. Permissions in `.claude/settings.json`.
|
||||
- **OpenCode** — reads `.claude/skills/<name>/SKILL.md` natively (multi-location search up the worktree), so the same `mt-*` skills work with zero duplication. `opencode.json` mirrors the Claude allow-list intent (node/git/hugo/edit). No MCP block — relies on OpenCode's built-in web fetch/search. Restart OpenCode after editing `opencode.json` (config is not hot-reloaded).
|
||||
- **Codex** — reads repository skills from `.agents/skills/` and can invoke them explicitly (`$mt-add-url`) or implicitly from their descriptions. The checked-in Codex skills delegate to the canonical workflow definitions in `.claude/skills/`, so workflow behavior stays shared without an install or sync step.
|
||||
- **Codex** — reads repository skills from `.agents/skills/` and can invoke them explicitly (`$mt-add-url`) or implicitly from their descriptions. Each `.agents/skills/<name>/SKILL.md` is a **symlink** to its `.claude/skills/` counterpart, so there is one file to edit and nothing that can drift. If a checkout cannot follow symlinks (Windows without developer mode), replace them with thin pointer files that say "read and follow `.claude/skills/<name>/SKILL.md`".
|
||||
|
||||
## Teardown — pick one, remove the rest
|
||||
|
||||
Once you commit to one tool, delete the others' config:
|
||||
|
||||
- **Keep Claude Code only** → remove `opencode.json` and `.agents/`. Keep `.claude/`.
|
||||
- **Keep Claude Code only** → remove `opencode.json` and `.agents/`. Keep `.claude/` and `docs/newsletter/`.
|
||||
- **Keep OpenCode only** → remove `.claude/settings*.json` and `.agents/`. **Keep `.claude/skills/`** — OpenCode reads it. If you want to drop the `.claude/` dir entirely, first move skills to `.opencode/skills/` (scripts are path-neutral in `scripts/newsletter/`, so the move breaks nothing). Optionally make `AGENTS.md` the only instruction file (delete `CLAUDE.md`).
|
||||
- **Keep Codex only** → keep `.agents/skills/`, `AGENTS.md`, `scripts/newsletter/`, and the canonical `.claude/skills/` files referenced by the Codex skills. You may remove Claude-specific settings and `opencode.json`.
|
||||
- **Keep Codex only** → keep `.agents/skills/`, `AGENTS.md`, `scripts/newsletter/`, `docs/newsletter/`, and the canonical `.claude/skills/` files the Codex skills symlink to. You may remove Claude-specific settings and `opencode.json`.
|
||||
|
||||
In every case: **keep `scripts/newsletter/`, `.claude/skills/`, and `AGENTS.md`** — they are the shared engine, canonical workflows, and instructions.
|
||||
In every case: **keep `scripts/newsletter/`, `.claude/skills/`, `docs/newsletter/`, and `AGENTS.md`** — they are the shared engine, canonical workflows, shared documentation, and instructions.
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
# Newsletter engine commands
|
||||
|
||||
The portable newsletter engine lives in `scripts/newsletter/` — Node ESM, one
|
||||
module per subcommand. **This file is the only place the commands are
|
||||
enumerated**; skills and docs link here rather than restating the list.
|
||||
|
||||
## Prerequisite
|
||||
|
||||
The engine has three runtime dependencies (`cheerio`, `defuddle`, `linkedom`), so
|
||||
a fresh clone needs one setup step:
|
||||
|
||||
```bash
|
||||
npm ci
|
||||
```
|
||||
|
||||
Skip it and the content-parsing subcommands print
|
||||
`newsletter engine: dependencies missing — run 'npm ci' from the repo root`
|
||||
and exit 1 rather than a module-resolution stack trace. `fetch-via-defuddle` is
|
||||
the exception: its local tier needs the dependencies, so without them it falls
|
||||
through to the `defuddle.md` proxy and still succeeds.
|
||||
|
||||
## Invocation
|
||||
|
||||
Always from the repo root — repo-relative paths (`content/post`) resolve from the
|
||||
process working directory:
|
||||
|
||||
```bash
|
||||
node scripts/newsletter <command> [args]
|
||||
```
|
||||
|
||||
JSON output is 2-space indented with a trailing newline, and is not HTML-escaped
|
||||
(URLs keep their `&` readable). Optional fields are **omitted** when empty rather
|
||||
than emitted as `""`.
|
||||
|
||||
## Commands
|
||||
|
||||
### `add-url <url>`
|
||||
|
||||
Classifies and deduplicates one URL — the single entry point per URL.
|
||||
|
||||
```json
|
||||
{
|
||||
"original_url": "…",
|
||||
"clean_url": "…",
|
||||
"http_status": "200",
|
||||
"accessible": true,
|
||||
"duplicate": false,
|
||||
"route": "article",
|
||||
"title": "…",
|
||||
"author": "…"
|
||||
}
|
||||
```
|
||||
|
||||
- `route` ∈ `youtube | image | video | document | article`
|
||||
- `clean_url` has tracking parameters stripped (`utm_*`, `fbclid`, `gclid`,
|
||||
`ref`, …); surviving parameters keep their original order and encoding
|
||||
- For `youtube`, `clean_url` is the canonical `https://www.youtube.com/watch?v=ID`
|
||||
and `title` / `author` come from oEmbed. Both keys are absent for every other
|
||||
route, and absent when oEmbed fails.
|
||||
- `duplicate` is boundary-aware: a URL that is merely a *prefix* of one already
|
||||
stored is not a duplicate. Substack images dedupe on their image uuid, so
|
||||
transform and size variants collapse onto one identity.
|
||||
|
||||
### `find-newsletter-number`
|
||||
|
||||
Prints the next newsletter number (plain text). Scans `content/post/YYYY/MM/DD/`
|
||||
newest-year-first and stops at the first year that holds a `Newsletter #N`
|
||||
heading, so a stale higher number in an older year does not win. An empty tree
|
||||
yields `1`.
|
||||
|
||||
### `list-existing-tags`
|
||||
|
||||
Prints tag frequencies, most-used first, top 40, as `%6d %s` — the count
|
||||
right-aligned in six columns. Ties keep first-seen (lexical walk) order.
|
||||
|
||||
### `detect-image-source <url>`
|
||||
|
||||
Detects a Substack-hosted image and extracts its S3 image uuid.
|
||||
|
||||
```json
|
||||
{
|
||||
"original_url": "…",
|
||||
"clean_url": "…",
|
||||
"isSubstack": true,
|
||||
"uuid": "…",
|
||||
"innerUrl": "…"
|
||||
}
|
||||
```
|
||||
|
||||
`uuid` and `innerUrl` are present only when the URL is a Substack image and the
|
||||
value is non-empty. A malformed percent sequence in the CDN wrapper falls back to
|
||||
the raw substring rather than failing.
|
||||
|
||||
### `find-substack-post --uuid <uuid> [--deep]`
|
||||
|
||||
Finds which Substack post embeds an image uuid, and extracts a label. Only the
|
||||
double-dash flag form is accepted.
|
||||
|
||||
Searches each publication in `scripts/newsletter/config/substack-publications.json`
|
||||
(editable, read at runtime) — the RSS feed first, then, with `--deep`, a sitemap
|
||||
crawl back ~3 months sharing a 40-fetch budget across all publications.
|
||||
|
||||
On a hit:
|
||||
|
||||
```json
|
||||
{
|
||||
"found": true,
|
||||
"source": "rss",
|
||||
"publication": "…",
|
||||
"postTitle": "…",
|
||||
"postUrl": "…",
|
||||
"caption": "…",
|
||||
"candidates": ["…"]
|
||||
}
|
||||
```
|
||||
|
||||
On a miss: `{"found": false}`, or with `--deep`
|
||||
`{"found": false, "source": "sitemap", "scanned": 0, "budget": 40, "cutoff": null}`
|
||||
where `cutoff` is the crawl boundary date, or `null` when no sitemap could be
|
||||
fetched at all.
|
||||
|
||||
### `fetch-via-defuddle <url>`
|
||||
|
||||
Fallback fetcher for pages that blocked the built-in fetch. Two tiers, in order:
|
||||
local defuddle extraction, then the `defuddle.md` proxy (which fetches from its
|
||||
own IP — the tier that matters when this machine's IP is the blocked one).
|
||||
|
||||
Both tiers write YAML frontmatter followed by the markdown body, so the caller
|
||||
parses one shape either way; stderr names which tier failed and why. An
|
||||
extraction that looks like a bot challenge rather than the requested page counts
|
||||
as a local-tier failure, so the proxy still gets its turn.
|
||||
|
||||
| Exit | Meaning |
|
||||
|---|---|
|
||||
| 0 | content returned |
|
||||
| 1 | every tier failed |
|
||||
| 2 | bad arguments |
|
||||
|
||||
### `post-stats <path/to/index.md>`
|
||||
|
||||
Counts what a post already holds, so a handler can report a running tally.
|
||||
|
||||
```json
|
||||
{
|
||||
"post": "content/post/2026/09/10/index.md",
|
||||
"newsletter": 132,
|
||||
"articles": 8,
|
||||
"images": 3,
|
||||
"videos": 1,
|
||||
"documents": 0,
|
||||
"total": 12
|
||||
}
|
||||
```
|
||||
|
||||
`newsletter` is `0` when the post carries no `Newsletter #N` heading.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
npm test # deterministic tier, offline, ~2s
|
||||
NEWSLETTER_NET=1 npm test # adds the live-upstream tier
|
||||
```
|
||||
|
||||
Golden outputs under `scripts/newsletter/__fixtures__/golden/` were captured from
|
||||
the previous Go implementation; see
|
||||
`plans/reports/parity-260918-newsletter-js-migration-report.md`.
|
||||
+4
-4
@@ -1,7 +1,7 @@
|
||||
# Newsletter Post Mechanics (shared)
|
||||
|
||||
Shared procedure used by the newsletter handler skills (`mt-add-post`, `mt-add-video`, `mt-add-image`).
|
||||
All shared scripts live in `scripts/newsletter/`.
|
||||
Shared procedure used by the newsletter handler skills (`mt-add-article`, `mt-add-video`, `mt-add-image`).
|
||||
The shared engine lives in `scripts/newsletter/` — see [engine-commands.md](engine-commands.md) for every command and the `npm ci` prerequisite.
|
||||
|
||||
**Project Context:**
|
||||
- Hugo static site, theme `hugo-theme-stack`
|
||||
@@ -26,7 +26,7 @@ The newsletter number always comes from the target post, not from today's date.
|
||||
## 2. Newsletter number
|
||||
|
||||
```bash
|
||||
go run ./scripts/newsletter find-newsletter-number
|
||||
node scripts/newsletter find-newsletter-number
|
||||
```
|
||||
Searches backwards from today for the most recent newsletter and returns the next number. Only needed when **creating** a new post — when the target post already exists, read its number from its own `title`.
|
||||
|
||||
@@ -88,7 +88,7 @@ When a subsection (e.g. `**Videos:**`) already exists, append under it; otherwis
|
||||
After every successful insertion, report the target post's running totals so the user can see what the post now holds:
|
||||
|
||||
```bash
|
||||
go run ./scripts/newsletter post-stats content/post/YYYY/MM/DD/index.md
|
||||
node scripts/newsletter post-stats content/post/YYYY/MM/DD/index.md
|
||||
```
|
||||
Output (JSON): `{ post, newsletter, articles, images, videos, documents, total }`.
|
||||
|
||||
Reference in New Issue
Block a user