refactor(semantle,doantu): drop ConceptNet vestiges, trim wordlist API

Now that both modules run on Workers AI embeddings, drop the legacy
Word2SimError alias, the unused wordlist helpers (getLine, LINE_COUNT,
pickFromPool), and every comment/README section still describing the
removed ConceptNet backend. Fix the bge-small doc typo in semantle/index.js
and align the semantle api-client test fake-vector dim with the real
384-dim output.

scripts/build-semantle-words.js

@@ -31,7 +31,7 @@ if (!res.ok) throw new Error(`fetch failed: ${res.status} ${res.statusText}`);
 const text = await res.text();
 
 // Normalize only: trim whitespace, lowercase, drop blanks, dedupe.
-// Preserve original frequency order so `getLine(n)` stays a frequency rank.
+// Preserve original frequency order — source is ranked by Google Ngram.
 const words = Array.from(
   new Set(
     text

src/modules/doantu/README.md

@@ -2,8 +2,8 @@
 
 Vietnamese "đoán từ" (guess-the-word) — same core mechanic as `semantle`,
 but targets come from a Vietnamese wordlist and similarity is computed
-against ConceptNet's `/c/vi/<term>` concept URIs. Unlimited guesses per
-round; solve on exact match.
+with a multilingual embedding model. Unlimited guesses per round; solve
+on exact match (case-insensitive, diacritic-sensitive).
 
 **Visibility: `protected`** — commands appear in `/help` but are hidden
 from Telegram's native `/` autocomplete menu while the module is still
@@ -18,40 +18,43 @@ experimental.
 | `/doantu_stats` | protected | Show per-subject stats |
 
 Submit with `/doantu <word>` (e.g. `/doantu con chó`). Multi-syllable words
-with single spaces between them are accepted. Matching is case-insensitive
-and diacritic-sensitive — `cá` and `ca` are different targets.
+with single spaces between them are accepted. `cá` and `ca` are different
+targets.
 
 ## Data source
 
-**Target pool:** [duyet/vietnamese-wordlist](https://github.com/duyet/vietnamese-wordlist)'s
-Viet22K list (~22k entries), sorted alphabetically. The raw list is
-normalized (lowercase + deduped) but otherwise used verbatim — ConceptNet's
-verify-and-fallback at round start rejects any pick that has no concept
-edges. License: GPL-2.0 (Ho Ngoc Duc).
+**Target + vocabulary:** [duyet/vietnamese-wordlist](https://github.com/duyet/vietnamese-wordlist)'s
+Viet22K list (~22k entries), lowercased and deduped. The same list is
+both the target pool and the vocabulary — OOV detection is `Set.has()`
+with no upstream call. License: GPL-2.0 (Ho Ngoc Duc).
+Regenerate with `node scripts/build-doantu-words.js`.
 
-**Similarity + vocabulary:** [ConceptNet 5](https://conceptnet.io).
-Multi-word Vietnamese terms are converted to underscore form (`con chó`
-→ `/c/vi/con_chó`) only when building URIs — the board keeps the
-space-separated display.
+**Similarity:** `@cf/baai/bge-m3` multilingual text embeddings via the
+`env.AI` binding. Chosen over the English-only `bge-small-en-v1.5`
+because that model's tokenizer shreds Vietnamese diacritics into noisy
+byte-level subwords. Each in-vocab guess runs one inference call
+batching target + guess (1024-dim vectors); the module scores them with
+local cosine similarity.
 
 ## Architecture
 
-- `api-client.js` — ConceptNet wrapper (`randomWord`, `similarity`, and
-  lower-level `concept` / `relatedness`). Hardcoded `LANG = "vi"`.
-- `wordlist.js` — three-function API (`LINE_COUNT`, `randomLine`, `getLine`)
-  over `words-data.js`.
-- `words-data.js` — auto-generated (regenerate via `npm run build:doantu-words`).
+- `api-client.js` — Workers AI wrapper: `randomWord()` picks from the
+  local pool, `similarity(a, b)` calls `env.AI.run()` and returns
+  `{ in_vocab_b, similarity }`. `UpstreamError` on inference failure.
+- `words-data.js` — auto-generated Viet22K dictionary.
+- `wordlist.js` — one-function module exposing `randomLine()`.
 - `state.js` — KV persistence for game + stats. Same shape as semantle.
 - `lookup.js` — guess normalization + shape validation. Accepts Unicode
   letters + combining marks + single internal spaces.
-- `format.js` — warmth-percent and emoji-bucket formatters (unchanged).
+- `format.js` — warmth-percent and emoji-bucket formatters (identical
+  to semantle/format.js — score display is language-agnostic).
 - `render.js` — Telegram HTML `<pre>` monospace board with a 🇻🇳 header.
 - `handlers.js` — subject resolution + the three command entry points.
 
 Near-clone of the semantle sibling — kept separate per the repo's
-existing one-module-per-game convention rather than factoring out a
-shared base. Diff your changes against `../semantle/` when fixing bugs
-that apply to both.
+one-module-per-game convention rather than factoring out a shared base.
+Diff your changes against `../semantle/` when fixing bugs that apply to
+both.
 
 ## Storage
 
@@ -64,11 +67,11 @@ KV namespace prefix: `doantu:`
 
 ## Config
 
-No env vars. ConceptNet base (`https://api.conceptnet.io`) is hardcoded;
-pass an override to `createClient(url)` if you need a mirror or test double.
+No env vars. Model defaults to `@cf/baai/bge-m3`; override with
+`createClient(env.AI, { model: "..." })` in a test or alternative deploy.
 
 ## Credits
 
+- Embeddings: [`@cf/baai/bge-m3`](https://developers.cloudflare.com/workers-ai/models/bge-m3/) on Cloudflare Workers AI (multilingual).
 - Wordlist: [duyet/vietnamese-wordlist](https://github.com/duyet/vietnamese-wordlist) by Ho Ngoc Duc (GPL-2.0).
-- Similarity + vocabulary: [ConceptNet 5](https://conceptnet.io) by Robyn Speer et al.
 - Game concept: [Semantle](https://semantle.com/) by David Turner.

src/modules/doantu/lookup.js

@@ -4,8 +4,7 @@
  * Allows Unicode letters (including diacritics via combining marks) plus
  * single spaces between syllables for compound words (`con chó`,
  * `máy bay`). Rejects digits, punctuation, and underscores so the board
- * stays clean; the api-client handles the space→underscore conversion
- * internally when building ConceptNet URIs.
+ * stays clean.
  */
 
 /** @param {string} raw */

src/modules/doantu/wordlist.js

@@ -4,23 +4,11 @@
  * Data lives in the sibling `words-data.js` (auto-generated by
  * `scripts/build-doantu-words.js` from duyet/vietnamese-wordlist's
  * Viet22K list, lowercased + deduped).
- *
- * API mirrors wordle/loldle and semantle's english sibling:
- *   `LINE_COUNT`  — total entries
- *   `randomLine()` — uniform random pick
- *   `getLine(n)`  — read the nth entry (alphabetical rank in the source list)
  */
 
 import LINES from "./words-data.js";
 
-export const LINE_COUNT = LINES.length;
-
-/** @param {number} n @returns {string | undefined} */
-export function getLine(n) {
-  return LINES[n];
-}
-
 /** @returns {string} */
 export function randomLine() {
-  return LINES[Math.floor(Math.random() * LINE_COUNT)];
+  return LINES[Math.floor(Math.random() * LINES.length)];
 }

src/modules/semantle/README.md

@@ -1,9 +1,9 @@
 # Semantle Module
 
 Semantic-similarity guessing game. A secret word is picked from a local
-curated pool and validated against ConceptNet; each guess is scored by
-ConceptNet's relatedness API against the target. Unlimited guesses per
-round — you play until you get the exact word (case-insensitive).
+curated pool and each guess is scored by cosine similarity between
+embedding vectors produced by Cloudflare Workers AI. Unlimited guesses
+per round — you play until you get the exact word (case-insensitive).
 
 ## Commands
 
@@ -20,47 +20,34 @@ ignored (no cost, no stat inflation).
 
 ## Data source
 
-**[ConceptNet 5](https://api.conceptnet.io/)** — free public API, no auth,
-~300k English concepts including multi-word phrases. Two endpoints:
-
-- `GET /relatedness?node1=/c/en/X&node2=/c/en/Y` — per-guess similarity,
-  returns `{ value: number ∈ [-1, 1] }`.
-- `GET /c/en/{term}` — vocabulary check: term is in vocab iff the response
-  carries at least one edge.
-
-Because ConceptNet has no random-word endpoint, the target pool ships in
-`words-data.js` — the full
-[google-10000-english-no-swears](https://github.com/first20hours/google-10000-english)
+**Target + vocabulary:** `words-data.js` ships the full
+[google-10000-english](https://github.com/first20hours/google-10000-english)
 list (~9.9k entries), ordered by Google Ngram frequency, normalized to
-lowercase and deduped but otherwise unfiltered.
+lowercase and deduped but otherwise unfiltered. The **same list is both
+the target pool and the vocabulary** — so every legal guess could itself
+have been the answer, and OOV detection is an O(1) `Set.has()` with no
+upstream round-trip. Regenerate with `node scripts/build-semantle-words.js`.
 
-`wordlist.js` exposes three accessors over the imported array:
-- `LINE_COUNT` — total entries
-- `randomLine()` — uniform random pick
-- `getLine(n)` — read the nth entry (n is the frequency rank)
+**Similarity:** `@cf/baai/bge-small-en-v1.5` text embeddings via the
+`env.AI` binding. Each in-vocab guess runs one inference call batching
+target + guess (384-dim vectors) and the module scores them with local
+cosine similarity. At ~0.0037 Neurons per guess, the Workers Free plan
+cap of 10k Neurons/day covers ~2.7M guesses/day.
 
-Each new round picks via `randomLine()`, verifies the candidate via
-ConceptNet's concept endpoint, and falls back to an unverified pick after
-a few misses (see `api-client.js`).
-
-Regenerate with `npm run build:semantle-words` (chained into the main
-`npm run build` that `npm run deploy` invokes).
-
-Every guess costs **two** ConceptNet calls (concept edges + relatedness)
-issued in parallel. Typical latency ~300–600ms round-trip from Cloudflare
-Workers; `api-client.js` enforces a 5s timeout and surfaces a "Upstream
-hiccup" message on failure.
+OOV guesses short-circuit before inference — the player sees
+"isn't in the vocabulary" instead of a noisy subword-based score.
 
 ## Architecture
 
-- `api-client.js` — ConceptNet HTTP wrapper (`randomWord`, `similarity`,
-  plus lower-level `concept` / `relatedness`) with `UpstreamError` metadata.
-  Preserves the earlier word2sim response shape so the rest of the module
-  didn't need rewriting.
-- `words-data.js` — auto-generated dictionary (regenerate via `npm run build:semantle-words`).
-- `wordlist.js` — thin wrapper exposing `TARGET_POOL` and `pickFromPool()` over the dictionary.
+- `api-client.js` — Workers AI wrapper: `randomWord()` picks from the
+  local pool, `similarity(a, b)` runs `env.AI.run()` and returns
+  `{ in_vocab_b, similarity }` along with canonical forms.
+  `UpstreamError` carries status/body metadata when inference fails.
+- `words-data.js` — auto-generated dictionary (~9.9k entries).
+- `wordlist.js` — one-function module exposing `randomLine()`.
 - `state.js` — KV persistence for game + stats. Target stored lowercased.
-- `lookup.js` — guess normalization and shape validation.
+- `lookup.js` — guess normalization (`trim + lowercase + collapse spaces`)
+  and shape validation (`/^[a-z]+$/`, max 64 chars).
 - `format.js` — warmth-percent and emoji-bucket formatters.
 - `render.js` — Telegram HTML `<pre>` monospace board, sorted by similarity
   desc, capped at top 15 rows to stay under Telegram's message-length limit.
@@ -79,24 +66,22 @@ KV namespace prefix: `semantle:`
 | `game:<subject>` | `{ target, startedAt, solved, guesses[] }` — active round (TTL 7 days). `target` stored lowercased. |
 | `stats:<subject>` | `{ played, solved, totalGuesses, bestGuessCount, lastResultAt }` |
 
-Each `guesses[]` entry is `{ word, canonical, similarity }`. The canonical
-form is lowercased on write so the solve check is a single string compare.
+Each `guesses[]` entry is `{ word, canonical, similarity }`.
 
 ## Config
 
-No env vars. ConceptNet's public API base (`https://api.conceptnet.io`) is
-hardcoded in `api-client.js`; pass an override to `createClient(url)` if you
-need to point at a mirror or test double.
+No env vars. Model defaults to `@cf/baai/bge-small-en-v1.5`; override with
+`createClient(env.AI, { model: "@cf/baai/bge-base-en-v1.5" })` in a test
+or alternative deploy.
 
 ## Why unlimited guesses?
 
 Classic Semantle offers up to 100s of guesses per day, and the fun is in
-the hunt — not the timer. We keep rounds open indefinitely (TTL 7 days on
-KV) and measure skill via `bestGuessCount`, the fewest guesses to solve
-across all rounds.
+the hunt — not the timer. Rounds stay open (TTL 7 days on KV) and skill is
+tracked via `bestGuessCount` — fewest guesses to solve across all rounds.
 
 ## Credits
 
-- Similarity + vocabulary: [ConceptNet 5](https://conceptnet.io) by Robyn Speer et al.
+- Embeddings: [`@cf/baai/bge-small-en-v1.5`](https://developers.cloudflare.com/workers-ai/models/bge-small-en-v1.5/) on Cloudflare Workers AI.
 - Target dictionary: [google-10000-english](https://github.com/first20hours/google-10000-english) by Josh Kaufman, derived from Peter Norvig's Google Ngram analysis.
 - Game concept: [Semantle](https://semantle.com/) by David Turner.

src/modules/semantle/api-client.js

@@ -8,12 +8,9 @@
  * Vocabulary: the curated `words-data.js` list (google-10k) doubles as our
  * in/out-of-vocabulary set — anything outside it is treated as OOV so players
  * get the "not in the vocabulary" reply instead of a noisy embedding score.
- *
- * The returned `similarity(a, b)` shape is kept identical to the prior
- * ConceptNet/word2sim contract so handlers/render/state stay untouched.
  */
 
-import { pickFromPool } from "./wordlist.js";
+import { randomLine } from "./wordlist.js";
 import WORDS from "./words-data.js";
 
 const DEFAULT_MODEL = "@cf/baai/bge-small-en-v1.5";
@@ -74,11 +71,10 @@ export function createClient(ai, { model = DEFAULT_MODEL } = {}) {
     /**
      * Pick a target word from the local pool. The pool IS our vocabulary,
      * so every pick is trivially verified — no upstream check needed.
-     * Shape matches the old word2sim `/random` response for handler reuse.
      * @returns {Promise<{ word: string, verified: boolean }>}
      */
     async randomWord() {
-      return { word: pickFromPool(), verified: true };
+      return { word: randomLine(), verified: true };
     },
 
     /**
@@ -106,6 +102,3 @@ export function createClient(ai, { model = DEFAULT_MODEL } = {}) {
     },
   };
 }
-
-// Backwards-compat alias — older imports referenced `Word2SimError`.
-export { UpstreamError as Word2SimError };

src/modules/semantle/index.js

@@ -4,7 +4,7 @@
  * Targets come from a curated local wordlist (same list doubles as the
  * vocabulary for OOV detection, so no upstream check is needed to pick or
  * validate a word). Similarity scores come from cosine distance between
- * `@cf/baai/bge-base-en-v1.5` embeddings produced by the `env.AI` binding.
+ * `@cf/baai/bge-small-en-v1.5` embeddings produced by the `env.AI` binding.
  */
 
 import { createClient } from "./api-client.js";

src/modules/semantle/lookup.js

@@ -1,9 +1,9 @@
 /**
  * @file Guess normalization + shape validation.
  *
- * Keeps obviously-bad input from hitting the API. The /random endpoint
- * already filters its output to ASCII letters only, so any guess outside
- * that shape can never equal the target — fail fast.
+ * Keeps obviously-bad input out of the VOCAB lookup and the embedding call.
+ * The wordlist is ASCII-letter-only at build time, so any guess outside
+ * that shape is guaranteed OOV — fail fast.
  */
 
 /** @param {string} raw */

src/modules/semantle/state.js

@@ -4,7 +4,7 @@
  *
  * Target is stored lowercased so the case-insensitive equality check
  * is a single compare. Unlimited guesses — no MAX cap; rounds end only
- * on solve, giveup, or `/semantle_new`.
+ * on solve or giveup.
  *
  * Key layout (inside the module-prefixed store):
  *   game:<subject>   -> { target, startedAt, solved, guesses[] }

src/modules/semantle/wordlist.js

@@ -4,26 +4,11 @@
  * Data lives in the sibling `words-data.js` (auto-generated by
  * `scripts/build-semantle-words.js` from google-10000-english, normalized
  * to lowercase + deduped but otherwise unfiltered).
- *
- * API mirrors the "count, random index, read line" access pattern:
- *   `LINE_COUNT`  — how many entries are in the dictionary
- *   `randomLine()` — pick a uniformly-random entry
- *   `getLine(n)`  — read the nth entry (n is the frequency rank)
  */
 
 import LINES from "./words-data.js";
 
-export const LINE_COUNT = LINES.length;
-
-/** @param {number} n @returns {string | undefined} */
-export function getLine(n) {
-  return LINES[n];
-}
-
 /** @returns {string} */
 export function randomLine() {
-  return LINES[Math.floor(Math.random() * LINE_COUNT)];
+  return LINES[Math.floor(Math.random() * LINES.length)];
 }
-
-// Backwards-compat alias for earlier callers.
-export { randomLine as pickFromPool };

tests/modules/semantle/api-client.test.js

@@ -1,15 +1,11 @@
 import { describe, expect, it, vi } from "vitest";
-import {
-  UpstreamError,
-  Word2SimError,
-  createClient,
-} from "../../../src/modules/semantle/api-client.js";
+import { UpstreamError, createClient } from "../../../src/modules/semantle/api-client.js";
 
 /**
- * Build a deterministic 768-dim vector from a seed so cosine scores are
- * reproducible in tests without hardcoding 768 floats.
+ * Build a deterministic 384-dim vector (bge-small output size) from a seed
+ * so cosine scores are reproducible without hardcoding 384 floats.
  */
-function fakeVector(seed, dim = 768) {
+function fakeVector(seed, dim = 384) {
   const out = new Array(dim);
   for (let i = 0; i < dim; i++) out[i] = Math.sin(seed * (i + 1));
   return out;
@@ -38,10 +34,6 @@ describe("semantle/api-client", () => {
       const err = new UpstreamError("wrapper", { cause });
       expect(err.cause).toBe(cause);
     });
-
-    it("is re-exported as Word2SimError alias for legacy callers", () => {
-      expect(Word2SimError).toBe(UpstreamError);
-    });
   });
 
   describe("createClient", () => {
@@ -53,7 +45,7 @@ describe("semantle/api-client", () => {
 
     it("similarity batches target + guess in a single run() call", async () => {
       const ai = fakeAi(async (_model, { text }) => ({
-        shape: [text.length, 768],
+        shape: [text.length, 384],
         data: text.map((_, i) => fakeVector(i + 1)),
       }));
       const client = createClient(ai);
@@ -115,7 +107,7 @@ describe("semantle/api-client", () => {
     });
 
     it("similarity returns null score when a vector norm is zero", async () => {
-      const zero = new Array(768).fill(0);
+      const zero = new Array(384).fill(0);
       const ai = fakeAi(async () => ({ data: [zero, fakeVector(1)] }));
       const client = createClient(ai);
       const res = await client.similarity("apple", "orange");

tests/modules/semantle/handlers.test.js

@@ -1,6 +1,6 @@
 import { beforeEach, describe, expect, it, vi } from "vitest";
 import { createStore } from "../../../src/db/create-store.js";
-import { Word2SimError } from "../../../src/modules/semantle/api-client.js";
+import { UpstreamError } from "../../../src/modules/semantle/api-client.js";
 import {
   handleGiveup,
   handleSemantle,
@@ -302,7 +302,7 @@ describe("semantle/handlers", () => {
     });
 
     it("replies with UPSTREAM_FAIL on randomWord error", async () => {
-      client.randomWord.mockRejectedValue(new Word2SimError("timeout", { status: 504 }));
+      client.randomWord.mockRejectedValue(new UpstreamError("timeout", { status: 504 }));
 
       const ctx = makeCtx(1, "private", "/semantle");
       await handleSemantle(ctx, { db, client });
@@ -312,7 +312,7 @@ describe("semantle/handlers", () => {
 
     it("replies with UPSTREAM_FAIL on similarity error", async () => {
       client.randomWord.mockResolvedValue({ word: "apple", rank: 1000 });
-      client.similarity.mockRejectedValue(new Word2SimError("network error"));
+      client.similarity.mockRejectedValue(new UpstreamError("network error"));
 
       const ctx = makeCtx(1, "private", "/semantle guess");
       await handleSemantle(ctx, { db, client });