docs(planning): add planning artifacts for scribe example implementation

Planning docs for the Go sender example feature, including: - Phase overviews and implementation steps - Research findings on Scribe protocol and Go integration - Task breakdown and success criteria
2026-07-17 04:17:23 +00:00 · 2026-05-27 17:49:05 +07:00
parent bd16c305eb
commit 9a47995d7c
3 changed files with 287 additions and 0 deletions
@@ -0,0 +1,123 @@
+---
+phase: 1
+title: Go raw-socket sender
+status: completed
+priority: P2
+effort: 2h
+dependencies: []
+---
+
+# Phase 1: Go raw-socket sender
+
+## Overview
+
+A tiny Go program that connects to the Scribe daemon over TCP and sends `LogEntry`
+messages by hand-encoding the Thrift framed binary `Log()` RPC — stdlib only, no Apache
+Thrift dependency, no generated bindings. Compiles to a small static binary in a
+multi-stage Docker build.
+
+## Requirements
+
+- Functional: connect to `SCRIBE_HOST:SCRIBE_PORT`, send `SCRIBE_COUNT` messages under
+  category `SCRIBE_CATEGORY`, parse the `ResultCode` reply, log OK/TRY_LATER, exit 0 on success.
+- Functional: retry the initial connect (~10s, ~1s interval) so a cold `docker compose up`
+  does not fail before Scribe is listening.
+- Non-functional: no external Go modules (stdlib `net`, `encoding/binary`, `os`, `time`, `log`).
+- Non-functional: static binary (`CGO_ENABLED=0`); minimal runtime image.
+
+## Architecture
+
+### Scribe Thrift contract (from facebookarchive/scribe IDL)
+
+```thrift
+enum ResultCode { OK = 0, TRY_LATER = 1 }
+struct LogEntry { 1: string category; 2: string message; }
+service scribe { ResultCode Log(1: list<LogEntry> messages); }
+```
+
+### Wire format to produce
+
+- **Transport:** `TFramedTransport` → 4-byte big-endian frame length prefixes the payload.
+- **Protocol:** `TBinaryProtocol`, **non-strict** message header (matches README client which
+  uses `strictWrite=False`; Scribe C++ server reads with `strict_read=false`, accepts it).
+
+Thrift type IDs used: `STOP=0`, `I32=8`, `STRING=11`, `STRUCT=12`, `LIST=15`. All ints big-endian.
+
+**Request payload (`Log` CALL):**
+1. Message begin (non-strict): `string "Log"` (i32 len + bytes) · `i8 msgType=1 (CALL)` · `i32 seqid=0`
+2. Args struct `Log_args`:
+   - field begin: `i8 type=15 (LIST)` · `i16 id=1`
+   - list begin: `i8 elemType=12 (STRUCT)` · `i32 size=N`
+   - per `LogEntry`:
+     - field: `i8 11 (STRING)` · `i16 1` · writeString(category)
+     - field: `i8 11 (STRING)` · `i16 2` · writeString(message)
+     - `i8 0 (STOP)`
+   - args struct: `i8 0 (STOP)`
+3. Prepend `i32` total length → frame.
+
+`writeString(s)` = `i32 len(s)` + UTF-8 bytes.
+
+**Response parse (minimal):**
+1. Read 4-byte frame length, then that many bytes.
+2. Message begin (non-strict): i32 name-len + name + `i8 msgType` + `i32 seqid`.
+3. Field begin `i8 type` · `i16 id`; for return value id=0 type=I32 → read `i32` ResultCode.
+4. Map 0→OK, 1→TRY_LATER; anything else → log raw + treat non-zero as failure.
+
+### Connect-retry
+
+Loop `net.DialTimeout("tcp", addr, 2s)` up to 10 attempts, 1s between failures; fatal after.
+
+## Related Code Files
+
+- Create: `scribe/example/sender/main.go` — sender program (encode/send/parse + retry).
+- Create: `scribe/example/sender/go.mod` — `module scribe-example-sender` / `go 1.22`, stdlib only.
+- Create: `scribe/example/sender/Dockerfile` — multi-stage build.
+
+### Dockerfile (multi-stage)
+
+```dockerfile
+FROM golang:1.22-alpine AS build
+WORKDIR /src
+COPY go.mod .
+COPY main.go .
+RUN CGO_ENABLED=0 go build -o /sender .
+
+FROM alpine:3.20
+COPY --from=build /sender /sender
+ENTRYPOINT ["/sender"]
+```
+
+### main.go env inputs (with defaults)
+
+| Env | Default | Purpose |
+|-----|---------|---------|
+| `SCRIBE_HOST` | `scribe` | daemon hostname (compose service name) |
+| `SCRIBE_PORT` | `1463` | Thrift port |
+| `SCRIBE_CATEGORY` | `example` | message category → log subdir |
+| `SCRIBE_COUNT` | `5` | number of LogEntry messages to send |
+
+## Implementation Steps
+
+1. Create `scribe/example/sender/go.mod` (module name + `go 1.22`).
+2. Write `main.go`:
+   - read env with defaults; build `bytes.Buffer` helpers for `i8/i16/i32/string`.
+   - `buildLogFrame(entries)` → payload then 4-byte length prefix.
+   - connect-retry loop → write frame → read+parse response frame → log ResultCode.
+   - `os.Exit(0)` on OK; non-zero exit on connect failure or non-OK.
+3. Write the multi-stage `Dockerfile`.
+4. Local sanity build: `docker build -t scribe-sender scribe/example/sender` (verify compiles).
+
+## Success Criteria
+
+- [ ] `docker build scribe/example/sender` succeeds (binary compiles, no module fetch).
+- [ ] Run against a live scribe container: sender logs `sent N messages, result=OK` and exits 0.
+- [ ] Sender survives a cold start (scribe not yet listening) via the retry loop.
+
+## Risk Assessment
+
+- **Hand-encoded Thrift correctness** — wire format above is the stable 0.9.1 binary layout;
+  keep helpers tiny and comment each field. Mitigation: validate end-to-end in Phase 2 by
+  confirming the written log file contains the exact messages.
+- **Strict vs non-strict header** — if the server unexpectedly rejects non-strict, switch
+  message-begin to strict (`i32 version=0x80010001`, name, `i32 seqid`). Documented here as
+  the fallback; default is non-strict to match the README client.
@@ -0,0 +1,111 @@
+---
+phase: 2
+title: Compose orchestration and docs
+status: completed
+priority: P2
+effort: 1h
+dependencies:
+  - 1
+---
+
+# Phase 2: Compose orchestration and docs
+
+## Overview
+
+Wire the Scribe daemon and the Go sender (Phase 1) into a single `docker compose up`
+demo, bind-mount logs to a host-visible folder, and document run + verify steps. End
+result: one command produces real log files under `scribe/example/logs/`.
+
+## Requirements
+
+- Functional: `docker compose up` (from `scribe/example/`) starts Scribe with the repo's
+  `config/scribe.conf`, runs the sender once, and writes files to host `./logs`.
+- Functional: logs are inspectable on the host without entering the container.
+- Non-functional: scribe service uses published `ghcr.io/tiennm99/scribe:2.2` (no local build).
+- Non-functional: example README is self-contained (run, verify, troubleshoot).
+
+## Architecture
+
+### docker-compose.yml
+
+```yaml
+services:
+  scribe:
+    image: ghcr.io/tiennm99/scribe:2.2
+    ports:
+      - "1463:1463"
+    volumes:
+      - ../config/scribe.conf:/etc/scribe/scribe.conf:ro
+      - ./logs:/var/log/scribe
+    restart: unless-stopped
+
+  sender:
+    build: ./sender
+    depends_on:
+      - scribe
+    environment:
+      - SCRIBE_HOST=scribe
+      - SCRIBE_PORT=1463
+      - SCRIBE_CATEGORY=example
+      - SCRIBE_COUNT=5
+```
+
+Notes:
+- Sender reaches Scribe over the compose network by service name `scribe:1463` (internal),
+  independent of the host `1463:1463` mapping.
+- `depends_on` only orders start, not readiness — the Phase 1 retry loop covers the gap.
+- Sender is one-shot: it sends then exits; scribe keeps running (`restart: unless-stopped`).
+
+### Expected on-disk result
+
+```
+scribe/example/logs/
+└── example/
+    └── example-<YYYY-MM-DD>_00000        # contains the 5 sent messages
+```
+
+(Category `example` → subdir under the default catch-all store's `file_path=/var/log/scribe`.)
+
+## Related Code Files
+
+- Create: `scribe/example/docker-compose.yml`
+- Create: `scribe/example/logs/.gitkeep` — keep the bind-mount dir in git, empty.
+- Create: `scribe/example/README.md` — run/verify/troubleshoot for the example.
+- Modify: `scribe/README.md` — add a short "Runnable example" pointer linking to `example/`.
+
+### scribe/example/README.md outline
+
+1. **What it does** — one command boots Scribe + sends test messages → log files appear.
+2. **Run** — `cd scribe/example && docker compose up --build`.
+3. **Verify** — show `scribe/example/logs/example/example-*_00000` and `cat` a line.
+4. **Customize** — env vars (`SCRIBE_CATEGORY`, `SCRIBE_COUNT`), pointer to `config/scribe.conf`.
+5. **Troubleshoot** — empty logs dir = no messages sent yet; Linux bind-mount permissions caveat.
+6. **Stop** — `docker compose down`.
+
+## Implementation Steps
+
+1. Create `scribe/example/docker-compose.yml` per spec above.
+2. Create `scribe/example/logs/.gitkeep`.
+3. Write `scribe/example/README.md` per outline.
+4. Add a "Runnable example" subsection to `scribe/README.md` linking to `example/`.
+5. End-to-end test: `cd scribe/example && docker compose up --build`; confirm a log file
+   appears under `logs/example/` containing the sent messages; `docker compose down`.
+
+## Success Criteria
+
+- [ ] `docker compose up --build` starts both services without manual steps.
+- [ ] After the sender exits, `scribe/example/logs/example/example-*_00000` exists on host.
+- [ ] That file contains the test messages sent by the Go sender.
+- [ ] `scribe/example/README.md` documents run, verify, and the Linux permission caveat.
+- [ ] `scribe/README.md` links to the example.
+
+## Risk Assessment
+
+- **Bind-mount permissions (native Linux)** — Scribe runs as non-root user `scribe`
+  (`scribe/Dockerfile:107-110`); host `./logs` may reject its writes. On Docker Desktop
+  (Windows/macOS) this is normally fine. Mitigation: document `chmod 777 ./logs` (or matching
+  uid) in the README troubleshoot section.
+- **Stale relative paths** — `../config/scribe.conf` and `./sender` build context are relative
+  to `scribe/example/`; compose must be run from that dir. Documented in README run step.
+- **Empty logs confusion** — if the sender fails silently, logs dir stays empty. Mitigation:
+  sender logs success/failure to stdout (`docker compose` output) and exits non-zero on failure.
@@ -0,0 +1,53 @@
+---
+title: Scribe runnable example with Go test sender
+description: >-
+  Self-contained scribe/example/ that boots the Scribe daemon and a Go sender so
+  docker compose up writes visible log files to host ./logs
+status: completed
+priority: P2
+branch: main
+tags:
+  - scribe
+  - example
+  - docker-compose
+  - golang
+  - thrift
+blockedBy: []
+blocks: []
+created: '2026-05-27T10:27:22.391Z'
+createdBy: 'ck:plan'
+source: skill
+---
+
+# Scribe runnable example with Go test sender
+
+## Overview
+
+Add `scribe/example/` — a one-command demo proving the Scribe config actually writes logs.
+`docker compose up` starts the Scribe daemon (published image, mounting the repo's
+`config/scribe.conf`) plus a tiny Go sender that connects over Thrift, pushes a few
+`LogEntry` messages, and exits. Resulting log files land in host-visible `./logs`.
+
+Scribe is receive-only: it writes nothing until a client sends Thrift messages. The Go
+sender is what makes the example "write logs". The sender hand-encodes the Thrift framed
+binary `Log()` call using only the Go stdlib (no codegen, no external modules) — chosen for
+the smallest, most deterministic build (approach G1 from brainstorm).
+
+## Phases
+
+| Phase | Name | Status |
+|-------|------|--------|
+| 1 | [Go raw-socket sender](./phase-01-go-raw-socket-sender.md) | Completed |
+| 2 | [Compose orchestration and docs](./phase-02-compose-orchestration-and-docs.md) | Completed |
+
+## Key Decisions (from brainstorm)
+
+- Sender language: **Go**, raw-socket stdlib only (no Apache Thrift lib, no generated bindings).
+- Logs: **host bind mount** `scribe/example/logs` → container `/var/log/scribe`.
+- Scribe service uses the **published image** `ghcr.io/tiennm99/scribe:2.2` (no local build needed).
+- README's existing Python client snippet stays as-is (documented "real API" reference).
+
+## Dependencies
+
+None. New folder only; touches no existing image build. Reuses `scribe/config/scribe.conf`
+(read-only mount) and the published `scribe:2.2` image.