From 9a47995d7ceebcc09eefd46f2dc31d8d5506540c Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Wed, 27 May 2026 17:49:05 +0700 Subject: [PATCH] docs(planning): add planning artifacts for scribe example implementation Planning docs for the Go sender example feature, including: - Phase overviews and implementation steps - Research findings on Scribe protocol and Go integration - Task breakdown and success criteria --- .../phase-01-go-raw-socket-sender.md | 123 ++++++++++++++++++ ...phase-02-compose-orchestration-and-docs.md | 111 ++++++++++++++++ .../plan.md | 53 ++++++++ 3 files changed, 287 insertions(+) create mode 100644 plans/260527-1702-scribe-example-go-sender/phase-01-go-raw-socket-sender.md create mode 100644 plans/260527-1702-scribe-example-go-sender/phase-02-compose-orchestration-and-docs.md create mode 100644 plans/260527-1702-scribe-example-go-sender/plan.md diff --git a/plans/260527-1702-scribe-example-go-sender/phase-01-go-raw-socket-sender.md b/plans/260527-1702-scribe-example-go-sender/phase-01-go-raw-socket-sender.md new file mode 100644 index 0000000..e719470 --- /dev/null +++ b/plans/260527-1702-scribe-example-go-sender/phase-01-go-raw-socket-sender.md @@ -0,0 +1,123 @@ +--- +phase: 1 +title: Go raw-socket sender +status: completed +priority: P2 +effort: 2h +dependencies: [] +--- + +# Phase 1: Go raw-socket sender + +## Overview + +A tiny Go program that connects to the Scribe daemon over TCP and sends `LogEntry` +messages by hand-encoding the Thrift framed binary `Log()` RPC — stdlib only, no Apache +Thrift dependency, no generated bindings. Compiles to a small static binary in a +multi-stage Docker build. + +## Requirements + +- Functional: connect to `SCRIBE_HOST:SCRIBE_PORT`, send `SCRIBE_COUNT` messages under + category `SCRIBE_CATEGORY`, parse the `ResultCode` reply, log OK/TRY_LATER, exit 0 on success. +- Functional: retry the initial connect (~10s, ~1s interval) so a cold `docker compose up` + does not fail before Scribe is listening. +- Non-functional: no external Go modules (stdlib `net`, `encoding/binary`, `os`, `time`, `log`). +- Non-functional: static binary (`CGO_ENABLED=0`); minimal runtime image. + +## Architecture + +### Scribe Thrift contract (from facebookarchive/scribe IDL) + +```thrift +enum ResultCode { OK = 0, TRY_LATER = 1 } +struct LogEntry { 1: string category; 2: string message; } +service scribe { ResultCode Log(1: list messages); } +``` + +### Wire format to produce + +- **Transport:** `TFramedTransport` → 4-byte big-endian frame length prefixes the payload. +- **Protocol:** `TBinaryProtocol`, **non-strict** message header (matches README client which + uses `strictWrite=False`; Scribe C++ server reads with `strict_read=false`, accepts it). + +Thrift type IDs used: `STOP=0`, `I32=8`, `STRING=11`, `STRUCT=12`, `LIST=15`. All ints big-endian. + +**Request payload (`Log` CALL):** +1. Message begin (non-strict): `string "Log"` (i32 len + bytes) · `i8 msgType=1 (CALL)` · `i32 seqid=0` +2. Args struct `Log_args`: + - field begin: `i8 type=15 (LIST)` · `i16 id=1` + - list begin: `i8 elemType=12 (STRUCT)` · `i32 size=N` + - per `LogEntry`: + - field: `i8 11 (STRING)` · `i16 1` · writeString(category) + - field: `i8 11 (STRING)` · `i16 2` · writeString(message) + - `i8 0 (STOP)` + - args struct: `i8 0 (STOP)` +3. Prepend `i32` total length → frame. + +`writeString(s)` = `i32 len(s)` + UTF-8 bytes. + +**Response parse (minimal):** +1. Read 4-byte frame length, then that many bytes. +2. Message begin (non-strict): i32 name-len + name + `i8 msgType` + `i32 seqid`. +3. Field begin `i8 type` · `i16 id`; for return value id=0 type=I32 → read `i32` ResultCode. +4. Map 0→OK, 1→TRY_LATER; anything else → log raw + treat non-zero as failure. + +### Connect-retry + +Loop `net.DialTimeout("tcp", addr, 2s)` up to 10 attempts, 1s between failures; fatal after. + +## Related Code Files + +- Create: `scribe/example/sender/main.go` — sender program (encode/send/parse + retry). +- Create: `scribe/example/sender/go.mod` — `module scribe-example-sender` / `go 1.22`, stdlib only. +- Create: `scribe/example/sender/Dockerfile` — multi-stage build. + +### Dockerfile (multi-stage) + +```dockerfile +FROM golang:1.22-alpine AS build +WORKDIR /src +COPY go.mod . +COPY main.go . +RUN CGO_ENABLED=0 go build -o /sender . + +FROM alpine:3.20 +COPY --from=build /sender /sender +ENTRYPOINT ["/sender"] +``` + +### main.go env inputs (with defaults) + +| Env | Default | Purpose | +|-----|---------|---------| +| `SCRIBE_HOST` | `scribe` | daemon hostname (compose service name) | +| `SCRIBE_PORT` | `1463` | Thrift port | +| `SCRIBE_CATEGORY` | `example` | message category → log subdir | +| `SCRIBE_COUNT` | `5` | number of LogEntry messages to send | + +## Implementation Steps + +1. Create `scribe/example/sender/go.mod` (module name + `go 1.22`). +2. Write `main.go`: + - read env with defaults; build `bytes.Buffer` helpers for `i8/i16/i32/string`. + - `buildLogFrame(entries)` → payload then 4-byte length prefix. + - connect-retry loop → write frame → read+parse response frame → log ResultCode. + - `os.Exit(0)` on OK; non-zero exit on connect failure or non-OK. +3. Write the multi-stage `Dockerfile`. +4. Local sanity build: `docker build -t scribe-sender scribe/example/sender` (verify compiles). + +## Success Criteria + +- [ ] `docker build scribe/example/sender` succeeds (binary compiles, no module fetch). +- [ ] Run against a live scribe container: sender logs `sent N messages, result=OK` and exits 0. +- [ ] Sender survives a cold start (scribe not yet listening) via the retry loop. + +## Risk Assessment + +- **Hand-encoded Thrift correctness** — wire format above is the stable 0.9.1 binary layout; + keep helpers tiny and comment each field. Mitigation: validate end-to-end in Phase 2 by + confirming the written log file contains the exact messages. +- **Strict vs non-strict header** — if the server unexpectedly rejects non-strict, switch + message-begin to strict (`i32 version=0x80010001`, name, `i32 seqid`). Documented here as + the fallback; default is non-strict to match the README client. diff --git a/plans/260527-1702-scribe-example-go-sender/phase-02-compose-orchestration-and-docs.md b/plans/260527-1702-scribe-example-go-sender/phase-02-compose-orchestration-and-docs.md new file mode 100644 index 0000000..c2af6cd --- /dev/null +++ b/plans/260527-1702-scribe-example-go-sender/phase-02-compose-orchestration-and-docs.md @@ -0,0 +1,111 @@ +--- +phase: 2 +title: Compose orchestration and docs +status: completed +priority: P2 +effort: 1h +dependencies: + - 1 +--- + +# Phase 2: Compose orchestration and docs + +## Overview + +Wire the Scribe daemon and the Go sender (Phase 1) into a single `docker compose up` +demo, bind-mount logs to a host-visible folder, and document run + verify steps. End +result: one command produces real log files under `scribe/example/logs/`. + +## Requirements + +- Functional: `docker compose up` (from `scribe/example/`) starts Scribe with the repo's + `config/scribe.conf`, runs the sender once, and writes files to host `./logs`. +- Functional: logs are inspectable on the host without entering the container. +- Non-functional: scribe service uses published `ghcr.io/tiennm99/scribe:2.2` (no local build). +- Non-functional: example README is self-contained (run, verify, troubleshoot). + +## Architecture + +### docker-compose.yml + +```yaml +services: + scribe: + image: ghcr.io/tiennm99/scribe:2.2 + ports: + - "1463:1463" + volumes: + - ../config/scribe.conf:/etc/scribe/scribe.conf:ro + - ./logs:/var/log/scribe + restart: unless-stopped + + sender: + build: ./sender + depends_on: + - scribe + environment: + - SCRIBE_HOST=scribe + - SCRIBE_PORT=1463 + - SCRIBE_CATEGORY=example + - SCRIBE_COUNT=5 +``` + +Notes: +- Sender reaches Scribe over the compose network by service name `scribe:1463` (internal), + independent of the host `1463:1463` mapping. +- `depends_on` only orders start, not readiness — the Phase 1 retry loop covers the gap. +- Sender is one-shot: it sends then exits; scribe keeps running (`restart: unless-stopped`). + +### Expected on-disk result + +``` +scribe/example/logs/ +└── example/ + └── example-_00000 # contains the 5 sent messages +``` + +(Category `example` → subdir under the default catch-all store's `file_path=/var/log/scribe`.) + +## Related Code Files + +- Create: `scribe/example/docker-compose.yml` +- Create: `scribe/example/logs/.gitkeep` — keep the bind-mount dir in git, empty. +- Create: `scribe/example/README.md` — run/verify/troubleshoot for the example. +- Modify: `scribe/README.md` — add a short "Runnable example" pointer linking to `example/`. + +### scribe/example/README.md outline + +1. **What it does** — one command boots Scribe + sends test messages → log files appear. +2. **Run** — `cd scribe/example && docker compose up --build`. +3. **Verify** — show `scribe/example/logs/example/example-*_00000` and `cat` a line. +4. **Customize** — env vars (`SCRIBE_CATEGORY`, `SCRIBE_COUNT`), pointer to `config/scribe.conf`. +5. **Troubleshoot** — empty logs dir = no messages sent yet; Linux bind-mount permissions caveat. +6. **Stop** — `docker compose down`. + +## Implementation Steps + +1. Create `scribe/example/docker-compose.yml` per spec above. +2. Create `scribe/example/logs/.gitkeep`. +3. Write `scribe/example/README.md` per outline. +4. Add a "Runnable example" subsection to `scribe/README.md` linking to `example/`. +5. End-to-end test: `cd scribe/example && docker compose up --build`; confirm a log file + appears under `logs/example/` containing the sent messages; `docker compose down`. + +## Success Criteria + +- [ ] `docker compose up --build` starts both services without manual steps. +- [ ] After the sender exits, `scribe/example/logs/example/example-*_00000` exists on host. +- [ ] That file contains the test messages sent by the Go sender. +- [ ] `scribe/example/README.md` documents run, verify, and the Linux permission caveat. +- [ ] `scribe/README.md` links to the example. + +## Risk Assessment + +- **Bind-mount permissions (native Linux)** — Scribe runs as non-root user `scribe` + (`scribe/Dockerfile:107-110`); host `./logs` may reject its writes. On Docker Desktop + (Windows/macOS) this is normally fine. Mitigation: document `chmod 777 ./logs` (or matching + uid) in the README troubleshoot section. +- **Stale relative paths** — `../config/scribe.conf` and `./sender` build context are relative + to `scribe/example/`; compose must be run from that dir. Documented in README run step. +- **Empty logs confusion** — if the sender fails silently, logs dir stays empty. Mitigation: + sender logs success/failure to stdout (`docker compose` output) and exits non-zero on failure. diff --git a/plans/260527-1702-scribe-example-go-sender/plan.md b/plans/260527-1702-scribe-example-go-sender/plan.md new file mode 100644 index 0000000..5270651 --- /dev/null +++ b/plans/260527-1702-scribe-example-go-sender/plan.md @@ -0,0 +1,53 @@ +--- +title: Scribe runnable example with Go test sender +description: >- + Self-contained scribe/example/ that boots the Scribe daemon and a Go sender so + docker compose up writes visible log files to host ./logs +status: completed +priority: P2 +branch: main +tags: + - scribe + - example + - docker-compose + - golang + - thrift +blockedBy: [] +blocks: [] +created: '2026-05-27T10:27:22.391Z' +createdBy: 'ck:plan' +source: skill +--- + +# Scribe runnable example with Go test sender + +## Overview + +Add `scribe/example/` — a one-command demo proving the Scribe config actually writes logs. +`docker compose up` starts the Scribe daemon (published image, mounting the repo's +`config/scribe.conf`) plus a tiny Go sender that connects over Thrift, pushes a few +`LogEntry` messages, and exits. Resulting log files land in host-visible `./logs`. + +Scribe is receive-only: it writes nothing until a client sends Thrift messages. The Go +sender is what makes the example "write logs". The sender hand-encodes the Thrift framed +binary `Log()` call using only the Go stdlib (no codegen, no external modules) — chosen for +the smallest, most deterministic build (approach G1 from brainstorm). + +## Phases + +| Phase | Name | Status | +|-------|------|--------| +| 1 | [Go raw-socket sender](./phase-01-go-raw-socket-sender.md) | Completed | +| 2 | [Compose orchestration and docs](./phase-02-compose-orchestration-and-docs.md) | Completed | + +## Key Decisions (from brainstorm) + +- Sender language: **Go**, raw-socket stdlib only (no Apache Thrift lib, no generated bindings). +- Logs: **host bind mount** `scribe/example/logs` → container `/var/log/scribe`. +- Scribe service uses the **published image** `ghcr.io/tiennm99/scribe:2.2` (no local build needed). +- README's existing Python client snippet stays as-is (documented "real API" reference). + +## Dependencies + +None. New folder only; touches no existing image build. Reuses `scribe/config/scribe.conf` +(read-only mount) and the published `scribe:2.2` image.