mirror of
https://github.com/tiennm99/goclaw.git
synced 2026-07-13 11:07:28 +00:00
Duy /zuey/
361f2abbf5
Squash merge PR #97 after resolving changelog conflict with current dev. PR CI run 26703171763 passed release-versioning, go, and web.
252 lines
10 KiB
Markdown
252 lines
10 KiB
Markdown
# GitHub Binary Installer
|
||
|
||
Install CLI tools directly from GitHub Releases at runtime. Covers Go, Rust,
|
||
shell, and other binary-distributed tools not available via `apk` / `pip` / `npm`.
|
||
|
||
Closes [#741](https://github.com/nextlevelbuilder/goclaw/issues/741).
|
||
|
||
## Install Syntax
|
||
|
||
```
|
||
github:owner/repo[@tag]
|
||
```
|
||
|
||
Examples:
|
||
- `github:cli/cli` → latest release
|
||
- `github:jesseduffield/lazygit@v0.42.0` → specific version
|
||
- `github:sharkdp/fd@v9.0.0` → specific version with dot separator
|
||
|
||
## How It Works
|
||
|
||
1. Fetches release metadata from the GitHub API
|
||
2. Auto-selects asset matching `linux` + current arch (amd64 / arm64)
|
||
3. Streams download to a temp file, enforcing a max size cap
|
||
4. Verifies SHA256 if the publisher ships `checksums.txt` / `SHA256SUMS`
|
||
5. Validates ELF magic bytes + 64-bit class + machine matches runtime arch
|
||
6. Extracts archive safely (tar.gz / zip / raw binary) with path-traversal + zip-bomb guards
|
||
7. Installs to `{runtimeDir}/bin/` (prepended to `$PATH`)
|
||
8. Persists a manifest for later listing + uninstall
|
||
|
||
## Usage
|
||
|
||
### Web UI
|
||
|
||
1. Admin Settings → Packages page
|
||
2. Scroll to **GitHub Binaries** section
|
||
3. Enter `owner/repo[@tag]` or click **Browse releases** to pick a version
|
||
4. Click **Install**
|
||
|
||
### HTTP API
|
||
|
||
```bash
|
||
# Install
|
||
curl -X POST http://gateway/v1/packages/install \
|
||
-H "Authorization: Bearer $ADMIN_TOKEN" \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"package": "github:jesseduffield/lazygit@v0.42.0"}'
|
||
|
||
# List installed (includes pip/npm/system + github)
|
||
curl http://gateway/v1/packages -H "Authorization: Bearer $ADMIN_TOKEN"
|
||
|
||
# Browse releases (picker UI uses this)
|
||
curl 'http://gateway/v1/packages/github-releases?repo=cli/cli&limit=10' \
|
||
-H "Authorization: Bearer $VIEWER_TOKEN"
|
||
|
||
# Uninstall
|
||
curl -X POST http://gateway/v1/packages/uninstall \
|
||
-H "Authorization: Bearer $ADMIN_TOKEN" \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"package": "github:lazygit"}'
|
||
```
|
||
|
||
## Admin Configuration
|
||
|
||
All configuration is driven by environment variables — **never** place the
|
||
token in `config.json`.
|
||
|
||
| Env var | Default | Notes |
|
||
|---------|---------|-------|
|
||
| `GOCLAW_PACKAGES_GITHUB_TOKEN` | `""` | Optional PAT: rate 60/hr → 5000/hr + private repo access |
|
||
| `GOCLAW_PACKAGES_MAX_ASSET_SIZE_MB` | `200` | Applies to both download cap and 2× uncompressed cap |
|
||
| `GOCLAW_PACKAGES_GITHUB_ALLOWED_ORGS` | `""` | Comma-separated allowlist (empty = all orgs allowed) |
|
||
| `GOCLAW_PACKAGES_GITHUB_BIN_DIR` | `{runtimeDir}/bin` | Where extracted binaries land |
|
||
| `GOCLAW_PACKAGES_GITHUB_MANIFEST` | `{bin_dir}/../github-packages.json` | Manifest path |
|
||
|
||
`packages.scratch_dir` in `config.json` is optional. If it is empty or cannot
|
||
be created, updates use `{runtimeDir}/tmp` so bare-metal services do not depend
|
||
on root-owned release directories such as `/opt/goclaw/tmp`.
|
||
|
||
Token scopes:
|
||
- public-only repos: no scopes required
|
||
- private repos: `repo`
|
||
- org-SSO-enforced repos: must be SSO-authorized PAT
|
||
|
||
## Security
|
||
|
||
- HTTPS-only downloads with SSRF host allowlist:
|
||
`github.com`, `api.github.com`, `objects.githubusercontent.com`,
|
||
`release-assets.githubusercontent.com`, `codeload.github.com`
|
||
- Every redirect hop re-validated (blocks redirect-based host escape)
|
||
- Literal IP hostnames (v4 / v6) always rejected (blocks cloud-metadata access)
|
||
- SHA256 verification when publisher ships `checksums.txt` / `SHA256SUMS` (constant-time compare)
|
||
- ELF magic + 64-bit class + machine-arch validation before `chmod +x`
|
||
- Path-traversal prevention in archive extraction (rejects `..`, absolute, Windows drive, null byte)
|
||
- Zip-bomb guard (cumulative uncompressed bytes capped at 2× max asset size)
|
||
- Symlink / hardlink entries skipped, never written
|
||
- Admin-only API + master-scope guard on install/uninstall
|
||
- Picker endpoint `/v1/packages/github-releases` throttled per user (30 req/min, burst 10) to protect GitHub API quota; anonymous fallback keyed by remote IP. Response is `429 Too Many Requests` with `Retry-After: 60` when tripped.
|
||
- Token never logged (startup log prints `token_set=bool`)
|
||
|
||
## Troubleshooting
|
||
|
||
### "glibc not found" / segfault on execution
|
||
|
||
GoClaw runs on Alpine Linux (musl libc). Many Go/Rust binaries target glibc.
|
||
|
||
**Fix:** pick a musl-compatible release asset. Look for names containing:
|
||
- `*-musl.tar.gz` (explicit musl)
|
||
- `*-linux-static*` (fully static)
|
||
- Go binaries with `CGO_ENABLED=0` typically work out of the box
|
||
|
||
Known-good musl releases:
|
||
- `ripgrep`: `ripgrep-*-x86_64-unknown-linux-musl.tar.gz`
|
||
- `starship`: `starship-x86_64-unknown-linux-musl.tar.gz`
|
||
- `gh`: `gh_*_linux_amd64.tar.gz` (static)
|
||
|
||
### "no matching asset found"
|
||
|
||
Asset naming doesn't fit the heuristic. Open the release page and confirm assets
|
||
exist for `linux` + your arch. Workaround: file an upstream issue asking for
|
||
standard `linux_amd64` / `linux_arm64` naming.
|
||
|
||
### "arch mismatch"
|
||
|
||
Binary is `amd64` but runtime is `arm64` (or vice versa). Pick a release asset
|
||
matching the host arch — the release picker UI filters automatically.
|
||
|
||
### "rate limit exceeded"
|
||
|
||
Anonymous GitHub API is capped at 60 req/hr. Set
|
||
`GOCLAW_PACKAGES_GITHUB_TOKEN` to bump to 5000/hr.
|
||
|
||
### "checksum mismatch"
|
||
|
||
Hard-fail. Indicates tampered download or publisher re-signing without updating
|
||
the release. Do not force-install; report upstream.
|
||
|
||
## Limitations (Phase 1)
|
||
|
||
- Linux-only (Lite/Desktop editions not yet supported)
|
||
- Docker and bare-metal gateway editions (default runtime dir resolves to `/app/data/.runtime/bin` in Docker or `/var/lib/goclaw/data/.runtime/bin` on bare-metal Linux)
|
||
- Installs all top-level executables in an archive (no interactive picker if
|
||
archive contains multiple binaries)
|
||
- No version history / rollback — re-installing replaces in place
|
||
- Global manifest (not per-tenant)
|
||
|
||
## Updating Installed Packages
|
||
|
||
Update flow is **Phase 1 GitHub-only** (pip/npm/apk deferred to Phase 2).
|
||
|
||
### UI
|
||
|
||
The Runtime & Packages page renders a summary bar above the GitHub Binaries
|
||
section when updates are available:
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────┐
|
||
│ 🟡 3 updates available │
|
||
│ Last checked 5m ago [Refresh] [Update All] │
|
||
└─────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
Per-row `[Update]` buttons appear next to each package with a newer release.
|
||
Clicking applies the update via atomic `.bak` swap with automatic rollback on
|
||
failure.
|
||
|
||
### API
|
||
|
||
All write endpoints require **master-scope admin** (tenant admins are denied):
|
||
|
||
| Endpoint | Purpose |
|
||
|---|---|
|
||
| `GET /v1/packages/updates` | Cache snapshot + `{stale, ageSeconds, ttlSeconds}` (operator+) |
|
||
| `POST /v1/packages/updates/refresh` | Force sync CheckAll — fetch from GitHub |
|
||
| `POST /v1/packages/update` | Apply one: body `{"package":"github:lazygit","toVersion":"v0.44.5"}` |
|
||
| `POST /v1/packages/updates/apply-all` | Sequential apply; body `{"packages":[...]}` (empty = all). Always returns 200 — inspect `failed[]` |
|
||
|
||
### Behaviour
|
||
|
||
- **Stale-while-revalidate**: `GET /updates` returns the cached snapshot
|
||
immediately and triggers a background refresh if the cache is older than
|
||
`packages.updates_check_ttl` (default `1h`).
|
||
- **ETag**: responses use `If-None-Match`, so repeated checks cost zero
|
||
rate-limit budget (304 responses don't count against 60/hr).
|
||
- **Pre-releases**: if your current tag matches `(-alpha|-beta|-rc|-pre|-preview|-dev|-nightly)`,
|
||
the checker polls both `/releases/latest` and `/releases?per_page=5` and
|
||
picks the newest via `golang.org/x/mod/semver.Compare`. This correctly
|
||
handles the `v1.0.0-rc.1 → v1.0.0` stable transition.
|
||
- **Non-semver tags** (e.g. `2024-01-15`): string-compare fallback. Never
|
||
downgrades — if the candidate string is lexically less than current, the
|
||
update is suppressed.
|
||
- **Atomic swap**: two-phase rename. Phase A renames ALL current binaries to
|
||
`{name}.bak.{unixNano}`; Phase B renames the new binaries in place. On any
|
||
failure during Phase B, Phase A's renames are rolled back. Manifest is
|
||
persisted AFTER all swaps succeed, with retries (100ms/500ms/1s).
|
||
|
||
### WebSocket events
|
||
|
||
Owner clients receive (non-owner master admins use the HTTP API directly):
|
||
|
||
```
|
||
package.update.checked {count, checked_at}
|
||
package.update.started {source, name, from_version, to_version}
|
||
package.update.succeeded {source, name, from_version, to_version, duration_ms}
|
||
package.update.failed {source, name, reason}
|
||
```
|
||
|
||
### Troubleshooting Updates
|
||
|
||
#### "Binary updated but manifest save failed" (manifestDesynced=true)
|
||
|
||
The `.bak` files are deleted but the manifest didn't record the new version.
|
||
Next update attempt will re-apply the same version. Manual recovery is not
|
||
required — just run the update again OR restart the gateway (which re-reads
|
||
the manifest). No data loss.
|
||
|
||
#### Corrupt updates cache
|
||
|
||
Symptom: UI shows no updates available despite newer releases.
|
||
|
||
Recovery: delete `/app/data/.runtime/updates-cache.json`, click `[Refresh]`.
|
||
|
||
#### Rate-limit exhaustion
|
||
|
||
Symptom: `Refresh` returns 429 or check returns partial results.
|
||
|
||
Check response header `X-RateLimit-Reset` (Unix epoch). Wait or set
|
||
`packages.github_token` in config (Phase 2 auth — unwired in Phase 1).
|
||
|
||
#### Scratch dir leftover after crash
|
||
|
||
Path: `{runtimeDir}/tmp/{name}-{tag}-{nanos}/` unless
|
||
`packages.scratch_dir` points to another writable directory.
|
||
|
||
Safe to remove any `{name}-*-*` directory under tmp after ensuring no active
|
||
update is in flight. Phase 2 will add startup GC.
|
||
|
||
#### Mid-swap process crash
|
||
|
||
Phase 1 leaves `.bak.{nanos}` files on disk. Manual recovery:
|
||
1. Check `{BinDir}` for `*.bak.*` files.
|
||
2. If the main binary is MISSING, rename the `.bak.{nanos}` back to the
|
||
original name.
|
||
3. If the main binary EXISTS but is the new version you wanted, delete the
|
||
`.bak.{nanos}`.
|
||
4. Re-run the update via UI — idempotent.
|
||
|
||
## See Also
|
||
|
||
- [`docs/packages-pip-npm.md`](./packages-pip-npm.md) — pip + npm package updates (Phase 2a)
|
||
- [`docs/14-skills-runtime.md`](./14-skills-runtime.md) — Overview of the runtime packages system
|
||
- Issue [#741](https://github.com/nextlevelbuilder/goclaw/issues/741) — Original feature request
|