diff --git a/README.md b/README.md index e535c19..e03afe6 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,28 @@ The bot validates syntax, tickers, and arithmetic safety, but does not look up notices or prevent duplicate calls. The caller is responsible for verifying the notice and avoiding accidental repeated adjustments. +### Stock and coin P&L accounting + +Stock and coin portfolios persist the total remaining cost basis for each open +position. Buys add their actual spend. Partial sells remove basis using the +weighted-average method and report realized P&L; full sells remove the position +and its basis. Stock share dividends add shares without adding cost, which +lowers the derived average price, while cash dividends do not change position +basis. + +`/stock_portfolio` and `/coin_portfolio` show average entry price and +unrealized P&L for each priced position. `Account P&L` remains the broader +account value minus all top-ups, so it also reflects realized proceeds, +dividend cash, and idle cash. If any current quote is unavailable, totals are +marked partial and numeric Account P&L is withheld. + +On startup, enabled stock and coin modules scan every stored portfolio. Legacy +holdings without basis are initialized at that startup's current market quote, +giving them zero initial unrealized P&L. The migration uses optimistic writes, +records completion in the shared `system` collection, and still verifies the +invariant on every boot. Startup fails rather than accepting trades when a +required legacy quote or migration write is unavailable. + ## Layout ``` @@ -60,7 +82,7 @@ internal/telegram/ Telegram long-polling bot wrapper internal/cron/ in-process cron scheduler internal/modules/ Module framework, registry, dispatchers, modules internal/storage/ typed DocStore[T] (Provider + Typed); mongodb runtime + memory (tests). Values persist as flattened native BSON root documents -internal/systemstate/ shared `system` collection helper for future startup migrations +internal/systemstate/ shared `system` collection helper for startup migration records compose.yml Coolify self-host stack (single bot service) docs/deploy-coolify-selfhosted.md Self-host deploy and operations guide ``` diff --git a/docs/deploy-coolify-selfhosted.md b/docs/deploy-coolify-selfhosted.md index 0dbcb52..6a1fda1 100644 --- a/docs/deploy-coolify-selfhosted.md +++ b/docs/deploy-coolify-selfhosted.md @@ -101,8 +101,12 @@ Successful GIF replies include the result behind Telegram spoiler formatting. > counts and creates indexes on startup. Deleted legacy command rows are > retained with `deleted: true`; `/stats` queries filter those rows from visible > results. A historical `system` collection may remain in MongoDB with completed -> migration records and can be reused if a future one-time startup migration is -> needed. +> migration records. Stock and coin documents store `costBasis` by symbol. On +> every startup, enabled paper-trading modules verify that each positive holding +> has a valid basis; legacy holdings are initialized from current quotes before +> Telegram handlers are installed. The bot intentionally fails startup if a +> required quote or migration write fails. Completed records remain in `system` +> as audit history, but do not suppress later invariant scans. ## 2. Coolify diff --git a/docs/journals/260721-1539-stock-coin-cost-basis-pnl.md b/docs/journals/260721-1539-stock-coin-cost-basis-pnl.md new file mode 100644 index 0000000..dcd5686 --- /dev/null +++ b/docs/journals/260721-1539-stock-coin-cost-basis-pnl.md @@ -0,0 +1,59 @@ +# Stock and Coin Cost Basis P&L Journal + +## Context + +Stock and coin portfolios tracked account funding and holdings but not the +remaining acquisition cost of each position, so they could not distinguish +realized sale results from unrealized open-position performance. + +## What Changed + +- Added persisted per-symbol total remaining `costBasis` to stock and coin + portfolios, with invariant checks on load, update, and save paths. +- Buys add actual spend; partial sells remove proportional weighted-average + basis and report realized P&L; full sells remove the position and its basis. +- Portfolio views now show average entry price and per-position unrealized P&L. + Account P&L remains total account value minus top-ups and is withheld when + missing quotes make valuation partial. +- Stock share dividends preserve total basis while increasing quantity; cash + dividends and top-ups remain outside position basis. +- Sorted, bounded portfolio replies retain summaries while omitting excess + position lines before Telegram's message limit. + +## Migration Safety + +- Enabled modules scan every portfolio before handlers are installed on every + boot, even when a completion marker already exists. +- Missing legacy basis is initialized from a complete current quote set, giving + each migrated position zero initial unrealized P&L without repricing rows that + already have basis. +- Versioned writes retry conflicts; the shared `system` marker is written only + after all rows succeed and remains an audit record rather than a scan bypass. +- A two-minute overall deadline bounds startup. Invalid data, noncanonical + symbols, missing quotes, exhausted conflicts, or storage failures abort + startup instead of allowing trades with unknown basis. + +## Decisions + +- Persist total remaining cost, deriving average entry price as basis divided by + held quantity; do not persist trade lots or cumulative realized P&L. +- Migrate only loaded modules, so disabled-module data waits until re-enabled. +- Preserve all command names, parameter contracts, and existing portfolio data. + +## Verification + +- Passed: focused stock and coin accounting, migration, output, and reply-budget + tests. +- Passed: MongoDB 8 Testcontainers migration and idempotency coverage. +- Passed: `go test ./...` +- Passed: `go vet ./...` +- Passed: `go build ./...` +- Passed: `golangci-lint run` + +## Operational Impact + +- Deployments with legacy holdings may perform market-price lookups and writes + during startup; a required provider or database failure intentionally keeps + the bot offline until initialization can complete safely. +- Subsequent healthy boots still verify invariants but do not reprice completed + positions. diff --git a/plans/260721-1508-stock-coin-cost-basis-pnl/phase-01-design-persistence-and-migration.md b/plans/260721-1508-stock-coin-cost-basis-pnl/phase-01-design-persistence-and-migration.md new file mode 100644 index 0000000..a80816d --- /dev/null +++ b/plans/260721-1508-stock-coin-cost-basis-pnl/phase-01-design-persistence-and-migration.md @@ -0,0 +1,84 @@ +--- +phase: 1 +title: Design persistence and migration +status: completed +priority: P1 +effort: medium +dependencies: [] +--- + +# Phase 1: Design persistence and migration + +## Overview + +Extend both portfolio documents and add retry-safe startup migrations that +initialize legacy holdings from current quotes before command handlers run. + +## Requirements + +- Functional: persist `costBasis` as ticker/coin → total remaining cost. +- Functional: initialize missing legacy basis as `quantity × current price`. +- Functional: preserve any positive basis already written by a previous retry. +- Functional: scan every boot even after completion; the marker is an audit + record and never suppresses invariant checks. +- Non-functional: startup fails on a required quote/storage error; migration is + idempotent and bounded by a migration-wide deadline. + +## Architecture + +`cmd/server` checks which modules are loaded, then invokes each module's +`InitStore` before `modules.Install`. Every boot lists `user:` documents and +checks `positive holding => finite positive basis`. Migration fetches each +required symbol once, verifies the complete requested set, and writes only +missing basis using bounded CAS reload/retry. The shared `system` marker records +completion but does not skip future scans. Existing populated basis acts as +per-position retry progress. + +## Related Code Files + +- Modify: `internal/modules/stock/portfolio.go` +- Modify: `internal/modules/coin/portfolio.go` +- Create: `internal/modules/stock/startup.go` +- Create: `internal/modules/coin/startup.go` +- Modify: `cmd/server/main.go` +- Create: stock/coin startup memory and MongoDB tests + +## Implementation Steps + +1. Add and defensively initialize `CostBasis map[string]float64` with + `json:"costBasis" bson:"costBasis"` in both portfolio types. +2. Validate before normalization. Reject corrupt basis, noncanonical symbols, + canonical-key collisions, and non-finite holdings; do not delete or merge + ambiguous legacy assets. +3. Implement injectable stock and coin migrations with stable `system` keys, + `user:` listing, unique-symbol quote caching, complete quote-set validation, + bounded CAS reload/retry, progress logs, and a completion marker written last. +4. Wire migration only for loaded modules after registry construction and + before handler installation. Return startup-fatal errors with module/symbol + context but no credentials. +5. Apply an overall startup-migration timeout; timeout is startup-fatal and + leaves the marker incomplete. +6. Test empty, legacy, mixed, partial-retry, post-marker missing rows, partial + quote maps, noncanonical/corrupt rows, CAS conflicts, timeout, storage + failure, and idempotent second-boot cases in memory and MongoDB 8. + +## Success Criteria + +- [ ] New portfolios always contain initialized basis maps. +- [ ] Existing documents without `costBasis` still decode. +- [ ] Each legacy symbol receives migration-time current-price basis. +- [ ] Retry never overwrites basis initialized by an earlier attempt. +- [ ] Any required failure prevents the completion marker and aborts startup. +- [ ] Disabled modules do not call external quote providers. +- [ ] A completion marker never hides a later holding with missing basis. + +## Risk Assessment + +- External quote outages can block startup by explicit owner choice; provider + timeouts bound the delay and logs identify the module/symbol. +- Partial writes are recoverable because populated per-symbol basis is never + repriced and the global marker is written only after all rows succeed. +- Runtime validation fails closed if an old writer or restored row reintroduces + a positive holding without valid basis after startup. +- Storage listing remains unpaginated; the migration-wide deadline bounds the + current small, one-replica deployment without broad storage-API scope. diff --git a/plans/260721-1508-stock-coin-cost-basis-pnl/phase-02-implement-weighted-average-accounting.md b/plans/260721-1508-stock-coin-cost-basis-pnl/phase-02-implement-weighted-average-accounting.md new file mode 100644 index 0000000..0087a09 --- /dev/null +++ b/plans/260721-1508-stock-coin-cost-basis-pnl/phase-02-implement-weighted-average-accounting.md @@ -0,0 +1,81 @@ +--- +phase: 2 +title: Implement weighted-average accounting +status: completed +priority: P1 +effort: medium +dependencies: + - 1 +--- + +# Phase 2: Implement weighted-average accounting + +## Overview + +Update stock and coin trade mutations to maintain remaining cost basis and show +realized P&L on every successful sale. + +## Requirements + +- Functional: buys add actual transaction spend to the symbol's total basis. +- Functional: sells remove proportional basis and report realized P&L. +- Functional: full exits remove both holding and basis keys. +- Functional: stock share dividends change quantity but not total basis. +- Non-functional: rejected trades and failed saves leave portfolio state intact. +- Non-functional: positive holdings with absent, nonpositive, or non-finite + basis fail closed instead of producing fabricated accounting. + +## Architecture + +For a sale, capture pre-sale holding and basis, then calculate: + +```text +sold_basis = total_basis × sold_quantity / held_quantity +realized_pnl = proceeds - sold_basis +realized_pct = realized_pnl / sold_basis × 100 +``` + +The remaining average price is unchanged after a partial sale. A share dividend +adds shares without cost, so `basis / new_quantity` automatically lowers the +average price. No lot ledger or cumulative realized-P&L field is introduced. + +## Related Code Files + +- Modify: stock/coin `portfolio.go`, trade handlers, and format helpers +- Modify: stock dividend handler tests to assert unchanged total basis +- Modify: stock/coin portfolio and handler tests + +## Implementation Steps + +1. Add small portfolio methods to add purchase basis and remove proportional + basis with finite/overflow guards appropriate to each module's precision. +2. Update buys so quantity, cash deduction, and basis addition persist in the + same mutation. +3. Update sells so quantity, cash credit, and basis reduction persist together; + append `Realized P&L` amount and percentage to the reply. +4. Determine coin full exit from normalized post-sale holdings. If quantity + disappears at the dust threshold, assign all remaining basis to that sale + and delete the basis key. Keep monetary validation separate from quantity + dust normalization. +5. Assert cash/share/combined dividend behavior: cash never changes basis and + share additions preserve total basis. +6. Cover weighted repeated buys, gain/loss/breakeven partial sells, full sells, + insufficient holdings, overflow/invalid data, conflicts, and save failure. + +## Success Criteria + +- [ ] `average price == remaining basis / remaining quantity` after every buy, + sell, and share dividend. +- [ ] Stock and coin sell replies report correct realized P&L. +- [ ] Full exits leave no holding or basis entry. +- [ ] Failed/rejected operations do not mutate persisted state. +- [ ] Missing or corrupt runtime basis blocks affected accounting operations + with a clear retry/operator message. + +## Risk Assessment + +- Stock uses float64 currency already; calculations must reject non-finite or + unsafe results. Coin retains its existing dust normalization. +- Mutation order must not delete quantity before sold basis is calculated. +- Existing account `Meta.Invested` continues to mean deposits and must not be + repurposed as position basis. diff --git a/plans/260721-1508-stock-coin-cost-basis-pnl/phase-03-verify-p-l-behavior-and-compatibility.md b/plans/260721-1508-stock-coin-cost-basis-pnl/phase-03-verify-p-l-behavior-and-compatibility.md new file mode 100644 index 0000000..710a915 --- /dev/null +++ b/plans/260721-1508-stock-coin-cost-basis-pnl/phase-03-verify-p-l-behavior-and-compatibility.md @@ -0,0 +1,86 @@ +--- +phase: 3 +title: Verify P&L behavior and compatibility +status: completed +priority: P1 +effort: medium +dependencies: + - 1 + - 2 +--- + +# Phase 3: Verify P&L behavior and compatibility + +## Overview + +Expose average entry price and unrealized position P&L in both portfolio views, +document the accounting model, and run full compatibility verification. + +## Requirements + +- Functional: each priced holding shows average price and unrealized P&L. +- Functional: retain cash, total value, invested deposits, and account P&L. +- Functional: unavailable quotes degrade gracefully without inventing P&L; + aggregate account P&L is suppressed when any holding is unpriced. +- Non-functional: output remains deterministic and within Telegram limits. + +## Architecture + +Position metrics derive from persisted remaining basis: + +```text +average_price = basis / quantity +unrealized_pnl = current_value - basis +unrealized_pct = unrealized_pnl / basis × 100 +``` + +Label position performance `Unrealized P&L` and the existing net-worth minus +top-ups metric `Account P&L` so dividends, cash, and realized proceeds are not +confused with open-position performance. If any holding lacks a quote, label +priced values as partial and do not render aggregate account P&L. + +## Related Code Files + +- Modify: `internal/modules/stock/handlers.go`, `stats_test.go` +- Modify: `internal/modules/coin/views.go` and view/handler tests +- Modify: `README.md`, `docs/deploy-coolify-selfhosted.md` +- Create: completion journal during finalization + +## Implementation Steps + +1. Render average basis and signed unrealized P&L for each priced stock/coin + position. Add explicit stock sorting; retain coin sorting. +2. Keep existing total-value and invested calculations; relabel the final line + as `Account P&L` and add aggregate `Unrealized P&L` for priced positions. +3. For unavailable current quotes, show stored average price when valid, mark + current value/P&L unavailable, label priced totals partial, and suppress + aggregate account P&L. +4. Add a conservative Telegram reply budget with deterministic truncation and + test worst-case multi-position output. +5. Update exact output, missing-price, reply-budget, migration compatibility, + and command behavior tests. +6. Document startup migration, weighted-average basis, realized vs unrealized + P&L, and dividend effects. +7. Run `gofmt`, focused stock/coin/server tests, MongoDB Testcontainers tests, + `go test -count=1 ./...`, `go vet ./...`, `go build ./...`, and + `golangci-lint run`; complete independent test/debug/review gates. + +## Success Criteria + +- [ ] Each open position shows average price and unrealized amount/percentage. +- [ ] Portfolio retains account-level P&L with a clearer label. +- [ ] Missing prices do not fabricate P&L or break the reply. +- [ ] Stock output is sorted and both portfolio replies remain within Telegram + limits under worst-case supported holdings. +- [ ] Legacy memory and MongoDB documents remain readable and migrate once. +- [ ] All focused and repository-wide verification passes. +- [ ] No command name, parameter, stats-history, or price-provider contract + changes. + +## Risk Assessment + +- Users may confuse account and position metrics; explicit labels and README + examples define both. +- Missing quotes make totals partial today; the new output must label them + partial and suppress account P&L rather than report a false loss. +- Longer replies require budget tests for multi-asset coin portfolios. diff --git a/plans/260721-1508-stock-coin-cost-basis-pnl/plan.md b/plans/260721-1508-stock-coin-cost-basis-pnl/plan.md new file mode 100644 index 0000000..4d419a6 --- /dev/null +++ b/plans/260721-1508-stock-coin-cost-basis-pnl/plan.md @@ -0,0 +1,108 @@ +--- +title: Stock and Coin Cost Basis P&L +description: >- + Persist weighted-average cost basis, migrate legacy holdings at startup + prices, and expose realized/unrealized P&L. +status: completed +priority: P1 +branch: main +tags: + - stock + - coin + - mongodb + - migration + - pnl +blockedBy: [] +blocks: [] +created: '2026-07-21T08:08:21.155Z' +createdBy: 'ck:plan' +source: skill +--- + +# Stock and Coin Cost Basis P&L + +## Overview + +Add a per-symbol remaining cost basis to stock and coin portfolios. New buys +increase basis, partial sells remove proportional weighted-average basis, and +full sells remove it. Portfolio views show average entry price and unrealized +P&L per position; sell replies show realized P&L for that sale while preserving +the existing account-level P&L. + +Legacy holdings migrate before handlers are installed. Each missing basis is +initialized from the current quote, so its unrealized P&L starts at zero. +Migration is idempotent, marker-backed, and fail-fast: the bot does not accept +trades if required quotes or storage writes fail. + +## Phases + +| Phase | Name | Status | +|-------|------|--------| +| 1 | [Design persistence and migration](./phase-01-design-persistence-and-migration.md) | Completed | +| 2 | [Implement weighted-average accounting](./phase-02-implement-weighted-average-accounting.md) | Completed | +| 3 | [Verify P&L behavior and compatibility](./phase-03-verify-p-l-behavior-and-compatibility.md) | Completed | + +## Dependencies + +- No active plan dependencies. The completed stock-dividend implementation + defines the share-dividend behavior this plan must preserve. +- MongoDB 8 Testcontainers are required for persisted migration verification; + existing Docker-unavailable warning/skip behavior remains unchanged. + +## Decisions + +- Persist total remaining cost per symbol, not a mutable average-price field. +- Derive average price as `basis / held quantity`. +- Use proportional weighted-average basis for partial sells. +- Keep total stock basis unchanged when share dividends add quantity. +- Keep cash dividends and top-ups outside position basis. +- Do not persist cumulative realized P&L or trade lots. +- Run a module migration only when that module is loaded; disabled-module data + migrates the next time the module is enabled. +- Abort startup on migration failure to avoid operating with unknown basis. + +## Acceptance Criteria + +- Stock and coin portfolios persist remaining cost basis without breaking + legacy BSON/JSON decoding. +- Repeated buys produce the correct weighted average; partial/full sells update + basis exactly and report realized gain or loss. +- Startup maintenance scans every boot, initializes only missing legacy basis + from one complete current quote per symbol, never reprices completed work, + and writes its system marker only after success. +- Share dividends add quantity without adding cost; cash dividends do not + change position basis. +- Portfolio output shows average price and per-position unrealized P&L plus the + existing account-level P&L. +- No command names or parameter contracts change. +- Focused tests, MongoDB migration tests, full tests, vet, build, and lint pass. + +## Red Team Review + +### Session — 2026-07-21 + +**Findings:** 12 deduplicated (9 accepted, 3 rejected) + +| Finding | Severity | Disposition | Applied To | +|---|---|---|---| +| Completion marker cannot enforce future row invariants | Critical | Accept | Phases 1–2 | +| Missing quotes fabricate aggregate account P&L | Critical | Accept | Phase 3 | +| Migration has no overall deadline | High | Accept | Phase 1 | +| Partial stock quote maps can look successful | High | Accept | Phase 1 | +| Corrupt/noncanonical legacy symbols are ambiguous | High | Accept | Phase 1 | +| Migration conflicts lack retry semantics | Medium | Accept | Phase 1 | +| Coin quantity dust can orphan monetary basis | Medium | Accept | Phase 2 | +| Invalid basis can be silently normalized | Medium | Accept | Phases 1–2 | +| Stock output order and reply size are unsafe | Medium | Accept | Phase 3 | +| Storage-wide pagination and migration lease | High | Reject | Deadline and fail-closed runtime validation fit the one-replica deployment | +| Persist quote manifests and correction tooling | High | Reject | Outside paper-trading scope; validate complete finite quotes and log context | +| Restrict portfolio commands to private chats | Medium | Reject | Pre-existing visibility contract outside this accounting change | + +### Whole-Plan Consistency Sweep + +- Files reread: `plan.md` and all three phase files. +- Decision deltas checked: every-boot invariant scan, fail-closed runtime checks, + bounded startup, quote completeness, corrupt-symbol rejection, CAS retry, + dust cleanup, missing-price suppression, sorting, and reply bounds. +- Reconciled stale references: marker short-circuiting and partial aggregate P&L. +- Unresolved contradictions: 0.