diff --git a/README.md b/README.md index f40541d..03fb1ce 100644 --- a/README.md +++ b/README.md @@ -53,9 +53,10 @@ notice and avoiding accidental repeated adjustments. ### Stock and coin P&L accounting -Stock and coin portfolios embed each open position under `assets.` with -`quantity`, total remaining `base`, and `dividendCheckedAt`. Stock cash is -stored directly as `vnd`; coin cash remains `usd`. Buys add their actual spend. Partial sells remove basis using the +Stock and coin portfolios embed each open position under `assets.`. +Both store `quantity` and total remaining `base`; stock positions additionally +store `dividendCheckedAt`. Stock cash is stored directly as `vnd`; coin cash +remains `usd`. Buys add their actual spend. Partial sells remove basis using the weighted-average method and report realized P&L; full sells remove the position and its basis. Stock share dividends add shares without adding cost, which lowers the derived average price, while cash dividends do not change position @@ -67,16 +68,11 @@ account value minus all top-ups, so it also reflects realized proceeds, dividend cash, and idle cash. If any current quote is unavailable, totals are marked partial and numeric Account P&L is withheld. -`dividendCheckedAt` is a future dividend-event cursor. It is initialized when a -position is first bought, preserved across later buys and sells, and advanced -when a stock dividend is recorded. A full exit removes the cursor; reopening -the position starts it again. - -On startup, enabled stock and coin modules migrate the previous flat -`assets`/`costBasis` shape into embedded asset documents. Stock also migrates -`currency.VND` to `vnd`. The migration preserves the already-established basis, -uses optimistic writes, records completion in the shared `system` collection, -and verifies the new invariant every boot. +For stock positions, `dividendCheckedAt` is a future dividend-event cursor. It +is initialized when a position is first bought, preserved across later buys and +sells, and advanced when a stock dividend is recorded. A full exit removes the +cursor; reopening the position starts it again. Coin positions do not store a +dividend cursor. ## Layout diff --git a/cmd/server/main.go b/cmd/server/main.go index 2ce26b8..692d175 100644 --- a/cmd/server/main.go +++ b/cmd/server/main.go @@ -29,7 +29,6 @@ import ( "github.com/tiennm99/miti99bot/internal/modules/wordle" "github.com/tiennm99/miti99bot/internal/server" "github.com/tiennm99/miti99bot/internal/storage" - "github.com/tiennm99/miti99bot/internal/systemstate" "github.com/tiennm99/miti99bot/internal/telegram" ) @@ -94,7 +93,6 @@ func factories() map[string]modules.Factory { // shutdown). Atlas SRV DNS + TLS handshake can take a couple seconds on a cold // container; 10s leaves headroom without hiding a wedged cluster. const mongodbInitTimeout = 10 * time.Second -const portfolioMigrationTimeout = 2 * time.Minute func main() { rootCtx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) @@ -115,7 +113,7 @@ func main() { } defer closeProvider() - if err := stats.InitStore(rootCtx, provider.Collection("stats"), provider.Collection(systemstate.CollectionName)); err != nil { + if err := stats.InitStore(rootCtx, provider.Collection("stats")); err != nil { log.Fatal("stats storage init failed", "err", err) } if err := lol.InitStore(rootCtx, provider.Collection(lol.CollectionName)); err != nil { @@ -133,20 +131,6 @@ func main() { if err != nil { log.Fatal("module registry build failed", "err", err) } - migrationCtx, cancelMigrations := context.WithTimeout(rootCtx, portfolioMigrationTimeout) - if moduleLoaded(reg, stock.CollectionName) { - if err := stock.InitStore(migrationCtx, provider.Collection(stock.CollectionName), provider.Collection(systemstate.CollectionName)); err != nil { - cancelMigrations() - log.Fatal("stock storage init failed", "err", err) - } - } - if moduleLoaded(reg, coin.CollectionName) { - if err := coin.InitStore(migrationCtx, provider.Collection(coin.CollectionName), provider.Collection(systemstate.CollectionName)); err != nil { - cancelMigrations() - log.Fatal("coin storage init failed", "err", err) - } - } - cancelMigrations() auth := modules.Auth{BotOwnerID: cfg.BotOwnerID, AdminUserIDs: cfg.AdminUserIDs} modules.Install(b, reg, auth) log.Info("modules loaded", @@ -225,15 +209,6 @@ func main() { } } -func moduleLoaded(reg *modules.Registry, name string) bool { - for _, module := range reg.Modules { - if module.Name == name { - return true - } - } - return false -} - // buildProvider picks the storage backend. Selection order: // 1. Explicit KV_PROVIDER env (memory|mongodb) wins. // 2. Auto-detect: MONGO_URL set → mongodb; otherwise memory. diff --git a/docs/deploy-coolify-selfhosted.md b/docs/deploy-coolify-selfhosted.md index 7dd20bf..ca98d5c 100644 --- a/docs/deploy-coolify-selfhosted.md +++ b/docs/deploy-coolify-selfhosted.md @@ -101,14 +101,11 @@ Successful GIF replies include the result behind Telegram spoiler formatting. > counts and creates indexes on startup. Deleted legacy command rows are > retained with `deleted: true`; `/stats` queries filter those rows from visible > results. A historical `system` collection may remain in MongoDB with completed -> migration records. Stock stores cash as `vnd`; coin stores cash as `usd`. -> Both embed positions as -> `assets..{quantity,base,dividendCheckedAt}`. On every startup, enabled -> paper-trading modules verify this shape and migrate the previous flat -> `assets`/`costBasis` documents before Telegram handlers are installed. The bot -> intentionally fails startup if validation or a migration write fails. -> Completed records remain in `system` as audit history, but do not suppress -> later invariant scans. +> migration records. Stock stores cash as `vnd` and embeds positions as +> `assets..{quantity,base,dividendCheckedAt}`. Coin stores cash as `usd` +> and embeds positions as `assets..{quantity,base}`. Completed migration +> records remain in `system` as audit history; the completed one-time migration +> code no longer runs at startup. ## 2. Coolify diff --git a/docs/journals/260721-1704-dividend-api-and-migration-cleanup.md b/docs/journals/260721-1704-dividend-api-and-migration-cleanup.md new file mode 100644 index 0000000..7a8e8c2 --- /dev/null +++ b/docs/journals/260721-1704-dividend-api-and-migration-cleanup.md @@ -0,0 +1,87 @@ +--- +type: technical-journal +topic: dividend-api-and-migration-cleanup +conducted_at: 2026-07-21T17:04:00+07:00 +status: complete +--- + +# Dividend API and Migration Cleanup Journal + +## Context + +Portfolio schema migrations were verified complete in production, so their +compatibility code had become permanent startup and maintenance overhead. Coin +positions also inherited a stock-only dividend cursor with no valid use. +Separately, research was needed before designing automatic discovery of +Vietnamese stock dividend events. + +## What Happened + +- Removed `dividendCheckedAt` from coin positions, validation, buy behavior, + and tests. Stock retains the cursor because dividend events apply there. +- Retired completed stock and coin portfolio migrations plus the completed + stats command-renaming migration and their migration-only tests. +- Preserved recurring startup maintenance: stats query indexes and the LoL + match-cache TTL index remain idempotently initialized and tested. +- Preserved historical migration records in MongoDB's `system` collection and + kept the reusable `internal/systemstate` helper for future migrations. +- Removed legacy numeric-position JSON/BSON decoders after production schema + completion was confirmed. Current models now describe only the supported + nested asset schema. +- Accepted lazy cleanup of stale coin cursor fields. BSON decoding ignores the + old unknown field, and the next portfolio mutation uses versioned + `ReplaceOne`, rewriting that portfolio in the current shape without a new + one-time migration. + +## Dividend API Research + +SSI iBoard's corporate-actions endpoint is the recommended initial source. It +currently returns anonymous JSON with ticker/date filters, pagination, stable +`CorId` values, cash amounts, ratios, and event dates. It covered verified cash +and share dividend examples. + +The endpoint is undocumented and has no published SLA, rate limit, or stability +contract. Any implementation should therefore isolate it behind a replaceable +provider, validate and locally classify events, deduplicate by `CorId`, and ask +the user to confirm before changing a portfolio. VSDC remains the authoritative +notice source; a licensed FiinGroup feed is the stronger future option if this +becomes production-critical. + +## Decisions + +- Keep dividend state stock-only; coin assets persist only `quantity` and + total remaining `base`. +- Remove completed one-time runtime code rather than continuing to scan already + migrated portfolios on every boot. +- Do not delete system history or general migration infrastructure. +- Do not introduce a cleanup migration solely for stale coin BSON fields; + normal writes remove them safely over time. +- Treat SSI iBoard as a replaceable prototype provider, not a guaranteed public + API contract. + +## Verification + +- Passed focused tests: + `go test -count=1 ./internal/modules/coin ./internal/modules/stock ./internal/modules/stats ./cmd/server` +- Passed full suite: `go test -count=1 ./...` +- MongoDB 8 tests executed and passed for stats indexes and the LoL TTL index. +- Passed: `go vet ./...` +- Passed: `go build ./...` +- Passed with zero issues: `golangci-lint run` +- Passed: `git diff --check` (expected LF/CRLF working-copy warnings only). +- Independent tester, debugger, and reviewer found no defects. + +## Reflection + +Migration code is operationally valuable only while incompatible data can +still exist. Removing it after verification narrows startup failure modes and +makes the active persistence contract explicit. Lazy removal is appropriate +for an ignored field because it does not affect reads or correctness; structural +schema changes still require guarded migrations. + +## Next + +Design the Telegram interaction for listing SSI dividend events, including date +windows, pagination, exact ratio conversion, ambiguity handling, and explicit +user confirmation. Preserve manual dividend commands and keep automatic event +application out of scope until that interaction is approved. diff --git a/internal/modules/coin/handlers.go b/internal/modules/coin/handlers.go index c054844..5bad019 100644 --- a/internal/modules/coin/handlers.go +++ b/internal/modules/coin/handlers.go @@ -102,7 +102,7 @@ func (s *state) handleBuy(ctx context.Context, b *bot.Bot, update *models.Update insufficientBalance = &balance return errInsufficientUSD } - return p.BuyTicker(coin.Symbol, qty, amount, now) + return p.BuyTicker(coin.Symbol, qty, amount) }) if errors.Is(err, errInsufficientUSD) && insufficientBalance != nil { return chathelper.Reply(ctx, b, update.Message, diff --git a/internal/modules/coin/handlers_test.go b/internal/modules/coin/handlers_test.go index ef40c2f..e5f1c90 100644 --- a/internal/modules/coin/handlers_test.go +++ b/internal/modules/coin/handlers_test.go @@ -156,7 +156,7 @@ func TestHandleBuyAndSell(t *testing.T) { if p.USD != 500 || p.Assets["BTC"].Quantity != 0.01 { t.Fatalf("after buy = %+v", p) } - if p.Assets["BTC"].Base != 500 || p.Assets["BTC"].DividendCheckedAt != 123 { + if p.Assets["BTC"].Base != 500 { t.Fatalf("buy asset = %+v", p.Assets["BTC"]) } s.prices.(fakePriceFetcher).prices["BTC"] = CoinPrice{USD: 60_000, Source: "Binance"} @@ -354,7 +354,7 @@ func TestStatsTreatsOverflowedValuationAsUnavailable(t *testing.T) { ctx := context.Background() s := newTestState(map[string]CoinPrice{"BTC": {USD: math.MaxFloat64, Source: "test"}}, nil) p := NewPortfolio(1) - p.Assets["BTC"] = AssetPosition{Quantity: 2, Base: 1, DividendCheckedAt: 1} + p.Assets["BTC"] = AssetPosition{Quantity: 2, Base: 1} if err := SavePortfolio(ctx, s.store, 7, p); err != nil { t.Fatal(err) } diff --git a/internal/modules/coin/portfolio.go b/internal/modules/coin/portfolio.go index fdeba9d..c828a53 100644 --- a/internal/modules/coin/portfolio.go +++ b/internal/modules/coin/portfolio.go @@ -1,64 +1,24 @@ package coin import ( - "bytes" "context" - "encoding/json" "errors" "fmt" "math" "strconv" - "go.mongodb.org/mongo-driver/v2/bson" - "github.com/tiennm99/miti99bot/internal/storage" ) const coinDustEpsilon = 1e-9 const portfolioUpdateAttempts = 5 +const CollectionName = "coin" type Store = storage.DocStore[Portfolio] type AssetPosition struct { - Quantity float64 `json:"quantity" bson:"quantity"` - Base float64 `json:"base" bson:"base"` - DividendCheckedAt int64 `json:"dividendCheckedAt" bson:"dividendCheckedAt"` - legacyQuantity bool -} - -func (p *AssetPosition) UnmarshalJSON(data []byte) error { - data = bytes.TrimSpace(data) - if len(data) > 0 && data[0] == '{' { - type plain AssetPosition - return json.Unmarshal(data, (*plain)(p)) - } - var quantity float64 - if err := json.Unmarshal(data, &quantity); err != nil { - return fmt.Errorf("coin: decode legacy asset quantity: %w", err) - } - *p = AssetPosition{Quantity: quantity, legacyQuantity: true} - return nil -} - -func (p *AssetPosition) UnmarshalBSONValue(valueType byte, data []byte) error { - raw := bson.RawValue{Type: bson.Type(valueType), Value: data} - if raw.Type == bson.TypeEmbeddedDocument { - type plain AssetPosition - return raw.Unmarshal((*plain)(p)) - } - if quantity, ok := raw.DoubleOK(); ok { - *p = AssetPosition{Quantity: quantity, legacyQuantity: true} - return nil - } - if quantity, ok := raw.Int64OK(); ok { - *p = AssetPosition{Quantity: float64(quantity), legacyQuantity: true} - return nil - } - if quantity, ok := raw.Int32OK(); ok { - *p = AssetPosition{Quantity: float64(quantity), legacyQuantity: true} - return nil - } - return fmt.Errorf("coin: unsupported legacy asset BSON type %s", raw.Type) + Quantity float64 `json:"quantity" bson:"quantity"` + Base float64 `json:"base" bson:"base"` } type Portfolio struct { @@ -148,7 +108,7 @@ func (p Portfolio) Validate() error { for symbol, position := range p.Assets { coin, err := ResolveCoinSymbol(symbol) if err != nil || coin.Symbol != symbol || !isPositiveFinite(position.Quantity) || - !isPositiveFinite(position.Base) || position.DividendCheckedAt <= 0 { + !isPositiveFinite(position.Base) { return fmt.Errorf("coin: %s has invalid position", symbol) } } @@ -169,8 +129,8 @@ func (p *Portfolio) DeductUSD(amount float64) (ok bool, balance float64) { return true, p.USD } -func (p *Portfolio) BuyTicker(symbol string, quantity, base float64, now int64) error { - if !isPositiveFinite(quantity) || !isPositiveFinite(base) || now <= 0 { +func (p *Portfolio) BuyTicker(symbol string, quantity, base float64) error { + if !isPositiveFinite(quantity) || !isPositiveFinite(base) { return fmt.Errorf("coin: invalid purchase position") } if p.Assets == nil { @@ -182,9 +142,6 @@ func (p *Portfolio) BuyTicker(symbol string, quantity, base float64, now int64) if !isPositiveFinite(position.Quantity) || !isPositiveFinite(position.Base) { return fmt.Errorf("coin: position overflows") } - if position.DividendCheckedAt == 0 { - position.DividendCheckedAt = now - } p.Assets[symbol] = position return nil } diff --git a/internal/modules/coin/portfolio_test.go b/internal/modules/coin/portfolio_test.go index 6a1bebf..c5789d3 100644 --- a/internal/modules/coin/portfolio_test.go +++ b/internal/modules/coin/portfolio_test.go @@ -6,6 +6,8 @@ import ( "math" "testing" + "go.mongodb.org/mongo-driver/v2/bson" + "github.com/tiennm99/miti99bot/internal/storage" ) @@ -16,35 +18,59 @@ func TestLoadPortfolioFirstTimeUser(t *testing.T) { } } -func TestCoinBuySellMathAndCursor(t *testing.T) { +func TestPortfolioDropsLegacyDividendCursorOnBSONRoundTrip(t *testing.T) { + legacy, err := bson.Marshal(bson.M{ + "usd": 100.0, + "assets": bson.M{"BTC": bson.M{ + "quantity": 0.5, + "base": 25_000.0, + "dividendCheckedAt": int64(123), + }}, + "meta": bson.M{"invested": 100.0, "createdAt": int64(1)}, + }) + if err != nil { + t.Fatal(err) + } + var portfolio Portfolio + if err := bson.Unmarshal(legacy, &portfolio); err != nil { + t.Fatal(err) + } + if err := portfolio.Validate(); err != nil { + t.Fatalf("legacy cursor prevented load: %v", err) + } + + current, err := bson.Marshal(portfolio) + if err != nil { + t.Fatal(err) + } + position := bson.Raw(current).Lookup("assets").Document().Lookup("BTC").Document() + if _, err := position.LookupErr("dividendCheckedAt"); err == nil { + t.Fatal("legacy dividend cursor survived current BSON encoding") + } +} + +func TestCoinBuySellMath(t *testing.T) { p := NewPortfolio(1) p.AddUSD(1000) p.Meta.Invested = 1000 if ok, balance := p.DeductUSD(250); !ok || balance != 750 { t.Fatalf("balance=%v ok=%v", balance, ok) } - if err := p.BuyTicker("BTC", 0.1, 250, 10); err != nil { + if err := p.BuyTicker("BTC", 0.1, 250); err != nil { t.Fatal(err) } - if err := p.BuyTicker("BTC", 0.05, 150, 20); err != nil { + if err := p.BuyTicker("BTC", 0.05, 150); err != nil { t.Fatal(err) } - position := p.Assets["BTC"] - if position.DividendCheckedAt != 10 { - t.Fatalf("cursor=%d", position.DividendCheckedAt) - } remaining, soldBase, ok, err := p.SellTicker("BTC", 0.06) if err != nil || !ok || math.Abs(remaining-0.09) > 1e-12 || math.Abs(soldBase-160) > 1e-9 { t.Fatalf("remaining=%v soldBase=%v ok=%v err=%v", remaining, soldBase, ok, err) } - if p.Assets["BTC"].DividendCheckedAt != 10 { - t.Fatal("sell changed dividend cursor") - } } func TestCoinFullSellRemovesTicker(t *testing.T) { p := NewPortfolio(1) - _ = p.BuyTicker("BTC", 0.3, 12_000, 10) + _ = p.BuyTicker("BTC", 0.3, 12_000) _, soldBase, ok, err := p.SellTicker("BTC", 0.3) if err != nil || !ok || math.Abs(soldBase-12_000) > 1e-9 || len(p.Assets) != 0 { t.Fatalf("portfolio=%+v soldBase=%v ok=%v err=%v", p, soldBase, ok, err) diff --git a/internal/modules/coin/startup.go b/internal/modules/coin/startup.go deleted file mode 100644 index 0a19b55..0000000 --- a/internal/modules/coin/startup.go +++ /dev/null @@ -1,118 +0,0 @@ -package coin - -import ( - "context" - "errors" - "fmt" - "time" - - "github.com/tiennm99/miti99bot/internal/log" - "github.com/tiennm99/miti99bot/internal/storage" - "github.com/tiennm99/miti99bot/internal/systemstate" -) - -const ( - CollectionName = "coin" - assetSchemaMarkerKey = "migration:coin-asset-schema-v2" - tickerMigrationRetries = 5 -) - -type legacyPortfolio struct { - USD float64 `json:"usd" bson:"usd"` - Assets map[string]AssetPosition `json:"assets" bson:"assets"` - CostBasis map[string]float64 `json:"costBasis,omitempty" bson:"costBasis,omitempty"` - Meta PortfolioMeta `json:"meta" bson:"meta"` -} - -func InitStore(ctx context.Context, portfolioColl, systemColl storage.Collection) error { - docs := storage.Typed[legacyPortfolio](portfolioColl) - system := systemstate.New(systemColl) - marker, markerExists, err := system.Get(ctx, assetSchemaMarkerKey) - if err != nil { - return fmt.Errorf("coin asset schema migration: read marker: %w", err) - } - keys, err := docs.List(ctx, "user:") - if err != nil { - return fmt.Errorf("coin asset schema migration: list portfolios: %w", err) - } - var migrated int64 - for index, key := range keys { - changed, err := migrateAssetSchema(ctx, docs, key, time.Now().UnixMilli()) - if err != nil { - return err - } - if changed { - migrated++ - log.Info("coin asset schema migrated", "portfolio", index+1, "total", len(keys)) - } - } - now := time.Now().UnixMilli() - if markerExists && marker.Status == "completed" && migrated == 0 { - return nil - } - if !markerExists { - marker = systemstate.Record{Kind: "migration", Name: "coin asset schema v2", CompletedAt: now} - } - marker.Status = "completed" - marker.Count += migrated - marker.UpdatedAt = now - if err := system.Put(ctx, assetSchemaMarkerKey, marker); err != nil { - return fmt.Errorf("coin asset schema migration: write marker: %w", err) - } - return nil -} - -func migrateAssetSchema(ctx context.Context, docs storage.DocStore[legacyPortfolio], key string, now int64) (bool, error) { - for attempt := 0; attempt < tickerMigrationRetries; attempt++ { - doc, version, err := docs.Get(ctx, key) - if err != nil { - return false, fmt.Errorf("coin asset schema migration: read %s: %w", key, err) - } - changed, err := doc.migrate(now) - if err != nil { - return false, fmt.Errorf("coin asset schema migration: %s: %w", key, err) - } - if !changed { - return false, nil - } - if err := docs.PutVersioned(ctx, key, version, doc); err == nil { - return true, nil - } else if !errors.Is(err, storage.ErrConflict) { - return false, fmt.Errorf("coin asset schema migration: write %s: %w", key, err) - } - } - return false, fmt.Errorf("coin asset schema migration: write %s: %w", key, storage.ErrConflict) -} - -func (p *legacyPortfolio) migrate(now int64) (bool, error) { - hasLegacyQuantity := false - for _, position := range p.Assets { - hasLegacyQuantity = hasLegacyQuantity || position.legacyQuantity - } - hasLegacy := p.CostBasis != nil || hasLegacyQuantity - if !hasLegacy { - return false, Portfolio{USD: p.USD, Assets: p.Assets, Meta: p.Meta}.Validate() - } - for symbol, position := range p.Assets { - if !position.legacyQuantity { - return false, fmt.Errorf("document mixes legacy and nested assets") - } - if position.Quantity == 0 { - delete(p.Assets, symbol) - continue - } - coin, err := ResolveCoinSymbol(symbol) - base := p.CostBasis[symbol] - if err != nil || coin.Symbol != symbol || !isPositiveFinite(position.Quantity) || !isPositiveFinite(base) { - return false, fmt.Errorf("invalid legacy position %q", symbol) - } - p.Assets[symbol] = AssetPosition{Quantity: position.Quantity, Base: base, DividendCheckedAt: now} - } - for symbol, base := range p.CostBasis { - if !isPositiveFinite(base) || p.Assets[symbol].Quantity <= 0 { - return false, fmt.Errorf("orphan or invalid legacy basis %q", symbol) - } - } - p.CostBasis = nil - return true, Portfolio{USD: p.USD, Assets: p.Assets, Meta: p.Meta}.Validate() -} diff --git a/internal/modules/coin/startup_mongo_test.go b/internal/modules/coin/startup_mongo_test.go deleted file mode 100644 index a160470..0000000 --- a/internal/modules/coin/startup_mongo_test.go +++ /dev/null @@ -1,71 +0,0 @@ -package coin - -import ( - "context" - "fmt" - "os" - "testing" - "time" - - "go.mongodb.org/mongo-driver/v2/bson" - - "github.com/tiennm99/miti99bot/internal/storage" - "github.com/tiennm99/miti99bot/internal/systemstate" - "github.com/tiennm99/miti99bot/internal/testutil/mongotest" -) - -var mongoTests mongotest.Manager - -func TestMain(m *testing.M) { - os.Exit(mongoTests.Run(m)) -} - -func TestInitStoreMigratesCoinBasisInMongoDB(t *testing.T) { - ctx, portfolioColl, systemColl := setupMongoCoinTest(t) - if err := storage.Typed[oldCoinPortfolio](portfolioColl).Put(ctx, "user:7", oldCoinPortfolio{ - USD: 500, Assets: map[string]float64{"BTC": 0.25}, CostBasis: map[string]float64{"BTC": 25_000}, - Meta: PortfolioMeta{CreatedAt: 1}, - }); err != nil { - t.Fatal(err) - } - if err := InitStore(ctx, portfolioColl, systemColl); err != nil { - t.Fatalf("InitStore: %v", err) - } - if err := InitStore(ctx, portfolioColl, systemColl); err != nil { - t.Fatalf("InitStore second run: %v", err) - } - got, _, err := storage.Typed[Portfolio](portfolioColl).Get(ctx, "user:7") - if err != nil || got.USD != 500 || got.Assets["BTC"].Quantity != 0.25 || got.Assets["BTC"].Base != 25_000 { - t.Fatalf("portfolio=%+v err=%v", got, err) - } - rawColl, _ := storage.MongoCollection(portfolioColl) - var raw bson.M - if err := rawColl.FindOne(ctx, bson.M{"_id": "user:7"}).Decode(&raw); err != nil { - t.Fatal(err) - } - for _, legacyField := range []string{"costBasis", "tickers"} { - if _, exists := raw[legacyField]; exists { - t.Fatalf("legacy field %q remains in %#v", legacyField, raw) - } - } -} - -func setupMongoCoinTest(t *testing.T) (context.Context, storage.Collection, storage.Collection) { - t.Helper() - uri := mongoTests.URI(t) - ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) - t.Cleanup(cancel) - client, err := storage.NewMongoClient(ctx, uri) - if err != nil { - t.Fatal(err) - } - db := client.Database(fmt.Sprintf("miti99bot_coin_test_%d", time.Now().UnixNano())) - t.Cleanup(func() { - cleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), 10*time.Second) - defer cleanupCancel() - _ = db.Drop(cleanupCtx) - _ = client.Disconnect(cleanupCtx) - }) - provider := storage.NewMongoProvider(db) - return ctx, provider.Collection(CollectionName), provider.Collection(systemstate.CollectionName) -} diff --git a/internal/modules/coin/startup_test.go b/internal/modules/coin/startup_test.go deleted file mode 100644 index 866e3bd..0000000 --- a/internal/modules/coin/startup_test.go +++ /dev/null @@ -1,74 +0,0 @@ -package coin - -import ( - "context" - "testing" - - "github.com/tiennm99/miti99bot/internal/storage" - "github.com/tiennm99/miti99bot/internal/systemstate" -) - -type oldCoinPortfolio struct { - USD float64 `json:"usd" bson:"usd"` - Assets map[string]float64 `json:"assets" bson:"assets"` - CostBasis map[string]float64 `json:"costBasis" bson:"costBasis"` - Meta PortfolioMeta `json:"meta" bson:"meta"` -} - -func TestInitStoreMigratesCoinNestedAssets(t *testing.T) { - ctx := context.Background() - provider := storage.NewMemoryProvider() - portfolioColl := provider.Collection(CollectionName) - systemColl := provider.Collection(systemstate.CollectionName) - if err := storage.Typed[oldCoinPortfolio](portfolioColl).Put(ctx, "user:7", oldCoinPortfolio{ - USD: 500, Assets: map[string]float64{"BTC": 0.25}, CostBasis: map[string]float64{"BTC": 20_000}, - Meta: PortfolioMeta{CreatedAt: 1}, - }); err != nil { - t.Fatal(err) - } - if err := InitStore(ctx, portfolioColl, systemColl); err != nil { - t.Fatalf("InitStore: %v", err) - } - got, err := LoadPortfolio(ctx, storage.Typed[Portfolio](portfolioColl), 7, 9) - if err != nil { - t.Fatal(err) - } - position := got.Assets["BTC"] - if got.USD != 500 || position.Quantity != 0.25 || position.Base != 20_000 || position.DividendCheckedAt <= 0 { - t.Fatalf("portfolio=%+v", got) - } - if err := InitStore(ctx, portfolioColl, systemColl); err != nil { - t.Fatalf("second InitStore: %v", err) - } - marker, exists, err := systemstate.New(systemColl).Get(ctx, assetSchemaMarkerKey) - if err != nil || !exists || marker.Count != 1 { - t.Fatalf("marker=%+v exists=%v err=%v", marker, exists, err) - } -} - -func TestCoinMigrationRejectsMissingBasis(t *testing.T) { - ctx := context.Background() - provider := storage.NewMemoryProvider() - coll := provider.Collection(CollectionName) - if err := storage.Typed[oldCoinPortfolio](coll).Put(ctx, "user:7", oldCoinPortfolio{ - Assets: map[string]float64{"BTC": 1}, CostBasis: map[string]float64{}, - }); err != nil { - t.Fatal(err) - } - if err := InitStore(ctx, coll, provider.Collection(systemstate.CollectionName)); err == nil { - t.Fatal("InitStore accepted legacy holding without basis") - } -} - -func TestCoinSchemaMigrationRejectsMixedAssetShapes(t *testing.T) { - doc := legacyPortfolio{ - Assets: map[string]AssetPosition{ - "BTC": {Quantity: 1, Base: 10, DividendCheckedAt: 1}, - "ETH": {Quantity: 1, legacyQuantity: true}, - }, - CostBasis: map[string]float64{"ETH": 2_000}, - } - if _, err := doc.migrate(123); err == nil { - t.Fatal("migration accepted mixed legacy and nested assets") - } -} diff --git a/internal/modules/stats/startup.go b/internal/modules/stats/startup.go index a6ab657..cbf71a9 100644 --- a/internal/modules/stats/startup.go +++ b/internal/modules/stats/startup.go @@ -2,253 +2,32 @@ package stats import ( "context" - "errors" "fmt" - "math" - "strconv" - "strings" - "time" "go.mongodb.org/mongo-driver/v2/bson" "go.mongodb.org/mongo-driver/v2/mongo" "go.mongodb.org/mongo-driver/v2/mongo/options" "github.com/tiennm99/miti99bot/internal/storage" - "github.com/tiennm99/miti99bot/internal/systemstate" ) const ( statsCommandUsersIndexName = "stats_cmd_n_user" statsUserCommandsIndexName = "stats_uid_n_cmd" statsUsernameLookupIndexName = "stats_user_uid" - - dividendStatsMigrationKey = "migration:stock-dividend-command-stats-v1" - dividendStatsRowMarkerPrefix = dividendStatsMigrationKey + ":row:" - migrationStatusPrepared = "prepared" - migrationStatusCompleted = "completed" ) -type commandRename struct { - from string - to string -} - -var dividendCommandRenames = []commandRename{ - {from: "stock_dividend", to: "stock_cash_dividend"}, - {from: "stock_bonus", to: "stock_share_dividend"}, -} - // InitStore performs stats collection startup maintenance. It is safe to call -// every boot: MongoDB indexes and the guarded stats migration are idempotent. -func InitStore(ctx context.Context, statsColl, systemColl storage.Collection) error { +// every boot because MongoDB index creation is idempotent. +func InitStore(ctx context.Context, statsColl storage.Collection) error { if mongoColl, ok := storage.MongoCollection(statsColl); ok { if err := ensureUsageIndexes(ctx, mongoColl); err != nil { return err } } - if err := migrateDividendCommandStats( - ctx, - storage.Typed[usageEntry](statsColl), - storage.Typed[systemstate.Record](systemColl), - ); err != nil { - return fmt.Errorf("stats dividend command migration: %w", err) - } return nil } -func migrateDividendCommandStats( - ctx context.Context, - statsDocs storage.DocStore[usageEntry], - systemDocs storage.DocStore[systemstate.Record], -) error { - global, exists, err := getSystemRecord(ctx, systemDocs, dividendStatsMigrationKey) - if err != nil { - return fmt.Errorf("read global marker: %w", err) - } - if exists && global.Status == migrationStatusCompleted { - return nil - } - - for _, rename := range dividendCommandRenames { - keys, err := statsDocs.List(ctx, rename.from) - if err != nil { - return fmt.Errorf("list %s rows: %w", rename.from, err) - } - for _, key := range keys { - userID, ok := usageUserIDForCommandKey(key, rename.from) - if !ok { - continue - } - if err := migrateDividendStatsRow(ctx, statsDocs, systemDocs, rename, userID); err != nil { - return err - } - } - } - - markerKeys, err := systemDocs.List(ctx, dividendStatsRowMarkerPrefix) - if err != nil { - return fmt.Errorf("list row checkpoints: %w", err) - } - for _, markerKey := range markerKeys { - rename, userID, ok := parseDividendStatsRowMarkerKey(markerKey) - if !ok { - continue - } - marker, exists, err := getSystemRecord(ctx, systemDocs, markerKey) - if err != nil { - return fmt.Errorf("read row checkpoint %s: %w", markerKey, err) - } - if !exists || marker.Status != migrationStatusPrepared { - continue - } - if err := migrateDividendStatsRow(ctx, statsDocs, systemDocs, rename, userID); err != nil { - return err - } - } - - now := time.Now().UnixMilli() - return systemDocs.Put(ctx, dividendStatsMigrationKey, systemstate.Record{ - Kind: "migration", - Name: "stock dividend command stats v1", - Status: migrationStatusCompleted, - CompletedAt: now, - UpdatedAt: now, - }) -} - -func migrateDividendStatsRow( - ctx context.Context, - statsDocs storage.DocStore[usageEntry], - systemDocs storage.DocStore[systemstate.Record], - rename commandRename, - userID int64, -) error { - markerKey := dividendStatsRowMarkerKey(rename.from, userID) - marker, markerExists, err := getSystemRecord(ctx, systemDocs, markerKey) - if err != nil { - return fmt.Errorf("read %s checkpoint for user %d: %w", rename.from, userID, err) - } - if markerExists && marker.Status == migrationStatusCompleted { - return nil - } - - sourceKey := usageKey(rename.from, userID) - targetKey := usageKey(rename.to, userID) - source, sourceExists, err := getUsageEntry(ctx, statsDocs, sourceKey) - if err != nil { - return fmt.Errorf("read source %s: %w", sourceKey, err) - } - target, targetExists, err := getUsageEntry(ctx, statsDocs, targetKey) - if err != nil { - return fmt.Errorf("read target %s: %w", targetKey, err) - } - - if !markerExists { - if !sourceExists { - return nil - } - if source.N < 0 || target.N < 0 || source.N > math.MaxInt64-target.N { - return fmt.Errorf("count overflow for %s user %d", rename.from, userID) - } - now := time.Now().UnixMilli() - marker = systemstate.Record{ - Kind: "migration-row", - Name: rename.from + " -> " + rename.to, - Status: migrationStatusPrepared, - Count: target.N + source.N, - UpdatedAt: now, - } - if err := systemDocs.Put(ctx, markerKey, marker); err != nil { - return fmt.Errorf("prepare %s user %d: %w", rename.from, userID, err) - } - } - - if !targetExists { - target = usageEntry{UserID: userID} - } - target.Cmd = rename.to - target.UserID = userID - target.N = marker.Count - target.Deleted = false - if sourceExists && source.Username != "" { - target.Username = source.Username - } - if userID == 0 { - target.Username = "" - } - if err := statsDocs.Put(ctx, targetKey, target); err != nil { - return fmt.Errorf("write target %s: %w", targetKey, err) - } - if sourceExists { - if err := statsDocs.Delete(ctx, sourceKey); err != nil { - return fmt.Errorf("delete source %s: %w", sourceKey, err) - } - } - - now := time.Now().UnixMilli() - marker.Status = migrationStatusCompleted - marker.CompletedAt = now - marker.UpdatedAt = now - if err := systemDocs.Put(ctx, markerKey, marker); err != nil { - return fmt.Errorf("complete %s user %d: %w", rename.from, userID, err) - } - return nil -} - -func getUsageEntry(ctx context.Context, docs storage.DocStore[usageEntry], key string) (usageEntry, bool, error) { - entry, _, err := docs.Get(ctx, key) - if errors.Is(err, storage.ErrNotFound) { - return usageEntry{}, false, nil - } - return entry, err == nil, err -} - -func getSystemRecord(ctx context.Context, docs storage.DocStore[systemstate.Record], key string) (systemstate.Record, bool, error) { - record, _, err := docs.Get(ctx, key) - if errors.Is(err, storage.ErrNotFound) { - return systemstate.Record{}, false, nil - } - return record, err == nil, err -} - -func usageUserIDForCommandKey(key, command string) (int64, bool) { - if key == command { - return 0, true - } - prefix := command + ":" - if !strings.HasPrefix(key, prefix) { - return 0, false - } - userID, err := strconv.ParseInt(strings.TrimPrefix(key, prefix), 10, 64) - return userID, err == nil && userID != 0 -} - -func dividendStatsRowMarkerKey(sourceCommand string, userID int64) string { - return dividendStatsRowMarkerPrefix + sourceCommand + ":" + strconv.FormatInt(userID, 10) -} - -func parseDividendStatsRowMarkerKey(key string) (commandRename, int64, bool) { - remainder := strings.TrimPrefix(key, dividendStatsRowMarkerPrefix) - if remainder == key { - return commandRename{}, 0, false - } - separator := strings.LastIndexByte(remainder, ':') - if separator < 1 { - return commandRename{}, 0, false - } - sourceCommand := remainder[:separator] - userID, err := strconv.ParseInt(remainder[separator+1:], 10, 64) - if err != nil { - return commandRename{}, 0, false - } - for _, rename := range dividendCommandRenames { - if rename.from == sourceCommand { - return rename, userID, true - } - } - return commandRename{}, 0, false -} - func ensureUsageIndexes(ctx context.Context, coll *mongo.Collection) error { models := []mongo.IndexModel{ { diff --git a/internal/modules/stats/startup_mongo_test.go b/internal/modules/stats/startup_mongo_test.go index 7e4c6a1..07d65cf 100644 --- a/internal/modules/stats/startup_mongo_test.go +++ b/internal/modules/stats/startup_mongo_test.go @@ -18,17 +18,17 @@ func TestMain(m *testing.M) { } func TestInitStore_MongoCreatesIndexes(t *testing.T) { - ctx, statsColl, systemColl := setupMongoStatsTest(t) + ctx, statsColl := setupMongoStatsTest(t) rawStatsColl, ok := storage.MongoCollection(statsColl) if !ok { t.Fatal("stats collection is not Mongo-backed") } - if err := InitStore(ctx, statsColl, systemColl); err != nil { + if err := InitStore(ctx, statsColl); err != nil { t.Fatalf("InitStore: %v", err) } - if err := InitStore(ctx, statsColl, systemColl); err != nil { + if err := InitStore(ctx, statsColl); err != nil { t.Fatalf("InitStore second run: %v", err) } @@ -58,25 +58,7 @@ func TestInitStore_MongoCreatesIndexes(t *testing.T) { } } -func TestInitStore_MongoMigratesDividendStatsIdempotently(t *testing.T) { - ctx, statsColl, systemColl := setupMongoStatsTest(t) - docs := storage.Typed[usageEntry](statsColl) - seedUsageEntries(t, docs, map[string]usageEntry{ - usageKey("stock_bonus", 0): {Cmd: "stock_bonus", N: 4}, - usageKey("stock_bonus", 7): {Cmd: "stock_bonus", UserID: 7, Username: "alice", N: 5}, - usageKey("stock_share_dividend", 7): {Cmd: "stock_share_dividend", UserID: 7, Username: "alice", N: 2}, - }) - if err := InitStore(ctx, statsColl, systemColl); err != nil { - t.Fatalf("InitStore: %v", err) - } - if err := InitStore(ctx, statsColl, systemColl); err != nil { - t.Fatalf("InitStore second run: %v", err) - } - assertUsageEntry(t, docs, usageKey("stock_share_dividend", 0), 4, "") - assertUsageEntry(t, docs, usageKey("stock_share_dividend", 7), 7, "alice") -} - -func setupMongoStatsTest(t *testing.T) (context.Context, storage.Collection, storage.Collection) { +func setupMongoStatsTest(t *testing.T) (context.Context, storage.Collection) { t.Helper() uri := mongoTests.URI(t) @@ -98,5 +80,5 @@ func setupMongoStatsTest(t *testing.T) (context.Context, storage.Collection, sto }) provider := storage.NewMongoProvider(db) - return ctx, provider.Collection("stats"), provider.Collection("system") + return ctx, provider.Collection("stats") } diff --git a/internal/modules/stats/startup_test.go b/internal/modules/stats/startup_test.go deleted file mode 100644 index dd1110f..0000000 --- a/internal/modules/stats/startup_test.go +++ /dev/null @@ -1,186 +0,0 @@ -package stats - -import ( - "context" - "errors" - "testing" - - "github.com/tiennm99/miti99bot/internal/storage" - "github.com/tiennm99/miti99bot/internal/systemstate" -) - -func TestInitStoreMigratesDividendCommandStats(t *testing.T) { - ctx := context.Background() - provider := storage.NewMemoryProvider() - statsColl := provider.Collection("stats") - systemColl := provider.Collection(systemstate.CollectionName) - statsDocs := storage.Typed[usageEntry](statsColl) - systemDocs := storage.Typed[systemstate.Record](systemColl) - - seedUsageEntries(t, statsDocs, map[string]usageEntry{ - usageKey("stock_dividend", 0): {Cmd: "stock_dividend", N: 10}, - usageKey("stock_cash_dividend", 0): {Cmd: "stock_cash_dividend", N: 3}, - usageKey("stock_dividend", 7): {Cmd: "stock_dividend", UserID: 7, Username: "alice", N: 4}, - usageKey("stock_cash_dividend", 7): {Cmd: "stock_cash_dividend", UserID: 7, Username: "old-alice", N: 2}, - usageKey("stock_bonus", 0): {Cmd: "stock_bonus", N: 8}, - usageKey("stock_bonus", 9): {Cmd: "stock_bonus", UserID: 9, Username: "bob", N: 5}, - usageKey("stock_share_dividend", 9): {Cmd: "stock_share_dividend", UserID: 9, Username: "previous", N: 1}, - }) - - if err := InitStore(ctx, statsColl, systemColl); err != nil { - t.Fatalf("InitStore: %v", err) - } - assertUsageEntry(t, statsDocs, usageKey("stock_cash_dividend", 0), 13, "") - assertUsageEntry(t, statsDocs, usageKey("stock_cash_dividend", 7), 6, "alice") - assertUsageEntry(t, statsDocs, usageKey("stock_share_dividend", 0), 8, "") - assertUsageEntry(t, statsDocs, usageKey("stock_share_dividend", 9), 6, "bob") - for _, key := range []string{ - usageKey("stock_dividend", 0), usageKey("stock_dividend", 7), - usageKey("stock_bonus", 0), usageKey("stock_bonus", 9), - } { - if _, _, err := statsDocs.Get(ctx, key); !errors.Is(err, storage.ErrNotFound) { - t.Fatalf("source %s still exists or get failed: %v", key, err) - } - } - global, _, err := systemDocs.Get(ctx, dividendStatsMigrationKey) - if err != nil || global.Status != migrationStatusCompleted { - t.Fatalf("global marker = %+v, %v", global, err) - } - markerKeys, err := systemDocs.List(ctx, dividendStatsRowMarkerPrefix) - if err != nil || len(markerKeys) != 4 { - t.Fatalf("row markers = %v, %v", markerKeys, err) - } - - if err := InitStore(ctx, statsColl, systemColl); err != nil { - t.Fatalf("InitStore second run: %v", err) - } - assertUsageEntry(t, statsDocs, usageKey("stock_cash_dividend", 0), 13, "") - assertUsageEntry(t, statsDocs, usageKey("stock_share_dividend", 9), 6, "bob") -} - -func TestDividendStatsMigrationResumesPreparedCheckpointWithoutSource(t *testing.T) { - ctx := context.Background() - provider := storage.NewMemoryProvider() - statsDocs := storage.Typed[usageEntry](provider.Collection("stats")) - systemDocs := storage.Typed[systemstate.Record](provider.Collection(systemstate.CollectionName)) - markerKey := dividendStatsRowMarkerKey("stock_dividend", 7) - - if err := statsDocs.Put(ctx, usageKey("stock_cash_dividend", 7), usageEntry{ - Cmd: "stock_cash_dividend", UserID: 7, Username: "alice", N: 12, - }); err != nil { - t.Fatalf("seed target: %v", err) - } - if err := systemDocs.Put(ctx, markerKey, systemstate.Record{ - Kind: "migration-row", Name: "stock_dividend -> stock_cash_dividend", Status: migrationStatusPrepared, Count: 12, - }); err != nil { - t.Fatalf("seed marker: %v", err) - } - - if err := migrateDividendCommandStats(ctx, statsDocs, systemDocs); err != nil { - t.Fatalf("migrateDividendCommandStats: %v", err) - } - assertUsageEntry(t, statsDocs, usageKey("stock_cash_dividend", 7), 12, "alice") - marker, _, err := systemDocs.Get(ctx, markerKey) - if err != nil || marker.Status != migrationStatusCompleted { - t.Fatalf("row marker = %+v, %v", marker, err) - } -} - -type faultDocStore[T any] struct { - storage.DocStore[T] - fail func(op, id string, value T) error -} - -func (s *faultDocStore[T]) Put(ctx context.Context, id string, value T) error { - if s.fail != nil { - if err := s.fail("put", id, value); err != nil { - return err - } - } - return s.DocStore.Put(ctx, id, value) -} - -func (s *faultDocStore[T]) Delete(ctx context.Context, id string) error { - var zero T - if s.fail != nil { - if err := s.fail("delete", id, zero); err != nil { - return err - } - } - return s.DocStore.Delete(ctx, id) -} - -func TestDividendStatsMigrationRetriesEveryWriteBoundaryWithoutDuplication(t *testing.T) { - for _, boundary := range []string{"prepare", "target", "delete", "complete"} { - t.Run(boundary, func(t *testing.T) { - ctx := context.Background() - provider := storage.NewMemoryProvider() - baseStats := storage.Typed[usageEntry](provider.Collection("stats")) - baseSystem := storage.Typed[systemstate.Record](provider.Collection(systemstate.CollectionName)) - seedUsageEntries(t, baseStats, map[string]usageEntry{ - usageKey("stock_dividend", 7): {Cmd: "stock_dividend", UserID: 7, Username: "alice", N: 5}, - usageKey("stock_cash_dividend", 7): {Cmd: "stock_cash_dividend", UserID: 7, Username: "alice", N: 7}, - }) - - failed := false - statsDocs := &faultDocStore[usageEntry]{DocStore: baseStats} - systemDocs := &faultDocStore[systemstate.Record]{DocStore: baseSystem} - forced := errors.New("forced boundary failure") - statsDocs.fail = func(op, id string, _ usageEntry) error { - if failed { - return nil - } - if boundary == "target" && op == "put" && id == usageKey("stock_cash_dividend", 7) || - boundary == "delete" && op == "delete" && id == usageKey("stock_dividend", 7) { - failed = true - return forced - } - return nil - } - systemDocs.fail = func(op, id string, record systemstate.Record) error { - if failed || op != "put" || id != dividendStatsRowMarkerKey("stock_dividend", 7) { - return nil - } - if boundary == "prepare" && record.Status == migrationStatusPrepared || - boundary == "complete" && record.Status == migrationStatusCompleted { - failed = true - return forced - } - return nil - } - - if err := migrateDividendCommandStats(ctx, statsDocs, systemDocs); err == nil { - t.Fatal("migration unexpectedly succeeded before injected failure") - } - statsDocs.fail = nil - systemDocs.fail = nil - if err := migrateDividendCommandStats(ctx, statsDocs, systemDocs); err != nil { - t.Fatalf("retry migration: %v", err) - } - assertUsageEntry(t, baseStats, usageKey("stock_cash_dividend", 7), 12, "alice") - if _, _, err := baseStats.Get(ctx, usageKey("stock_dividend", 7)); !errors.Is(err, storage.ErrNotFound) { - t.Fatalf("source remains after retry: %v", err) - } - }) - } -} - -func seedUsageEntries(t *testing.T, docs storage.DocStore[usageEntry], entries map[string]usageEntry) { - t.Helper() - for key, entry := range entries { - if err := docs.Put(context.Background(), key, entry); err != nil { - t.Fatalf("seed %s: %v", key, err) - } - } -} - -func assertUsageEntry(t *testing.T, docs storage.DocStore[usageEntry], key string, wantN int64, wantUsername string) { - t.Helper() - entry, _, err := docs.Get(context.Background(), key) - if err != nil { - t.Fatalf("get %s: %v", key, err) - } - if entry.N != wantN || entry.Username != wantUsername || entry.Deleted { - t.Fatalf("entry %s = %+v, want n=%d username=%q deleted=false", key, entry, wantN, wantUsername) - } -} diff --git a/internal/modules/stock/portfolio.go b/internal/modules/stock/portfolio.go index 71844a1..1c16399 100644 --- a/internal/modules/stock/portfolio.go +++ b/internal/modules/stock/portfolio.go @@ -1,21 +1,19 @@ package stock import ( - "bytes" "context" - "encoding/json" "errors" "fmt" "math" "strconv" - "go.mongodb.org/mongo-driver/v2/bson" - "github.com/tiennm99/miti99bot/internal/storage" ) type Store = storage.DocStore[Portfolio] +const CollectionName = "stock" + // AssetPosition keeps the complete persisted state for one stock ticker. // Base is total remaining VND cost, not average price. DividendCheckedAt is // the cursor for future dividend-event discovery. @@ -23,38 +21,6 @@ type AssetPosition struct { Quantity int64 `json:"quantity" bson:"quantity"` Base float64 `json:"base" bson:"base"` DividendCheckedAt int64 `json:"dividendCheckedAt" bson:"dividendCheckedAt"` - legacyQuantity bool -} - -func (p *AssetPosition) UnmarshalJSON(data []byte) error { - data = bytes.TrimSpace(data) - if len(data) > 0 && data[0] == '{' { - type plain AssetPosition - return json.Unmarshal(data, (*plain)(p)) - } - var quantity int64 - if err := json.Unmarshal(data, &quantity); err != nil { - return fmt.Errorf("stock: decode legacy asset quantity: %w", err) - } - *p = AssetPosition{Quantity: quantity, legacyQuantity: true} - return nil -} - -func (p *AssetPosition) UnmarshalBSONValue(valueType byte, data []byte) error { - raw := bson.RawValue{Type: bson.Type(valueType), Value: data} - if raw.Type == bson.TypeEmbeddedDocument { - type plain AssetPosition - return raw.Unmarshal((*plain)(p)) - } - if quantity, ok := raw.Int64OK(); ok { - *p = AssetPosition{Quantity: quantity, legacyQuantity: true} - return nil - } - if quantity, ok := raw.Int32OK(); ok { - *p = AssetPosition{Quantity: int64(quantity), legacyQuantity: true} - return nil - } - return fmt.Errorf("stock: unsupported legacy asset BSON type %s", raw.Type) } type Portfolio struct { diff --git a/internal/modules/stock/startup.go b/internal/modules/stock/startup.go deleted file mode 100644 index 4d4c705..0000000 --- a/internal/modules/stock/startup.go +++ /dev/null @@ -1,163 +0,0 @@ -package stock - -import ( - "context" - "errors" - "fmt" - "math" - "time" - - "github.com/tiennm99/miti99bot/internal/log" - "github.com/tiennm99/miti99bot/internal/storage" - "github.com/tiennm99/miti99bot/internal/systemstate" -) - -const ( - CollectionName = "stock" - assetSchemaMarkerKey = "migration:stock-asset-schema-v2" - tickerMigrationRetries = 5 -) - -type legacyPortfolio struct { - VND float64 `json:"vnd,omitempty" bson:"vnd,omitempty"` - Currency map[string]float64 `json:"currency,omitempty" bson:"currency,omitempty"` - Assets map[string]AssetPosition `json:"assets" bson:"assets"` - CostBasis map[string]float64 `json:"costBasis,omitempty" bson:"costBasis,omitempty"` - Meta PortfolioMeta `json:"meta" bson:"meta"` -} - -// InitStore migrates the old currency/assets/costBasis shape into the nested -// asset schema before handlers can load portfolios. It remains an every-boot -// invariant scan; the system marker is audit history, not a bypass. -func InitStore(ctx context.Context, portfolioColl, systemColl storage.Collection) error { - docs := storage.Typed[legacyPortfolio](portfolioColl) - system := systemstate.New(systemColl) - marker, markerExists, err := system.Get(ctx, assetSchemaMarkerKey) - if err != nil { - return fmt.Errorf("stock asset schema migration: read marker: %w", err) - } - keys, err := docs.List(ctx, "user:") - if err != nil { - return fmt.Errorf("stock asset schema migration: list portfolios: %w", err) - } - var migrated int64 - for index, key := range keys { - changed, err := migrateAssetSchema(ctx, docs, key, time.Now().UnixMilli()) - if err != nil { - return err - } - if changed { - migrated++ - log.Info("stock asset schema migrated", "portfolio", index+1, "total", len(keys)) - } - } - now := time.Now().UnixMilli() - if markerExists && marker.Status == "completed" && migrated == 0 { - return nil - } - if !markerExists { - marker = systemstate.Record{Kind: "migration", Name: "stock asset schema v2", CompletedAt: now} - } - marker.Status = "completed" - marker.Count += migrated - marker.UpdatedAt = now - if err := system.Put(ctx, assetSchemaMarkerKey, marker); err != nil { - return fmt.Errorf("stock asset schema migration: write marker: %w", err) - } - return nil -} - -func migrateAssetSchema(ctx context.Context, docs storage.DocStore[legacyPortfolio], key string, now int64) (bool, error) { - for attempt := 0; attempt < tickerMigrationRetries; attempt++ { - doc, version, err := docs.Get(ctx, key) - if err != nil { - return false, fmt.Errorf("stock asset schema migration: read %s: %w", key, err) - } - changed, err := doc.migrate(now) - if err != nil { - return false, fmt.Errorf("stock asset schema migration: %s: %w", key, err) - } - if !changed { - return false, nil - } - if err := docs.PutVersioned(ctx, key, version, doc); err == nil { - return true, nil - } else if !errors.Is(err, storage.ErrConflict) { - return false, fmt.Errorf("stock asset schema migration: write %s: %w", key, err) - } - } - return false, fmt.Errorf("stock asset schema migration: write %s: %w", key, storage.ErrConflict) -} - -func (p *legacyPortfolio) migrate(now int64) (bool, error) { - if !p.hasLegacyFields() { - return false, p.currentPortfolio().Validate() - } - if err := p.migrateVND(); err != nil { - return false, err - } - if err := p.migrateAssets(now); err != nil { - return false, err - } - p.Currency = nil - p.CostBasis = nil - return true, p.currentPortfolio().Validate() -} - -func (p *legacyPortfolio) hasLegacyFields() bool { - if p.Currency != nil || p.CostBasis != nil { - return true - } - for _, position := range p.Assets { - if position.legacyQuantity { - return true - } - } - return false -} - -func (p *legacyPortfolio) migrateVND() error { - if math.IsNaN(p.VND) || math.IsInf(p.VND, 0) || p.VND < 0 { - return fmt.Errorf("invalid VND balance") - } - for currency, balance := range p.Currency { - if currency != "VND" && balance != 0 { - return fmt.Errorf("unsupported legacy currency %q", currency) - } - } - if legacyVND := p.Currency["VND"]; legacyVND != 0 { - if p.VND != 0 && p.VND != legacyVND { - return fmt.Errorf("conflicting VND balances") - } - p.VND = legacyVND - } - return nil -} - -func (p *legacyPortfolio) migrateAssets(now int64) error { - for symbol, position := range p.Assets { - if !position.legacyQuantity { - return fmt.Errorf("document mixes legacy and nested assets") - } - if position.Quantity == 0 { - delete(p.Assets, symbol) - continue - } - canonical, err := normalizeStockSymbol(symbol) - base := p.CostBasis[symbol] - if err != nil || canonical != symbol || position.Quantity < 0 || !isPositiveFiniteCost(base) { - return fmt.Errorf("invalid legacy position %q", symbol) - } - p.Assets[symbol] = AssetPosition{Quantity: position.Quantity, Base: base, DividendCheckedAt: now} - } - for symbol, base := range p.CostBasis { - if !isPositiveFiniteCost(base) || p.Assets[symbol].Quantity <= 0 { - return fmt.Errorf("orphan or invalid legacy basis %q", symbol) - } - } - return nil -} - -func (p *legacyPortfolio) currentPortfolio() Portfolio { - return Portfolio{VND: p.VND, Assets: p.Assets, Meta: p.Meta} -} diff --git a/internal/modules/stock/startup_mongo_test.go b/internal/modules/stock/startup_mongo_test.go deleted file mode 100644 index 78bbd47..0000000 --- a/internal/modules/stock/startup_mongo_test.go +++ /dev/null @@ -1,71 +0,0 @@ -package stock - -import ( - "context" - "fmt" - "os" - "testing" - "time" - - "go.mongodb.org/mongo-driver/v2/bson" - - "github.com/tiennm99/miti99bot/internal/storage" - "github.com/tiennm99/miti99bot/internal/systemstate" - "github.com/tiennm99/miti99bot/internal/testutil/mongotest" -) - -var mongoTests mongotest.Manager - -func TestMain(m *testing.M) { - os.Exit(mongoTests.Run(m)) -} - -func TestInitStoreMigratesStockBasisInMongoDB(t *testing.T) { - ctx, portfolioColl, systemColl := setupMongoStockTest(t) - if err := storage.Typed[oldStockPortfolio](portfolioColl).Put(ctx, "user:7", oldStockPortfolio{ - Currency: map[string]float64{"VND": 500_000}, Assets: map[string]int64{"TCB": 100}, - CostBasis: map[string]float64{"TCB": 3_000_000}, Meta: PortfolioMeta{CreatedAt: 1}, - }); err != nil { - t.Fatal(err) - } - if err := InitStore(ctx, portfolioColl, systemColl); err != nil { - t.Fatalf("InitStore: %v", err) - } - if err := InitStore(ctx, portfolioColl, systemColl); err != nil { - t.Fatalf("InitStore second run: %v", err) - } - got, _, err := storage.Typed[Portfolio](portfolioColl).Get(ctx, "user:7") - if err != nil || got.VND != 500_000 || got.Assets["TCB"].Quantity != 100 || got.Assets["TCB"].Base != 3_000_000 { - t.Fatalf("portfolio=%+v err=%v", got, err) - } - rawColl, _ := storage.MongoCollection(portfolioColl) - var raw bson.M - if err := rawColl.FindOne(ctx, bson.M{"_id": "user:7"}).Decode(&raw); err != nil { - t.Fatal(err) - } - for _, legacyField := range []string{"currency", "costBasis", "tickers"} { - if _, exists := raw[legacyField]; exists { - t.Fatalf("legacy field %q remains in %#v", legacyField, raw) - } - } -} - -func setupMongoStockTest(t *testing.T) (context.Context, storage.Collection, storage.Collection) { - t.Helper() - uri := mongoTests.URI(t) - ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) - t.Cleanup(cancel) - client, err := storage.NewMongoClient(ctx, uri) - if err != nil { - t.Fatal(err) - } - db := client.Database(fmt.Sprintf("miti99bot_stock_test_%d", time.Now().UnixNano())) - t.Cleanup(func() { - cleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), 10*time.Second) - defer cleanupCancel() - _ = db.Drop(cleanupCtx) - _ = client.Disconnect(cleanupCtx) - }) - provider := storage.NewMongoProvider(db) - return ctx, provider.Collection(CollectionName), provider.Collection(systemstate.CollectionName) -} diff --git a/internal/modules/stock/startup_test.go b/internal/modules/stock/startup_test.go deleted file mode 100644 index bf8770d..0000000 --- a/internal/modules/stock/startup_test.go +++ /dev/null @@ -1,95 +0,0 @@ -package stock - -import ( - "context" - "testing" - - "github.com/tiennm99/miti99bot/internal/storage" - "github.com/tiennm99/miti99bot/internal/systemstate" -) - -type oldStockPortfolio struct { - Currency map[string]float64 `json:"currency" bson:"currency"` - Assets map[string]int64 `json:"assets" bson:"assets"` - CostBasis map[string]float64 `json:"costBasis" bson:"costBasis"` - Meta PortfolioMeta `json:"meta" bson:"meta"` -} - -func TestInitStoreMigratesStockNestedAssetsAndVND(t *testing.T) { - ctx := context.Background() - provider := storage.NewMemoryProvider() - portfolioColl := provider.Collection(CollectionName) - systemColl := provider.Collection(systemstate.CollectionName) - oldDocs := storage.Typed[oldStockPortfolio](portfolioColl) - if err := oldDocs.Put(ctx, "user:7", oldStockPortfolio{ - Currency: map[string]float64{"VND": 2_000_000}, - Assets: map[string]int64{"TCB": 100}, CostBasis: map[string]float64{"TCB": 3_000_000}, - Meta: PortfolioMeta{CreatedAt: 1}, - }); err != nil { - t.Fatal(err) - } - if err := InitStore(ctx, portfolioColl, systemColl); err != nil { - t.Fatalf("InitStore: %v", err) - } - got, err := LoadPortfolio(ctx, storage.Typed[Portfolio](portfolioColl), 7, 9) - if err != nil { - t.Fatal(err) - } - position := got.Assets["TCB"] - if got.VND != 2_000_000 || position.Quantity != 100 || position.Base != 3_000_000 || position.DividendCheckedAt <= 0 { - t.Fatalf("portfolio=%+v", got) - } - if err := InitStore(ctx, portfolioColl, systemColl); err != nil { - t.Fatalf("second InitStore: %v", err) - } - marker, exists, err := systemstate.New(systemColl).Get(ctx, assetSchemaMarkerKey) - if err != nil || !exists || marker.Status != "completed" || marker.Count != 1 { - t.Fatalf("marker=%+v exists=%v err=%v", marker, exists, err) - } -} - -func TestStockMigrationRejectsMissingBasis(t *testing.T) { - ctx := context.Background() - provider := storage.NewMemoryProvider() - coll := provider.Collection(CollectionName) - if err := storage.Typed[oldStockPortfolio](coll).Put(ctx, "user:7", oldStockPortfolio{ - Currency: map[string]float64{"VND": 1}, Assets: map[string]int64{"TCB": 100}, CostBasis: map[string]float64{}, - }); err != nil { - t.Fatal(err) - } - if err := InitStore(ctx, coll, provider.Collection(systemstate.CollectionName)); err == nil { - t.Fatal("InitStore accepted legacy holding without basis") - } -} - -type conflictOnceSchemaStore struct { - storage.DocStore[legacyPortfolio] - conflicted bool -} - -func (s *conflictOnceSchemaStore) PutVersioned(ctx context.Context, key string, version int64, value legacyPortfolio) error { - if !s.conflicted { - s.conflicted = true - return storage.ErrConflict - } - return s.DocStore.PutVersioned(ctx, key, version, value) -} - -func TestStockSchemaMigrationRetriesConflict(t *testing.T) { - ctx := context.Background() - provider := storage.NewMemoryProvider() - coll := provider.Collection(CollectionName) - if err := storage.Typed[oldStockPortfolio](coll).Put(ctx, "user:7", oldStockPortfolio{ - Currency: map[string]float64{"VND": 1}, - Assets: map[string]int64{"TCB": 10}, - CostBasis: map[string]float64{"TCB": 300_000}, - }); err != nil { - t.Fatal(err) - } - docs := storage.Typed[legacyPortfolio](coll) - store := &conflictOnceSchemaStore{DocStore: docs} - changed, err := migrateAssetSchema(ctx, store, "user:7", 123) - if err != nil || !changed || !store.conflicted { - t.Fatalf("changed=%v conflicted=%v err=%v", changed, store.conflicted, err) - } -} diff --git a/plans/reports/pm-260721-1705-portfolio-cleanup.md b/plans/reports/pm-260721-1705-portfolio-cleanup.md new file mode 100644 index 0000000..7b4ae55 --- /dev/null +++ b/plans/reports/pm-260721-1705-portfolio-cleanup.md @@ -0,0 +1,48 @@ +# Portfolio Cleanup Completion Report + +## Summary + +| Item | Result | +|---|---| +| Coin dividend cursor | Removed from schema, API, validation, callers, tests | +| Stock dividend cursor | Retained unchanged | +| Portfolio migrations | Completed runtime paths and migration-only tests removed | +| Stats rename migration | Removed; recurring indexes retained | +| LoL startup maintenance | TTL index retained | +| Historical system data | Untouched; reusable helper retained | +| Dividend API research | SSI iBoard recommended behind provider interface | + +## Verification + +- Focused and full Go tests passed. +- MongoDB 8 stats-index and LoL TTL-index tests executed and passed. +- `go vet ./...`, `go build ./...`, and `golangci-lint run` passed. +- `git diff --check` passed. +- Independent tester, debugger, and reviewer reported no defects. +- Coin BSON regression coverage proves a stale cursor loads safely and is + omitted by the next whole-document encoding/write. + +## Documentation + +- README and deployment guide now distinguish stock and coin asset schemas. +- Standalone dividend API research report added under `plans/reports/`. +- Existing completed implementation plans remain historical records; no phase + statuses changed. + +## Known Limitations + +- Untouched MongoDB coin documents retain the ignored cursor until their next + portfolio write. +- SSI iBoard corporate actions are undocumented and have no published SLA. + +## Next Steps + +1. Commit the approved cleanup when requested. +2. Design the Telegram dividend-event selection/confirmation interaction. +3. Implement SSI behind a replaceable provider interface after command design + approval. + +## Unresolved Questions + +- Should dividend lookup inspect one ticker per command or all stock assets? +- Should an event only prefill guidance or execute after explicit confirmation?