From 949bb01427f0c6a68d4c92e014bbe30b8ad90ec6 Mon Sep 17 00:00:00 2001 From: tiennm99 Date: Thu, 25 Jun 2026 17:17:08 +0700 Subject: [PATCH] docs(stock): add price provider research --- ...stable-vn-stock-price-provider-research.md | 292 ++++++++++++++++++ ...1704-kbs-vs-vci-price-provider-research.md | 164 ++++++++++ 2 files changed, 456 insertions(+) create mode 100644 plans/reports/260625-0936-stable-vn-stock-price-provider-research.md create mode 100644 plans/reports/260625-1704-kbs-vs-vci-price-provider-research.md diff --git a/plans/reports/260625-0936-stable-vn-stock-price-provider-research.md b/plans/reports/260625-0936-stable-vn-stock-price-provider-research.md new file mode 100644 index 0000000..0d1c58b --- /dev/null +++ b/plans/reports/260625-0936-stable-vn-stock-price-provider-research.md @@ -0,0 +1,292 @@ +--- +type: research-report +topic: stable-vn-stock-price-provider +created: 2026-06-25 09:36 UTC +status: done +--- + +# Research Report: Stable Free VN Stock Price Provider + +## Executive Summary + +Best fit: **EODHD free plan as primary EOD provider**, with DynamoDB daily cache. It is the only provider found with explicit Vietnam exchange support (`VN` / MIC `XSTC`) and a real free API-key plan. It is not real-time on free tier, but this bot is paper trading, so previous/official-ish EOD close is acceptable if rendered as EOD/last close. + +Do not depend on SSI/KBS broker-app endpoints as primary. Live evidence from Lambda: SSI returns Cloudflare security pages, KBS times out from AWS egress. Yahoo works today, but it is not a stable contracted API for VN stocks. Keep Yahoo/SSI/KBS only as emergency fallback. + +If user wants intraday/current price with a stable contract, no truly free option found. The free stable path is EOD only. Paid floor appears around $19.99/mo for EODHD all-world EOD; intraday/live is higher. + +## Methodology + +- Date: 2026-06-25. +- Scope: Vietnam listed equities for `stock` module: `/stock_stats`, `/stock_buy`, `/stock_sell`. +- Criteria: explicit VN coverage, free API key, documented REST API, low-volume Lambda-safe, legal/terms risk, implementation effort. +- Sources consulted: EODHD, Marketstack, Twelve Data, FMP, Alpha Vantage, existing production probes. +- Direct probes: + - Twelve Data `/stocks?country=Vietnam`, `/stocks?country=Viet%20Nam`, `/stocks?exchange=HOSE` returned empty arrays. + - Twelve Data `/stocks?symbol=FPT` resolved Thailand `FPT`, not Vietnam FPT Corp. + - EODHD demo token returned `403 Forbidden` for `exchange-symbol-list/VN`; expected, demo token not real free key. + - Marketstack without key returned `missing_access_key`; expected. + +## Key Findings + +### 1. EODHD + +EODHD explicitly lists **Vietnam Stocks - VN**, country Vietnam, MIC `XSTC`, timezone `Asia/Ho_Chi_Minh`, and 593 active tickers. Example listed format is `AAA.VN`, `ACB.VN`, etc. Source: https://eodhd.com/exchange/VN + +Free plan: +- $0/mo. +- 20 calls/day. +- EOD data only, past-year depth. +- Personal use. +- Requires registration/API key. + +Docs/pricing sources: +- https://eodhd.com/financial-apis/api-for-historical-data-and-volumes +- https://eodhd.com/pricing +- https://eodhd.com/list-of-stock-markets + +Pros: +- Explicit VN coverage. +- Documented REST API. +- Free key allowed. +- Enough for bot if we cache daily and request only held/watchlist symbols. +- Symbol format simple: `FPT.VN`, `TCB.VN`, etc. + +Cons: +- Free quota low: 20/day. +- EOD, not live current price. +- Terms disclaim data may not be real-time/accurate and is indicative. +- Commercial/display use may require paid/commercial agreement. + +Verdict: **Primary recommendation**. + +### 2. Marketstack + +Marketstack has a documented free plan and stable API infra, but no confirmed Vietnam symbol coverage from public unauthenticated probe. It advertises global EOD data, stock tickers info, and exchange info. + +Free plan: +- 100 requests/month. +- EOD data and up to 12 months history. +- API key required. + +Sources: +- https://marketstack.com/ +- https://marketstack.com/pricing +- https://docs.apilayer.com/marketstack/docs/api-endpoints-v1 + +Pros: +- Documented. +- Stable SaaS provider. +- Supports multi-symbol EOD requests via `symbols` parameter according to docs/search result. + +Cons: +- Free quota worse than EODHD for daily bot use unless multi-symbol request covers all holdings. +- VN coverage not verified without real key. +- Intraday/real-time mostly paid/US-oriented. + +Verdict: **Candidate backup only after registering a free key and proving `FPT`, `TCB`, `HPG`, `MSN`, `MWG`, `SSI`, `VND` coverage**. + +### 3. Twelve Data + +Twelve Data has good free quota reputation and strong docs, but public reference data did not show Vietnam equities in probes. + +Source: +- https://support.twelvedata.com/en/articles/5620513-how-to-find-all-available-symbols-at-twelve-data +- https://twelvedata.com/ + +Probe results: +- `https://api.twelvedata.com/stocks?country=Vietnam` -> `[]` +- `https://api.twelvedata.com/stocks?country=Viet%20Nam` -> `[]` +- `https://api.twelvedata.com/stocks?exchange=HOSE` -> `[]` +- `https://api.twelvedata.com/stocks?symbol=FPT` -> Thailand `FPT`, not Vietnam. + +Verdict: **Reject for now** unless support confirms VN coverage on a plan/add-on. + +### 4. Financial Modeling Prep + +FMP free plan is attractive on paper: 250 calls/day and EOD/reference data. But the public docs/pricing suggest global coverage only at higher tiers, and demo API key did not allow exchange/symbol coverage validation. + +Sources: +- https://site.financialmodelingprep.com/developer/docs +- https://site.financialmodelingprep.com/pricing-plans + +Pros: +- 250 calls/day free. +- Strong docs/API shape. + +Cons: +- Vietnam coverage not verified. +- Pricing page implies Basic/free has limited symbols and EOD only; global coverage appears in Ultimate paid tier. +- Display/redistribution needs licensing agreement. + +Verdict: **Reject until a real free key confirms VN symbols**. + +### 5. Alpha Vantage + +Alpha Vantage is reputable and free-key based, but no evidence found for Vietnam exchange coverage. It is more useful for US/global large markets than VN equities. + +Source: +- https://www.alphavantage.co/ + +Verdict: **Reject for VN stocks**. + +### 6. Broker/App Internal Endpoints + +Current/existing providers: +- SSI direct quote: `iboard-query.ssi.com.vn`. +- SSI chart history: `iboard-api.ssi.com.vn`. +- KBS data-day: `kbbuddywts.kbsec.com.vn`. +- Yahoo chart: `query1.finance.yahoo.com`. + +Findings: +- SSI now returns 403 Cloudflare security page from this workspace and Lambda. +- KBS returns locally but times out from Lambda. +- Yahoo works from Lambda today, but is not a contracted API. + +Verdict: **fallback only, never primary**. + +## Comparative Analysis + +| Provider | VN coverage proven | Free key | Free quota | Current price | Stable API | Fit | +|---|---:|---:|---:|---:|---:|---| +| EODHD | yes | yes | 20/day | EOD only | yes | best | +| Marketstack | unknown | yes | 100/month | EOD free | yes | backup candidate | +| Twelve Data | no in probes | yes | likely generous | unknown | yes | reject | +| FMP | unknown | yes | 250/day | EOD free | yes | reject until proven | +| Alpha Vantage | no evidence | yes | free | unknown | yes | reject | +| SSI/KBS/Yahoo | yes-ish | no | unlimited-ish | yes | no | fallback only | + +## Implementation Recommendation + +### Provider Order + +```text +EODHD EOD cache -> Yahoo emergency -> SSI emergency -> KBS emergency -> no price +``` + +Do not put Yahoo before EODHD once EODHD key exists. Yahoo is useful operationally but not stable. + +### Data Model + +```go +type StockQuote struct { + Symbol string + PriceVND float64 + Source string // eodhd, yahoo, ssi, kbs + AsOfDate string // YYYY-MM-DD + RetrievedAt int64 + Stale bool +} +``` + +Cache keys: + +```text +stock-price:FPT:2026-06-25 +stock-price:TCB:2026-06-25 +``` + +### API Shape + +EODHD EOD endpoint: + +```text +GET https://eodhd.com/api/eod/FPT.VN?api_token=$EODHD_API_KEY&fmt=json&period=d&from=YYYY-MM-DD&to=YYYY-MM-DD +``` + +Use latest returned row: + +```json +{ + "date": "2026-06-25", + "open": 71000, + "high": 71700, + "low": 70800, + "close": 71000, + "adjusted_close": 71000, + "volume": 6592100 +} +``` + +For portfolio valuation, use `close`. For paper buy/sell, either: +- Use same `close` and label trade as "last close", or +- Keep Yahoo as live-ish fallback only for trade commands. + +### Quota Strategy + +Free EODHD quota is 20/day. Therefore: + +1. Cache per symbol per trading date. +2. Fetch only missing cache entries. +3. For `/stock_stats`, batch unique held symbols but call EODHD sequentially under a short timeout. +4. Do not refresh more than once/day per symbol. +5. If quota exceeded, use stale cache and label `(stale YYYY-MM-DD)`. + +With current portfolio of 7 symbols, daily refresh costs 7 calls/day. Fits. + +### Env Config + +Add: + +```text +STOCK_EODHD_API_KEY_PARAMETER_NAME=/miti99bot/prod/eodhd-api-key +STOCK_EODHD_API_URL=https://eodhd.com/api +``` + +Store key in SSM SecureString, same pattern as Telegram/Gemini secrets. + +### User-Facing Copy + +When using EODHD: + +```text +FPT x2300 @ 71.000 VND (EOD 2026-06-25) = ... +``` + +When fallback/stale: + +```text +FPT x2300 @ 71.000 VND (stale EOD 2026-06-24) = ... +``` + +## Common Pitfalls + +- Calling EODHD on every `/stock_stats`: burns quota fast. +- Treating EOD price as live market price without label. +- Assuming Marketstack/Twelve/FMP support Vietnam because marketing says "global". +- Keeping broker-app endpoints as primary. +- No stale cache. The bot should degrade to cached prices, not zero portfolio value. + +## Recommended Next Steps + +1. Register EODHD free key. +2. Manually verify these URLs with real key: + - `FPT.VN` + - `HPG.VN` + - `MSN.VN` + - `MWG.VN` + - `SSI.VN` + - `TCB.VN` + - `VND.VN` +3. If all seven work, implement `EODHDPriceProvider` + daily cache. +4. Change `/stock_stats` labels to show source/date. +5. Keep Yahoo/SSI/KBS emergency fallbacks after EODHD/cache. + +## Resources + +- EODHD Vietnam exchange: https://eodhd.com/exchange/VN +- EODHD supported exchanges: https://eodhd.com/list-of-stock-markets +- EODHD EOD API: https://eodhd.com/financial-apis/api-for-historical-data-and-volumes +- EODHD pricing: https://eodhd.com/pricing +- Marketstack pricing: https://marketstack.com/pricing +- Marketstack docs: https://docs.apilayer.com/marketstack/docs/api-endpoints-v1 +- Twelve Data symbol reference: https://support.twelvedata.com/en/articles/5620513-how-to-find-all-available-symbols-at-twelve-data +- FMP docs: https://site.financialmodelingprep.com/developer/docs +- FMP pricing: https://site.financialmodelingprep.com/pricing-plans +- Alpha Vantage: https://www.alphavantage.co/ + +## Unresolved Questions + +- Does EODHD free key return all current holdings (`FPT.VN`, `HPG.VN`, `MSN.VN`, `MWG.VN`, `SSI.VN`, `TCB.VN`, `VND.VN`) via API, not just website pages? +- Is "last close" acceptable for `/stock_buy` and `/stock_sell`, or should those commands keep live-ish Yahoo fallback? +- Is bot use personal/non-commercial under EODHD terms, or does Telegram display to group chats require paid/commercial terms? diff --git a/plans/reports/260625-1704-kbs-vs-vci-price-provider-research.md b/plans/reports/260625-1704-kbs-vs-vci-price-provider-research.md new file mode 100644 index 0000000..4373a29 --- /dev/null +++ b/plans/reports/260625-1704-kbs-vs-vci-price-provider-research.md @@ -0,0 +1,164 @@ +--- +type: research-report +topic: kbs-vs-vci-price-provider +created: 2026-06-25 17:04 UTC +status: done +--- + +# Research Report: KBS vs VCI Price Provider + +## Executive Summary + +For this bot, if we only need **current price**, the comparison is **KBS `/stock/iss` vs VCI `/price/symbols/getList`**. Both return all needed symbols in one batch request and returned identical prices in live probe. + +**Best practical answer: test KBS price-board first, then VCI fallback.** KBS current-price response is simpler and is vnstock's default for Vietnam market quote. Our previous Lambda failure was against KBS `data_day` history endpoint with weaker headers, not the KBS current price-board endpoint. Therefore KBS current quote is still worth a Lambda probe before replacing it. + +Do not treat either as stable. Both are unofficial web/app endpoints, not contracted API-key services. Both rejected raw curl fingerprints during probe: KBS returned `400 Request Blocked`, VCI returned `403 Error Page`. With browser-like headers, both worked. That means both can break by WAF/fingerprint changes. + +Best stable architecture remains: **EODHD EOD/cache primary when key exists -> KBS current quote after Lambda proof -> VCI current quote fallback -> SSI direct fallback -> no price**. + +## Research Methodology + +- Timestamp: 2026-06-25 17:04 UTC. +- Scope: VN equity **current price only** for `miti99bot` stock module on AWS Lambda. +- Sources consulted: + - `vnstock` source at commit `5bf05e3e494b1d109750c143e964945ed7be3f7d`. + - Live endpoint probes from this workspace. + - Prior production evidence: KBS timed out from Lambda during `/stock_stats` probe. +- Key terms: KBS, VCI, Vietcap, vnstock quote, price board, OHLCChart, Lambda egress. + +## Key Findings + +### KBS + +vnstock default Vietnam market quote route uses KBS: + +```text +Market().quote("SSI") + -> Trading.price_board() + -> POST https://kbbuddywts.kbsec.com.vn/iis-server/investment/stock/iss + -> {"code":"SSI"} +``` + +Live probe with browser headers: + +| Probe | Result | +|---|---:| +| Current quote, 7 symbols | `200`, `0.297s` | +| Raw/default curl | `400 Request Blocked` | + +Pros: +- Simple payload. +- Flat fields: `CP` close/current price, `B1..B3` bid, `S1..S3` ask. +- vnstock uses it as default for unified market quote. + +Cons: +- KBS `data_day` history endpoint timed out from our AWS Lambda environment; `stock/iss` current quote still needs Lambda proof. +- Current Go fallback currently uses per-symbol `data_day`, not the KBS batch price-board endpoint. +- WAF/fingerprint sensitive. Needs browser-like headers. +- No official read-only API key contract. + +### VCI + +vnstock supports VCI/Vietcap current quote endpoint: + +```text +POST https://trading.vietcap.com.vn/api/price/symbols/getList +``` + +Live probe with browser headers: + +| Probe | Result | +|---|---:| +| Current quote, 7 symbols | `200`, `0.357s` | +| Raw/default curl | `403 Error Page` | + +Current quote returned same current prices as KBS: + +| Symbol | KBS | VCI | +|---|---:|---:| +| FPT | 71000 | 71000 | +| HPG | 23400 | 23400 | +| MSN | 71500 | 71500 | +| MWG | 77200 | 77200 | +| SSI | 26500 | 26500 | +| TCB | 33400 | 33400 | +| VND | 17350 | 17350 | + +Pros: +- Better fit for `/stock_stats`: one batch request for all current quotes. +- Rich structured response: `listingInfo`, `bidAsk`, `matchPrice`. +- Gives matched price, bid/ask, session, reference, floor, ceiling, volume. +- Same observed prices as KBS. + +Cons: +- 403 without browser-like headers. +- More nested parsing than KBS. +- No official read-only API key contract. +- Needs Lambda proof before promoting. + +## Comparative Analysis + +| Criteria | KBS | VCI | Winner | +|---|---|---|---| +| Current quote batch | Yes | Yes | Tie | +| Payload simplicity | Flat fields | Nested objects | KBS | +| Parser safety | Easy | Moderate | KBS | +| Lambda evidence | History endpoint timed out; price-board not tested | Not tested yet | Unknown | +| Browser/WAF sensitivity | Yes | Yes | Tie | +| vnstock default | Yes | No | KBS | +| Current price richness | Good | Better structured | VCI | +| Official/stable API | No | No | Neither | +| Best for current price only | First Lambda probe | Fallback probe | KBS | + +## Recommendation + +For current price only, use **KBS price-board first**, but only after a Lambda probe passes. Use **VCI second**. + +Recommended order: + +```text +EODHD daily cache -> KBS stock/iss current quote -> VCI current quote -> SSI direct quote -> no price +``` + +If no EODHD key yet: + +```text +KBS stock/iss -> VCI price/symbols/getList -> SSI direct quote +``` + +Implementation notes: +- Replace current KBS `data_day` fallback with KBS `stock/iss` batch endpoint for current price. +- Add VCI only as current quote fallback: `price/symbols/getList`. +- Add browser-like headers exactly; `Mozilla/5.0 (miti99bot)` is probably too bot-looking. +- Cache result briefly per command or per minute to avoid repeated WAF pressure. +- Log provider/source/date in stock output. + +## Common Pitfalls + +- Assuming vnstock's default means stable in Lambda. It does not. +- Calling KBS `data_day` per symbol when price-board can batch current price. +- Promoting either provider without Lambda proof. +- Calling either "official API". These are web/app backend endpoints. +- Using weak headers. Both rejected raw/default curl. + +## Resources & References + +- vnstock KBS constants: https://github.com/thinh-vu/vnstock/blob/5bf05e3e494b1d109750c143e964945ed7be3f7d/vnstock/explorer/kbs/const.py +- vnstock KBS price board: https://github.com/thinh-vu/vnstock/blob/5bf05e3e494b1d109750c143e964945ed7be3f7d/vnstock/explorer/kbs/trading.py +- vnstock VCI price board: https://github.com/thinh-vu/vnstock/blob/5bf05e3e494b1d109750c143e964945ed7be3f7d/vnstock/explorer/vci/trading.py +- vnstock UI routing defaults: https://github.com/thinh-vu/vnstock/blob/5bf05e3e494b1d109750c143e964945ed7be3f7d/vnstock/ui/_registry.py + +## Next Steps + +1. Add KBS `stock/iss` batch current-price provider behind env flag or provider order config. +2. Deploy to Lambda and fake-probe `/stock_stats`. +3. If KBS price-board passes Lambda, place it before VCI/SSI. +4. If KBS fails from Lambda, add VCI current-price fallback and probe. +5. Keep EODHD as stable primary when key is available. + +## Unresolved Questions + +- Does KBS `stock/iss` work from AWS Lambda `ap-southeast-1` with browser-like headers? +- Does VCI work from AWS Lambda `ap-southeast-1` with browser-like headers if KBS fails? +- Is live-ish price necessary for `/stock_buy` and `/stock_sell`, or is EOD close acceptable?