Commit Graph
8 Commits
Author SHA1 Message Date
Alex 8aa0facf4a feat(wiki): wiki_pages table + repository + targeted chunk delete
Migration 0023_wiki_pages (source-scoped; version/content_hash/embed_status; FK source_id->sources ON DELETE CASCADE; UNIQUE(source_id,path) + prefix index). WikiPagesRepository: path-keyed CRUD with content-hash short-circuit, version bump, and move-with-reject. BaseVectorStore.delete_chunks_by_source_path (loop default) + parameterized pgvector override (DELETE WHERE source_id=%s AND metadata->>'source'=%s). Unit 1 of F-Wiki.
2026-06-22 17:40:41 +01:00
Alex a709b7ccdc feat: semantic chunking + pgvector hybrid (BM25+vector) retriever (#2553) 2026-06-22 17:04:13 +01:00
Alex f6400cd736 feat: per-source RAG configuration (retrieval strategies, chunking, exposure, prescreen)
Introduces a per-source config contract that makes RAG behavior strategy-dispatched instead of a single hardcoded path. Every source gains a validated JSONB config; an empty/absent config reproduces current behavior byte-for-byte, and the whole path is gated by PER_SOURCE_RETRIEVAL_ENABLED.

Foundation: sources.config JSONB column + migration 0022_source_config; SourceConfig/ChunkingConfig/RetrievalConfig pydantic models (strict on write, lenient on read); ChunkerCreator and RetrieverCreator.register registries; config threaded through the upload routes, ingest/remote/connector workers, and reingest.

Retrieval: a Dispatcher groups sources by retriever key (all-classic collapses to today's single ClassicRAG under one shared token budget; non-classic retrievers get their own instance), removing the previous single-global-retriever collapse in stream_processor. Per-source chunks, score_threshold (honored for pgvector/mongodb, safely ignored elsewhere), and rephrase_query toggle. New PATCH /api/sources/<id>/config with team-aware (effective_write_owner) authz and a requires_reingest signal.

Chunking strategies: recursive, markdown, parent_child (selectable per source; re-ingest to apply). Search exposure: per-source prefetch vs agentic_tool for agentic/research agents. Map-reduce prescreen: optional LLM relevance pre-filter implemented as a composable post-retrieval stage that wraps any retriever.

Backend and frontend (shared Retrieval options panel + edit modal) with tests; backend suite and frontend vitest green. Excludes the wiki and GraphRAG flagships.
2026-06-20 21:54:23 +01:00
Alex e692c645b9 fix: pgvec 2026-05-05 01:55:23 +01:00
Alex a9761061fc fix: mini issues 2026-04-12 12:24:58 +01:00
Alex ececcb8b17 feat: init pg migration 2026-04-12 00:07:24 +01:00
Alex aacf281222 fix: improve remote embeds (#2193) 2025-12-16 13:59:17 +02:00
Alex b1d8266eef feat: implement PGVectorStore for PostgreSQL vector storage 2025-08-05 13:54:39 +01:00