mirror of
https://github.com/tiennm99/DocsGPT.git
synced 2026-10-04 16:13:23 +00:00
Alex
f6400cd736
Introduces a per-source config contract that makes RAG behavior strategy-dispatched instead of a single hardcoded path. Every source gains a validated JSONB config; an empty/absent config reproduces current behavior byte-for-byte, and the whole path is gated by PER_SOURCE_RETRIEVAL_ENABLED. Foundation: sources.config JSONB column + migration 0022_source_config; SourceConfig/ChunkingConfig/RetrievalConfig pydantic models (strict on write, lenient on read); ChunkerCreator and RetrieverCreator.register registries; config threaded through the upload routes, ingest/remote/connector workers, and reingest. Retrieval: a Dispatcher groups sources by retriever key (all-classic collapses to today's single ClassicRAG under one shared token budget; non-classic retrievers get their own instance), removing the previous single-global-retriever collapse in stream_processor. Per-source chunks, score_threshold (honored for pgvector/mongodb, safely ignored elsewhere), and rephrase_query toggle. New PATCH /api/sources/<id>/config with team-aware (effective_write_owner) authz and a requires_reingest signal. Chunking strategies: recursive, markdown, parent_child (selectable per source; re-ingest to apply). Search exposure: per-source prefetch vs agentic_tool for agentic/research agents. Map-reduce prescreen: optional LLM relevance pre-filter implemented as a composable post-retrieval stage that wraps any retriever. Backend and frontend (shared Retrieval options panel + edit modal) with tests; backend suite and frontend vitest green. Excludes the wiki and GraphRAG flagships.
42 lines
1.1 KiB
Python
42 lines
1.1 KiB
Python
"""0022 source config — per-source behavior contract JSONB column.
|
|
|
|
Adds ``sources.config`` (JSONB, NOT NULL, default ``{}``). The column holds a
|
|
Pydantic-validated ``SourceConfig`` (chunking + retrieval knobs) separate from
|
|
the display/provenance ``metadata`` bag. The server default backfills every
|
|
existing row with ``{}``, which ``SourceConfig.parse()`` reads as classic
|
|
defaults — so existing sources behave byte-for-byte as before.
|
|
|
|
No new Postgres extensions (keeps it Neon/DBngin-portable).
|
|
|
|
Revision ID: 0022_source_config
|
|
Revises: 0021_teams
|
|
"""
|
|
|
|
from typing import Sequence, Union
|
|
|
|
import sqlalchemy as sa
|
|
from alembic import op
|
|
from sqlalchemy.dialects import postgresql
|
|
|
|
|
|
revision: str = "0022_source_config"
|
|
down_revision: Union[str, None] = "0021_teams"
|
|
branch_labels: Union[str, Sequence[str], None] = None
|
|
depends_on: Union[str, Sequence[str], None] = None
|
|
|
|
|
|
def upgrade() -> None:
|
|
op.add_column(
|
|
"sources",
|
|
sa.Column(
|
|
"config",
|
|
postgresql.JSONB,
|
|
nullable=False,
|
|
server_default=sa.text("'{}'::jsonb"),
|
|
),
|
|
)
|
|
|
|
|
|
def downgrade() -> None:
|
|
op.drop_column("sources", "config")
|