Commit Graph
4925 Commits
Author SHA1 Message Date
Pavel 5c19d972d2 Docs revamp 2026-06-26 11:56:20 +04:00
Alex 1acb836332 feat: guard on remote embedding size 2026-06-24 09:28:39 +01:00
Alex 872152e62a Merge pull request #2559 from arc53/fix-embeds-remote
fix: remote embeds
2026-06-23 21:43:48 +01:00
Alex df5582db37 fix: remote embeds 2026-06-23 21:43:11 +01:00
Alex 9c0803f017 Merge pull request #2557 from ManishMadan2882/main
minor ui, filter users intead of exact match
2026-06-23 21:32:39 +01:00
Alex f45b8df4e5 Merge pull request #2558 from arc53/graph-mini-fixes
fix: GraphRAG follow-ups (rebuild + progress streaming), Brave graph view, compression null guard
2026-06-23 20:53:35 +01:00
Manish Madan 23cd3d2e50 Merge pull request #2446 from arc53/dependabot/npm_and_yarn/extensions/react-widget/globals-17.6.0
chore(deps-dev): bump globals from 17.5.0 to 17.6.0 in /extensions/react-widget
2026-06-24 01:09:37 +05:30
ManishMadan2882 aa10146e25 (fix)ui/settings: prompts dropdown 2026-06-24 01:08:02 +05:30
Alex d7bbfcfe17 fix: minor graph rag improvements 2026-06-23 20:10:41 +01:00
Alex ae3e79bf8f fix: graph view on brave 2026-06-23 19:15:43 +01:00
Alex 6e4480b61b fix: mini compression issues 2026-06-23 19:15:22 +01:00
Alex 8c312ed311 Merge pull request #2556 from arc53/rag-graphrag
feat: GraphRAG retriever: per-source knowledge graph + PPR retrieval
2026-06-23 18:56:40 +01:00
Alex db3af415ac fix: scale content on graph rag 2026-06-23 18:50:27 +01:00
Alex ba5c9a8910 fix: graph rag ui improvements 2026-06-23 17:11:49 +01:00
ManishMadan2882 2965e1d5ac (feat:admin) filter users intead of exact match, minor ui 2026-06-23 14:48:56 +05:30
Alex 92e64cc192 fix(ui): wrap long source-card titles instead of overflowing under the menu
A long underscore-heavy filename (no natural break points) overflowed the card toward the 3-dots menu: the title <h3> was a flex item without min-w-0, so it couldn't shrink below its min-content width, and wrap-break-word doesn't reduce flex min-content. Add min-w-0 flex-1 + wrap-anywhere on the title and shrink-0 on the menu column so the title shrinks, breaks anywhere, and stays left of the menu (still line-clamp-3).
2026-06-23 08:50:51 +01:00
Alex 91d17694b3 fix(graphrag): graph view rendered blank — measure container after load
The ResizeObserver/size effect ran once on mount with [] deps while loading=true, so containerRef was null (the graph container isn't in the DOM yet) and size.width stayed 0, gating out ForceGraph2D (blank canvas) even though the data loaded. Re-run the measure effect on [loading, data.nodes.length] so it attaches + measures once the container actually mounts.
2026-06-23 08:46:03 +01:00
Alex 7e7f34ee9c test(graphrag): address code-quality nits in graph tests
Meaningful self-loop degree assertion (was always-true); single import style per module; explicit returns in the live-store helper. (CI code-quality bot.)
2026-06-23 03:48:21 +01:00
Alex 2594bb0bed feat(graphrag): graph-view endpoints + network visualization
GraphStore.get_graph_overview (top-N by degree, bounded) + get_node_detail (description + linked chunks). GET /api/sources/<id>/graph + /graph/node/<id> (read-access gated, node scoped to source_id; empty graph -> {nodes:[],edges:[]}). Frontend GraphView (react-force-graph-2d) wired to config.kind=graphrag via card-click/'View graph'; node-click -> description + chunks; tooltip renders untrusted names as text (no innerHTML XSS). All read-only GraphStore methods rollback their txn (no idle-in-transaction lock). Unit G7. (Also a pre-existing prettier fix in WorkflowPreview to keep lint green.)
2026-06-23 02:48:18 +01:00
Alex daddbeb062 chore: fix pre-existing prettier formatting in TeamSwitcher (unblock CI lint) 2026-06-23 01:57:00 +01:00
Alex 8ccc291ef5 feat(graphrag): frontend Enable GraphRAG action + status + badge
/api/config exposes graphrag_available. 'Enable GraphRAG' source action (owner/editor + pgvector + not-already-graphrag) -> POST graphrag/enable, confirm + task-poll + 'Building graph...' + summary (EnableGraphRAGModal, unmount-guarded; pure graphragEnableUtils). GraphRAG badge for config.kind==graphrag. Mirrors the wiki convert UX. Unit G6.
2026-06-23 01:57:00 +01:00
Alex 4e0c792a6d feat(graphrag): GraphRAGRetriever (PPR local) + register + ClassicRAG fallback
Query-side: composes ClassicRAG (rephrase/budget/fallback). Per source: pgvector entity-name seed -> bounded subgraph -> networkx personalized PageRank (IDF hub down-weight) -> rank graph_node_chunks -> chunk text -> shared token budget; no LLM at query time. Citations derived from chunk metadata (matches ClassicRAG). Falls back to ClassicRAG per-source when no graph rows / unavailable / error (retrieval never breaks). get_chunk_texts queries the co-located documents table by configured table/column names (parameterized). Registered 'graphrag'. Unit G5.
2026-06-23 01:45:14 +01:00
Alex 4742aec4c6 feat(graphrag): durable extract_graph task + ingest-path enqueue + enable route
graph_enabled() sets kind=graphrag + retriever=graphrag. extract_graph_worker fetches the source's pgvector chunks and runs G3 extraction (graphrag_available guard, empty no-op). extract_graph durable+idempotent task; key varies with source updated_at so re-ingest/re-enable re-run incrementally (G3 checkpoint skips done chunks) while concurrent same-state enqueues dedup. The 4 ingest paths enqueue after embed when kind=graphrag (isolated in try/except so a broker hiccup can't fail the ingest). POST /api/sources/<id>/graphrag/enable: pgvector+GRAPHRAG_ENABLED gated, owner/editor write-authz; PATCH config still can't flip kind->graphrag. Unit G4.
2026-06-23 01:29:51 +01:00
Alex 22582e900c feat(graphrag): ingest-time extraction pipeline (entities/relations -> graph)
Per-chunk schema-constrained LLM extraction (gleanings off, one .gen() per chunk) into the GraphStore: model resolves config.graph.extraction_model -> GRAPHRAG_EXTRACTION_MODEL -> instance LLM_NAME; token_usage logged per call (_token_usage_source=graph_extraction, owner-attributed via decoded_token); chunk text fenced as untrusted; robust JSON parse (malformed/errors skip the chunk, never crash); resumable via graph_ingest_progress; max_chunks cap; entity names embedded + merged by normalized_name; chunk_id = doc_id so links match retrieval. Unit G3.
2026-06-23 01:00:21 +01:00
Alex 6687d85085 feat(graphrag): GraphStore — per-source graph tables in the pgvector DB
On-demand graph_nodes/graph_edges/graph_node_chunks/graph_ingest_progress in the same DB as pgvector documents (no Alembic, no cross-DB FK; Python uuids; parameterized SQL). upsert_node (merge by normalized_name, doc_freq, dedup description), add_edge (+degree), link_node_chunk, search_nodes_by_embedding (cosine), bounded get_subgraph, checkpoint, delete_by_source. Embedding dim derived from the configured model (768 fallback). Unit G2.
2026-06-23 00:51:41 +01:00
Alex b4c465e608 feat(graphrag): config.graph sub-model + settings + pgvector availability gate
GraphConfig (extraction_model?, max_chunks?, gleanings=0; extra=forbid) on SourceConfig; settings GRAPHRAG_EXTRACTION_MODEL (None=instance default) + GRAPHRAG_MAX_CHUNKS_FOR_EXTRACTION; graphrag_available() = GRAPHRAG_ENABLED and VECTOR_STORE==pgvector (D27-D32 foundation). Unit G1.
2026-06-23 00:39:27 +01:00
Alex acb7956fe0 Merge pull request #2554 from arc53/rag-wiki
feat: LLM-editable wiki source (F-Wiki v1)
2026-06-23 00:20:24 +01:00
Alex ef44459984 fix: small wiki fixes 2026-06-23 00:19:56 +01:00
Alex 8b099ef1aa fix(wiki): monochrome icon for the New wiki source option
The add-source grid inverts black icons to white (dark:invert), but the wiki used book.svg (fill #6A4DF4), which inverted to lime. Point the wiki ingestor at a new black book-mono.svg so it renders white like the others; book.svg stays purple for its other use (PromptsModal, no invert there).
2026-06-22 23:39:03 +01:00
Alex f809f660f4 fix(wiki): maintain sources.tokens for wikis (sum of page token_counts)
sources.tokens was only set at ingest, so wikis showed no/stale token count on the card (blank wikis showed nothing; converted ones showed the stale original count). rebuild_wiki_directory_structure (called after every wiki mutation) now also sets sources.tokens to the sum of wiki_pages.token_count; blank wiki create sets tokens=0. So the card reflects live wiki content.
2026-06-22 23:33:47 +01:00
Alex 2616ec5d1b fix(wiki): detect wiki sources by config.kind, not type
A converted source keeps its origin type (e.g. crawler) with the wiki-ness in config.kind=wiki (D5/D20), so the UI's type==='wiki' checks treated converted sources as normal (wrong view, convert action still offered). Sources.tsx now keys isWiki + the view branch on config.kind==='wiki' (with a type fallback for blank-created wikis). The paginated endpoint already returns config.
2026-06-22 23:26:12 +01:00
Alex 54773cdf0f feat(wiki): open source view on card click (wiki viewer or normal view)
Clicking a source card opens its view via the existing setDocumentToView handler (wiki -> WikiViewer, nested -> FileTree/ConnectorTree, else -> Chunks). The 3-dots Button and both DropdownMenuContent menus stopPropagation so clicks on the trigger or menu items don't bubble (through the React tree, even though Radix portals them) to the card's onClick. Adds cursor + keyboard/role a11y matching AgentCard.
2026-06-22 23:21:11 +01:00
Alex 8a0f7024e3 fix(wiki): convert reassembles pages from existing chunks (crawler/remote)
convert_source_to_wiki now builds pages from the source's already-ingested vector-store chunks instead of re-parsing storage files, so it works for crawler/remote/connector sources (which have no stored files; file_path empty). Groups by metadata file_path/file_name (URL- and connector-aware; URLs normalized to virtual paths) rather than the raw source. Passes embeddings_key to create_vectorstore in BOTH convert and reembed_wiki_page (fixes a TypeError crash on faiss/elasticsearch). Conservative overlap-trim (min length). Deletes the original chunks after reassembling so retrieval has no duplicate/stale content.
2026-06-22 23:03:52 +01:00
Alex 0a73381457 Merge pull request #2555 from ManishMadan2882/main
Minor UI polish
2026-06-22 22:38:21 +01:00
Manish Madan fb5b5e61a8 Merge branch 'arc53:main' into main 2026-06-23 01:48:24 +05:30
ManishMadan2882 7b75bfd36f (feat:teams) minor polish/fe 2026-06-23 01:47:57 +05:30
Alex 80533f9383 feat(wiki): convert-to-wiki UX, living-wiki naming, surface agent edits
Unit 7 of F-Wiki (D20/D23/D24). Source row action 'Convert to wiki' (owner/editor only) -> confirm naming the cost (lossy/one-way/skipped) -> poll task status -> after-summary; blank source enables inline. Agent wiki writes (create/str_replace/insert/delete/rename) render as a prominent top-level card in the conversation (outside the collapsed tool-calls accordion) with the page path; reads stay normal. 'Living wiki'/agent-editable naming + explainer; 'New wiki' blank-source fast path retained.
2026-06-22 21:05:35 +01:00
Alex 919fe10ab9 feat(wiki): human page editing + provenance stamps
Unit 6 of F-Wiki (D22 + D24). Migration 0024 adds wiki_pages.updated_via; set to 'agent' by WikiTool + convert, 'human' by the edit/seed endpoints (content-hash short-circuit preserves it). WikiViewer gains a markdown editor (Save via PUT with expected_version; 409 reloads latest while keeping the draft; 403 graceful) and a per-page provenance stamp (editor/when/version). Edit gated on write access client-side; backend remains the real authz.
2026-06-22 20:42:37 +01:00
Alex 0c9e0313bd feat(wiki): convert existing source to wiki + human page edit endpoint
Unit 5 of F-Wiki (D20-D23). convert_source_to_wiki task reuses reingest's storage file-load + parser to materialize files->wiki_pages (one page/file), skips/reports non-text, re-embeds per page, and flips kind=wiki + exposure=agentic_tool only when pages were created. POST /wiki/convert (explicit, write-authz; blank source enables inline, fileful enqueues the task; rejects mid-ingest). PUT /wiki/page for human edits (write-authz, optimistic version -> 409, re-embed). PATCH /config preserves kind (kind changes only via convert).
2026-06-22 20:16:18 +01:00
Alex 5110189143 feat(wiki): create-wiki route + read endpoints + minimal read-only UI
Unit 4 of F-Wiki. POST /api/sources/wiki creates a type=wiki/kind=wiki source with no ingest task (optional seed page enqueues reembed). GET /wiki/pages + /wiki/page serve the tree + fresh page content, read-access gated (owner or team grant), path-validated. Frontend: 'Create Wiki' ingestor entry, read-only WikiViewer (FileTree + react-markdown, no raw HTML), sync/reingest hidden for wiki sources.
2026-06-22 18:41:54 +01:00
Alex 98aa8db953 feat(wiki): WikiTool with injection + write authz
Unit 3 of F-Wiki. WikiTool (view/create/str_replace/insert/delete/rename) over wiki_pages: exact-case unique str_replace (no silent multi-replace), optimistic version on edits (WikiPageConflict), reads served fresh from Postgres, untrusted-content fencing on reads, 1MB page cap. Injected via add_wiki_tool only for writable wiki sources (effective_write_owner: owner/team-editor; viewers get nothing), scoped to one source_id. Each mutation enqueues reembed_wiki_page (owner as user, per-page idempotency key) and rebuilds directory_structure. Shared validate_tool_path extracted from MemoryTool.
2026-06-22 18:18:04 +01:00
Alex d68be86244 feat(wiki): reembed_wiki_page durable task
Per-page re-embed (Unit 2 of F-Wiki): targeted delete of the page's old chunks, re-chunk via the source's chunking config, add_chunk with reingest-matching metadata (source=path), set embed_status embedded/failed. Durable + idempotent (key=content_hash), mirroring reingest_source_task. Missing page => purge only.
2026-06-22 17:49:04 +01:00
Alex 8aa0facf4a feat(wiki): wiki_pages table + repository + targeted chunk delete
Migration 0023_wiki_pages (source-scoped; version/content_hash/embed_status; FK source_id->sources ON DELETE CASCADE; UNIQUE(source_id,path) + prefix index). WikiPagesRepository: path-keyed CRUD with content-hash short-circuit, version bump, and move-with-reject. BaseVectorStore.delete_chunks_by_source_path (loop default) + parameterized pgvector override (DELETE WHERE source_id=%s AND metadata->>'source'=%s). Unit 1 of F-Wiki.
2026-06-22 17:40:41 +01:00
Alex a709b7ccdc feat: semantic chunking + pgvector hybrid (BM25+vector) retriever (#2553) 2026-06-22 17:04:13 +01:00
Alex c6a33eca32 Merge pull request #2552 from arc53/rag-per-source-config
feat: per-source RAG configuration (retrieval strategies, chunking, exposure, prescreen)
2026-06-22 14:00:02 +01:00
Alex 057b510adf fix: better error logging 2026-06-22 13:47:09 +01:00
Alex a9068faefa feat: small mixes on mixes sources 2026-06-22 13:13:51 +01:00
Alex babc067aa4 feat: remove old chunk management 2026-06-22 10:40:33 +01:00
Alex f2e6967ec9 feat: redesign advanced settings panel as settings rows with dividers
Group eyebrow headers with short live/re-ingest tags, lighter field labels, hairline dividers between rows, compact right-aligned controls, and hide the single-option Retriever select.
2026-06-21 14:15:09 +01:00
Alex e18886b4f1 fix: rename create-flow section to "Advanced settings" with neutral toggle color 2026-06-21 14:02:44 +01:00