Source access control
---------------------
`active_docs` is client-supplied and reached the retriever unchecked, and the
retriever queries `WHERE source_id = <id>` with no owner predicate — so any
caller could pass any source id to /stream or /api/answer and have another
tenant's documents quoted back, while /api/sources/<id>/search correctly
refused the same id. Gate it through `can_access`, the helper the guarded
endpoints already use, and filter `self.source` down to the authorized set.
Fails closed: no principal, or a check that errors, drops the source.
Three sibling paths had the same gap:
- workflow agent nodes: `AgentNodeConfig.sources` is written verbatim from
client JSON at save time and nothing validated it, so a node could name any
tenant's source. Gate against the workflow owner, so shared workflows keep
reading their owner's sources like shared agents do.
- /api/share: `_resolve_source_pg_id` resolved any id with no ownership
predicate and baked it into the agent the share creates; /api/search then
searched it. Authorize before attaching.
- search_service: re-resolve the ids stored on an agent row instead of
trusting them, so a row written by any future path with the same gap cannot
be read back.
Team grantees previously lost their source's retrieval config: the post-check
read was still owner-scoped, so it missed and fell back to defaults (an
`agentic_tool` source was bulk-prefetched for every grantee). Read unscoped
after `can_access` passes.
Retrieval
---------
`PGVectorStore._ensure_table_exists` created an IVFFlat index on the empty
table it had just created. IVFFlat computes centroids at build time, so those
centroids were random, and combined with the `source_id` post-filter a source
with hundreds of embedded chunks returned zero rows — retrieval reported no
documents, the model answered from memory, and nothing was logged. Stop
creating the index (exact search is correct and fast well past the sizes most
deployments reach); raise `ivfflat.probes` to sqrt(lists) where an index still
exists; and re-run a short indexed search exactly, since post-filtering means
no index setting can guarantee a full result. `graphrag` had the same
empty-table index with no fallback at all.
Also: bound `chunks` to 0-500 on both the request and agent paths (0 still
means "skip retrieval"), let a source's configured `retrieval.chunks` outrank
the request body, and cap ClassicRAG's per-source floor at
max(top_k, n_sources) so attaching sources cannot inflate the result set.
Silent failures
---------------
An empty retrieval was invisible to both the model and the client: the `source`
event was suppressed when the list was empty, so "searched and found nothing"
looked identical to "no source attached", and the prompt said nothing at all.
Emit the event always, and tell the model when a search ran and returned
nothing. A file that parses to nothing now fails ingest with a message naming
the cause instead of storing an embedding of the empty string. `score_threshold`
returns warnings when the active store or retriever cannot honour it.
Prompt structure
----------------
Retrieved documents move from the system prompt into the user turn, with the
injection guard restated next to them: they change every turn (defeating prefix
caching), they are third-party text that should not carry system authority, and
routing them through the query budget makes them truncatable rather than
silently crowding it out. Documents are shed lowest-ranked-first before the
question is touched.
The six chat presets (3 tones x 2 retrieval modes) differed only in their
Answering section; they are now composed from single-source fragments at load
time, not through Jinja inheritance, which would have opened a file-read
surface in the template sandbox and broken the tool-prefetch parser. Per-tool
guidance moves out of the prompt into tool schemas, so it travels with the tool
and cannot render when the tool is absent. A plain-text custom prompt is staged
as a persona value inside the skeleton instead of replacing it wholesale — it
used to silently lose the injection guard, platform block, memory and
attachments, and its braces are now inert.
Other fixes
-----------
- agents/base: an oversized system prompt drove the query budget negative and
dispatched a full-price request with an empty question; raise instead.
- llm/anthropic: migrate off the retired Text Completions API. It flattened
history to first+last message and ignored tools entirely. Adds the missing
Anthropic handler, without which every tool call was silently dropped.
- sources/upload: `sitemap` had no branch, so every sitemap ingest died on a
TypeError; `validate_url` now rejects a falsy URL cleanly.
- workflow nodes: retrieved documents never reached the node agent, so a
classic node with a source and an ordinary prompt answered "I have no
documents" while the run reported completed.
- parser/bulk: copy the metadata dict, or every chunk reports the last chunk's
token_count.
- crawler_loader: carry the page title, or citations render the whole chunk
body as the label.
Workflows ship two expression languages in adjacent fields. Agent
`prompt_template` and end `output_template` are Jinja2, so `{{query}}` is
correct there. State and condition `expression` fields are bare CEL,
where the same string is a parse error.
Nothing told users this. docs/content/Agents/nodes.mdx documented the
pre-CEL Set State node with `{{variable_name}}` examples and Set /
Increment / Append operations that no longer exist — "CEL" appeared
nowhere in the docs. The builder's own placeholder was `input.foo + 1`,
referencing an `input` namespace that is not in workflow state, while the
panel beside it advertised `{{ agent.variable }}`.
There was also no validation anywhere: neither validateWorkflow nor
validate_workflow_structure looked at state nodes at all, and conditions
were only checked for a non-empty string. A workflow containing
`{{query}}` saved and published clean, then aborted on the first message
with a bare caret dump that sanitize_api_error collapsed into "An error
occurred … please try again later" — advice that sends the user round the
same loop, which is what the 2026-08-01 report shows.
Adds validate_cel_expression (compile-only, since state is built at run
time and unresolved names are not knowable when saving) and wires it into
the workflow save path for both node types, plus the half-configured
state operations the engine silently skips. `{{ }}` gets a targeted hint,
raised only after compilation has already failed so valid CEL that
contains braces — `x == "{{y}}"`, or a nested map literal — is not
rejected.
celpy errors are now summarized: undeclared-reference messages embed a
repr of the entire activation, thousands of characters including the
user's own query, and others interpolate the offending state value.
Quoted fragments that are not bare identifiers are redacted, since
config errors now bypass sanitize_api_error to reach the user, and a
shared agent's runner is not its owner.
Docs rewritten with a table of which field takes which syntax, CEL
examples for Set State, and the Condition node section that basics.mdx
has been linking to all along.
/api/models reports `display_provider` when a catalog YAML sets one
(`foundry`, `azure_foundry`, `cloudflare`), and the builder persists that
string as a node's `llm_name`. The engine passed it straight to
LLMCreator, which only knows the names in PROVIDERS_BY_NAME and raises
`No LLM class found for type <label>`.
That fails the node before any LLM call, so the turn ends in ~60ms with
an empty answer and no tokens generated. It hits the *default* path: the
platform-default model's label is stamped into every newly dragged agent
node and validateWorkflow requires an agent node, so a new user's
untouched workflow could not produce a token regardless of what they
typed. Ordinary chat was unaffected because it resolves the provider from
the model registry and never reads the stored name.
Adds `resolve_dispatch_provider`, which prefers a stored name that is a
real dispatch provider, then the registry lookup, then the parent agent.
Nodes already saved with a label are repaired at run time, so no
migration is needed. The api_key now resolves from the normalized name
too — `get_api_key_for_provider` falls back to settings.API_KEY for names
it does not recognize, which would have sent the deployment key to
whatever endpoint the label happened to select.
Replace the sandbox Docling extractor with read_document, backed by the in-process
backend parser (the same one ingestion uses) and offloaded to a dedicated
'parsing' Celery queue so it can run on GPU-capable workers with predictable RAM.
The tool resolves the input ref under the run-scoped gate, enqueues the parse,
and awaits it with a timeout (degrading to an error rather than hanging); the
worker independently re-resolves the artifact through the same gate and never
trusts a raw path. Untrusted files get the upload path's safeguards (extension
whitelist, size cap, sanitized temp file, cleanup). Options: output
(markdown/text/structured/chunks), ocr, pages, engine, max_chars, include_tables,
persist, json_schema. The workflow native-file 'extract' fallback now uses the
same worker path, so document parsing no longer needs the sandbox and works on
every backend.
Also fixes the branch's periodic-task test (the sandbox reaper made it 12) and
points the dev and e2e Celery workers at the parsing queue.
An agent node can list input_documents (short refs like A1, a literal "*" for all
inputs, or state variables holding refs produced by an upstream node). The engine
resolves each through the run-scoped artifact gate and feeds it to the node's LLM
natively when the model supports the format (PDFs included via the provider's
image synthesis), otherwise extracts it to text with Docling. A file_passing
policy (auto/native/extract) and per-node count + per-file byte caps bound cost;
the native decision uses the same supported-types source the provider handler
uses, so a document is never silently dropped. Bytes never enter run state.
Let workflow runs consume and produce documents end to end: bridge uploaded
attachments into run-scoped artifacts so nodes receive the input documents
(with a per-run cap and server-computed size/sha256, and the run row pre-created
so produced artifacts are authorized during the run); emit the run id to the
client and add a builder panel that lists, previews, and downloads a run's
artifacts; and allow attaching documents to a Preview run via the existing
upload flow.
Also fixes issues a compliance workflow surfaced: attachment ownership now keys
on the raw identity instead of a sanitized one (the sanitized form could not be
read back and could collide across users); workflow code nodes read prior state
from a state.json data file instead of templating it into the program, so
untrusted document content can never be interpolated into executed code;
structured node output wrapped in code fences is recovered; and the live
speech-to-text ownership check compares the raw identity.
Stamp the engine's workflow_run_id into a workflow agent node's tool config, so
artifact_generator, code_executor, and document_extractor address artifacts by
the workflow run. An artifact (and its short ref like A1) created by one agent
node is then visible to a later code node and editable by a later agent node in
the same run - enabling an 'agent generates, a code node transforms, a second
agent edits' flow. Workflow agent nodes carry no conversation id, so only the
run scope applies.
Give each produced artifact a short virtual handle - A1, A2, ... - the n-th
artifact in the conversation or workflow run (case-insensitive, not stored). The
tools return it in their results, and the edit, rewrite, and input parameters
accept either the handle or the full id. A handle resolves only within the
caller's own conversation or run, and the resolved id is still checked against
that parent before any read, so it cannot reach another tenant's artifact. This
fixes the model creating a duplicate instead of a new version when asked to
edit an artifact.
Add a code workflow node that runs code in the run-scoped sandbox session and
writes produced files as artifact references into workflow state, passing them
by reference (only id and metadata, never bytes) so downstream nodes and CEL
conditions can branch on them. Add an artifacts.* templating namespace that
resolves those references to metadata via a run-scoped lookup, available to
both the workflow engine and the prompt renderer. Extract the sandbox-to-
artifact persistence into a shared helper reused by the code node and the
code_executor tool.
* feat: add condition node functionality with CEL evaluation
- Introduced ConditionNode to support conditional branching in workflows.
- Implemented CEL evaluation for state updates and condition expressions.
- Updated WorkflowEngine to handle condition nodes and their execution logic.
- Enhanced validation for workflows to ensure condition nodes have at least two outgoing edges and valid expressions.
- Modified frontend components to support new condition node type and its configuration.
- Added necessary types and interfaces for condition cases and state operations.
- Updated requirements to include cel-python for expression evaluation.
* mini-fixes
* feat(workflow): improve UX
---------
Co-authored-by: Alex <a@tushynski.me>
* feat: implement WorkflowAgent and GraphExecutor for workflow management and execution
* refactor: workflow schemas and introduce WorkflowEngine
- Updated schemas in `schemas.py` to include new agent types and configurations.
- Created `WorkflowEngine` class in `workflow_engine.py` to manage workflow execution.
- Enhanced `StreamProcessor` to handle workflow-related data.
- Added new routes and utilities for managing workflows in the user API.
- Implemented validation and serialization functions for workflows.
- Established MongoDB collections and indexes for workflows and related entities.
* refactor: improve WorkflowAgent documentation and update type hints in WorkflowEngine
* feat: workflow builder and managing in frontend
- Added new endpoints for workflows in `endpoints.ts`.
- Implemented `getWorkflow`, `createWorkflow`, and `updateWorkflow` methods in `userService.ts`.
- Introduced new UI components for alerts, buttons, commands, dialogs, multi-select, popovers, and selects.
- Enhanced styling in `index.css` with new theme variables and animations.
- Refactored modal components for better layout and styling.
- Configured TypeScript paths and Vite aliases for cleaner imports.
* feat: add workflow preview component and related state management
- Implemented WorkflowPreview component for displaying workflow execution.
- Created WorkflowPreviewSlice for managing workflow preview state, including queries and execution steps.
- Added WorkflowMiniMap for visual representation of workflow nodes and their statuses.
- Integrated conversation handling with the ability to fetch answers and manage query states.
- Introduced reusable Sheet component for UI overlays.
- Updated Redux store to include workflowPreview reducer.
* feat: enhance workflow execution details and state management in WorkflowEngine and WorkflowPreview
* feat: enhance workflow components with improved UI and functionality
- Updated WorkflowPreview to allow text truncation for better display of long names.
- Enhanced BaseNode with connectable handles and improved styling for better visibility.
- Added MobileBlocker component to inform users about desktop requirements for the Workflow Builder.
- Introduced PromptTextArea component for improved variable insertion and search functionality, including upstream variable extraction and context addition.
* feat(workflow): add owner validation and graph version support
* fix: ruff lint
---------
Co-authored-by: Alex <a@tushynski.me>
Alex